DOS ain't dead - BSD alternatives to coreutils

lmemsm

19.01.2014, 17:32

BSD alternatives to coreutils (Developers)

Just wondering if anyone's considered using some of the BSD alternatives to common utilities instead of GNU coreutils and other similar GNU tools. From my personal experiences so far, some of the GNU developers don't seem too helpful about supporting DOS or other ports or too eager to accept patches for them. They seem more concerned with Linux/POSIX systems. Was thinking that some of the BSD alternatives are typically less bloated and complex than the GNU varieties and might be easier to maintain.

There's a project called Obase that ports some of the BSD utilities to Linux. There are also several BSD licensed Minix utilities that are used by embedded systems like Elks. I've been working on porting some of them and some of the NetBSD and FreeBSD utilities to Windows. Don't think it would be that much harder to add DOS support. Currently have utilities like find and xargs working in DOS. I'm also using pkgconf instead of GNU pkg-config with djgpp.

I think one can find BSD alternatives for most of the coreutils and other basic utilities. Some of the other tools I've run across BSD alternatives to are gettext/libintl and gzip. I believe libarchive (which includes a BSD version of tar) is already available and working on several operating systems as well.

Was just wondering if anyone (other than me) would be interested in this sort of thing. I'd be interested in discussing the topic with others.

RayeR

CZ,
19.01.2014, 17:40

@ lmemsm

BSD alternatives to coreutils

Post reply

Until DJGPP ports will work I don't need to search for alternative...

---
DOS gives me freedom to unlimited HW access.

Doug

19.01.2014, 21:42

@ lmemsm

BSD alternatives to coreutils

Post reply

> Was just wondering if anyone (other than me) would be interested in this
> sort of thing.

Sure -- i'm always interested in new stuff for my fave o/s! :) Especially
if more DOS-like than the GNU utils (which tend to give me a headache).
However, i have minimal programming skills, so i doubt that i could
contribute much to development beyond testing.

> I've been working on porting some of them and some of the NetBSD and
> FreeBSD utilities to Windows. Don't think it would be that much harder
> to add DOS support.

Many Win9x command-line executables (and even some gui) will work in real
DOS with Japheth's HX runtime. Undoubtedly, pure DOS ports would be a
more-ideal solution (?), but HX-friendly ones could be a viable
alternative.

- Doug B.

lmemsm

20.01.2014, 13:24

@ Doug

BSD alternatives to coreutils

Post reply

My preference would be pure DOS ports. Have about a dozen utilities building on DOS so far. Still several I want to port to Windows and DOS. Also need to do more testing and debugging to make sure they're working properly. I don't like releasing anything until I feel it's fairly stable.

Was working on it yesterday. On Windows, I'm using regex and fnmatch code ported from musl C library. Tried compiling that code with djgpp yesterday to see if it would work. Had issues with not finding wide character versions of routines (like towupper, iswalnum, etc.). Left me wondering if there was UTF-8 support in the regex routines and the gnu utilities built with djgpp or whether it just supported the current code page. Does anyone know more on that?

I'm currently trying to decide whether to use my own utf-8 and wide character routines to add better utf-8 support or just stick with code page support only. Would be curious how others are handling internationalization issues like these.

bocke

20.01.2014, 15:04

@ lmemsm

BSD alternatives to coreutils

Post reply

> Just wondering if anyone's considered using some of the BSD alternatives to
> common utilities instead of GNU coreutils and other similar GNU tools.
> From my personal experiences so far, some of the GNU developers don't seem
> too helpful about supporting DOS or other ports or too eager to accept
> patches for them. They seem more concerned with Linux/POSIX systems. Was
> thinking that some of the BSD alternatives are typically less bloated and
> complex than the GNU varieties and might be easier to maintain.
>

I don't think BSD guys care any more. :) And utilites are usually not much simpler and still require POSIX compatibility.

> There's a project called Obase that ports some of the BSD utilities to
> Linux. There are also several BSD licensed Minix utilities that are used
> by embedded systems like Elks. I've been working on porting some of them
> and some of the NetBSD and FreeBSD utilities to Windows. Don't think it
> would be that much harder to add DOS support. Currently have utilities
> like find and xargs working in DOS. I'm also using pkgconf instead of GNU
> pkg-config with djgpp.
>

Well find and xargs are pretty generic utilties. Pkgconf is meant to be used on non-GNU systems anyway, so Windows support was probably pretty much planed in.

Elks utilities are hodge-podge of different utilties from all over the 80's/90's *nix eco-system. Some of it is a "public domain" code posted to Minix newsgroup in 80s, some of it comes from early x86 based BSDs and NET2, some of it is early GNU stuff. Pretty obsolete stuff.

> I think one can find BSD alternatives for most of the coreutils and other
> basic utilities. Some of the other tools I've run across BSD alternatives
> to are gettext/libintl and gzip. I believe libarchive (which includes a
> BSD version of tar) is already available and working on several operating
> systems as well.
>
> Was just wondering if anyone (other than me) would be interested in this
> sort of thing. I'd be interested in discussing the topic with others.

Well, I had a phase when I was interested in porting *nix stuff to DOS. But it proved too easy with DJGPP. :) Honestly most of the users of these forums are technical enough to do that themselves. So I quit messing around and wasting time with something nobody would use.

Well, anyway I doubt BSD utilties are that much lighter. But if you could port the base BSD userspace go for it. If, on the other side, you mean to take a bit from here, a bit from there, making a hodge-podge of "alotasources" that's not worth doing.

About unicode. No, it's not supported. For full unicode support you would have to use graphic mode (like in blocek). Text mode is limited and can have only one subset active at the time.

bocke 20.01.2014, 16:41 @ bocke	BSD alternatives to coreutils Post reply
	On the other tought, don't let me discourage you from your personal projects. :) Have fun. ;)

lmemsm

21.01.2014, 13:59

@ bocke

BSD alternatives to coreutils

Post reply

> I don't think BSD guys care any more. :) And utilites are usually not much
> simpler and still require POSIX compatibility.

I've been looking at the POSIX specifications versus what GNU supports and POSIX compatibility appears to be much less complicated than the extra features added by typical GNU tools.

> Well find and xargs are pretty generic utilties. Pkgconf is meant to be
> used on non-GNU systems anyway, so Windows support was probably pretty much
> planed in.

pkgconf is used on Gentoo and some other Linux systems to replace pkg-config. Pkg-config has a circular dependency with glib which makes it a nuisance to build. I'm also currently working on building a gtk+ free systems (which means no glib). Windows support wasn't planned from the beginning as far as I can tell. I sent the project the patches to get that support working properly on Windows using MinGW.

> Elks utilities are hodge-podge of different utilties from all over the
> 80's/90's *nix eco-system. Some of it is a "public domain" code posted to
> Minix newsgroup in 80s, some of it comes from early x86 based BSDs and
> NET2, some of it is early GNU stuff. Pretty obsolete stuff.

For pretty obsolete stuff, some of it seems fairly portable and has enough POSIX compatibility to be used as decent substitutes in many cases.

> Well, I had a phase when I was interested in porting *nix stuff to DOS. But
> it proved too easy with DJGPP. :) Honestly most of the users of these
> forums are technical enough to do that themselves. So I quit messing around
> and wasting time with something nobody would use.

I'm trying to get to the point where I have all the base utilities and libraries I need to build the Open Source applications I want on a DOS system. That means being able to run tools to build Open Source projects like gnu autotools. Another main goal of mine is to be able to build everything in a repeatable manner. Right now, my DOS development system is a collection of some older djgpp programs and utilities, some of Georg Potthast's build environment and some utilities I built from source (like make 4.0). I'm also using a portable package management system similar to Slackware's slackbuilds. I'm attempting to consolidate so that I can build the entire environment (everything I need to build Open Source applications) from source and build scripts. My goal isn't to make something useful for someone else. I'm working on something I personally need in order to port applications I use more easily to DOS. However, if anyone else is interested in this sort of thing, it would be nice to be able to discuss design ideas and share code and results.

> About unicode. No, it's not supported. For full unicode support you would
> have to use graphic mode (like in blocek). Text mode is limited and can
> have only one subset active at the time.

Was planning on using FLTK and possibly a few other choice GUI libraries. Was beginning to think most Open Source programs wouldn't port to DOS when Georg got me started looking into nano-x and FLTK.

marcov

21.01.2014, 15:30

@ lmemsm

BSD alternatives to coreutils

Post reply

> > I don't think BSD guys care any more. :) And utilites are usually not
> much
> > simpler and still require POSIX compatibility.
>
> I've been looking at the POSIX specifications versus what GNU supports and
> POSIX compatibility appears to be much less complicated than the extra
> features added by typical GNU tools.

In recent years (say after 2005), more and more GNUisms have been implemented in the BSD tools.

My guess would be that in the (early-mid) nineties, people prefered the BSD tools for dos porting because they had simpler buildsystem (just make) than GNU with all its autoconf/configure stuff.

> > Well find and xargs are pretty generic utilties.

Both require decent pipe implementations.

> I'm trying to get to the point where I have all the base utilities and
> libraries I need to build the Open Source applications I want on a DOS
> system.

_on_ or _for_ ? Many issues can be avoided by crosscompiling I assume.

P.s. afaik the last non msys (pure mingw) coreutils is also already quite old, 2005 or so.

Oso2k 21.01.2014, 23:01 @ marcov	BSD alternatives to coreutils Post reply
	The suckless project has sbase. The rob landley had toybox (busybox replacement) and there was also beastiebox and some other small package at one time for use as a busybox replacement on bsd.

Ibidem

22.01.2014, 04:05

@ Oso2k

BSD alternatives to coreutils

Post reply

> The suckless project has sbase. The rob landley had toybox (busybox
> replacement) and there was also beastiebox and some other small package at
> one time for use as a busybox replacement on bsd.

Though I do some work on toybox, I'd strongly recommend not starting there for DOS related projects. Toybox expects a SUS4 (POSIX2008/XSI option) development environment, and has several places where it expects Linux-specific features (well, BSD does some things similarly, but...).
Also, Rob Landley has specifically stated that he will not accept ports to various old platforms.

If any project in this area would be willing to consider a DOS port, I'd expect it to be Busybox; they have some support for newlib+libgloss already.

Also, I suggest that you not bother with sbase if you aim to get scripts working easily; it's a minimalist setup that does not support many options.

bocke 22.01.2014, 21:26 @ marcov	BSD alternatives to coreutils Post reply
	> > > Well find and xargs are pretty generic utilties. > > Both require decent pipe implementations. > True. Also enlarged command environment. But doesn't seem to be too much of a problem with DJGPP.

georgpotthast Germany, 22.01.2014, 08:18 @ lmemsm	BSD alternatives to coreutils Post reply
	>That means being able to run tools to build Open Source projects like gnu autotools. As you are probably aware djgpp has autotools. However, these seem to be very old versions which do not work well (i.e.not) with current software packages. If you could upgrade these to the new autotools versions djgpp developers would appreciate it.

lmemsm

22.01.2014, 14:33

@ georgpotthast

BSD alternatives to coreutils

Post reply

marcov wrote:
>_on_ or _for_ ? Many issues can be avoided by crosscompiling I assume

As I mentioned, my preference would be for native DOS (although definitely with LFN support). One of my project goals is cross-platform portability. If I need to set up a djgpp build environment on Linux in order to build the Open Source applications I want, I might as well just build those applications for Linux and use them there. It's certainly less effort and less porting.

Toybox and Busybox are very similar to what I have in mind only I'm not sure if it really needs to be one monolithic program. Busybox has been ported to Windows (more recently than coreutils), but both Busybox and Toybox seem to be very POSIX oriented (as in, they expect standard POSIX functions as part of the C library). I agree with Ibidem's assessments. Based on some of Rob Landley's comments on the musl mailing list, I got the idea it wasn't a very good base for cross-platform work. I do like the goals of the project though. Sbase looks a lot more promising from a portability standpoint but, as Ibidem mentioned, it's a very minimal implementation without full POSIX compatibility support. I didn't want to start completely from scratch, so I'm currently considering starting with bits and pieces of the Minix utilities and some of the Obase/NetBSD/FreeBSD utilities along with some base libraries to factor out common functionality. Would like UTF-8 support so utilities like grep or wc can work on UTF-8 files. I'm using some of the data from files at http://www.unicode.org/Public/ and modeling parts of the utf-8 library on something similar to Plan 9's Rune library. Have functions like towupper, towlower and iswalnum working for utf-8/Unicode (32 bit), but the library is still a work in progress.

Georg wrote:
>As you are probably aware djgpp has autotools. However, these seem to be very old versions which do not work well (i.e.not) with current software packages.

I've been using GNU autotools with djgpp to some extent, but seems to fail with more projects than it succeeds. Thought perhaps I was missing something and that it worked better. Anyone else having good luck with it?

It would definitely be nice to have GNU autotools working reliably so one doesn't have to keep creating/editing makefiles every time a program has updates. The CoApps project is attempting to go from GNU autotools scripts to native Visual C++ build files so they don't have to continually create build files when Open Source projects update.

With djgpp, at first, I couldn't get past some of the configure files trying to find programs like sed and grep. I set the SHELL environment variable to bash and that got me farther. Now some of the configure scripts are failing when they run utilities like sed, ln, cp, mv. I think it's an issue with the file names using Unix style slashes rather than backslashes. On Windows, some C functions (although not all) work fine with slashes or backslashes in file names. Would like to add that type of support to the base utilities I'm working with. Hoping that will get things a little farther.

It would be nice to update some of the versions of the autotools programs, but it's certainly going to be a lot of work to update the entire autotools system. I have m4, autoconf, automake and make building natively on Windows. Haven't been able to test or debug the ports because you can't mix msys utilities with native ones. The paths get all messed up when you try. The djgpp environment, however, is completely native. So, if I can get some of these tools building and ported for DOS, might be easier to debug things there. I also did some experimenting with building Perl and micro-Perl on Windows, but getting a later version of Perl to work does not look like an easy goal for non-POSIX compatible systems. I do like the micro-Perl concept, but it's not enough to run GNU autotools. One needs several Perl scripts in the proper locations for GNU autotools to work. Rather than a complete update of GNU autotools and related utilities, I'd settle for just enough support to be able to build programs in a reliable fashion with minimal intervention from the developer. (Still not a small project though.)

I know where source code is for the djgpp programs/utilities, where Georg's Nano-x/FLTK programs are and a few other DOS code repositories. If there are any other good locations to find ports of Open Source programs and the patches to get them working on DOS, please let me know. I'd be very interested in looking at some of the patches needed for DOS. Thanks. On a more positive note, I am getting a little farther at getting some Open Source programs to build using my automated package management/build system and my current djgpp environment. It's still buggy, but I got Emilia Pinball building and working well enough to play pinball.

marcov 23.01.2014, 21:33 @ georgpotthast	BSD alternatives to coreutils Post reply
	> >That means being able to run tools to build Open Source projects like gnu > autotools. > > As you are probably aware djgpp has autotools. I was just trying to provide some historical perspective. As most will know, my codebases do not require gcc, let alone autotools ;_)

bocke

22.01.2014, 21:45

@ lmemsm

BSD alternatives to coreutils

Post reply

> I'm trying to get to the point where I have all the base utilities and
> libraries I need to build the Open Source applications I want on a DOS
> system. That means being able to run tools to build Open Source projects
> like gnu autotools. Another main goal of mine is to be able to build
> everything in a repeatable manner. Right now, my DOS development system is
> a collection of some older djgpp programs and utilities, some of Georg
> Potthast's build environment and some utilities I built from source (like
> make 4.0). I'm also using a portable package management system similar to
> Slackware's slackbuilds. I'm attempting to consolidate so that I can build
> the entire environment (everything I need to build Open Source
> applications) from source and build scripts. My goal isn't to make
> something useful for someone else. I'm working on something I personally
> need in order to port applications I use more easily to DOS. However, if
> anyone else is interested in this sort of thing, it would be nice to be
> able to discuss design ideas and share code and results.
>

So basically you want to create a POSIX compatibillity layer on top of the DOS? DJGPP already does a big part of that. Having in mind the OS limitations, they did pretty good job with it.

To make a real POSIX compatible environment you would need to write a 32-bit multiprocess (and multithreading), networked and multi-user DOS multitasker extension. But problem is: when you load this multitasking core, you don't run DOS anymore. This core would have its own APIs and DOS software won't be "native" anymore. You 'd probably run them in the emulated DOS VMs like in Windows. It's basically like writing a new operating system. :)

But you would most likely be able to exit it and switch to "pure" DOS if needed.

lmemsm

23.01.2014, 23:25

@ bocke

BSD alternatives to coreutils

Post reply

> So basically you want to create a POSIX compatibillity layer on top of the
> DOS?

Not really what I'm working towards. If I wanted that, there's a musl port to Windows which will give a POSIX compatibility layer. I could just use it. From what I've read about it, it may be much better than the Cygwin alternative.

I think Georg, Oso2K and Ibidem got the idea of what I'm trying to accomplish. So, if anyone's interested in discussing it further, feel free to e-mail me. Was just curious if anyone was already working along those lines or was interested in projects of that nature.

Oso2k

24.01.2014, 01:28

@ lmemsm

BSD alternatives to coreutils

Post reply

bocke

14.02.2014, 02:59

@ lmemsm

BSD alternatives to coreutils

Post reply

> > So basically you want to create a POSIX compatibillity layer on top of
> the
> > DOS?
>
> Not really what I'm working towards. If I wanted that, there's a musl port
> to Windows which will give a POSIX compatibility layer. I could just use
> it. From what I've read about it, it may be much better than the Cygwin
> alternative.
>
> I think Georg, Oso2K and Ibidem got the idea of what I'm trying to
> accomplish. So, if anyone's interested in discussing it further, feel free
> to e-mail me. Was just curious if anyone was already working along those
> lines or was interested in projects of that nature.

I'm sorry. I don't get you. I tought you meant DOS, but after some tought I am not sure if you meant win32 command line or a native DOS. Also I don't get what scope do you expect.

If you only want to port some Unix utilites, there are already working and functioning ports.

Win32: Microsoft's "own" SUA/Interix, Cygwin, Uwin, Gnuwin32, Unixutils and probably others. MinGW also includes some utilites based on an older version of Cygwin they call msys.

DOS: DJGPP has a nice collection of GNU utilites. FreeDOS server (and mirrors) also contain the older GNUish project. But there were also other collections. You'll find them if you search ftp servers for the mirrors of the dos shareware archives (unfortunately rapidly dissapearing). Some names are: berk, danix, dantools, diskutil, dosix, pcunix, picnix, unix4dos, unixlike, uutil, uxutils, etc.

Anyway DOS tools still work with 32-bit Windows. But don't expect them to work with 64-bit. Actually it's good they don't because people will stop calling Win NT+ command line "a DOS". I think it's worth it. :)