DOS ain't dead

I promised early june to generate a snapshot for 16-bit FPC. Due to holidays and some problems (the default way of building generated non-functional binaries) it was delayed, but Nicolay solved the problem last weekend.

I experimented a bit, and it seems that this is a working win32-> i8086-msdos crosscompiler for tiny and small memory models. Officially, current FPC builds don't work on win9x anymore, so you might need WinXP+.

http://www.stack.nl/~marcov/fpc-i8086-msdos-r25379.zip (4MB)

The demo/ directory contains some precompiled examples (tetris, samegame and hello world). The tetris was compiled with tiny mm. I tested them both in Win XP cmd.exe and dosbox.

There is a readme.txt which are my cleaned notes from building the snapshot, but basically:

- check dofpc16.bat and correct paths/drives.
- check bin/pp16.bat and bin/pp16t.bat
- put dofpc16.bat somewhere central (e.g. in the %PATH%)
- run dofpc16.bat
- compile your apps with pp16 <file> or pp16t <file>

No need to install watcom and/or nasm they are included.

It is certainly nice but when will we have the GO32V2 --> DOS16 crosscompiler?

---
DOS-u-akbar!

> It is certainly nice but when will we have the GO32V2 --> DOS16
> crosscompiler?

I've no concrete planning currently. I might do an attempt if I find some spare time, but since I haven't really used 2.6.x and 2.7.x for much go32v2 work yet, I may run into problems. Like dos watcom tools not running on win64 ;)

> > It is certainly nice but when will we have the GO32V2 --> DOS16
> > crosscompiler?
>
> I've no concrete planning currently. I might do an attempt if I find some
> spare time, but since I haven't really used 2.6.x and 2.7.x for much go32v2
> work yet, I may run into problems. Like dos watcom tools not running on
> win64 ;)

I tried to do something similar for go32v2 but failed.

The trouble is that wlib bombs out with

[7.850] Using util d:\pp16\binw\wlib.exe
Specified COMMAND search directory bad
Too many parameters
Too many parameters
Too many parameters
Microsoft(R) Windows DOS
(C)Copyright Microsoft Corp 1990-2001.

D>exit

I tried shortening paths (not in zip) but that didn't help

Nasm sometimes also exits (run too quickly in succession, memory problems?) Testing was done under Windows XP.

I uploaded the snapshot anyway for more knowledgable people.

http://www.stack.nl/~marcov/fpc-i8086-msdos-go32v2hosted.zip

Note that the pp16\dopp16.bat file sets some paths for compiling, and the
pp16\bin\i386-go32v2\pp16.bat should also be checked.

After that it should be running pp16\dopp16.bat, changing to \pp16\demo and running

pp16 fpctris

should do the trick.

> I may run into problems. Like dos watcom tools not running on win64 ;)

Note that I've not studied your buildfaq nor tried rebuilding FPC myself. Also, the Win32 ("XP") build apparently does not run under HX, no surprise. I guess building with older FPC that did still work (2.2.x?) is out of the question?

> I tried to do something similar for go32v2 but failed.
>
> The trouble is that wlib bombs out with ... [snip]

There are apparently three distinct types of .EXE in there: FPC/GO32V2/COFF (nasm and most everything), Watcom/LE (wlib,wlink), and Win32/PE (make3, not needed?).

These do not by default play well together. The problem is probably passing long cmdlines to each other, e.g. GO32V2-only "!PROXY" (via direct memory address?), which Watcom doesn't support.

Well, and the second problem is incompatible extenders (although GO32V2 is really just pure DPMI only, no DOS extensions). You can't really mix Watcom and DJGPP .EXEs without crashes unless you use "HDPMI32 -r" first or similar. (Maybe Causeway or D3X would play well too, based upon limited experience, but others like DOS/4GW and WDOSX? Not so much.)

Unlike DJGPP, Watcom output .EXEs don't by default support certain features: passing long cmdlines (response files or env. vars) or LFN support. The OpenWatcom compiler and associated select essential tools themselves do manually support passing long cmdlines (see WLIB) via '@env_var' or '@cmd_file'. You can use wmake to work around cmdline limits for some of those select tools (via env_var) if you prefix the tool with '*'. E.g., quoting from NASM's Mkfiles/openwcom.mak (which did build in pure DOS, last I checked, but only with this feature) :

CC = *wcl386

And yes, this means that you can build NASM via OpenWatcom, but like I said, that won't help you here. On Linux, allegedly (I've never tried, but so I've heard) you can bootstrap all of the tools with GCC, so maybe this problem could be indirectly solved by building WLIB and WLINK with DJGPP.

Of course, an easier solution might just involve the compiler driver writing an explicit .BAT file of what individual (non-mixed .EXE !) steps are needed to build the entire project, esp. if it uses response files only (no !PROXY via direct memory access). Not sure of what can be done already, I'm only vaguely aware of -sh and -st and similar switches.

> I tried shortening paths (not in zip) but that didn't help

%PATH% can only be approx. 128 bytes long. Actually, for DOS/4GW .EXEs (whose stub call external DOS4GW.EXE or similar extender), it might be even lower, like 126 or 122.

> Nasm sometimes also exits (run too quickly in succession, memory problems?)
> Testing was done under Windows XP.

I once encountered some oddball errors trying to run (can't remember) either DOS NASM within subshell of Win32 program or maybe vice versa. Though that's probably not an issue here.

As much as NTVDM (and v86 mode) were successful, MS just didn't keep it up, so there are some known major bugs, which can cause problems, even in (best environment) WinXP. (I don't have barely any access to XP anymore.) E.g. DOS/4GW tools sometimes hung the console for me when run under XP, sadly. (DJGPP tools work better overall, due to CWS' hacks.) Though this can't be that big a shock, even you previously said GO32V2 was only for DOS and didn't care about any known regressions under NTVDM (and such GO32V2 binaries don't run at all under DOSEMU, last I checked, dunno why).

> I uploaded the snapshot anyway for more knowledgable people.
>
> http://www.stack.nl/~marcov/fpc-i8086-msdos-go32v2hosted.zip
>
> Note that the pp16\dopp16.bat file sets some paths for compiling, and the
> pp16\bin\i386-go32v2\pp16.bat should also be checked.
>
> After that it should be running pp16\dopp16.bat, changing to \pp16\demo
> and running
>
> pp16 fpctris
>
> should do the trick.

Your .BAT files aren't DOS-shell friendly anyways, they use non-portable Win32-isms:


REM pp16.bat
REM (this file doesn't have any line ending, last byte is '*')
REM ppc8086 -XX -CX -XP %*
REM normal DOS shells don't support %* (except maybe 4DOS)
ppc8086 -XX -CX -XP %1 %2 %3 %4 %5 %6 %7 %8 %9
REM if you want more than nine, you'll have to "shift" and
REM manually handle saving the parameters for later use



REM dopp16.bat
REM (thankfully does end in CR+LF, most DOS shells don't accept otherwise)
@echo off

REM SET MYDRIVE=d
if "%MYDRIVE%"=="" set MYDRIVE=d

SET WATCOM=%MYDRIVE%:\pp16
Rem we save oldpath only once, so we can always revert to it.

REM if "%OLDPATH%" neq "" goto :nosave
if not "%OLDPATH%"=="" goto nosave
set OLDPATH=%PATH%
:nosave
REM SET PATH=%OLDPATH%
PATH %OLDPATH%
REM ... but then you just overwrite it? oops ...
REM ... LFNs assumed? ...
PATH %MYDRIVE%:\pp16\bin\i386-go32v2;%MYDRIVE%:\pp16\binw;%PATH%
REM ... expect user to manually restore OLDPATH later?


> > I may run into problems. Like dos watcom tools not running on win64 ;)
>
> Note that I've not studied your buildfaq nor tried rebuilding FPC myself.
> Also, the Win32 ("XP") build apparently does not run under HX, no surprise.
> I guess building with older FPC that did still work (2.2.x?) is out of the
> question?

Yes. But note that I have a go32v2-i8086 crosscompiler. It just fails to compile the 8086 RTL.

> > The trouble is that wlib bombs out with ... [snip]
>
> There are apparently three distinct types of .EXE in there:
> FPC/GO32V2/COFF (nasm and most everything), Watcom/LE (wlib,wlink), and
> Win32/PE (make3, not needed?).

Yes, make3 is unused. I tried to use it while compiling the RTL using the go32 crosscompiler. I solved that by using a win32-crosscompiled RTL.

> These do not by default play well together. The problem is probably passing
> long cmdlines to each other, e.g. GO32V2-only "!PROXY" (via direct memory
> address?), which Watcom doesn't support.

go32v2 or DJGPP libc?

Anyway, your problem descriptions are more or less what I expected. There is some ! and @ in go32v2 I guess, that will probably have to be ported to call the i8086 backend files.

Since afaik nobody is working on a binwriter (internal assembler+archiver/librarian) yet, we'll probably have to do with these suboptimal solutions a while longer. (though maybe it is not as bad under pure dos as under WinXP. IIRC NTs are noticably slower to start dos binaries. It might be worthwhile testing under dosbox even)

Internal linkers are considerably more complicated, and are only secondary (wrt speed) to the internal assembler/linker. This goes doubly for Dos, since the binaries won't be big, but smartlinking is paramount.

An alternate avenue might be using some more efficient form of smartlinking supported by the nasm-watcom combination. (so that you don't have an nasm invocation per symbol)

> You can use wmake to work around cmdline limits
> for some of those select tools (via env_var) if you prefix the tool with
> '*'. E.g., quoting from NASM's Mkfiles/openwcom.mak (which did build in
> pure DOS, last I checked, but only with this feature) :
>
> CC = *wcl386

Not really an option. That only brings in more alien tools, and the idea is too reduce them in time, not inflate them. Effort can be better spent eliminating nasm-wlink altogether.

> Of course, an easier solution might just involve the compiler driver

(I don't know what you mean with compiler driver. I know GCC is spread out over more binaries, but FPC isn't. fpc.exe could be seen as a driver, but it is more something for make and to have multiple compiler versions and architectures (crosscompilers) in the path. It doesn't contain real functionality (like relating to compiling/assembling/linking))

> writing an explicit .BAT file of what individual (non-mixed .EXE !) steps
> are needed to build the entire project, esp. if it uses response files only
> (no !PROXY via direct memory access). Not sure of what can be done already,
> I'm only vaguely aware of -sh and -st and similar switches.

Yes, in the compiler there used to be support for that (-al/-s) before most targets got the binwriter. (-al/-s<x> etc). Worth a try. Still doesn't solve wlib though (since the cmdline would be too long anyway). Will have to look it up, it is over 10 years ago that I used such kludges. (I assume the go32v2 linker still does, but that is probably pretty invariant and hidden)

> > I tried shortening paths (not in zip) but that didn't help
>
> %PATH% can only be approx. 128 bytes long.

(I thought LFN api under windows allowed 255 char limits? Maybe the lfn driver under pure dos too?)

> > Nasm sometimes also exits (run too quickly in succession, memory
> problems?)
> > Testing was done under Windows XP.
>
> I once encountered some oddball errors trying to run (can't remember)
> either DOS NASM within subshell of Win32 program or maybe vice versa.
> Though that's probably not an issue here.

That's pretty why I posted the whole lot here, hoping sb could try to compile in pure dos.

> As much as NTVDM (and v86 mode) were successful, MS just didn't keep it up,
> so there are some known major bugs, which can cause problems, even in (best
> environment) WinXP.

Well. Under WinXP, one can use the win32 variant. It is just that I didn't have anything else to prepare the go32v2 snapshot.

> Your .BAT files aren't DOS-shell friendly anyways, they use non-portable
> Win32-isms:

They are more meant as examples and reflect my local build systems. Feel free to customize;-0

No Rugxulo, problem is elsewhere.

The compiler does not work because it uses the EXEC function (from DOS unit) to call WLINK.EXE.

But the Exec function is broken and does not correctly pass commandline string from FreePascal into DOS.
There is simply again present the bug 21502 (http://bugs.freepascal.org/view.php?id=21502)

It was corrected in some snapshots but some time ago was merged some branch versions into trunk and apparently something got wrong and in this crosscompiler the wrong version is present.

---
DOS-u-akbar!

> > These do not by default play well together. The problem is probably
> passing
> > long cmdlines to each other, e.g. GO32V2-only "!PROXY" (via direct
> memory
> > address?), which Watcom doesn't support.
>
> go32v2 or DJGPP libc?

Both, most likely, AFAIK, but I'm not familiar with the details (how/when).

> Anyway, your problem descriptions are more or less what I expected. There
> is some ! and @ in go32v2 I guess, that will probably have to be ported to
> call the i8086 backend files.

The problem is the bad assumption that all DOS tools play well together and use the same workarounds. Unfortunately, they don't. Again, it may be possible to cross-compile WLIB and WLINK with DJGPP, that would definitely work better (without manual kludges), but honestly (again) such a thing may not be supported. You know, if they didn't explicitly test it, it probably doesn't work, probably assumes too many *nix things and/or hardcodes #ifdef __unix or #ifdef MSDOS or such (educated guess).

Maybe Laaca is right, it's some Exec bug, but that doesn't mean there aren't other issues that have to be worked around here.

> (though maybe it is not as bad under pure dos as under WinXP.

Like I said, I don't have XP anymore. My two P4 machines had failures, and I can only (barely) maybe test it at my aunts' house. But even then, like I said, XP is no perfect solution, as even its "mostly" working environment has bugs with DOS/4GW (OpenWatcom).

> IIRC NTs are noticably slower to start dos
> binaries. It might be worthwhile testing under dosbox even)

DOSBox is almost entirely out of the question due to various limitations (and "only for games" bugs). DOSEMU too, as mentioned, any FPC/GO32V2 binary just crashes the VM there.

> Internal linkers are considerably more complicated, and are only secondary
> (wrt speed) to the internal assembler/linker. This goes doubly for Dos,
> since the binaries won't be big, but smartlinking is paramount.

I assume you mean smartlinking is 1000x more crucial for 16-bit than otherwise. I don't think it's very important (functionally speaking) for 32-bit DOS as there is no hardcoded .EXE size limit nor total RAM limit.

> An alternate avenue might be using some more efficient form of smartlinking
> supported by the nasm-watcom combination. (so that you don't have an nasm
> invocation per symbol)

It's not bad how it's done now, just tedious. The main problem is Watcom tools not supporting LFNs, the cmdline (120-ish bytes) limit, and apparently not supporting "*.o" wildcards either (but response files are okay).

> Not really an option. That only brings in more alien tools, and the idea is
> too reduce them in time, not inflate them. Effort can be better spent
> eliminating nasm-wlink altogether.

The point was that it will figure out when the cmdline is too long for you, if you're too lazy to do it manually. That's the real problem, you have to not go over the limit. (I know Windows supports like 8000 bytes, but c'mon, you can do without that much.)

> > Of course, an easier solution might just involve the compiler driver
>
> (I don't know what you mean with compiler driver. I know GCC is spread out
> over more binaries, but FPC isn't. fpc.exe could be seen as a driver ...

Yes, I meant the handling of all these stages and needs to be tweaked slightly for DOS limits. Or else use all tools that support the same workarounds (e.g. DJGPP-built, no OpenWatcom-built stuff, even if it is pieces from OpenWatcom itself!).

P.S. Before I forget, maybe it is easier to build JWlink / JWlib with DJGPP than otherwise. At least that is separate and doesn't need the "full" Watcom sources (and weird cryptic makefiles) just to build those two tools.

> > writing an explicit .BAT file of what individual (non-mixed .EXE !)
> steps
> > are needed to build the entire project, esp. if it uses response files
> only
> > (no !PROXY via direct memory access). Not sure of what can be done
> already,
> > I'm only vaguely aware of -sh and -st and similar switches.
>
> Yes, in the compiler there used to be support for that (-al/-s) before most
> targets got the binwriter. (-al/-s<x> etc). Worth a try. Still doesn't
> solve wlib though (since the cmdline would be too long anyway). Will have
> to look it up, it is over 10 years ago that I used such kludges. (I assume
> the go32v2 linker still does, but that is probably pretty invariant and
> hidden)

Wildcards and LFNs and long cmdlines (120+) aren't supported in wlib / wlink for DOS, but response files work fine.

> > > I tried shortening paths (not in zip) but that didn't help
> >
> > %PATH% can only be approx. 128 bytes long.
>
> (I thought LFN api under windows allowed 255 char limits? Maybe the lfn
> driver under pure dos too?)

Not sure, but IIRC, only Win9x (not XP) had such a workaround (also supported by 4DOS??) where if the cmdline count is 128 (80h) or higher, it reads from %CMDLINE% env_var for the "full" cmdline entered. Maybe FreeCOM (FreeDOS shell) supports it too, dunno, but most apps don't support it either. Though DJGPP should support it too, but I'm not sure, never delved too deeply into it.

> That's pretty why I posted the whole lot here, hoping sb could try to
> compile in pure dos.

Yes, with a few workarounds, it works.

> Well. Under WinXP, one can use the win32 variant. It is just that I didn't
> have anything else to prepare the go32v2 snapshot.

I know, but my point is that even DJGPP barely works due to bugs that MS flat out refused to fix. I don't want to rag on them, but it's just gotten worse and worse since XP (2k3, Vista, 7) for NTVDM. It's just sad, honestly, unless (like most) you don't care about losing years worth of binary compatibility.

> > Your .BAT files aren't DOS-shell friendly anyways, they use non-portable
> > Win32-isms:
>
> They are more meant as examples and reflect my local build systems. Feel
> free to customize;-0

Well, here's a .BAT that shows how to get it working. Feel free to ask more questions. I'm no pro, but such workarounds are "par for the course", so to speak! This should give you an idea of what needs fixing.


@echo off
REM Thursday, September 5, 2013

if "%GSED%"=="" set GSED=gsed
if "%RAMDRIVE%"=="" goto end
%RAMDRIVE%:
cd\
REM ... LFNs are mandatory ...
doslfn
set F1=c:\temp\fpc-i8086-msdos-go32v2hosted.zip
if not "%1"=="" if exist %1 set F1=%1
if not exist %F1% goto end
unzip -q %F1%
set F1=
cd pp16

:begin
REM ... use Causeway (p.d. w/ srcs) to avoid out of memory errors ...
copy /b c:\utils\cwstub.exe binw\dos4gw.exe >NUL
REM ... Ignore spurious time warnings from sed + DOSLFN ...
ctty nul
%GSED% -i~ -e "s/\(%\)\*/\11 \12 \13 \14 \15/" .../pp16.bat
%GSED% -i~ -e "s/^\(SET MYDRIVE=\)d/\1%RAMDRIVE%/" dopp16.bat
%GSED% -i -e "s/\\i386-go32v2/\\i386-g~1/" dopp16.bat
%GSED% -i -e "s/.* neq .*/if not x%%OLDPATH%%x==xx goto nosave/" dopp16.bat
ctty con
path c:\utils;c:\;c:\fdos
call dopp16
cd demo

REM ... -st = "Generate script to link on target" ...
echo on
call pp16 -WmTiny -st fpctris
@echo off

REM ... begin sed hack for fpctris_ppas.bat here ...
echo #n>>make.sed
echo s/\(wlib\.exe -q -fo -c \)\(.*\)\.lib.*/cd \2.sl\>>make.sed
echo "dir \/b *.o > \2.rsp\"|%GSED% -e "s/^.//" -e "s/.$//" >>make.sed
echo \1 ..\\\2.lib @\2.rsp\>>make.sed
echo del \2.rsp\>>make.sed
echo cd../>>make.sed
echo /echo Assembling/s/^/REM />>make.sed
echo s/^\(wlink\.exe\)  */\1 op q />>make.sed
echo w make.bat>>make.sed
REM ... end sed hack for fpctris_ppas.bat here ...

%GSED% -f make.sed fpctris_ppas.bat

REM ... DOS wlink accepts SFN only ...
ren fpctri*.res fpctris_.res

if not exist make.bat goto end

:make
call make
if not exist fpctris.exe goto end
dir fpctris.exe
comp fpctris.exe c:\temp\fpctris.old

echo ... output is same except for month/date of FPC 2.7.1 snapshot ...
::
::Byte    File 1  File 2
::AF17    39 '9'  38 '8'
::AF19    30 '0'  32 '2'
::AF1A    34 '4'  37 '7'
::3 error(s).

:end
if "%GSED%"=="gsed" set GSED=
doslfn -u


> But the Exec function is broken and does not correctly pass commandline
> string from FreePascal into DOS.
> There is simply again present the bug 21502
> (http://bugs.freepascal.org/view.php?id=21502)

I checked that bugreport, and the only fix named was r50099, and that is present in both 2.6.x branch and trunk. So it doesn't seem to be a regression from that.