DOS ain't dead - OS/2 extender

kerravon

Ligao, Free World North,
10.09.2025, 21:05

OS/2 extender (Developers)

The mainframe work has created an unexpected development.

So OS/2 2.0 is a 32-bit OS.

In the same way that we have 32-bit DOS extenders to allow access to memory above 1 MiB - is it technically feasible to have a 64-bit OS/2 extender to allow access to memory above 4 GiB?

What would that look like?

Are there 64-bit DOS extenders? If so, maybe it would look like that?

Any reason to not use the PE32+ Win64 API as the DOS and/or OS2 extender? What would the alternative be? Is there a 64-bit LX format?

Rugxulo

Usono,
10.09.2025, 21:30

@ kerravon

OS/2 extender

Post reply

> Physical Address Extension (PAE): To allow 32-bit systems to use
> more than 4 GB of physical RAM, an extension called PAE was developed.
> With a PAE-enabled kernel, the OS could access up to 64 GB of physical
> memory, but each individual 32-bit process was still limited to its
> 4 GB virtual address space.
>
> For a time, there was a specialized Application Binary Interface (ABI)
> called "x32" that ran on 64-bit AMD64 machines.
> Pointers and performance: The x32 ABI used 32-bit pointers for
> compatibility and efficiency, which limited the memory of a single
> process to 4 GB. However, it still provided access to the full set of
> 64-bit registers, which could offer a performance boost for specific
> workloads.

Realistically, if there is a speed advantage to AMD64 (despite wasting more memory with bigger pointers), it just varies. The apps also tend to be noticeably larger.

For DOS, I imagine you'd load the data into memory, switch to 64-bit mode to do calculations, then switch back and write the data in memory to file (if needed).

Then again, yet another option is to use SIMD (SSE2) for 64-bit math.
Although 64-bit integer math is also easy to do with 32-bit regs.

Yet another idea is to use VT-X in 64-bit mode to "emulate" DOS.

So there is no huge need for AMD64 support in DOS.

Ringding

11.09.2025, 17:44

@ kerravon

OS/2 extender

Post reply

> In the same way that we have 32-bit DOS extenders to allow access to memory
> above 1 MiB - is it technically feasible to have a 64-bit OS/2 extender to
> allow access to memory above 4 GiB?
>
> What would that look like?

It would be very difficult, as the kernel would need to be made aware of it. In DOS, the extender itself (or the DPMI host, which might be the same thing) is the kernel, which can do whatever it wants to. Under OS/2, user programs are not allowed to exert this kind of control over the machine.

kerravon

Ligao, Free World North,
11.09.2025, 17:57

@ Ringding

OS/2 extender

Post reply

> > In the same way that we have 32-bit DOS extenders to allow access to
> memory
> > above 1 MiB - is it technically feasible to have a 64-bit OS/2 extender
> to
> > allow access to memory above 4 GiB?
> >
> > What would that look like?
>
> It would be very difficult, as the kernel would need to be made aware of
> it. In DOS, the extender itself (or the DPMI host, which might be the same
> thing) is the kernel, which can do whatever it wants to. Under OS/2,
> user programs are not allowed to exert this kind of control over the
> machine.

I'm not sure what OS/2 needs to be able to run a privileged program.

Some sort of "administrator command prompt"?

I was considering disabling interrupts, entering 64-bit mode, doing all my work above 4 GiB, and then when I need any service, switching back to 32-bit mode and reenabling interrupts.

Presumably exactly the same as DOS extenders did, especially DOS extenders written prior to DPMI existing (I think that's a thing).

You think the above scheme won't work?

I'm not sure how to find out what memory above 4 GiB I can use - I might need the user to provide address ranges which they can get from some other mechanism. E.g. there is a DOS interrupt that gives you all available memory, including memory above 4 GiB. The output from that interrupt could be put into a flat file when I run my OS/2 extender.

I have a new version of the cc64 compiler to try out - I'm hoping that that is more robust than the one I currently use. So that's a public domain x64 compiler. But it is generated code, so is not a lot different from just having a binary.

So I'm expecting to have a mini-Win64 under OS/2, as there doesn't appear to be such a thing as a 64-bit LX executable.

Ringding

11.09.2025, 20:41

@ kerravon

OS/2 extender

Post reply

> I was considering disabling interrupts, entering 64-bit mode, doing all my
> work above 4 GiB, and then when I need any service, switching back to
> 32-bit mode and reenabling interrupts.
>
> Presumably exactly the same as DOS extenders did, especially DOS extenders
> written prior to DPMI existing (I think that's a thing).
>
> You think the above scheme won't work?

No. That's the protection in "protected mode" at work.

It works in DOS, as there is no protected mode operating system running that prevents programs from switching processor modes like crazy.

kerravon

Ligao, Free World North,
12.09.2025, 00:08

@ Ringding

OS/2 extender

Post reply

> > I was considering disabling interrupts, entering 64-bit mode, doing all
> my
> > work above 4 GiB, and then when I need any service, switching back to
> > 32-bit mode and reenabling interrupts.
> >
> > Presumably exactly the same as DOS extenders did, especially DOS
> extenders
> > written prior to DPMI existing (I think that's a thing).
> >
> > You think the above scheme won't work?
>
> No. That's the protection in "protected mode" at work.
>
> It works in DOS, as there is no protected mode operating system running
> that prevents programs from switching processor modes like crazy.

I don't understand.

First of all - both PM32 and LM64 are "protected mode", I believe.

Secondly, so long as I have privilege, there is nothing preventing me from switching from PM32 to LM64.

You could argue that when I return to PM32, everything will blow up because I've messed everything up.

But I believe everything can be restored to how it was before.

So long as I only touch memory above 4 GiB, OS/2 won't even be aware that anything has happened. The most it could possibly detect - if it tried hard - was that interrupts were disabled for "quite a while".

Rugxulo Usono, 12.09.2025, 04:07 @ kerravon	OS/2 extender Post reply
	Have you seen EMS64 - EMS 4.0 simulator on FASM's forum?

kerravon

Ligao, Free World North,
12.09.2025, 16:55

@ Rugxulo

OS/2 extender

Post reply

> Have you seen EMS64
> - EMS 4.0 simulator on FASM's forum?

I just took a look. Doesn't look like a DOS extender to me, which is what I'm after. It will most likely be an x64 DOS or OS/2 extender to run Win64 executables.

BTW, I was scratching my head to try to find out what an issue might be - and I had a thought that maybe the thinking is that OS/2 will fail to execute the code that I need to switch to LM64 because it isn't paged in.

In which case, there are a couple of solutions.

One would be to access one byte in each 4k page of the code I wish to execute and use to make sure it is paged in at the time I go to execute it.

Another would be to allocate 8K of memory from OS/2 to ensure that I have a full 4k page, and then place code there that is designed to switch to and from LM64 as well as provide 512 bytes of memory (so rwx) that can be used as a buffer for when the Win64 program wishes to write to a file.

PDOS/386 alread does everything 512 bytes at a time, switching from PM32 to RM16 to read/write a sector at a time.

Isn't an issue in practice. The biggest task I do is do optimized gcc 3.2.3 C compiles under an S/370 mainframe emulator and the bottleneck is all in the CPU (actually, memory access I believe - but it shows up as CPU). Mode switching and writing sectors to an SSD isn't an issue.

Ringding

12.09.2025, 17:20

@ kerravon

OS/2 extender

Post reply

> Secondly, so long as I have privilege, there is nothing preventing me from
> switching from PM32 to LM64.
>
> You could argue that when I return to PM32, everything will blow up because
> I've messed everything up.
>
> But I believe everything can be restored to how it was before.
>
> So long as I only touch memory above 4 GiB, OS/2 won't even be aware that
> anything has happened. The most it could possibly detect - if it tried hard
> - was that interrupts were disabled for "quite a while".

It is an interesting thought experiment, but ultimately completely useless. In theory, it might just work. You just don't gain the privilege to switch modes so easily. I assume you would need a ring 0 kernel driver for this.

I envision the architecture like this: For every 64 bit task to run, there is a small, normal 32-bit OS/2 part that connects to and communicates with the kernel mode driver. It will continuously ask the kernel mode driver to run the 64 bit task until it exits, similar to how virtualized tasks are run today.

Long mode would have to have its separate set of page tables and interrupt descriptors. The kernel driver would have to act as a kernel for the 64 bit task and handle its synchronous interrupts like invalid instruction or arithmetic exception as well as syscalls. Youl would need to come up with some kind of syscall and API thunking scheme. There would also need to be some kind of communication buffer that both the 32 bit and the 64 bit part can access. This could be allocated by the kernel driver in a way so that it will not be moved around by OS/2. Then it could be mapped into the 64 bit address space.

Whenever a hardware interrupt comes in that is intended for OS/2, it would need to quickly save the state of the 64 bit task, switch back to legacy mode and deliver the interrupt to OS/2 by manually interpreting its IDT. Or maybe you could get away with disabling interrupts, but you would need to have a way to interrupt the running 64 bit task anyway to prevent it from stalling the OS indefinitely. I could think of using a timer for this, but this would have to be stolen from OS/2 and put back into the state that OS/2 expects it to be in before returning control -- quite a mess!

kerravon

Ligao, Free World North,
12.09.2025, 18:03

@ Ringding

OS/2 extender

Post reply

> It is an interesting thought experiment, but ultimately completely useless.

Why useless? It's somewhat similar to S/380.

https://mvs380.sourceforge.net/

It is nominally impossible to run 31-bit tasks under a 24-bit operating system.

But it was possible anyway, and enabled me to access large amounts of memory. And the executables I run on MVS/380 are able to be run on modern z/OS.

And what I'm now talking about is running Win64 executables. Similar but not identical to HX.

> In theory, it might just work. You just don't gain the privilege to switch
> modes so easily. I assume you would need a ring 0 kernel driver for this.

I would have thought any privileged executable could execute the instructions. Who's going to stand in the way? I have asked:

https://www.os2world.com/forum/index.php/topic,3980.0.html

> I envision the architecture like this: For every 64 bit task to run, there
> is a small, normal 32-bit OS/2 part that connects to and communicates with
> the kernel mode driver. It will continuously ask the kernel mode driver to
> run the 64 bit task until it exits, similar to how virtualized tasks are
> run today.

Yes.

> Long mode would have to have its separate set of page tables and interrupt
> descriptors. The kernel driver would have to act as a kernel for the 64 bit
> task and handle its synchronous interrupts like invalid instruction or
> arithmetic exception as well as syscalls.

Ok, first of all, I have the limited goal of running error-free programs. If there is a bug, I don't particularly care what happens.

In addition, there won't be any syscalls (I assume you mean INT instructions). It is the Win64 API that will be called, and that will have callback functions plugged in that naturally return to the 32-bit portion (ie switch from LM64 to PM32) to get executed.

> Youl would need to come up with
> some kind of syscall and API thunking scheme.

Yes, I already have that - I call it a pseudobios. And I make good use of that in mfemul where the mainframe code does a call into the pseudobios which is then resolved by native (e.g. 386) code.

> There would also need to be
> some kind of communication buffer that both the 32 bit and the 64 bit part
> can access. This could be allocated by the kernel driver in a way so that
> it will not be moved around by OS/2. Then it could be mapped into the 64
> bit address space.

Sure. I will need a way of knowing the real address for a PM32 virtual page.

> Whenever a hardware interrupt comes in that is intended for OS/2, it would
> need to quickly save the state of the 64 bit task, switch back to legacy
> mode and deliver the interrupt to OS/2 by manually interpreting its IDT. Or
> maybe you could get away with disabling interrupts,

Yes - disabling interrupts was what I had in mind. At least for proof of concept and to start running Win64 apps. Bells and whistles is an exercise for the reader (or a commercial vendor).

> but you would need to
> have a way to interrupt the running 64 bit task anyway to prevent it from
> stalling the OS indefinitely.

I don't mind buggy programs hanging the OS. There are already bugs in OS/2 that cause it to hang anyway.

> I could think of using a timer for this, but
> this would have to be stolen from OS/2 and put back into the state that
> OS/2 expects it to be in before returning control -- quite a mess!

I don't consider it a mess. I don't consider DOS extenders to be a mess either.

I'm only expecting one task to run, and like MSDOS, I'm expecting buggy apps to be able to hang/crash the system.

I simply debug buggy apps. Works fine for me - for decades.

Ringding

01.10.2025, 18:54

@ kerravon

OS/2 extender

Post reply

> https://www.os2world.com/forum/index.php/topic,3980.0.html

I tried to register an account at the OS2 world forum specifically to reply to the thread there, but that would require admin approval, which is not forthcoming.

From a reply in this thread:

> You can have 16/32 code or 32/64 code but not 16/32/64.

That’s what I thought, but it is incorrect. Long mode can deal with all three code segment sizes at the same time, as indicated by the very first table in the system programming volume of AMD’s architecture manual (“Operating Modes”). What you cannot have is VM86 mode. I verified PM16 on my current Linux installations last week (from a 32-bit task). One significant piece of functionality is missing, although this is just a limitation of Linux, not the machine architecture: having your own SIGSEGV handler intercept general protection faults from that 16-bit code segment. Unfortunately, the kernel does not understand this kind of code and will kill the faulting process directly instead of invoking its signal handler.

kerravon

Ligao, Free World North,
17.10.2025, 17:41

@ Ringding

OS/2 extender

Post reply

> > https://www.os2world.com/forum/index.php/topic,3980.0.html
>
> I tried to register an account at the OS2 world forum specifically to reply
> to the thread there, but that would require admin approval, which is not
> forthcoming.

I have asked the question myself.

> From a reply in this thread:
>
> > You can have 16/32 code or 32/64 code but not 16/32/64.
>
> That’s what I thought, but it is incorrect. Long mode can deal with all
> three code segment sizes at the same time, as indicated by the very first
> table in the system programming volume of AMD’s architecture manual
> (“Operating Modes”). What you cannot have is VM86 mode. I verified PM16
> on my current Linux installations last week (from a 32-bit task). One
> significant piece of functionality is missing, although this is just a
> limitation of Linux, not the machine architecture: having your own SIGSEGV
> handler intercept general protection faults from that 16-bit code segment.
> Unfortunately, the kernel does not understand this kind of code and will
> kill the faulting process directly instead of invoking its signal handler.

I'm having difficulty understanding the replies - to the point where
I'm not even sure if people are talking cross-purposes.

But what are your thoughts about TWO replies that were given in the above
thread?