DOS ain't dead

Howto receive 4GB memory as one chunk, DPMI + Watcom C (Developers)

posted by Rugxulo , Usono, 26.10.2019, 18:13

Sorry I'm late! (FYI, I'm pretty much a noob, so I can only provide vague hints.)

> I had develop software (text editor) and I want open very huge files.

*nix "Ed" kept the whole file in memory! Fast but bad if you want to edit huge files (since it requires enough RAM, but that's less of a problem these days). For huge files, something like Sed was recommended (a line at a time!), and it only copied, didn't truly edit in-place.

In-place editing is almost impossible for plain text files anyways due to added or deleted lines or changing line length. If you wanted to hex edit a few bytes in-place (change 0xF to 0xA), that would work, but that kind of editing is somewhat rare (especially if done via batch, aka non-interactively).

> My problem is - dpmi hosts does not return memory as one chunk.
> So, watcom memory manager use memory only from one chunk (~64mb, with causeway extender - 220mb).

Be careful! Causeway supports virtual memory (but will swap out the *entire* amount, unlike CWSDPMI which only requests the extra it needs).

> How to received full memory as one chunk?

XMS is easily fragmented. EMS is probably worse. Virtual memory also makes everything more complicated!
So you can only grab a big chunk, not literally everything in one piece. Heck, BIOS holes make it
where you only get 2.9 GB or such on a 4 GB machine anyways (among other problems ...)!

* http://www.delorie.com/djgpp/doc/dpmi/api/310500.html
"Get Free Memory Information"
"Largest available free block in bytes"
"[T]he information returned by this function should only be considered as advisory."

> PPPS: I had receive big speed degradation for watcom's malloc, after 500k allocated strings.
> I think, it will be unusable after 1m malloc's.

I noticed back in 2010 that DJGPP 2.04's malloc was much better than OpenWatcom (at least, more efficient in space). DJGPP 2.05 switched to CBFalconer's nmalloc (faster for lots of free() calls). CWSDPMI r7 (2010) also added optional support for 4 MB pages (faster on supported 586+ cpus but can't swap, doesn't play well with EMM386 as it's non-standard size). More complications, but at least it doesn't overflow the page table (or whatever) in low/conventional memory as easily!

(historical only! but still interesting)
* http://www.delorie.com/djgpp/malloc/

> With DJGPP/CWSDPMI you should be able to malloc something around 2GB

CWS said to call sbrk() twice to get more than 2 GB total since DJGPP libc does assume 2 GB address space.
(Of course, he also offered to help anyone use 686+ PAE, if desired, but no one did.)

> djgpp build "hello world" >20s on my k6/300. I want see my texteditor.exe before I am die)

This is a separate problem. Not totally dire, but you have to use an appropriate GCC version for your machine (sadly). Also, enable Ultra DMA, use a software cache, use a RAM disk, use a well-written GNUmakefile (to avoid redundant compilations), use libraries (*.a instead of a billion separate *.o), adjust your PATH, split up larger files (-Q to see what funcs are being compiled, IIRC), etc. etc. etc. GCC 3.4.6 is probably max newest you should consider (oh, and you don't have to compile every module with optimizations nor everything with -O2, you can even preprocess and then compile later separately).

> First of all you should probably forget about editing any files larger than 2GB-1. That's a FAT16 maximum.
> FAT32 may let you have 4GB-1-long files, but...

DJGPP 2.04+ should (maybe?) support 4 GB files, but it wasn't widely tested (at least, not by me). I believe that can only be done with int 21h, 716Ch. I don't think NT/XP supported it properly either.

To be honest, unless you have external/third-party data, you shouldn't really have files that big (especially not for text). I'm saying don't do what's hard if you don't have to, then you don't have to support it. You don't have to support literally everything, only reasonable things. It's best to keep texts small and self-contained (especially source code). You can only see a screen of text at a time anyways, so it makes little sense to fill your whole memory up (except for speed, which doesn't matter except with huge batch operations, changing all matching lines or whatever).

Complete thread:

• Howto receive 4GB memory as one chunk, DPMI + Watcom C - DosWorld, 06.02.2019, 23:06 (Developers)
  • Howto receive 4GB memory as one chunk, DPMI + Watcom C - RayeR, 07.02.2019, 02:03
    • Howto receive 4GB memory as one chunk, DPMI + Watcom C - DosWorld, 14.02.2019, 01:36
  • Howto receive 4GB memory as one chunk, DPMI + Watcom C - alexfru, 07.02.2019, 09:55
    • Howto receive 4GB memory as one chunk, DPMI + Watcom C - DosWorld, 14.02.2019, 01:30
      • Howto receive 4GB memory as one chunk, DPMI + Watcom C - alexfru, 14.02.2019, 10:32
        • Howto receive 4GB memory as one chunk, DPMI + Watcom C - Laaca, 14.02.2019, 17:40
        • Howto receive 4GB memory as one chunk, DPMI + Watcom C - DosWorld, 14.02.2019, 22:05
  • Howto receive 4GB memory as one chunk, DPMI + Watcom C - CandyMan, 14.02.2019, 18:08
  • Howto receive 4GB memory as one chunk, DPMI + Watcom C - Rugxulo, 26.10.2019, 18:13