Back to home page

DOS ain't dead

Forum index page

Log in | Register

Back to the board
Thread view  Mix view  Order
DosWorld(R)

06.02.2019, 23:06
 

Howto receive 4GB memory as one chunk, DPMI + Watcom C (Developers)

Hello!

I had develop software (text editor) and I want open very huge files. My problem is - dpmi hosts does not return memory as one chunk. So, watcom memory manager use memory only from one chunk (~64mb, with causeway extender - 220mb). How to received full memory as one chunk?

My target - open 3-3.5gb text file. Now, I can open only little bit more 40mb :(
May be somebody have any advise..

As I understood, need something like virtual memory mapping. (But what is it and how prepare it?)

PS: One time I receive all memory as one chunk, but in very strange way: I compile programm for watcom "compact" memory model for dos4g-extender and run it via HX's dpmild32. I had read recomendations does not use this memory model (because library is not ready, but printf works for me!) and I dont known how to receive access to first megabyte (0000:004xx, b800:0000) in this memory model. At the same time, programm compiled for "hxdos" had 64mb chunks.

PPS: For memory size check I am use source code from Watcom Programmers Guide (I hope, many people see it).

PPPS: I had receive big speed degradation for watcom's malloc, after 500k allocated strings. I think, it will be unusable after 1m malloc's. It is looks like watcom manage memory via linked list structure. So, I want refuse using watcom malloc and switch to my own memory manager (may be he not so fast, but he remember where is stored first free memory block).

RayeR(R)

Homepage

CZ,
07.02.2019, 02:03

@ DosWorld

Howto receive 4GB memory as one chunk, DPMI + Watcom C

With DJGPP/CWSDPMI you should be able to malloc something around 2GB... There's also new API for 64bit long mode, see some older posts here...

---
DOS gives me freedom to unlimited HW access.

alexfru(R)

USA,
07.02.2019, 09:55

@ DosWorld

Howto receive 4GB memory as one chunk, DPMI + Watcom C

> I had develop software (text editor) and I want open very huge files. My
> problem is - dpmi hosts does not return memory as one chunk. So, watcom
> memory manager use memory only from one chunk (~64mb, with causeway
> extender - 220mb). How to received full memory as one chunk?
>
> My target - open 3-3.5gb text file. Now, I can open only little bit more
> 40mb :(
> May be somebody have any advise..

If the problem is in the standard library, you should try allocating memory directly either via DPMI functions or HIMEM.SYS (possibly, before the main .EXE, and then map it via DPMI).

Alex

DosWorld(R)

14.02.2019, 01:30

@ alexfru

Howto receive 4GB memory as one chunk, DPMI + Watcom C

> If the problem is in the standard library, you should try allocating memory
> directly either via DPMI functions or HIMEM.SYS (possibly, before the main
> .EXE, and then map it via DPMI).

yes :( (I have no enough expirience for it, now)

Also... What about SmallerC's dpstub.asm ? (is this mapping available? may be more easy way - just switch to SmallerC?)

DosWorld(R)

14.02.2019, 01:36

@ RayeR

Howto receive 4GB memory as one chunk, DPMI + Watcom C

> With DJGPP/CWSDPMI you should be able to malloc something around 2GB...
> There's also new API for 64bit long mode, see some older posts here...

Hmm... It will be "plan B". DJGPP is good compiler, but... too modern.

(it other words - djgpp build "hello world" >20s on my k6/300. I want see my texteditor.exe before I am die)

alexfru(R)

USA,
14.02.2019, 10:32

@ DosWorld

Howto receive 4GB memory as one chunk, DPMI + Watcom C

> > If the problem is in the standard library, you should try allocating
> memory
> > directly either via DPMI functions or HIMEM.SYS (possibly, before the
> main
> > .EXE, and then map it via DPMI).
>
> yes :( (I have no enough expirience for it, now)
>
> Also... What about SmallerC's dpstub.asm ? (is this mapping available? may
> be more easy way - just switch to SmallerC?)

First of all you should probably forget about editing any files larger than 2GB-1. That's a FAT16 maximum. FAT32 may let you have 4GB-1-long files, but...

... Secondly, in 32-bit compilers (especially old ones or developed for old systems) long and fpos_t likely won't let you handle more than 2GB-1 for file sizes and offsets. malloc() may also fail for larger sizes (due to possible implementation bugs, even if the underlying DPMI host allows it).

Third, I don't know what size limitations are in e.g. CWSDPMI or DOS4GW. There may be some. Btw, NTVDM on Windows Vista (or was it 7?) and up also limits DPMI memory by default to something like 32MB (can be fixed by poking in the registry, AFAIR).

In reality I wouldn't want to deal with anything larger than 1GB in DOS at all.

Fourth, Smaller C isn't particularly good if you want to do lots of processing. It optimizes nearly nothing. And its malloc() is primitive and not suitable for many allocations (very likely much worse than Watcom's or DJGPP's). Smaller C's <string.h> functions (actually, most of the library functions) are also implemented in slow C.

So, yeah, you'll likely need a custom memory manager for your task.

Another thing to point out is that you shouldn't be using primitive algorithms when working with large text files. You definitely don't want to move 1GB of data in memory every time you insert or delete a character in the middle of a 2GB file. https://en.wikipedia.org/wiki/Gap_buffer may help here (it's a very simple data structure with a simple implementation).

Speaking of algorithms, if you can't allocate most of free RAM as one block due to fragmentation, you may want to represent your text as https://en.wikipedia.org/wiki/Rope_(data_structure). Then you can use non-contiguous buffers. The rope is more complex than the gap buffer and needs more memory for the meta data.

As for dpstub.asm, if you specify a large value in -maxheap (when linking), you should be able to malloc() that much memory at once unless the DPMI host has some limitation of its own.

Laaca(R)

Homepage

Czech republic,
14.02.2019, 17:40

@ alexfru

Howto receive 4GB memory as one chunk, DPMI + Watcom C

Yes! For text editor is definitely not needed the allocation of one huge buffer. Much better is to split the file into more buffers.
For example in my editor Blocek is every line on own buffer and these buffers are connected via structure called linked-list (in my implementation is it encapsulated double linked list).
Other possible solution is to connect lines by dynamic array (has some advantages and disadvantages over linked lists).

I didn't want to enter this discussion earlier because I considered it more as a theoretical discussion about allocating giant memory blocks.

---
DOS-u-akbar!

CandyMan(R)

14.02.2019, 18:08
(edited by CandyMan, 14.02.2019, 18:22)

@ DosWorld

Howto receive 4GB memory as one chunk, DPMI + Watcom C

Tomasz Grysztar (author of flat assembler) wrote a text editor using a one-way list. Each line of text consists of blocks of "segments" of fixed length. The first block points to the next one and the next block to the next one. As a result, a continuous memory area is not needed.
In the long mode it is possible to obtain a continuous memory area of up to 64GiB thanks to mapping.
It is possible to download this text editor from (search for the file FASMD3X.EXE in archive FASM.7Z):
https://drive.google.com/drive/folders/0B_wEiYjzVkC0ZGtkbENENzF1Nms

DosWorld(R)

14.02.2019, 22:05

@ alexfru

Howto receive 4GB memory as one chunk, DPMI + Watcom C

> First of all
> ... Secondly
> Third
> Fourth

Thanks for hightlithing of problems, I have solution for ALL of this problems.

> So, yeah, you'll likely need a custom memory manager for your task.

I am already have.

> Another thing to point out is that you shouldn't be using primitive
> algorithms when working with large text files. You definitely don't want to
> move 1GB of data in memory every time you insert or delete a character

solved.

> Speaking of algorithms, if you can't allocate most of free RAM as one block
> due to fragmentation, you may want to represent your text as
> https://en.wikipedia.org/wiki/Rope_(data_structure).

It is solved - it work into my custom memory manager (something sort of "rope" technique). BUT, for this custom memory manager's sanbox- need memory. One big piece of memory. Only this is my problem.

Back to the board
Thread view  Mix view  Order
15764 Postings in 1445 Threads, 262 registered users, 7 users online (0 registered, 7 guests)
DOS ain't dead | Admin contact
RSS Feed
powered by my little forum