DOS ain't dead

Johnson Lam has posted a new DRIVERS.ZIP file on his website at --

http://johnson.tmfc.net/dos/driver.html

The 16-Nov-2009 UIDE can cache up to 4 GIGABYTES of data! This now allows
a 4-GB PC system to have, for example, a 500-MB RAMdisk using RDISK, 500-MB
of XMS saved for other drivers/programs, and a 3-GB UIDE cache! That will
provide a "SCREAMING Fast!" DOS system!

I have only 1-GB, so I tested the driver "patched" for 16K-byte data blocks
(not its usual 64K-byte blocks), and UIDE handled all possible cache blocks
properly. If anyone has a REAL 4-GB system, I would appreciate hearing if
UIDE in fact works O.K. with it.

UIDE also adds the /P switch, for users of JEMM386 and other protected-mode
systems. Its usual binary-search table in XMS memory does demand much XMS
work, making the driver a bit slower in protected-mode. /P makes UIDE set
its "old style" binary-search table at the end of driver memory. This can
give up to 10% better speed in protected-mode due to far less XMS accesses.

Though faster, UIDE's /P switch has memory limits. DOS systems that offer
under 14K of free HMA (V7.10 MS-DOS etc.) are limited to 170-MB caches when
UIDE loads with /H in the HMA, to avoid V7.10 MS-DOS "bugs"! A DOS system
with more HMA, e.g. V6.22 MS-DOS, can have a 350-MB+ cache using HMA space.
FreeDOS does not make HMA available when CONFIG.SYS runs, but FreeDOS users
can run DEVLOAD from AUTOEXEC.BAT to put UIDE in the HMA. UIDE /P can set
up to 400-MB HMA caches for FreeDOS, depending on BUFFERS= and other values
that also take HMA space.

With /P, UIDE sets a 5-MB HMA cache only if /S5 is used and free HMA allows
this cache. The 5-MB cache runs slower, so it is "by request only". For
larger /S values, if that cache cannot fit in free HMA, UIDE /P defaults to
a 15-MB or 80-MB cache depending on available HMA space, or UIDE loads only
in upper/DOS memory if its 15-MB HMA cache cannot be set. Users can avoid
these defaults by omitting /H and so "ordering" the driver to use upper/DOS
memory. The maximum UIDE /P cache size is 1900-MB if loaded in memory, as
the driver and its search table must go in one 64K program segment. Users
needing bigger caches must omit /P and run UIDE with regular 4-GB caching.

Real-mode UMBPCI users do not need UIDE /P. A protected-mode JEMM386 user
can now choose between UIDE /P, for more speed using medium-size caches, or
UIDE without its /P switch when a larger cache is required.

---
(Account disabled on user's request.)

Today, after announcing the 16-Nov-2009 UIDE with 4-GB caching, Steve Burd of
this forum wrote me and offered to test the new UIDE on his REAL 4-GB system!
I accepted his gracious offer, and this evening, Steve has replied:

> ... I created a 2047-MB file (just short of 2GB as my dos partition is
> fat32) and copied the file from the root to a secondary folder. First
> read of the new copy read at normal drive speed, 2nd read at 397mb/
> second, 3rd read 600mb/second. Caching fine!
>
> Next a compare of the two copies using Freedos FC v2.22 "FC" reported
> no differences.

THANK YOU, Steve -- Exactly what I wanted to hear, and other users can now be
assured that the new UIDE with up to 4-GB caching works fine!

---
(Account disabled on user's request.)

> Johnson Lam has posted a new DRIVERS.ZIP file on his website at --
>
> http://johnson.tmfc.net/dos/driver.html
>
> The 16-Nov-2009 UIDE can cache up to 4 GIGABYTES of data! This now
> allows
> a 4-GB PC system to have, for example, a 500-MB RAMdisk using RDISK,
> 500-MB
> of XMS saved for other drivers/programs, and a 3-GB UIDE cache! That
> will
> provide a "SCREAMING Fast!" DOS system!

I thought due to PCI space (or whatever) that only 3.1 GB or such was generably accessible?

> I have only 1-GB, so I tested the driver "patched" for 16K-byte data
> blocks
> (not its usual 64K-byte blocks), and UIDE handled all possible cache
> blocks
> properly. If anyone has a REAL 4-GB system, I would appreciate hearing
> if
> UIDE in fact works O.K. with it.

I know you already had one success story, but try contacting Zyzzle. IIRC, he was the guy with lots of RAM.

> Though faster, UIDE's /P switch has memory limits.
> FreeDOS does not make HMA available when CONFIG.SYS runs, but FreeDOS
> users
> can run DEVLOAD from AUTOEXEC.BAT to put UIDE in the HMA. UIDE /P can
> set
> up to 400-MB HMA caches for FreeDOS, depending on BUFFERS= and other
> values
> that also take HMA space.

I will have to try DEVLOADing this to see if it works. (I don't remember it being successful in the past, but I admit it's been a while.)

> With /P, UIDE sets a 5-MB HMA cache only if /S5 is used and free HMA
> allows
> this cache. The 5-MB cache runs slower, so it is "by request only".

I don't understand, 5 MB cache runs slower with /P or just slower than bigger caches in general? For instance, I always run a 5 MB cache on my P166 (only 32 MB RAM total). We really need a good but simple benchmark for UIDE changes (e.g. compiling TDE is what I typical would use).

> Real-mode UMBPCI users do not need UIDE /P. A protected-mode JEMM386
> user
> can now choose between UIDE /P, for more speed using medium-size caches,
> or
> UIDE without its /P switch when a larger cache is required.

Slightly complex, but good to know. Thanks!

>> The 16-Nov-2009 UIDE can cache up to 4 GIGABYTES of data!
>
> I thought due to PCI space (or whatever) that only 3.1 GB or such was
> generally accessible?

PCI addresses usually do not take more than 1-MB, unless a system has MANY
I-O devices. On my 1-GB system I can assign up to 1021 out of 1024-MB to
UIDE. DOS uses 1-MB, plus 64K for its HMA, and UIDE uses some XMS as its
cache-data tables. This is why I limit a 4-GB system to 4093-MB.

> I will have to try DEVLOADing this to see if it works. (I don't remember
> it being successful in the past, but I admit it's been a while.)

I last loaded UIDE with FreeDOS DEVLOAD about May of 2009, and I told Eric
Auer that his DEVLOAD worked O.K. This should still be so. If not, let
me know, and I will "Pull out a few more gray hairs!" and FIX this!

>> With /P, UIDE sets a 5-MB HMA cache only if /S5 is used and free HMA
>> allows this cache. The 5-MB cache runs slower, so it is "by request
>> only".
>
> I don't understand, 5 MB cache runs slower with /P or just slower than
> bigger caches in general? For instance, I always run a 5 MB cache on
> my P166 (only 32 MB RAM total).

Smaller caches must "discard" data more often, to make room for new data
being read or written. The 5-MB cache must "discard" almost everything
including directory data, if a file of over 2-MB is copied. UIDE needs
2-MB for the input copy, 2-MB more for the output copy, and shall likely
have to "give up" most stored directory sectors as well. The next file
will thus run slower, as DOS must re-cache all directory sectors. Only
then is DOS speed restored.

All of this is true with /P or without it. UIDE has a "least-recently-
used" (LRU) list that threads through all its cache-table entries, so it
"discards" only those sectors which are least-used on that list. But a
BIG new file can make UIDE discard EVERYTHING, as is true for any cache!

A "Rule of Thumb" for UIDE caches is that you should have (if possible!)
at least 3 times the cache space as your largest data file. Thus, UIDE
has room for an input copy, an output copy, and room so that directories
do NOT have to be "discarded", thus more data files are handled quickly!

Since "Vista" and other systems have MANY files at or near 2-MB, I chose
to have UIDE "default" to at least a 15-MB cache, if using HMA space, as
the 5-MB cache simply loses too much performance. Users that want 5-MB
can still ask for it with /S5, but UIDE will otherwise "prefer" 15-MB if
/H is given and it loads in HMA space. Note also that if a 15-MB cache
cannot fit in the HMA, UIDE will default to loading in upper/DOS memory.
"Not TOO bad!", as the memory needed for up to a 100-MB UIDE /P cache is
only about 7.5K! [Up to a 250-MB /P cache is only about 12K, as well!]

On your 32-MB Pentium system, unless you have other uses for XMS memory,
I would set at least UIDE's 15-MB cache, for better performance.

>> Real-mode UMBPCI users do not need UIDE /P. A protected-mode JEMM386
>> user can now choose between UIDE /P, for more speed using medium-size
>> caches, or UIDE without its /P switch when a larger cache is required.
>
> Slightly complex, but good to know. Thanks!

You are welcome! I know the rules for using UIDE /P are a bit complex,
but it is "about all I had LEFT" to give UIDE more protected-mode speed!
Japheth had to delete direct Int 15h "calls" (for XMS moves) from HIMEMX
same as I had to delete /K from UIDE, since neither worked with Win/311.
UIDE /P "buys back" some of that lost speed for JEMM386 users and others
who run DOS in protected-mode.

---
(Account disabled on user's request.)

> PCI addresses usually do not take more than 1-MB, unless a system has
> MANY
> I-O devices. On my 1-GB system I can assign up to 1021 out of 1024-MB
> to
> UIDE. DOS uses 1-MB, plus 64K for its HMA, and UIDE uses some XMS as
> its
> cache-data tables. This is why I limit a 4-GB system to 4093-MB.

BTW on new systems you need to count huge framebuffer of VGA (typ. 256-512MB) which is also memory mapped and cut down free RAM...

---
DOS gives me freedom to unlimited HW access.

>> PCI addresses usually do not take more than 1-MB, unless a system has
>> MANY I-O devices. On my 1-GB system, I can assign up to 1021 out of
>> 1024-MB to UIDE ...
>
> BTW on new systems you need to count huge framebuffer of VGA (typ.
> 256-512MB) which is also memory-mapped and cuts down free RAM ...

Understood, but not a problem. It is a "job" for any XMS manager to
know which areas over 1-MB can be used by drivers or programs wanting
XMS memory. Since the 1990s, we have had "Int 15h, AX=E820h" calls
to the BIOS which post a "list" of extended memory areas: I-O ports,
memory, a "hole" in addressing to avoid, etc. On systems with a VGA
frame buffer, the extended-memory "list" should denote a block of I-O
addresses for it. XMS managers only want MEMORY, so they will avoid
the VGA frame-buffer addresses.

Most drivers or applications, like my own UIDE, are not concerned re:
exact memory specifics. UIDE simply asks the XMS manager for an XMS
memory block to use, goes on if it GETS the memory, or ABORTS if not!
I expect most other drivers/applications do nothing more complex than
that. Exact memory specifics ought to be "worried about" only by an
XMS manager itself.

---
(Account disabled on user's request.)

Johnson Lam has posted another DRIVERS.ZIP file, now dated 11-22-2009, on
his website at --

http://johnson.tmfc.net/dos/driver.html

The 11-22-2009 UIDE/UIDEJR, and all later releases, shall now be supplied
without UPX packing. This prevents DOS system loaders, FreeDOS DEVLOAD,
and other loaders from placing the drivers in too-small a memory area due
to their "packed" size. When the drivers are unpacked, this can DESTROY
whatever follows them in memory! If UIDE/UIDEJR are "seen" full-size by
DOS loaders, this unpacking problem is eliminated.

The 11-22-2009 UIDE also has more optimal cache-block sizes. My testing
showed that when UIDE copied my 63-MB WINNT\SYSTEM32 directory to another
hard-disk area, the total 126-MB (input and output data) needed 180-MB of
cache to compare at XMS-memory speeds, i.e. all files were "in cache" and
compared quickly. Since this is only 70% efficient, I tried 32K and 16K
cache-block sizes. 32K blocks required only 150-MB of cache, 16K blocks
needed only 140-MB! MUCH more efficient!

So, UIDE shall now use 16K-byte cache blocks for 40-MB to 1023-MB caches,
32K blocks for 1024-MB (1 Gigabyte) to 2047-MB caches, and 64K blocks for
2-GB caches or more, which must use 64K blocks. I noted no visible loss
of speed in my tests, from reading or writing smaller blocks. UIDE also
has a new /F switch to set 64K blocks as before, for "odd" cases when 64K
blocks may provide a tiny bit more speed.

My apologies to all, for failing to realize until now that smaller blocks
in fact DO provide up 25% better cache capacity for UIDE!

Note that these changes are for the standard XMS-memory caches only. /P
protected mode caches have far more limited binary-search table space and
their total blocks cannot be doubled/quadrupled! So, the /P caches will
continue to use 64K blocks for all caches above 50-MB.

---
(Account disabled on user's request.)

Many thanks for your continued dedication and improvements. Yes, it does seem that 16-KB cache blocks provide more efficient transfers when many small files are involved, such as exist in your \windows\system32\ files directory. Am I correct that the logic works the same as a cluster-size on large hard disks, ie there is always some "slack" space involved with every file transfered? So, it would seem that 64-kb block sizes will be most efficient in terms of speed, for large files and 16-kb best for smaller files?

I am the guy with lots of memory (8 GB RAM) and can confirm that the 4 GB cache option works well and properly. As I am only able to access 3584 MB total RAM in DOS due to framebuffer limits, I did test on files of 2047 MB and 1024 MB (3 GB total) and all was cached at blazing speeds (at least 1.2 GB/sec) to the best of my timing ability.

I would like to test QCACHE on a *single* file > 2 GB, but this requires a 64-kb cluster size FAT16 partition, which I will set up and try... I'll start with a 3072 MB file and work from there.

> Many thanks for your continued dedication and improvements.

You are welcome! Happy to see that the new UIDE seems to have helped you.

> Yes, it does seem that 16-KB cache blocks provide more efficient transfers
> when many small files are involved ...

Actually, the 16K-byte blocks may "transfer" slightly slower. Their intent
is to provide more "net" cache capacity.

> Am I correct that the logic works the same as a cluster-size on large hard
> disks, ie there is always some "slack" space involved with every file ...

There is always some "slack" in the last cache block used to store any file.
But it was not my intent to "emulate" a hard disk's clusters. UIDE's newer
cache blocks provide the minimum blocksize which its 65536 blocks can allow.
A 16-bit word is used for UIDE's binary-search indexes, so it cannot "count"
higher than 65536 blocks. 65K blocks times 16K bytes per block allows 1-GB
caches, times 32K bytes allows 2-GB caches and times 64K allows 4-GB caches.
"Nothing more complex" than that is involved!

> So, it would seem that 64-kb block sizes will be most efficient in terms
> of speed, for large files and 16-kb best for smaller files?

I have noted very little speed difference with 16K/32K/64K blocks. Again,
my intent was to provide the best CAPACITY, and I am in fact SURPRISED that
the 16K blocks seem to perform almost equally fast as larger ones!

> I am the guy with lots of memory (8 GB RAM) and can confirm that the 4 GB
> cache option works well and properly. As I am only able to access 3584 MB
> total RAM in DOS due to framebuffer limits, I did test on files of 2047 MB
> and 1024 MB (3 GB total) and all was cached at blazing speed (at least 1.2
> GB/sec) to the best of my timing ability.

Steve Burd, on his own 4-GB system, also seems to be limited by frame-buffer
"losses" to only 3.5-GB. You both ought to try DISABLING the frame buffer,
when you run DOS, if that is possible.

> I would like to test QCACHE on a *single* file > 2 GB, but this requires a
> 64-kb cluster size FAT16 partition, which I will set up and try ... I'll
> start with a 3072 MB file and work from there.

Do call the driver UIDE! The original QCACHE came not-even close to UIDE's
capabilities, has not existed for almost 3 years, and I do not even have the
QCACHE source files any more!

---
(Account disabled on user's request.)