DOS ain't dead

New kernel compression methods - Updated lzd, new lzo (Announce)

posted by ecm  , Düsseldorf, Germany, 25.04.2020, 22:23

I updated the lzd depacker somewhat, in particular to use repeated string instructions for copying over matches. The saving is about 3 seconds shaved off for the test case, from 16.4s to 13.0s (lDebug) or 15.3s to 11.5s (RxDOS kernel).

I also added an LZO depacker, based on the lzo.txt file from the Linux kernel. The resulting size is fairly competitive, compressing 83 kB to 61 kB (lDebug) or 101 kB to 53 kB (RxDOS kernel). The decompression time places it among the faster methods, around 3.8s for both test cases. The LZ4 and Snappy depackers are only slightly faster.

I mailed the authors of the lzo.txt description regarding two issues with the text, one concerning the first byte interpretation (18 = 1 literal) and the other the LZO-RLE extension parsing of the L L L bits of the 0 0 0 1 H L L L instructions.

In the following I list all new test results again. Other than the LZD time optimisation and adding LZO not much has changed.

ldebug/tmp$ hg id
24b1bbf37614 tip
ldebug/tmp$ fname="ldebug.com"; fname2="debug"; fname3="debug.big"; fnameu="ldebugu.com"; arg="/C=Q"; upcase=0; method=none; ((upcase)) && method="$(echo "$method" | tr 'a-z' 'A-Z')"; mkdir -p "$method"; cp -a ../bin/"$fnameu" "$method"/"$fname"; cp -a "$fname3" "$method/$fname2.$method"; for method in none blz lz4 sz exo x hs lz lzo; do ((upcase)) && method="$(echo "$method" | tr 'a-z' 'A-Z')"; echo -e "\nmethod=$method\nsize=$(stat -c %s "$method/$fname")"; echo -e "ini size=$(( $(stat -c %s "$method/$fname") - ( ($(stat -c %s "$method/$fname2.$method") + 15) / 16 * 16 ) ))"; echo -e "alloc=$(exememls "$method/$fname")"; echo "@echo off" > test.bat; for jj in $(seq 0 511); do echo "@$method\\$fname $arg"; done >> test.bat; (export TIMEFORMAT='%3R'; time dosemu -dumb -quiet -K "$PWD" -E "test.bat" 2> /dev/null > /dev/null); done

method=none
size=82944
ini size=4496
alloc=96912 bytes = 6057 paragraphs
2.225

method=blz
size=65024
ini size=6160
alloc=97232 bytes = 6077 paragraphs
5.059

method=lz4
size=63488
ini size=6064
alloc=97248 bytes = 6078 paragraphs
3.373

method=sz
size=70656
ini size=6192
alloc=97360 bytes = 6085 paragraphs
2.941

method=exo
size=55296
ini size=6048
alloc=97312 bytes = 6082 paragraphs
6.159

method=x
size=72192
ini size=6512
alloc=344496 bytes = 21531 paragraphs
13.263

method=hs
size=64000
ini size=5632
alloc=96992 bytes = 6062 paragraphs
7.850

method=lz
size=53248
ini size=7488
alloc=98064 bytes = 6129 paragraphs
12.910

method=lzo
size=60928
ini size=6416
alloc=96960 bytes = 6060 paragraphs
3.786
ldebug/tmp$

rxdos/tmp$ hg id
d441a721725f tip
rxdos/tmp$ fname="RxDOS.COM"; fname2="RxDOS"; fname3="RxDOS.BIN"; fnameu="RxDOSU.COM"; arg="version"; upcase=1; method=none; ((upcase)) && method="$(echo "$method" | tr 'a-z' 'A-Z')"; mkdir -p "$method"; cp -a ../bin/"$fnameu" "$method"/"$fname"; cp -a "$fname3" "$method/$fname2.$method"; for method in none blz lz4 sz exo x hs lz lzo; do ((upcase)) && method="$(echo "$method" | tr 'a-z' 'A-Z')"; echo -e "\nmethod=$method\nsize=$(stat -c %s "$method/$fname")"; echo -e "ini size=$(( $(stat -c %s "$method/$fname") - ( ($(stat -c %s "$method/$fname2.$method") + 15) / 16 * 16 ) ))"; echo -e "alloc=$(exememls "$method/$fname")"; echo "@echo off" > test.bat; for jj in $(seq 0 511); do echo "@$method\\$fname $arg"; done >> test.bat; (export TIMEFORMAT='%3R'; time dosemu -dumb -quiet -K "$PWD" -E "test.bat" 2> /dev/null > /dev/null); done

method=NONE
size=101376
ini size=4528
alloc=98992 bytes = 6187 paragraphs
2.184

method=BLZ
size=56320
ini size=6240
alloc=100480 bytes = 6280 paragraphs
4.762

method=LZ4
size=54784
ini size=5712
alloc=99968 bytes = 6248 paragraphs
3.240

method=SZ
size=59904
ini size=5760
alloc=100000 bytes = 6250 paragraphs
3.087

method=EXO
size=47616
ini size=5776
alloc=100016 bytes = 6251 paragraphs
5.712

method=X
size=68608
ini size=6336
alloc=362736 bytes = 22671 paragraphs
24.434

method=HS
size=57344
ini size=5504
alloc=99744 bytes = 6234 paragraphs
6.839

method=LZ
size=46592
ini size=7824
alloc=116800 bytes = 7300 paragraphs
11.539

method=LZO
size=52736
ini size=6336
alloc=100576 bytes = 6286 paragraphs
3.761
rxdos/tmp$

---
l

Complete thread:

• New kernel compression methods - ecm, 13.04.2020, 00:05 (Announce)
  • New kernel compression methods - ecm, 13.04.2020, 00:17
  • New kernel compression methods - tkchia, 13.04.2020, 14:23
    • New kernel compression methods - ecm, 13.04.2020, 18:31
      • New kernel compression methods - ecm, 13.04.2020, 18:35
      • New kernel compression methods - tkchia, 14.04.2020, 14:46
        • New kernel compression methods - fdkernpl - ecm, 14.04.2020, 17:05
        • New kernel compression methods - exeExtraBytes jump - ecm, 14.04.2020, 17:06
      • New kernel compression methods - tom, 14.04.2020, 20:06
  • New kernel compression methods - Updated lDebug lzd results - ecm, 16.04.2020, 17:29
  • New kernel compression methods - Corrections - ecm, 16.04.2020, 22:06
    • New kernel compression methods - X compressor layers - ecm, 16.04.2020, 22:22
    • New kernel compression methods - Updated lzd, new lzo - ecm, 25.04.2020, 22:23
      • New kernel compression methods - Updated lzd, new lzo - tom, 26.04.2020, 13:46
        • New kernel compression methods - UPX, 512 runs clarification - ecm, 26.04.2020, 20:17