DOS ain't dead

New kernel compression methods - UPX, 512 runs clarification (Announce)

posted by ecm  , Düsseldorf, Germany, 26.04.2020, 20:17

> what hardware is this?
> using 10 seconds to decompress 100K, or 10KB/s sounds tremendously slow.

The answer is given in the scriptlet, and in my original post describing that scriptlet:

> > The last line for each method gives the time in seconds
> > for a quick test, running the executable 512 times from
> > a single batch file,

In addition, note that even without any compression the test takes about 2.2 seconds, which presumably includes overhead from the dosemu startup as well as the repeated actual execution of the debugger or kernel (EXE mode, ie COM loader) initialisation and quitting.

The platform is an "AMD A10-7870K Radeon R7, 12 Compute Cores 4C+8G" (a quadcore CPU) running Debian testing (bullseye) amd64, with a recent dosemu2 built from its git repo.


> could you run UPX and show results to as comparison?

Sure, good idea. This is the UPX from the Debian package version 3.95-2+b1, which I believe is an UPX-UCL (FLOSS) build. I included only the tests of the uncompressed and the UPX-compressed executable.

Note that the 4.5 kB initial loader stage is dropped by UPX. Thus the filesize should be interpreted with that in mind. The "ini size" result is not meaningful for these UPX runs because UPX doesn't give us the same way to determine how large its depacker stub is. I went with the --best --ultra-brute --8086 switches. As mentioned in the other post, "The --8086 option does not appreciably alter the result." As also mentioned in that post, "I had to patch out the short jump in the "last page size" bytes of the EXE header."

The results, compared to those in the "Updated lzd, new lzo" post, have UPX as a very fast depacker, the fastest of all of them in fact. The compression ratio is competitive. The debugger is packed from 83 kB to 51 kB. However, with the size of the dropped initial loader, the value to compare should be 55 kB. The Exomizer 3 (55 kB) and LZMA-lzip (53 kB) methods beat this calculated value. The kernel is packed from 101 kB to 43 kB. This gives 47 kB as the value to compare. Again Exomizer 3 (48 kB) and LZMA-lzip (47 kB) are very close.


ldebug/tmp$ upx --version
upx 3.95
UCL data compression library 1.03
zlib data compression library 1.2.11
LZMA SDK version 4.43
Copyright (C) 1996-2018 Markus Franz Xaver Johannes Oberhumer
Copyright (C) 1996-2018 Laszlo Molnar
Copyright (C) 2000-2018 John F. Reiser
Copyright (C) 2002-2018 Jens Medoch
Copyright (C) 1995-2005 Jean-loup Gailly and Mark Adler
Copyright (C) 1999-2006 Igor Pavlov
UPX comes with ABSOLUTELY NO WARRANTY; for details type 'upx -L'.
ldebug/tmp$ hg id
24b1bbf37614 tip
ldebug/tmp$ fname="ldebug.com"; fname2="debug"; fname3="debug.big"; fnameu="ldebugu.com"; arg="/C=Q"; upcase=0; method=none; ((upcase)) && method="$(echo "$method" | tr 'a-z' 'A-Z')"; mkdir -p "$method"; cp -a ../bin/"$fnameu" "$method"/"$fname"; cp -a "$fname3" "$method/$fname2.$method"; method=upx; ((upcase)) && method="$(echo "$method" | tr 'a-z' 'A-Z')"; mkdir -p "$method"; cp -a ../bin/"$fnameu" "$method"/"$fnameu"; echo -ne "File: $method/$fnameu\n000002: 00 00 [ EB 16 ]\n" >> "$method"/"$fname2".pat; bpatch -f "$method"/"$fname2".pat; upx --best --ultra-brute --8086 -fvvvv -o "$method"/"$fname" "$method"/"$fnameu"; cp -a "$method"/"$fname" "$method"/"$fname2.$method"; for method in none upx; do ((upcase)) && method="$(echo "$method" | tr 'a-z' 'A-Z')"; echo -e "\nmethod=$method\nsize=$(stat -c %s "$method/$fname")"; echo -e "ini size=$(( $(stat -c %s "$method/$fname") - ( ($(stat -c %s "$method/$fname2.$method") + 15) / 16 * 16 ) ))"; echo -e "alloc=$(exememls "$method/$fname")"; echo "@echo off" > test.bat; for jj in $(seq 0 511); do echo "@$method\\$fname $arg"; done >> test.bat; (export TIMEFORMAT='%3R'; time dosemu -dumb -quiet -K "$PWD" -E "test.bat" 2> /dev/null > /dev/null); done
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2018
UPX 3.95        Markus Oberhumer, Laszlo Molnar & John Reiser   Aug 26th 2018

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
     82944 ->     51427   62.00%     dos/exe     ldebug.com                   

Packed 1 file.

method=none
size=82944
ini size=4496
alloc=96912 bytes = 6057 paragraphs
2.301

method=upx
size=51427
ini size=-13
alloc=97200 bytes = 6075 paragraphs
2.778
ldebug/tmp$


rxdos/tmp$ upx --version
upx 3.95
UCL data compression library 1.03
zlib data compression library 1.2.11
LZMA SDK version 4.43
Copyright (C) 1996-2018 Markus Franz Xaver Johannes Oberhumer
Copyright (C) 1996-2018 Laszlo Molnar
Copyright (C) 2000-2018 John F. Reiser
Copyright (C) 2002-2018 Jens Medoch
Copyright (C) 1995-2005 Jean-loup Gailly and Mark Adler
Copyright (C) 1999-2006 Igor Pavlov
UPX comes with ABSOLUTELY NO WARRANTY; for details type 'upx -L'.
rxdos/tmp$ hg id
d441a721725f tip
rxdos/tmp$ fname="RxDOS.COM"; fname2="RxDOS"; fname3="RxDOS.BIN"; fnameu="RxDOSU.COM"; arg="version"; upcase=1; method=none; ((upcase)) && method="$(echo "$method" | tr 'a-z' 'A-Z')"; mkdir -p "$method"; cp -a ../bin/"$fnameu" "$method"/"$fname"; cp -a "$fname3" "$method/$fname2.$method"; method=upx; ((upcase)) && method="$(echo "$method" | tr 'a-z' 'A-Z')"; mkdir -p "$method"; cp -a ../bin/"$fnameu" "$method"/"$fnameu"; echo -ne "File: $method/$fnameu\n000002: 00 00 [ EB 16 ]\n" >> "$method"/"$fname2".pat; bpatch -f "$method"/"$fname2".pat; upx --best --ultra-brute --8086 -fvvvv -o "$method"/"$fname" "$method"/"$fnameu"; cp -a "$method"/"$fname" "$method"/"$fname2.$method"; for method in none upx; do ((upcase)) && method="$(echo "$method" | tr 'a-z' 'A-Z')"; echo -e "\nmethod=$method\nsize=$(stat -c %s "$method/$fname")"; echo -e "ini size=$(( $(stat -c %s "$method/$fname") - ( ($(stat -c %s "$method/$fname2.$method") + 15) / 16 * 16 ) ))"; echo -e "alloc=$(exememls "$method/$fname")"; echo "@echo off" > test.bat; for jj in $(seq 0 511); do echo "@$method\\$fname $arg"; done >> test.bat; (export TIMEFORMAT='%3R'; time dosemu -dumb -quiet -K "$PWD" -E "test.bat" 2> /dev/null > /dev/null); done
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2018
UPX 3.95        Markus Oberhumer, Laszlo Molnar & John Reiser   Aug 26th 2018

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
    101376 ->     43133   42.55%     dos/exe     RxDOS.COM                     

Packed 1 file.

method=NONE
size=101376
ini size=4528
alloc=98992 bytes = 6187 paragraphs
2.290

method=UPX
size=43133
ini size=-3
alloc=99376 bytes = 6211 paragraphs
2.735
rxdos/tmp$

---
l

Complete thread:

• New kernel compression methods - ecm, 13.04.2020, 00:05 (Announce)
  • New kernel compression methods - ecm, 13.04.2020, 00:17
  • New kernel compression methods - tkchia, 13.04.2020, 14:23
    • New kernel compression methods - ecm, 13.04.2020, 18:31
      • New kernel compression methods - ecm, 13.04.2020, 18:35
      • New kernel compression methods - tkchia, 14.04.2020, 14:46
        • New kernel compression methods - fdkernpl - ecm, 14.04.2020, 17:05
        • New kernel compression methods - exeExtraBytes jump - ecm, 14.04.2020, 17:06
      • New kernel compression methods - tom, 14.04.2020, 20:06
  • New kernel compression methods - Updated lDebug lzd results - ecm, 16.04.2020, 17:29
  • New kernel compression methods - Corrections - ecm, 16.04.2020, 22:06
    • New kernel compression methods - X compressor layers - ecm, 16.04.2020, 22:22
    • New kernel compression methods - Updated lzd, new lzo - ecm, 25.04.2020, 22:23
      • New kernel compression methods - Updated lzd, new lzo - tom, 26.04.2020, 13:46
        • New kernel compression methods - UPX, 512 runs clarification - ecm, 26.04.2020, 20:17