DOS ain't dead - New kernel compression methods

New kernel compression methods (Announce)

posted by ecm , Düsseldorf, Germany, 13.04.2020, 18:31

> > With enough contiguous low memory area space
> > permitting, my decompressors all handle compressed and uncompressed file
> > lengths exceeding 64 KiB well. (I think the FreeDOS kernel compression
> > doesn't properly allow that.)
>
> Thanks for your work. From what I see, the FreeDOS kernel uses UPX, which
> has no problems with uncompressed kernels larger than 64 KiB. However, the
> stub that is added after UPX-ing assumes that the compressed kernel
> is smaller than 64 KiB.
>
> (The sizes for my kernel build: 77,180 bytes when uncompressed (.exe),
> 52,043 when compressed (as .sys).)

Yep, something like that. I don't remember the exact details but I came away with the impression it won't work. But FreeDOS isn't alone in thinking its kernels should be smaller than 64 KiB.

> So I would guess that any compression method that compresses even better
> than UPX should not have much of a problem with the FreeDOS kernel.

By the way, you can put a FreeDOS kernel (better to use an uncompressed one then) into ldosboot's fdkernpl stage, then can use that payload for inicomp. You'd use _IMAGE_EXE=0 then because the FreeDOS kernel cannot be entered as an .EXE file. However, the FreeDOS kernel entrypoint (and the lDOS/RxDOS and PC-DOS/MS-DOS 6/MS-DOS 7 kernel entrypoints) would still work. The only thing that that does not support is the kernel CONFIG section expected fixed at the start of the file by FreeDOS's kernel configuration (which is part of the FreeDOS SYS program).

Funny, I don't actually know how much UPX would compress things. So I made a test. I had to patch out the short jump in the "last page size" bytes of the EXE header. (These are used for the MS-DOS 6.x and FreeDOS kernel entrypoints.) The invalid value of the short jump instruction in that field otherwise leads to "CantPackException: exe header corrupted". (Obviously, packing the kernel file with UPX without special handling makes it no longer work as a kernel file.)

Here are the results. They aren't entirely fair to my compression stage because my files have not only the depacker in the uncompressed initial part but also the iniload loader stage, which according to my OP is 4944 bytes for the uncompressed ldebugu.com file. So the 51 kB really should be read as 51 + 5 kB, which is indeed beaten by the Exomizer 3 and LZMA-lzip methods. The --8086 option does not appreciably alter the result. (My depackers are all 8086 safe.)

testupx$ upx --version; hg id; for file in *.pat; do echo "=== $file"; cat "$file"; done; echo "==="; cp -at . ../ldebug.com ../ldebugu.com ../debug.com; bpatch -f ldebug.pat; bpatch -f ldebugu.pat; ls -lgG *.com; for x in --8086 ""; do for y in debug.com ldebugu.com ldebug.com; do upx --best --ultra-brute $x -fvvvv -o $y.upx $y; done; done upx 3.95 UCL data compression library 1.03 zlib data compression library 1.2.11 LZMA SDK version 4.43 Copyright (C) 1996-2018 Markus Franz Xaver Johannes Oberhumer Copyright (C) 1996-2018 Laszlo Molnar Copyright (C) 2000-2018 John F. Reiser Copyright (C) 2002-2018 Jens Medoch Copyright (C) 1995-2005 Jean-loup Gailly and Mark Adler Copyright (C) 1999-2006 Igor Pavlov UPX comes with ABSOLUTELY NO WARRANTY; for details type 'upx -L'. a6c6df3e2820 === ldebug.pat File: ldebug.com 000002: 00 00 [ EB 16 ] === ldebugu.pat File: ldebugu.com 000002: 00 00 [ EB 16 ] === -rw-r--r-- 1 78032 Apr 13 17:59 debug.com -rw-r--r-- 1 53760 Apr 13 17:59 ldebug.com -rw-r--r-- 1 82944 Apr 13 17:59 ldebugu.com Ultimate Packer for eXecutables Copyright (C) 1996 - 2018 UPX 3.95 Markus Oberhumer, Laszlo Molnar & John Reiser Aug 26th 2018 File size Ratio Format Name -------------------- ------ ----------- ----------- 78032 -> 51015 65.38% dos/exe debug.com.upx Packed 1 file. Ultimate Packer for eXecutables Copyright (C) 1996 - 2018 UPX 3.95 Markus Oberhumer, Laszlo Molnar & John Reiser Aug 26th 2018 File size Ratio Format Name -------------------- ------ ----------- ----------- 82944 -> 51070 61.57% dos/exe ldebugu.com.upx Packed 1 file. Ultimate Packer for eXecutables Copyright (C) 1996 - 2018 UPX 3.95 Markus Oberhumer, Laszlo Molnar & John Reiser Aug 26th 2018 File size Ratio Format Name -------------------- ------ ----------- ----------- upx: ldebug.com: NotCompressibleException Packed 1 file: 0 ok, 1 error. Ultimate Packer for eXecutables Copyright (C) 1996 - 2018 UPX 3.95 Markus Oberhumer, Laszlo Molnar & John Reiser Aug 26th 2018 File size Ratio Format Name -------------------- ------ ----------- ----------- 78032 -> 51004 65.36% dos/exe debug.com.upx Packed 1 file. Ultimate Packer for eXecutables Copyright (C) 1996 - 2018 UPX 3.95 Markus Oberhumer, Laszlo Molnar & John Reiser Aug 26th 2018 File size Ratio Format Name -------------------- ------ ----------- ----------- 82944 -> 51059 61.56% dos/exe ldebugu.com.upx Packed 1 file. Ultimate Packer for eXecutables Copyright (C) 1996 - 2018 UPX 3.95 Markus Oberhumer, Laszlo Molnar & John Reiser Aug 26th 2018 File size Ratio Format Name -------------------- ------ ----------- ----------- upx: ldebug.com: NotCompressibleException Packed 1 file: 0 ok, 1 error. testupx$

---
l

Post reply

Complete thread:

New kernel compression methods - ecm, 13.04.2020, 00:05 (Announce)
- New kernel compression methods - ecm, 13.04.2020, 00:17
- New kernel compression methods - tkchia, 13.04.2020, 14:23
  - New kernel compression methods - ecm, 13.04.2020, 18:31
    - New kernel compression methods - ecm, 13.04.2020, 18:35
    - New kernel compression methods - tkchia, 14.04.2020, 14:46
      - New kernel compression methods - fdkernpl - ecm, 14.04.2020, 17:05
      - New kernel compression methods - exeExtraBytes jump - ecm, 14.04.2020, 17:06
    - New kernel compression methods - tom, 14.04.2020, 20:06
- New kernel compression methods - Updated lDebug lzd results - ecm, 16.04.2020, 17:29
- New kernel compression methods - Corrections - ecm, 16.04.2020, 22:06
  - New kernel compression methods - X compressor layers - ecm, 16.04.2020, 22:22
  - New kernel compression methods - Updated lzd, new lzo - ecm, 25.04.2020, 22:23
    - New kernel compression methods - Updated lzd, new lzo - tom, 26.04.2020, 13:46
      - New kernel compression methods - UPX, 512 runs clarification - ecm, 26.04.2020, 20:17