Back to home page

DOS ain't dead

Forum index page

Log in | Register

Search:  
Back to the forum
Board view  Mix view

Snappy compression - script to build a compressed executable (Announce)

posted by ecm Homepage E-mail, Düsseldorf, Germany, 08.02.2025, 18:02

> Hi ECM,
>
> > > https://en.wikipedia.org/wiki/Snappy_(compression) is a fast LZ style
> > > algorithm without bitstrings.
> >
> > I have a Snappy depacker in inicomp already:
> > https://hg.pushbx.org/ecm/inicomp/file/536526f7c49c/snappy.asm
>
> I was curious about the binary size of that, but neither NASM nor the TASM
> mode of YASM want to assemble it, so I just have to ask you how big your
> Snappy depacker is :-)
In an lDebug source directory, running a command like INICOMP_METHOD=snappy use_build_compress_only=1 build_name=cdebugx ./mak.sh -D_PM=1 -D_DEBUG -D_DEBUG_COND -D_DEFAULTSHOWSIZE will pack the (existing) .big executable image using the Snappy method. (Drop "use_build_compress_only" or set it to zero to actually build the debugger first.) The final inicomp and iniload stages display messages like these:

    148 progress dots, method           snappy
../../inicomp/inicomp.asm:2040: warning: inisz: 1952 bytes used for depacker [-w+us
er]
../../ldosboot/iniload.asm:864: warning: 1 bytes in front of ms7_entry [-w+user]
../../ldosboot/iniload.asm:1271: warning: 9 bytes in front of ldos_entry [-w+user]
../../ldosboot/iniload.asm:1687: warning: 10 bytes in front of end [-w+user]
../../ldosboot/iniload.asm:1750: warning: 428 bytes in front of end2 [-w+user]
  130560 bytes ( 90.10%), method           snappy
Note: Method snappy selected.
-rw-r--r-- 1 130560  ../bin/lcdebugx.com

So the entire triple-mode (bootloader, DOS device driver, DOS executable) inicomp depacker stage, including progress display, takes 1952 bytes.

> The unpacker also seems to rely on normalise_dssi_pointer,
> ..._both_pointers, ...pointer_with_displacement_bxcx, pointer_to_linear,
> check_pointers_not_overlapping, disp_al, disp_al_counter and
> disp_al_for_progress to be provided elsewhere.
Yes, these are all defined in inicomp.asm, the main assembly source that includes the method-specific source text file and contains the code to call the depacker. The canonical source of truth on how to build is the mak.sh script, either the one from lDebug or the one from the kernwrap repo as used by lDOS/Enhanced DR-DOS and lDOS/MS-DOS.

I'll step you through the steps it takes to assemble the Snappy depacker:

* https://hg.pushbx.org/ecm/ldebug/file/81124309228f/source/mak.sh#l586 this sets some variables for the Snappy method, importantly it tells you inicomp.asm requires the -D_SNAPPY switch to choose the correct depacker. It also tells us the packer executable snzip is used (by default).
* https://github.com/kubo/snzip this is the repo that I use to pack.
* https://hg.pushbx.org/ecm/ldebug/file/81124309228f/source/mak.sh#l680 here we actually call the packer, as in snzip -ck file.big > file.sz
* https://hg.pushbx.org/ecm/ldebug/file/81124309228f/source/mak.sh#l739 this builds the "test program" executable image, which is the MZ executable stage used in an iniload wrapper stage to create the test program. The test program can tell us three things: Whether the depacker works at all and produces the exact expected output, how much memory the depacker needs to succeed, and how many progress units (dots) there are for this specific compressed image. As a fourth choice (not used by default) it can also run a benchmark (repeatedly depacking) which can be used as a speed test, usually with at least 16 runs and up to as many as 1024 to reduce the fraction of the time spent in startup cost. _PAYLOAD_FILE must refer to the compressed payload file.
* https://hg.pushbx.org/ecm/ldebug/file/81124309228f/source/mak.sh#l936 here we build the actual final inicomp stage. Again -D_SNAPPY must be passed and _PAYLOAD_FILE specifies the compressed payload file. Some more options are used as well. $inicomp_def_progress $inicomp_def_progressamount are used to pass the amount of progress dots determined from running the test program.
* After inicomp.asm has been assembled, it is used as the executable image payload to lDOS boot's iniload.asm, which also supports triple-mode execution.

> And while I am at it, how much RAM does it need? Apparently just the output
> buffer and less than 40 bytes of stack "lvars"?
Yes, there's the lframe variables (these are specified using macros that are defined in lmacros2.mac of my lDOS macro collection) and normal stack usage during the run. The inicomp stage usually supports overlapping buffers. The output buffer starts in memory below the packed payload. While depacking, it is valid for the depacked image to grow into the space used by the packed payload, but it must never cross into parts of the payload not yet consumed by the depacker. The exact boundary of how much memory is needed is determined by the test program, and is encoded into the MZ executable's minimum allocation field by the scripting. (The tellsize program is also used on the debugger .big image to determine how much memory is needed for the debugger init, and the bigger of the two sizes is encoded into the final executable.)

---
l

Post reply
 

Complete thread:

Back to the forum
Board view  Mix view
22287 Postings in 2062 Threads, 399 registered users, 197 users online (0 registered, 197 guests)
DOS ain't dead | Admin contact
RSS Feed
powered by my little forum