DOS ain't dead - CRITICAL OFF-by-32-KiB-BUG

CRITICAL OFF-by-32-KiB-BUG (Users)

posted by Rugxulo , Usono, 05.04.2013, 19:15

(quoting Eric Auer):

> I just had an idea ... what if a pointer or counter is updated in a
> non- atomic way where some block count or the upper N bits could get
> updated concurrently while multi-threading, somehow skipping 32 k of data?

I don't know all the details. LZMA2 is supposed to be better for multithreaded compression. Not sure about decompression, IIRC it sounded like it only normally used like maybe two threads, one for I/O and one for disk writes.

EDIT:
LZMA2 ... Better multithreading support. If you compress big file, LZMA2 can split that file to chunks and compress these chunks in multiple threads. LZMA2 uses: 1 thread for each chunk in x1 and x3 modes; and 2 threads for each chunk in x5, x7 and x9 modes. If LZMA2 is set to use only such number of threads required for one chunk, it doesn't split stream to chunks. So you can get different compression ratio for different number of threads. You can get the best compression ratio, when you use 1 or 2 threads.

(LZMA2 is default in latest alphas, but I think all our older versions default to original LZMA only.)

Maybe "-mtt" can configure how many threads, dunno, lemme check docs ...

(from p7zip_9.20.1/DOCS/MANUAL/switches/method.htm): mt=[off | on | {N}] Sets multithread mode. If you have a multiprocessor or multicore system, you can get a speed increase with this switch. This option affects only compression (with any method) and decompression of BZip2 streams. Each thread in the multithread mode uses 32 MB of RAM for buffering. If you specify {N}, 7-Zip tries to use N threads.

EDIT: Apparently this actually means "-mmt=whatever". Try "7za b -mmt=2" on a four-core machine, etc. etc.

Dunno if the docs are truly consistent with latest code. But, if any of you can reliably reproduce the failure (and please mention which OS build and version and which HX version and which OS tested and similar stats), please try again with "-mmt=off" or similar. (And of course report bugs to Igor Pavlov or p7zip dude "myspace" on SourceForge.)

Post reply

Complete thread:

CRITICAL OFF-by-32-KiB-BUG - DOS386, 27.03.2013, 09:10 (Users)
- CRITICAL OFF-by-32-KiB-BUG - RayeR, 28.03.2013, 02:16
  - CRITICAL OFF-by-32-KiB-BUG - DOS386, 29.03.2013, 09:26
    - CRITICAL OFF-by-32-KiB-BUG - RayeR, 29.03.2013, 11:02
      - CRITICAL OFF-by-32-KiB-BUG - DOS386, 29.03.2013, 11:19
        CRITICAL OFF-by-32-KiB-BUG (threads) - DOS386, 31.03.2013, 17:42
        CRITICAL OFF-by-32-KiB-BUG (threads) - RayeR, 03.04.2013, 12:35
        CRITICAL OFF-by-32-KiB-BUG (threads) - Zyzzle, 04.04.2013, 04:17
        CRITICAL OFF-by-32-KiB-BUG (threads) - RayeR, 04.04.2013, 13:51
        CRITICAL OFF-by-32-KiB-BUG (threads) - DOS386, 04.04.2013, 04:37
- CRITICAL OFF-by-32-KiB-BUG - Rugxulo, 05.04.2013, 19:15
  - CRITICAL OFF-by-32-KiB-BUG | Eric | LZMA2 - DOS386, 10.04.2013, 12:00