Improve performance in a few key areas:
authorsobomax <sobomax@FreeBSD.org>
Sat, 23 Apr 2016 07:23:43 +0000 (07:23 +0000)
committersobomax <sobomax@FreeBSD.org>
Sat, 23 Apr 2016 07:23:43 +0000 (07:23 +0000)
commit5be3733785c9b1121d206292b84efb473e6f227f
treed37338a897dafbbdef6403827c5d6ba1242ea9eb
parent577d7115d64ea2b414c8ea52cc5c956b38c3f93f
Improve performance in a few key areas:

 o Split the compression across several worker threads. By default, "several"
   matches number of CPUs, capped at 24 for sanity when running on a very big
   hardwares. Provide option to set that number manually;

 o Fix bug inherited from the mkulzma (R.I.P) which degraded already slow LZMA
   compression even further by calling function to release compression state
   after processing each block.

   It is neither documented as required nor actually required by the LZMA
   library. This caused spree of system calls to release memory and then map
   it again for every block. LZMA compression is more than 2x faster after this
   change alone;

 o Record time it takes to do compression and report throughput achieved.

 o Add simple first-level 256 entry hash table for de-dup code, so it's not
   becoming a bottleneck at big files.
21 files changed:
usr.bin/mkuzip/Makefile
usr.bin/mkuzip/mkuz_blk.c [copied from usr.bin/mkuzip/mkuz_zlib.h with 80% similarity]
usr.bin/mkuzip/mkuz_blk.h [copied from usr.bin/mkuzip/mkuz_zlib.h with 78% similarity]
usr.bin/mkuzip/mkuz_blk_chain.h [copied from usr.bin/mkuzip/mkuz_blockcache.h with 88% similarity]
usr.bin/mkuzip/mkuz_blockcache.c
usr.bin/mkuzip/mkuz_blockcache.h
usr.bin/mkuzip/mkuz_cfg.h [copied from usr.bin/mkuzip/mkuz_blockcache.h with 87% similarity]
usr.bin/mkuzip/mkuz_conveyor.c [new file with mode: 0644]
usr.bin/mkuzip/mkuz_conveyor.h [copied from usr.bin/mkuzip/mkuz_blockcache.h with 73% similarity]
usr.bin/mkuzip/mkuz_format.h [copied from usr.bin/mkuzip/mkuz_blockcache.h with 80% similarity]
usr.bin/mkuzip/mkuz_fqueue.c [new file with mode: 0644]
usr.bin/mkuzip/mkuz_fqueue.h [copied from usr.bin/mkuzip/mkuz_zlib.h with 64% similarity]
usr.bin/mkuzip/mkuz_lzma.c
usr.bin/mkuzip/mkuz_lzma.h
usr.bin/mkuzip/mkuz_time.c [copied from usr.bin/mkuzip/mkuzip.h with 82% similarity]
usr.bin/mkuzip/mkuz_time.h [copied from usr.bin/mkuzip/mkuzip.h with 84% similarity]
usr.bin/mkuzip/mkuz_zlib.c
usr.bin/mkuzip/mkuz_zlib.h
usr.bin/mkuzip/mkuzip.8
usr.bin/mkuzip/mkuzip.c
usr.bin/mkuzip/mkuzip.h