hammer2 - Merge Daniel Flores's HAMMER2 GSOC project into the main tree
authorMatthew Dillon <dillon@apollo.backplane.com>
Thu, 19 Sep 2013 02:56:07 +0000 (19:56 -0700)
committerMatthew Dillon <dillon@apollo.backplane.com>
Thu, 19 Sep 2013 02:56:07 +0000 (19:56 -0700)
commit355d67fcd81e0a7b17007d691bb00bdd151f3d28
treea608516960d1aa099f9b631c2d8e22945d52124f
parent7e3f82112b228b5819b5885e02e3a1c2b09f50e8
hammer2 - Merge Daniel Flores's HAMMER2 GSOC project into the main tree

* This merge contains work primarily by Daniel Flores, and also by
  Matthew Dillon.  Daniel's work is focused around adding the compression
  infrastructure while my work was focused around changing HAMMER2's
  I/O infrastructure to work with the compression code.

  Daniel's work involved adding the compression functions and heuristics,
  modifying mainly vnops and vfsops to use them, adding the new buffer
  cache write thread, and adding the new hammer2 utility directives and
  related ioctls.

  My work involved changing the H2 I/O infrastructure to always double-buffer
  (i.e. logical buffers vs device buffers) because device buffers can now
  wind up being a different size than the related logical buffers.  I also
  had to make changes to the hammer2_mount and hammer2_pfsmount mechanics
  and other things to prevent deadlocks.

    Daniel's Work

* Add the hammer2 setcomp directive which sets the compression mode on a
  directory or file.  If applie to a directory, the compression mode is
  inherited by any new files or directories created under the directory.
  Pre-existing subdirectories and files are not affected.

  The directive has a recursive option to recurse down and set the mode on
  everything underneath.

* Add wthread_bioq and related fields to hammer2_mount to support a
  buffer cache buffer writing thread.  This thread is responsible for
  calculating compression sizes, allocating device buffer blocks, and
  compressing logical buffers into the device buffers.

* Implement HAMMER2_COMP_AUTOZERO, HAMMER2_COMP_LZ4, and HAMMER2_COMP_ZLIB
  compression modes.  AUTOZERO is the zero-block detection code.  LZ4 will
  do zero-block-detection and LZ4 otherwise, and ZLIB will do
  zero-block-detection and gzip otherwise.

  This work entails a ton of new files imported from the LZ4 and ZLIB
  projects plus lots of wiring.

  The new files had to be cleaned up considerably, as well, since they
  were originally intended for userland.

* Move synchronous device buffer handling out of hammer2_vop_write() and
  into the support thread.  Numerous procedures were moved out of
  hammer2_vnops.c and into hammer2_vfsops.c as well.

  This greatly simplifies hammer2_vop_write() as well as the truncate and
  extend code, and improves the critical-path performance for write()
  (at least until the buffer cache fills up or gets too far behind).

* Implement semi-synchronous decompression callbacks for read I/O and
  read-ahead I/O.

* Add HAMMER2IOC_INODE_COMP_REC_SET and HAMMER2IOC_INODE_COMP_REC_SET2
  ioctls to support the setcomp directive.

  Matthew's Work

* The hammer2_inode copies additional fields from the inode data, allowing
  the inode data to be deallocated after use.

* Due to the way the buffer cache now operates, multiple deletions of the
  same chain key can occur within the same transaction.  Adjust the RBTREE
  compare code to handle the case.

* Track chain structure use on a pfsmount-by-pfsmount basis for current
  and future management of the kmalloc pools used by hammer2.

* Rework the way inodes are locked to track chain modifications.

* Rewrite hammer2_chain_delete_duplicate().

* Rewrite hammer2_trans_init() and the flush code primarily to fix
  deadlocks in the flush synchronization mechanics.

* Interlock very low level chain operations with a spin lock instead
  of the full-blown chain lock to deal with potential deadlocks and
  fix a few SMP races.

* For the moment make all logical buffers 64KB.  Not efficient for small
  files and will be changed back at some point but necessary for efficient
  compression at the moment.

* Implement an asynchronous buffer cache callback feature.

* Use a localized in-hammer2_inode-structure size field for frontend
  operations, including extend and truncate, to remove confusion against
  backend flushes.  This way the inode data can be modified only during
  the flush and not before where it might cause confusion with previously
  staged flushes.
37 files changed:
sbin/hammer2/Makefile
sbin/hammer2/cmd_setcomp.c [new file with mode: 0644]
sbin/hammer2/cmd_stat.c
sbin/hammer2/hammer2.h
sbin/hammer2/main.c
sbin/hammer2/print_inode.c [new file with mode: 0644]
sys/vfs/hammer2/Makefile
sys/vfs/hammer2/hammer2.h
sys/vfs/hammer2/hammer2_chain.c
sys/vfs/hammer2/hammer2_disk.h
sys/vfs/hammer2/hammer2_flush.c
sys/vfs/hammer2/hammer2_freemap.c
sys/vfs/hammer2/hammer2_inode.c
sys/vfs/hammer2/hammer2_ioctl.c
sys/vfs/hammer2/hammer2_ioctl.h
sys/vfs/hammer2/hammer2_lz4.c [new file with mode: 0644]
sys/vfs/hammer2/hammer2_lz4.h [new file with mode: 0644]
sys/vfs/hammer2/hammer2_lz4_encoder.h [new file with mode: 0644]
sys/vfs/hammer2/hammer2_subr.c
sys/vfs/hammer2/hammer2_vfsops.c
sys/vfs/hammer2/hammer2_vnops.c
sys/vfs/hammer2/zlib/hammer2_zlib.h [new file with mode: 0644]
sys/vfs/hammer2/zlib/hammer2_zlib_adler32.c [new file with mode: 0644]
sys/vfs/hammer2/zlib/hammer2_zlib_deflate.c [new file with mode: 0644]
sys/vfs/hammer2/zlib/hammer2_zlib_deflate.h [new file with mode: 0644]
sys/vfs/hammer2/zlib/hammer2_zlib_inffast.c [new file with mode: 0644]
sys/vfs/hammer2/zlib/hammer2_zlib_inffast.h [new file with mode: 0644]
sys/vfs/hammer2/zlib/hammer2_zlib_inffixed.h [new file with mode: 0644]
sys/vfs/hammer2/zlib/hammer2_zlib_inflate.c [new file with mode: 0644]
sys/vfs/hammer2/zlib/hammer2_zlib_inflate.h [new file with mode: 0644]
sys/vfs/hammer2/zlib/hammer2_zlib_inftrees.c [new file with mode: 0644]
sys/vfs/hammer2/zlib/hammer2_zlib_inftrees.h [new file with mode: 0644]
sys/vfs/hammer2/zlib/hammer2_zlib_trees.c [new file with mode: 0644]
sys/vfs/hammer2/zlib/hammer2_zlib_trees.h [new file with mode: 0644]
sys/vfs/hammer2/zlib/hammer2_zlib_zconf.h [new file with mode: 0644]
sys/vfs/hammer2/zlib/hammer2_zlib_zutil.c [new file with mode: 0644]
sys/vfs/hammer2/zlib/hammer2_zlib_zutil.h [new file with mode: 0644]