X-Git-Url: https://gitweb.dragonflybsd.org/ikiwiki.git/blobdiff_plain/d007b8cb15f2658d5305f72c3929e12d6e450c54..HEAD:/hammer/index.mdwn diff --git a/hammer/index.mdwn b/hammer/index.mdwn index ba9e106a..6284caab 100644 --- a/hammer/index.mdwn +++ b/hammer/index.mdwn @@ -1,32 +1,96 @@ -# The HAMMER Filesystem +## HAMMER2 -## What is HAMMER? +HAMMER2, the successor of HAMMER, is under active development since 2012 and now the official default filesystem for DragonFly. -HAMMER is a file system written for DragonFly that provides instant crash recovery, multi-volume file systems, integrity checking, fine grained history, networked mirroring, and historical snapshots. HAMMER is the default file system for DragonFly. +### General Details + +* Block copy-on-write filesystem +* Instant recovery on mount +* Instant snapshots +* Mounted snapshots are writable +* Automatic snapshotting can be enabled at the system level via periodic scripts +* Default Periodic also does daily bulk pass on the meta-data to free space +* Automatic compression (controllable on directory recursion and per-file basis) +* Automatic de-duplication +* Future master/slave mechanism +* Utilizes a dynamic radix tree +* 64-bit hardlink counter +* 2^63 logical file size limit +* Recursive check codes to detect corruption +* Any number of pseudo-filesystem volumes for each physical hammer2 disk image (also used by snapshots). + +Because HAMMER2 is a block copy-on-write filesystem, the "atime" field is not supported and will typically just reflect local system in-memory caches or mtime. + +The radix tree is dynamic in that each entry can dynamically control how many bits it chops off. This allows small files to be contained in just one or two levels regardless of the block seek positions. The depth of the radix tree is increased as needed via a splitting mechanism, and will also be recombined if it grows smaller. All block references are 64-bit aligned-byte-indexed references and thus portable regardless of physical sector size changes between underlying block devices. + +Inodes are 1KB of which 512 bytes are used for the top-level radix tree OR 512 bytes of data. Any file less than or equal to 512 bytes stores its data directly in the inode. Files up to 256KB can be accommodated with direct inode block references. + +Directory entries are hashed (semi-sorted hash algorithm), and directly embedded in the radix table's blockref structure for maximum performance. Files with very long filenames will contain a dataref, otherwise filenames are embedded in the directory entry itself. Because directory entries are hashed, seeking and lookups are able to use a radix search and no linear scan of the directory is needed. + +The inode and directory entry structure is extremely well suited for any file size or directory size, from tiny to huge. + +Because of the block-copy-on-write nature of the filesystem, the filesystem is able to create a snapshot trivially simply by copying the volume header's root block table (4 blockref entries). The directory topology actually starts with a SUPERROOT, and volume ROOTs are directory entries under the SUPERROOT. Though the entries are actually special-cased a bit and actually part of the root inode for each filesystem root. And since physical freeing of space is handled via a bulk meta-data scan, destroying a snapshot or volume can be done simply by wiping the inode and ignoring everything under it... the next bulkfree scan will reclaim any reclaimable space. Similarly with file deletions... the top-level data blockrefs can simply be removed. The inode can simply be removed from the radix tree. + +Performance is very good. HAMMER2 uses a variable-sized block in powers of two, starting at 1KB, up to 64KB, for the last block of the file (straddling EOF). All earlier blocks in the file, if any, use 64KB blocks. The freemap is organized by domain to cluster various meta-data types together. Indirect blocks can be one of two sizes: 16KB or 64KB, allowing medium-sized files and directories to be optimally allocated. In addition, file data compression of a logical block can result in a smaller physical block. The physical layer *always* does 64KB I/O and can cluster the I/O on top of that. + +### Future and TBD Features + +* HAMMER2 implements a fat blockref in its radix tables which allows for up to a 512-bit hash or cryptographic signature. One planned future feature is blind de-duplication (using the signature for de-duplication only without double checking the data content). Current de-duplication always double-checks the data content. + +* There also fields reserved for multi-master, master-slave, and other replication and mirroring mechanisms that have not yet been implemented. Including voting and healing. + +### Automatic snapshotting in `/etc/periodic.conf` + + daily_snapshot_hammer2_enable="YES" + daily_snapshot_hammer2_dirs="/" # optional + weekly_snapshot_hammer2_enable="YES" + weekly_snapshot_hammer2_dirs="/" # optional + monthly_snapshot_hammer2_enable="YES" + monthly_snapshot_hammer2_dirs="/" # optional + +### Additional Reference Material + +For more information, you can consult the following resources: + +* [HAMMER2 Design v1 (08-Feb-2012)](https://gitweb.dragonflybsd.org/dragonfly.git/blob/05af5bd1aeaf8bf30d36fc65103097415b943700:/sys/vfs/hammer2/DESIGN), [Mailing list](http://leaf.dragonflybsd.org/mailarchive/users/2012-02/msg00020.html) +* [HAMMER2 Design v2 (14-May-2013)](https://gitweb.dragonflybsd.org/dragonfly.git/blob/1a7cfe5ae3c897f704a358fd3e556a55e430dcb1:/sys/vfs/hammer2/DESIGN) +* [HAMMER2 Design v3 (03-Apr-2015)](https://gitweb.dragonflybsd.org/dragonfly.git/blob/b93cc2e0815ec1ad6d6f8e60cc0becbdee247679:/sys/vfs/hammer2/DESIGN) +* [HAMMER2 Design v4 (09-Jul-2016)](https://gitweb.dragonflybsd.org/dragonfly.git/blob/7fece146f268d677c46bde997079860f4cf553d0:/sys/vfs/hammer2/DESIGN) +* [HAMMER2 Design v5 (24-Jul-2017)](https://gitweb.dragonflybsd.org/dragonfly.git/blob/b7910865ff18fc68fa4af19f7c7ad1b021a346be:/sys/vfs/hammer2/DESIGN) +* [HAMMER2 Design v6 (08-Dec-2018)](https://gitweb.dragonflybsd.org/dragonfly.git/blob/57614c517203403e7fe4a34c370c7151ba52c539:/sys/vfs/hammer2/DESIGN) + + +## What is HAMMER1? + +HAMMER1 was the original iteration of the hammer series of file-systems written for DragonFly. It provides instant crash recovery, multi-volume file systems, integrity checking, fine grained history/undo, networked mirroring, and historical read-only snapshots. Note that HAMMER2 is the default file system for DragonFly. ### General details -* HAMMER file systems are immediately available after a crash, and do not require fsck. -* A single HAMMER file system can be up to 1 exabyte in size, and can encompass up to 256 volumes, of which each can be up to 4 petabytes. -* Historical views of every file on the system are saved with every sync to disk. +* HAMMER file systems are immediately available after a crash. There is no fsck. +* A single HAMMER file system can be up to 1 exabyte in size, and can encompass up to 256 volumes, each of which can be up to 4 petabytes (4096 terabytes). +* HAMMER retains a fine-grained history. The state of the filesystem can be accessed live on 30-60 second boundaries without having to make explicit snapshots, up to a configurable fine-grained retention time. +* Coarse-grained history is controlled by snapshots. By default the system cron generates one snapshot a day and retains 60 days worth. Snapshots can be accessed live. +* A convenient undo command is provided for single-file history, diffs, and extractions. Snapshots may be used to access entire directory trees. +* Data and meta-data is CRC-checked for integrity. +* Data block deduplication ### Snapshots -* Snapshots of the file system can be taken at any time and in any amount. -* Snapshots take only the space needed to describe changes to disk, not the actual size of the volume. +* Snapshots of the file system can be taken at any time, with no limitations. +* Snapshots are indexed by the on-media B-Tree and are extremely storage-efficient. * Snapshots are "live", and can be accessed at any time. * Snapshot and historical data retention are controlled through a config file kept in meta-data - no manual maintenance is required for historical files. ### Backups and history -* HAMMER file systems can be split up into multiple pseudo-file systems, or PFS. -* HAMMER volumes can be continually backed up to other HAMMER volumes on a per-PFS basis. -* Backup volumes (slaves) are functionally identical to the original (master) and can be promoted to a master volume. -* Slave PFSs can retain file history independent of the master volume, saving more or less snapshots as desired. -* Backups can be on a local network or over the Internet. No minimum data speed required; the backup will catch up as bandwidth allows. -* Data is CRC-checked for integrity. - -**Note: HAMMER should be used on 50G or larger disks.** - +* HAMMER file systems can be split up into multiple pseudo-file systems, or PFSs. Snapshots and backups can be different for each individual PFS. +* HAMMER PFSs can be backed up continuously or in batch to other HAMMER PFSs, on a per-PFS basis. +* Backup PFSs (slaves) are functionally identical to the original (master) and can be promoted to a master. +* Slave PFSs can retain file history independent of the master volume's settings. +* HAMMER can efficiently stream bandwidth-controlled near-real-time backup data to slave PFSs on remote hosts. +* 1 master PFS can stream backups to any number of slave PFSs. +* Slave-to-slave mirroring streams are supported, allowing mirrors to be chained together. + For more details, please read the [hammer(5)](http://leaf.dragonflybsd.org/cgi/web-man?command=hammer§ion=5) man page. People interested in porting HAMMER to other operating systems should contact Matthew Dillon at dillon at backplane.com. + ## Documentation [[!table data=""" @@ -40,4 +104,10 @@ Document|Description """]] +### General Administrative Notes +* ***HAMMER is designed for use on storage media greater than 50G.*** Snapshots and history require significantly different space management than usual, and HAMMER will recover space based on each PFS's pruning schedule. +* By default 60-days worth of snapshots are retained; adjust this based on disk size and activity. Free space will stabilize once the snapshot limit is reached. These parameters are programmable. For example, snapshots are typically turned off on /tmp, /var/tmp, /var/crash, and /usr/obj (each of which is a PFS in a typical DragonFly installation). +* Various cleanup directives can be specified manually via [hammer(5)](http://leaf.dragonflybsd.org/cgi/web-man?command=hammer§ion=5) for situations which do not fit default expectations. +* The system cron automatically runs HAMMER cleanup functions at least once a day. + More help topics involving HAMMER can be found in the [documentation section](/docs/documentation/) of this website.