hammer - Add tunable vfs.hammer.skip_redo

[dragonfly.git] / share / man / man5 / hammer.5

.\"
.\" Copyright (c) 2008
.\"	The DragonFly Project.  All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions
.\" are met:
.\"
.\" 1. Redistributions of source code must retain the above copyright
.\"    notice, this list of conditions and the following disclaimer.
.\" 2. Redistributions in binary form must reproduce the above copyright
.\"    notice, this list of conditions and the following disclaimer in
.\"    the documentation and/or other materials provided with the
.\"    distribution.
.\" 3. Neither the name of The DragonFly Project nor the names of its
.\"    contributors may be used to endorse or promote products derived
.\"    from this software without specific, prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
.\" FOR A PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE
.\" COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
.\" INCIDENTAL, SPECIAL, EXEMPLARY OR CONSEQUENTIAL DAMAGES (INCLUDING,
.\" BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
.\" LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
.\" AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
.\" OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
.\" OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
.\" SUCH DAMAGE.
.\"
.Dd August 14, 2012
.Dt HAMMER 5
.Os
.Sh NAME
.Nm HAMMER
.Nd HAMMER file system
.Sh SYNOPSIS
To compile this driver into the kernel,
place the following line in your
kernel configuration file:
.Bd -ragged -offset indent
.Cd "options HAMMER"
.Ed
.Pp
Alternatively, to load the driver as a
module at boot time, place the following line in
.Xr loader.conf 5 :
.Bd -literal -offset indent
hammer_load="YES"
.Ed
.Pp
To mount via
.Xr fstab 5 :
.Bd -literal -offset indent
/dev/ad0s1d[:/dev/ad1s1d:...]	/mnt hammer rw 2 0
.Ed
.Sh DESCRIPTION
The
.Nm
file system provides facilities to store file system data onto disk devices
and is intended to replace
.Xr ffs 5
as the default file system for
.Dx .
.Pp
Among its features are instant crash recovery,
large file systems spanning multiple volumes,
data integrity checking,
data deduplication,
fine grained history retention and snapshots,
pseudo-filesystems (PFSs),
mirroring capability and
unlimited number of files and links.
.Pp
All functions related to managing
.Nm
file systems are provided by the
.Xr newfs_hammer 8 ,
.Xr mount_hammer 8 ,
.Xr hammer 8 ,
.Xr sysctl 8 ,
.Xr chflags 1 ,
and
.Xr undo 1
utilities.
.Pp
For a more detailed introduction refer to the paper and slides listed in the
.Sx SEE ALSO
section.
For some common usages of
.Nm
see the
.Sx EXAMPLES
section below.
.Pp
Description of
.Nm
features:
.Ss Instant Crash Recovery
After a non-graceful system shutdown,
.Nm
file systems will be brought back into a fully coherent state
when mounting the file system, usually within a few seconds.
.Pp
In the unlikely case
.Nm
mount fails due redo recovery (stage 2 recovery) being corrupted, a
workaround to skip this stage can be applied by setting the following tunable:
.Bd -literal -offset indent
vfs.hammer.skip_redo=<value>
.Ed
.Pp
Possible values are:
.Bl -tag -width indent
.It 0
Run redo recovery normally and fail to mount in the case of error (default).
.It 1
Run redo recovery but continue mounting if an error appears.
.It 2
Completely bypass redo recovery.
.El
.Pp
Related commands:
.Xr mount_hammer 8
.Ss Large File Systems & Multi Volume
A
.Nm
file system can be up to 1 Exabyte in size.
It can span up to 256 volumes,
each volume occupies a
.Dx
disk slice or partition, or another special file,
and can be up to 4096 TB in size.
Minimum recommended
.Nm
file system size is 50 GB.
For volumes over 2 TB in size
.Xr gpt 8
and
.Xr disklabel64 8
normally need to be used.
.Pp
Related
.Xr hammer 8
commands:
.Cm volume-add ,
.Cm volume-del ,
.Cm volume-list ;
see also
.Xr newfs_hammer 8
.Ss Data Integrity Checking
.Nm
has high focus on data integrity,
CRC checks are made for all major structures and data.
.Nm
snapshots implements features to make data integrity checking easier:
The atime and mtime fields are locked to the ctime
for files accessed via a snapshot.
The
.Fa st_dev
field is based on the PFS
.Ar shared-uuid
and not on any real device.
This means that archiving the contents of a snapshot with e.g.\&
.Xr tar 1
and piping it to something like
.Xr md5 1
will yield a consistent result.
The consistency is also retained on mirroring targets.
.Ss Data Deduplication
To save disk space data deduplication can be used.
Data deduplication will identify data blocks which occur multiple times
and only store one copy, multiple reference will be made to this copy.
.Pp
Related
.Xr hammer 8
commands:
.Cm dedup ,
.Cm dedup-simulate ,
.Cm cleanup ,
.Cm config
.Ss Transaction IDs
The
.Nm
file system uses 64-bit transaction ids to refer to historical
file or directory data.
Transaction ids used by
.Nm
are monotonically increasing over time.
In other words:
when a transaction is made,
.Nm
will always use higher transaction ids for following transactions.
A transaction id is given in hexadecimal format
.Li 0x016llx ,
such as
.Li 0x00000001061a8ba6 .
.Pp
Related
.Xr hammer 8
commands:
.Cm snapshot ,
.Cm snap ,
.Cm snaplo ,
.Cm snapq ,
.Cm snapls ,
.Cm synctid
.Ss History & Snapshots
History metadata on the media is written with every sync operation, so that
by default the resolution of a file's history is 30-60 seconds until the next
prune operation.
Prior versions of files and directories are generally accessible by appending
.Ql @@
and a transaction id to the name.
The common way of accessing history, however, is by taking snapshots.
.Pp
Snapshots are softlinks to prior versions of directories and their files.
Their data will be retained across prune operations for as long as the
softlink exists.
Removing the softlink enables the file system to reclaim the space
again upon the next prune & reblock operations.
In
.Nm
Version 3+ snapshots are also maintained as file system meta-data.
.Pp
Related
.Xr hammer 8
commands:
.Cm cleanup ,
.Cm history ,
.Cm snapshot ,
.Cm snap ,
.Cm snaplo ,
.Cm snapq ,
.Cm snaprm ,
.Cm snapls ,
.Cm config ,
.Cm viconfig ;
see also
.Xr undo 1
.Ss Pruning & Reblocking
Pruning is the act of deleting file system history.
By default only history used by the given snapshots
and history from after the latest snapshot will be retained.
By setting the per PFS parameter
.Cm prune-min ,
history is guaranteed to be saved at least this time interval.
All other history is deleted.
Reblocking will reorder all elements and thus defragment the file system and
free space for reuse.
After pruning a file system must be reblocked to recover all available space.
Reblocking is needed even when using the
.Cm nohistory
.Xr mount_hammer 8
option or
.Xr chflags 1
flag.
.Pp
Related
.Xr hammer 8
commands:
.Cm cleanup ,
.Cm snapshot ,
.Cm prune ,
.Cm prune-everything ,
.Cm rebalance ,
.Cm reblock ,
.Cm reblock-btree ,
.Cm reblock-inodes ,
.Cm reblock-dirs ,
.Cm reblock-data
.Ss Pseudo-Filesystems (PFSs)
A pseudo-filesystem, PFS for short, is a sub file system in a
.Nm
file system.
Each PFS has independent inode numbers.
All disk space in a
.Nm
file system is shared between all PFSs in it,
so each PFS is free to use all remaining space.
A
.Nm
file system supports up to 65536 PFSs.
The root of a
.Nm
file system is PFS# 0, it is called the root PFS and is always a master PFS.
.Pp
A PFS can be either master or slave.
Slaves are always read-only,
so they can't be updated by normal file operations, only by
.Xr hammer 8
operations like mirroring and pruning.
Upgrading slaves to masters and downgrading masters to slaves are supported.
.Pp
It is recommended to use a
.Nm null
mount to access a PFS, except for root PFS;
this way no tools are confused by the PFS root being a symlink
and inodes not being unique across a
.Nm
file system.
.Pp
Many
.Xr hammer 8
operations operates per PFS,
this includes mirroring, offline deduping, pruning, reblocking and rebalancing.
.Pp
Related
.Xr hammer 8
commands:
.Cm pfs-master ,
.Cm pfs-slave ,
.Cm pfs-status ,
.Cm pfs-update ,
.Cm pfs-destroy ,
.Cm pfs-upgrade ,
.Cm pfs-downgrade ;
see also
.Xr mount_null 8
.Ss Mirroring
Mirroring is copying of all data in a file system, including snapshots
and other historical data.
In order to allow inode numbers to be duplicated on the slaves
.Nm
mirroring feature uses PFSs.
A master or slave PFS can be mirrored to a slave PFS.
I.e.\& for mirroring multiple slaves per master are supported,
but multiple masters per slave are not.
.Pp
Related
.Xr hammer 8
commands:
.Cm mirror-copy ,
.Cm mirror-stream ,
.Cm mirror-read ,
.Cm mirror-read-stream ,
.Cm mirror-write ,
.Cm mirror-dump
.Ss Fsync Flush Modes
The
.Nm
file system implements several different
.Fn fsync
flush modes, the mode used is set via the
.Va vfs.hammer.flush_mode
sysctl, see
.Xr hammer 8
for details.
.Ss Unlimited Number of Files and Links
There is no limit on the number of files or links in a
.Nm
file system, apart from available disk space.
.Ss NFS Export
.Nm
file systems support NFS export.
NFS export of PFSs is done using
.Nm null
mounts (for file/directory in root PFS
.Nm null
mount is not needed).
For example, to export the PFS
.Pa /hammer/pfs/data ,
create a
.Nm null
mount, e.g.\& to
.Pa /hammer/data
and export the latter path.
.Pp
Don't export a directory containing a PFS (e.g.\&
.Pa /hammer/pfs
above).
Only
.Nm null
mount for PFS root
(e.g.\&
.Pa /hammer/data
above) should be exported (subdirectory may be escaped if exported).
.Ss File System Versions
As new features have been introduced to
.Nm
a version number has been bumped.
Each
.Nm
file system has a version, which can be upgraded to support new features.
.Pp
Related
.Xr hammer 8
commands:
.Cm version ,
.Cm version-upgrade ;
see also
.Xr newfs_hammer 8
.Sh EXAMPLES
.Ss Preparing the File System
To create and mount a
.Nm
file system use the
.Xr newfs_hammer 8
and
.Xr mount_hammer 8
commands.
Note that all
.Nm
file systems must have a unique name on a per-machine basis.
.Bd -literal -offset indent
newfs_hammer -L HOME /dev/ad0s1d
mount_hammer /dev/ad0s1d /home
.Ed
.Pp
Similarly, multi volume file systems can be created and mounted by
specifying additional arguments.
.Bd -literal -offset indent
newfs_hammer -L MULTIHOME /dev/ad0s1d /dev/ad1s1d
mount_hammer /dev/ad0s1d /dev/ad1s1d /home
.Ed
.Pp
Once created and mounted,
.Nm
file systems need periodic clean up making snapshots, pruning and reblocking,
in order to have access to history and file system not to fill up.
For this it is recommended to use the
.Xr hammer 8
.Cm cleanup
metacommand.
.Pp
By default,
.Dx
is set up to run
.Nm hammer Cm cleanup
nightly via
.Xr periodic 8 .
.Pp
It is also possible to perform these operations individually via
.Xr crontab 5 .
For example, to reblock the
.Pa /home
file system every night at 2:15 for up to 5 minutes:
.Bd -literal -offset indent
15 2 * * * hammer -c /var/run/HOME.reblock -t 300 reblock /home \e
	>/dev/null 2>&1
.Ed
.Ss Snapshots
The
.Xr hammer 8
utility's
.Cm snapshot
command provides several ways of taking snapshots.
They all assume a directory where snapshots are kept.
.Bd -literal -offset indent
mkdir /snaps
hammer snapshot /home /snaps/snap1
(...after some changes in /home...)
hammer snapshot /home /snaps/snap2
.Ed
.Pp
The softlinks in
.Pa /snaps
point to the state of the
.Pa /home
directory at the time each snapshot was taken, and could now be used to copy
the data somewhere else for backup purposes.
.Pp
By default,
.Dx
is set up to create nightly snapshots of all
.Nm
file systems via
.Xr periodic 8
and to keep them for 60 days.
.Ss Pruning
A snapshot directory is also the argument to the
.Xr hammer 8
.Cm prune
command which frees historical data from the file system that is not
pointed to by any snapshot link and is not from after the latest snapshot
and is older than
.Cm prune-min .
.Bd -literal -offset indent
rm /snaps/snap1
hammer prune /snaps
.Ed
.Ss Mirroring
Mirroring is set up using
.Nm
pseudo-filesystems (PFSs).
To associate the slave with the master its shared UUID should be set to
the master's shared UUID as output by the
.Nm hammer Cm pfs-master
command.
.Bd -literal -offset indent
hammer pfs-master /home/pfs/master
hammer pfs-slave /home/pfs/slave shared-uuid=<master's shared uuid>
.Ed
.Pp
The
.Pa /home/pfs/slave
link is unusable for as long as no mirroring operation has taken place.
.Pp
To mirror the master's data, either pipe a
.Cm mirror-read
command into a
.Cm mirror-write
or, as a short-cut, use the
.Cm mirror-copy
command (which works across a
.Xr ssh 1
connection as well).
Initial mirroring operation has to be done to the PFS path (as
.Xr mount_null 8
can't access it yet).
.Bd -literal -offset indent
hammer mirror-copy /home/pfs/master /home/pfs/slave
.Ed
.Pp
It is also possible to have the target PFS auto created
by just issuing the same
.Cm mirror-copy
command, if the target PFS doesn't exist you will be prompted
if you would like to create it.
You can even omit the prompting by using the
.Fl y
flag:
.Bd -literal -offset indent
hammer -y mirror-copy /home/pfs/master /home/pfs/slave
.Ed
.Pp
After this initial step
.Nm null
mount can be setup for
.Pa /home/pfs/slave .
Further operations can use
.Nm null
mounts.
.Bd -literal -offset indent
mount_null /home/pfs/master /home/master
mount_null /home/pfs/slave /home/slave

hammer mirror-copy /home/master /home/slave
.Ed
.Ss NFS Export
To NFS export from the
.Nm
file system
.Pa /hammer
the directory
.Pa /hammer/non-pfs
without PFSs, and the PFS
.Pa /hammer/pfs/data ,
the latter is
.Nm null
mounted to
.Pa /hammer/data .
.Pp
Add to
.Pa /etc/fstab
(see
.Xr fstab 5 ) :
.Bd -literal -offset indent
/hammer/pfs/data /hammer/data null rw
.Ed
.Pp
Add to
.Pa /etc/exports
(see
.Xr exports 5 ) :
.Bd -literal -offset indent
/hammer/non-pfs
/hammer/data
.Ed
.Sh DIAGNOSTICS
.Bl -diag
.It "hammer: System has insuffient buffers to rebalance the tree.  nbuf < %d"
Rebalancing a
.Nm
PFS uses quite a bit of memory and
can't be done on low memory systems.
It has been reported to fail on 512MB systems.
Rebalancing isn't critical for
.Nm
file system operation;
it is done by
.Nm hammer
.Cm rebalance ,
often as part of
.Nm hammer
.Cm cleanup .
.El
.Sh SEE ALSO
.Xr chflags 1 ,
.Xr md5 1 ,
.Xr tar 1 ,
.Xr undo 1 ,
.Xr exports 5 ,
.Xr ffs 5 ,
.Xr fstab 5 ,
.Xr disklabel64 8 ,
.Xr gpt 8 ,
.Xr hammer 8 ,
.Xr mount_hammer 8 ,
.Xr mount_null 8 ,
.Xr newfs_hammer 8 ,
.Xr periodic 8 ,
.Xr sysctl 8
.Rs
.%A Matthew Dillon
.%D June 2008
.%O http://www.dragonflybsd.org/hammer/hammer.pdf
.%T "The HAMMER Filesystem"
.Re
.Rs
.%A Matthew Dillon
.%D October 2008
.%O http://www.dragonflybsd.org/hammer/nycbsdcon/
.%T "Slideshow from NYCBSDCon 2008"
.Re
.Rs
.%A Michael Neumann
.%D January 2010
.%O http://www.ntecs.de/sysarch09/HAMMER.pdf
.%T "Slideshow for a presentation held at KIT (http://www.kit.edu)"
.Re
.Sh FILESYSTEM PERFORMANCE
The
.Nm
file system has a front-end which processes VNOPS and issues necessary
block reads from disk, and a back-end which handles meta-data updates
on-media and performs all meta-data write operations.
Bulk file write operations are handled by the front-end.
Because
.Nm
defers meta-data updates virtually no meta-data read operations will be
issued by the frontend while writing large amounts of data to the file system
or even when creating new files or directories, and even though the
kernel prioritizes reads over writes the fact that writes are cached by
the drive itself tends to lead to excessive priority given to writes.
.Pp
There are four bioq sysctls, shown below with default values,
which can be adjusted to give reads a higher priority:
.Bd -literal -offset indent
kern.bioq_reorder_minor_bytes: 262144
kern.bioq_reorder_burst_bytes: 3000000
kern.bioq_reorder_minor_interval: 5
kern.bioq_reorder_burst_interval: 60
.Ed
.Pp
If a higher read priority is desired it is recommended that the
.Va kern.bioq_reorder_minor_interval
be increased to 15, 30, or even 60, and the
.Va kern.bioq_reorder_burst_bytes
be decreased to 262144 or 524288.
.Sh HISTORY
The
.Nm
file system first appeared in
.Dx 1.11 .
.Sh AUTHORS
.An -nosplit
The
.Nm
file system was designed and implemented by
.An Matthew Dillon Aq dillon@backplane.com ,
data deduplication was added by
.An Ilya Dryomov .
This manual page was written by
.An Sascha Wildner
and updated by
.An Thomas Nikolajsen .
.Sh CAVEATS
Data deduplication is considered experimental.