.\" Copyright (c) 2001 Matthew Dillon.  Terms and conditions are those of
.\" the BSD Copyright as specified in the file "/usr/src/COPYRIGHT" in
.\" the source tree.
.\"
.\" $DragonFly: src/share/man/man7/tuning.7,v 1.21 2008/10/17 11:30:24 swildner Exp $
.\"
.Dd October 24, 2010
.Dt TUNING 7
.Os
.Sh NAME
.Nm tuning
.Nd performance tuning under DragonFly
.Sh SYSTEM SETUP - DISKLABEL, NEWFS, TUNEFS, SWAP
Modern
.Dx
systems typically have just three partitions on the main drive.
In order, a UFS
.Pa /boot ,
.Pa swap ,
and a HAMMER
.Pa / .
The installer usually creates a multitude of PFSs (pseudo filesystems) on
the HAMMER partition for /var, /tmp, and numerous other sub-trees.
These PFSs exist to ease the management of snapshots and backups.
.Pp
Generally speaking the
.Pa /boot
partition should be around 768MB in size.  The minimum recommended
is around 350MB, giving you room for backup kernels and alternative
boot schemes.
.Pp
In the old days we recommended that swap be sized to at least 2x main
memory.  These days swap is often used for other activities, including
.Xr tmpfs 5 .
We recommend that swap be sized to the larger of 2x main memory or
1GB if you have a fairly small disk and up to 16GB if you have a
moderately endowed system and a large drive.
If you are on a minimally configured machine you may, of course,
configure far less swap or no swap at all but we recommend at least
some swap.
The kernel's VM paging algorithms are tuned to perform best when there is
at least 2x swap versus main memory.
Configuring too little swap can lead to inefficiencies in the VM
page scanning code as well as create issues later on if you add
more memory to your machine.
Swap is a good idea even if you don't think you will ever need it as it allows the
machine to page out completely unused data from idle programs (like getty),
maximizing the ram available for your activities.
.Pp
If you intend to use the
.Xr swapcache 8
facility with a SSD we recommend the SSD be configured with at
least a 32G swap partition (the maximum default for i386).
If you are on a moderately well configured 64-bit system you can
size swap even larger.
.Pp
Finally, on larger systems with multiple drives, if the use
of SSD swap is not in the cards, we recommend that you
configure swap on each drive (up to four drives).
The swap partitions on the drives should be approximately the same size.
The kernel can handle arbitrary sizes but
internal data structures scale to 4 times the largest swap partition.
Keeping
the swap partitions near the same size will allow the kernel to optimally
stripe swap space across the N disks.
Do not worry about overdoing it a
little, swap space is the saving grace of
.Ux
and even if you do not normally use much swap, it can give you more time to
recover from a runaway program before being forced to reboot.
.Pp
Most
.Dx
systems have a single HAMMER root and use PFSs to break it up into
various administrative domains.
All the PFSs share the same allocation layer so there is no longer a need
to size each individual mount.
Instead you should review the
.Xr hammer 8
manual page and use the 'hammer viconfig' facility to adjust snapshot
retention and other parameters.
By default
HAMMER keeps 60 days worth of snapshots.
Usually snapshots are not desired on PFSs such as
.Pa /usr/obj
or
.Pa /tmp
since data on these partitions cycles a lot.
.Pp
If a very large work area is desired it is often beneficial to
configure it as a separate HAMMER mount.  If it is integrated into
the root mount it should at least be its own HAMMER PFS.
We recommend naming the large work area
.Pa /build .
Similarly if a machine is going to have a large number of users
you might want to separate your
.Pa /home
out as well.
.Pp
A number of run-time
.Xr mount 8
options exist that can help you tune the system.
The most obvious and most dangerous one is
.Cm async .
Do not ever use it; it is far too dangerous.
A less dangerous and more
useful
.Xr mount 8
option is called
.Cm noatime .
.Ux
filesystems normally update the last-accessed time of a file or
directory whenever it is accessed.
This operation is handled in
.Dx
with a delayed write and normally does not create a burden on the system.
However, if your system is accessing a huge number of files on a continuing
basis the buffer cache can wind up getting polluted with atime updates,
creating a burden on the system.
For example, if you are running a heavily
loaded web site, or a news server with lots of readers, you might want to
consider turning off atime updates on your larger partitions with this
.Xr mount 8
option.
You should not gratuitously turn off atime updates everywhere.
For example, the
.Pa /var
filesystem customarily
holds mailboxes, and atime (in combination with mtime) is used to
determine whether a mailbox has new mail.
.Sh STRIPING DISKS
In larger systems you can stripe partitions from several drives together
to create a much larger overall partition.
Striping can also improve
the performance of a filesystem by splitting I/O operations across two
or more disks.
The
.Xr vinum 8 ,
.Xr lvm 8 ,
and
.Xr dm 8
subsystems may be used to create simple striped filesystems.
We have deprecated
.Xr ccd 4 .
Generally
speaking, striping smaller partitions such as the root and
.Pa /var/tmp ,
or essentially read-only partitions such as
.Pa /usr
is a complete waste of time.
You should only stripe partitions that require serious I/O performance.
We recommend that such partitions be completely separate mounts
and not use the same storage media as your root mount.
.Pp
I should reiterate that last comment.  Do not stripe your boot+root.
Just about everything in there will be cached in system memory anyway.
Neither would I recommend RAIDing your root.
If robustness is needed placing your boot, swap, and root on a SSD
which has about the same MTBF as your motherboard, then RAIDing everything
which requires significant amounts of storage, should be sufficient.
There isn't much point making the boot/swap/root storage even more redundant
when the motherboard itself has no redundancy.
When a high level of total system redundancy is required you need to be
thinking more about having multiple physical machines that back each
other up.
.Pp
When striping multiple disks always partition in multiples of at
least 8 megabytes and use at least a 128KB stripe.
A 256KB stripe is probably even better.
This will avoid mis-aligning HAMMER big-blocks (which are 8MB)
or causing a single I/O cluster from crossing a stripe boundary.
.Dx
will issue a significant amount of read-ahead, upwards of a megabyte
or more if it determines accesses are linear enough, which is
sufficient to issue concurrent I/O across multiple stripes.
.Sh SYSCTL TUNING
.Xr sysctl 8
variables permit system behavior to be monitored and controlled at
run-time.
Some sysctls simply report on the behavior of the system; others allow
the system behavior to be modified;
some may be set at boot time using
.Xr rc.conf 5 ,
but most will be set via
.Xr sysctl.conf 5 .
There are several hundred sysctls in the system, including many that appear
to be candidates for tuning but actually are not.
In this document we will only cover the ones that have the greatest effect
on the system.
.Pp
The
.Va kern.ipc.shm_use_phys
sysctl defaults to 0 (off) and may be set to 0 (off) or 1 (on).
Setting
this parameter to 1 will cause all System V shared memory segments to be
mapped to unpageable physical RAM.
This feature only has an effect if you
are either (A) mapping small amounts of shared memory across many (hundreds)
of processes, or (B) mapping large amounts of shared memory across any
number of processes.
This feature allows the kernel to remove a great deal
of internal memory management page-tracking overhead at the cost of wiring
the shared memory into core, making it unswappable.
.Pp
The
.Va vfs.write_behind
sysctl defaults to 1 (on).  This tells the filesystem to issue media
writes as full clusters are collected, which typically occurs when writing
large sequential files.  The idea is to avoid saturating the buffer
cache with dirty buffers when it would not benefit I/O performance.  However,
this may stall processes and under certain circumstances you may wish to turn
it off.
.Pp
The
.Va vfs.hirunningspace
sysctl determines how much outstanding write I/O may be queued to
disk controllers system wide at any given instance.  The default is
usually sufficient but on machines with lots of disks you may want to bump
it up to four or five megabytes.  Note that setting too high a value
(exceeding the buffer cache's write threshold) can lead to extremely
bad clustering performance.  Do not set this value arbitrarily high!  Also,
higher write queueing values may add latency to reads occurring at the same
time.
.Pp
There are various other buffer-cache and VM page cache related sysctls.
We do not recommend modifying these values.
As of
.Fx 4.3 ,
the VM system does an extremely good job tuning itself.
.Pp
The
.Va net.inet.tcp.sendspace
and
.Va net.inet.tcp.recvspace
sysctls are of particular interest if you are running network intensive
applications.
They control the amount of send and receive buffer space
allowed for any given TCP connection.
The default sending buffer is 32K; the default receiving buffer
is 64K.
You can often
improve bandwidth utilization by increasing the default at the cost of
eating up more kernel memory for each connection.
We do not recommend
increasing the defaults if you are serving hundreds or thousands of
simultaneous connections because it is possible to quickly run the system
out of memory due to stalled connections building up.
But if you need
high bandwidth over a fewer number of connections, especially if you have
gigabit Ethernet, increasing these defaults can make a huge difference.
You can adjust the buffer size for incoming and outgoing data separately.
For example, if your machine is primarily doing web serving you may want
to decrease the recvspace in order to be able to increase the
sendspace without eating too much kernel memory.
Note that the routing table (see
.Xr route 8 )
can be used to introduce route-specific send and receive buffer size
defaults.
.Pp
As an additional management tool you can use pipes in your
firewall rules (see
.Xr ipfw 8 )
to limit the bandwidth going to or from particular IP blocks or ports.
For example, if you have a T1 you might want to limit your web traffic
to 70% of the T1's bandwidth in order to leave the remainder available
for mail and interactive use.
Normally a heavily loaded web server
will not introduce significant latencies into other services even if
the network link is maxed out, but enforcing a limit can smooth things
out and lead to longer term stability.
Many people also enforce artificial
bandwidth limitations in order to ensure that they are not charged for
using too much bandwidth.
.Pp
Setting the send or receive TCP buffer to values larger than 65535 will result
in a marginal performance improvement unless both hosts support the window
scaling extension of the TCP protocol, which is controlled by the
.Va net.inet.tcp.rfc1323
sysctl.
These extensions should be enabled and the TCP buffer size should be set
to a value larger than 65536 in order to obtain good performance from
certain types of network links; specifically, gigabit WAN links and
high-latency satellite links.
RFC 1323 support is enabled by default.
.Pp
The
.Va net.inet.tcp.always_keepalive
sysctl determines whether or not the TCP implementation should attempt
to detect dead TCP connections by intermittently delivering
.Dq keepalives
on the connection.
By default, this is disabled for all applications, only applications
that specifically request keepalives will use them.
In most environments, TCP keepalives will improve the management of
system state by expiring dead TCP connections, particularly for
systems serving dialup users who may not always terminate individual
TCP connections before disconnecting from the network.
However, in some environments, temporary network outages may be
incorrectly identified as dead sessions, resulting in unexpectedly
terminated TCP connections.
In such environments, setting the sysctl to 0 may reduce the occurrence of
TCP session disconnections.
.Pp
The
.Va net.inet.tcp.delayed_ack
TCP feature is largely misunderstood.  Historically speaking this feature
was designed to allow the acknowledgement to transmitted data to be returned
along with the response.  For example, when you type over a remote shell
the acknowledgement to the character you send can be returned along with the
data representing the echo of the character.   With delayed acks turned off
the acknowledgement may be sent in its own packet before the remote service
has a chance to echo the data it just received.  This same concept also
applies to any interactive protocol (e.g. SMTP, WWW, POP3) and can cut the
number of tiny packets flowing across the network in half.   The
.Dx
delayed-ack implementation also follows the TCP protocol rule that
at least every other packet be acknowledged even if the standard 100ms
timeout has not yet passed.  Normally the worst a delayed ack can do is
slightly delay the teardown of a connection, or slightly delay the ramp-up
of a slow-start TCP connection.  While we aren't sure we believe that
the several FAQs related to packages such as SAMBA and SQUID which advise
turning off delayed acks may be referring to the slow-start issue.
.Pp
The
.Va net.inet.tcp.inflight_enable
sysctl turns on bandwidth delay product limiting for all TCP connections.
The system will attempt to calculate the bandwidth delay product for each
connection and limit the amount of data queued to the network to just the
amount required to maintain optimum throughput.  This feature is useful
if you are serving data over modems, GigE, or high speed WAN links (or
any other link with a high bandwidth*delay product), especially if you are
also using window scaling or have configured a large send window.  If
you enable this option you should also be sure to set
.Va net.inet.tcp.inflight_debug
to 0 (disable debugging), and for production use setting
.Va net.inet.tcp.inflight_min
to at least 6144 may be beneficial.  Note, however, that setting high
minimums may effectively disable bandwidth limiting depending on the link.
The limiting feature reduces the amount of data built up in intermediate
router and switch packet queues as well as reduces the amount of data built
up in the local host's interface queue.  With fewer packets queued up,
interactive connections, especially over slow modems, will also be able
to operate with lower round trip times.  However, note that this feature
only affects data transmission (uploading / server-side).  It does not
affect data reception (downloading).
.Pp
Adjusting
.Va net.inet.tcp.inflight_stab
is not recommended.
This parameter defaults to 20, representing 2 maximal packets added
to the bandwidth delay product window calculation.  The additional
window is required to stabilize the algorithm and improve responsiveness
to changing conditions, but it can also result in higher ping times
over slow links (though still much lower than you would get without
the inflight algorithm).  In such cases you may
wish to try reducing this parameter to 15, 10, or 5, and you may also
have to reduce
.Va net.inet.tcp.inflight_min
(for example, to 3500) to get the desired effect.  Reducing these parameters
should be done as a last resort only.
.Pp
The
.Va net.inet.ip.portrange.*
sysctls control the port number ranges automatically bound to TCP and UDP
sockets.  There are three ranges:  A low range, a default range, and a
high range, selectable via an IP_PORTRANGE
.Fn setsockopt
call.
Most network programs use the default range which is controlled by
.Va net.inet.ip.portrange.first
and
.Va net.inet.ip.portrange.last ,
which defaults to 1024 and 5000 respectively.  Bound port ranges are
used for outgoing connections and it is possible to run the system out
of ports under certain circumstances.  This most commonly occurs when you are
running a heavily loaded web proxy.  The port range is not an issue
when running serves which handle mainly incoming connections such as a
normal web server, or has a limited number of outgoing connections such
as a mail relay.  For situations where you may run yourself out of
ports we recommend increasing
.Va net.inet.ip.portrange.last
modestly.  A value of 10000 or 20000 or 30000 may be reasonable.  You should
also consider firewall effects when changing the port range.  Some firewalls
may block large ranges of ports (usually low-numbered ports) and expect systems
to use higher ranges of ports for outgoing connections.  For this reason
we do not recommend that
.Va net.inet.ip.portrange.first
be lowered.
.Pp
The
.Va kern.ipc.somaxconn
sysctl limits the size of the listen queue for accepting new TCP connections.
The default value of 128 is typically too low for robust handling of new
connections in a heavily loaded web server environment.
For such environments,
we recommend increasing this value to 1024 or higher.
The service daemon
may itself limit the listen queue size (e.g.\&
.Xr sendmail 8 ,
apache) but will
often have a directive in its configuration file to adjust the queue size up.
Larger listen queues also do a better job of fending off denial of service
attacks.
.Pp
The
.Va kern.maxfiles
sysctl determines how many open files the system supports.
The default is
typically a few thousand but you may need to bump this up to ten or twenty
thousand if you are running databases or large descriptor-heavy daemons.
The read-only
.Va kern.openfiles
sysctl may be interrogated to determine the current number of open files
on the system.
.Pp
The
.Va vm.swap_idle_enabled
sysctl is useful in large multi-user systems where you have lots of users
entering and leaving the system and lots of idle processes.
Such systems
tend to generate a great deal of continuous pressure on free memory reserves.
Turning this feature on and adjusting the swapout hysteresis (in idle
seconds) via
.Va vm.swap_idle_threshold1
and
.Va vm.swap_idle_threshold2
allows you to depress the priority of pages associated with idle processes
more quickly than the normal pageout algorithm.
This gives a helping hand
to the pageout daemon.
Do not turn this option on unless you need it,
because the tradeoff you are making is to essentially pre-page memory sooner
rather than later, eating more swap and disk bandwidth.
In a small system
this option will have a detrimental effect but in a large system that is
already doing moderate paging this option allows the VM system to stage
whole processes into and out of memory more easily.
.Sh LOADER TUNABLES
Some aspects of the system behavior may not be tunable at runtime because
memory allocations they perform must occur early in the boot process.
To change loader tunables, you must set their values in
.Xr loader.conf 5
and reboot the system.
.Pp
.Va kern.maxusers
controls the scaling of a number of static system tables, including defaults
for the maximum number of open files, sizing of network memory resources, etc.
On
.Dx ,
.Va kern.maxusers
is automatically sized at boot based on the amount of memory available in
the system, and may be determined at run-time by inspecting the value of the
read-only
.Va kern.maxusers
sysctl.
Some sites will require larger or smaller values of
.Va kern.maxusers
and may set it as a loader tunable; values of 64, 128, and 256 are not
uncommon.
We do not recommend going above 256 unless you need a huge number
of file descriptors; many of the tunable values set to their defaults by
.Va kern.maxusers
may be individually overridden at boot-time or run-time as described
elsewhere in this document.
.Pp
The
.Va kern.dfldsiz
and
.Va kern.dflssiz
tunables set the default soft limits for process data and stack size
respectively.
Processes may increase these up to the hard limits by calling
.Xr setrlimit 2 .
The
.Va kern.maxdsiz ,
.Va kern.maxssiz ,
and
.Va kern.maxtsiz
tunables set the hard limits for process data, stack, and text size
respectively; processes may not exceed these limits.
The
.Va kern.sgrowsiz
tunable controls how much the stack segment will grow when a process
needs to allocate more stack.
.Pp
.Va kern.ipc.nmbclusters
may be adjusted to increase the number of network mbufs the system is
willing to allocate.
Each cluster represents approximately 2K of memory,
so a value of 1024 represents 2M of kernel memory reserved for network
buffers.
You can do a simple calculation to figure out how many you need.
If you have a web server which maxes out at 1000 simultaneous connections,
and each connection eats a 16K receive and 16K send buffer, you need
approximately 32MB worth of network buffers to deal with it.
A good rule of
thumb is to multiply by 2, so 32MBx2 = 64MB/2K = 32768.
So for this case
you would want to set
.Va kern.ipc.nmbclusters
to 32768.
We recommend values between
1024 and 4096 for machines with moderates amount of memory, and between 4096
and 32768 for machines with greater amounts of memory.
Under no circumstances
should you specify an arbitrarily high value for this parameter, it could
lead to a boot-time crash.
The
.Fl m
option to
.Xr netstat 1
may be used to observe network cluster use.
.Sh KERNEL CONFIG TUNING
There are a number of kernel options that you may have to fiddle with in
a large-scale system.
In order to change these options you need to be
able to compile a new kernel from source.
The
.Xr config 8
manual page and the handbook are good starting points for learning how to
do this.
Generally the first thing you do when creating your own custom
kernel is to strip out all the drivers and services you do not use.
Removing things like
.Dv INET6
and drivers you do not have will reduce the size of your kernel, sometimes
by a megabyte or more, leaving more memory available for applications.
.Pp
If your motherboard is AHCI-capable then we strongly recommend turning
on AHCI mode.
.Sh CPU, MEMORY, DISK, NETWORK
The type of tuning you do depends heavily on where your system begins to
bottleneck as load increases.
If your system runs out of CPU (idle times
are perpetually 0%) then you need to consider upgrading the CPU or moving to
an SMP motherboard (multiple CPU's), or perhaps you need to revisit the
programs that are causing the load and try to optimize them.
If your system
is paging to swap a lot you need to consider adding more memory.
If your
system is saturating the disk you typically see high CPU idle times and
total disk saturation.
.Xr systat 1
can be used to monitor this.
There are many solutions to saturated disks:
increasing memory for caching, mirroring disks, distributing operations across
several machines, and so forth.
If disk performance is an issue and you
are using IDE drives, switching to SCSI can help a great deal.
While modern
IDE drives compare with SCSI in raw sequential bandwidth, the moment you
start seeking around the disk SCSI drives usually win.
.Pp
Finally, you might run out of network suds.
The first line of defense for
improving network performance is to make sure you are using switches instead
of hubs, especially these days where switches are almost as cheap.
Hubs
have severe problems under heavy loads due to collision backoff and one bad
host can severely degrade the entire LAN.
Second, optimize the network path
as much as possible.
For example, in
.Xr firewall 7
we describe a firewall protecting internal hosts with a topology where
the externally visible hosts are not routed through it.
Use 100BaseT rather
than 10BaseT, or use 1000BaseT rather than 100BaseT, depending on your needs.
Most bottlenecks occur at the WAN link (e.g.\&
modem, T1, DSL, whatever).
If expanding the link is not an option it may be possible to use the
.Xr dummynet 4
feature to implement peak shaving or other forms of traffic shaping to
prevent the overloaded service (such as web services) from affecting other
services (such as email), or vice versa.
In home installations this could
be used to give interactive traffic (your browser,
.Xr ssh 1
logins) priority
over services you export from your box (web services, email).
.Sh SEE ALSO
.Xr netstat 1 ,
.Xr systat 1 ,
.Xr dummynet 4 ,
.Xr nata 4 ,
.Xr login.conf 5 ,
.Xr rc.conf 5 ,
.Xr sysctl.conf 5 ,
.Xr firewall 7 ,
.Xr hier 7 ,
.Xr boot 8 ,
.Xr ccdconfig 8 ,
.Xr config 8 ,
.Xr disklabel 8 ,
.Xr fsck 8 ,
.Xr ifconfig 8 ,
.Xr ipfw 8 ,
.Xr loader 8 ,
.Xr mount 8 ,
.Xr newfs 8 ,
.Xr route 8 ,
.Xr sysctl 8 ,
.Xr tunefs 8 ,
.Xr vinum 8
.Sh HISTORY
The
.Nm
manual page was originally written by
.An Matthew Dillon
and first appeared
in
.Fx 4.3 ,
May 2001.