BIOQ - Create a more sophisticated bursting mechanic for writes.
* Add sysctls:
vfs.bioq_reorder_minor_interval
vfs.bioq_reorder_minor_bytes
vfs.bioq_reorder_burst_interval
vfs.bioq_reorder_burst_bytes
* Reads are prioritized over writes. Every N (minor_interval) reads
up to (minor_bytes) worth of writes are allowed in order to avoid
complete write starvation (which is just as bad as complete read
starvation).
* If a large amount of write data is queued to devices the related
buffer cache buffers will be locked. This is not a good place to
be as any readers trying to access those buffers will then also
block.
When the amount of data is excessive the (burst_interval) and
(burst_bytes) comes into play. This primarily helps HAMMER
flush cycles.
* Try to keep vfs.hirunningspace reasonably bounded. We do not want
too large a value because of the above buffer locking issue. On
the flip side, we can have large numbers of delayed-write dirty buffers
sitting around because they aren't locked.
* When the buffer cache is hitting hidirtybufspace the buffer daemons
now try to keep the vfs.runningbufspace at hirunningspace levels
instead of lorunningspace levels in order to trigger the bioq's
bursting mode.