HAMMER 24/many: Clean up edge cases
Hammer now survives many cpdup/rm-rf loops with recovery forced on.
* Fix an edge case with historical lookups. When pushing down through a
spike an exact match on a SPIKE_END element is no guarantee of success,
so check for and flag the delete boundary in that case.
* Delay physical cluster deletions (marking the cluster as free in the
Volume's A-list) until the last ref on the cluster goes away.
* Fix a race between cluster deletion, filesystem sync, and recovery.
Do not attempt to recover a cluster marked as undergoing deletion.
* Enable cluster recovery by default. It is still fairly primitive but
it should work well enough to put the filesystem in a working state
once the remaining debug shims are removed.
* Clean up sequencing of io->modified to ensure that clusters are properly
closed during normal operation. Recovery operations should only occur
when encountering an open cluster after a crash.
* Do not open a cluster or create a cluster->buffer dependancy when only
updating the mtime or atime of an inode.
* In the hammer_sync_*() procedures, ref the volume/cluster prior to
scanning its RB tree, use hammer_ref() instead of hammer_ref_*() to
avoid unnecessary loads. This also fixes a stale pointer bug in the
same code.
* Remove the stack recursion in btree_remove(). btree_remove() is now able
to clean up empty nodes going all the way up the tree (at least until it
hits a deadlock, which is not yet handled).
* Fix primary filesystem statistics (they were getting off because
whole clusters were being deleted without bothering to free up internal
on-disk structures).
* Fix a minor bug in the cluster's stat_records tracking field. 'df' no
longer shows a bleeding capacity. Statistics will still get off due
to crashes and such (the balancer will probably have to deal with that).
* Add more assertions. How do you debug complex code? You assert that
everything is in its proper state, everywhere you can.