2 HAMMER2 DESIGN DOCUMENT
11 Current Status as of document date
13 * Filesystem Core - operational
14 - bulkfree - operational
15 - Compression - operational
16 - Snapshots - operational
18 - Subhierarchy quotas - specced
19 - Logical Encryption - not specced yet
20 - Copies - not specced yet
21 - fsync bypass - not specced yet
24 - Network msg core - operational
25 - Network blk device - operational
26 - Error handling - under development
27 - Quorum Protocol - under development
28 - Synchronization - under development
29 - Transaction replay - not specced yet
30 - Cache coherency - not specced yet
34 * Multiple roots, with many features. This is implemented via the super-root
35 concept. When mounting a HAMMER2 filesystem you specify a device path and
36 a directory name in the super-root. (HAMMER1 had only one root).
38 * All cluster types and multiple PFSs (belonging to the same or different
39 clusters) can be mixed on one physical filesystem.
41 This allows independent cluster components to be configured within a
42 single formatted H2 filesystem. Each component is a super-root entry,
43 a cluster identifier, and a unique identifier. The network protocl
44 integrates the component into the cluster when it is created
46 * Roots are really no different from snapshots (HAMMER1 distinguished between
47 its root mount and its PFS's. HAMMER2 does not).
49 * Snapshots are writable (in HAMMER1 snapshots were read-only).
51 * Snapshots are explicit but trivial to create. In HAMMER1 snapshots were
52 both explicit and fine-grained/automatic. HAMMER2 does not implement
53 automatic fine-grained snapshots. H2 snapshots are cheap enough that you
54 can create fine-grained snapshots if you desire.
56 * HAMMER2 formalizes a synchronization point for the flush, does a pre-flush
57 that does not update the volume root, then waits for all running modifying
58 operations to complete to memory (not to disk) while temporarily stalling
59 new modifying operation initiations. The final flush is then executed.
61 At the moment we do not allow concurrent modifying operations during the
62 final flush phase. Ultimately I would like to, but doing so can be complex.
64 * HAMMER2 flushes and synchronization points do not bisect VOPs (system calls).
65 (HAMMER1 flushes could wind up bisecting VOPs). This means the H2 flushes
66 leave the filesystem in a far more consistent state than H1 flushes did.
68 * Directory sub-hierarchy-based quotas for space and inode usage tracking.
69 Any directory can be used.
71 * Low memory footprint. Except for the volume header, the buffer cache
72 is completely asynchronous and dirty buffers can be retired by the OS
73 directly to backing store with no further interactions with the filesystem.
75 * Background synchronization and mirroring occurs at the logical level.
76 When a failure occurs or a normal validation scan comes up with
77 discrepancies, the synchronization thread will use the quorum to figure
78 out which information is not correct and update accordingly.
80 * Support for multiple compression algorithms configured on subdirectory
81 tree basis and on a file basis. Block compression up to 64KB will be used.
82 Only compression ratios at powers of 2 that are at least 2:1 (e.g. 2:1,
83 4:1, 8:1, etc) will work in this scheme because physical block allocations
84 in HAMMER2 are always power-of-2. Modest compression can be achieved with
85 low overhead, is turned on by default, and is compatible with deduplication.
87 * Encryption. Whole-disk encryption is supported by another layer, but I
88 intend to give H2 an encryption feature at the logical layer which works
89 approximately as follows:
91 - Encryption controlled by the client on an inode/sub-tree basis.
92 - Server has no visibility to decrypted data.
93 - Encrypt filenames in directory entries. Since the filename[] array
94 is 256 bytes wide, client can add random bytes after the normal
95 terminator to make it virtually impossible for an attacker to figure
97 - Encrypt file size and most inode contents.
98 - Encrypt file data (holes are not encrypted).
99 - Encryption occurs after compression, with random filler.
100 - Check codes calculated after encryption & compression (not before).
102 - Blockrefs are not encrypted.
103 - Directory and File Topology is not encrypted.
104 - Encryption is not sub-topology validation. Client would have to keep
105 track of that itself. Server or other clients can still e.g. remove
108 In particular, note that even though the file size field can be encrypted,
109 the server does have visibility on the block topology and thus has a pretty
110 good idea how big the file is. However, a client could add junk blocks
111 at the end of a file to make this less apparent, at the cost of space.
113 If a client really wants a fully validated H2-encrypted space the easiest
114 solution is to format a filesystem within an encrypted file by treating it
115 as a block device, but I digress.
117 * Zero detection on write (writing all-zeros), which requires the data
118 buffer to be scanned, is fully supported. This allows the writing of 0's
121 * Copies support for redundancy within a single physical filesystem.
122 Up to 256 physical disks and/or partitions can be ganged to form a
123 single physical filesystem. If you use a disk or RAID aggregation
124 layer then the actual number of physical disks that can be associated
125 with a single H2 filesystem is unbounded.
127 H2 handles this by interleaving the 64-bit address space 256-ways on
128 level-1 (2GB) boundaries. Thus, a single physical disk from H2's point
129 of view can be sized up to 2^56 which is 64 Petabytes, and the total
130 filesystem size can be up to 16 Exabytes.
132 Copies support is implemented by having multiple blockref entries for
133 the same key, each with a different copyid. The copyid represents which
134 of the 256 slots is used. Meta-data is also subject to the copies
135 mechanism. However, for both meta-data and data, each copy should be
136 identical so the check fields in the blockref for all copies should wind
137 up being the same, and any valid copy can be used by the block-level
138 hammer2_chain code to access the filesystem. File accesses will attempt
139 to use the same copy. If an I/O read error occurs, a different copy will
140 be chosen. Modifying operations must update all copies and/or create
141 new copies as needed. If a write error occurs on a copy and other copies
142 are available, the errored target will be taken offline.
144 It is possible to configure H2 to write out fewer copies on-write and then
145 use a background scan to beef-up the number of copies to improve real-time
148 * MESI Cache coherency for multi-master/multi-client clustering operations.
149 The servers hosting the MASTERs are also responsible for keeping track of
152 * Hardlinks and softlinks are supported. Hardlinks are somewhat complex to
153 deal with and there is still an edge case. I am trying to avoid storing
154 the hardlinks at the root level because that messes up my concept for
155 sub-tree quotas and is unnecessarily burdensome in terms of SMP collisions
158 * The media blockref structure is now large enough to support up to a 192-bit
159 check value, which would typically be a cryptographic hash of some sort.
160 Multiple check value algorithms will be supported with the default being
161 a simple 32-bit iSCSI CRC.
163 * Fully verified deduplication will be supported and automatic (and
164 necessary in many respects).
166 * Unverified de-duplication will be supported as a configurable option on a
167 file or subdirectory tree. Unverified deduplication must use the largest
168 available check code (192 bits). It will not verify that data content with
169 the same check code is actually identical during the dedup pass, resulting
170 in approximately 100x to 1000x the deduplication performance but at the cost
171 of potentially corrupting some data.
173 The Unverified dedup feature is intended only for those files where
174 occassional corruption is ok, such as in a web-crawler data store or
175 other situations where the data content is not critically important
176 or can be externally recovered if it becomes corrupt.
180 HAMMER2 generally implements a copy-on-write block design for the filesystem,
181 which is very different from HAMMER1's B-Tree design. Because the design
182 is copy-on-write it can be trivially snapshotted simply by referencing an
183 existing block, and because the media structures logically match a standard
184 filesystem directory/file hierarchy snapshots and other similar operations
185 can be trivially performed on an entire subdirectory tree at any level in
188 The copy-on-write nature of the filesystem implies that any modification
189 whatsoever will have to eventually synchronize new disk blocks all the way
190 to the super-root of the filesystem and the volume header itself. This forms
191 the basis for crash recovery and also ensures that recovery occurs on a
192 completed high-level transaction boundary. All disk writes are to new blocks
193 except for the volume header (which cycles through 4 copies), thus allowing
194 all writes to run asynchronously and concurrently prior to and during a flush,
195 and then just doing a final synchronization and volume header update at the
198 Clearly this method requires intermediate modifications to the chain to be
199 cached so multiple modifications can be aggregated prior to being
200 synchronized. One advantage, however, is that the normal buffer cache can
201 be used and intermediate elements can be retired to disk by H2 or the OS
202 at any time. This means that HAMMER2 has very resource overhead from the
203 point of view of the operating system. Unlike HAMMER1 which had to lock
204 dirty buffers in memory for long periods of time, HAMMER2 has no such
207 Buffer cache overhead is very well bounded and can handle filesystem
208 operations of any complexity, even on boxes with very small amounts
209 of physical memory. Buffer cache overhead is significantly lower with H2
210 than with H1 (and orders of magnitude lower than ZFS).
212 At some point I intend to implement a shortcut to make fsync()'s run fast,
213 and that is to allow deep updates to blockrefs to shortcut to auxillary
214 space in the volume header to satisfy the fsync requirement. The related
215 blockref is then recorded when the filesystem is mounted after a crash and
216 the update chain is reconstituted when a matching blockref is encountered
217 again during normal operation of the filesystem.
219 DIRECTORIES AND INODES
221 Directories are hashed, and another major design element is that directory
222 entries ARE INODES. They are one and the same, with a special placemarker
223 for hardlinks. Inodes are 1KB.
225 Half of the inode structure (512 bytes) is used to hold top-level blockrefs
226 to the radix block tree representing the file contents. Files which are
227 less than or equal to 512 bytes in size will simply store the file contents
228 in this area instead of a blockref array. So files <= 512 bytes take only
229 1KB of space inclusive of the inode.
231 Inode numbers are not spatially referenced, which complicates NFS servers
232 but doesn't complicate anything else. The inode number is stored in the
233 inode itself, an absolute necessity required to properly support HAMMER2s
234 hugely flexible snapshots.
238 H2 allows freemap flushes to lag behind topology flushes. The freemap flush
239 tracks a separate transaction id in the volume header.
241 On mount, HAMMER2 will first locate the highest-sequenced check-code-validated
242 volume header from the 4 copies available (if the filesystem is big enough,
243 e.g. > ~10GB or so, there will be 4 copies of the volume header).
245 HAMMER2 will then run an incremental scan of the topology for transaction ids
246 between the last freemap flush and the current topology in order to update
247 the freemap. Because this scan is incremental the worst-case time to run
248 the scan is about the time it takes to run one flush.
250 The filesystem is then ready for use.
252 DISK I/O OPTIMIZATIONS
254 The freemap implements a 1KB allocation resolution. Each 2MB segment managed
255 by the freemap is zoned and has a tendancy to collect inodes, small data,
256 indirect blocks, and larger data blocks into separate segments. The idea is
257 to greatly improve I/O performance (particularly by laying inodes down next
258 to each other which has a huge effect on directory scans).
260 The current implementation of HAMMER2 implements a fixed block size of 64KB
261 in order to allow aliasing of hammer2_dio's in its IO subsystem. This way
262 we don't have to worry about matching the buffer cache / DIO cache to the
263 variable block size of underlying elements.
265 HAMMER2 also allows OS support for ganging buffers together into even
266 larger blocks for I/O, OS-supported read-ahead, and other performance
267 features typically provided by the OS at the block-level.
271 Hardlinks are a particularly sticky problem for HAMMER2 due to the lack of
272 a spatial reference to the inode number. We do not want to have to have
273 an index of inode numbers for any basic HAMMER2 feature if we can help it.
275 Hardlinks are handled by placing the inode for a multiply-hardlinked file
276 in the closest common parent directory. If "a/x" and "a/y" are hardlinked
277 the inode for the hardlinked file will be placed in directory "a", e.g.
278 "a/<inode_number>". The actual file inode will be an invisible out-of-band
279 entry in the directory. The directory entries "a/x" and "a/y" will be given
280 the same inode number but in fact they are only placemarks that cause
281 HAMMER2 to recurse upwards through the directory tree to find the invisible
284 Because directories are hashed and a different namespace (hash key range)
285 is used for hardlinked inodes, standard directory scans are able to trivially
286 skip this invisible namespace and inode-specific lookups can restrict their
287 lookup to within this space. No linear scans are needed.
291 The freemap is stored in the reserved blocks situated in the ~4MB reserved
292 area at the baes of every ~2GB level-1 zone. The current implementation
293 reserves 8 copies of every freemap block and cycles through them in order
294 to make the freemap operate in a copy-on-write fashion.
296 - Freemap is copy-on-write.
297 - Freemap operations are transactional, same as everything else.
298 - All backup volume headers are consistent on-mount.
300 The Freemap is organized using the same radix blockmap algorithm used for
301 files and directories, but with fixed radix values. For a maximally-sized
302 filesystem the Freemap will wind up being a 5-level-deep radix blockmap,
303 but the top-level is embedded in the volume header so insofar as performance
304 goes it is really just a 4-level blockmap.
306 The freemap radix allocation mechanism is also the same, meaning that it is
307 bottom-up and will not allocate unnecessary intermediate levels for smaller
308 filesystems. A 16GB filesystem uses a 2-level blockmap (volume header + one
309 level), a 16TB filesystem uses a 3-level blockmap (volume header + two levels),
312 The Freemap has bitmap granularity down to 16KB and a linear iterator that
313 can linearly allocate space down to 1KB. Due to fragmentation it is possible
314 for the linear allocator to become marginalized, but it is relatively easy
315 to for a reallocation of small blocks every once in a while (like once a year
316 if you care at all) and once the old data cycles out of the snapshots, or you
317 also rewrite the snapshots (which you can do), the freemap should wind up
318 relatively optimal again. Generally speaking I believe that algorithms can
319 be developed to make this a non-problem without requiring any media structure
322 In order to implement fast snapshots (and writable snapshots for that
323 matter), HAMMER2 does NOT ref-count allocations. All the freemap does is
324 keep track of 100% free blocks plus some extra bits for staging the bulkfree
325 scan. The lack of ref-counting makes it possible to:
327 - Completely trivialize HAMMER2s snapshot operations.
328 - Allows any volume header backup to be used trivially.
329 - Allows whole sub-trees to be destroyed without having to scan them.
330 - Simplifies normal crash recovery operations.
331 - Simplifies catastrophic recovery operations.
333 Normal crash recovery is simply a matter of doing an incremental scan
334 of the topology between the last flushed freemap TID and the last flushed
335 topology TID. This usually takes only a few seconds and allows:
337 - Freemap flushes to be be deferred for any number of topology flush
339 - Does not have to be flushed for fsync, reducing fsync overhead.
343 Blocks are freed via a bulkfree scan, which is a two-stage meta-data scan.
344 Blocks are first marked as being possibly free and then finalized in the
345 second scan. Live filesystem operations are allowed to run during these
346 scans and any freemap block that is allocated or adjusted after the first
347 scan will simply be re-marked as allocated and the second scan will not
348 transition it to being free.
350 The cost of not doing ref-count tracking is that HAMMER2 must perform two
351 bulkfree scans of the meta-data to determine which blocks can actually be
352 freed. This can be complicated by the volume header backups and snapshots
353 which cause the same meta-data topology to be scanned over and over again,
354 but mitigated somewhat by keeping a cache of higher-level nodes to detect
355 when we would scan a sub-topology that we have already scanned. Due to the
356 copy-on-write nature of the filesystem, such detection is easy to implement.
358 Part of the ongoing design work is finding ways to reduce the scope of this
359 meta-data scan so the entire filesystem's meta-data does not need to be
360 scanned (though in tests with HAMMER1, even full meta-data scans have
361 turned out to be fairly low cost). In other words, its an area where
362 improvements can be made without any media format changes.
364 Another advantage of operating the freemap like this is that some future
365 version of HAMMER2 might decide to completely change how the freemap works
366 and would be able to make the change with relatively low downtime.
370 Clustering, as always, is the most difficult bit but we have some advantages
371 with HAMMER2 that we did not have with HAMMER1. First, HAMMER2's media
372 structures generally follow the kernel's filesystem hiearchy which allows
373 cluster operations to use topology cache and lock state. Second,
374 HAMMER2's writable snapshots make it possible to implement several forms
375 of multi-master clustering.
377 The mount device path you specify serves to bootstrap your entry into
378 the cluster. This is typically local media. It can even be a ram-disk
379 that only contains placemarkers that help HAMMER2 connect to a fully
382 With HAMMER2 you mount a directory entry under the super-root. This entry
383 will contain a cluster identifier that helps HAMMER2 identify and integrate
384 with the nodes making up the cluster. HAMMER2 will automatically integrate
385 *all* entries under the super-root when you mount one of them. You have to
386 mount at least one for HAMMER2 to integrate the block device in the larger
387 cluster. This mount will typically be a SOFT_MASTER, DUMMY, SLAVE, or CACHE
388 mount that simply serves to cause hammer to integrate the rest of the
389 represented cluster. ALL CLUSTER ELEMENTS ARE TREATED ACCORDING TO TYPE
390 NO MATTER WHICH ONE YOU MOUNT.
392 For cluster servers every HAMMER2-formatted partition has a "LOCAL" MASTER
393 which can be mounted in order to make the rest of the elements under the
394 super-root available to the network. (In a prior specification I emplaced
395 the cluster connections in the volume header's configuration space but I no
398 Connecting to the wider networked cluster involves setting up the /etc/hammer2
399 directory with appropriate IP addresses and keys. The user-mode hammer2
400 service daemon maintains the connections and performs graph operations
403 Node types within the cluster:
405 DUMMY - Used as a local placeholder (typically in ramdisk)
406 CACHE - Used as a local placeholder and cache (typically on a SSD)
407 SLAVE - A SLAVE in the cluster, can source data on quorum agreement.
408 MASTER - A MASTER in the cluster, can source and sink data on quorum
410 SOFT_SLAVE - A SLAVE in the cluster, can source data locally without
411 quorum agreement (must be directly mounted).
412 SOFT_MASTER - A local MASTER but *not* a MASTER in the cluster. Can source
413 and sink data locally without quorum agreement, intended to
414 be synchronized with the real MASTERs when connectivity
415 allows. Operations are not coherent with the real MASTERS
416 even when they are available.
418 NOTE: SNAPSHOT, AUTOSNAP, etc represent sub-types, typically under a
419 SLAVE. A SNAPSHOT or AUTOSNAP is a SLAVE sub-type that is no longer
420 synchronized against current masters.
422 NOTE: Any SLAVE or other copy can be turned into its own writable MASTER
423 by giving it a unique cluster id, taking it out of the cluster that
424 originally spawned it.
426 There are four major protocols:
430 This protocol is used between MASTER nodes to vote on operations
431 and resolve deadlocks.
433 This protocol is used between SOFT_MASTER nodes in a sub-cluster
434 to vote on operations, resolve deadlocks, determine what the latest
435 transaction id for an element is, and to perform commits.
439 This is the MESI sub-protocol which runs under the Quorum
440 protocol. This protocol is used to maintain cache state for
441 sub-trees to ensure that operations remain cache coherent.
443 Depending on administrative rights this protocol may or may
444 not allow a leaf node in the cluster to hold a cache element
445 indefinitely. The administrative controller may preemptively
446 downgrade a leaf with insufficient administrative rights
447 without giving it a chance to synchronize any modified state
452 The Quorum and Cache protocols only operate between MASTER
453 and SOFT_MASTER nodes. All other node types must use the
454 Proxy protocol to perform similar actions. This protocol
455 differs in that proxy requests are typically sent to just
456 one adjacent node and that node then maintains state and
457 forwards the request or performs the required operation.
458 When the link is lost to the proxy, the proxy automatically
459 forwards a deletion of the state to the other nodes based on
460 what it has recorded.
462 If a leaf has insufficient administrative rights it may not
463 be allowed to actually initiate a quorum operation and may only
464 be allowed to maintain partial MESI cache state or perhaps none
465 at all (since cache state can block other machines in the
466 cluster). Instead a leaf with insufficient rights will have to
467 make due with a preemptive loss of cache state and any allowed
468 modifying operations will have to be forwarded to the proxy which
469 continues forwarding it until a node with sufficient administrative
470 rights is encountered.
472 To reduce issues and give the cluster more breath, sub-clusters
473 made up of SOFT_MASTERs can be formed in order to provide full
474 cache coherent within a subset of machines and yet still tie them
475 into a greater cluster that they normally would not have such
476 access to. This effectively makes it possible to create a two
477 or three-tier fan-out of groups of machines which are cache-coherent
478 within the group, but perhaps not between groups, and use other
479 means to synchronize between the groups.
483 This is basically the physical media protocol.
485 There are lots of ways to implement multi-master environments using the
486 above core features but the implementation is going to be fairly complex
487 even with HAMMER2's feature set.
489 Keep in mind that modifications propagate all the way to the super-root
490 and volume header, so in any clustered arrangement the use of (modify_tid)
491 and (mirror_tid) is critical in determining the synchronization state of
492 portion(s) of the filesystem.
494 Specifically, since any modification propagates to the root the (mirror_tid)
495 in higher level directories is going to be in a constant state of flux. This
496 state of flux DOES NOT invalidate the cache state for these higher levels
497 of directories. Instead, the (modify_tid) is used on a node-by-node basis
498 to determine cache state at any given level, and (mirror_tid) is used to
499 determine whether any recursively underlying state is desynchronized.
500 The inode structure also has two additional transaction ids used to optimize
501 path lookups, stat, and directory lookup/scan operations.
503 MASTER & SLAVE SYNCHRONIZATION
505 With HAMMER2 I really want to be hard-nosed about the consistency of the
506 filesystem, including the consistency of SLAVEs (snapshots, etc). In order
507 to guarantee consistency we take advantage of the copy-on-write nature of
508 the filesystem by forking consistent nodes and using the forked copy as the
509 source for synchronization.
511 Similarly, the target for synchronization is not updated on the fly but instead
512 is also forked and the forked copy is updated. When synchronization is
513 complete, forked sources can be thrown away and forked copies can replace
514 the original synchronization target.
516 This may seem complex, but 'forking a copy' is actually a virtually free
517 operation. The top-level inode (under the super-root), on-media, is simply
518 copied to a new inode and poof, we have an unchanging snapshot to work with.
520 - Making a snapshot is fast... almost instantanious.
522 - Snapshots are used for various purposes, including synchronization
523 of out-of-date nodes.
525 - A snapshot can be converted into a MASTER or some other PFS type.
527 - A snapshot can be forked off from its parent cluster entirely and
528 turned into its own writable filesystem, either as a single MASTER
529 or this can be done across the cluster by forking a quorum+ of
530 existing MASTERs and transfering them all to a new cluster id.
532 More complex is reintegrating the target once the synchronization is complete.
533 For SLAVEs we just delete the old SLAVE and rename the copy to the same name.
534 However, if the SLAVE is mounted and not optioned as a static mount (that is
535 the mounter wants to see updates as they are synchronized), a reconciliation
536 must occur on the live mount to clean up the vnode, inode, and chain caches
537 and shift any remaining vnodes over to the updated copy.
539 - A mounted SLAVE can track updates made to the SLAVE but the
540 actual mechanism is that the SLAVE PFS is replaced with an
541 updated copy, typically every 30-60 seconds.
543 Reintegrating a MASTER which has fallen out of the quorum due to being out
544 of date is also somewhat more complex. The same updating mechanic is used,
545 we actually have to throw the 'old' MASTER away once the new one has been
546 updated. However if the cluster is undergoing heavy modifications the
547 updated MASTER will be out of date almost the instant its source is
548 snapshotted. Reintegrating a MASTER thus requires a somewhat more complex
551 - If a MASTER is really out of date we can run one or more
552 synchronization passes concurrent with modifying operations.
553 The quorum can remain live.
555 - A final synchronization pass is required with quorum operations
556 blocked to reintegrate the now up-to-date MASTER into the cluster.
561 Quorum operations can be broken down into HARD BLOCK operations and NETWORK
562 operations. If your MASTERs are all local mounts, then failures and
563 sequencing is easy to deal with.
565 Quorum operations on a networked cluster are more complex. The problems:
567 - Masters cannot rely on clients to moderate quorum transactions.
568 Apart from the reliance being unsafe, the client could also
569 lose contact with one or more masters during the transaction and
570 leave one or more masters out-of-sync without the master(s) knowing
571 they are out of sync.
573 - When many clients are present, we do not want a flakey network
574 link from one to cause one or more masters to go out of
575 synchronization and potentially stall the whole works.
577 - Normal hammer2 mounts allow a virtually unlimited number of modifying
578 transactions between actual flushes. The media flush rolls everything
579 up into a single transaction id per flush. Detection of 'missing'
580 transactions in a concurrent multi-client setup when one or more client
581 temporarily loses connectivity is thus difficult.
583 - Clients have a limited amount of time to reconnect to a cluster after
584 a network disconnect before their MESI cache states are lost.
586 - Clients may proceed with several transactions before knowing for sure
587 that earlier transactions were completely successful. Performance is
588 important, we won't be waiting for a full quorum-verified synchronous
589 flush to media before allowing a system call to return.
591 - Masters can decide that a client's MESI cache states were lost (i.e.
592 that the transaction was too slow) as well.
594 The solutions (for modifying transactions):
596 - Masters handle quorum confirmation amongst themselves and do not rely
597 on the client for that purpose.
599 - A client can connect to one or more masters regardless of the size of
600 the quorum and can submit modifying operations to a single master if
601 desired. The master will take care of the rest.
603 A client must still validate the quorum (and obtain MESI cache states)
604 when doing read-only operations in order to present the correct data
605 to the user process for the VOP.
607 - Masters will run a 2-phase commit amongst themselves, often concurrent
608 with other non-conflicting transactions, and will serialize operations
609 and/or enforce synchronization points for 2-phase completion on
610 serialized transactions from the same client or when cache state
611 ownership is shifted from one client to another.
613 - Clients will usually allow operations to run asynchronously and return
614 from system calls more or less ASAP once they own the necessary cache
615 coherency locks. The client can select the validation mode to wait for
618 (1) Fully async (mount -o async)
619 (2) Wait for phase-1 ack (mount)
620 (3) Wait for phase-2 ack (mount -o sync) (fsync - wait p2ack)
621 (4) Wait for flush (mount -o sync) (fsync - wait flush)
623 Modifying system calls cannot be told to wait for a full media
624 flush, as full media flushes are prohibitively expensive. You
625 still have to fsync().
627 The fsync wait mode for network links can be selected, either to
628 return after the phase-2 ack or to return after the media flush.
629 The default is to wait for the phase-2 ack, which at least guarantees
630 that a network failure after that point will not disrupt operations
631 issued before the fsync.
633 - Clients must adjust the chain state for modifying operations prior to
634 releasing chain locks / returning from the system call, even if the
635 masters have not finished the transaction. A late failure by the
636 cluster will result in desynchronized state which requires erroring
637 out the whole filesystem or resynchronizing somehow.
639 - Clients can opt to keep a record of transactions through the phase-2
640 ack or the actual media flush on the masters.
642 However, replaying/revalidating the log cannot necessarily guarantee
643 success. If the masters lose synchronization due to network issues
644 between masters (or if the client was mounted fully-async), or if enough
645 masters crash simultaniously such that a quorum fails to flush even
646 after the phase-2 ack, then it is possible that by the time a client
647 is able to replay/revalidate, some other client has squeeded in and
648 committed something that would conflict.
650 If the client crashes it works similarly to a crash with a local storage
651 mount... many dirty buffers might be lost. And the same happens in
656 Keeping a short-term transaction log, much less being able to properly replay
657 it, is fraught with difficulty and I've made it a separate development task.