sys/vfs/hammer: Use HAMMER_ENCODE() for hammer_mod_rb_compare()

sys/vfs/hammer: Use HAMMER_ENCODE() for hammer_mod_rb_compare()

Try not to hardcode 8 for volume fields.

The 60 bits offset format changes, but the change doesn't affect
RB-Tree's elements comparison. It works as long as it has volume
bits and 52 bits offset within that volume. Whether HAMMER_ENCODE()
is used or not, io[12]_offset are not valid offsets anyway.
These offset values just need to be unique for physical locations.

(A) before this commit
----oooooooooooooooooooooooooooooooooooooooooooooooooooovvvvvvvv
(B) this commit
----vvvvvvvvoooooooooooooooooooooooooooooooooooooooooooooooooooo