kernel - Greatly improve shared memory fault rate concurrency / shared tokens

kernel - Greatly improve shared memory fault rate concurrency / shared tokens

This commit rolls up a lot of work to improve postgres database operations
and the system in general.  With this changes we can pgbench -j 8 -c 40 on
our 48-core opteron monster at 140000+ tps, and the shm vm_fault rate
hits 3.1M pps.

* Implement shared tokens.  They work as advertised, with some cavets.

  It is acceptable to acquire a shared token while you already hold the same
  token exclusively, but you will deadlock if you acquire an exclusive token
  while you hold the same token shared.

  Currently exclusive tokens are not given priority over shared tokens so
  starvation is possible under certain circumstances.

* Create a critical code path in vm_fault() using the new shared token
  feature to quickly fault-in pages which already exist in the VM cache.
  pmap_object_init_pt() also uses the new feature.

  This increases fault-in concurrency by a ridiculously huge amount,
  particularly on SHM segments (say when you have a large number of postgres
  clients).  Scaling for large numbers of clients on large numbers of
  cores is significantly improved.

  This also increases fault-in concurrency for MAP_SHARED file maps.

* Expand the breadn() and cluster_read() APIs.  Implement breadnx() and
  cluster_readx() which allows a getblk()'d bp to be passed.  If *bpp is not
  NULL a bp is being passed in, otherwise the routines call getblk().

* Modify the HAMMER read path to use the new API.  Instead of calling
  getcacheblk() HAMMER now calls getblk() and checks the B_CACHE flag.
  This gives getblk() a chance to regenerate a fully cached buffer from
  VM backing store without having to acquire any hammer-related locks,
  resulting in even faster operation.

* If kern.ipc.shm_use_phys is set to 2 the VM pages will be pre-allocated.
  This can take quite a while for a large map and also lock the machine
  up for a few seconds.  Defaults to off.

* Reorder the smp_invltlb()/cpu_invltlb() combos in a few places, running
  cpu_invltlb() last.

* An invalidation interlock might be needed in pmap_enter() under certain
  circumstances, enable the code for now.

* vm_object_backing_scan_callback() was failing to properly check the
  validity of a vm_object after acquiring its token.  Add the required
  check + some debugging.

* Make vm_object_set_writeable_dirty() a bit more cache friendly.

* The vmstats sysctl was scanning every process's vm_map (requiring a
  vm_map read lock to do so), which can stall for long periods of time
  when the system is paging heavily.  Change the mechanic to a LWP flag
  which can be tested with minimal locking.

* Have the phys_pager mark the page as dirty too, to make sure nothing
  tries to free it.

* Remove the spinlock in pmap_prefault_ok(), since we do not delete page
  table pages it shouldn't be needed.

* Add a required cpu_ccfence() in pmap_inval.c.  The code generated prior
  to this fix was still correct, and this makes sure it stays that way.

* Replace several manual wiring cases with calls to vm_page_wire().