udp: Make udp pcbinfo and portinfo per-cpu; greatly improve performance

udp: Make udp pcbinfo and portinfo per-cpu; greatly improve performance

MAJOR CHANGES:

- Add token to protect pcbinfo's inpcb list and wildcard hash table.
  Currently only udp per-cpu pcbinfo sets this token.  udp serializer
  and netisr barrier are nuked.

  o  udp inpcb list:

     Under most cases, udp inpcb list is operated in its owner netisr.
     However, it is also accessed and modified (no effiective udp inpcb
     will be unlinked though) in netisr0 to adjust multicast options if
     one interface is to be detached.  So protecting udp inpcb list
     accessing and modification w/ token is necessary.

     At udp inpcb detach time, the udp inpcb is first removed from the
     udp inpcb list, then a message will go through all netisrs, which
     makes sure that no netisrs are using or can find this udp inpcb
     from the udp inpcb list.  After all these, this udp inpcb is
     destroyed in its owner netisr.

     In netisrs, it is MP safe to find a udp inpcb from udp inpcb list,
     then release the token and process the found udp inpcb.

     In other threads, it is MP safe to find a udp inpcb from udp inpcb
     list, then release the token and process the found udp inpcb in
     non-blocking fashion.

     See also the usage of inpcb marker.

  o  udp wildcard hash table:

     On input path, udp wildcard hash table is searched in its owner
     netisr.  In order to ease implicit binding (bind during send),
     connect after binding, and disconnect, udp inpcb are inserted
     into and removed from other udp pcbinfos' wildcard hash table in
     its owner netisr.  Thus the udp wildcard hash table must be
     protected w/ token.

     At udp inpcb detach time, a message will go through all netisrs,
     and this udp inpcb will be removed from the udp wildcard hash
     table belonging to the current netisr.  This makes sure that once
     the current netisr runs the message handler, this udp inpcb will
     not be used and be found in the current netisr.  When the message
     reaches the last netisr, this udp inpcb is redispatched to its
     owner netisr to be destroyed.

     In netisrs, it is MP safe to find a udp inpcb from udp wildcard
     hash table, then release the token and process the found udp inpcb,
     e.g. use udp inpcb found by in_pcblookuphash().

     In other threads, it is MP safe to find a udp inpcb from udp
     wildcard hash table, then release the token and process the found
     udp inpcb in non-blocking fashion.

     See also the usage of inpcb container marker.

  o  udp connect hash table:

     It is lockless MP safe, and only accessed and modified in its owner
     netisr.

- During inpcb iteration through inpcb list, use inpcb marker when
  calling functions, which may block, e.g. in_pcbpurgeif0(), so the
  inpcb iteration will not stop prematurely, if the inpcb being
  processed is removed from the inpcb list.

- Use udp inpcb wildcard table and udp inpcb connect hash table to
  dispatch input multicast and broadcast udp datagrams.  Using udp inpcb
  list could be time consume, since we need to check udp inpcb lists on
  all cpus; and secondly, once udp inpcb has a local port, it will be in
  either udp wildcard hash table or udp connect hash table.

  Since the socket buffer operation on input path may block, inpcb
  container marker is used when iterating inpcbs from udp inpcb wildcard
  hash table.  in_pcblookup_pkthash() is adjusted to skip inpcb
  container marker.

- udp socket so_port is no longer fixed to netisr0 msgport
  o  Initial udp socket so_port is the current cpu's netisr msgport.
  o  Bound but unconnected udp socket so_port is selected according to
     local port hash.
  o  Connected udp socket so_port is selected according to the udp hash,
     i.e. laddr/faddr toeplitz hash (exception: multicast laddr or
     multicast faddr, is hashed to netisr0).
  o  Multicast socket options are forced to be handled in netisr0, since
     udp socket so_port may not be netisr0 msgport.

- In order to support asynchronized udp inpcb detach:
  o  EJUSTRETURN from pru_detach method now means protocol will call
     sodiscard() and sofree() for soclose().  udp pru_detach method
     returns EJUSTRETURN as of this commit.
  o  SS_ISCLOSING socket state is set before calling pru_detach method,
     so protocol could avoid certain expensive, unnecessary or
     disallowed operation in pru_disconnect or pru_detach method, e.g.
     udp pru_disconnect method avoids putting udp inpcb back to udp
     wildcard hash table, if SS_ISCLOSING is set.

MISC CHANGES:

- pcbinfo's cpu id must be set now; -1 is disallowed.
- udp pru_abort method should never be called; it panicks now.
- Restore traditional BSD behaviour, if unbound udp socket connect
  fails: if local port of the udp socket has been selected, its inpcb
  should be in wildcard hash table, i.e. the udp inpcb should be visible
  on udp datagrams input path.
- Make sure multicast stuffs are adjusted only in netisr0 for inet6, if
  one interface is about to be detached.

PERFORMANCE IMPROVEMENT:

For 'kq_connect_client -u' test, this commit gives 400% performance
improvement (31Kconns/s -> 160Kconns/s).