kernel - Fix issue with ARP packets stalling out entire network
* ARP packets can cause ARP routing table updates to occur. An ARP
routing table update is an expensive synchronous netmsg that is
forwarded through *ALL* cpus.
* ARP was previously being handled by netisr 0 and on large multi-way
machines (aka monster the 48-way opteron) under very heavy loads this
could result in very long stalls for any packet processing forwarded
to cpu 0.
Stalls exceeding 200 seconds were observed on monster when a large
number of ARP packets had to be processed.
* Implement a dedicated thread feature for the NETISR mechanism and
modify NETISR_ARP to use it. This takes the expensive synchronous
ARP packet processing off the general per-cpu netisr threads.
This thread currently runs on cpu (18 % ncpus) (NETISR_ARP == 18).
Thus the general per-cpu (netisr 0) thread will no longer stall
on ARP packets.
* ping latencies under extreme loads improved to (approximately):
ping -i 0.001 monster-nr
11735 packets transmitted, 11735 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.073/0.190/27.019/0.382 ms