ifnet: Allow drivers to adjust mbuf cluster/jcluster limits This is mainly for raising mbuf clusters/jclusters limits to a high enough value for device reception queues, e.g. modern network devices w/ multiple reception queues and each reception queue could consume >=512 mbuf clusters.
ifnet: Make blocking operation in if_addrheads iteration MPSAFE Two methods are used: - Dispatch blocking operation in if_addrheads iteration to netisr0 to run for functions which do not copy data into user space. - Use ifaddr marker to make sure that the ifaddr list is not broken if the ifaddr list is changed during the blocking operation. And reference the ifaddr being used so it will not be destroyed during the blocking operation.
ifnet: Make ifnet and ifindex2ifnet MPSAFE - Accessing to these two global variables from non-netisr threads uses ifnet lock. This kind of accessing is from - Accessing to ifindex2ifnet from netisrs are lockless MPSAFE. - Netisrs no longer access ifnet, instead they access ifnet array as of this commit, which is lockless MPSAFE. Rules for accessing ifnet and ifindex2ifnet is commented near the declaration of the related global variables/functions in net/if_var.h.
ifnet: Properly protect if_multiaddrs using ifnet serializers - Protect ifnet.if_multiaddrs using ifnet serializers. Add some comment in the places, where only main serailizer is necessary. - Fix if_delallmulti(). Using TAILQ_FOREACH_MUTABLE is incorrect for deleting an ifmultiaddr from ifnet.if_multiaddrs. Since deleting one ifmultiaddr may cause additional ifmultiaddr deletion (e.g. the AF_LINK ifmultiaddr for AF_INET ifmultiaddr). - Change IN_LOOKUP_MULTI and IN6_LOOKUP_MULTI macros into inline functions. - Redispatch multicast IP packets to netisr0 for further processing. Software based IP packet hash function is changed. And hash value fixup for multicast IP packets is added to the beginning of ip_input(); this is mainly for IP packets, whose hash is calculated by hardware. - For wlan's multicast hardware filter updating, we no longer need to release wlan serializer and mess up w/ the if_ioctl setting. In netisr0, read and test ifma_refcount for AF_INET ifmultiaddr is MPSAFE w/o ifnet serializers, since its ifma_refcount is only altered in netisr0. In netisr0, any operation on in_multi, which is obtained from the corresponding ifmuliaddr's ifma_protospec, is MPSAFE w/o ifnet serializers, since ifmultiaddr for AF_INET is only set and cleared in netisr0. While I'm here also redispatch IP packets w/o hash to the proper netisrs, on ip_input() path. And unnecessary critical sections in in_{add,del}multi() are removed.
ethernet: If caller thread cpu is fixed, pass cpuid to ether_input_pkt() So we could use optimized lwkt_sendmsg_oncpu() instead of lwkt_sendmsg(), if the target netisr is on the same cpu as caller thread cpu. Mainly to avoid unnecessary wakeup() IPIs to other cpus. THE RESULT: On i7-3770 w/ HT enabled (8 logical cpus); NIC is 82599ES w/ 8 RX rings and 8 TX rings. Run: repeat 10 tcp_stream -H ... -i 256 -l 10 -r (256 netperf TCP_MAERTS instances for 10 seconds, 10 rounds) Total amount of cross IPIs before this commit is 6946097. Total amount of cross IPIs as of this commit is 5445324. ~22% unnecessary wakeup() IPIs are avoided!
ifnet: Update and add comment; no functional changes - Update various comment - Remove no longer applied comment - Add comment for ifaddr per-cpu container - Add comment for per-cpu ifnet and ifaddr statistics - Add comment for ifnet single serializer mode and multiple serializers mode - Add pseudo code in comment for ifnet multiple transmit queue support
ifsq: Let ifaltq_subque know its related hardware TX queue's serializer This avoids following operations on packet transmission hot path: - Dereferening device driver supplied serialize function pointers - Locating hardware TX queue's serializer Comparing to the lwkt_serialize functions, the above two operations are costful. Driver changes: - For device drivers which use the default ifnet serializer, no additional code will be needed, if_attach() will assign ifnet serializer to ifaltq_subque. - For device drivers which use independent serializers for main function, RX queues and TX queues, ifsq_set_hw_serialize() must be called to properly assign the hardware TX queue's serializer to ifaltq_subque. Drivers in this category are bce(4), emx(4), igb(4) and jme(4).
if: Per-cpu ifnet/ifaddr statistics, step 3/3 Make ifnet stats per-cpu. The per-cpu stats are stored in cache aligned ifdata_pcpu struct. - if_mib code is adjusted, so callers of the IFMIB_IFDATA sysctl will not aware of the ifnet per-cpu stats, e.g. systat -if - User space programs that directly access ifnet.if_data are modified to aware of the new location of the ifnet stats, e.g. netstat -in