Modify ktr(4) to be typesafe - Use a struct copy to place the logged values directly into the destination buffer without forcing a memcpy. - This also fixes event logging on x86_64, as we no longer rely on the i386 calling convention for functions with a variable number of arguments. - Verify that the arguments match the specified format. - Update callsites. Note that this requires userspace changes! ktrdump(8) needs to be adjusted even on i386 (%c and %hd are no longer promoted as per the C rules). We need very special handling on x86_64 too.
systimer: Fix statclock() intr% counting for i8254 interrupt cputimer If i8254 is used as interrupt cputimer, it will send IPIs to APs to get statclock() called. In order to detect fast interrupt, statclock() tests gd_intr_nesting_level to see whether it should bump intr% or not, however, during IPI processing, gd_intr_nesting_level is bumped up, which tricks statclock() to believe that intr% should be incremented. This usually leads to 100% interrupt time on APs reported by utilities like systat(1). To solve this problem: - Pass in_ipi to systimer functions to indicate whether the functions are called during IPI processing or not. - In statclock(): if it is called during IPI processing, using (gd_intr_nesting_level - 1) to test whether intr% should be incremented or not. Additional cleanup: - Make systimer_init_*() interface stricter, by using "systimer_func_t" instead of "void *"
kernel - Add support for up to 63 cpus & 512G of ram for 64-bit builds. * Increase SMP_MAXCPU to 63 for 64-bit builds. * cpumask_t is 64 bits on 64-bit builds now. It remains 32 bits on 32-bit builds. * Add #define's for atomic_set_cpumask(), atomic_clear_cpumask, and atomic_cmpset_cpumask(). Replace all use cases on cpu masks with these functions. * Add CPUMASK(), BSRCPUMASK(), and BSFCPUMASK() macros. Replace all use cases on cpu masks with these functions. In particular note that (1 << cpu) just doesn't work with a 64-bit cpumask. Numerous bits of assembly also had to be adjusted to use e.g. btq instead of btl, etc. * Change __uint32_t declarations that were meant to be cpu masks to use cpumask_t (most already have). Also change other bits of code which work on cpu masks to be more agnostic. For example, poll_cpumask0 and lwp_cpumask. * 64-bit atomic ops cannot use "iq", they must use "r", because most x86-64 do NOT have 64-bit immediate value support. * Rearrange initial kernel memory allocations to start from KvaStart and not KERNBASE, because only 2GB of KVM is available after KERNBASE. Certain VM allocations with > 32G of ram can exceed 2GB. For example, vm_page_array[]. 2GB was not enough. * Remove numerous mdglobaldata fields that are not used. * Align CPU_prvspace[] for now. Eventually it will be moved into a mapped area. Reserve sufficient space at MPPTDI now, but it is still unused. * When pre-allocating kernel page table PD entries calculate the number of page table pages at KvaStart and at KERNBASE separately, since the KVA space starting at KERNBASE caps out at 2GB. * Change kmem_init() and vm_page_startup() to not take memory range arguments. Instead the globals (virtual_start and virtual_end) are manipualted directly.
network - Major netmsg retooling, part 1 * Remove all the netmsg shims and make all pr_usrreqs and some proto->pr_* requests directly netmsg'd. * Fix issues with tcp implied connects and tcp6->tcp4 fallbacks with implied connects. * Fix an issue with a stack-based udp netmsg (allocate it) * Consolidate struct ip6protosw and struct protosw into a single structure and normalize the API functions which differed between the two (primarily proto->pr_input()). * Remove protosw->pr_soport() * Replace varargs protocol *_input() functions (ongoing) with fixed arguments.
network - Completely revamp the netisr / dispatch code * All netisrs are dispatched MPSAFE (followup in later commits) * Centralize the protocol threads. There is now just one thread per cpu managed by the netisr code. No more separate tcp/udp threads on each cpu. * Make the mbuf M_HASH/m_pkthdr.hash mechanic the central routing mechanic for netmsgs. * Remove the netisr ip_mport and pktinfo_portfn stuff and replace with a cpufn function which handles M_HASH/m_pkthdr.hash when M_HASH is not already set. * Seriously clean up the packet input paths. Adjust ether_input_chain() and friends to not have to adjust the mbuf forwards and backwards, instead pass a header offset to the ni_cpufn function. The ip pullup and other related code will use the offset to determine where the ip header is within the packet.
network - Move socket from netmsg ext to netmsg header, add port to socket These changes should make it easier to debug mbufs routed to the wrong protocol threads. Note that IPV6 is routed to netisr0, even for tcp and udp packets, so for now we do not KKASSERT that the port matches directly. The TCP code still KKASSERTs that the timers are running on the correct cpu, since the cpu is still correct for IPV6 TCP/UDP packets even when the protocol thread is wrong (netisr_cpu 0 instead of tcp_thread 0 or udp_thread 0). * Instead of recalculating the port based on the inp or mbuf all the time, add a so_port field to the socket structure directly. * The socket pointer is now part of the netmsg header, even though some subsystems do not need it. This allows us to validate the message port more easily.
Split ifnet serialize step 1/many: Add if_{serialize,deserialize,tryserialize}() function pointers to ifnet. These three function pointers accept ifnet struct and ifnet_serialize enumeration. The ifnet_serialize enumeration indicates the serialization type: IFNET_SERIALIZE_ALL: All of the serializers should be held. Except for if_start and if_input, this enumeration must be used when call ifnet function pointers. IFNET_SERIALIZE_TX: Only transmit serializer should be held. This enumeration could be used when calling ifnet.if_start. IFNET_SERIALIZE_RX: Only receive serializer should be held. This enumeration could be used when calling ifnet.if_input. If the NIC driver does not set these three function pointer, then if_attach() will set them to the default ones: only one serializer (if_serializer) is used and ifnet_serialize parameter is ignored. Following several inline functions are added which are sheer wrappers of the three ifnet serialize function pointers: ifnet_serialize_{all,tx,rx}() ifnet_deserialize_{all,tx,rx}() ifnet_tryserialize_{all,tx,rx}() All of the protocol layers and most of the pseudo drivers are converted. Discussed-with: dillon@
- Move polling disablement, i.e. deregistering all NICs, out of hot code path. - When the last NIC is deregistered or polling is disabled, reset all interval state. - Return immediately in netisr_poll/netisr_pollmore if no NIC is registered. - Add assertion in netisr_poll/netisr_pollmore: it is not possible that polling is not enabled but there are registered NIC. - Comment pollctx structure fields.