Reduce the cost of TLB invalidation on x86 by using per-CPU completion flags
authorcem <cem@FreeBSD.org>
Fri, 30 Sep 2016 18:12:16 +0000 (18:12 +0000)
committercem <cem@FreeBSD.org>
Fri, 30 Sep 2016 18:12:16 +0000 (18:12 +0000)
commit22e3a710d0213e9f7472d30d8a1c6631091d6d77
tree765794f0fc46ef30ad12d910baf132f68752f163
parent2d5d973470dd4e8db861b126e8ebcc5559af5ff2
Reduce the cost of TLB invalidation on x86 by using per-CPU completion flags

Reduce contention during TLB invalidation operations by using a per-CPU
completion flag, rather than a single atomically-updated variable.

On a Westmere system (2 sockets x 4 cores x 1 threads), dtrace measurements
show that smp_tlb_shootdown is about 50% faster with this patch; observations
with VTune show that the percentage of time spent in invlrng_single_page on an
interrupt (actually doing invalidation, rather than synchronization) increases
from 31% with the old mechanism to 71% with the new one.  (Running a basic file
server workload.)

Submitted by: Anton Rang <rang at acm.org>
Reviewed by: cem (earlier version), kib
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D8041
sys/amd64/amd64/mp_machdep.c
sys/amd64/include/pcpu.h
sys/x86/include/x86_smp.h
sys/x86/x86/mp_x86.c