From: Sepherosa Ziehau <sephe@dragonflybsd.org>
Date: Thu, 5 Oct 2017 06:06:11 +0000 (+0800)
Subject: socket: Limit the number of accepted sockets that kevent reports.
X-Git-Tag: v5.0.0~38
X-Git-Url: https://gitweb.dragonflybsd.org/dragonfly.git/commitdiff_plain/5fcc484ea290d8d3f6c0a15ea6bbce166ddf6dbf

socket: Limit the number of accepted sockets that kevent reports.

By default it is limited to 32.  It can be changed through:
sysctl kern.ipc.soavailconn=X

This change does _not_ affect userland using accept(2) in the following
way:
    for (;;) {
        s = accept();
        if (s < 0 && errno == EAGAIN)
            break;
        /* Processing accepted socket. */
    }

This change only affects optimized userland using kevent.data to avoid
extra accept(2) syscall:
    for (i = 0; i < kevent.data; ++i) {
        s = accept();
        /* Processing accepted socket. */
    }

The above logic is applied by nginx.  However, due to the cost of the
"Processing accepted socket" parts, this kinda of loop can increase
latency and destablize latency.

The comparison w/ 30K concurrent connections, 1 request/connection.

 1K web object
         |  performance |  lat-avg | lat-stdev |  lat-99%
---------+--------------+----------+-----------+----------
no limit | 210279.88tps |  59.19ms |    4.60ms |  69.02ms
---------+--------------+----------+-----------+----------
32 limit | 217599.01tps |  32.00ms |    2.35ms |  35.59ms

========

 8K web object
         |  performance |  lat-avg | lat-stdev |  lat-99%
---------+--------------+----------+-----------+----------
no limit | 180627.61tps |  70.53ms |    4.95ms |  80.61ms
---------+--------------+----------+-----------+----------
32 limit | 186324.41tps |  37.41ms |    4.81ms |  48.69ms

========

16K web object
         |  performance |  lat-avg | lat-stdev |  lat-99%
---------+--------------+----------+-----------+----------
no limit | 138667.84tps |  95.93ms |   14.90ms | 135.47ms
---------+--------------+----------+-----------+----------
32 limit | 138778.11tps |  60.90ms |   11.80ms |  92.07ms

This change significantly reduces average latency and .99 latency,
and performance is improved slightly.
---

diff --git a/sys/kern/uipc_socket.c b/sys/kern/uipc_socket.c
index 7b46953e23..aef6a69de5 100644
--- a/sys/kern/uipc_socket.c
+++ b/sys/kern/uipc_socket.c
@@ -152,6 +152,10 @@ static int use_socreate_fast = 1;
 SYSCTL_INT(_kern_ipc, OID_AUTO, socreate_fast, CTLFLAG_RW,
     &use_socreate_fast, 0, "Fast socket creation");
 
+static int soavailconn = 32;
+SYSCTL_INT(_kern_ipc, OID_AUTO, soavailconn, CTLFLAG_RW,
+    &soavailconn, 0, "Maximum available socket connection queue size");
+
 /*
  * Socket operation routines.
  * These routines are called by the routines in
@@ -2615,7 +2619,11 @@ static int
 filt_solisten(struct knote *kn, long hint __unused)
 {
 	struct socket *so = (struct socket *)kn->kn_fp->f_data;
+	int qlen = so->so_qlen;
+
+	if (soavailconn > 0 && qlen > soavailconn)
+		qlen = soavailconn;
+	kn->kn_data = qlen;
 
-	kn->kn_data = so->so_qlen;
-	return (! TAILQ_EMPTY(&so->so_comp));
+	return (!TAILQ_EMPTY(&so->so_comp));
 }