socket: Limit the number of accepted sockets that kevent reports.
authorSepherosa Ziehau <sephe@dragonflybsd.org>
Thu, 5 Oct 2017 06:06:11 +0000 (14:06 +0800)
committerSepherosa Ziehau <sephe@dragonflybsd.org>
Thu, 5 Oct 2017 06:09:10 +0000 (14:09 +0800)
commit65e531c6c247b945f241dc189d6cffd14e2bb893
tree741408ab1256fd14c4395a43f84b6d630d0231f5
parentba3d86e75043bb4772f7fabd852c89abdc6d3d5b
socket: Limit the number of accepted sockets that kevent reports.

By default it is limited to 32.  It can be changed through:
sysctl kern.ipc.soavailconn=X

This change does _not_ affect userland using accept(2) in the following
way:
    for (;;) {
        s = accept();
        if (s < 0 && errno == EAGAIN)
            break;
        /* Processing accepted socket. */
    }

This change only affects optimized userland using kevent.data to avoid
extra accept(2) syscall:
    for (i = 0; i < kevent.data; ++i) {
        s = accept();
        /* Processing accepted socket. */
    }

The above logic is applied by nginx.  However, due to the cost of the
"Processing accepted socket" parts, this kinda of loop can increase
latency and destablize latency.

The comparison w/ 30K concurrent connections, 1 request/connection.

 1K web object
         |  performance |  lat-avg | lat-stdev |  lat-99%
---------+--------------+----------+-----------+----------
no limit | 210279.88tps |  59.19ms |    4.60ms |  69.02ms
---------+--------------+----------+-----------+----------
32 limit | 217599.01tps |  32.00ms |    2.35ms |  35.59ms

========

 8K web object
         |  performance |  lat-avg | lat-stdev |  lat-99%
---------+--------------+----------+-----------+----------
no limit | 180627.61tps |  70.53ms |    4.95ms |  80.61ms
---------+--------------+----------+-----------+----------
32 limit | 186324.41tps |  37.41ms |    4.81ms |  48.69ms

========

16K web object
         |  performance |  lat-avg | lat-stdev |  lat-99%
---------+--------------+----------+-----------+----------
no limit | 138667.84tps |  95.93ms |   14.90ms | 135.47ms
---------+--------------+----------+-----------+----------
32 limit | 138778.11tps |  60.90ms |   11.80ms |  92.07ms

This change significantly reduces average latency and .99 latency,
and performance is improved slightly.
sys/kern/uipc_socket.c