$ guides / varnish / varnish-sess-fail ▌

Operations Guides

Varnish sess_fail: session accept failures at the front door

When MAIN.sess_fail starts incrementing in Varnish, new TCP connections are failing at the accept() call. Varnish never brings them into the worker pipeline. Clients experience connection resets, timeouts, or refused connections. Unlike sess_dropped (where the connection was accepted but the thread queue was full), sess_fail means Varnish never got far enough to process the request.

The counter is an aggregate. Since Varnish 6.1, it decomposes into sub-counters that isolate the cause: sess_fail_emfile for file descriptor exhaustion, sess_fail_econnaborted for client-side aborts, sess_fail_enomem for memory pressure, and several others. Reading only the aggregate counter is a common diagnostic mistake.

There is also a class of connection failures that never reaches Varnish counters. If the kernel’s listen backlog (net.core.somaxconn or net.ipv4.tcp_max_syn_backlog) overflows, the kernel drops SYN packets or half-open connections before Varnish’s accept thread sees them. These drops are invisible to sess_fail and to every other Varnish counter. You must check kernel state separately.

What this means

MAIN.sess_fail is defined by the Varnish documentation as: “Count of failures to accept TCP connection. This counter is the sum of the sess_fail_* counters, which give more detailed information.”

The accept thread is the entry point for all client traffic. It runs in a tight loop calling accept() on the listening socket. Each successful accept() returns a new file descriptor, which is then handed to a worker thread. When accept() returns an error instead of a file descriptor, sess_fail increments and the connection is lost.

Varnish applies backpressure when certain accept errors recur. For sess_fail_emfile, sess_fail_ebadf, and sess_fail_enomem, Varnish calls an internal pacing function (vca_pace_bad) that accumulates a delay before retrying the accept loop. The delay is controlled by two runtime parameters: acceptor_sleep_incr (default 0, meaning pacing is disabled by default) and acceptor_sleep_max (default 0.05s ceiling). This prevents a tight spin on a persistent error when pacing is enabled, but means that under sustained FD exhaustion, new connections are not just rejected once but throttled.

The harmless sub-counters, sess_fail_econnaborted and sess_fail_eintr, do not trigger pacing. sess_fail_econnaborted means the client closed the connection before Varnish finished accepting it. sess_fail_eintr means a signal interrupted the accept() call. Both are routine on any production server with real traffic.

flowchart TD
    A[New TCP connection] --> B{Kernel backlog full?}
    B -- yes --> C[Dropped by kernel\nInvisible to Varnish counters]
    B -- no --> D[Varnish accept thread]
    D --> E{accept returns?}
    E -- error --> F[sess_fail increments]
    F --> G[Decompose via sess_fail sub-counters]
    E -- success --> H{Thread available or queue space?}
    H -- no --> I[sess_dropped or req_dropped]
    H -- yes --> J[Request processed]

Common causes

Cause	What it looks like	First thing to check
FD exhaustion	`sess_fail_emfile` incrementing; FD count near `RLIMIT_NOFILE`	`/proc/$CHILD_PID/limits` for Max open files
Kernel backlog overflow	Connections dropped but `sess_fail` is zero; clients see timeouts	`/proc/sys/net/core/somaxconn` and `tcp_max_syn_backlog`
Memory pressure	`sess_fail_enomem` incrementing; system memory constrained	`dmesg` for OOM activity; process RSS
Client aborts (harmless)	`sess_fail_econnaborted` incrementing at low rate	Normal on production servers; no action needed
Other accept errors	`sess_fail_other` incrementing	`varnishlog -g raw -i SessError` for the errno string
Listen socket invalid	`sess_fail_ebadf` incrementing	Should never happen; indicates bug or corruption

Quick checks

# Decompose sess_fail into sub-counters
varnishstat -1 -f MAIN.sess_fail -f 'MAIN.sess_fail_*'

# Child process FD usage (newest varnishd process is the child)
CHILD_PID=$(pgrep -n varnishd)
ls /proc/$CHILD_PID/fd | wc -l
cat /proc/$CHILD_PID/limits | grep 'Max open files'

# Kernel listen backlog limits
cat /proc/sys/net/core/somaxconn
cat /proc/sys/net/ipv4/tcp_max_syn_backlog

# Varnish listen_depth parameter
varnishadm param.show listen_depth

# Thread queue and drops (related but distinct from sess_fail)
varnishstat -1 -f MAIN.thread_queue_len -f MAIN.sess_dropped -f MAIN.req_dropped

# Kernel-level socket overflows (invisible to Varnish)
nstat -az TcpExtListenOverflows TcpExtListenDrops

# Inspect sess_fail_other errors with errno strings
varnishlog -g raw -i SessError

# Total sessions accepted vs failed
varnishstat -1 -f MAIN.sess_conn -f MAIN.sess_fail

How to diagnose it

Read the sub-counters, not just the aggregate. Run varnishstat -1 -f 'MAIN.sess_fail_*' and identify which sub-counter is incrementing.
If sess_fail_emfile is the culprit, check the child process FD limit and current usage:
```
CHILD_PID=$(pgrep -n varnishd)
echo "FDs in use: $(ls /proc/$CHILD_PID/fd | wc -l)"
cat /proc/$CHILD_PID/limits | grep 'Max open files'
```
If usage is near the limit, the problem is FD exhaustion. Both client connections and backend connections consume FDs. Check the backend connection reuse ratio to rule out a connection leak.
If clients report connection failures but sess_fail is zero or only shows harmless sub-counters (sess_fail_econnaborted, sess_fail_eintr), the drops may be happening at the kernel level. Check:
```
cat /proc/sys/net/core/somaxconn
cat /proc/sys/net/ipv4/tcp_max_syn_backlog
nstat -az TcpExtListenOverflows TcpExtListenDrops
```
If TcpExtListenOverflows is incrementing, the kernel is dropping connections because its backlog queue is full. Varnish never sees these connections, and no Varnish counter will reflect the loss.
If sess_fail_enomem is incrementing, the system is under memory pressure. The Varnish documentation describes this as “most likely insufficient socket buffer memory.” Check dmesg for OOM killer activity and monitor process RSS versus available system memory.
If sess_fail_other is incrementing, capture the specific error:
```
varnishlog -g raw -i SessError
```
The SessError log tag shows the errno string, which identifies the exact system call failure.
Distinguish from thread pool drops. If clients are experiencing failures but sess_fail is zero, check sess_dropped and req_dropped instead. Those counters track sessions that were accepted but dropped because the thread queue (thread_queue_limit, default 20) was full. The failure mode, the counters, and the fix are all different.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`MAIN.sess_fail`	Summary of all accept() failures	Any sustained nonzero rate
`MAIN.sess_fail_emfile`	Confirms FD exhaustion as the cause	Any nonzero value
`MAIN.sess_fail_enomem`	Socket buffer memory pressure	Any nonzero value
`MAIN.sess_fail_other`	Unclassified accept errors needing log investigation	Any nonzero value
`MAIN.sess_fail_econnaborted`	Client aborted before accept completed	Normal at low rates; investigate spikes
`/proc/$PID/fd` count vs FD limit	Approaching FD ceiling	Usage above 80% of limit
`net.core.somaxconn`	Kernel listen queue depth	Value lower than Varnish `listen_depth` (default 1024)
`TcpExtListenOverflows` (nstat)	Kernel dropping connections before Varnish sees them	Any increment
`MAIN.sess_conn` vs `MAIN.sess_fail`	Ratio of successful to failed accepts	Increasing failure ratio
`MAIN.sess_dropped` / `MAIN.req_dropped`	Thread queue full (different failure from sess_fail)	Any sustained nonzero rate

Fixes

File descriptor exhaustion (sess_fail_emfile)

The fix is to raise the FD limit for the Varnish child process. The FD limit is governed by RLIMIT_NOFILE, set via ulimit -n or systemd LimitNOFILE.

Check the current limit:

cat /proc/$CHILD_PID/limits | grep 'Max open files'

If the limit is low (for example 1024 or 4096 on older configurations), raise it. The method depends on how Varnish is started:

systemd: Set LimitNOFILE=131072 (or higher) in the service unit file, then restart Varnish. Restarting flushes the cache and resets in-flight connections.
SysV init or custom script: Add ulimit -n 131072 to the init script before the Varnish start command.

After restarting, verify the new limit took effect by reading /proc/$CHILD_PID/limits.

If FD usage is high even with a generous limit, investigate a connection leak. Both client keepalive connections and backend connections consume FDs. A high backend request rate with low connection reuse can burn through FDs quickly. Check MAIN.backend_conn versus MAIN.backend_reuse. If the reuse ratio is low, every backend fetch opens a new TCP connection, consuming an FD that may not be released promptly.

Kernel backlog overflow (invisible to Varnish)

If connections are being dropped at the kernel level, Varnish counters will not show it. The kernel’s listen queue depth is controlled by two parameters:

net.core.somaxconn: the maximum backlog for all listening sockets. Default was 128 on kernels before 5.4 and 4096 on Linux 5.4 and later.
net.ipv4.tcp_max_syn_backlog: the maximum number of remembered connection requests in SYN-RECEIVED state.

Varnish’s listen_depth parameter defaults to 1024 connections. The kernel’s somaxconn value caps the effective backlog. If somaxconn is lower than listen_depth, the kernel value wins. A server running an older kernel with the default somaxconn=128 effectively limits Varnish to a 128-connection backlog regardless of the Varnish configuration.

Fix:

# These take effect immediately but are not persistent across reboot
sysctl -w net.core.somaxconn=4096
sysctl -w net.ipv4.tcp_max_syn_backlog=8192

Persist these in /etc/sysctl.d/ so they survive reboot. Verify that the new somaxconn is at least as high as Varnish’s listen_depth.

Changing somaxconn does not affect already-open listening sockets. Varnish must be restarted for the new value to take effect on its listen socket.

Memory pressure (sess_fail_enomem)

This counter is described in the Varnish documentation as “most likely insufficient socket buffer memory” and is annotated “should never happen.” When it does appear, it points to system-level memory exhaustion. Check:

dmesg for OOM killer activity targeting the Varnish process or other system processes
Process RSS versus available system memory
Whether transient storage (SMA.Transient.g_bytes) is consuming unbounded memory through pass or pipe traffic

If Varnish itself is consuming too much memory through transient storage growth, investigate the VCL pass rate and consider sizing transient storage explicitly with -s Transient=malloc,1G (available in Varnish 6.1 and later).

Other accept errors (sess_fail_other)

Run varnishlog -g raw -i SessError to capture the errno string. This shows the specific error code from the failed accept() call. Act based on the errno value. If the error is persistent and the cause is unclear from the errno, it may indicate a kernel-level socket issue or a Varnish bug worth reporting upstream.

Distinguish from thread queue drops

If the real problem is sess_dropped or req_dropped (sessions accepted but dropped because the thread queue is full), the fix is different. The session was successfully accepted, so sess_fail is not involved. The fix involves increasing thread_pool_max, reducing backend response latency so threads are released faster, or raising thread_queue_limit. These are concurrency capacity problems, not accept() resource problems.

Prevention

Monitor the sub-counters independently. Alert on sess_fail_emfile and sess_fail_enomem specifically, not just the aggregate. A nonzero sess_fail_econnaborted rate is normal background noise; a nonzero sess_fail_emfile rate is an actionable FD exhaustion condition.
Set FD limits generously. A modern Varnish server handling high traffic should have LimitNOFILE of at least 131072. Both client and backend connections consume FDs, and idle keepalive connections can accumulate.
Verify kernel backlog after kernel upgrades or new deployments. The somaxconn default changed from 128 to 4096 in Linux 5.4, but custom sysctl configurations or containerized environments may override this with lower values. Ensure somaxconn is at least as high as Varnish’s listen_depth.
Monitor TcpExtListenOverflows via nstat. This is the only signal that catches kernel-level connection drops invisible to Varnish counters.
Track backend connection reuse. Low reuse means more open FDs. If the ratio backend_reuse / (backend_reuse + backend_conn) is consistently low, investigate backend keepalive configuration and backend_idle_timeout.
Alert on sess_fail and sess_dropped separately. Both cause client-visible failures but require different fixes. sess_fail is an accept() resource problem (FDs, memory, kernel backlog). sess_dropped is a thread pool capacity problem.

How Netdata helps

Netdata correlates the signals needed to diagnose sess_fail without manual command-line triage:

Per-second sess_fail_* sub-counters show the exact failure cause. Rate views distinguish a brief spike from a sustained problem.
Process file descriptor monitoring correlates FD usage with sess_fail_emfile directly, so the cause is visible without checking /proc by hand.
Kernel TCP metrics (TcpExtListenOverflows, TcpExtListenDrops) catch the class of drops Varnish cannot see. Correlating kernel overflows with Varnish counters separates a kernel backlog problem from a Varnish resource problem.
Thread pool metrics (thread_queue_len, sess_dropped, req_dropped) distinguish accept failures from thread queue drops, which require different fixes.
Backend connection counters (backend_conn, backend_reuse, backend_recycle) reveal whether FD exhaustion is driven by poor backend connection reuse.