$ guides / varnish / varnish-too-many-open-files ▌

Operations Guides

Varnish Too many open files: EMFILE, sess_fail_emfile, and the FD ceiling

sess_fail_emfile is climbing and new client connections are being refused. The Varnish child process has hit its file descriptor ceiling. This is a cliff-edge failure: once the FD limit is exhausted, Varnish cannot accept new connections, cannot open backend connections, and can panic if internal operations that require FDs fail. There is no graceful degradation and no queuing.

The default ulimit -n of 1024 on most Linux distributions is far too low for production Varnish. The Varnish package’s default systemd unit sets LimitNOFILE=131072, but inherited or misconfigured environments often leave the lower default in place.

What this means

EMFILE is the errno for “too many open files.” When a process hits its RLIMIT_NOFILE ceiling, every open(), socket(), or accept() call returns EMFILE.

MAIN.sess_fail_emfile (Varnish 6.1+) is the dedicated counter for session accept failures caused by FD exhaustion. The official counter documentation describes it as “Session accept failures: too many open files” with the advice “Consider raising RLIMIT_NOFILE (see ulimit -n).” On Varnish versions before 6.1, only the aggregate MAIN.sess_fail counter exists, and you must infer FD exhaustion from process-level FD counts.

A nonzero sess_fail_emfile rate means connections are already being refused.

FD exhaustion can also crash the child process. Internal shared-memory operations (VSM/VSL) need FDs, and exhaustion in the wrong path can trigger a panic in vsmw_append_record() (GitHub issue #3663, reported 2021). Varnish has no built-in FD reservation mechanism to protect internal operations from exhaustion.

flowchart TD
    A["Client connections"] --> FD["Process FD pool"]
    B["Backend connections"] --> FD
    C["Storage and log handles"] --> FD
    D["Worker thread FDs"] --> FD
    FD --> E["usage < 0.8: normal"]
    FD --> F["usage > 0.8: sess_fail_emfile"]
    FD --> G["usage = 100%: cliff-edge failure"]

Every FD-consuming resource contributes to the ceiling: accepted client connections, backend connections, storage file handles (especially with file-backed storage), VSM/VSL shared memory segment handles, and worker thread internal FDs.

Common causes

Cause	What it looks like	First thing to check
Default ulimit too low	FD count hits ~1024, `sess_fail_emfile` climbs	`cat /proc/$(pgrep -n varnishd)/limits`
Idle keepalive connections holding FDs	FD count grows slowly, plateaus at limit	`timeout_idle` parameter and `WAITER.*.conns`
Backend connection churn	`backend_conn` rate high, `backend_reuse` low	`backend_reuse / (backend_reuse + backend_conn)` ratio
High `thread_pools` inflating FD usage	FD count far exceeds concurrent connections	`MAIN.threads` and `thread_pools` parameter
Connection leak in VMOD or backend	FD count grows monotonically without cleanup	FD count trend over time vs traffic pattern

Quick checks

All read-only and safe in production.

# sess_fail_emfile rate (Varnish 6.1+) and aggregate sess_fail
varnishstat -1 -f MAIN.sess_fail_emfile -f MAIN.sess_fail

# CHILD process FD count (not the manager)
CHILD_PID=$(pgrep -n varnishd)
ls /proc/$CHILD_PID/fd | wc -l

# Child's FD limit
cat /proc/$CHILD_PID/limits | grep 'Max open files'

# Connection-related counters
varnishstat -1 -f MAIN.sess_conn -f MAIN.backend_conn -f MAIN.backend_reuse -f 'WAITER.*.conns'

# timeout_idle controls how long idle keepalive FDs stay open
varnishadm param.show timeout_idle

# thread_pools value
varnishadm param.show thread_pools

How to diagnose it

Confirm FD exhaustion. Run varnishstat -1 -f MAIN.sess_fail_emfile. A nonzero rate on Varnish 6.1+ confirms FD exhaustion. On pre-6.1 with only sess_fail nonzero, proceed to process-level checks.
Check the right process. Varnish runs two processes: a management process (root-owned, supervises the child) and a child/worker process (handles all cache operations). The child holds the FDs. Use pgrep -n varnishd (newest) to get the child PID. pgrep -o varnishd (oldest) gives the management process, which has very few FDs and will mislead you.
Compute FD utilization. Compare ls /proc/$CHILD_PID/fd | wc -l against the limit in /proc/$CHILD_PID/limits. Below 50% is comfortable. Above 80% is the action threshold. At 100%, connections are already being refused.
Find where FDs are going. If the FD count is high but client and backend connection counts do not account for it, check thread_pools. Each worker thread holds internal FDs, so high thread_pools values inflate FD usage. Maintainers rarely increase thread_pools beyond the default of 2, even at scale.
Check timeout_idle. This controls how long idle keepalive client connections stay open. A high value (for example, 1800 seconds) means each idle client holds an FD for 30 minutes. The default is typically 5 seconds. Long keepalive timeouts are fine with FD headroom but become a liability near the ceiling.
Check backend connection reuse. Low backend_reuse relative to backend_conn means Varnish opens new backend connections instead of reusing them. Each new connection consumes an FD and adds TCP handshake latency. The reuse ratio should be above 0.5 in steady state.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`MAIN.sess_fail_emfile` (V6.1+)	Confirms FD exhaustion specifically	Any nonzero rate
`MAIN.sess_fail`	Aggregate accept failures, includes FD exhaustion and other causes	Any nonzero rate
Process FD count / FD limit	Direct utilization of the FD ceiling	Ratio above 0.8
`MAIN.sess_conn` rate	Incoming connection pressure	Rate exceeding baseline
`MAIN.backend_conn` vs `MAIN.backend_reuse`	Backend connection churn consuming FDs	Reuse ratio below 0.5
`MAIN.backend_fail`	Backend connections failing, may be FD-starved	Any nonzero rate
`MAIN.sess_dropped`	Sessions dropped as consequence of accept failures	Any nonzero rate

Fixes

Raise the OS file descriptor limit

This is the primary fix. For systemd-managed Varnish, create a drop-in override:

systemctl edit varnish

Add or modify:

[Service]
LimitNOFILE=131072

Reload and restart:

systemctl daemon-reload
systemctl restart varnish

Restarting Varnish empties the cache. Schedule during a maintenance window or when cache warmup is acceptable.

For non-systemd deployments, set the ulimit before starting varnishd:

ulimit -n 131072
varnishd ...

For Docker, set the ulimit via docker run:

docker run --ulimit nofile=131072:131072 ...

Or in docker-compose:

services:
  varnish:
    ulimits:
      nofile:
        soft: 131072
        hard: 131072

After raising the limit, verify the child process picked it up:

cat /proc/$(pgrep -n varnishd)/limits | grep 'Max open files'

Reduce idle keepalive FD consumption

If timeout_idle is high, idle client connections hold FDs for a long time. Lowering it reclaims FDs faster:

varnishadm param.set timeout_idle 10

Live change, no restart required. The tradeoff: clients with keepalive connections re-establish more frequently, adding a small latency cost for returning visitors.

Improve backend connection reuse

If backend_reuse is low relative to backend_conn, Varnish is churning backend connections. Check:

The backend_idle_timeout parameter (too short means connections closed before reuse)
Backend support for keepalive (HTTP/1.1 with Connection: keep-alive)
Whether the backend is closing connections prematurely

Reduce thread_pools

If thread_pools has been increased well beyond the default of 2 with high thread_pool_min, reducing it back can cut FD consumption significantly.

varnishadm param.set thread_pools 2

This is disruptive. Changing the number of pools requires Varnish to destroy and recreate thread pools, causing a temporary stall.

Investigate connection leaks

If FD count grows monotonically regardless of traffic, suspect a connection leak in a custom VMOD or a backend that does not properly close connections, leaving Varnish holding stale FDs.

Monitor FD count over time:

while true; do
  echo "$(date +%s) $(ls /proc/$(pgrep -n varnishd)/fd | wc -l)"
  sleep 60
done

If the count grows without bound, inspect individual FDs with ls -la /proc/$CHILD_PID/fd/ to see what types of handles are accumulating.

Prevention

Set LimitNOFILE to at least 131072 in the systemd unit or equivalent. The default 1024 is a guaranteed cliff for production Varnish.
Alert on FD utilization at 0.8. Do not wait for sess_fail_emfile to increment. By then connections are already being refused.
Monitor sess_fail_emfile directly on Varnish 6.1+. Any nonzero rate is a confirmed FD exhaustion event.
Keep thread_pools at 2 unless you have a measured reason to increase it.
Audit timeout_idle when traffic patterns change. Long keepalive timeouts are fine with headroom but become a liability near the ceiling.
Monitor backend connection reuse ratio. Low reuse burns FDs on new connections and adds latency.

How Netdata helps

Per-second FD monitoring. Netdata collects process-level file descriptor counts at 1-second resolution, catching FD growth before the cliff.
sess_fail_emfile and sess_fail tracking. Varnish counters are collected per second, letting you distinguish gradual FD pressure from sudden spikes.
Correlation with connection metrics. Cross-referencing MAIN.sess_conn, MAIN.backend_conn, MAIN.backend_reuse, and MAIN.threads against process FD count pinpoints which consumer is driving the ceiling.
Alerts on FD utilization. Configurable alerts on process FD usage ratio let you page before sess_fail_emfile starts incrementing.