Varnish is dropping client connections and CPU looks idle. The process is running, the management CLI responds, and backends are technically up, but users see connection resets or stream resets. This is thread pool exhaustion: all worker threads are busy, the request queue is full, and Varnish has started refusing traffic.

Varnish’s concurrency model is thread-per-request with a bounded pool. Each pool holds between thread_pool_min and thread_pool_max threads, and Varnish runs thread_pools pools (default 2). When a request arrives and no idle worker is available, it enters a bounded queue (thread_queue_limit, default 20 per pool). When the queue is full, the session or stream is dropped.

The most common root cause is slow backends. Each worker thread blocked on a backend fetch is one fewer thread available for new requests. As backend response times increase, threads accumulate at thread_pool_max, the queue fills, and drops begin. CPU stays low because the bottleneck is thread availability, not processing power.

Confirming thread pool exhaustion

Thread pool exhaustion means Varnish has reached its configured concurrency ceiling. The system transitions from healthy to dropping connections in seconds because the degradation curve is cliff-edge, not gradual.

Three conditions confirm the diagnosis:

  • MAIN.threads equals thread_pool_max multiplied by thread_pools (pool saturated).
  • MAIN.thread_queue_len is greater than zero (requests waiting).
  • MAIN.threads_limited is incrementing (Varnish tried to create more threads but hit the configured maximum).

When drops start, MAIN.sess_dropped (HTTP/1 connections) or MAIN.req_dropped (HTTP/2 streams) will be nonzero. Both must be checked because HTTP/2 multiplexes streams over connections and the failure mode differs.

flowchart TD
    A[Backend response time increases] --> B[Worker threads held longer per request]
    B --> C[Threads accumulate toward thread_pool_max x pools]
    C --> D[No idle workers for incoming requests]
    D --> E[Requests enter queue: thread_queue_len > 0]
    E --> F{Queue at thread_queue_limit?}
    F -->|Not yet| E
    F -->|Queue full| G["sess_dropped / req_dropped increment"]
    G --> H[Client receives connection or stream reset]

The bottleneck is thread availability, not CPU or memory. If you see drops with idle CPU, thread pool exhaustion is the primary suspect.

Common causes

CauseWhat it looks likeFirst thing to check
Slow backend responsesBackend TTFB elevated, backend_fail or backend_busy nonzero, threads at maxvarnishadm backend.list -p and backend fetch timing
Undersized thread_pool_maxthreads_limited incrementing at moderate traffic, drops during peaksCurrent thread_pool_max vs peak concurrency
OS-level thread creation failurethreads_failed incrementing, no corresponding threads_limited spikedmesg for cgroup or pids errors, ulimit -u
Traffic spike exceeding thread ramp-upDrops during sudden burst, thread_queue_len spikes then recoversthread_pool_add_delay setting, traffic pattern
HTTP/2 protocol abusereq_dropped spiking, sess_dropped stable, possibly sc_rapid_reset nonzeroHTTP/2-specific counters and access logs

Quick checks

These commands are safe and read-only. Run them in order.

# Confirm thread pool saturation
varnishstat -1 -f MAIN.threads -f MAIN.thread_queue_len -f MAIN.threads_limited -f MAIN.threads_failed -f MAIN.pools

# Check for active drops
varnishstat -1 -f MAIN.sess_dropped -f MAIN.req_dropped

# Check backend health and probe details
varnishadm backend.list -p

# Check backend connection failures
varnishstat -1 -f MAIN.backend_fail -f MAIN.backend_unhealthy -f MAIN.backend_busy

# Check thread creation failures (OS-level)
varnishstat -1 -f MAIN.threads_failed

# Check session close reasons for overload signals
varnishstat -1 -f 'MAIN.sc_*'

# Check fetch failures related to thread starvation
varnishstat -1 -f MAIN.fetch_failed -f MAIN.fetch_no_thread -f MAIN.bgfetch_no_thread

# Check VCL state for recent reloads
varnishadm vcl.list

# Check process file descriptor usage (child PID, not management)
CHILD_PID=$(pgrep -n varnishd)
ls /proc/$CHILD_PID/fd | wc -l
cat /proc/$CHILD_PID/limits | grep 'Max open files'

# Check system logs for OOM or cgroup limits
dmesg | grep -iE 'oom|pids|cgroup' | tail -20

How to diagnose it

  1. Confirm the pool is at capacity. Run varnishstat -1 -f MAIN.threads -f MAIN.pools and compare MAIN.threads to thread_pool_max multiplied by MAIN.pools. If threads equal the ceiling, the pool is saturated.

  2. Distinguish threads_limited from threads_failed. If threads_limited is incrementing, Varnish hit its configured maximum. If threads_failed is incrementing, the operating system refused to create the thread. These require different fixes: one is a Varnish parameter, the other is a system limit.

  3. Check whether drops are HTTP/1 or HTTP/2. Run varnishstat -1 -f MAIN.sess_dropped -f MAIN.req_dropped. If only req_dropped is nonzero, the problem is HTTP/2 stream drops, not HTTP/1 connection drops. Teams monitoring only sess_dropped miss HTTP/2 traffic loss entirely.

  4. Investigate backend response time. Varnish does not expose backend latency as a varnishstat counter. Use varnishlog to inspect fetch timing. The Timestamp tags in a backend transaction show Bereq (request sent) and Beresp (first byte received). The delta is backend TTFB.

# Backend fetch timing breakdown
varnishlog -g request -i Timestamp -i FetchError -q 'BerespStatus gt 0'

  1. Check for VCL performing blocking operations. DNS lookups in VCL, external VMOD calls, or complex regex evaluation can hold threads even when backends are fast. Look at the Process time in varnishlog -i Timestamp -g request to see VCL processing overhead separate from backend fetch time.

  2. Check for OS-level thread creation limits. If threads_failed is nonzero, the OS refused pthread_create(). On systemd-based systems, check TasksMax in the service unit. The default cgroup pids.max limit can silently cap thread creation. Check dmesg for “fork rejected by pids controller.”

# Get the child process PID
CHILD_PID=$(pgrep -n varnishd)

# Check systemd task limit
systemctl show varnish.service -p TasksMax

# Check ulimits on the child process
cat /proc/$CHILD_PID/limits | grep -E 'processes|open files'
  1. Verify the thread_pool_add_delay setting. In modern Varnish the default is 0ms, but older versions defaulted to 2ms. Under sudden load, a conservative thread_pool_add_delay means threads ramp up too slowly, causing temporary queue saturation and drops even when the maximum is adequate.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
MAIN.threads vs thread_pool_max x poolsPool saturation is the direct leading indicatorThreads at ceiling, sustained queue follows
MAIN.thread_queue_lenQueue depth is the last buffer before dropsAny sustained nonzero value
MAIN.sess_queuedCumulative count of requests that entered the queue, smoother than the gaugeRate greater than zero indicates sustained saturation
MAIN.threads_limitedConfirms Varnish wanted more threads but hit config limitRate greater than zero sustained
MAIN.threads_failedOS refused thread creation, different remediation neededAny nonzero value
MAIN.sess_droppedHTTP/1 connections being droppedAny sustained nonzero rate with uptime > 300
MAIN.req_droppedHTTP/2 streams being droppedAny sustained nonzero rate
MAIN.fetch_no_threadThread starvation affecting backend fetchesAny nonzero rate
MAIN.backend_fail / MAIN.backend_busyBackend connectivity issues holding threadsSustained nonzero rate
MAIN.sc_overloadSessions closed due to Varnish overloadAny nonzero rate

Fixes

Raise thread_pool_max (stop-gap)

The immediate mitigation when threads_limited is incrementing:

# Increase thread pool maximum at runtime
varnishadm param.set thread_pool_max 8000

This takes effect for new threads. Existing threads are unaffected. This buys time but does not fix the root cause. If backends are slow, a larger pool simply delays the cliff without preventing it.

Memory cost: each thread consumes stack memory (thread_pool_stack, default 80kB in Varnish 7.0 and later, previously 48kB). A pool of 10,000 threads at 80kB per stack consumes approximately 800MB just for stacks. Account for this in capacity planning.

To make the change permanent, add the parameter to your Varnish startup configuration via the -p flag in the systemd unit or startup script.

Fix slow backends (root cause)

If backend TTFB is elevated, fixing the backend is the real fix. Identify which backend is slow:

# Check per-backend health and probe details
varnishadm backend.list -p
# Check per-backend happy counts
varnishstat -1 -f 'VBE.*.happy'

If one backend is the culprit, mark it sick to stop sending traffic:

# WARNING: disruptive - removes the backend from rotation immediately.
# Verify the backend name first with: varnishadm backend.list
varnishadm backend.set_health <backend_name> sick

Review backend timeout parameters. first_byte_timeout and between_bytes_timeout control how long Varnish waits before giving up on a backend. If these are set too high (for example, 600 seconds based on outdated advice), threads can be held for minutes on a single slow request. Lower them to fail fast and release the thread.

Check OS-level limits

If threads_failed is nonzero, the OS is blocking thread creation. Check systemd TasksMax and process limits:

CHILD_PID=$(pgrep -n varnishd)

# Check systemd TasksMax
systemctl show varnish.service -p TasksMax
# Check process limits
cat /proc/$CHILD_PID/limits | grep processes

On systemd-based distributions, the default TasksMax for the varnish service unit can cap thread creation at around 4900 threads. If threads_failed increments and dmesg shows “fork rejected by pids controller,” set TasksMax=infinity in the service unit override.

Also check ulimit -u (RLIMIT_NPROC) and available memory for thread stacks. Under vm.overcommit_memory=2, Linux may block thread creation even with apparent free memory. Check /proc/meminfo for CommitLimit versus Committed_AS.

Tune thread_pool_add_delay

If drops occur during traffic spikes but the pool recovers, thread_pool_add_delay may be too conservative:

# Check current setting
varnishadm param.show thread_pool_add_delay

In modern Varnish the default is 0ms. If your deployment was configured with an older default of 2ms, threads ramp up slowly under sudden load. Set it to 0 to allow immediate thread creation.

Consider thread_pool_reserve (Varnish 6.5+)

The thread_pool_reserve parameter reserves threads for high-priority tasks like backend fetches. The default is 0, which auto-tunes to 5% of thread_pool_min (minimum effective value is 5). When the pool is near exhaustion, reserved threads can still perform critical work like backend fetches, preventing a deadlock where all threads are busy on client requests and none can fetch from backends.

Consider backend connection queuing (Varnish 7.6+)

Varnish 7.6 introduced backend_wait_timeout and backend_wait_limit, which allow tasks to queue when a backend’s max_connections is reached instead of immediately failing. The counters MAIN.backend_wait and MAIN.backend_wait_fail track this behavior. If backend connection limits are contributing to thread pressure, this feature can reduce the cascade.

Handle HTTP/2 abuse

If req_dropped is spiking without a corresponding sess_dropped increase, investigate HTTP/2-specific causes. Check MAIN.sc_rapid_reset for signs of the Rapid Reset attack pattern (CVE-2023-44487). Ensure you are running a patched Varnish version if HTTP/2 abuse is suspected.

Prevention

  • Monitor thread pool saturation continuously. Track MAIN.threads as a ratio of thread_pool_max multiplied by thread_pools. Alert when sustained above 80%.
  • Monitor thread_queue_len at sub-second intervals. The gauge oscillates rapidly and one-second sampling can miss sub-second spikes that cause drops.
  • Alert on combined sess_dropped plus req_dropped rate. Use a sustained window of more than 120 seconds with uptime > 300 to exclude cold start, and confirm with thread_queue_len greater than zero.
  • Track backend TTFB independently. Varnish does not expose latency as a counter. Parse varnishncsa output (%D for request duration, %{Varnish:time_firstbyte}x for TTFB) to build latency monitoring.
  • Audit thread_pool_add_delay. Ensure it is 0 on modern Varnish. Older deployments may carry legacy values that slow thread ramp-up.
  • Set TasksMax=infinity in the systemd unit. Prevents cgroup pids limits from silently capping thread creation.
  • Review first_byte_timeout and between_bytes_timeout. Ensure they are not set excessively high. Threads held for minutes on a slow backend will exhaust any pool.
  • Distinguish threads_limited from threads_failed in alerts. They require different fixes: one is a Varnish parameter, the other is a system limit.
  • On Varnish before 6.5.1, be aware of the threads_limited race condition. A known bug caused threads_limited to increment without actually reaching thread_pool_max under spiky traffic. If you see threads_limited incrementing but MAIN.threads is well below the ceiling on an older Varnish, upgrade.
  • Ensure grace and stale configuration is in place. Without grace and stale-if-error, any backend hiccup immediately cascades into cache misses that hold threads on slow fetches.

How Netdata helps

Netdata’s Varnish collector surfaces thread pool metrics at per-second resolution, which matters because thread_queue_len is a point-in-time gauge that can spike and recover within a single second.

  • Thread pool saturation correlation. Netdata charts MAIN.threads, MAIN.thread_queue_len, MAIN.threads_limited, and MAIN.threads_failed alongside each other, making it immediately visible whether saturation is Varnish-configured (threads_limited) or OS-limited (threads_failed).
  • Drop rate visibility. MAIN.sess_dropped and MAIN.req_dropped are charted separately, so HTTP/1 and HTTP/2 traffic loss are distinguishable without manual queries.
  • Backend health context. Per-backend VBE.*.happy counters and MAIN.backend_fail or MAIN.backend_busy appear on the same dashboard, letting you correlate thread pool exhaustion with backend degradation in seconds rather than switching between tools.
  • Anomaly detection on thread counts. Netdata’s ML-based anomaly detection flags unusual thread pool behavior before drops begin, giving lead time during gradual backend slowdown.
  • Cold start filtering. Correlating MAIN.uptime with thread metrics prevents false alerts during the warmup period after a child restart.