Varnish threads_limited climbing: hitting thread_pool_max

MAIN.threads_limited increments each time Varnish wanted to create a new worker thread but thread_pool_max prevented it. Movement during a traffic burst is expected. A sustained nonzero rate during steady-state operation means the configured thread pool ceiling is a binding constraint on concurrency, and if the bounded queue behind the pool fills, Varnish will drop sessions.

The default thread_pool_max is 5000 threads per pool. With the default thread_pools value of 2, the absolute ceiling is 10,000 worker threads. For most workloads that ceiling is high enough that the counter stays at zero. The problem arises when backends are slow: each thread is held longer per request, effective concurrency drops, and the pool fills at a request rate that would be trivial if backends responded in single-digit milliseconds.

What this means

When a client connection arrives, the accept thread hands it to an idle worker. If no worker is available and the pool has not reached thread_pool_max, Varnish creates a new thread. If the pool is at the maximum, the request enters a bounded queue tracked by MAIN.thread_queue_len. MAIN.threads_limited increments at the moment Varnish decides it cannot create a thread because the pool is full.

The failure cascade:

flowchart TD
    A["Client request"] --> B{"Idle worker?"}
    B -- yes --> C["Handled"]
    B -- no --> D{"Below thread_pool_max?"}
    D -- yes --> E["Thread created"]
    D -- no --> F["MAIN.threads_limited++"]
    F --> G["Queue: thread_queue_len grows"]
    G --> H{"Below thread_queue_limit?"}
    H -- yes --> I["Wait for worker"]
    H -- no --> J["sess_dropped / req_dropped"]
    I --> C

threads_limited by itself is not a user-facing failure. No client sees an error when this counter ticks up. The damage happens downstream: if requests queue faster than workers free up, thread_queue_len grows toward thread_queue_limit (default 20 per pool). When the queue fills, sessions are dropped for HTTP/1.1 (MAIN.sess_dropped) and requests are dropped for HTTP/2 (MAIN.req_dropped).

Key distinction during diagnosis:

• MAIN.threads_limited incrementing: Varnish’s configuration ceiling (thread_pool_max) prevented thread creation. Fix: raise the ceiling or reduce demand.
• MAIN.threads_failed incrementing: The OS refused pthread_create(). This is a system-level resource limit (nproc, insufficient memory for thread stacks, systemd TasksMax, or cgroup pids.max). Fix: raise OS limits.

Both can be nonzero at the same time. Check both.

Common causes

Quick checks

All read-only and safe to run at any time.

# Current thread state: count, pool ceiling, queue depth, failure counters
varnishstat -1 -f MAIN.threads -f MAIN.pools -f MAIN.thread_queue_len -f MAIN.threads_limited -f MAIN.threads_failed

# Configured ceiling per pool
varnishadm param.show thread_pool_max
varnishadm param.show thread_pools

# Whether sessions or requests are being dropped
varnishstat -1 -f MAIN.sess_dropped -f MAIN.req_dropped

# Backend health
varnishadm backend.list -p

# threads_limited rate: two readings 10 seconds apart
varnishstat -1 -f MAIN.threads_limited; sleep 10; varnishstat -1 -f MAIN.threads_limited

# OS thread limits for the child process (newest varnishd is the child)
CHILD_PID=$(pgrep -n varnishd)
cat /proc/$CHILD_PID/status | grep Threads
cat /proc/$CHILD_PID/limits | grep -i processes

# systemd TasksMax if running under systemd
systemctl show varnish.service | grep TasksMax

How to diagnose it

Step 1: Confirm the pool is actually at the ceiling.

Compare MAIN.threads against thread_pool_max x MAIN.pools. If threads is at or near the ceiling, the pool is genuinely full. If threads is well below the ceiling but threads_limited is still incrementing, you may be hitting the race condition on older Varnish (see step 5) or threads_failed is the real problem.

Step 2: Check whether drops are happening.

If sess_dropped or req_dropped is incrementing, users are being rejected. Treat this as urgent.

Step 3: Distinguish threads_limited from threads_failed.

Run varnishstat -1 -f MAIN.threads_limited -f MAIN.threads_failed. If threads_failed is nonzero, the OS is refusing thread creation. This is a different problem with a different fix (OS limits, not Varnish config). If threads_failed is zero and threads_limited is climbing, the issue is the Varnish config ceiling.

Step 4: Check backend response time.

Backend latency is the most common root cause. Slow backends hold each worker thread longer, reducing effective concurrency. Use varnishlog to inspect fetch timing:

# Backend fetch timing (Bereq to Beresp delta)
varnishlog -g request -i Timestamp -q 'BerespStatus gt 0'

If backend TTFB is in the hundreds of milliseconds or higher during the threads_limited event, slow backends are consuming your thread budget. Raising thread_pool_max buys time but does not fix the root cause.

Step 5: Check Varnish version for the known race condition.

On Varnish <= 6.5.1, a race condition (GitHub issue #3531) could cause threads_limited to increment even when thread_pool_max had not been reached. The maintainer noted: “There is a race in current code which can lead to the counter being increased even if everything is ok.” The fix landed in commit 2bd5d2a (February 2021) and shipped in Varnish 6.6 and later.

If you are on an older version and threads_limited is incrementing while threads is well below thread_pool_max x pools, this is a known false positive.

Step 6: Check for cold-start transients.

After a child process restart, the thread pool ramps up from thread_pool_min (default 100 per pool). If traffic hits during warmup, threads_limited may bump briefly. This is normal and self-resolves. Check MAIN.uptime to rule this out.

Metrics and signals to monitor

Fixes

Raise thread_pool_max

The direct fix when the pool ceiling is genuinely too low for your traffic volume.

# Live change, takes effect immediately, no restart required
varnishadm param.set thread_pool_max 8000

This does not persist across restarts. Add the parameter to your startup configuration:

• In the systemd unit file or command line: -p thread_pool_max=8000
• Verify after restart: varnishadm param.show thread_pool_max

Memory implications. Each thread consumes a stack of thread_pool_stack (default 80k on 64-bit systems in Varnish 7.0 and later).

With the defaults of thread_pools=2 and thread_pool_max=5000, thread stacks alone account for 2 x 5000 x 80k = 800 MB. Raising thread_pool_max to 8000 per pool increases stack memory to 2 x 8000 x 80k = 1.28 GB. Verify the system has headroom before raising the limit. Do not starve the cache store (-s malloc,SIZE) or transient storage.

Increase thread_pools

Controls how many independent thread pools Varnish maintains. Increasing it can reduce per-pool lock contention at very high concurrency.

# Live change
varnishadm param.set thread_pools 4

Can be increased on the fly. Decreasing it requires a restart. Each pool maintains its own thread_pool_min through thread_pool_max range, so adding pools also adds thread_pool_min threads at minimum. The total thread ceiling changes accordingly: with thread_pool_max=5000 and thread_pools=4, the ceiling is 20,000 threads.

Reduce demand: fix slow backends

If backend response time is the root cause, raising thread_pool_max is a stopgap. Each thread held for 500ms on a slow backend fetch is unavailable for the next 500ms. Reducing backend TTFB from 500ms to 50ms lets the same pool handle 10x the request rate.

Check for:

• Backend database slowdowns
• Backend connection pool exhaustion (MAIN.backend_busy)
• Missing cache grace or stale configuration (without grace, every cache miss hits the backend synchronously)
• VCL performing blocking operations (DNS lookups, external calls in VMODs)

Fix OS-level thread limits (if threads_failed is nonzero)

The OS refused pthread_create(). The fix is at the OS level.

Common causes on modern Linux:

• systemd TasksMax: The Varnish unit may have TasksMax set too low. Set it in a unit override:

  [Service]
  TasksMax=infinity
  

  This removes the per-unit task safety net. Only do this if you have verified that system-wide PID and memory limits are sufficient for the thread counts you expect.
• ulimit -u (RLIMIT_NPROC): Check /proc/<child_pid>/limits for the actual effective limit on the Varnish child process.
• cgroup pids.max: If Varnish runs in a cgroup with a PID limit, thread creation fails when the limit is reached.
• Memory for thread stacks: Under memory pressure, pthread_create() can fail because there is no room for the new thread’s stack.

Prevention

• Monitor the rate, not the counter. MAIN.threads_limited is cumulative and never decreases. Track the delta per second. A counter at 5000 since the last restart that is not growing is not a problem.
• Watch thread_queue_len as the leading indicator. thread_queue_len > 0 means the pool is saturated. If it grows toward thread_queue_limit (default 20), drops are imminent.
• Correlate with backend latency. The most common root cause of thread pool pressure is not an undersized pool but slow backends. Track backend TTFB alongside thread metrics.
• Capacity-plan for peak concurrency. Estimate peak concurrent requests as (request_rate x average_response_time). If that number approaches thread_pool_max x pools, you need a bigger pool, faster backends, or better caching.
• Account for thread stack memory. Before raising thread_pool_max, verify system memory can absorb the additional stack allocation.
• Keep Varnish current. The threads_limited race condition in <= 6.5.1 produced false positives. Running 6.6+ or 7.x eliminates this source of noise.

How Netdata helps

Netdata collects Varnish counters at per-second resolution, which matters because thread_queue_len is a point-in-time gauge that can spike and resolve between coarser polling intervals.

• threads_limited rate: Netdata computes the per-second rate automatically, so you see the climb as it happens rather than discovering a stale cumulative counter.
• threads and thread_queue_len correlation: Thread count pinned at the ceiling alongside a growing queue confirms pool saturation in a single view.
• threads_failed separation: Surfaced independently, so you can distinguish a Varnish config ceiling from an OS-level resource limit immediately.
• sess_dropped and req_dropped: Confirm when thread pool pressure has escalated to user-visible failures. Correlating their onset with threads_limited growth identifies exactly when the queue filled.
• Backend latency correlation: Backend latency spikes alongside thread pool growth identify slow backends as the root cause.

Related guides

• How Varnish actually works in production: a mental model for operators
• Varnish monitoring checklist: the signals every production cache needs
• Varnish monitoring maturity model: from survival to expert
• Varnish thread pool exhaustion: workers all busy, queue full, sessions dropped