Varnish thread_queue_len above zero: requests waiting for a worker

MAIN.thread_queue_len is the point-in-time count of client sessions sitting in Varnish’s bounded worker queue, waiting for an idle thread. In normal operation it is zero. Any sustained nonzero value means every worker thread across all pools is busy and incoming requests are piling up in the last buffer before Varnish starts dropping them.

The thread pool has a cliff-edge failure curve: performance is fine until the pool is saturated, then requests queue, then the queue fills, then sessions are dropped. The distance between “queue length is 1” and “clients are getting connection resets” can be seconds if the queue limit is small (the default thread_queue_limit is 20 per pool).

varnishstat samples this counter at 1-second intervals by default. The underlying value oscillates rapidly under pressure; monitoring tools with longer polling intervals (15s, 30s, 60s) can miss spikes that still cause drops. For alerting, track the rate of MAIN.sess_queued (the cumulative counter of sessions that entered the queue) rather than relying on the gauge alone. At approximately 50% of thread_queue_limit sustained, you are within the buffer before drops begin.

What this means

Varnish uses a thread-per-request model with a bounded worker pool. An accept thread receives connections and hands each one to an idle worker. If no worker is available, the request enters the session queue. If the queue is full (thread_queue_limit reached), the session or request is dropped: MAIN.sess_dropped for HTTP/1 connections, MAIN.req_dropped for HTTP/2 streams.

When thread_queue_len is nonzero, the pool herder (the thread that manages pool sizing) is trying to create new workers up to thread_pool_max per pool but cannot keep up with demand. There are two sub-cases:

1. Pool not yet at max. Varnish is creating threads as fast as thread_pool_add_delay allows, but demand outpaces creation. Transient if traffic stabilizes. Persistent if traffic is genuinely above capacity.
2. Pool at max. All threads are busy, no more can be created, and the queue is absorbing overflow. MAIN.threads_limited will be incrementing. This is the direct precursor to drops.

The key diagnostic question is: why are threads busy long enough for the queue to fill? The answer is almost always slow backends holding threads hostage, not CPU exhaustion. CPU may be completely idle while Varnish refuses connections.

flowchart LR
    A[Accept thread] -->|idle worker| B[Worker processes request]
    A -->|no worker| C[Session queue]
    C -->|worker frees up| B
    C -->|queue full| D[Drops: sess_dropped, req_dropped]
    B -->|slow backend| E[Thread held on fetch]
    C -->|herder creates thread| B

Common causes

Quick checks

# Current thread pool state
varnishstat -1 -f MAIN.threads -f MAIN.thread_queue_len -f MAIN.threads_limited -f MAIN.threads_failed -f MAIN.pools

# Cumulative queue entries (rate this, do not alert on the gauge alone)
varnishstat -1 -f MAIN.sess_queued

# Session and request drops (the failure that follows queue saturation)
varnishstat -1 -f MAIN.sess_dropped -f MAIN.req_dropped

# Backend health
varnishadm backend.list -p

# Current thread pool parameters
varnishadm param.show thread_pool_max
varnishadm param.show thread_pool_min
varnishadm param.show thread_queue_limit
varnishadm param.show thread_pool_add_delay

All read-only and safe to run at any time.

How to diagnose it

1. Confirm pool saturation. Check whether MAIN.threads is at or near thread_pool_max x MAIN.pools. If yes, the pool is at capacity. If no, the pool herder may be ramping threads too slowly, or the OS is refusing creation.

2. Check for thread creation failures. If MAIN.threads_failed is incrementing, the OS is blocking pthread_create(). This is a system-level problem, not a Varnish tuning problem. Check ulimit -u (max user processes), available memory for thread stacks, and vm.max_map_count (rule of thumb: Varnish needs roughly 2 memory maps per thread). On systemd-managed hosts, also check the cgroup pids.max limit, which can block thread creation even when ulimits are generous.

3. Check for thread creation limits. If MAIN.threads_limited is incrementing but threads_failed is zero, Varnish itself is the limit: thread_pool_max is too low for current traffic. Note: on some Varnish versions (see GitHub issue #3531), a race condition in the pool herder could cause threads_limited to increment spuriously without actually reaching thread_pool_max. If you see threads_limited incrementing while threads is well below thread_pool_max x pools, consider this a known issue and verify against your Varnish version’s changelog.

4. Identify why threads are busy. If the pool is at max and no creation failures exist, threads are occupied for too long. The dominant cause is slow backend responses. Check backend health and fetch time:

# Backend health with probe details
varnishadm backend.list -p

# Per-request timing breakdown (look at Fetch vs Process deltas)
varnishlog -i Timestamp -g request | head -100

5. Check backend response time. Use varnishncsa to inspect per-request duration:

# Request duration in microseconds, with URL and status
varnishncsa -F '%D %U %s' -q 'ReqMethod ne "PURGE"'

Cache hits should be sub-millisecond to low single-digit milliseconds. If cache misses show multi-second durations, backends are slow and threads are being held during each fetch.

6. Check for VCL blocking. If backends are healthy and fast but threads still exhaust, review VCL for operations that block the worker thread: DNS resolution via VMODs, external calls, or complex regex evaluation.

Metrics and signals to monitor

Fixes

Slow backends (most common root cause)

Threads block waiting for backend responses, the pool fills, the queue fills, drops begin. CPU is idle the entire time.

• Immediate: mark a known-slow backend sick to stop sending it traffic. Warning: this is disruptive and takes effect instantly, halting all traffic to that backend: varnishadm backend.set_health <name> sick. Only use this if the alternative is worse (total pool exhaustion affecting all backends).
• Short-term: increase thread_pool_max to buy more concurrency headroom: varnishadm param.set thread_pool_max 8000. This applies live. It gives more threads to absorb slow backend responses but increases memory consumption (thread stacks) and does not fix the backend.
• Root cause: fix the backend. Investigate database queries, application GC pauses, connection pool limits, and network latency between Varnish and the origin.

Undersized thread pool

If threads_limited is incrementing and threads is pinned at thread_pool_max x pools, the pool is too small for current traffic.

• Increase thread_pool_max via varnishadm param.set thread_pool_max N. The default is 5000 per pool. With 2 pools, that is 10,000 threads maximum. Each thread consumes stack memory (default thread_pool_stack is 80 KB on 64-bit systems in current Varnish versions, 64 KB on 32-bit), so 10,000 threads consume roughly 800 MB of stack alone at the 64-bit default.
• Verify the change took effect: varnishadm param.show thread_pool_max.
• Make the change persistent in your Varnish startup parameters or systemd unit. Parameters set via param.set do not survive a restart.

OS refusing thread creation

If threads_failed is incrementing, the problem is not Varnish configuration. The OS or container runtime is denying pthread_create().

• Check ulimit -u for the Varnish user.
• Check vm.max_map_count: sysctl vm.max_map_count. Varnish needs roughly 2 maps per thread. With 10,000 threads, you need at least 20,000 maps.
• On systemd hosts: check the unit’s TasksMax directive and the cgroup pids.max. Run systemctl show varnish -p TasksMax to see the limit.
• Check available memory. Thread stacks require committed memory. Under memory pressure, thread creation fails.

Thread ramp too slow

If thread_queue_len spikes during traffic bursts but settles once threads catch up, thread_pool_add_delay may be too conservative. This parameter controls the delay between thread creation attempts under pressure.

• Check current value: varnishadm param.show thread_pool_add_delay.
• Reduce it (e.g., to 0) to allow faster thread ramp: varnishadm param.set thread_pool_add_delay 0. Warning: this applies live and allows rapid thread creation, which can spike memory and CPU if the pool grows quickly.
• Consider raising thread_pool_min so more threads are pre-allocated and ready before traffic spikes. The default is 100 per pool. Raising it to 200-500 keeps more idle threads warm at the cost of baseline memory.

Queue limit too small

The thread_queue_limit parameter (default 20 per pool) controls how many requests queue before drops begin. Increasing it gives more buffer but also means requests wait longer in the queue, adding latency.

• Increasing thread_queue_limit does not fix saturation. It only delays the onset of drops.
• If you increase it, also monitor request latency for queued requests. A longer queue means higher latency even if requests are eventually served.

Prevention

• Monitor sess_queued rate, not just the gauge. The gauge oscillates too rapidly for reliable alerting at typical sampling intervals. Alert on the cumulative rate instead.
• Track thread utilization ratio. threads / (thread_pool_max x pools) above 0.8 during peak means you are approaching the cliff. Capacity-plan before you reach it.
• Monitor backend TTFB trends. Backend slowdown is the leading indicator of thread pool exhaustion. If backend response times are trending up, threads will follow.
• Set thread_pool_min above your idle baseline. Pre-allocated threads eliminate the ramp-up delay during traffic spikes.
• Keep thread_pool_add_delay low. A conservative delay causes avoidable queue spikes during bursts.
• Exclude cold-start from drop alerts. After a child restart, the thread pool ramps from thread_pool_min. If traffic is high during warmup and thread_pool_add_delay is conservative, brief session drops are possible. Gate alerts on MAIN.uptime > 300 to suppress the warmup window.

Monitoring with Netdata

Netdata collects Varnish counters at 1-second intervals, which is the sampling frequency this problem demands. What matters for thread queue diagnosis:

• Per-second MAIN.thread_queue_len and MAIN.sess_queued capture queue spikes that 15-60 second polling intervals miss entirely.
• Thread pool metrics (MAIN.threads, MAIN.threads_limited, MAIN.threads_failed) appear on the same timeline as backend health and backend request rate, so you can distinguish slow-backend saturation from pool-sizing limits from OS-level thread creation failures without switching tools.
• Drop counters (MAIN.sess_dropped, MAIN.req_dropped) have sustained-duration alert conditions (>120 seconds) with a traffic-floor guard (client_req > 0) to suppress false positives on idle or cold-starting nodes.

Related guides

• How Varnish actually works in production: a mental model for operators
• Varnish monitoring checklist: the signals every production cache needs
• Varnish monitoring maturity model: from survival to expert
• Varnish thread pool exhaustion: workers all busy, queue full, sessions dropped