$ guides / varnish / varnish-backend-ttfb-high ▌

Operations Guides

Varnish backend TTFB high: the leading indicator of thread pool death

Backend time-to-first-byte (TTFB) is the time from Varnish sending a backend request to receiving the first response header byte. Varnish uses a thread-per-request model with a bounded pool. Every backend fetch holds a worker thread for the entire fetch duration: TCP connect, wait for first byte, transfer body, process headers. When TTFB rises, each fetch holds a thread longer, and the pool fills faster.

The cascade is deterministic. Backend TTFB climbs, threads accumulate at the pool ceiling, the session queue fills, and sessions start dropping. CPU and memory may look healthy while Varnish refuses connections. TTFB degrades before any availability counter moves.

TTFB is Timestamp:Beresp minus Timestamp:Bereq in a backend (BeReq) transaction. It does not include body transfer time. A fast TTFB with slow body delivery still holds the thread, but TTFB is the signal that warns the backend is in trouble before body timeouts or fetch failures appear.

flowchart TD
    A[Backend TTFB increases] --> B[Each fetch holds a worker thread longer]
    B --> C[Thread pool reaches thread_pool_max]
    C --> D[threads_limited increments]
    C --> E[thread_queue_len rises]
    E --> F[Queue hits thread_queue_limit]
    F --> G[sess_dropped / req_dropped]
    D --> G

What this means

Each pool has thread_pool_max threads (default 5000 per pool), with thread_pools pools (default 2). When a client request results in a cache miss, a worker thread performs a backend fetch and is occupied for the entire duration. If the backend takes 5 seconds to return the first byte, that thread is unavailable for 5 seconds.

The steady-state thread demand is approximately miss rate times fetch duration. With 1000 concurrent cache misses and a TTFB of 3 seconds, you need 1000 threads held for 3 seconds each. With 2 pools of 5000 threads (10,000 total), there is headroom. If TTFB climbs to 8 seconds, those same 1000 misses hold threads for 8 seconds. New misses arriving during that window pile up, and the pool fills.

At that point, MAIN.threads reaches thread_pool_max * thread_pools. MAIN.threads_limited increments: Varnish wanted to create a thread and was refused. MAIN.thread_queue_len rises above zero. When the queue hits thread_queue_limit (default 20 per pool), sessions are dropped. The client gets nothing.

Common causes

Cause	What it looks like	First thing to check
Backend application slowdown	TTFB rises across all backends; health probes may still pass	Backend-side metrics: DB query time, CPU, GC pauses
Database bottleneck	TTFB spikes on specific URL patterns hitting slow queries	Backend application logs for slow queries
Network latency increase	Consistently elevated TTFB with no backend-side cause	`backend_conn` vs `backend_reuse` ratio; new connections add handshake latency
Backend connection limit	`backend_busy` incrementing; TTFB rises under load	Backend `.max_connections` setting and concurrent connection count
Java backend GC pauses	Periodic TTFB spikes at regular intervals	JVM GC logs and pause durations
Backend deploy with cold start	TTFB spike correlating with deployment time	Deployment timeline and backend warmup status
Low connection reuse	High `backend_conn` relative to `backend_reuse`	`backend_idle_timeout` and backend keepalive configuration

Quick checks

# Thread pool state
varnishstat -1 -f MAIN.threads -f MAIN.thread_queue_len -f MAIN.threads_limited -f MAIN.threads_failed

# Backend TTFB for recent requests: delta between Timestamp:Bereq and Timestamp:Beresp
varnishlog -g request -i Timestamp -q 'BerespStatus gt 0'

# Filter for backend responses with TTFB over 1 second
varnishlog -d -g request -q "Timestamp:Beresp[2] > 1.0"

# Backend health and probe details
varnishadm backend.list -p

# Connection reuse ratio
varnishstat -1 -f MAIN.backend_conn -f MAIN.backend_reuse -f MAIN.backend_recycle

# Fetch failures and thread starvation on the fetch path
varnishstat -1 -f MAIN.fetch_failed -f MAIN.fetch_no_thread -f MAIN.bgfetch_no_thread

# Fetch error strings for timeout diagnosis
varnishlog -g request -q 'FetchError'

# Session and request drops
varnishstat -1 -f MAIN.sess_dropped -f MAIN.req_dropped

# Current thread pool and timeout configuration
varnishadm param.show thread_pool_max
varnishadm param.show thread_pools
varnishadm param.show thread_queue_limit
varnishadm param.show first_byte_timeout
varnishadm param.show between_bytes_timeout

How to diagnose it

Confirm thread pool saturation. Compare MAIN.threads against thread_pool_max * thread_pools. If threads are at the ceiling and thread_queue_len is nonzero, the pool is exhausted. threads_limited incrementing confirms Varnish hit the configured maximum.
Measure backend TTFB. Varnish does not expose backend latency as a counter in varnishstat. Parse it from the shared memory log. Run varnishlog -g request -i Timestamp -q 'BerespStatus gt 0' and check the delta between Timestamp:Bereq and Timestamp:Beresp for each backend transaction. For high-volume systems, filter directly for slow responses using the query in Quick checks above.
Identify which backends are slow. In a multi-backend director, aggregate TTFB can hide a single bad backend. Use varnishadm backend.list -p for per-backend health and probe details. Cross-reference slow TTFB in varnishlog with the specific backend each request hit.
Check connection reuse. New backend connections add TCP handshake (and TLS where applicable) latency to every fetch. Compare MAIN.backend_conn (new connections) to MAIN.backend_reuse (reused). If reuse is low, Varnish is opening fresh connections repeatedly, inflating TTFB. Check backend_idle_timeout and backend keepalive settings.
Look for fetch errors. When TTFB exceeds first_byte_timeout (default 60 seconds), the fetch fails. Check FetchError in varnishlog for specific error strings. On current Varnish, a first-byte timeout is logged as FetchError: first byte timeout; response-header stalls and closures can also be logged as timeout, backend closed, overflow, or an HTC <status> (<description>) error. A between-bytes stall is a read timeout after the first byte. These errors mean slow TTFB has crossed a hard cutoff and turned into fetch failures, which surface as 503s to clients.
Distinguish TTFB from body transfer time. A backend can return the first byte quickly but stall during body transfer. The thread is still held. Compare the TTFB delta (Bereq to Beresp) against the full fetch time in varnishlog timestamps. If TTFB is fast but total fetch time is slow, the problem is body transfer governed by between_bytes_timeout, not initial backend response.
Check for OS-level thread creation failures. If MAIN.threads_failed is incrementing, the OS is refusing thread creation. This is not a Varnish configuration problem. Check ulimit -u for the Varnish process, available memory for thread stacks, and whether systemd’s TasksMax cgroup limit is blocking creation. The symptom in dmesg is cgroup: fork rejected by pids controller.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Backend TTFB (varnishlog)	Leading indicator of thread pool death. Only available via log parsing, not in varnishstat.	P99 above 2x baseline. P99 measured in seconds means thread exhaustion is imminent.
`MAIN.threads`	Current worker thread count. At `thread_pool_max * pools`, pool is saturated.	Consistently above 80% of max during peak.
`MAIN.thread_queue_len`	Instantaneous queue depth. Nonzero means all threads are busy.	Any sustained nonzero value. Approaching `thread_queue_limit` (default 20) means drops are imminent.
`MAIN.threads_limited`	Counter for times thread creation was refused by `thread_pool_max`.	Rate above zero.
`MAIN.threads_failed`	Counter for times the OS refused thread creation.	Any nonzero value. System-level problem, not Varnish config.
`MAIN.sess_dropped` / `MAIN.req_dropped`	Sessions (HTTP/1) and streams (HTTP/2) dropped because no thread was available.	Any sustained nonzero rate. Users get nothing.
`MAIN.fetch_no_thread` / `MAIN.bgfetch_no_thread`	Backend fetches that failed because no thread could be dispatched.	Any nonzero rate. Thread starvation has reached the fetch path.
`MAIN.backend_busy`	Backend `.max_connections` limit reached.	Nonzero rate. Backend is rejecting connections.
`MAIN.backend_reuse` / `MAIN.backend_conn`	Connection pool efficiency. Low reuse means more handshake latency inflating TTFB.	Reuse ratio below 50%.
`MAIN.fetch_failed`	Fetch operations that failed after connection. Combined with FetchError logs for diagnosis.	Rate above zero.

Fixes

Backend is slow (the most common cause)

Fix the backend. Teams frequently respond to thread pool exhaustion by tuning Varnish parameters instead of addressing the backend. Increasing thread_pool_max buys time but does not fix the root cause. If backend TTFB is 8 seconds, you need 8x the threads to handle the same concurrency as a 1-second TTFB.

If a single backend in a director is the culprit, mark it sick to stop sending traffic:

# Disruptive: traffic shifts to remaining backends immediately
varnishadm backend.set_health <backend_name> sick

If all backends are slow, investigate backend-side: database queries, GC pauses, connection pool limits, CPU saturation.

Thread pool is too small

As a stop-gap, increase thread_pool_max at runtime:

# Applies immediately, consumes memory for thread stacks
varnishadm param.set thread_pool_max 8000

Each thread consumes stack memory. The default thread_pool_stack is 80kB on 64-bit systems since Varnish 7.0, increased from 48kB for PCRE2 JIT compatibility. Increasing thread_pool_max from 5000 to 8000 per pool with 2 pools adds roughly 480MB of stack memory, before accounting for per-thread workspace. Verify the system has the RAM before raising this parameter. Note that param.set changes do not persist across restarts.

Timeout cutoffs

first_byte_timeout (default 60 seconds) and between_bytes_timeout (default 60 seconds) are hard cutoffs. If the backend exceeds first_byte_timeout, the fetch fails and the client gets a 503 or a stale object if grace is configured.

If backends are legitimately slow and you cannot fix them, you face a tradeoff:

Increase first_byte_timeout to accommodate slowness: threads held even longer, worse pool pressure.
Decrease first_byte_timeout to fail fast: clients get errors sooner, threads are freed faster.

Failing fast with a shorter timeout combined with grace and stale-if-error protects the thread pool at the cost of some user-visible errors. The right answer is still to fix the backend.

Low connection reuse

If backend_reuse is low relative to backend_conn, new connections are inflating TTFB with TCP and TLS handshake overhead. Check:

backend_idle_timeout (default 60 seconds): if too short, connections are closed before reuse.
Backend keepalive settings: the backend must support connection reuse.
Backend connection limits: if the backend closes connections under load, reuse suffers.

See Varnish backend connection reuse low for a deeper treatment.

Prevention

Monitor backend TTFB continuously. Varnish does not expose this as a counter. You need log-based monitoring (varnishncsa or varnishlog parsing) to track TTFB percentiles over time. Without this, the first sign of trouble is sess_dropped.
Alert on P99 above 2x baseline. A P99 measured in seconds means thread exhaustion is approaching. The exact threshold depends on your thread_pool_max and traffic volume.
Track thread pool headroom. Monitor MAIN.threads / (thread_pool_max * thread_pools). Alert above 80% sustained.
Configure grace and stale-if-error. When backends slow down, grace serves stale content while threads wait. This reduces user-visible errors and buys time.
Right-size thread_pool_max for worst-case TTFB, not average. If your worst-case TTFB is 5x your average, you need 5x the thread headroom during degradation.
Check OS thread limits. On systemd-managed systems, verify the service unit’s TasksMax is not blocking thread creation. Check ulimit -u for the Varnish user.

How Netdata helps

Per-second metric collection means thread pool saturation (MAIN.threads, thread_queue_len, threads_limited) is visible with sub-second granularity. Thread pool exhaustion is a cliff edge: minute-averaged metrics can miss the entire spike.
Anomaly detection on MAIN.threads and thread_queue_len surfaces the accumulation pattern before it reaches the drop threshold.
Correlation across signals. Backend TTFB is not a counter, but its consequences are. Netdata correlates MAIN.threads rising with cache_hit dropping (more misses means more backend fetches), backend_req increasing, and eventually sess_dropped incrementing. Seeing these together on one timeline shortens root-cause analysis.
Per-backend visibility. VBE.*.happy counters distinguish a single sick backend from a systemic slowdown.
Connection pool efficiency tracking. MAIN.backend_conn vs MAIN.backend_reuse over time reveals whether connection reuse degradation contributes to TTFB inflation.