$ guides / varnish / varnish-backend-connection-reuse-low ▌

Operations Guides

Varnish backend connection reuse low: keepalive not working and slow TTFB

When Varnish reuses backend TCP connections through HTTP keepalive, cache misses skip connection setup: no TCP handshake, no optional TLS negotiation, no kernel connection-tracking overhead. When reuse collapses, every backend fetch pays that cost, and it shows up directly in time-to-first-byte.

The reuse ratio is backend_reuse / (backend_reuse + backend_conn) from varnishstat. A ratio below 50% when your backend supports keepalive means the majority of fetches are opening fresh TCP connections. This adds latency on every cache miss, increases backend CPU and connection-tracking load, and accelerates file descriptor consumption on the Varnish child process.

What this means

Varnish maintains a pool of backend connections reusable across requests via HTTP keepalive. The connection lifecycle involves four counters:

backend_conn: a new TCP connection was opened to the backend
backend_recycle: a connection was returned to the idle pool after a fetch completed
backend_reuse: a pooled connection was picked up for a new fetch

When the reuse ratio is healthy (above 80% for most workloads), most fetches pick up a recycled connection. When it drops below 50%:

TTFB increases. Each new connection pays the TCP handshake cost (at least one round trip) plus any TLS handshake cost (one to two additional round trips if the backend is HTTPS). This overhead is per-miss, not amortized.
File descriptor pressure rises. Short-lived connections cycle faster, and each one occupies a socket pair on both Varnish and the backend. At high backend request rates, low reuse can push the Varnish child process toward its FD limit.

The critical diagnostic split is backend_recycle versus backend_reuse, plus the close reason in BackendClose records. If recycled connections are not reused, or BackendClose ... close RESP_CLOSE records recur, the backend is closing idle connections before Varnish reuses them. This points to a timeout mismatch between Varnish’s backend_idle_timeout and the backend’s own idle connection timeout.

flowchart TD
    A["Reuse ratio below 50%"] --> B{"reuses lag recycles or\nBackendClose RESP_CLOSE?"}
    B -->|Yes| C["Keepalive timeout mismatch:\nbackend closes before Varnish reuses"]
    B -->|No| D{"Connection: close\nor HTTP/1.0 from backend?"}
    D -->|Yes| E["Backend disables keepalive\nby policy or protocol"]
    D -->|No| F["Firewall or NAT dropping\nidle connections early"]
    C --> G["Align backend_idle_timeout\nwith backend keepalive timeout"]
    E --> H["Fix backend config or\nupgrade to HTTP/1.1"]
    F --> I["Increase firewall idle timeout\nor decrease backend_idle_timeout"]

Common causes

Cause	What it looks like	First thing to check
Backend keepalive timeout shorter than `backend_idle_timeout`	`backend_reuse` lags `backend_recycle`; `BackendClose ... close RESP_CLOSE`	Backend keepalive timeout (Apache `KeepAliveTimeout`, Nginx `keepalive_timeout`)
Backend sending `Connection: close`	`backend_conn` high, `backend_reuse` near zero, no `backend_recycle`	`varnishlog -b -i RxHeader` for Connection header from backend
Backend using HTTP/1.0 without explicit keepalive	`backend_reuse` near zero, all connections are new	`varnishtop -b -i RxProtocol` for HTTP/1.0 responses
Stateful firewall or NAT idle timeout too low	Sporadic `backend_fail`, intermittent fetch failures	Firewall/NAT TCP idle timeout vs `backend_idle_timeout`
`backend_idle_timeout` too short for traffic pattern	Low recycle volume, `backend_conn` high, backend keepalive is long	`varnishadm param.show backend_idle_timeout`
High response-time variance	Recycles do not become reuses during slow periods	Backend TTFB distribution via varnishlog Timestamp deltas

Quick checks

# Core backend connection counters
varnishstat -1 -f MAIN.backend_conn -f MAIN.backend_reuse \
  -f MAIN.backend_recycle

# Inspect backend close reasons (RESP_CLOSE means the peer or response
# asked for the close; recycle means Varnish returned it to the pool)
varnishlog -g raw -i BackendClose

# Compute the reuse ratio from the above:
# reuse_ratio = backend_reuse / (backend_reuse + backend_conn)

# Varnish idle timeout parameter
varnishadm param.show backend_idle_timeout

# Check for Connection: close from backend (runs until interrupted)
varnishlog -b -i RxHeader | grep -i connection

# Check backend HTTP protocol version (look for HTTP/1.0)
varnishtop -b -i RxProtocol

# Backend connection failures (timeout, refused, reset)
varnishstat -1 -f MAIN.backend_fail

# Fetch failures (connection OK but fetch broke)
varnishstat -1 -f MAIN.fetch_failed

# File descriptor usage on the child process (newest PID = child)
CHILD_PID=$(pgrep -n varnishd)
ls /proc/$CHILD_PID/fd | wc -l
cat /proc/$CHILD_PID/limits | grep 'Max open files'

How to diagnose it

Compute the reuse ratio from counter deltas. Take two readings of backend_reuse and backend_conn spaced 60 seconds apart. Compute the ratio from the deltas, not the cumulative values. All MAIN.* counters reset on child restart, so cumulative ratios are unreliable after a restart until the counters accumulate meaningful volume.
Compare backend_recycle to backend_reuse and inspect BackendClose reasons. This is the most informative comparison. If many recycled connections never become reuses, or BackendClose ... close RESP_CLOSE records recur, the backend is closing idle connections before Varnish reuses them. This almost always points to a keepalive timeout mismatch.
Inspect what the backend is actually sending. Run varnishlog -b -i RxHeader and look for a Connection: close header. Check varnishtop -b -i RxProtocol to see whether the backend responds with HTTP/1.0, which cannot do keepalive by default. Both conditions force Varnish to open a new connection on every fetch.
Check backend keepalive settings. On Apache, check KeepAliveTimeout (default 5 seconds) and MaxKeepAliveRequests (default 100, meaning the server sends Connection: close after 100 requests on the same connection). On Nginx, check keepalive_timeout (default 75 seconds). If the backend’s idle timeout is shorter than Varnish’s backend_idle_timeout (default 60 seconds), the backend wins the race and closes the connection first.
Check for network-layer idle timeouts. Stateful firewalls, load balancers, and NAT devices between Varnish and the backend may track TCP connections with idle timeouts shorter than both Varnish’s and the backend’s keepalive timeouts. They silently drop or reset connections that Varnish thinks are still alive. If the backend closes keepalive connections faster than Varnish’s backend_idle_timeout, Varnish will get ECONNRESET on reused connections, appearing as sporadic fetch errors that surface during quiet periods between bursts. Check for intermittent backend_fail or fetch_failed increments that correlate with those quiet periods.
Check file descriptor pressure. Low reuse means higher FD churn. Each backend connection consumes one FD on the child process. Use pgrep -n varnishd to identify the child PID (newest process), then compare /proc/$PID/fd count to /proc/$PID/limits. If MAIN.sess_fail_emfile is incrementing, acceptor-level FD exhaustion (EMFILE) is confirmed. MAIN.sess_fail is the aggregate acceptor failure counter.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`backend_reuse / (backend_reuse + backend_conn)`	Core reuse ratio; tells you if keepalive pooling is working	Below 50% when backend supports keepalive
`BackendClose` reason rate	Distinguishes peer-closed connections from normal recycles	Repeated `close RESP_CLOSE`; `backend_reuse` far below `backend_recycle`
`backend_conn` rate	Rate of new TCP connections to backends	Sustained high rate relative to `backend_reuse`
`backend_fail`	Connection-level failures (refused, timeout, reset)	Sustained nonzero rate; may indicate firewall or NAT drops
`fetch_failed`	Fetch broke after connection succeeded	Correlates with stale-connection reuse races
Backend TTFB (varnishlog Timestamp deltas)	First-byte latency, sensitive to connection setup cost	Elevated P99 on cache misses during low-reuse periods
Child process FD count	Each connection consumes one FD	Approaching `ulimit -n` limit
`backend_busy`	Backend `.max_connections` reached	Nonzero rate; backend rejecting connections

Fixes

Backend keepalive timeout shorter than backend_idle_timeout

The most common cause. Varnish defaults backend_idle_timeout to 60 seconds. If the backend closes idle connections faster (Apache KeepAliveTimeout defaults to 5 seconds, some application servers default even lower), Varnish will frequently discard or fail to recycle the closed connection, fall back to a new connection, and increment backend_conn.

Two fixes, with different tradeoffs:

Increase the backend’s keepalive timeout to exceed Varnish’s backend_idle_timeout. Recommended when the backend can afford to hold idle connections longer. On Apache, set KeepAliveTimeout above 60. On Nginx, set keepalive_timeout above 60. The tradeoff is more memory consumed on the backend for idle worker threads or processes holding those connections.

Decrease Varnish’s backend_idle_timeout to be shorter than the backend’s timeout:

# Runtime only; persist via -p in your varnishd startup parameters
varnishadm param.set backend_idle_timeout <seconds>

This makes Varnish proactively close connections before the backend does. The tradeoff is more frequent reconnections if traffic is bursty with gaps longer than the new timeout.

For optimal reuse, set both timeouts as long as your traffic patterns allow, with Varnish’s slightly shorter than the backend’s.

Backend sending Connection: close

Some backends send Connection: close on every response, or after a fixed number of requests. This forces Varnish to close the connection after each qualifying fetch, making reuse impossible. Common sources:

Apache MaxKeepAliveRequests at its default of 100. After 100 requests on a single connection, the server sends Connection: close. On high-traffic paths that sustain more than 100 sequential requests per connection, this cycles frequently. Set MaxKeepAliveRequests 0 for unlimited requests per connection.
Application frameworks that explicitly disable keepalive on specific code paths (error responses, certain endpoints).
Backend proxies or load balancers that strip or override keepalive headers between the origin and Varnish.

Check varnishlog -b -i RxHeader | grep -i connection to see what the backend is actually sending. Fix the backend configuration, not Varnish.

Backend using HTTP/1.0

HTTP/1.0 does not negotiate persistent connections by default; it requires explicit keepalive handling. An HTTP/1.0 backend without that behavior cannot be reused by Varnish. Check with varnishtop -b -i RxProtocol. This is typically a backend misconfiguration or an old application server that predates HTTP/1.1. Upgrading the backend to HTTP/1.1 or later resolves it. If the backend cannot be upgraded, accept that reuse will be low for those endpoints and size file descriptor limits accordingly.

Stateful firewall or NAT idle timeout

Network devices between Varnish and the backend may track TCP connections with idle timeouts shorter than the application-level keepalive timeouts. When the firewall’s timer fires, it silently drops or resets the connection. Varnish discovers this only when it tries to reuse the connection and gets an error.

This manifests as sporadic backend_fail or fetch_failed increments, often during quiet periods between traffic bursts. The fix is to increase the firewall or NAT TCP idle timeout to exceed the longest keepalive timeout in the chain. Alternatively, decrease backend_idle_timeout so Varnish closes before the firewall does.

backend_idle_timeout too short for traffic patterns

If backend_idle_timeout was tuned aggressively low (for example, 5 seconds to match a fast backend), Varnish proactively closes connections before they can be reused during bursty traffic with short gaps. If backend_recycle is also low (the backend is not closing usable connections) but backend_conn is still high, Varnish is closing too aggressively. Compare backend_idle_timeout to your traffic pattern’s inter-request gaps. A backend that supports long keepalive but receives bursty traffic with 10-second gaps will show low reuse if backend_idle_timeout is set to 5.

Prevention

Monitor the reuse ratio continuously. Track backend_reuse / (backend_reuse + backend_conn) as a gauge and alert when it drops below 50% for a sustained period.
Watch BackendClose reasons. Repeated peer-initiated closes are an early signal of a keepalive timeout mismatch, often appearing after a backend configuration change that nobody communicated to the cache team.
Audit backend keepalive settings during changes. Backend teams rarely include KeepAliveTimeout, MaxKeepAliveRequests, or equivalent settings in their change checklist.
Include firewall and NAT timeouts in the path audit. Any device with a shorter timeout than Varnish or the backend will break reuse silently.
Track file descriptor consumption. Low reuse burns FDs faster than steady-state keepalive. Monitor the child process FD count relative to the limit.

How Netdata helps

The Varnish collector exposes backend_conn, backend_reuse, and backend_recycle as per-second metrics. Correlating the reuse ratio trend with backend TTFB in a single timeline makes the keepalive-to-latency relationship visible without manual counter math.
backend_reuse and backend_recycle rates are charted independently, so you can see a keepalive timeout mismatch the moment it starts.
File descriptor pressure on the Varnish child process is tracked alongside backend connection counters, making the downstream effect of low reuse on FD consumption visible.
backend_fail, fetch_failed, and backend health probe status are correlated with connection reuse metrics, helping distinguish a keepalive mismatch from a genuine backend connectivity problem.