When Varnish reuses backend TCP connections through HTTP keepalive, cache misses skip connection setup: no TCP handshake, no optional TLS negotiation, no kernel connection-tracking overhead. When reuse collapses, every backend fetch pays that cost, and it shows up directly in time-to-first-byte.

The reuse ratio is backend_reuse / (backend_reuse + backend_conn) from varnishstat. A ratio below 50% when your backend supports keepalive means the majority of fetches are opening fresh TCP connections. This adds latency on every cache miss, increases backend CPU and connection-tracking load, and accelerates file descriptor consumption on the Varnish child process.

What this means

Varnish maintains a pool of backend connections reusable across requests via HTTP keepalive. The connection lifecycle involves four counters:

  • backend_conn: a new TCP connection was opened to the backend
  • backend_recycle: a connection was returned to the idle pool after a fetch completed
  • backend_reuse: a pooled connection was picked up for a new fetch
  • backend_toolate: Varnish attempted to reuse a pooled connection, but the backend had already closed it

When the reuse ratio is healthy (above 80% for most workloads), most fetches pick up a recycled connection. When it drops below 50%:

  1. TTFB increases. Each new connection pays the TCP handshake cost (at least one round trip) plus any TLS handshake cost (one to two additional round trips if the backend is HTTPS). This overhead is per-miss, not amortized.
  2. File descriptor pressure rises. Short-lived connections cycle faster, and each one occupies a socket pair on both Varnish and the backend. At high backend request rates, low reuse can push the Varnish child process toward its FD limit.

The critical diagnostic split is backend_recycle versus backend_toolate. If backend_recycle is healthy (connections are being returned to the pool) but backend_toolate is high relative to it, Varnish is doing its part but the backend is closing idle connections before Varnish reuses them. This points to a timeout mismatch between Varnish’s backend_idle_timeout and the backend’s own idle connection timeout.

flowchart TD
    A["Reuse ratio below 50%"] --> B{"backend_toolate high\nvs backend_recycle?"}
    B -->|Yes| C["Keepalive timeout mismatch:\nbackend closes before Varnish reuses"]
    B -->|No| D{"Connection: close\nor HTTP/1.0 from backend?"}
    D -->|Yes| E["Backend disables keepalive\nby policy or protocol"]
    D -->|No| F["Firewall or NAT dropping\nidle connections early"]
    C --> G["Align backend_idle_timeout\nwith backend keepalive timeout"]
    E --> H["Fix backend config or\nupgrade to HTTP/1.1"]
    F --> I["Increase firewall idle timeout\nor decrease backend_idle_timeout"]

Common causes

CauseWhat it looks likeFirst thing to check
Backend keepalive timeout shorter than backend_idle_timeoutbackend_toolate high relative to backend_recycleBackend keepalive timeout (Apache KeepAliveTimeout, Nginx keepalive_timeout)
Backend sending Connection: closebackend_conn high, backend_reuse near zero, backend_toolate lowvarnishlog -b -i RxHeader for Connection header from backend
Backend using HTTP/1.0backend_reuse near zero, all connections are newvarnishtop -b -i RxProtocol for HTTP/1.0 responses
Stateful firewall or NAT idle timeout too lowSporadic backend_fail or backend_toolate, intermittent fetch failuresFirewall/NAT TCP idle timeout vs backend_idle_timeout
backend_idle_timeout too short for traffic patternbackend_toolate low, backend_conn high, backend keepalive is longvarnishadm param.show backend_idle_timeout
High response-time variancebackend_toolate intermittent, correlates with slow backend periodsBackend TTFB distribution via varnishlog Timestamp deltas

Quick checks

# Core backend connection counters
varnishstat -1 -f MAIN.backend_conn -f MAIN.backend_reuse \
  -f MAIN.backend_recycle -f MAIN.backend_toolate

# Compute the reuse ratio from the above:
# reuse_ratio = backend_reuse / (backend_reuse + backend_conn)

# Varnish idle timeout parameter
varnishadm param.show backend_idle_timeout

# Check for Connection: close from backend (runs until interrupted)
varnishlog -b -i RxHeader | grep -i connection

# Check backend HTTP protocol version (look for HTTP/1.0)
varnishtop -b -i RxProtocol

# Backend connection failures (timeout, refused, reset)
varnishstat -1 -f MAIN.backend_fail

# Fetch failures (connection OK but fetch broke)
varnishstat -1 -f MAIN.fetch_failed

# File descriptor usage on the child process (newest PID = child)
CHILD_PID=$(pgrep -n varnishd)
ls /proc/$CHILD_PID/fd | wc -l
cat /proc/$CHILD_PID/limits | grep 'Max open files'

How to diagnose it

  1. Compute the reuse ratio from counter deltas. Take two readings of backend_reuse and backend_conn spaced 60 seconds apart. Compute the ratio from the deltas, not the cumulative values. All MAIN.* counters reset on child restart, so cumulative ratios are unreliable after a restart until the counters accumulate meaningful volume.

  2. Compare backend_recycle to backend_toolate. This is the single most informative comparison. If backend_toolate is climbing at a rate comparable to or exceeding backend_recycle, the backend is closing idle connections before Varnish reuses them. This almost always points to a keepalive timeout mismatch.

  3. Inspect what the backend is actually sending. Run varnishlog -b -i RxHeader and look for a Connection: close header. Check varnishtop -b -i RxProtocol to see whether the backend responds with HTTP/1.0, which cannot do keepalive by default. Both conditions force Varnish to open a new connection on every fetch.

  4. Check backend keepalive settings. On Apache, check KeepAliveTimeout (default 5 seconds) and MaxKeepAliveRequests (default 100, meaning the server sends Connection: close after 100 requests on the same connection). On Nginx, check keepalive_timeout (default 75 seconds). If the backend’s idle timeout is shorter than Varnish’s backend_idle_timeout (default 60 seconds), the backend wins the race and closes the connection first.

  5. Check for network-layer idle timeouts. Stateful firewalls, load balancers, and NAT devices between Varnish and the backend may track TCP connections with idle timeouts shorter than both Varnish’s and the backend’s keepalive timeouts. They silently drop or reset connections that Varnish thinks are still alive. If the backend closes keepalive connections faster than Varnish’s backend_idle_timeout, Varnish will get ECONNRESET on reused connections, appearing as sporadic fetch errors that surface during quiet periods between bursts. Check for intermittent backend_fail or fetch_failed increments that correlate with those quiet periods.

  6. Check file descriptor pressure. Low reuse means higher FD churn. Each backend connection consumes one FD on the child process. Use pgrep -n varnishd to identify the child PID (newest process), then compare /proc/$PID/fd count to /proc/$PID/limits. If MAIN.sess_fail or equivalent session failure counters are incrementing, FD exhaustion is confirmed.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
backend_reuse / (backend_reuse + backend_conn)Core reuse ratio; tells you if keepalive pooling is workingBelow 50% when backend supports keepalive
backend_toolate vs backend_recycle rateDistinguishes backend-closed-from-pool from never-pooledbackend_toolate rate approaching backend_recycle rate
backend_conn rateRate of new TCP connections to backendsSustained high rate relative to backend_reuse
backend_failConnection-level failures (refused, timeout, reset)Sustained nonzero rate; may indicate firewall or NAT drops
fetch_failedFetch broke after connection succeededCorrelates with stale-connection reuse races
Backend TTFB (varnishlog Timestamp deltas)First-byte latency, sensitive to connection setup costElevated P99 on cache misses during low-reuse periods
Child process FD countEach connection consumes one FDApproaching ulimit -n limit
backend_busyBackend .max_connections reachedNonzero rate; backend rejecting connections

Fixes

Backend keepalive timeout shorter than backend_idle_timeout

The most common cause. Varnish defaults backend_idle_timeout to 60 seconds. If the backend closes idle connections faster (Apache KeepAliveTimeout defaults to 5 seconds, some application servers default even lower), Varnish will frequently pick up a dead connection, increment backend_toolate, fall back to a new connection, and increment backend_conn.

Two fixes, with different tradeoffs:

Increase the backend’s keepalive timeout to exceed Varnish’s backend_idle_timeout. Recommended when the backend can afford to hold idle connections longer. On Apache, set KeepAliveTimeout above 60. On Nginx, set keepalive_timeout above 60. The tradeoff is more memory consumed on the backend for idle worker threads or processes holding those connections.

Decrease Varnish’s backend_idle_timeout to be shorter than the backend’s timeout:

# Runtime only; persist via -p in your varnishd startup parameters
varnishadm param.set backend_idle_timeout <seconds>

This makes Varnish proactively close connections before the backend does. The tradeoff is more frequent reconnections if traffic is bursty with gaps longer than the new timeout.

For optimal reuse, set both timeouts as long as your traffic patterns allow, with Varnish’s slightly shorter than the backend’s.

Backend sending Connection: close

Some backends send Connection: close on every response, or after a fixed number of requests. This forces Varnish to close the connection after each qualifying fetch, making reuse impossible. Common sources:

  • Apache MaxKeepAliveRequests at its default of 100. After 100 requests on a single connection, the server sends Connection: close. On high-traffic paths that sustain more than 100 sequential requests per connection, this cycles frequently. Set MaxKeepAliveRequests 0 for unlimited requests per connection.
  • Application frameworks that explicitly disable keepalive on specific code paths (error responses, certain endpoints).
  • Backend proxies or load balancers that strip or override keepalive headers between the origin and Varnish.

Check varnishlog -b -i RxHeader | grep -i connection to see what the backend is actually sending. Fix the backend configuration, not Varnish.

Backend using HTTP/1.0

HTTP/1.0 does not support keepalive by default. If the backend responds with HTTP/1.0, Varnish cannot reuse the connection. Check with varnishtop -b -i RxProtocol. This is typically a backend misconfiguration or an old application server that predates HTTP/1.1. Upgrading the backend to HTTP/1.1 or later resolves it. If the backend cannot be upgraded, accept that reuse will be low for those endpoints and size file descriptor limits accordingly.

Stateful firewall or NAT idle timeout

Network devices between Varnish and the backend may track TCP connections with idle timeouts shorter than the application-level keepalive timeouts. When the firewall’s timer fires, it silently drops or resets the connection. Varnish discovers this only when it tries to reuse the connection and gets an error.

This manifests as sporadic backend_fail or fetch_failed increments, often during quiet periods between traffic bursts. The fix is to increase the firewall or NAT TCP idle timeout to exceed the longest keepalive timeout in the chain. Alternatively, decrease backend_idle_timeout so Varnish closes before the firewall does.

backend_idle_timeout too short for traffic patterns

If backend_idle_timeout was tuned aggressively low (for example, 5 seconds to match a fast backend), Varnish proactively closes connections before they can be reused during bursty traffic with short gaps. If backend_toolate is low (the backend is not closing connections) but backend_conn is still high, Varnish is closing too aggressively. Compare backend_idle_timeout to your traffic pattern’s inter-request gaps. A backend that supports long keepalive but receives bursty traffic with 10-second gaps will show low reuse if backend_idle_timeout is set to 5.

Prevention

  • Monitor the reuse ratio continuously. Track backend_reuse / (backend_reuse + backend_conn) as a gauge and alert when it drops below 50% for a sustained period.
  • Watch backend_toolate independently. A rising backend_toolate rate is the earliest signal of a keepalive timeout mismatch, often appearing after a backend configuration change that nobody communicated to the cache team.
  • Audit backend keepalive settings during changes. Backend teams rarely include KeepAliveTimeout, MaxKeepAliveRequests, or equivalent settings in their change checklist.
  • Include firewall and NAT timeouts in the path audit. Any device with a shorter timeout than Varnish or the backend will break reuse silently.
  • Track file descriptor consumption. Low reuse burns FDs faster than steady-state keepalive. Monitor the child process FD count relative to the limit.

How Netdata helps

  • The Varnish collector exposes backend_conn, backend_reuse, backend_recycle, and backend_toolate as per-second metrics. Correlating the reuse ratio trend with backend TTFB in a single timeline makes the keepalive-to-latency relationship visible without manual counter math.
  • backend_toolate and backend_recycle rates are charted independently, so you can see a keepalive timeout mismatch the moment it starts.
  • File descriptor pressure on the Varnish child process is tracked alongside backend connection counters, making the downstream effect of low reuse on FD consumption visible.
  • backend_fail, fetch_failed, and backend health probe status are correlated with connection reuse metrics, helping distinguish a keepalive mismatch from a genuine backend connectivity problem.