$ guides / varnish / varnish-backend-conn-failures ▌

Operations Guides

Varnish backend_fail, backend_unhealthy, and backend_busy: three different backend problems

Three varnishstat counters track backend problems, and operators routinely conflate them. backend_fail, backend_unhealthy, and backend_busy each fire at a different point in the backend connection decision flow, have different root causes, and need different fixes.

The most dangerous confusion is also the subtlest: a backend can be completely offline with backend_fail sitting at zero. If the backend is probe-sick, Varnish never attempts a connection, so backend_fail never increments. Only backend_unhealthy reveals this state. An operator who monitors backend_fail alone sees a healthy system while the backend is dark.

What these counters actually measure

Counter	What happened	Connection attempted?	Typical root cause
`MAIN.backend_fail`	TCP connection to backend was attempted and failed	Yes	Backend crash, network partition, firewall, listen queue overflow
`MAIN.backend_unhealthy`	Varnish skipped the backend because its probe marked it sick	No	Backend unresponsive to health probes, probe misconfiguration
`MAIN.backend_busy`	Backend’s `.max_connections` ceiling was already reached	No	Too many concurrent connections open to this backend

backend_fail means Varnish tried and the connection failed. The other two mean Varnish decided not to try. They measure different layers of the stack.

How each counter fires

When Varnish needs to fetch from a backend, it runs through a decision chain. Each counter fires at a specific branch. If the chain stops early, later counters never see the traffic.

flowchart TD
    A["Varnish needs backend content"] --> B{"Backend probe healthy?"}
    B -- No --> C["backend_unhealthy increments\nconnection NOT attempted"]
    B -- Yes --> D{"At .max_connections?"}
    D -- Yes --> E["backend_busy increments\nconnection NOT attempted"]
    D -- No --> F["TCP connect to backend"]
    F -- Success --> G["Fetch proceeds"]
    F -- Fail --> H["backend_fail increments\nconnection attempted, failed"]

The counters are mutually exclusive for a given request. A single fetch hits exactly one path. If the probe is sick, backend_fail and backend_busy stay at zero for that request regardless of what the backend is doing. If the probe is healthy but max_connections is saturated, backend_fail stays at zero. Only when the probe passes and the connection pool has room does Varnish attempt a TCP connect, and only then can backend_fail fire.

backend_fail: the connection was attempted and failed

MAIN.backend_fail increments when Varnish attempts a TCP connection to a backend and the connection fails. The SYN was sent, and something went wrong at or below the TCP layer.

Common causes: backend process crash, network partition between Varnish and the backend, firewall or security group blocking the port, or the backend’s listen queue overflowing under load.

The varnishlog output for these failures includes the errno, which narrows the root cause:

errno	Meaning	Likely cause
111	Connection refused	Backend process not listening (crashed, not started, wrong port)
110	Connection timed out	Network partition, firewall dropping packets, backend too slow to accept
104	Connection reset	Backend crashed mid-handshake, TLS mismatch
101	Network unreachable	Routing failure, interface down

# See backend connection failures with errno detail
varnishlog -g request -q 'FetchError ~ "fail"'

# Check the global counter rate
varnishstat -1 -f MAIN.backend_fail

The FetchError log line for a connection failure includes the backend name, the string fail, the errno number, and a short reason string. The errno tells you whether the problem is local (refused = process not running), network-level (timeout = partition or firewall), or protocol-level (reset = TLS or application crash).

backend_fail counts only connection failures, not HTTP-level errors. If the backend accepts the TCP connection and returns a 500 or 502, backend_fail does not increment. Those failures show up in fetch_failed and the FetchError log tags instead.

backend_unhealthy: the connection was never attempted

MAIN.backend_unhealthy increments when Varnish decides not to attempt a connection because the backend’s health probe has marked it sick. No SYN packet is sent. No TCP handshake begins.

This counter is the only varnishstat signal that directly reflects the probe-based health system. If all backends are sick and grace is configured, Varnish serves stale content and clients see no error. backend_unhealthy silently climbs while everything looks fine from the client side. When grace expires, 503s cascade.

# Check backend health with probe details
varnishadm backend.list -p

# See which backends are sick
varnishstat -1 -f 'VBE.*.happy'

# Check the unhealthy counter
varnishstat -1 -f MAIN.backend_unhealthy

The VBE.<backend>.happy counter shows the number of successful probes within the probe window. A value below the probe’s threshold means the backend is sick. The probe uses a threshold/window model: a backend is healthy if at least threshold out of the last window probes succeeded.

The probe requirement

backend_unhealthy requires a configured health probe. If no .probe block is defined for a backend in VCL, the backend is always considered healthy. In that case, backend_unhealthy will never increment, even if the backend is completely offline.

This is the most common gotcha. Operators see backend_unhealthy = 0 and assume the backend is fine. Without a probe, Varnish has no mechanism to detect backend sickness proactively. The only signal would be backend_fail incrementing as connections fail, which happens later in the decision chain and gives no early warning.

If you use a VMOD like libvmod-dynamic that manages backends programmatically, confirm that probes are being attached. A dynamically created backend without a probe will never produce backend_unhealthy increments.

backend_busy: the connection ceiling was hit

MAIN.backend_busy increments when the backend’s .max_connections limit has been reached and Varnish cannot open another connection. This counter is only meaningful if .max_connections is explicitly set in the backend definition. Without it, the default is unlimited connections, and backend_busy will never fire.

# Check if any backends have max_connections configured
# (look in your VCL for .max_connections in backend definitions)

# Check the busy counter
varnishstat -1 -f MAIN.backend_busy

The FetchError log line for this condition is FetchError backend <name>: busy.

When backend_busy is climbing, either the backend is genuinely overloaded and cannot keep up with the connection rate, or .max_connections is set too low for the traffic volume. The fix is either raising the limit (if the backend can handle more) or reducing backend load (improving cache hit rate, adding backends, distributing traffic).

Varnish 7.6 connection queuing

Varnish 7.6 introduced backend connection queuing. With the new .wait_timeout and .wait_limit backend properties, tasks that would have immediately hit backend_busy can instead queue and wait for an available connection slot. Two new counters track this: MAIN.backend_wait (tasks that queued) and MAIN.backend_wait_fail (tasks that waited but failed because wait_timeout was reached or the backend went sick).

If you are running Varnish 7.6+, check whether connection queuing is configured. If it is, backend_busy may stay low while backend_wait_fail climbs. The diagnostic approach changes: look at wait counters, not just the busy counter.

MAIN vs VBE: aggregate vs per-backend

The MAIN.backend_* counters are global aggregates across all backends. In a multi-backend setup (directors with multiple backends, multiple named backends), the MAIN counters tell you something is wrong but not which backend.

For per-backend granularity, use the VBE counters:

# Per-backend failures
varnishstat -1 -f 'VBE.*.fail'

# Per-backend health
varnishstat -1 -f 'VBE.*.happy'

VBE counter names include the backend name and connection details in the format VBE.<vcl_name>.<backend_name>(<ip>,,<port>).<metric>. The backend names are VCL-defined and may change on VCL reload, so VBE counter names are not stable across VCL changes.

VBE-level busy counters exist per backend. They increment only for that backend’s .max_connections ceiling.

One note from the official documentation: the VBE-level fail_* counters may be slightly inaccurate for efficiency reasons. Use them for diagnosis and trend detection, not for precise accounting.

Where teams get confused

Monitoring only backend_fail. Teams that watch backend_fail and nothing else can miss a completely offline backend. If the backend is probe-sick, Varnish never attempts a connection, so backend_fail stays at zero. Only backend_unhealthy reveals the problem.

No probe, no backend_unhealthy. Without a configured .probe block, backend_unhealthy will never increment. The backend could be down for hours and the counter stays at zero. Common with dynamically created backends or simple single-backend setups where the team assumed Varnish would detect sickness automatically.

backend_busy with no max_connections. If backend_busy is always zero, the backend might be fine, or .max_connections was never set. Check the VCL. Without the limit, Varnish opens unlimited connections and the backend itself becomes the bottleneck. You will see slow responses and thread pool exhaustion instead of backend_busy.

Conflating backend_fail with fetch failures. backend_fail counts connection-level failures. If the TCP connection succeeds but the fetch fails (truncated response, protocol error, timeout after connect), the counter is fetch_failed, not backend_fail. Cross-check with varnishlog -q 'FetchError' to see which layer is failing.

Signals to watch

Signal	Why it matters	Warning sign
`MAIN.backend_fail` rate	TCP connections to backends are failing	Any sustained nonzero rate
`MAIN.backend_unhealthy` rate	Backends are probe-sick, traffic is being skipped	Any sustained nonzero rate
`MAIN.backend_busy` rate	`.max_connections` ceiling is saturated	Any sustained nonzero rate (only fires if max_connections is set)
`VBE.<name>.happy`	Per-backend probe success count within window	Value below probe threshold means backend is sick
`MAIN.backend_wait_fail` rate (7.6+)	Queued connections timed out waiting for a slot	Any sustained nonzero rate
`MAIN.s_synth` rate	Varnish generating synthetic responses (often 503s)	Spike correlates with backend problems
`MAIN.fetch_failed` rate	Fetch operations failed after connection succeeded	Distinguishes connection failures from fetch failures
`MAIN.cache_hit_grace` rate	Serving stale content from grace	High rate may indicate backend is down and grace is masking it

Correlating with Netdata

These counters can spike and recover within a single minute, so per-second resolution matters for catching short backend_fail bursts that 60-second polling windows miss.

If backend_fail is zero but backend_unhealthy is climbing, the backend is being skipped by probes, not unreachable at the network layer. Viewing both on one timeline makes this immediately obvious.
Per-backend VBE metrics in Netdata let you identify which backend in a director is degrading before the aggregate MAIN counter reflects it. A single backend going sick in a multi-backend director is invisible in the aggregate until traffic concentrates on the remaining backends.
Correlating backend counters with s_synth (synthetic 503 responses) and cache_hit_grace shows whether backend problems are reaching users or being masked by stale content.
Alerting on each counter independently ensures a probe-sick backend is not hidden behind a healthy backend_fail reading.