$ guides / varnish / varnish-fetch-failed ▌

Operations Guides

Varnish fetch_failed: backend connected but the fetch broke

The MAIN.fetch_failed counter is climbing in varnishstat. Clients are seeing intermittent 503 responses. Backend health probes report healthy, and MAIN.backend_fail is zero. The TCP connection to the backend succeeded, but the fetch itself broke.

The problem is not connectivity. It is what happens after: malformed response headers, a truncated body, broken chunked encoding, a failed gzip decompression, or a timeout between bytes. Varnish established the connection, began the HTTP transaction, and something failed before a complete response was received.

Each fetch_failed is a potential client-facing 503 unless grace or stale content covers the gap. At low rates, grace masks the problem. At higher rates, users see errors and cache effectiveness drops because failed fetches produce nothing to store.

What this means

MAIN.fetch_failed counts backend fetch operations that failed after the TCP connection was established. This is distinct from MAIN.backend_fail, which counts connection-level failures (TCP refused, timeout during connect, network unreachable). With fetch_failed, the TCP handshake completed. The failure happened during the HTTP response phase.

Related counters that help isolate the cause:

MAIN.fetch_no_thread: fetch failed because no worker thread was available to handle the backend response. Thread starvation affecting backend fetches specifically.
MAIN.bgfetch_no_thread: background fetch (a grace refresh) failed due to thread starvation. These are asynchronous fetches for stale objects.

The diagnostic key is the FetchError tag in varnishlog. This tag carries a human-readable string describing the specific failure. Without it, fetch_failed tells you “something broke” but not what. The counters fetch_eof, fetch_length, and fetch_chunked track successful body transfers by framing type, giving you a baseline for comparison. fetch_bad counts responses with bad or unknown framing.

A ratio of fetch_failed / backend_req > 0.01 (more than 1% of backend fetches failing) is significant. At that rate, users are regularly hitting errors unless grace covers them.

flowchart TD
    A["Varnish fetches from backend"] --> B{TCP connect succeeded?}
    B -- No --> C["backend_fail (connection-level)"]
    B -- Yes --> D{HTTP response complete and valid?}
    D -- No --> E["fetch_failed (transaction-level)"]
    E --> F["FetchError in varnishlog gives the reason"]
    D -- Yes --> G[Object cached and served]
    F --> H{Grace or stale available?}
    H -- Yes --> I[Stale content served to client]
    H -- No --> J[Client receives 503]

Common causes

Cause	What it looks like	First thing to check
Backend premature close	FetchError references premature close or EOF before complete response	Backend keepalive settings, connection recycling on the origin
Content-Length mismatch	FetchError references insufficient bytes or body length mismatch	Backend gzip or mod_deflate interaction with dynamic content
HTTP header limits exceeded	FetchError “http format error” or “overflow”	`http_max_hdr`, `http_resp_hdr_len`, `http_resp_size` parameters
Gzip/gunzip failure	FetchError references gzip, gunzip, or TestGunzip	`beresp.do_gzip` / `beresp.do_gunzip` in `vcl_backend_response`
Chunked encoding error	FetchError references chunked framing	Backend response framing, proxy layers between Varnish and origin
Thread starvation	`fetch_no_thread` incrementing alongside `fetch_failed`	Thread pool saturation metrics
Timeout between bytes	FetchError references timeout after headers received	`between_bytes_timeout` parameter

Quick checks

# Check the fetch_failed rate and related counters
varnishstat -1 -f MAIN.fetch_failed -f MAIN.fetch_no_thread -f MAIN.bgfetch_no_thread

# Check the ratio denominator (backend requests)
varnishstat -1 -f MAIN.backend_req

# Get the specific FetchError strings from recent transactions
varnishlog -g request -q 'FetchError'

# Filter for transactions hitting vcl_backend_error
varnishlog -g request -q "VCL_call eq 'BACKEND_ERROR'"

# Check for 503s being returned to clients
varnishlog -m TxStatus:503

# Check backend workspace overflow (can cause fetch failures)
varnishstat -1 -f 'MAIN.ws_*_overflow'

# Check backend connection health and reuse ratios
varnishstat -1 -f MAIN.backend_fail -f MAIN.backend_busy -f MAIN.backend_conn -f MAIN.backend_reuse

# Check synthetic response rate (Varnish-generated error pages)
varnishstat -1 -f MAIN.s_synth

How to diagnose it

Confirm the failure rate. Take two readings of fetch_failed and backend_req 10 seconds apart. Compute delta(fetch_failed) / delta(backend_req). If the ratio is above 0.01, you have a real problem, not noise.
Capture the FetchError string. Run varnishlog -g request -q 'FetchError' during the failure window. The FetchError tag appears in backend fetch transactions and names the specific failure. Collect several samples because different URLs or backends may produce different errors.
Check for thread starvation. If fetch_no_thread is incrementing alongside fetch_failed, the root cause is thread pool exhaustion, not a backend protocol issue. Check MAIN.threads, MAIN.thread_queue_len, and MAIN.threads_limited. See Varnish thread pool exhaustion for the full diagnostic path.
Correlate with backend connection behavior. A backend that aggressively closes keepalive connections can cause premature close errors. Check the connection reuse ratio: backend_reuse / (backend_reuse + backend_conn). Low reuse with high fetch_failed suggests the backend is resetting connections that Varnish expected to be reusable.
Check workspace overflow. If ws_backend_overflow is incrementing, the backend response headers are exceeding workspace_backend. This is a configuration issue, not a backend bug.
Isolate by backend. If you have multiple backends in a director, check per-backend health with varnishadm backend.list -p. A single misbehaving backend can produce all the fetch failures while others are clean.
Check for recent VCL or backend changes. A new beresp.do_gunzip directive, a backend deploy that changed response headers, or a proxy layer that strips or alters Content-Length can all trigger fetch failures that were not present before.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`MAIN.fetch_failed` rate	Core counter for this failure mode	Sustained nonzero rate, especially `fetch_failed / backend_req > 0.01`
`MAIN.fetch_no_thread`	Thread starvation preventing fetch dispatch	Any nonzero rate
`MAIN.bgfetch_no_thread`	Background grace refreshes failing silently	Any nonzero rate
FetchError tag in varnishlog	Specific failure reason, the diagnostic key	Any new or recurring error string
`MAIN.backend_fail`	Connection-level failures, distinct from fetch failures	If nonzero alongside fetch_failed, connectivity and transaction issues coexist
`MAIN.ws_backend_overflow`	Backend workspace exhaustion causing fetch failures	Any nonzero rate
`MAIN.s_synth`	Synthetic responses, typically 503s from failed fetches	Spike correlating with fetch_failed
`MAIN.backend_reuse` ratio	Low reuse indicates connection churn	`backend_reuse / (backend_reuse + backend_conn) < 0.5` with rising fetch_failed
`MAIN.cache_hit_grace`	Grace serving is masking the backend problem	Elevated rate while fetch_failed is nonzero

Fixes

Backend premature close

The backend closed the TCP connection before sending a complete HTTP response. This happens when the backend terminates keepalive connections aggressively, or when a proxy layer between Varnish and the origin times out.

Check the backend_idle_timeout parameter in Varnish (default 60 seconds). If the backend closes idle connections faster than this value, Varnish will attempt to reuse a connection that the backend has already closed, resulting in a premature close. Reduce backend_idle_timeout to be shorter than the backend’s own keepalive timeout.

If the backend itself is closing connections mid-response due to resource pressure (memory limits, worker timeouts), that is a backend problem, not a Varnish tuning problem. Check backend logs for worker process exits or memory limits.

Content-Length mismatch

The backend advertises a Content-Length header claiming N bytes but sends fewer (or more). This is common with backend gzip compression modules that compress the body after computing Content-Length, or with FastCGI handlers that miscalculate body length for compressed responses.

If the backend is applying gzip compression and sending a Content-Length that does not match the compressed body, consider disabling gzip at the backend and letting Varnish handle compression instead via set beresp.do_gzip = true; in vcl_backend_response. Varnish’s compression is deterministic and will not produce length mismatches.

Alternatively, return (pipe) in vcl_recv for the affected request paths bypasses Varnish’s fetch logic entirely, passing the response through without parsing. This avoids the fetch failure but also means the response is not cached.

HTTP header limits exceeded

Varnish enforces limits on response header count and size. The defaults are:

http_max_hdr: 64 headers
http_resp_hdr_len: 8 KB per header
http_resp_size: 32 KB total response header size

Exceeding these produces FetchError http format error or FetchError overflow. If the backend legitimately sends many headers (deep proxy chains, extensive tracking headers), increase the relevant parameter via varnishadm param.set. Each increase costs workspace memory per request, so raise only what you need.

Check MAIN.losthdr as well. If headers are being dropped before the fetch fails, losthdr will be nonzero.

Gzip/gunzip failure

If VCL instructs Varnish to decompress (beresp.do_gunzip = true) or compress (beresp.do_gzip = true) a backend response, and the response body is corrupt or truncated, the compression or decompression will fail mid-stream. This is common when the backend sends Content-Encoding: gzip with a zero-byte or incomplete body.

Stop the gzip/gunzip processing for the affected responses. If the backend already sends compressed content, let it pass through without beresp.do_gunzip. If Varnish is compressing (beresp.do_gzip) and the backend body is incomplete, the underlying issue is a Content-Length mismatch or premature close, not the compression itself.

Thread starvation

If fetch_no_thread is incrementing, the thread pool cannot dispatch backend fetches. This is a thread pool sizing problem, not a backend protocol problem. Increase thread_pool_max or reduce the load holding threads. See Varnish thread pool tuning for the parameter guide and Varnish thread pool exhaustion for the diagnostic flow.

Timeout between bytes

If the backend sends response headers promptly but then stalls during body transfer, between_bytes_timeout fires and the fetch fails. Backends with slow streaming responses (large database exports, slow file generation) can trip it legitimately.

Increase between_bytes_timeout if the backend legitimately produces responses with long pauses between bytes. But investigate the backend first: a backend that stalls mid-body is usually a sign of resource contention, not a timeout tuning problem.

Prevention

Configure grace. Without beresp.grace (or the grace parameter), every fetch_failed immediately becomes a client-visible 503. With grace, Varnish serves a stale cached object while the fetch fails, and the user sees nothing wrong. Grace is the single most effective mitigation for intermittent fetch failures. The tradeoff is serving potentially stale content for the grace duration.

Monitor the fetch_failed to backend_req ratio. A sustained ratio above 0.01 is the threshold for investigation. Track this as a trend, not just an alert. Gradually rising ratios often indicate backend degradation before probes catch it.

Run a persistent log consumer. The FetchError string is the diagnostic key, but it only exists in the shared memory log. If no varnishlog or varnishncsa process is writing to persistent storage, the error details are overwritten and lost. Always run at least one persistent log consumer.

Audit backend compression. If the backend applies gzip and sends Content-Length headers, verify the lengths are accurate. The most common pattern is a backend that computes Content-Length before compression, then compresses the body, producing a mismatch.

Check backend keepalive compatibility. If the backend or any proxy layer between Varnish and the origin closes idle connections faster than Varnish’s backend_idle_timeout, Varnish will reuse dead connections and see premature close errors. Align the timeout values.

How Netdata helps

The varnish.fetch_failed metric from the Netdata Varnish collector provides per-second visibility into the fetch failure rate without manual varnishstat polling. Correlate it with varnish.backend_req for the failure ratio.
Cross-referencing fetch_failed with fetch_no_thread and thread pool saturation metrics in a single dashboard distinguishes backend protocol errors from thread starvation.
The varnish.s_synth metric tracks synthetic error responses. When it spikes alongside fetch_failed, the failures are reaching clients as 503s, not masked by grace.
varnish.backend_fail and varnish.backend_busy plotted alongside fetch_failed help determine whether the problem is connectivity-level, capacity-level, or transaction-level.
Workspace overflow counters (ws_backend_overflow, ws_client_overflow) alongside fetch failures reveal whether backend response headers are exceeding workspace allocation.
Netdata anomaly detection on the fetch_failed rate can surface a gradual increase before it crosses the 0.01 ratio threshold.