Varnish Error 503 Backend fetch failed: what the error page actually means

The “Error 503 Backend fetch failed” page is the default synthetic error Varnish serves when it cannot get a usable response from any backend. It displays “Guru Meditation” and an XID identifier, but nothing about the actual failure cause. The error page is a symptom, not a diagnosis.

The root cause is always in the FetchError tag in the shared memory log. Every Varnish-synthesised 503 is preceded by a backend transaction that logged a specific FetchError string: a timeout, a premature close, a protocol violation, or a health-probe failure that left no healthy backend to try. Reading that string is the single most important diagnostic step, and the one most operators skip.

What this means

Varnish generates the “Error 503 Backend fetch failed” page in the vcl_backend_error subroutine. This runs when a backend fetch fails in a way Varnish cannot recover from: the connection was refused, the response timed out, the HTTP was malformed, or no healthy backend existed to receive the request. The built-in VCL produces the HTML page with the title and Guru Meditation heading, then delivers it to the client as a synthetic 503 response.

This is fundamentally different from a backend-originated 503. If the backend itself returns HTTP 503 as a complete, valid response, Varnish does not generate the synthetic error page. It passes the backend’s response through to the client (and may cache it, depending on VCL). The FetchError tag only appears for Varnish-synthesised 503s where the fetch failed at the communication layer.

How grace interacts with the 503 path. When a backend fetch fails and a stale object exists in cache, Varnish serves the stale object first if grace is configured in VCL. The client gets a 200 OK with valid but aged content. The 503 only appears once grace expires and no stale object remains. This means the 503 rate you observe may lag the actual backend failure by the duration of the grace window. MAIN.cache_hit_grace (Varnish 7+) tracks hits served from grace. A spike there with low or zero 503s means Varnish is masking a backend problem that will surface when grace runs out.

flowchart TD
    A["Client receives 503"] --> B{"FetchError in backend log?"}
    B -->|Yes| C["Varnish-synthesised 503"]
    B -->|No| D["Backend-originated 503\n- investigate origin"]
    C --> E{"FetchError string?"}
    E -->|unhealthy / no backend| F["All backends sick\n- check probes"]
    E -->|timeout / EOF / HTC idle| G["Backend slow or\ndropping connections"]
    E -->|http format error| H["Malformed HTTP\nfrom backend"]
    E -->|busy| I["max_connections\nreached"]
    E -->|overflow / workspace| J["Workspace\nexhausted"]

Common causes

Quick checks

Run these read-only commands to narrow the problem immediately.

# Get the specific FetchError string - the single most important check
varnishlog -b -q 'FetchError'

# Confirm whether the 503 is Varnish-synthesised (look for FetchError)
# or backend-originated (no FetchError, backend returned 503)
varnishlog -q 'RespStatus == 503' -g request

# Check backend health with probe details
varnishadm backend.list -p

# Check fetch failure, backend health, and busy counters
varnishstat -1 -f MAIN.fetch_failed -f MAIN.backend_fail -f MAIN.backend_unhealthy -f MAIN.backend_busy

# Check if grace is masking the backend problem
varnishstat -1 -f MAIN.cache_hit_grace -f MAIN.s_synth

# Check workspace overflow and lost headers
varnishstat -1 -f 'MAIN.ws_*_overflow' -f MAIN.losthdr

# Check thread starvation affecting backend fetches
varnishstat -1 -f MAIN.fetch_no_thread -f MAIN.bgfetch_no_thread

How to diagnose it

Step 1: Determine whether the 503 is Varnish-synthesised or backend-originated.

Filter varnishlog for 503 responses and look for a FetchError tag in the same request transaction:

varnishlog -q 'RespStatus == 503' -g request

If you see a FetchError line in the backend transaction, Varnish generated the 503 because the fetch failed. If you see a complete backend response with status 503 and no FetchError, the backend itself returned 503 and you need to investigate the origin, not Varnish.

Step 2: Read the FetchError string.

varnishlog -b -q 'FetchError'

The -b flag filters to backend transactions. The query matches any FetchError record. The output gives you the exact failure string, which maps directly to a cause. Common strings and their meanings:

• HTC idle: the backend accepted the TCP connection but did not send response headers within first_byte_timeout (default 60s). The backend is alive on TCP but too slow to respond.
• http first read error: EOF: the backend closed the connection before sending any response headers.
• http read error: EOF or HTC eof: the backend closed the connection mid-response, after headers but before the body completed. Often a backend crash, OOM kill, or keepalive mismatch.
• backend <name>: unhealthy: the health probe marked this backend sick. Varnish did not attempt the fetch.
• Director returned no backend or No backend: all backends in the director are sick. No fetch was possible.
• backend <name>: busy: the backend reached its configured max_connections limit.
• http format error: the backend sent a response that Varnish’s HTTP parser rejected. Check for HTTP/0.9 responses, missing headers, or oversized headers exceeding http_resp_hdr_len.
• overflow or out of workspace: the backend response exceeded available workspace memory.

Step 3: Check backend health.

varnishadm backend.list -p

Look at the probe status for each backend. The happy column shows how many recent probes succeeded within the window. If happy is below the configured threshold, the backend is sick. Verify the probe URL: a common mistake is a probe pointing at an endpoint that returns 404 or 500, which marks the backend sick even though the application works for real traffic.

Step 4: Assess grace runway.

If MAIN.cache_hit_grace is elevated but 503s are low, Varnish is serving stale content and the backend problem is deferred, not resolved. Check how long the grace window is in your VCL (beresp.grace value) to estimate when the 503s will start cascading. The keep parameter controls how long objects are retained after TTL and grace expire; once both are exhausted, Varnish has nothing to serve and falls through to vcl_backend_error.

Step 5: Check workspace and thread pool.

If the FetchError is overflow or out of workspace, check workspace overflow counters:

varnishstat -1 -f 'MAIN.ws_*_overflow' -f MAIN.losthdr

If MAIN.fetch_no_thread is incrementing, the thread pool cannot dispatch backend fetches. This is a thread pool saturation problem, not a backend problem:

varnishstat -1 -f MAIN.threads -f MAIN.thread_queue_len -f MAIN.threads_limited

Metrics and signals to monitor

Fixes

All backends sick

The backend health probe is failing. Verify the probe configuration. Check that the probe URL returns 200 from the backend’s perspective:

# Verify the probe endpoint returns 200 from the backend
curl -I http://<backend_host>:<backend_port><probe_url>

If the probe URL is wrong (returning 404, 500, or a redirect), fix the probe definition in VCL and reload. If the probe URL is correct but the backend is genuinely down, investigate the backend application and infrastructure.

Backend timeout after connect

The backend accepted the TCP connection but was too slow to send headers. The relevant VCL parameters are connect_timeout, first_byte_timeout, and between_bytes_timeout, set per-backend in the backend definition.

Increasing first_byte_timeout gives the backend more time, but the real fix is almost always backend performance. If the backend is slow because it is overloaded (cache miss storm, database lock), increasing the timeout just delays the failure. Investigate backend response time:

# Check backend TTFB from varnishlog timestamps
varnishlog -g request -i Timestamp -q 'BerespStatus gt 0'

Look at the Bereq to Beresp delta for backend time-to-first-byte.

Backend premature close

The backend closed the connection before completing the response. Common causes: backend OOM kill, backend crash during response generation, or keepalive mismatch where the backend closes idle connections faster than Varnish reuses them. Check backend error logs and system logs (dmesg, journalctl) for crashes or OOM events.

max_connections reached

If you configured .max_connections on the backend definition and traffic exceeds it, Varnish logs FetchError: backend ...: busy. Either raise the limit in the VCL backend definition or add more backend capacity. MAIN.backend_busy tracks this condition. Note that max_connections is not enabled by default; if you never set it, this is not your cause.

Workspace exhausted

Large response headers, excessive Set-Cookie headers, or complex VCL string operations can exhaust the per-request workspace. Increase the backend workspace:

# Increase backend workspace (runtime, does not persist across restart)
varnishadm param.set workspace_backend 128k

This takes effect for new connections. Increasing workspace raises per-connection memory usage, so calculate the impact against your maximum concurrent connection count. workspace_client (default 64k) is a separate parameter for the client side; large request headers or cookies exhaust that instead.

Thread pool exhaustion affecting fetches

If MAIN.fetch_no_thread is nonzero, the worker pool is saturated and cannot dispatch backend fetches. This is not a backend problem. Increase thread_pool_max or investigate what is holding worker threads (typically slow backend responses on cache miss paths). See the related guide on thread pool exhaustion for detailed tuning guidance.

Prevention

• Configure grace in VCL. Set beresp.grace in vcl_backend_response to a duration that covers typical backend recovery time. This turns sudden backend failures into deferred failures, giving you time to fix the backend before clients see 503s. Without grace, any backend hiccup immediately produces 503s for all cache misses.
• Persist varnishncsa logs to disk. The shared memory log is ephemeral. Without a log consumer writing to persistent storage, you lose the FetchError data you need for post-incident analysis. Always run varnishncsa writing to disk with rotation.
• Monitor backend health independently of client-facing metrics. Grace can mask backend failures for extended periods. If you only alert on 503 rate, you will not discover the backend is down until grace expires and 503s cascade.
• Audit health probe configuration. Verify that the probe URL exercises real application health, not just TCP reachability. A probe that checks a static endpoint can return “healthy” while the application is broken for real traffic. Verify the probe URL returns 200 from the backend itself.

How Netdata helps

Netdata collects the counters that distinguish a Varnish-synthesised 503 from other failure modes, at per-second resolution:

• MAIN.fetch_failed and MAIN.fetch_no_thread appear alongside backend connection counters (backend_fail, backend_unhealthy, backend_busy) on the same timeline, so you can correlate fetch failures with sick backends, connection failures, or thread starvation.
• MAIN.cache_hit_grace is tracked independently of error rates. A spike in grace hits with a stable 503 rate reveals that Varnish is masking a backend outage.
• MAIN.s_synth correlates with backend health signals, making it clear when synthetic responses are driven by backend unreachability versus VCL logic.
• MAIN.ws_backend_overflow and MAIN.losthdr are collected continuously, so workspace-related 503s do not require a manual varnishstat check to discover.
• Per-backend VBE.*.happy counters let you see which specific backend is failing its probes, not just an aggregate health summary.

Related guides

• Varnish cache hit ratio dropped: hit rate collapse and backend overload
• Varnish cache stampede: a popular object expires and the herd hits the backend
• Varnish cache_hitpass / cache_hitmiss climbing: uncacheable content bleeding to the backend
• How Varnish actually works in production: a mental model for operators
• Varnish monitoring checklist: the signals every production cache needs
• Varnish monitoring maturity model: from survival to expert
• Varnish pass vs miss: why s_pass and cache_miss are not the same thing
• Varnish sess_dropped vs req_dropped: HTTP/1 connection drops and HTTP/2 stream drops
• Varnish thread pool exhaustion: workers all busy, queue full, sessions dropped
• Varnish thread pool tuning: thread_pool_min, thread_pool_max, and thread_pools
• Varnish thread_queue_len above zero: requests waiting for a worker
• Varnish threads_failed: the OS refusing to create worker threads