Varnish Guru Meditation: reading the XID and tracing the failing request

The Guru Meditation page is Varnish’s default 503 response. It fires when Varnish cannot serve from cache and cannot fetch from a backend. The page carries a transaction ID (XID) that maps directly to the shared memory log entries for that failed request. The XID is the fastest path from “users are seeing 503s” to “here is the FetchError, the backend, and the VCL subroutine where the failure happened.”

The core procedure works on Varnish 5.1 and later. The vxid field was added to the VSL query language in Varnish 5.1, making it possible to filter by transaction ID directly with -q. On earlier versions, you cannot filter by XID in the query and must rely on timestamp correlation or piping varnishlog through grep.

Where the 503 originates

Two built-in VCL subroutines generate the Guru Meditation page, depending on where the failure occurred:

• vcl_synth handles client-side synthetic errors: all backends are sick with no grace available, VCL explicitly calls return(synth(503)), or the request was blocked by VCL rules.
• vcl_backend_error handles backend fetch failures: connection refused, timeout, truncated response, or protocol error.

Both render the same HTML template and both include the XID. The distinction that matters for tracing is which XID appears on the page:

• In vcl_synth, the page uses req.xid (the client request transaction ID).
• In vcl_backend_error, the page uses bereq.xid (the backend fetch transaction ID).

If the 503 came from a backend fetch failure, the XID on the page is the backend transaction ID, not the client transaction ID. Both exist in the shared memory log, but only the one printed on the page leads directly to the FetchError.

flowchart TD
    A["Error 503 page
XID: 987654"] --> B["varnishlog -g request
-q 'vxid == 987654'"]
    B --> C{"FetchError tag
present?"}
    C -->|Yes| D["Backend fetch failed
path: vcl_backend_error"]
    C -->|No| E["Client-side synth
path: vcl_synth"]
    D --> F["Check backend.list
and backend_fail rate"]
    E --> G["Check vcl_recv logic
and all-backends-sick state"]

Where to find the XID

The XID is available from three sources:

1. The Guru Meditation HTML page, under the error text. The line reads XID: NNNNNN.
2. The X-Varnish response header. The first number is the current request’s vxid. On a cache hit, a second number appears, which is the vxid of the cached object that was served.
3. varnishlog output, in the vxid field of each transaction record.

If the XID is null or missing, the request failed before Varnish assigned a transaction ID. This is rare and typically indicates a very early failure in request processing.

Quick checks

These commands are read-only and safe to run at any time.

# Count 503 responses over a 60-second window
timeout 60 varnishncsa -F '%s' 2>/dev/null | grep -c '^503$'

# Check backend health with probe details
varnishadm backend.list -p

# Check synthetic response rate (includes 503 pages)
varnishstat -1 -f MAIN.s_synth

# Check fetch failure rate
varnishstat -1 -f MAIN.fetch_failed

# Check backend connection failures
varnishstat -1 -f MAIN.backend_fail -f MAIN.backend_unhealthy

Trace the failing request

Step 1: extract the XID

Copy the numeric value from the XID: line on the Guru Meditation page. If you can reproduce the error, curl -I against the failing URL returns the X-Varnish response header, whose first number carries the same vxid.

Step 2: query varnishlog by vxid

# Trace a specific transaction by XID
varnishlog -g request -q 'vxid == NNNNNN'

Replace NNNNNN with the XID from the error page. The -g request flag groups output so the full request lifecycle appears as one block, with the linked backend fetch (if any) indented underneath. You will see the client request, the VCL subroutine calls, the backend fetch attempt, and the error.

Step 3: find the FetchError

Within the grouped output, look for the FetchError tag. This tag appears in the backend fetch transaction (BeReq) and contains the human-readable reason the fetch failed. Common examples:

• backend boot.default: fail errno 111 (Connection refused) means the backend refused the TCP connection.
• backend closed means the peer closed while Varnish was reading the response headers. A body that ends early is logged as straight insufficient bytes or chunked insufficient bytes.
• overflow means the backend response headers exceeded workspace_backend.

The FetchError tag has been available since Varnish Cache 2.1.1, whose changes added a log tag “which tries to explain why we could not fetch an object from the backend.” On very old versions, infer the failure from backend fetch timestamps and the absence of a BerespStatus tag.

Step 4: identify the VCL path

The grouped output also shows which VCL subroutines executed. Look for VCL_call entries:

• VCL_call: BACKEND_ERROR means the failure was in the backend fetch path.
• VCL_call: SYNTH means the failure was client-side.

This tells you whether to investigate the backend infrastructure or the VCL logic itself.

Step 5: check backend state

Cross-reference the FetchError with backend state:

# Per-backend health with probe details
varnishadm backend.list -p

# See which backend and URL the failing request used
varnishlog -g request -q 'vxid == NNNNNN' -i BackendOpen -i BereqURL

Common FetchError patterns

Live capture during an incident

If you do not have a specific XID but need to catch failing requests as they happen, filter for the VCL call that generates the error page:

# Catch backend fetch errors live
varnishlog -g request -q "VCL_call eq 'BACKEND_ERROR'"

# Catch client-side synthetic errors live
varnishlog -g request -q "VCL_call eq 'SYNTH'"

# Catch all 503 responses
varnishlog -g request -q 'RespStatus == 503'

Run these in a terminal during the incident window. They are useful when errors are intermittent or you cannot reproduce them.

Customizing the error page

You can override the default Guru Meditation page by defining your own vcl_synth and/or vcl_backend_error in your VCL file. Your custom subroutine replaces the built-in one for matching status codes. This split replaced the old single vcl_error subroutine starting in Varnish 4.0.

A minimal custom vcl_synth that preserves the XID for support tickets:

sub vcl_synth {
    if (resp.status == 503) {
        synthetic({"<!DOCTYPE html>
<html><body>
<h1>Service temporarily unavailable</h1>
<p>Reference: "} + req.xid + {"</p>
</body></html>"});
        return(deliver);
    }
}

Use req.xid in vcl_synth and bereq.xid in vcl_backend_error if you want to keep the reference number for diagnosis.

The vcl_backend_error caching trap

Objects generated by vcl_backend_error can end up in the cache. This is unlike vcl_synth, whose responses are never stored. Before vcl_backend_error runs, Varnish creates its synthetic backend-error object with TTL, grace, and keep set to zero; it does not automatically mark that object uncacheable. If your custom vcl_backend_error raises the TTL and returns deliver, the error page can be cached and served to subsequent clients hitting the same URL.

To prevent this:

sub vcl_backend_error {
    set beresp.ttl = 0s;
    set beresp.uncacheable = true;
}

If users report Guru Meditation pages for URLs that should work, check whether a backend error was cached during a transient outage and is now being served from cache. A varnishlog -g request -q 'RespStatus == 503' capture during the issue will show whether the response is a cache hit or a fresh fetch.

Metrics and signals to monitor

Prevention

Configure grace mode. Without grace, any backend hiccup immediately produces 503s for cache misses. Grace lets Varnish serve stale content while the backend recovers. Set beresp.grace in vcl_backend_response to a duration that covers typical backend recovery time.

Monitor backend health independently of client-facing metrics. Grace mode can mask backend failures completely until the grace window expires. Do not rely on 503 rate alone to detect backend problems. A sick backend with backend_unhealthy incrementing but grace still serving is a ticking clock.

Keep varnishncsa logging to persistent storage. The shared memory log is a circular buffer by design. Without a persistent log consumer, you lose the ability to trace specific XIDs after the fact. During high-traffic incidents, the buffer wraps faster, and the transactions you need are overwritten first.

Size workspace_backend correctly. The default may be too small for backends that send large response headers. The overflow FetchError indicates workspace exhaustion. Increase it via the workspace_backend runtime parameter.

Correlating with monitoring

At per-second resolution, the signals above tell you whether 503s are a backend problem or a VCL problem before you open varnishlog:

• MAIN.s_synth rate spiking correlates directly with user-visible Guru Meditation pages.
• MAIN.fetch_failed and MAIN.backend_fail rates show whether the 503s are backend connectivity or fetch errors, narrowing the investigation.
• Per-backend VBE.*.happy counters reveal which specific backend is failing its health probes, often before the fetch failures that produce 503s.
• MGT.child_panic indicates child process crashes, which can produce brief 503 bursts during restart cycles when the cache is cold.
• Anomaly detection on these signals catches subtle increases in synthetic response rate or fetch failures before they reach user-visible severity.
• Correlating s_synth spikes with backend_fail or backend_unhealthy changes in the same time window confirms whether the root cause is backend health, reducing the number of XIDs you need to trace manually.

Related guides

• How Varnish actually works in production: a mental model for operators
• Varnish thread pool exhaustion: workers all busy, queue full, sessions dropped
• Varnish sess_dropped vs req_dropped: HTTP/1 connection drops and HTTP/2 stream drops
• Varnish monitoring checklist: the signals every production cache needs
• Varnish cache hit ratio dropped: hit rate collapse and backend overload
• Varnish pass vs miss: why s_pass and cache_miss are not the same thing