$ guides / varnish / varnish-vcl-fail ▌

Operations Guides

Varnish vcl_fail: VCL runtime errors during request processing

MAIN.vcl_fail is incrementing. The VCL loaded and compiled successfully at vcl.load time, but something is breaking at runtime while real requests flow through the VCL subroutines.

The vcl_fail counter (Varnish 6.0 and later) counts failures that prevented VCL from completing. Each increment means a request could not finish its normal VCL lifecycle. On the client side, the request is diverted to vcl_synth with a 503 status. On the backend side, the fetch fails. In both cases the user gets an error response instead of the content they asked for.

The vcl_fail counter is a summary. It tells you VCL broke, but not why. The root cause lives in the shared memory log (VSL) under the VCL_Error tag, in workspace overflow counters, or in the specific VMOD that called VRT_fail(). Correlating vcl_fail with s_synth, ws_*_overflow, and recent VCL changes is the diagnostic path.

What this means

VCL executes through a sequence of subroutines during request processing: vcl_recv, vcl_hash, cache lookup, vcl_hit / vcl_miss / vcl_pass, then optionally vcl_backend_fetch / vcl_backend_response, and finally vcl_deliver. At any point, the VCL can hit a runtime error condition that prevents completion.

return(fail) can appear in any VCL subroutine, including inside VMODs. When a failure occurs on the client side (in vcl_recv, vcl_hash, vcl_hit, vcl_miss, vcl_pass, vcl_deliver, vcl_pipe, or vcl_purge), the request is sent to vcl_synth with a 503 status and reason “VCL failed”. When a failure occurs on the backend side (in vcl_backend_fetch, vcl_backend_response, or vcl_backend_error), the fetch fails. If return(fail) is called from within vcl_synth itself, Varnish sends a minimal 500 response and forces the connection closed.

flowchart TD
    A["VCL execution during request"] --> B{"Runtime error condition?"}
    B -- No --> C["Normal request flow"]
    B -- Yes --> D["MAIN.vcl_fail increments"]
    D --> E["Client side: diverted to vcl_synth 503"]
    D --> F["Backend side: fetch fails"]
    D --> G{"Severe VCL or VMOD bug?"}
    G -- Yes --> H["Child process panic"]
    G -- No --> I["Synthetic error to client"]
    E --> J["MAIN.s_synth increments"]

A vcl_fail increment is distinct from a VCL compilation error. Compilation errors happen at vcl.load time and prevent the VCL from being activated at all. Runtime VCL failures happen against already-loaded, successfully compiled VCL.

It is also distinct from backend_fail (TCP connection to backend failed) and fetch_failed (backend connected but the fetch broke). Those are backend communication problems. The backend might be perfectly healthy while VCL is failing.

Common causes

Cause	What it looks like	First thing to check
Workspace exhaustion	`ws_client_overflow` or `ws_backend_overflow` incrementing alongside `vcl_fail`. Large Cookie headers, many synthetic headers added by VCL.	`varnishstat -1 -f 'MAIN.ws_*_overflow'`
VMOD failure	A VMOD called `VRT_fail()` or hit an internal error. `VCL_Error` entries in the log reference the VMOD.	`varnishlog -q 'VCL_Error' -g request`
Illegal VCL operation	VCL attempted something not allowed in the current subroutine context, or `req.restarts` exceeded `max_restarts`.	`varnishlog -q 'VCL_Error' -g request`
ESI processing failure	`esi_errors` incrementing. ESI includes failing, sometimes cascading into VCL failures on sub-requests.	`varnishstat -1 -f MAIN.esi_errors`
Child panic (escalation)	`MGT.child_panic` incrementing. Severe VCL or VMOD bugs escalate beyond `vcl_fail` to a full child crash.	`varnishstat -1 -f MGT.child_panic`

Quick checks

Run these read-only checks to establish the scope of the problem.

# Check vcl_fail rate (take two readings 10 seconds apart, compute delta)
varnishstat -1 -f MAIN.vcl_fail

# Check for workspace overflows (the most common vcl_fail cause)
varnishstat -1 -f 'MAIN.ws_*_overflow' -f MAIN.losthdr

# Check synthetic response rate (vcl_fail produces synthetic responses)
varnishstat -1 -f MAIN.s_synth

# Check for child panics (severe escalation)
varnishstat -1 -f MGT.child_panic -f MGT.child_died -f MGT.child_start

# Check VCL state and loaded versions
varnishadm vcl.list

# Check loaded VCL count (multiple VCLs may indicate failed reloads)
varnishstat -1 -f MAIN.n_vcl -f MAIN.n_vcl_avail -f MAIN.n_vcl_discard

# Search the shared memory log for VCL_Error entries
varnishlog -q 'VCL_Error' -g request

# Check ESI errors if Edge Side Includes are in use
varnishstat -1 -f MAIN.esi_errors -f MAIN.esi_warnings

How to diagnose it

Confirm vcl_fail is actually incrementing. Take two varnishstat readings 10 seconds apart. Compute the delta. If the rate is zero, the failures may have been transient or already resolved.
Check workspace overflow counters. Run varnishstat -1 -f 'MAIN.ws_*_overflow'. If ws_client_overflow or ws_backend_overflow are incrementing, workspace exhaustion is the cause. This is the most common trigger for vcl_fail.
Search for VCL_Error log entries. Run varnishlog -q 'VCL_Error' -g request. The VCL_Error tag records the specific error string when return(fail) is called with an argument, or when a VMOD calls VRT_fail(). The error string tells you exactly what broke.
Correlate with s_synth. Run varnishstat -1 -f MAIN.s_synth. If s_synth is incrementing at a similar rate to vcl_fail, the failures are producing synthetic 503 responses to clients.
Check for recent VCL changes. Run varnishadm vcl.list. Look at timestamps. A recently activated VCL is the most likely culprit. If the VCL status is “available” (not “active”), it is not serving traffic and cannot cause runtime failures.
Check for child panics. Run varnishstat -1 -f MGT.child_panic. If panics are incrementing alongside vcl_fail, the VCL or VMOD bug is severe enough to crash the child process. Check for _.panic files in /var/lib/varnish/.
Identify the failing subroutine or VMOD. Use varnishlog -g request -q 'VCL_Error' and examine the surrounding transaction context. The log entry shows which VCL subroutine was executing when the failure occurred, and if a VMOD was involved, which VMOD function failed.
Check if max_restarts is the trigger. If VCL uses return(restart) and req.restarts exceeds max_restarts, control passes to vcl_synth with “Too many restarts”. Search for restart-related VCL_Error entries.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`MAIN.vcl_fail`	Direct counter of VCL runtime failures	Any nonzero sustained rate
`MAIN.s_synth`	Synthetic responses generated by Varnish, including those from vcl_fail	Spike correlating with vcl_fail
`MAIN.ws_client_overflow`	Client workspace exhausted during request processing	Any nonzero rate
`MAIN.ws_backend_overflow`	Backend workspace exhausted during fetch processing	Any nonzero rate
`MAIN.losthdr`	HTTP headers dropped due to limit exceeded	Any nonzero rate
`MGT.child_panic`	Severe VCL/VMOD bug escalating to child crash	Any increment
`MAIN.esi_errors`	ESI parse errors that may cascade into VCL failures	Any nonzero rate
`MAIN.n_vcl`	Number of loaded VCLs (failed reloads accumulate)	Unexpected growth

Fixes

Workspace exhaustion

If ws_client_overflow or ws_backend_overflow are incrementing, the per-request workspace is too small for the headers, cookies, or VCL-added data in your traffic. Large Cookie headers are the most common cause.

# Check current workspace sizes
varnishadm param.show workspace_client
varnishadm param.show workspace_backend

# Increase workspace (live change, no restart needed)
varnishadm param.set workspace_client 96k
varnishadm param.set workspace_backend 96k

The default is typically 64k for both. Increasing to 96k or 128k resolves most Cookie-driven overflows. The tradeoff is memory: workspace is allocated per session, so the cost scales with concurrent connections. On a system with 5000 worker threads, going from 64k to 128k adds roughly 320MB.

Make the change persistent by adding -p workspace_client=96k -p workspace_backend=96k to the Varnish startup parameters.

VMOD failure

If VCL_Error log entries point to a specific VMOD, the VMOD is calling VRT_fail() or encountering an internal error. Common scenarios: a VMOD making external calls (DNS, Redis, HTTP) hitting a timeout or connection error, or a VMOD receiving unexpected input data.

Identify the VMOD from the VCL_Error message.
Check if the VMOD call is wrapped in error handling in VCL. If not, add one.
If the VMOD itself has a bug, check for an updated version.
As a stopgap, remove the VMOD call from the hot path and reload VCL.

Too many restarts

If VCL uses return(restart) and req.restarts exceeds max_restarts, the restart loop triggers vcl_fail.

varnishadm param.show max_restarts

The fix is to fix the VCL logic causing excessive restarts, not to raise max_restarts. A restart loop usually indicates a logic error where vcl_recv keeps redirecting back to itself.

If esi_errors is incrementing alongside vcl_fail, Edge Side Includes are breaking. ESI processing increases workspace pressure: each ESI sub-request runs its own VCL cycle, multiplying workspace and thread consumption.

See the related guide on Varnish ESI errors for detailed troubleshooting.

Severe bugs escalating to child panic

If MGT.child_panic is incrementing, the VCL or VMOD bug is crashing the child process entirely. The child restarts automatically under management process supervision, but each restart loses the entire cache.

Check for panic files: ls /var/lib/varnish/*/_.panic
Read the panic message to identify the failing component.
Roll back to the previous VCL if a recent change triggered it: varnishadm vcl.use <previous_vcl_name>
If a VMOD is causing the panic, remove the VMOD call from VCL and reload.

Prevention

Monitor vcl_fail continuously. It should be zero in steady state. Alert on sustained nonzero rate once MAIN.uptime > 300 (past warmup).
Monitor workspace overflow counters. ws_client_overflow and ws_backend_overflow are leading indicators. If they start incrementing before vcl_fail, you have early warning.
Test VCL changes before activation. Use varnishd -C -f your.vcl to compile-check VCL before loading. This catches compilation errors but not runtime errors. For runtime testing, load the new VCL without making it active, then use varnishadm vcl.use to activate after verification.
Track VCL versions. Use varnishadm vcl.list regularly. Accumulating loaded VCLs can indicate failed reload attempts or old VCLs not being discarded. Each loaded VCL consumes resources.
Size workspace for your traffic. If your application sets large Cookie headers, proactively increase workspace_client above the default. Monitor losthdr as well: if headers are being dropped, workspace pressure is already building.
Wrap VMOD calls with error handling. Any VMOD that can fail (network calls, external lookups) should have its return value checked in VCL.

How Netdata helps

Per-second resolution. Netdata collects MAIN.vcl_fail at one-second resolution, so you can pinpoint the exact moment failures start and correlate with deployments or traffic changes.
Workspace correlation. When vcl_fail spikes, overlay ws_client_overflow, ws_backend_overflow, and losthdr in the same dashboard. If workspace counters lead vcl_fail by even a few seconds, you have the root cause.
s_synth correlation. Overlaying MAIN.s_synth against vcl_fail confirms whether failures are reaching clients as synthetic 503 responses or whether vcl_synth is handling them gracefully.
Child panic escalation. Netdata monitors MGT.child_panic and MGT.child_died alongside MAIN.uptime vs MGT.uptime. If vcl_fail escalates to child panics, you see the crash-restart cycle immediately.
VCL lifecycle tracking. Changes in MAIN.n_vcl and MAIN.n_vcl_avail appear alongside vcl_fail rate. If a VCL reload precedes a spike, the new VCL is the likely trigger.