$ guides / varnish / varnish-grace-masking-backend-failure ▌

Operations Guides

Varnish grace masking a backend outage: the ticking-clock incident

Your origin servers went down 12 minutes ago. The dashboard shows a 97% cache hit rate, zero 503s, and normal response times.

This is grace masking. Varnish is serving cached objects past their TTL because no healthy backend can refresh them. Clients see stale 200s. Monitoring sees healthy cache traffic. The outage stays invisible until grace expires on enough popular objects, at which point 503s cascade.

Grace buys time proportional to the shortest grace TTL across your working set. When that runway runs out, the hit rate can drop from 97% to near zero in minutes. The impact is deferred, not prevented.

What this means

Grace mode lets Varnish serve a stale object while dispatching a background fetch to refresh it. When the backend is unreachable, the background fetch fails and Varnish keeps serving the stale object until its grace period expires. From the client side, the response is a valid 200 with an elevated Age header. From the monitoring side, the request counts as a cache_hit.

In Varnish 7.x, the builtin vcl_hit delivers an object when obj.ttl >= 0s (fresh) or when obj.ttl + obj.grace > 0s (stale but within grace). When TTL has expired but grace has not, the object is delivered as a grace hit and a background fetch is dispatched. If the backend is healthy, the refresh succeeds silently. If the backend is sick, the refresh fails and Varnish keeps serving the stale object, consuming the grace runway with each request.

Grace hits are included in the cache_hit counter, so standard client-facing signals stay green:

Cache hit ratio holds steady or drops only slightly.
Client 503 rate stays at zero.
Response latency stays low, because Varnish is serving from memory.

Only signals that measure backend state independently reveal the truth.

There is a second mechanism that can shorten the runway. When a background fetch receives a 5xx response from the backend (as opposed to a connection timeout, which fails without a response), the builtin vcl_backend_response may store that error response, overwriting the graced object in cache. For the TTL of that error object (commonly 120 seconds, depending on default_ttl and response headers), Varnish serves the error instead of the stale 200. This creates a 503 window embedded inside the grace period, appearing before you would expect grace to expire.

flowchart TD
    A[Backend outage] --> B[Probes mark backends sick]
    B --> C[Grace serves stale objects]
    C --> D[Client metrics: hit rate stable, zero 503s]
    C --> E[Backend signals: VBE happy drops, backend_unhealthy climbs]
    D --> F{Backend recovers before grace expires?}
    F -->|Yes| G[Background fetches refresh objects]
    F -->|No| H[Grace expires on popular objects]
    H --> I[503 cascade begins]

Common causes

The backend outage itself can have many root causes. What matters operationally is that grace masks all of them equally, and the fix for the masking problem is the same regardless of the underlying failure.

Cause	What it looks like	First thing to check
Backend application crash or deploy failure	`VBE.*.happy` drops to zero across all backends; `backend_unhealthy` increments	`varnishadm backend.list -p` for probe details
Network partition between Varnish and backend	`backend_fail` increments (TCP connect failures); probes time out	Network connectivity from Varnish host to backend port
Backend overload causing probe timeouts	Backend slow but eventually responds; probes fail intermittently	Backend response time and load metrics
5xx overwrite during outage	503s for specific objects while grace should still cover them	Whether 5xx background fetch responses overwrite graced objects

Quick checks

Run these read-only commands to confirm whether grace is masking a backend outage.

# Check backend health with probe details
varnishadm backend.list -p

# Check grace-serving activity (V7+)
varnishstat -1 -f MAIN.cache_hit_grace

# Check backend sick counters (connections not attempted)
varnishstat -1 -f MAIN.backend_unhealthy -f MAIN.backend_fail

# Check per-backend probe success counts
varnishstat -1 -f 'VBE.*.happy'

# Check object expiry rate (declining rate signals objects surviving via grace)
varnishstat -1 -f MAIN.n_expired

# Check background fetch thread failures (V7+)
varnishstat -1 -f MAIN.bgfetch_no_thread

# Check synthetic response rate (503s generated by Varnish)
varnishstat -1 -f MAIN.s_synth

# Inspect object TTL and grace values to estimate runway
varnishlog -i TTL

How to diagnose it

Confirm backend health independently. Run varnishadm backend.list -p and check whether backends show probe failures. Look at VBE.*.happy values: if happy is below the probe threshold, the backend is sick. Check MAIN.backend_unhealthy to see how many connections Varnish skipped because backends were marked unhealthy. A quiet backend_fail counter does not mean the backend is fine. backend_unhealthy counts connections that were never attempted, so a completely offline backend shows zero backend_fail but climbing backend_unhealthy.
Check if grace is actively serving. On Varnish 7+, check MAIN.cache_hit_grace. If this counter is climbing while backends are sick, grace is masking the outage. Each grace hit is also counted in cache_hit, which is why hit ratio looks normal.
Check the natural expiry rate. MAIN.n_expired counts objects expiring from cache by TTL. During a grace-masking event, this rate declines because objects that would normally expire are kept alive by grace serving. A declining n_expired rate alongside sick backends is a tell.
Estimate the grace runway. Run varnishlog -i TTL to inspect the TTL and grace values of objects being served. The output shows remaining TTL, grace, and keep for each object. The shortest grace values across your popular objects define how long the runway lasts. If most objects have a 30-second grace and the backend has been down for 25 minutes, you are already past the cliff for those objects.
Check for 5xx overwrite. If you see 503s for specific objects while other objects still serve via grace, check whether background fetches receiving 5xx responses are overwriting graced objects. Look for patterns where 503s appear in bursts for specific URLs, then stop when the error object TTL expires and grace serving resumes.
Verify VCL grace configuration. Check whether your VCL sets beresp.grace explicitly or relies on the default_grace parameter (default: 10 seconds). A 10-second grace runway masks very little. Production setups typically set grace to several minutes or more depending on content type and staleness tolerance.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`VBE.*.happy`	Per-backend probe success count within the window	Drops below probe threshold means backend is sick
`MAIN.backend_unhealthy`	Connections not attempted because backend is sick	Sustained rate above zero means backends marked sick
`MAIN.cache_hit_grace` (V7+)	Objects served from grace rather than fresh cache	Climbing while backends are sick means grace masking active
`MAIN.n_expired`	Objects expiring naturally from cache by TTL	Declining rate means objects surviving past TTL via grace
`MAIN.backend_fail`	TCP connection failures to backends	Sustained rate above zero means backend unreachable
`MAIN.bgfetch_no_thread` (V7+)	Background fetches that failed for lack of threads	Rate above zero means thread starvation preventing refreshes
`MAIN.s_synth`	Synthetic responses, typically 503 error pages	Spiking means grace exhausted and Varnish is generating errors

Fixes

Add return(abandon) for 5xx background fetches

The single most important VCL change to prevent the 5xx overwrite problem:

sub vcl_backend_response {
    if (beresp.status >= 500 && bereq.is_bgfetch) {
        return (abandon);
    }
}

When a background fetch gets a 5xx from the backend, return (abandon) discards the response without storing it. The graced object stays in cache and Varnish continues serving it. Without this, the error response can overwrite the graced object, forcing Varnish to serve the error for that object’s TTL.

This is the standard pattern recommended in the Varnish grace documentation.

Configure dynamic grace based on backend health

Set req.grace in vcl_recv to use a short grace when the backend is healthy and a longer grace when it is sick:

sub vcl_recv {
    if (req.backend.healthy) {
        set req.grace = 10s;
    } else {
        set req.grace = 24h;
    }
}

When the backend is healthy, a short grace is enough for background fetches to complete. When the backend is sick, the full grace value applies, giving Varnish a longer runway. The grace runway extends automatically when backends fail.

The tradeoff: longer grace means serving staler content during transient backend issues. Set the sick-backend grace based on the maximum staleness your application can tolerate.

Extend grace during an active incident

If a backend outage is detected and the estimated grace runway is shorter than the expected recovery time, reload VCL with extended grace values.

Use varnishadm vcl.load and varnishadm vcl.use to load and activate the new VCL without dropping connections. The grace extension applies to objects fetched after the VCL change. Objects already in cache retain their original grace values, so this extends the runway for future requests but does not retroactively save objects whose grace has already expired.

Set appropriate default grace values

The default_grace parameter defaults to 10 seconds. Set beresp.grace explicitly in vcl_backend_response based on your application’s staleness tolerance. Common production values range from 2 minutes to several hours, depending on content type. Static assets can tolerate long grace periods. Personalized or time-sensitive content needs shorter grace or none at all.

Prevention

Monitor backend health independently of client-facing metrics. If you only alert on hit ratio, 503 rate, and response latency, you will not detect a grace-masking event until it is too late. Alert on VBE.*.happy and MAIN.backend_unhealthy directly.
Alert on cache_hit_grace climbing (V7+). Grace serving is normal in small amounts during background refreshes. A sustained climb while backends are healthy can indicate intermittent backend issues. A climb while backends are sick is the ticking clock.
Include the abandon pattern in all production VCL. return (abandon) for 5xx background fetches prevents the overwrite that creates embedded 503 windows during outages.
Track grace runway as a capacity metric. Know the distribution of grace TTLs across your working set. If your shortest grace values are 30 seconds, a backend outage longer than 30 seconds will produce user-visible failures. Size grace values to cover realistic backend recovery times.
Audit health probe configuration. A probe testing a static endpoint that always returns 200 will report healthy even when the application is broken. Ensure probes test real application health. Check the probe threshold, window, and interval to understand how quickly Varnish detects failures.

How Netdata helps

Netdata collects the backend-independent signals that reveal grace masking, with per-second resolution.

Backend health. Netdata monitors VBE.*.happy per backend and MAIN.backend_unhealthy rate, so you see backends go sick immediately, independent of what the client-facing hit ratio shows.
Grace serving. The MAIN.cache_hit_grace counter (V7+) is collected natively. A spike in grace hits while backends are sick is the earliest warning that the clock is ticking.
Expiry rate correlation. MAIN.n_expired is tracked alongside MAIN.n_lru_nuked, letting you distinguish natural TTL expiry from grace-extended survival. A declining n_expired rate during a backend outage confirms grace masking.
Background fetch failures. MAIN.bgfetch_no_thread and MAIN.fetch_failed are monitored, catching the thread starvation that prevents grace refreshes from completing.
Anomaly detection. ML-based anomaly detection flags the correlation of sick backends plus stable hit ratio plus rising grace hits as abnormal, even when no individual static threshold is breached.