Varnish backend is sick: health probes, all-backends-sick, and grace

A sick Varnish backend receives zero traffic. When all backends in a director go sick, Varnish either serves stale content via grace or returns 503 on every cache miss. The health transition is governed by a threshold/window probe model, not a simple pass/fail. That distinction matters when you are reading backend.list output during an incident.

What this means

Varnish marks a backend healthy or sick based on a sliding window of probe results. A probe is an HTTP request Varnish sends to the backend at a configurable interval. The backend is healthy if at least threshold out of the last window probes returned the expected response (default 200) within the timeout. The defaults are stable across Varnish 6.x through 9.x:

• .window = 8: Varnish tracks the last 8 probe results.
• .threshold = 3: at least 3 of the last 8 must succeed.
• .interval = 5s: probes are sent every 5 seconds.
• .timeout = 2s: a probe that does not respond within 2 seconds fails.
• .expected_response = 200: only HTTP 200 counts as success.
• .initial: assumed successes before real probes complete. The default is threshold - 1, so with the default threshold of 3 the backend starts unhealthy until the first successful probe response arrives.

VBE.<name>.happy is not a boolean. It is the count of successful probes within the current window, ranging from 0 to .window. A backend with happy = 3 and threshold = 3 is healthy. A backend with happy = 2 and threshold = 3 is sick. happy = 0 means every recent probe failed, but happy = 2 with threshold = 3 is also sick and easy to miss if you are scanning for zeros.

When a backend is sick, directors skip it entirely. The counter MAIN.backend_unhealthy increments for each fetch not attempted because the backend was sick. This is distinct from MAIN.backend_fail, which counts backend fetches that were attempted and failed (connection errors, timeouts during transfer, and similar). A backend can be completely offline with backend_fail = 0 because Varnish never tried to connect: the probe already told it not to.

Common causes

Quick checks

# Check current backend health with probe details
varnishadm backend.list -p

# Check per-backend probe success counts (happy = successes in window)
varnishstat -1 -f 'VBE.*.happy'

# Check connections not attempted due to sick backends
varnishstat -1 -f MAIN.backend_unhealthy

# Check connection-level failures (fetch attempted, failed)
varnishstat -1 -f MAIN.backend_fail

# Watch real-time health transitions
varnishlog -g raw -i Backend_health

# Check grace-served hits - if sustained, backends are down but masked
varnishstat -1 -f MAIN.cache_hit_grace

# Check synthetic responses (typically 503 when all backends sick)
varnishstat -1 -f MAIN.s_synth

# Verify the probe URL from the Varnish host
curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' http://<backend>:<port>/

# List loaded VCLs to check for recent changes
varnishadm vcl.list

How to diagnose it

flowchart TD
    A["backend.list shows sick"] --> B{"happy < threshold?"}
    B -->|"happy = 0"| C["curl probe URL from Varnish host"]
    B -->|"happy > 0 but < threshold"| D["Intermittent probe failures"]
    C --> E{"curl returns 200?"}
    E -->|"No"| F["Fix probe URL or expected_response"]
    E -->|"Yes, but slow"| G["Check probe timeout vs backend TTFB"]
    E -->|"Yes, fast"| H["Network or probe config issue"]
    D --> I["Check backend load and response time"]
    F --> J["Reload VCL"]
    G --> K["Increase .timeout or fix backend"]
    H --> L["Verify backend IP and port in VCL"]
    I --> M["Scale backend or adjust threshold and window"]

1. Read backend.list -p output carefully. The -p flag shows the probe state in detail, including the shift register of recent probe results. A row of dashes or zeros means probes are failing. Check whether happy is 0 (all probes failing) or a low number (intermittent failures).

2. Test the probe URL from the Varnish host. Run the exact URL, port, and Host header the probe uses. If the backend responds 301 or 302, the probe sees a non-200 and marks the backend sick. If the backend responds 200 but the body contains an error page, the probe reports healthy while the application is broken.

3. Distinguish backend_unhealthy from backend_fail. backend_unhealthy means Varnish did not attempt a connection because the probe window says sick. backend_fail means Varnish attempted a fetch and it failed. If backend_unhealthy is incrementing but backend_fail is zero, the backend might be fine and the probe is misconfigured.

4. Check whether error responses are blocking grace. If all backends are sick and you have grace configured but clients get 503s instead of stale content, a cached error response from a background fetch may have overwritten the good object. When a background fetch returns 500 and that response gets cached with a TTL, it replaces the stale object grace would have served. Subsequent requests find the error object, not the good one. Check cache_hitpass rate and inspect your vcl_backend_response for how 5xx is handled.

5. Verify .initial after restart. If all backends show sick immediately after a Varnish restart and recover within 15-40 seconds, .initial is set lower than .threshold or explicitly to 0. Backends start unhealthy until enough real probes fill the window.

6. Watch health transitions in real time. varnishlog -g raw -i Backend_health shows messages like “Still sick”, “Back healthy”, and “Went sick”. This is the fastest way to see if probes are flapping or consistently failing.

Metrics and signals to monitor

Fixes

Probe returns non-200 (redirect, 401, 500)

Fix the probe URL or the expected_response in the probe definition. Common issues:

• The probe URL is / and the backend redirects to /login with 301 or 302. Set .url to a dedicated health endpoint that returns 200 directly.
• The backend requires authentication and returns 401 on the probe URL. Use an endpoint that does not require auth.
• The backend returns 200 on / but the application is broken. Point the probe at an endpoint that exercises real application logic, such as /healthcheck or /internal/status.

After changing the probe configuration, reload VCL. Varnish coalesces probes that appear identical across loaded VCLs, so discard old VCLs after reloading to avoid stale probe definitions.

Probes timing out on overloaded backend

If the backend is genuinely slow and probes time out at the default 2s timeout:

• Increase .timeout to give the backend more room. This delays the sick transition but also delays the healthy transition after recovery.
• Widen the window and lower the threshold to tolerate intermittent slowness. For example, .window = 10, .threshold = 5 tolerates more consecutive failures before marking sick.

Neither fix addresses the root cause. If the backend cannot respond to a lightweight probe within 2 seconds, it likely cannot serve real requests either. Fix the backend first.

Setting .initial to prevent post-restart false-sick

Set .initial explicitly in the probe definition to match threshold so the backend starts healthy immediately after VCL load:

probe myprobe {
    .url = "/healthcheck";
    .interval = 5s;
    .timeout = 2s;
    .window = 8;
    .threshold = 3;
    .initial = 3;
}

This eliminates the startup window where backends are marked sick before real probes complete. For backends created dynamically by VMODs, .initial must be set explicitly.

All backends sick with 503s despite grace configured

The most likely cause is cached error responses from background fetches overwriting the stale objects grace would serve. When a background fetch returns 500, the response can replace the cached good object with an error page. Subsequent requests find the error object, not the stale content.

Abandon backend fetches that return 5xx during background fetches:

sub vcl_backend_response {
    if (beresp.status >= 500 && bereq.is_bgfetch) {
        return (abandon);
    }
}

This tells Varnish to discard the error response from background fetches and keep the existing stale object, which grace can then serve. Apply this only for background fetches (bereq.is_bgfetch) so that foreground fetches still report errors to the client when there is no stale object available.

Also verify that grace is actually configured. Set beresp.grace in vcl_backend_response:

sub vcl_backend_response {
    set beresp.grace = 6h;
}

And use std.healthy() in vcl_recv to conditionally extend grace when backends are sick:

sub vcl_recv {
    if (!std.healthy(req.backend_hint)) {
        set req.grace = 24h;
    }
}

This pattern stores 6 hours of grace on objects during normal operation but allows serving stale content for up to 24 hours when backends are down.

Administratively marking a backend sick

To drain traffic from a backend without changing probe configuration:

# WARNING: This immediately stops all traffic to the backend.
# Use during deployments or when a backend is serving errors but passing probes.
varnishadm backend.set_health <backend_name> sick

To return it to probe-controlled health:

# Return backend to automatic probe-based health
varnishadm backend.set_health <backend_name> auto

backend.set_health accepts patterns, so use a precise name to avoid affecting unintended backends.

Prevention

• Audit probe URLs regularly. The probe should test an endpoint that exercises real application logic, not a static page that returns 200 regardless of state. A probe on / returning 200 while the application is broken gives false confidence.
• Set .initial explicitly. Avoid the post-restart false-sick window by setting .initial equal to threshold in every probe definition.
• Configure grace with 5xx abandonment. Without return (abandon) on 5xx background fetches, error responses overwrite cached objects and silently block grace during backend outages.
• Monitor backend_unhealthy independently of backend_fail. A nonzero backend_unhealthy rate with zero backend_fail is the signature of a sick backend that Varnish is not even trying to reach. This pattern is easy to miss if you only watch backend_fail.
• Monitor cache_hit_grace. If this counter is nonzero during normal operation, backends are failing and grace is masking it. Investigate before grace expires and 503s cascade.
• Tolerate brief probe failures during deploys. With threshold=3 and window=8, the backend absorbs 5 consecutive probe failures before going sick. The 6th consecutive failure tips happy to 2, below threshold. Tune window and threshold to match your deploy cadence and expected transient failures.

How Netdata helps

• Per-second VBE.*.happy collection shows the probe window filling and draining in real time. You can watch happy drop toward the threshold before the backend goes sick, giving you lead time that cumulative counters miss.
• Correlating backend_unhealthy with backend_fail and s_synth distinguishes three failure modes: probe misconfiguration (unhealthy only), fetch failure (fail incrementing), and user-visible 503s (s_synth spiking).
• cache_hit_grace trended alongside backend health reveals when Varnish is surviving on stale content. Sustained grace hits mean backends are down even if clients see no errors.
• ML anomaly detection on probe success rates catches gradual degradation where happy slowly drops from 8 toward the threshold over minutes or hours, before the binary sick transition fires.
• Alerting on backend_unhealthy rate with per-backend breakdown means you see which specific backend is being skipped, not just an aggregate signal.

Related guides

• Varnish cache hit ratio dropped: hit rate collapse and backend overload
• Varnish cache stampede: a popular object expires and the herd hits the backend
• Varnish ESI errors: broken pages and workspace pressure from Edge Side Includes
• Varnish cache_hitpass / cache_hitmiss climbing: uncacheable content bleeding to the backend
• How Varnish actually works in production: a mental model for operators
• Varnish monitoring checklist: the signals every production cache needs
• Varnish monitoring maturity model: from survival to expert
• Varnish not caching: Set-Cookie, Vary, and Cache-Control killing your hit rate
• Varnish pass vs miss: why s_pass and cache_miss are not the same thing
• Varnish sess_dropped vs req_dropped: HTTP/1 connection drops and HTTP/2 stream drops
• Varnish thread pool exhaustion: workers all busy, queue full, sessions dropped
• Varnish thread pool tuning: thread_pool_min, thread_pool_max, and thread_pools