Varnish cache hit ratio dropped: hit rate collapse and backend overload

When cache hit ratio drops, every missed request that previously served from memory now hits the backend. If your backends are sized for cached traffic, not the raw request rate, they saturate quickly. Backend TTFB rises, worker threads are held longer, the thread pool fills, and sessions drop.

The critical diagnostic signal is temporal ordering. Hit rate collapse always precedes backend degradation when the cache is the root cause. If backend TTFB rises first and hit rate falls second, the backend is the problem. If hit rate drops first and backend metrics follow, the cache stopped being effective and the backend is collateral damage. This distinction determines whether you fix VCL, storage, and invalidation logic, or investigate the origin.

This article covers the first pattern: cache-driven hit rate collapse. For the thread pool exhaustion that follows, see Varnish thread pool exhaustion.

What this means

Compute hit ratio from varnishstat counters:

hit_ratio = cache_hit / (cache_hit + cache_miss + cache_hitpass)

The counters are MAIN.cache_hit, MAIN.cache_miss, and MAIN.cache_hitpass. On Varnish 5.1+, a fourth counter MAIN.cache_hitmiss also exists and some operators include it in the denominator. MAIN.cache_hit_grace was added in Varnish 6.0 and is included in MAIN.cache_hit. Regardless of the formula variant, a declining ratio means more requests reach backends.

The cascade is mechanical. When hit ratio drops, MAIN.backend_req spikes proportionally. Backends handling 5-10% of traffic suddenly handle 30-50% or more. Response times increase. Each worker thread stays busy longer waiting for the backend. The thread pool, adequate when most requests were sub-millisecond cache hits, now holds threads for hundreds of milliseconds or seconds per miss. The pool fills, the queue grows, then sessions start dropping.

flowchart TD
    A["Hit rate drops
cache_hit rate falls"] --> B["backend_req rate spikes"]
    B --> C["Backend TTFB rises"]
    C --> D["Worker threads held longer"]
    D --> E["Thread pool fills to max"]
    E --> F["thread_queue_len above 0"]
    F --> G["sess_dropped / req_dropped"]
    C --> H["backend_busy / backend_fail"]

On high-traffic sites with backends tuned for cached load, the cascade can complete in under a minute.

Common causes

Quick checks

These commands are read-only and safe to run in production:

# Core hit ratio counters
varnishstat -1 -f MAIN.cache_hit -f MAIN.cache_miss -f MAIN.cache_hitpass

# Backend request rate (should be low relative to client_req)
varnishstat -1 -f MAIN.backend_req -f MAIN.client_req

# Deliberate pass count (VCL bypassing cache)
varnishstat -1 -f MAIN.s_pass

# LRU eviction vs natural expiry
varnishstat -1 -f MAIN.n_lru_nuked -f MAIN.n_expired

# Storage utilization
varnishstat -1 -f 'SMA.*.g_bytes' -f 'SMA.*.g_space'

# Ban list size and activity
varnishstat -1 -f 'MAIN.bans*'

# Recent VCL changes
varnishadm vcl.list

# Backend health
varnishadm backend.list -p

# Grace serving activity
varnishstat -1 -f MAIN.cache_hit_grace

How to diagnose it

1. Confirm the hit rate drop is real, not a cold-cache artifact. Check MAIN.uptime. If it is under 300 seconds, the cache is still warming after a restart and hit ratio will be near zero and climbing. Do not alert on hit rate for the first 15-30 minutes after a restart.

2. Check for recent VCL reloads. Run varnishadm vcl.list and examine timestamps. If a new VCL was loaded shortly before the hit rate drop, the VCL is the primary suspect. Common mistakes: return (pass) added too broadly, or vcl_hash changes that produce unique cache keys per request.

3. Distinguish miss from pass. MAIN.cache_miss means Varnish looked up the object and found nothing. MAIN.s_pass and MAIN.cache_hitpass mean VCL decided not to cache. If cache_hitpass is climbing, the application is likely emitting headers that prevent caching: Set-Cookie on cacheable responses, Cache-Control: private, or broad Vary headers. Inspect backend response headers with varnishlog -i BerespHeader -q 'BerespStatus == 200'.

4. Check for cache-key explosion. Run varnishlog -i Hash and look at what data goes into the hash. If query strings with varying parameter order, tracking parameters, or session cookies are in the hash, each variant creates a separate cache entry. MAIN.n_object will be high and climbing while hit rate stays low.

5. Check ban list size. Run varnishadm ban.list. If the list has thousands of entries, the ban lurker may be falling behind. Check MAIN.bans vs MAIN.bans_completed in varnishstat. If bans is growing and bans_completed is not keeping up, the lurker is stuck. Req-level bans (those referencing req.*) can only be evaluated at lookup time because the lurker lacks request context. This means they accumulate until each cached object is tested.

6. Check storage pressure. Run varnishstat -1 -f 'SMA.*.g_space'. If g_space is near zero, storage is full and LRU eviction is active. Confirm with MAIN.n_lru_nuked rate: if it is sustained above zero, the cache is too small for the working set.

7. Check if grace is masking a backend problem. If MAIN.cache_hit_grace is elevated, Varnish is serving stale content because backends are unhealthy. The hit rate may look stable while backends are actually down. Cross-check with varnishadm backend.list -p.

Metrics and signals to monitor

Alert on rate-of-change from a rolling baseline, not a static floor. A gradual decline from 95% to 82% over weeks goes unnoticed if the threshold is 80%. Use a > 10-20% sustained drop from a 1-hour rolling average as the trigger, and suppress alerts for 15-30 minutes after restart.

Fixes

VCL change passing everything

If varnishadm vcl.list shows a recent reload and MAIN.s_pass is elevated, roll back to the previous VCL:

# List loaded VCLs with timestamps
varnishadm vcl.list

# Activate the previous VCL (use the name from vcl.list)
varnishadm vcl.use <previous_vcl_name>

This is non-destructive and takes effect immediately. Once traffic stabilizes, diff the VCL files to find the offending change. Common culprits: return (pass) added to vcl_recv with too broad a condition, or a cookie-stripping regex that was removed.

Cache-key explosion

Inspect hash data with varnishlog -i Hash. If variable data is in the hash, normalize it:

• Query strings: Sort query parameters with set req.url = std.querysort(req.url); in vcl_recv. This makes ?b=2&a=1 and ?a=1&b=2 resolve to the same key without modifying the default hash logic.
• Cookies: Strip non-essential cookies before hashing, or exclude cookies from the hash entirely for public content.
• Vary headers: Vary: User-Agent causes Varnish to cache a separate copy per variant. A single browser patchlevel can generate 10 or more distinct User-Agent strings. Normalize the User-Agent in VCL or strip the Vary header for responses where it is not needed.

After fixing, ban the affected objects so old cache entries are evicted:

varnishadm ban 'req.url ~ /'

Warning: This empties the cache for all matching objects. Use during a maintenance window or target the specific URL pattern.

Mass invalidation (ban storm)

If MAIN.bans is high and growing:

# Check ban list size and contents
varnishadm ban.list

# Check lurker activity
varnishstat -1 -f 'MAIN.bans*'

Short-term: restarting the child process clears the ban list but loses the entire cache. You trade a cold cache for a clean ban list.

Long-term: switch from req-level bans to obj-level bans or hash-based purging using the xkey VMOD. The ban lurker can process obj-level bans proactively because they reference only object data. Req-level bans persist until each cached object is tested at lookup time.

Identify the source of excessive bans. Common causes: a CMS publishing hook that bans on every content update, or an automated purge system with a bug causing repeated invalidation.

Storage pressure

If SMA.*.g_space is near zero and MAIN.n_lru_nuked is elevated:

# Check storage utilization
varnishstat -1 -f 'SMA.*.g_bytes' -f 'SMA.*.g_space'

# Sample object sizes being fetched (shows live traffic, not all stored objects)
varnishtop -I ObjHeader:Content-Length

Short-term: exclude large, rarely-repeated objects from caching in VCL (return (pass) for objects above a size threshold). Long-term: increase the storage allocation (-s malloc,SIZE), but reserve 20-30% of system RAM for the OS, Varnish overhead, thread stacks, workspace, and transient storage. Allocating all system RAM to Varnish storage is a common path to OOM.

Also monitor SMA.Transient.g_bytes. Transient storage holds pass, hit-for-pass, and hit-for-miss objects. It is unbounded by default and can grow until the OOM killer fires.

TTL too short

If MAIN.n_expired rate is high and objects expire before being re-requested:

varnishlog -i TTL

Increase TTLs in vcl_backend_response for content that changes infrequently. If the backend sends correct Cache-Control or Expires headers, Varnish respects them. If not, set a default TTL in VCL.

Grace and stale-while-revalidate

Regardless of root cause, configure grace to protect backends during future incidents:

sub vcl_backend_response {
    set beresp.grace = 1h;
}

Grace allows Varnish to serve stale content while a background fetch refreshes the object. Without grace, any backend hiccup produces 503 errors for all cache misses. Monitor MAIN.cache_hit_grace to detect when Varnish is serving stale content, which indicates backend degradation.

Prevention

• Alert on rate-of-change. A static 80% threshold misses a gradual decline from 95% to 82%. Alert on sustained drops of 10-20% from a rolling baseline.
• Monitor cache_hitpass independently. A slow increase means the application is making more content uncacheable. Investigate Set-Cookie, Cache-Control: private, and broad Vary headers.
• Track ban list length. Alert when MAIN.bans exceeds 500. Prefer obj-level bans or the xkey VMOD over req-level bans.
• Diff VCL between deploys. VCL changes are the most common cause of sudden hit rate collapse. Track VCL in version control and watch hit rate for 15-30 minutes after every reload.
• Size storage for the working set. Monitor n_lru_nuked alongside hit rate. If nuking is active and hit rate is declining, the cache is undersized.
• Configure grace mode. Without beresp.grace, backend problems cascade immediately into user-facing errors.

How Netdata helps

Netdata surfaces the counters needed to detect hit rate collapse early and correlate with downstream effects:

• Per-second hit ratio computation from MAIN.cache_hit, MAIN.cache_miss, and MAIN.cache_hitpass, showing rate-of-change without manual varnishstat polling.
• Backend request rate (MAIN.backend_req) shown alongside hit rate, making the inverse relationship visible in a single view.
• Thread pool signals (MAIN.threads, MAIN.thread_queue_len, MAIN.threads_limited) next to cache metrics show whether collapse has progressed to thread pool exhaustion.
• Storage indicators (SMA.*.g_space, MAIN.n_lru_nuked) correlated with hit rate trends distinguish storage-driven decline from VCL-driven decline.
• ML-based anomaly detection on hit rate and backend_req rate flags gradual declines that static thresholds miss.
• Composite alerting combining hit rate drop with backend TTFB increase and thread queue growth provides early warning of the full cascade.

Related guides

• How Varnish actually works in production: a mental model for operators
• Varnish monitoring checklist: the signals every production cache needs
• Varnish monitoring maturity model: from survival to expert
• Varnish sess_dropped vs req_dropped: HTTP/1 connection drops and HTTP/2 stream drops
• Varnish thread pool exhaustion: workers all busy, queue full, sessions dropped
• Varnish thread_queue_len above zero: requests waiting for a worker
• Varnish threads_failed: the OS refusing to create worker threads
• Varnish threads_limited climbing: hitting thread_pool_max