Varnish cache stampede: a popular object expires and the herd hits the backend

A popular cached object hits its TTL and expires. In the next second, hundreds of concurrent requests for that object all miss simultaneously. Varnish forwards all of them to the backend, which was sized for the small fraction of traffic that normally leaks through, not a synchronized burst of identical requests. The backend slows. Worker threads pile up waiting for responses. If the backend cannot recover quickly, thread exhaustion follows and sessions start dropping.

The signature is a sudden, sharp spike in MAIN.cache_miss and MAIN.backend_req that mirror each other, often correlating with a deployment, a purge, or synchronized TTL expiry across a group of objects.

Varnish has a built-in defense: request coalescing. When multiple clients request the same uncached URL, Varnish sends one request to the backend and puts the rest to sleep on a “busy object,” waking them when the fetch completes. This works when the herd is asking for the same object. But when hundreds of unique URLs expire at the same time, coalescing cannot help. The number of concurrent backend fetches equals the unique-URL cardinality of the expiring set.

The fix is not more threads. More threads paper over the symptom while the backend drowns. The fix is grace mode, staggered TTLs, and soft-purge: mechanisms that either prevent the synchronized miss entirely or serve stale content while the cache refills.

What this means

The cascade:

1. A popular object’s TTL expires, or it is invalidated by a ban or purge.
2. Concurrent requests that would have been cache hits all become misses.
3. Each miss generates a backend fetch. Request coalescing collapses requests for the same URL, but different URLs generate different fetches.
4. The backend receives a spike of concurrent requests, often exceeding its capacity.
5. Backend response time increases, holding worker threads longer.
6. The thread pool fills, the queue grows, and sessions or requests are dropped.

The critical distinction from other failure patterns: the cache miss spike happens first, before backend degradation. If the backend degraded first and then hit rate dropped, you have a different problem. In a stampede, the cache is the trigger, not the backend.

flowchart TD
    A[Popular object expires] --> B[Concurrent requests all miss]
    B --> C{Coalescing applicable?}
    C -->|Same URL, few unique| D[One backend fetch, others wait]
    C -->|Many unique URLs| E[Many concurrent backend fetches]
    D --> F[Backend load manageable]
    E --> G[Backend load spike]
    G --> H[Backend response time rises]
    H --> I[Worker threads held longer]
    I --> J[thread_queue_len rises]
    J --> K[sess_dropped / req_dropped]

Common causes

Request coalescing only works for cacheable objects. If VCL returns pass for a request, or if a response is marked beresp.uncacheable = true with a TTL of zero, every concurrent request for that URL bypasses coalescing and hits the backend independently. A single URL that should be cached but is accidentally passed can amplify a stampede from one backend fetch to hundreds.

Quick checks

All commands below are read-only and safe to run on a production Varnish instance:

# Check miss rate and backend request rate together
varnishstat -1 -f MAIN.cache_hit -f MAIN.cache_miss -f MAIN.cache_hitpass -f MAIN.backend_req

# Check request coalescing activity
varnishstat -1 -f MAIN.busy_sleep -f MAIN.busy_wakeup -f MAIN.busy_killed

# Check thread pool saturation
varnishstat -1 -f MAIN.threads -f MAIN.thread_queue_len -f MAIN.threads_limited

# Check if grace is already serving stale content
varnishstat -1 -f MAIN.cache_hit_grace

# Check for recent ban/purge activity
varnishstat -1 -f MAIN.bans_added -f MAIN.n_purges

# Check backend health under load
varnishadm backend.list -p

# Check object expiry patterns
varnishlog -i TTL -g request | head -100

# Check session/request drops (cascade confirmation)
varnishstat -1 -f MAIN.sess_dropped -f MAIN.req_dropped

How to diagnose it

1. Confirm the miss spike mirrors the backend_req spike. If cache_miss and backend_req spike together at the same ratio, the cache is the trigger. Take two readings 5 to 10 seconds apart to compute the rate.

2. Check request coalescing counters. Rising busy_sleep with matching busy_wakeup means coalescing is working: requests for the same URL are collapsed into a single fetch. If busy_sleep is low but backend_req is high, the herd is hitting many unique URLs and coalescing cannot help.

3. Check for busy_killed. Any nonzero rate means requests timed out waiting for a busy object to be fetched. Clients received 503s because Varnish ran out of resources to manage the coalescing queue. The stampede has outgrown the coalescing safety net.

4. Check temporal correlation. Did the spike start immediately after a VCL reload, a deployment, a purge operation, or a restart? Match the timestamp of the first miss spike to operational events. A stampede that starts at a predictable interval (every 5 minutes, every hour) points to synchronized TTL expiry.

5. Check thread pool state. If thread_queue_len is rising, the stampede is cascading into thread starvation. The problem has moved from “backend is slow” to “users get nothing.” If sess_dropped or req_dropped is incrementing, the cascade has reached the user.

6. Check whether grace is active. cache_hit_grace tracks hits served from stale objects. If this counter is rising during the spike, grace mode is mitigating the stampede. If it is zero and you have grace configured, the grace period may be too short, or the objects may not have had a previous cached version to serve as stale.

Metrics and signals to monitor

Fixes

Grace mode (stale-while-revalidate)

Grace is the primary defense. When an object’s TTL expires, grace allows Varnish to serve the stale object to clients while a single background fetch refreshes it. The herd gets stale content instantly. Only one request goes to the backend.

sub vcl_backend_response {
    set beresp.grace = 6h;
}

The default grace is 10 seconds , which is too short for a popular object with high request rates. A grace period of minutes to hours (depending on content staleness tolerance) means that even if the backend is slow or down, Varnish continues serving stale content and does not generate a stampede.

Tradeoff: longer grace means clients may see staler content. For rapidly-changing content, shorter grace is appropriate. For versioned assets or API responses with cache headers, longer grace is safe.

Soft-purge

A hard purge removes the object from cache immediately. The next request misses and fetches from the backend. If the object is popular, the stampede begins.

Soft-purge reduces the object’s TTL without removing it. The object stays available for grace serving while a background fetch refreshes it. Using the purge VMOD :

import purge;

sub vcl_recv {
    if (req.method == "PURGE") {
        purge.soft(ttl = 0s, grace = 6h, keep = 1h);
        return (synth(200, "Soft purge"));
    }
}

This sets the object’s TTL to zero so it is immediately stale, but preserves its grace and keep values. Varnish serves it as stale content while refreshing in the background.

Staggered TTLs

If many objects share the same TTL, they all expire at the same moment. Adding jitter to each object’s TTL spreads expiry across a window, preventing synchronized misses.

The exact implementation depends on your VCL and available VMODs, but the principle is simple: 1000 objects with a nominal 5-minute TTL should expire anywhere between 5 and 6 minutes, not all at the 5-minute mark. Never let a large set of popular objects share a single expiry instant.

Fix hit-for-pass misconfiguration

If VCL is accidentally passing cacheable content, or if uncacheable responses have a zero TTL, request coalescing is disabled for those URLs and every concurrent request hits the backend independently.

For responses that should not be cached, set a non-zero TTL on the uncacheable marker:

sub vcl_backend_response {
    if (beresp.status >= 400) {
        set beresp.uncacheable = true;
        set beresp.ttl = 120s;
    }
}

This creates a hit-for-miss object: Varnish remembers the decision not to cache and serves it from cache for 120 seconds. During that window, request coalescing applies to the hit-for-miss object, preventing repeated backend fetches for the same uncacheable URL.

Prevention

• Set grace on every cacheable object type. Default 10 seconds is insufficient for most production traffic. Use 5 to 30 minutes for most content, hours for static assets.
• Use soft-purge instead of hard purge for popular objects. Reserve hard purge for content that must be immediately unavailable.
• Add TTL jitter. Even 30 to 60 seconds of jitter across objects with the same nominal TTL prevents the herd from forming.
• Watch coalescing counters proactively. busy_sleep, busy_wakeup, and busy_killed tell you whether coalescing is engaging and succeeding. Most teams do not watch these until after their first stampede.
• Audit VCL for accidental pass. Any return(pass) in vcl_recv disables coalescing. Verify that pass is only used for genuinely uncacheable content, and that uncacheable responses have a non-zero TTL to create hit-for-miss objects.
• Do not respond to a stampede by adding threads. More worker threads means more concurrent backend fetches and more backend load. The backend is already the bottleneck.

How Netdata helps

• Per-second metric collection means the cache_miss to backend_req correlation is visible at the resolution needed to confirm a stampede pattern, not smoothed away by 60-second polling.
• Correlating cache_miss, backend_req, busy_sleep, and thread_queue_len on a single timeline shows the full cascade: cache miss triggers backend spike, which triggers coalescing activity, which (if insufficient) triggers thread saturation.
• cache_hit_grace alongside cache_miss reveals when grace is absorbing the stampede versus when misses are reaching the backend.
• Anomaly detection on backend_req rate can flag the spike before thread saturation begins.
• busy_killed as an alert signal catches the moment coalescing fails and users start seeing 503s.

Related guides

• How Varnish actually works in production: a mental model for operators
• Varnish monitoring checklist: the signals every production cache needs
• Varnish monitoring maturity model: from survival to expert
• Varnish sess_dropped vs req_dropped: HTTP/1 connection drops and HTTP/2 stream drops
• Varnish thread pool exhaustion: workers all busy, queue full, sessions dropped
• Varnish thread_queue_len above zero: requests waiting for a worker
• Varnish threads_failed: the OS refusing to create worker threads
• Varnish threads_limited climbing: hitting thread_pool_max