$ guides / varnish ▌

VARNISH CACHE · OPERATIONS PLAYBOOK

Varnish's cliff-edges: a bounded thread pool, an unbounded memory path, and a cache that fails by going quiet

A reverse HTTP cache that serves from memory at wire speed, runs one worker thread per request from a bounded pool, splits into a management and a child process, and routes every uncacheable response into memory that has no ceiling. We trace how that design behaves under load, where a slow backend or a bad deploy turns it from fast into dropping traffic, and what to do when it does.

> Start with the monitoring checklist → # Jump to the full guide list

Varnish's defaults get you a blisteringly fast cache in an afternoon, then hand you a set of cliff-edges that most teams only discover during an incident.

The defaults work. Until a backend slows down and, because a worker thread is held for the whole request including the fetch, every thread ends up blocked on the origin, the queue fills, and sess_dropped climbs while the CPU sits idle — Varnish looks down, but it is really just full of threads waiting on your backend. Until transient storage — unbounded by default and invisible in the SMA counters — swells under a pass storm and the OOM killer takes the whole process with no warning. Until a popular object expires and hundreds of concurrent misses stampede a backend sized only for the cached load. Until a deploy adds a Set-Cookie to every response and the hit ratio quietly bleeds out. Until the child process panics, the management process restarts it, the cache empties, and — if nobody is watching MGT.child_panic — you never notice it happened.

These guides are written for engineers who already run Varnish, not for people deciding whether to. The goal is the mental model of how the cache actually behaves under load, the failure patterns that keep recurring, the monitoring story that catches them before they page anyone, and the runbooks you wish someone had handed you before your last 503.

How Varnish actually runs in production

Varnish is not just a cache. It is a thread-per-request proxy where a single accept thread feeds a bounded worker pool, a compiled VCL state machine decides the fate of every request, objects live in a fixed store while uncacheable bodies go to an unbounded one, and a root management process supervises a privilege-dropped child that does all the work. Most production failures live between these layers, not inside any one of them.

accept thread + listener

A single accept thread takes new connections and hands each to an idle worker. If none is free, the session enters a bounded queue (<code>thread_queue_len</code>); if the queue is full, the session is dropped and the client gets nothing. This is the front door, and it is the first thing to saturate.

worker thread pool

Thread-per-request, bounded by <code>thread_pool_max</code> × <code>thread_pools</code>. A worker is held for the entire request — including the backend fetch — so a slow origin, not CPU, is what exhausts the pool. <code>threads_limited</code> means you hit the ceiling; <code>threads_failed</code> means the OS refused.

WORKER

VCL state machine

VCL is compiled to C and loaded as a shared object: <code>vcl_recv</code> → <code>vcl_hash</code> → cache lookup → <code>vcl_hit</code>/<code>vcl_miss</code>/<code>vcl_pass</code> → optionally <code>vcl_backend_fetch</code>/<code>vcl_backend_response</code> → <code>vcl_deliver</code>. Hit-rate collapse, pass storms, and backend overload almost always trace back to VCL logic.

VCL

cache lookup + object store

A hash lookup into <code>malloc</code> or <code>file</code> storage. Objects carry three clocks — TTL (fresh), grace (serve stale while refreshing), keep (retain for conditional fetches). When the store fills, LRU eviction (<code>n_lru_nuked</code>) makes room; malloc fragmentation can make eviction happen with free space still reported.

CACHE

ban list + lurker

Invalidation rules. Bans referencing <code>req.*</code> are tested at lookup time; bans referencing <code>obj.*</code> can be processed by the background lurker and removed. A ban list that grows faster than the lurker clears it turns every cache lookup into an O(n) scan.

BANS

backend connections + probes

Pooled connections to origins, reused when keepalive allows. Health probes mark each backend healthy or sick on a threshold/window model. A backend marked sick gets no traffic; all backends sick returns 503 unless grace serves stale — which hides the outage until grace runs out.

BACKEND

workspace + transient memory

Each request allocates from a fixed per-request workspace for headers and VCL string work; overflow is a 500. Uncacheable bodies (pass, pipe, hit-for-pass) go to transient storage, which is unbounded by default — the quiet path to OOM. Everything is recorded to the shared-memory log (VSM).

MEMORY

management + child process

A root-owned management process compiles VCL, hosts the CLI, and supervises a privilege-dropped child that does all the caching. When the child panics, the management process restarts it and the cache empties — <code>MAIN.uptime</code> resets while <code>MGT.uptime</code> keeps counting, the one reliable sign a restart happened.

MGT

Why this matters: 'Varnish is slow' or 'we're getting 503s' can come from the thread pool exhausted by a slow backend, a hit-rate collapse dumping load on the origin, an LRU nuke storm, a ban list gone O(n), a workspace overflow, a file-descriptor ceiling, every backend probe-sick behind grace, or a child crash loop. The symptom rhymes but each layer has a different signal — and a different fix.

The failures you'll actually see

Most Varnish incidents fall into a small set of recurring patterns. Recognise the shape, and triage gets dramatically faster.

CRITICAL

The thread pool death spiral

A backend slows down. Because a worker is held for the whole request including the fetch, threads accumulate, threads reaches thread_pool_max × pools, the queue fills, and sess_dropped / req_dropped start climbing. The tell is that the CPU is idle — Varnish is not busy, it is blocked. It looks exactly like 'Varnish is down' when it is actually full of threads waiting on your origin.

sess_dropped + req_dropped incrementing with live traffic
threads at thread_pool_max × pools, thread_queue_len > 0
threads_limited incrementing (pool at its ceiling)
Low CPU utilisation despite refused connections; backend TTFB elevated

Investigate →

CRITICAL

The 503 backend wall

Clients get Error 503 Backend fetch failed with a Guru Meditation line. Varnish could not get a usable response from any backend — every backend probe-sick, a FetchError (premature close, timeout after connect, invalid HTTP), or a VCL vcl_backend_error. The XID on the error page is the thread to pull: it takes you straight to the failing transaction in varnishlog.

Error 503 Backend fetch failed served to clients
s_synth rising, backend_fail or fetch_failed nonzero
FetchError entries in varnishlog for the failing XID
backend_unhealthy nonzero (fetches to a sick backend)

Investigate →

IMMINENT

Backends all sick, hidden by grace

Backends are down but Varnish keeps serving stale content via grace, so hit ratio and every client-facing metric look fine. Only the backend-health signals are red. This is grace working as designed — but it is a runway, not a fix. When grace expires, 503s cascade all at once, and teams that only watch client metrics get blindsided.

backend.list shows all backends sick, backend_unhealthy climbing
cache_hit_grace elevated (V7+); serving stale objects
n_expired rate declining (objects kept past TTL by grace)
Client 503 rate near zero — until grace runs out

Investigate →

ACTIVE

Hit rate collapse into backend overload

Something stops the cache being effective — a VCL reload that passes everything, a mass ban/purge, a cold cache after restart, a deploy that adds Set-Cookie, or synchronized TTL expiry. backend_req spikes, backends sized for the cached load are overwhelmed, TTFB rises, and it cascades into the thread pool. The sequence always starts with hit rate changing before backend metrics degrade.

cache_hit rate dropping, cache_miss and backend_req spiking together
cache_hitpass / cache_hitmiss climbing (content gone uncacheable)
n_lru_nuked spiking if storage pressure is the cause
Recent vcl.load, mass ban, or app deploy in the timeline

Investigate →

CRITICAL

Transient storage OOM

A large fraction of traffic is being passed, and every passed body lands in transient storage — unbounded by default and not counted in the SMA storage counters. Process RSS climbs past the configured cache size until the OS OOM killer takes Varnish. It looks fine — no 503s, no thread pressure — right up until it is dead. RSS is the only leading indicator.

Process RSS growing beyond configured storage size
SMA.Transient.g_bytes climbing (if exposed by your version)
High cache_hitpass / s_pass rate; VCL passing most traffic
OOM-kill entry in dmesg, then a child restart

Investigate →

ACTIVE

The child crash loop

The child process panics, the management process restarts it, and the cache empties on each restart. A single crash self-recovers and is easy to miss. But when panics repeat faster than the cache can warm, effective caching is zero and the backend takes full traffic — while the listening port stays open, so it does not look like an outage from the outside.

MGT.child_panic or MGT.child_died incrementing repeatedly
MAIN.uptime staying far below MGT.uptime (recent restarts)
cache_hit rate dropping to zero on each restart
_.panic files in /var/lib/varnish/, or OOM entries in dmesg

Investigate →

Choosing a tool

Best Varnish Cache Monitoring Tools

A ranked review of the tools teams actually shortlist here, what each one is genuinely good at, and how the pricing behaves as you scale.

Read the buyer's guide →

Varnish monitoring maturity levels

Varnish observability works in four practical levels. Each is a complete operation, not a stepping stone. Pick the level that matches how much your cache matters. Most production caches should land at the second level.

Level 1: Survival

Know that something is wrong

Survival monitoring is the floor. With these signals you can answer one question: is Varnish alive and serving traffic? You will not learn what broke, but you will learn that something broke before users do. Survival is enough for a cache in front of non-critical, easily-regenerated content.

Process alive / child not crash-looping MAIN.uptime far below MGT.uptime means the child keeps restarting.
Sessions / requests dropped sess_dropped + req_dropped: clients being turned away at the door.
Backend reachable Is any backend healthy, or is everyone sick?
Cache hit ratio Is the cache working at all, or passing everything to the origin?
Process RSS The only leading indicator for the transient-storage OOM cliff.

Level 2: Operational

Diagnose most incidents on your own

Operational monitoring is what most production caches should target. Survival tells you something is wrong; operational tells you what. With this coverage your team can usually diagnose an incident on its own: thread saturation, storage pressure, backend trouble, workspace failures.

Threads and thread_queue_len The bounded pool and its overflow queue — the primary bottleneck.
threads_limited Hitting thread_pool_max; the pool needs resizing or the backend needs fixing.
Storage g_bytes / g_space How full the cache is; near-full triggers LRU nuking.
n_lru_nuked Eviction pressure; a spike with declining hit rate means undersized.
backend_unhealthy / backend_busy Fetches blocked by sick backends or a max_connections ceiling.
client_resp_500 / ws_*_overflow Workspace exhaustion returning 500s to clients.
fetch_failed Backend connected but the fetch broke — premature close, bad framing.
sess_fail / sess_fail_emfile Accept failures; emfile confirms the file-descriptor ceiling.

Level 3: Mature

Catch problems before they become incidents

Mature monitoring catches problems before they wake anyone up. A ban list creeping toward O(n) lookups, headers silently dropped, connection reuse quietly failing, a single backend degrading behind the aggregate, log records being lost. None of these page you on day one. They become page-out incidents on day thirty.

Ban list length and lurker activity bans growing with bans_lurker_contention means every lookup slows.
losthdr Headers dropped past http_max_hdr; a lost Vary means cache poisoning.
n_expired vs n_lru_nuked ratio TTL-bound (healthy) vs storage-bound (undersized) eviction.
Backend connection reuse ratio Low reuse means every fetch pays a fresh TCP/TLS handshake.
Per-backend VBE.happy Aggregate health hides one backend in trouble.
shm_flushes / vsm_overflowed Log records lost — a monitoring blind spot during incidents.
busy_sleep / busy_wakeup / busy_killed Request coalescing under load; killed means requests timed out waiting.
Transient.g_bytes and c_fail Transient growth and allocation failures (fragmentation).

Level 4: Expert

Reactive instrumentation after real incidents

Expert signals enter your stack the day after a specific incident proved you needed them. Hit-path latency regressions, lock contention, HTTP/2 abuse, cacheability drift, VCL lifecycle. Most teams never need every signal here. Add the ones your incident history says you do.

Hit vs miss latency, tracked separately varnishstat has no timing — this comes from varnishlog/varnishncsa.
Backend TTFB percentiles The leading indicator of thread pool pressure; from varnishlog.
LCK.* lock contention sma, wq, exp, ban lock rates when the hit path degrades.
sc_rapid_reset / sc_bankrupt HTTP/2 Rapid Reset (CVE-2023-44487) and stream-credit abuse.
beresp_uncacheable drift Backend responses slowly becoming uncacheable, bleeding to origin.
n_vcl and vcl_fail Cold VCLs never discarded; runtime VCL execution failures.
Grace runway estimation How long can you serve stale if every backend dies right now?

Operating mistakes worth avoiding

The traps Varnish teams keep falling into. Each has a clear, well-known fix. Most teams only learn it after an incident.

Not monitoring transient storage or process RSS

The number-one missed signal. Teams set <code>-s malloc,8G</code>, watch the <code>SMA</code> counters, and believe memory is covered. Transient storage — where every passed body goes — grows separately and unbounded, invisible in <code>SMA</code>. One day process RSS exceeds the box and the OOM killer fires with no warning. Monitor process RSS; the delta over configured storage is transient plus overhead. On 6.1+, cap it with <code>-s Transient=malloc,SIZE</code>.

Allocating all system RAM to storage

Setting <code>-s malloc,32G</code> on a 32GB machine ignores the OS, the Varnish process, thread stacks, per-request workspace, and transient storage. The result is that the OOM killer targets Varnish. Reserve 20-30% of system RAM for everything that is not the object store, and size storage against the working set rather than the box.

Ignoring ban list accumulation

Scripting a <code>ban</code> for every content update feels clean until updates arrive faster than the lurker clears them. The list grows, every cache lookup becomes an O(n) scan, and hit latency and CPU climb while hit rate looks fine. Monitor <code>MAIN.bans</code>, prefer <code>obj.*</code> bans the lurker can process, and use the <code>xkey</code> VMOD or hash-based purging for tag invalidation.

Watching hit ratio but not hitpass/hitmiss

A hit-ratio floor alert misses a slow rise in <code>cache_hitpass</code>/<code>cache_hitmiss</code> because the denominator includes them — the ratio reads 'fine' at 75% while 20% of traffic is passed to the backend unnecessarily. Track uncacheable rate independently, and when it climbs, look at the application's <code>Set-Cookie</code>, <code>Vary</code>, and <code>Cache-Control</code> headers.

Running with no grace or stale configuration

Without <code>grace</code> and <code>stale-if-error</code>, any backend hiccup turns every cache miss into an immediate 503. Grace gives a buffer that dramatically improves perceived reliability during backend blips — but it must be paired with monitoring backend health directly, because grace also hides that the backend is down until it expires.

Blaming Varnish for slow responses

Varnish adds microseconds to a cache hit. When responses are slow, the cause is almost always the backend — a hit-rate collapse pushing traffic to a slow origin, or slow backend TTFB holding worker threads. Check backend TTFB in <code>varnishlog</code> before touching Varnish tuning; Varnish is usually the messenger, not the cause.

Not monitoring the management process

Teams watch <code>MAIN.*</code> counters and miss <code>MGT.child_panic</code> and <code>MGT.child_died</code>. The child can crash and be restarted transparently — the port stays open — but the cache is lost each time. Without <code>MGT.*</code> monitoring, a crash loop that is silently sending full traffic to your backend goes unnoticed.

Ignoring workspace failures and losthdr

<code>client_resp_500</code>, the <code>ws_*_overflow</code> counters, and <code>losthdr</code> are rarely watched but signal active user-facing errors. Large <code>Cookie</code> headers are the top cause of workspace overflow; a header dropped past <code>http_max_hdr</code> can silently break caching — a lost <code>Vary</code> poisons the cache, a lost <code>Authorization</code> bypasses auth. Alert on any nonzero value.

Varnish runbooks in this section

Each guide is a focused runbook for one symptom or topic. Pick one when you have an incident, or use the categories to learn the area.

▸

Start here

▸

Worker threads and dropped connections

▸

Cache hit ratio and efficiency

▸

Backends, health, and 503s

▸

Storage, eviction, and memory

▸

Workspace, headers, and 500 errors

▸

Bans, purges, and the ban lurker

▸

Child process, VCL, and crashes

▸

Connections, file descriptors, and sessions

▸

Latency and shared-memory logging

▸

Security and HTTP/2 abuse

WHERE TO GO NEXT

Setting up Varnish monitoring, or putting out a fire?

If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, jump straight to the symptom that matches what you're seeing.

> Start with the checklist > Back to Operations Guides