$ guides / varnish / varnish-monitoring-maturity-model ▌

Operations Guides

Varnish monitoring maturity model: from survival to expert

Most Varnish deployments start with the same three questions: is the process alive, is the hit ratio acceptable, are backends reachable. That is enough to catch a full outage. It is not enough to catch the slow failures that actually dominate production incidents: thread pool exhaustion with idle CPU, ban list growth turning every cache hit into a linear scan, transient storage OOM with no SMA counter movement, or grace mode silently masking a dead backend until the stale content expires.

This article maps four monitoring maturity levels. Each level adds signals that answer questions the previous level could not. The levels are cumulative: Level 2 includes everything in Level 1, and so on. If your team was paged for an incident that monitoring did not predict, find the signal that would have caught it and work backward to the missing level.

Metrics referenced here are varnishstat counters unless otherwise noted. Counter names follow the MAIN.*, SMA.*, VBE.*, MGT.*, and LCK.* namespaces. Version-specific counters are noted where relevant.

flowchart TD
    L1["Level 1: Survival
alive, drops, hit ratio, backend health"]
    L2["Level 2: Operational
threads, storage, workspace, fetch failures"]
    L3["Level 3: Mature
bans, grace, coalescing, per-backend health"]
    L4["Level 4: Expert
contention, abuse, cacheability, latency"]

    L1 --> L2
    L2 --> L3
    L3 --> L4

Level 1: Survival

Level 1 answers one question: is Varnish serving traffic, or is it down? Every team running Varnish in production needs at least this coverage.

Signal	What it tells you	Why it matters
`MAIN.uptime` / `MGT.uptime`	Process alive, child not crash-looping	`MGT.uptime` continues through child restarts; `MAIN.uptime` resets on each restart. If `MAIN.uptime` is much smaller than `MGT.uptime`, the child recently restarted and the cache was lost.
`MAIN.sess_dropped` / `MAIN.req_dropped`	Sessions or requests dropped	Zero is the only acceptable sustained value. Nonzero means clients got nothing: no response, connection reset.
`MGT.child_panic` / `MGT.child_died`	Child process crash detection	Any increment is abnormal. Repeated increments with low `MAIN.uptime` indicate a crash loop where effective caching is zero.
`MAIN.cache_hit` / `MAIN.cache_miss`	Basic hit ratio	A drop means more traffic reaching backends. Do not alert on hit ratio for 15-30 minutes after restart (cold cache warmup).
`MAIN.backend_fail`	Backend connection failures	TCP connections attempted and failed. Sustained nonzero means backends are unreachable or refusing connections.
Process RSS	Memory pressure approaching OOM	Compare to configured storage size. RSS far exceeding `-s malloc,SIZE` signals transient storage growth or overhead that SMA counters do not track.

What Level 1 cannot see: thread pool saturation, storage pressure before it becomes critical, workspace overflow causing errors, ban list growth, per-backend health differences, and any failure that Varnish masks via grace mode.

Transition trigger: you get paged for 503 errors or slow responses. Hit ratio looks fine, backends look healthy, but users are complaining. The problem is thread pool exhaustion or backend latency holding threads, and Level 1 signals do not surface it.

Level 2: Operational

Level 2 adds the signals that explain why Varnish is failing, not just that it is failing. This is the level a competent team needs to diagnose most production incidents without guessing.

Signal	What it tells you	Warning sign
`MAIN.threads`	Current worker thread count	Approaching `thread_pool_max * thread_pools`, new requests queue.
`MAIN.thread_queue_len`	Requests waiting for a worker thread	Any sustained nonzero value means the system is at capacity. Drops follow when the queue fills.
`MAIN.threads_limited`	Thread creation blocked by `thread_pool_max` ceiling	Rate > 0 means the configured maximum is too low for current traffic.
`MAIN.threads_failed`	OS refused thread creation	Any nonzero value indicates system-level limits: `ulimit`, memory for thread stacks, cgroup constraints.
`SMA.{name}.g_bytes` / `g_space`	Storage used vs available	`g_space` approaching 0 triggers aggressive LRU eviction of cached objects.
`MAIN.n_lru_nuked`	Objects evicted to make room for new ones	Rate > 0 with declining hit ratio means cache is undersized for the working set.
`MAIN.backend_unhealthy`	Connections not attempted because backend is sick	Sick backends get zero traffic. If all backends are sick, Varnish serves stale or returns 503.
`MAIN.backend_busy`	Backend connection limit reached	Only relevant if `.max_connections` is configured on the backend definition.
`MAIN.client_resp_500`	500-class responses delivered to clients	Clients receiving 500 errors.
`MAIN.ws_*_overflow`	Per-workspace memory exhausted	Rate > 0 means request headers, cookies, or VCL data exceed workspace allocation. Large Cookie headers are the most common cause.
`MAIN.fetch_failed`	Backend fetch failed after TCP connect succeeded	Protocol errors, timeouts after connection, truncated responses, or thread starvation preventing fetch dispatch.
`MAIN.sess_fail` / `sess_fail_emfile`	Accept failures, file descriptor exhaustion	`sess_fail_emfile` (V6+) specifically confirms FD exhaustion as the cause.

Key diagnostic patterns at this level:

Thread starvation spiral: threads at max, thread_queue_len greater than zero, threads_limited incrementing, and sess_dropped beginning. CPU may be completely idle because the bottleneck is thread availability, not processing power. The root cause is almost always slow backends holding threads. Increasing thread_pool_max is a stopgap. Fixing backend latency is the actual fix.

Storage exhaustion cascade: g_space approaching 0, n_lru_nuked rate spiking, hit ratio declining, backend_req increasing. The cache is too small for the working set. Evicted objects get re-requested, sending more traffic to backends, which slows them, which holds threads longer, which can cascade into thread starvation.

What Level 2 cannot see: ban list growth degrading lookup performance, per-backend health granularity (only aggregate backend signals are visible), grace mode masking backend failures, transient storage growth toward OOM, request coalescing pressure, and shared memory log data loss.

Transition trigger: Varnish is slow but hit ratio and error rates look normal. The ban list has grown to thousands of entries and every cache hit does a linear scan against outstanding bans. Or: backends went down 20 minutes ago but nobody noticed because grace mode is silently serving stale content.

Level 3: Mature

Level 3 adds internal state visibility, leading indicators, and the signals that distinguish a well-run cache from one that is quietly degrading. Teams reach this level after their second or third incident that Level 2 could not explain.

Signal	What it tells you	Warning sign
`MAIN.bans` + lurker counters	Ban list size and lurker processing health	Growing `bans` with `bans_added` rate far exceeding `bans_deleted` means the lurker is falling behind.
`MAIN.losthdr`	HTTP headers dropped because `http_max_hdr` was exceeded	Any nonzero value. Dropped `Vary` headers cause cache poisoning. Dropped `Authorization` headers cause security bypass.
`backend_reuse` / `backend_conn` ratio	Backend connection pool efficiency	Ratio below 0.5 means excessive new TCP connections to backends, adding handshake latency.
`VBE.{name}.happy`	Per-backend probe success count within the probe window	Per-backend granularity reveals individual backend problems hidden by aggregate metrics.
`MAIN.shm_flushes`	Shared memory log buffer overruns	Rate > 0 means log consumers are losing records. You lose observability data during exactly the incidents you need it most.
`MAIN.busy_sleep` / `busy_wakeup` / `busy_killed`	Request coalescing activity for busy objects	`busy_killed` greater than zero means requests timed out waiting for a coalesced backend fetch to complete.
`SMA.Transient.g_bytes`	Transient storage size (uncacheable response bodies)	Unbounded growth path to OOM. Pass storms, pipe traffic, and hit-for-pass objects all consume this storage.
`SMA.{name}.c_fail`	Storage allocation failures	Any nonzero value means storage cannot satisfy an allocation even after LRU eviction.
`MAIN.vcl_fail`	VCL execution failures during request processing	Rate > 0 means the compiled VCL hit a runtime error condition.
`MAIN.cache_hit_grace`	Hits served from stale content via grace mode	High rate means the cache is masking backend problems. Correlate with backend health to catch this early.
`MAIN.fetch_no_thread` / `bgfetch_no_thread`	Thread starvation spreading to backend fetches	Any nonzero value means thread exhaustion has reached the point where even backend operations cannot dispatch.

Key patterns at this level:

Ban lurker stall: MAIN.bans growing, bans_completed rate near zero, bans_lurker_contention rate high. Every cache lookup tests the request against all outstanding req-level bans. Performance degrades across all URLs even though hit rate and error rates look normal. This is unique: most patterns affect hit rate or error rate, not pure hit-path latency. Switch from req.* bans (which the lurker cannot process proactively) to obj.* bans, or use the xkey VMOD for hash-based purging.

Grace masking: Backends are sick, VBE.{name}.happy is below threshold for all backends, but cache_hit_grace is high and client 503 rate is zero. Everything looks fine from the client side. This is grace working as designed, but it is a ticking clock. When grace periods expire, 503s cascade. Monitor backend health metrics independently of client-facing metrics.

Transient storage OOM: SMA.Transient.g_bytes growing monotonically while cache operations look normal. The process RSS creeps upward past the configured storage size. One day the OOM killer fires with no warning from standard SMA storage counters. This is the most common mystery crash in Varnish. If your Varnish is older than 6.1, transient storage is completely unbounded. In 6.1+, you can cap it with -s Transient=malloc,SIZE.

What Level 3 cannot see: lock contention inside Varnish internals, HTTP/2 protocol abuse patterns, cacheability drift where backend responses that should be cached are not, and per-URL latency breakdowns that would identify specific slow routes.

Transition trigger: you need to understand where latency is coming from on a per-URL basis, or you are investigating a security incident involving HTTP/2 abuse, or you suspect internal lock contention is causing performance degradation that no counter at Level 3 explains.

Level 4: Expert

Level 4 signals are the deep internals that experienced operators add after repeated production incidents. These counters and analysis techniques surface problems that are invisible to standard monitoring.

Signal or technique	What it tells you	When to use it
`LCK.{name}.locks` rates	Lock contention on internal locks (sma, wq, exp, ban)	When performance degrades and no other signal explains it. Requires the `lck` debug bit enabled, otherwise these counters never increment even under active contention.
`MAIN.sc_rapid_reset`	HTTP/2 Rapid Reset attack detection (CVE-2023-44487)	Security monitoring. Any sustained nonzero rate warrants investigation.
`MAIN.beresp_uncacheable`	Backend responses marked uncacheable	Gradual increase means cacheability drift. Responses that should be cached are not, and hit ratio drops slowly while individual metrics look normal.
Per-URL latency analysis	Latency broken down by URL pattern	Via `varnishncsa -F '%D %U %s'` for request duration in microseconds. Identifies slow URL patterns and per-route cache effectiveness.
Grace runway estimation	How long stale content can survive if all backends fail	Critical for incident readiness. Calculate from grace TTLs and object age distribution.
`MAIN.hcb_lock` vs `hcb_nolock` ratio	Hash lookup lock contention	High lock ratio relative to nolock means the hash table is contended under high request rates.
`MAIN.vsm_overflowed`	VSM space overflowed	Log subsystem under memory pressure. Observability data is being lost at the VSM level.
Malloc fragmentation tracking	RSS growth vs object churn over weeks	RSS growing while object count and `g_bytes` remain stable indicates malloc fragmentation in the storage allocator.

Important gotcha for LCK counters: the lck debug bit must be explicitly enabled for LCK.* counters to function. Without it, they stay at zero regardless of actual lock contention. If you are investigating a performance mystery and these counters show nothing, verify the debug bit is set before concluding contention is absent.

Expert-level analysis also includes techniques that go beyond varnishstat counters:

Per-URL latency: Varnish does not expose request latency as a varnishstat counter. Use varnishncsa -F '%D' for request duration in microseconds, or %T for seconds. For backend time-to-first-byte, use %{Varnish:time_firstbyte}x. Track hit and miss latency distributions separately. Cache misses dominate the tail and will hide a genuine hit-path regression such as VCL complexity growth or ban list overhead if you only look at aggregate P99.

Grace runway: When backends fail, grace mode serves stale content until the grace TTL expires. Estimate runway by examining object grace values via varnishlog -i TTL. If your average grace window is 10 minutes and backends take 30 minutes to recover, users will see 503s before recovery. Knowing this number before an incident lets you decide whether to extend grace in VCL proactively.

Malloc fragmentation: With malloc storage, the allocator can report free space via g_space that cannot actually be used because freed memory blocks are too fragmented to satisfy a contiguous allocation request. Track process RSS over weeks. If RSS grows while object count and g_bytes remain stable, fragmentation is accumulating. A scheduled restart during a maintenance window may be preferable to waiting for allocation failures or OOM.

Maturity level summary

Level	Core question answered	Key new signals added
1 Survival	Is Varnish up?	uptime, sess_dropped, hit ratio, backend_fail, RSS
2 Operational	Why is Varnish failing?	threads, queue, storage, workspace, fetch_failed
3 Mature	What is degrading silently?	bans, grace, transient storage, per-backend health, busy_sleep
4 Expert	Where is the hidden bottleneck?	LCK contention, sc_rapid_reset, beresp_uncacheable, per-URL latency

How Netdata helps

Netdata’s Varnish collector surfaces counters from all four maturity levels in a single per-second pipeline, which shortens the diagnostic path in several specific ways:

Correlating thread saturation with backend latency: Per-second MAIN.threads, MAIN.thread_queue_len, and MAIN.threads_limited alongside backend request rates let you see thread pool exhaustion develop in real time, not just after drops begin.
Distinguishing hit-path from miss-path degradation: Tracking MAIN.cache_hit, MAIN.cache_miss, and MAIN.cache_hit_grace together lets you detect grace masking before grace expires and 503s cascade.
Transient storage as a first-class signal: SMA.Transient.g_bytes monitored alongside process RSS catches the OOM path that standard SMA storage monitoring misses entirely.
Ban list growth detection: MAIN.bans with lurker counters visible on the same dashboard as request latency lets you correlate ban list growth with performance degradation that would otherwise be unexplained.
Per-backend health granularity: VBE.*.happy counters broken out per backend prevent aggregate health metrics from hiding individual backend failures during partial outages.