Varnish monitoring checklist: the signals every production cache needs

Varnish Cache is a reverse HTTP proxy that serves cached content from memory. It uses a dual-process architecture: a management process (root-owned, handles VCL compilation, child supervision, and the CLI) and a worker/child process (drops privileges, handles all cache operations). The management process restarts the child automatically on crash, so a child crash is not always a full outage. Repeated child restarts indicate a systemic problem.

Varnish’s concurrency model is thread-per-request with a bounded pool. When the pool is exhausted and the queue is full, sessions are dropped. Most production incidents trace back to thread pool exhaustion, storage pressure, ban list growth, transient memory growth, or backend failure cascades.

This checklist organizes signals into four maturity levels. Know which level you are at and which signals you are missing.

Signal taxonomy at a glance

Varnish exposes counters through varnishstat, which reads live values from a shared memory segment (VSM). Use varnishstat -1 -f <counter_name> for point-in-time snapshots in monitoring scripts. Counters use a prefix convention: MAIN.* for the worker/child process, MGT.* for the management process, SMA.* for malloc storage, SMF.* for file storage, and VBE.* for per-backend metrics.

The signals fall into five operational domains:

Domain	Core question	Primary counters
Availability	Is Varnish serving traffic?	`MAIN.sess_dropped`, `MAIN.req_dropped`, `MGT.child_panic`
Throughput and efficiency	How well is the cache working?	`MAIN.cache_hit`, `MAIN.cache_miss`, `MAIN.cache_hitpass`, `MAIN.backend_req`
Saturation and resources	Are we approaching limits?	`MAIN.threads`, `MAIN.thread_queue_len`, `SMA..g_bytes`, `SMA..g_space`
Backend health	Are origins reachable?	`MAIN.backend_fail`, `MAIN.backend_unhealthy`, `VBE.*.happy`
Internal state	Is the machinery healthy?	`MAIN.bans`, `MAIN.ws_*_overflow`, `MAIN.fetch_failed`

Maturity levels

flowchart TD
    L1["Level 1: Survival
Is it alive? Is traffic lost?"]
    L2["Level 2: Operational
Are we approaching limits?"]
    L3["Level 3: Mature
Leading indicators and internals"]
    L4["Level 4: Expert
Deep contention, security, forensics"]

    L1 --> L2 --> L3 --> L4

Level 1: Survival

Minimum signals to confirm Varnish is alive and not dropping traffic.

MAIN.uptime vs MGT.uptime. The management process stays alive across child restarts, but the child resets on each crash. If MAIN.uptime is much smaller than MGT.uptime, the child recently restarted. The cache is cold after every restart.
MAIN.sess_dropped and MAIN.req_dropped. Sessions dropped (HTTP/1 connections) and requests dropped; req_dropped counts refused HTTP/2 streams specifically. Any nonzero sustained rate means clients receive nothing. Monitor the rate, not the cumulative counter. Note: sess_drop (singular) is a deprecated counter, removed in Varnish 6.4. Use sess_dropped (plural).
MGT.child_panic and MGT.child_died. The child process crashed. A single event is a ticket. Repeated increments with low MAIN.uptime indicate a crash loop with zero effective caching.
MAIN.backend_fail. Backend connection failures (TCP connections attempted and failed). Should be zero in steady state.
Cache hit ratio. Calculated as cache_hit / (cache_hit + cache_miss). Some operators include cache_hitpass in the denominator, but cache_hitpass is a cached decision to bypass the cache, not a miss. Track deviation from your baseline, not an absolute threshold.
MAIN.cache_hit and MAIN.cache_miss. The raw counters behind the hit ratio. Monitor these independently: a declining ratio combined with rising cache_hitpass tells a different story than a decline driven by genuine misses.

Level 2: Operational

Signals to catch problems before they become outages.

MAIN.threads and MAIN.thread_queue_len. Current thread count and session queue depth. When threads reaches thread_pool_max multiplied by the number of pools, the pool is saturated. thread_queue_len should be zero. This gauge updates once per second, so sub-second spikes may not be visible.
MAIN.threads_limited and MAIN.threads_failed. threads_limited means Varnish hit the configured thread_pool_max. threads_failed means the OS refused thread creation (ulimit, memory, cgroup limits). Both are counters; any nonzero rate is actionable.
SMA.{name}.g_bytes and SMA.{name}.g_space. Bytes in use and bytes available for each storage segment. When g_space approaches zero, LRU eviction begins. The {name} is typically s0 for primary storage.
SMA.{name}.c_fail. Allocation failures. If this increments, storage cannot satisfy requests even after eviction. Hard failure.
MAIN.n_lru_nuked. Objects forcefully evicted because storage is full. Moderate nuking is normal for a right-sized cache. Nuking combined with a declining hit rate means the cache is undersized for the working set.
MAIN.backend_unhealthy and MAIN.backend_busy. backend_unhealthy counts connections not attempted because the backend is marked sick. backend_busy means too many outstanding connections. A sick backend with zero backend_fail looks fine if you only monitor connection failures.
MAIN.fetch_failed and MAIN.fetch_no_thread. Fetch failures after a backend connection was established. fetch_no_thread means the thread pool cannot dispatch backend fetches.
MAIN.sess_fail and MAIN.sess_fail_emfile. Session accept failures. sess_fail_emfile confirms file descriptor exhaustion.
MAIN.client_resp_500 and MAIN.ws_*_overflow. Workspace overflow causing 500 errors to clients. Large Cookie headers are the most common cause.
Process RSS. Compare process RSS against configured storage size. The delta is transient storage plus overhead. Transient storage (SMA.Transient.g_bytes) has no upper limit with malloc and can grow without bound, making this the most common OOM path.

Level 3: Mature

Leading indicators, internal machinery, and composite pattern detection.

MAIN.bans and ban lurker counters. Every cache lookup checks against active bans. A growing ban list makes each lookup O(n). Track MAIN.bans, MAIN.bans_added, MAIN.bans_deleted, and MAIN.bans_lurker_contention. If bans_added rate exceeds bans_deleted rate, the lurker is falling behind.
VBE.{name}.happy. Per-backend probe success count within the probe window. Not a boolean: it is a bitmap of recent probe results. Track the rate of change, not the absolute value.
MAIN.losthdr. HTTP headers dropped because the request or response exceeded http_max_hdr (default 64). A dropped Vary header can cause cache poisoning. A dropped Authorization header can cause security bypasses.
MAIN.backend_reuse / backend_conn ratio. Connection pool efficiency. Low reuse means every backend fetch opens a new TCP connection, adding latency and backend load.
MAIN.busy_sleep, MAIN.busy_wakeup, MAIN.busy_killed. Request coalescing activity. Normal in steady state. busy_killed means requests timed out waiting for a coalesced fetch.
SMA.Transient.g_bytes and SMA.{name}.c_fail. Transient storage growth and allocation failures. Transient storage is the silent OOM killer. Monitor it explicitly.
MAIN.cache_hit_grace. Hits served from stale objects via grace mode. A high rate means the cache is masking backend problems. Grace is a runway, not a fix.
MAIN.fetch_no_thread (Varnish ≤ 7.1) and MAIN.bgfetch_no_thread (7.2+, the renamed counter). Thread starvation preventing fetch dispatches, including background fetches (asynchronous refreshes of stale objects). Failure here means objects are not refreshed before they expire.
MAIN.vcl_fail. VCL execution errors during request processing.
MAIN.shm_flushes and MAIN.shm_cycles. Shared memory log subsystem health. Each shm_cycles increment means the log wrapped back to the start — records were overwritten before slow consumers read them, creating monitoring blind spots during incidents. (vsm_overflowed was a Varnish 4.x counter and does not exist in supported releases.)

Level 4: Expert

Deep diagnostics, lock contention, and security signals.

MAIN.hcb_lock vs MAIN.hcb_nolock ratio. Hash lookup contention. High lock ratios indicate the hash table is a bottleneck under high cache hit rates.
LCK.{name}.locks rates. Internal lock contention for key subsystems: sma (storage), wq (work queue), exp (expiry), ban (ban system). Rising rates indicate contention that does not show up in higher-level counters.
MAIN.sc_rapid_reset and MAIN.sc_bankrupt. HTTP/2 protocol abuse signals (both live under MAIN.). sc_rapid_reset, added with the Rapid Reset mitigation (7.3.1, 7.4.2, 6.0 LTS 6.0.12), detects the CVE-2023-44487 pattern. sc_bankrupt, added with the Broke Window mitigation (7.3.2, 7.4.3, 6.0 LTS 6.0.13), indicates an HTTP/2 session failed because all its streams were waiting for flow-control window credits when h2_window_timeout triggered.
MAIN.beresp_uncacheable (6.5+; not in the 6.0 LTS branch). Backend responses marked uncacheable. A rising trend means the application is increasingly preventing caching, often due to new Set-Cookie headers or Cache-Control: private directives.
MAIN.esi_errors and MAIN.esi_warnings. ESI parse errors. Only relevant if Edge Side Includes are configured in VCL.
Per-backend and per-URL latency via varnishlog. Varnish does not expose request latency or backend fetch time as varnishstat counters. Use varnishncsa -F '%D' for request duration in microseconds, or varnishlog -i Timestamp -g request for a detailed timing breakdown separating cache lookup time from backend fetch time.
MAIN.n_vcl and VCL lifecycle. Number of loaded VCLs. n_vcl > 1 can indicate failed reload attempts or old VCLs not being discarded. Check varnishadm vcl.list for timestamps.
OS-level signals. On Linux: net.core.somaxconn saturation, TCP listen overflows, major page faults for file-backed storage, and cgroup memory limits in containers.

Core signal reference

Signal	What it tells you	Warning sign	Minimum level
`MAIN.sess_dropped` + `MAIN.req_dropped`	Clients getting nothing	Any nonzero sustained rate	Survival
`MGT.child_panic` / `MGT.child_died`	Child process crashed	Any increment	Survival
`MAIN.uptime` vs `MGT.uptime`	Recent child restart	`MAIN.uptime` much smaller than `MGT.uptime`	Survival
`MAIN.backend_fail`	Backend unreachable	Any nonzero sustained rate	Survival
`cache_hit / (cache_hit + cache_miss)`	Cache effectiveness	Sustained drop from baseline	Survival
`MAIN.threads` vs `thread_pool_max * pools`	Thread pool saturation	Ratio above 0.8	Operational
`MAIN.thread_queue_len`	Requests waiting for threads	Any sustained nonzero value	Operational
`MAIN.threads_limited`	Hit thread cap	Any nonzero rate	Operational
`SMA.{name}.g_space`	Storage headroom	Approaching zero	Operational
`SMA.{name}.c_fail`	Allocation failure	Any increment	Operational
`MAIN.n_lru_nuked`	Eviction pressure	Sustained rate with declining hit ratio	Operational
`MAIN.backend_unhealthy`	Backend marked sick	Any nonzero rate	Operational
`MAIN.fetch_failed`	Backend fetch failure	Any nonzero sustained rate	Operational
`MAIN.ws_*_overflow`	Workspace exhausted	Any nonzero rate	Operational
Process RSS vs configured storage	Transient + overhead growth	RSS growing beyond expected	Operational
`MAIN.bans`	Ban list size	Growing trend	Mature
`VBE.{name}.happy`	Per-backend health	Dropping below threshold	Mature
`MAIN.losthdr`	Headers dropped	Any increment	Mature
`MAIN.cache_hit_grace`	Serving stale content	High rate (masking backend failure)	Mature
`MAIN.shm_flushes`	Log data loss	Any increment	Mature

What most teams get wrong

Recurring blind spots from production incidents:

Not monitoring process RSS separately from SMA storage counters. Teams configure -s malloc,8G and monitor storage gauges, believing memory is covered. Transient storage grows independently and unbounded. The OOM killer fires with no warning from varnishstat.
Not monitoring MGT. counters.* The child can crash and restart transparently. Without MGT.child_panic and MGT.child_died, repeated crashes go unnoticed. The cache is cold after every restart.
Treating hit ratio as a vanity metric. A 95% hit ratio means nothing if the 5% misses are the most expensive backend queries. Correlate hit ratio with backend_req rate and backend health.
Ignoring ban list accumulation. Application logic issues bans faster than the lurker can process them. The ban list grows until every cache lookup becomes O(n). Use xkey VMOD or hash-based purging instead of regex bans.
Monitoring only sess_dropped and missing HTTP/2 traffic loss. With HTTP/2, overload manifests as req_dropped (stream drops), not sess_dropped (connection drops). Both must be monitored.
Allocating all system RAM to Varnish storage. The OS, thread stacks, workspace memory, and transient storage all need room. Reserve 20-30% of system RAM for non-storage use.

How Netdata helps

Netdata collects Varnish counters per second, which matters because several key signals are point-in-time gauges that oscillate rapidly. Per-second resolution lets you:

Correlate thread_queue_len with sess_dropped and req_dropped in the same time window to confirm whether queue saturation caused traffic loss, or whether drops have a different root cause such as file descriptor exhaustion.
Track MAIN.uptime against MGT.uptime across time, making child restarts visible as step changes rather than requiring manual comparison after the fact.
Layer cache hit ratio with backend_req rate and n_lru_nuked to distinguish hit rate decline caused by storage pressure from decline caused by VCL changes or mass invalidation.
Monitor process RSS alongside SMA.*.g_bytes to surface transient storage growth that varnishstat alone does not make obvious, giving early warning before OOM.
Run anomaly detection on MGT.child_panic, MAIN.backend_fail, and MAIN.bans to flag deviations that may not cross a fixed threshold but represent a meaningful change from the node’s normal pattern.