How Varnish actually works in production: a mental model for operators

Varnish Cache is a reverse HTTP proxy that serves cached content from memory. Behind that description sits a set of interacting subsystems, each with its own failure modes and saturation points. Before you can interpret a counter or diagnose an incident, you need to understand how a request moves through the process, where worker threads are consumed, how the cache store is managed, and what runs in the background.

This article covers the architecture you need before the runbooks: the dual-process split, the accept thread and bounded worker pool, the VCL state machine, the object store with its unbounded transient sibling, the ban list and its lurker, backend connection pools with health probes, and the shared-memory log.

What it is and why it matters

Varnish sits between clients and origin backends, intercepting HTTP requests and serving responses from an in-memory or file-backed cache. When it fails, it can amplify load onto backends, drop connections silently, or consume all available memory and get killed by the OOM killer.

Varnish’s failure modes are not obvious from surface metrics. CPU can be idle while sessions are being dropped. Hit rate can be high while the cache serves stale content. Backend failure counters can read zero while backends are completely offline. Each scenario makes sense only if you understand which internal subsystem produces the signal and how those subsystems interact.

How it works

The dual-process split

Varnish runs as two cooperating processes. The management process runs as root and handles VCL compilation, child supervision, and the management CLI (varnishadm). The child process drops privileges after startup and handles all cache operations: accepting connections, executing VCL, serving objects, and fetching from backends.

This split matters because the management process supervises the child and can restart it automatically if it crashes. A single child crash is not a full outage: the listening port stays open under the management process, and the child comes back. But each restart empties the cache. Repeated crash-restart cycles mean zero effective caching with full backend load.

The counter namespaces reflect this split. MGT.* counters (management process) persist across child restarts. MAIN.* counters (child process) reset to zero on each restart. If MAIN.uptime is much smaller than MGT.uptime, the child has restarted recently. This is the first diagnostic check when hit rate suddenly drops to zero.

Accept thread and worker pool

A single accept thread listens on configured sockets and accepts new connections. Each accepted connection is handed to a worker thread from a bounded pool. Varnish uses a thread-per-request concurrency model: one worker thread handles one request from receipt to response delivery.

The pool has configurable bounds. thread_pools sets the number of pools (default 2). Each pool maintains thread_pool_min to thread_pool_max worker threads (default maximum is 5000 per pool). When all workers in a pool are busy, new requests enter a bounded queue (thread_queue_len). If the queue is full, the session is dropped for HTTP/1 or the stream is dropped for HTTP/2. The client gets nothing.

This is the single most important bottleneck to understand. The constraint is thread availability, not processing power. Slow backends hold threads longer, reducing effective concurrency without touching CPU. A system with idle CPU and zero backend errors can still be dropping connections because every worker thread is blocked waiting for a slow backend response.

flowchart TD
    MGT["Management process (root)"] -->|"supervises"| CHILD["Child process"]
    CHILD --> AT["Accept thread"]
    AT --> WP["Worker pool (bounded)"]
    WP --> VCL["VCL: recv, hash, lookup"]
    VCL -->|"hit"| DELIVER["vcl_deliver to client"]
    VCL -->|"miss"| BACKEND["Backend pool + probes"]
    BACKEND --> DELIVER

VCL: the compiled request lifecycle

VCL (Varnish Configuration Language) defines the request lifecycle. VCL source is compiled to C and loaded as a shared object, which makes execution fast but means severe VCL bugs can crash the child process through the compiled C path.

The request flows through subroutines in a defined order:

vcl_recv -> vcl_hash -> cache lookup -> vcl_hit / vcl_miss / vcl_pass -> optionally vcl_backend_fetch / vcl_backend_response -> vcl_deliver

Each subroutine can alter flow. return(pass) bypasses the cache entirely. return(pipe) sends the request raw to the backend without caching the response. return(synth) generates a synthetic response. The vcl_hash subroutine determines the cache key, which controls what counts as a hit.

Misconfigured VCL is the most common source of operational problems. Hit rate collapse, pass storms, and backend overload all trace back to VCL logic. A VCL change that hashes on the wrong variable, strips the wrong cookie, or passes traffic that should be cached can shift load patterns immediately after a reload.

Object store and transient storage

Cached objects live in a storage backend selected at startup:

• malloc: malloc()-based, fastest, hard-capped by -s malloc,SIZE. Subject to fragmentation over weeks or months of churn.
• file: memory-mapped file on disk. Larger capacity but subject to OS page cache pressure. Performs like malloc until memory pressure forces paging.
• persistent: deprecated. Avoid in production.

Each storage segment tracks g_bytes (bytes in use), g_space (bytes available), and c_fail (allocation failures). When storage fills, the LRU (Least Recently Used) list triggers eviction. Objects are “nuked” to make room for new ones. Some nuking is normal in a well-utilized cache: the LRU tail should be unpopular objects. Excessive nuking that correlates with declining hit rate means the cache is undersized for the working set.

Transient storage is the silent sibling. Objects that will never be cached (pass traffic, hit-for-pass, hit-for-miss, pipe bodies) go to transient storage, which uses malloc and is unbounded by default. A pass storm, a stream of large uncacheable responses, or a VCL change that accidentally passes all traffic can cause transient storage to grow without limit and OOM the process. Monitor SMA.Transient.g_bytes if your version exposes it, or track process RSS against the configured storage size. The delta is transient storage plus overhead.

Ban list and lurker

Bans are invalidation rules. When you purge or invalidate cached content, Varnish adds a ban expression to a list. Every cache lookup checks the object against all active bans. If the list grows large, each lookup becomes O(n) in the list length.

The ban lurker is a background thread that walks the object store, testing objects against bans and evicting matches proactively. This is the mechanism that eventually removes bans from the list. But the lurker can only evaluate bans that reference obj.* variables. Bans that reference req.* variables cannot be tested by the lurker because the original request context is gone. These bans persist until every relevant object has been checked at lookup time by a real request.

If bans arrive faster than the lurker processes them, the list grows without bound. The operational signature is gradually increasing latency on all requests, including cache hits, with no corresponding increase in backend request rate or error rate. CPU usage on Varnish rises from ban evaluation overhead.

Backend connection pools and health probes

Varnish maintains connection pools to backend servers. Connections are reused (backend_reuse), recycled (backend_recycle), or opened fresh (backend_conn). A high reuse ratio relative to new connections indicates efficient keepalive. Low reuse means every fetch opens a new TCP connection, adding handshake latency.

Backend health is assessed via probes: configurable HTTP checks that run on a timer. A backend is marked healthy or sick based on a threshold/window model. If at least threshold out of the last window probes succeeded, the backend is healthy. A sick backend receives zero traffic.

This creates a monitoring trap. A sick backend generates no connection failures because Varnish never attempts a connection to it. backend_fail stays at zero. Only backend_unhealthy reveals that connections were not attempted because the backend was marked sick. An operator checking only backend_fail will see a healthy system while backends are completely offline.

If all backends in a director are sick, Varnish either serves stale content via grace (if configured) or returns 503 to all cache misses. Grace mode can mask a backend outage for minutes or hours, depending on configured grace periods. The cache hit rate may look unchanged while the backend is down.

Shared-memory log (VSM)

Varnish does not write logs to disk by default. All transaction-level logging goes to a shared-memory segment called VSM, structured as a circular buffer. External tools (varnishlog, varnishncsa, varnishstat) read from this segment as clients.

If log consumers read too slowly, or if no consumer is running, the circular buffer wraps and log records are overwritten. This does not affect request processing, but it destroys observability data during high-traffic incidents when log volume is highest.

varnishstat reads live counters from a separate part of VSM. These counters update in batches, not in real time. Expect sub-second lag between an event and the counter increment.

Where it shows up in production

Several deployment choices change how the mental model applies:

• Storage type (malloc vs file): malloc is volatile (lost on restart) and RAM-bound. file adds disk I/O and OS page cache pressure. The monitoring signals differ: malloc requires watching for fragmentation, file requires watching for I/O latency.
• HTTP/1 vs HTTP/2: HTTP/2 multiplexes streams over a single connection. Overload manifests as req_dropped (stream drops) rather than sess_dropped (connection drops). A team monitoring only sess_dropped will miss HTTP/2 traffic loss entirely.
• Behind a load balancer: health checks come from the LB, not just Varnish’s own probes. Source IPs require X-Forwarded-For parsing. If the LB marks a Varnish node unhealthy during thread pool exhaustion, traffic concentrates on remaining nodes, creating a cascade.
• With ESI: Edge Side Includes add sub-requests, each running a full VCL cycle. This increases thread utilization, workspace pressure, and backend request count. ESI parse errors produce broken pages silently.
• With VMODs: custom C modules extend VCL but introduce their own failure modes. A VMOD segfault crashes the child process.

Failure patterns this model explains

Each failure archetype maps to a specific subsystem reaching its limit:

• Thread pool exhaustion: all workers busy, typically blocked on slow backend responses. Queue fills, sessions dropped. CPU is idle. Root cause is almost always backend latency holding threads, not Varnish itself.
• Cache stampede: popular objects expire simultaneously. Concurrent requests all miss, all fetch from backend. Without grace or stale-while-revalidate, this cascades into backend overload and then thread exhaustion.
• Ban list explosion: bans arrive faster than the lurker processes them. List grows, every lookup becomes expensive, latency increases across all requests. Hit rate may be fine while every hit is slower.
• Storage exhaustion: malloc or file storage fills. Aggressive LRU eviction evicts useful objects. Hit rate drops, backend load surges, which can cascade into thread exhaustion.
• Transient storage OOM: pass traffic or large uncacheable responses fill transient storage, which has no upper bound by default. Process RSS grows until the OOM killer fires. Looks fine until it dies.
• Workspace overflow: HTTP headers or cookies exceed per-request workspace allocation (workspace_client, workspace_backend). Requests fail with 500 errors. Large Cookie headers are the most common cause.
• Child crash loop: the child process panics or is killed. Management process restarts it. Cache is lost on each restart. If restarts happen faster than warmup, effective caching is zero.

Signals to watch in production

How Netdata helps

Netdata’s Varnish integration collects per-second metrics that correlate these subsystems in real time:

• Thread pool saturation: MAIN.threads, MAIN.thread_queue_len, and MAIN.sess_dropped are collected together. Per-second granularity shows the progression from saturation to drops before users report failures.
• Hit rate decomposition: cache_hit, cache_miss, cache_hitpass, and cache_hitmiss (where available) are tracked independently. A slow increase in hitpass, invisible in a single hit-rate percentage, becomes visible as its own trend.
• Storage and transient monitoring: SMA counters for primary and transient storage are collected alongside process RSS. The relationship between configured storage, transient growth, and actual memory usage is visible without manual computation.
• Ban list growth: MAIN.bans and related lurker counters are tracked over time, so gradual ban accumulation that takes weeks to become noticeable is visible as a trend.
• Backend health vs connection failures: backend_unhealthy, backend_fail, and per-backend VBE.*.happy are collected together. The distinction between “backend is sick, no connection attempted” and “connection attempted, failed” is clear without cross-referencing separate tools.
• Dual-process stability: MGT.* counters are tracked alongside MAIN.* counters. The relationship between MGT.uptime and MAIN.uptime reveals child restarts even when the management process has been running stably.

Related guides

• Varnish monitoring checklist: the signals every production cache needs
• Varnish monitoring maturity model: from survival to expert
• Varnish thread pool exhaustion: workers all busy, queue full, sessions dropped
• Varnish sess_dropped vs req_dropped: HTTP/1 connection drops and HTTP/2 stream drops
• Varnish thread_queue_len above zero: requests waiting for a worker
• Varnish threads_limited climbing: hitting thread_pool_max
• Varnish threads_failed: the OS refusing to create worker threads
• Varnish thread pool tuning: thread_pool_min, thread_pool_max, and thread_pools
• Varnish cache hit ratio dropped: hit rate collapse and backend overload
• Varnish cache stampede: a popular object expires and the herd hits the backend
• Varnish cache_hitpass / cache_hitmiss climbing: uncacheable content bleeding to the backend
• Varnish not caching: Set-Cookie, Vary, and Cache-Control killing your hit rate