Varnish Cache Operations Guides on Netdata

How Varnish actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

Varnish Cache is a reverse HTTP proxy that serves cached content from memory. Behind that description sits a set of interacting subsystems, each with its own failure modes and saturation points. Before you can interpret a counter or diagnose an incident, you need to understand how a request moves through the process, where worker threads are consumed, how the cache store is managed, and what runs in the background.

This article covers the architecture you need before the runbooks: the dual-process split, the accept thread and bounded worker pool, the VCL state machine, the object store with its unbounded transient sibling, the ban list and its lurker, backend connection pools with health probes, and the shared-memory log.

Varnish 400 Bad Request spike: malformed requests, scanning, and smuggling

Mon, 01 Jan 0001 00:00:00 +0000

A spike in MAIN.client_req_400 means Varnish is rejecting an unusual volume of client requests at the HTTP parsing layer, before VCL runs. A low steady rate of 400s is normal. Bots, scanners, and sloppy clients are constant background radiation on any public-facing proxy. What matters is the delta from your baseline.

The threshold for concern is a sustained rate spike greater than 5x your established baseline. At that volume, the causes narrow to: active scanning or fuzzing, a broken client SDK pushing malformed requests, request-smuggling attempts probing parser discrepancies, or a configuration change that made Varnish reject traffic it previously tolerated.

Varnish backend connection reuse low: keepalive not working and slow TTFB

Mon, 01 Jan 0001 00:00:00 +0000

When Varnish reuses backend TCP connections through HTTP keepalive, cache misses skip connection setup: no TCP handshake, no optional TLS negotiation, no kernel connection-tracking overhead. When reuse collapses, every backend fetch pays that cost, and it shows up directly in time-to-first-byte.

The reuse ratio is backend_reuse / (backend_reuse + backend_conn) from varnishstat. A ratio below 50% when your backend supports keepalive means the majority of fetches are opening fresh TCP connections. This adds latency on every cache miss, increases backend CPU and connection-tracking load, and accelerates file descriptor consumption on the Varnish child process.

Varnish backend is sick: health probes, all-backends-sick, and grace

Mon, 01 Jan 0001 00:00:00 +0000

A sick Varnish backend receives zero traffic. When all backends in a director go sick, Varnish either serves stale content via grace or returns 503 on every cache miss. The health transition is governed by a threshold/window probe model, not a simple pass/fail. That distinction matters when you are reading backend.list output during an incident.

What this means

Varnish marks a backend healthy or sick based on a sliding window of probe results. A probe is an HTTP request Varnish sends to the backend at a configurable interval. The backend is healthy if at least threshold out of the last window probes returned the expected response (default 200) within the timeout. The defaults are stable across Varnish 6.x through 9.x:

Varnish backend probe configuration: threshold, window, interval, and initial

Mon, 01 Jan 0001 00:00:00 +0000

A Varnish backend probe is a periodic HTTP request that determines whether a backend receives traffic. Its parameters control detection latency, flapping behavior, and false health states.

Tighter windows detect failures faster but flap on network hiccups. Longer intervals reduce probe load but extend the detection blind spot. The default .initial prevents false-sick states at startup, but a freshly loaded VCL marks backends unhealthy until the first real probe succeeds.

Varnish backend TTFB high: the leading indicator of thread pool death

Mon, 01 Jan 0001 00:00:00 +0000

Backend time-to-first-byte (TTFB) is the time from Varnish sending a backend request to receiving the first response header byte. Varnish uses a thread-per-request model with a bounded pool. Every backend fetch holds a worker thread for the entire fetch duration: TCP connect, wait for first byte, transfer body, process headers. When TTFB rises, each fetch holds a thread longer, and the pool fills faster.

The cascade is deterministic. Backend TTFB climbs, threads accumulate at the pool ceiling, the session queue fills, and sessions start dropping. CPU and memory may look healthy while Varnish refuses connections. TTFB degrades before any availability counter moves.

Varnish backend_fail, backend_unhealthy, and backend_busy: three different backend problems

Mon, 01 Jan 0001 00:00:00 +0000

Three varnishstat counters track backend problems, and operators routinely conflate them. backend_fail, backend_unhealthy, and backend_busy each fire at a different point in the backend connection decision flow, have different root causes, and need different fixes.

The most dangerous confusion is also the subtlest: a backend can be completely offline with backend_fail sitting at zero. If the backend is probe-sick, Varnish never attempts a connection, so backend_fail never increments. Only backend_unhealthy reveals this state. An operator who monitors backend_fail alone sees a healthy system while the backend is dark.

Varnish ban list growing: O(n) lookups and the lurker falling behind

Mon, 01 Jan 0001 00:00:00 +0000

Varnish’s ban list is a linear chain of invalidation rules. Every cache lookup tests the requested object against all active bans before serving it. When the list grows, each lookup pays O(n) cost. Hit latency rises. CPU climbs. But hit ratio stays flat, backend request rate stays flat, and error counts stay flat. This makes a ban list explosion hard to detect with standard monitoring, because most teams alert on hit rate and error rate, not hit latency.

Varnish ban lurker not keeping up: contention and ban_lurker_sleep

Mon, 01 Jan 0001 00:00:00 +0000

The ban lurker is Varnish’s background invalidation worker. When it falls behind, the ban list grows and every cache lookup must test the object against all uncompleted bans. The signature pattern is MAIN.bans climbing steadily while MAIN.bans_lurker_contention rises and MAIN.bans_lurker_tested stays low or stalls entirely.

This is a slow-motion degradation. Hit rate often looks fine, but per-request latency on cache hits creeps upward because each lookup scans the full ban list. A ban list in the tens of thousands turns each cache hit into an O(n) scan.

Varnish cache hit ratio dropped: hit rate collapse and backend overload

Mon, 01 Jan 0001 00:00:00 +0000

When cache hit ratio drops, every missed request that previously served from memory now hits the backend. If your backends are sized for cached traffic, not the raw request rate, they saturate quickly. Backend TTFB rises, worker threads are held longer, the thread pool fills, and sessions drop.

The critical diagnostic signal is temporal ordering. Hit rate collapse always precedes backend degradation when the cache is the root cause. If backend TTFB rises first and hit rate falls second, the backend is the problem. If hit rate drops first and backend metrics follow, the cache stopped being effective and the backend is collateral damage. This distinction determines whether you fix VCL, storage, and invalidation logic, or investigate the origin.

Varnish cache stampede: a popular object expires and the herd hits the backend

Mon, 01 Jan 0001 00:00:00 +0000

A popular cached object hits its TTL and expires. In the next second, hundreds of concurrent requests for that object all miss simultaneously. Varnish forwards all of them to the backend, which was sized for the small fraction of traffic that normally leaks through, not a synchronized burst of identical requests. The backend slows. Worker threads pile up waiting for responses. If the backend cannot recover quickly, thread exhaustion follows and sessions start dropping.

Varnish cache_hitpass / cache_hitmiss climbing: uncacheable content bleeding to the backend

Mon, 01 Jan 0001 00:00:00 +0000

cache_hitmiss and cache_hitpass are climbing. Backend request rate is growing. Cache hit rate looks stable. This is Varnish’s learned “do not cache” mechanism silently routing traffic past the cache.

When Varnish fetches an object and determines it cannot be cached, it caches that decision itself. For the next 120 seconds (the builtin.vcl default), every request for that URL bypasses the cache and goes straight to the backend. Unlike normal cache misses, hit-for-miss requests are not coalesced: concurrent requests for the same URL each generate an independent backend fetch. The result is silent, compounding backend load that your hit-rate alert probably cannot see.

Varnish child panic: Child died signal, core dumps, and the crash loop

Mon, 01 Jan 0001 00:00:00 +0000

Child (NNNN) died signal=N means the Varnish child (worker) process crashed. The management process supervises and restarts it automatically, so a single crash is self-recovering. But each restart empties the cache, and if the child keeps crashing, caching drops to zero and backends absorb full uncached traffic.

The immediate question: one-off crash or crash loop? If the child recovers and the cache warms back up, it is a TICKET. If MAIN.uptime never stabilizes and stays far below MGT.uptime, the child is dying before the cache can warm. That is a PAGE.

Varnish Error 503 Backend fetch failed: what the error page actually means

Mon, 01 Jan 0001 00:00:00 +0000

The “Error 503 Backend fetch failed” page is the default synthetic error Varnish serves when it cannot get a usable response from any backend. It displays “Guru Meditation” and an XID identifier, but nothing about the actual failure cause. The error page is a symptom, not a diagnosis.

The root cause is always in the FetchError tag in the shared memory log. Every Varnish-synthesised 503 is preceded by a backend transaction that logged a specific FetchError string: a timeout, a premature close, a protocol violation, or a health-probe failure that left no healthy backend to try. Reading that string is the single most important diagnostic step, and the one most operators skip.

Varnish ESI errors: broken pages and workspace pressure from Edge Side Includes

Mon, 01 Jan 0001 00:00:00 +0000

Pages assembled with Edge Side Includes (ESI) can fail in ways that look like application bugs but are cache-layer problems. When MAIN.esi_errors increments, users receive partial pages with missing fragments or HTTP 500 responses from workspace exhaustion. MAIN.esi_warnings counts ESI tags that Varnish skipped, silently dropping content from otherwise valid responses.

Each <esi:include> tag in a backend response triggers a full sub-request through Varnish’s VCL pipeline. A page with five includes means six VCL cycles: the parent plus five children. Each sub-request consumes a worker thread for its full lifecycle, allocates workspace memory for headers and VCL processing, and can itself contain ESI tags that spawn further sub-requests. The recursion is bounded by max_esi_depth, but hitting that bound produces an error, not graceful degradation.

Varnish fetch_failed: backend connected but the fetch broke

Mon, 01 Jan 0001 00:00:00 +0000

The MAIN.fetch_failed counter is climbing in varnishstat. Clients are seeing intermittent 503 responses. Backend health probes report healthy, and MAIN.backend_fail is zero. The TCP connection to the backend succeeded, but the fetch itself broke.

The problem is not connectivity. It is what happens after: malformed response headers, a truncated body, broken chunked encoding, a failed gzip decompression, or a timeout between bytes. Varnish established the connection, began the HTTP transaction, and something failed before a complete response was received.

Varnish grace masking a backend outage: the ticking-clock incident

Mon, 01 Jan 0001 00:00:00 +0000

Your origin servers went down 12 minutes ago. The dashboard shows a 97% cache hit rate, zero 503s, and normal response times.

This is grace masking. Varnish is serving cached objects past their TTL because no healthy backend can refresh them. Clients see stale 200s. Monitoring sees healthy cache traffic. The outage stays invisible until grace expires on enough popular objects, at which point 503s cascade.

Grace buys time proportional to the shortest grace TTL across your working set. When that runway runs out, the hit rate can drop from 97% to near zero in minutes. The impact is deferred, not prevented.

Varnish Guru Meditation: reading the XID and tracing the failing request

Mon, 01 Jan 0001 00:00:00 +0000

The Guru Meditation page is Varnish’s default 503 response. It fires when Varnish cannot serve from cache and cannot fetch from a backend. The page carries a transaction ID (XID) that maps directly to the shared memory log entries for that failed request. The XID is the fastest path from “users are seeing 503s” to “here is the FetchError, the backend, and the VCL subroutine where the failure happened.”

The core procedure works on Varnish 5.1 and later. The vxid field was added to the VSL query language in Varnish 5.1, making it possible to filter by transaction ID directly with -q. On earlier versions, you cannot filter by XID in the query and must rely on timestamp correlation or piping varnishlog through grep.

Varnish losthdr: HTTP headers silently dropped past http_max_hdr

Mon, 01 Jan 0001 00:00:00 +0000

MAIN.losthdr increments every time Varnish drops an HTTP header because the request or response exceeded the http_max_hdr limit (default 64). No error reaches the client in many cases. The request appears to succeed, but the header is gone, and Varnish does not reveal which one without querying the shared memory log.

The consequences depend on which header was lost. A dropped Vary header causes cache poisoning: the wrong content variant gets cached and served to subsequent users for that URL. A dropped Authorization header means the backend never sees credentials, causing authentication failures or, depending on backend behavior, authentication bypass. A dropped Cache-Control header changes how the response is cached. None produce a Varnish error. They produce incorrect behavior that is difficult to trace unless you know this counter exists.

Varnish management CLI exposure: a -T bound to the world is full control

Mon, 01 Jan 0001 00:00:00 +0000

The Varnish management CLI (varnishadm, the -T flag, default port 6082) is not a monitoring endpoint. It is a full administrative control plane. Through it, an authenticated user can load arbitrary VCL, inject bans, change every runtime parameter, and stop the cache child process. If VCL inline-C is enabled, loading a VCL is code execution on the Varnish host, running as the cache child user.

When -T is bound to 0.0.0.0 or a public IP, any network actor who can reach the port and authenticate gets total control of the service. The authentication mechanism is a shared-secret (pre-shared key) handshake over a plaintext TCP connection. It provides authentication but not encryption. If the secret file is missing, readable by unauthorized users, or authentication is disabled, the CLI is either unauthenticated or trivially compromisable.

Varnish monitoring checklist: the signals every production cache needs

Mon, 01 Jan 0001 00:00:00 +0000

Varnish Cache is a reverse HTTP proxy that serves cached content from memory. It uses a dual-process architecture: a management process (root-owned, handles VCL compilation, child supervision, and the CLI) and a worker/child process (drops privileges, handles all cache operations). The management process restarts the child automatically on crash, so a child crash is not always a full outage. Repeated child restarts indicate a systemic problem.

Varnish’s concurrency model is thread-per-request with a bounded pool. When the pool is exhausted and the queue is full, sessions are dropped. Most production incidents trace back to thread pool exhaustion, storage pressure, ban list growth, transient memory growth, or backend failure cascades.

Varnish monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Most Varnish deployments start with the same three questions: is the process alive, is the hit ratio acceptable, are backends reachable. That is enough to catch a full outage. It is not enough to catch the slow failures that actually dominate production incidents: thread pool exhaustion with idle CPU, ban list growth turning every cache hit into a linear scan, transient storage OOM with no SMA counter movement, or grace mode silently masking a dead backend until the stale content expires.

Varnish n_lru_nuked vs n_expired: healthy eviction or an undersized cache

Mon, 01 Jan 0001 00:00:00 +0000

Every Varnish cache evicts objects. The operational question is whether those evictions are healthy housekeeping or evidence that storage is too small for the working set. Two counters hold the answer. MAIN.n_expired tracks objects that aged out past their TTL. MAIN.n_lru_nuked tracks objects forcibly evicted from storage because a new object needed the space.

The distinction matters because nuking is not inherently a problem. A right-sized cache continuously nukes the unpopular tail of its working set, which is correct LRU behavior. The problem arrives when nuking removes objects that would still serve hits, driving cache hit rate down and pushing more traffic to backends.

Varnish n_vcl accumulation: cold VCLs never discarded

Mon, 01 Jan 0001 00:00:00 +0000

MAIN.n_vcl is climbing steadily in varnishstat. Each deploy loads a new VCL, and the count never comes back down. varnishadm vcl.list shows dozens of VCLs in “available” state, most of them cold. Process RSS is creeping upward. This is VCL accumulation: automated reload pipelines that load new VCLs on every deploy but never run vcl.discard on old ones.

Cold VCLs (no longer serving traffic) have released some runtime resources per the VCL temperature system, but the compiled shared object remains loaded until explicitly discarded. A deployment pipeline that only runs vcl.load and vcl.use without vcl.discard leaves every old VCL resident indefinitely.

Varnish not caching: Set-Cookie, Vary, and Cache-Control killing your hit rate

Mon, 01 Jan 0001 00:00:00 +0000

Your hit rate is low and backend_req tracks client_req almost one-to-one. Varnish is up, storage has room, there are no error spikes, but cache_hitpass or cache_hitmiss dominates the cache outcome counters. The cache is running but not caching.

The root cause is almost never Varnish itself. It is the interaction between backend response headers and the built-in VCL rules that determine cacheability. Three header problems account for the vast majority of cases: Set-Cookie on every response, Cache-Control: private or no-cache on public content, and an overly broad Vary header that fragments the cache into non-shareable variants.

Varnish pass vs miss: why s_pass and cache_miss are not the same thing

Mon, 01 Jan 0001 00:00:00 +0000

A Varnish cache_miss and an s_pass both result in a backend fetch, and that surface similarity is where the confusion starts. Teams see backend request rates climbing, glance at the counters, and reach for the wrong fix. A miss is a cache lookup that found nothing and will try to cache the response. A pass is a deliberate VCL decision to bypass the cache entirely via return(pass). The difference matters because the two paths have different storage behavior, different tuning levers, and different operational consequences.

Varnish purge vs ban vs xkey: choosing an invalidation method that scales

Mon, 01 Jan 0001 00:00:00 +0000

Varnish offers three mechanisms for invalidating cached objects: purge, ban, and the xkey VMOD (surrogate-key invalidation). Each has a fundamentally different cost model. Purge is O(1) and immediate but works only on a single exact hash. Bans are expression-based and flexible but accumulate in a list that every cache lookup must evaluate. xkey operates on secondary key indexes and avoids list growth entirely, but the open-source implementation has known scaling limits.

Varnish req.* vs obj.* bans: why req-level bans never leave the list

Mon, 01 Jan 0001 00:00:00 +0000

req.* bans persist on the ban list because the ban lurker thread has no request context and cannot evaluate them asynchronously. The list grows, lookup latency degrades across every request, and the fix is the same in nearly every case: rewrite req-based bans as obj-based bans so the lurker can process them in the background. For the broader failure pattern, see Varnish ban list growing.

What it is and why it matters

A ban is a regex-based cache invalidation rule. When you issue ban('req.url ~ "/product/123"') or ban('obj.http.x-url ~ "/product/123"'), Varnish adds the expression to a linked list. Every cached object is tested against outstanding bans, either by the lurker in the background or by a worker thread at lookup time.

Varnish sc_rapid_reset: the HTTP/2 Rapid Reset DDoS (CVE-2023-44487)

Mon, 01 Jan 0001 00:00:00 +0000

sc_rapid_reset incrementing on a Varnish instance means the HTTP/2 Rapid Reset rate limiter has closed at least one session for exceeding its configured reset budget. This is the primary detection signal for CVE-2023-44487, the HTTP/2 Rapid Reset DDoS disclosed in October 2023. The attack exploits an asymmetry in HTTP/2: a client can open hundreds of streams per connection and immediately cancel each one with an RST_STREAM frame, forcing the server to do real work (stream setup, header parsing, resource allocation) at near-zero cost to the attacker.

Varnish sess_dropped vs req_dropped: HTTP/1 connection drops and HTTP/2 stream drops

Mon, 01 Jan 0001 00:00:00 +0000

Two Varnish counters track the worst outcome a cache can produce: a client gets nothing. MAIN.sess_dropped counts HTTP/1 sessions dropped because the worker thread queue was full. MAIN.req_dropped counts HTTP/2 streams and other request types dropped for the same reason. Both mean Varnish refused to serve a client because no thread was available and the bounded queue was at its limit.

The distinction matters because HTTP/1 and HTTP/2 multiplex differently. An HTTP/1 client occupies a connection. An HTTP/2 client multiplexes many concurrent streams over a single connection. When Varnish runs out of threads, the rejection mechanism differs by protocol, and so does the counter that increments. Teams that monitor only sess_dropped can see zero drops while HTTP/2 streams are silently refused.

Varnish sess_fail: session accept failures at the front door

Mon, 01 Jan 0001 00:00:00 +0000

When MAIN.sess_fail starts incrementing in Varnish, new TCP connections are failing at the accept() call. Varnish never brings them into the worker pipeline. Clients experience connection resets, timeouts, or refused connections. Unlike sess_dropped (where the connection was accepted but the thread queue was full), sess_fail means Varnish never got far enough to process the request.

The counter is an aggregate. Since Varnish 6.1 , it decomposes into sub-counters that isolate the cause: sess_fail_emfile for file descriptor exhaustion, sess_fail_econnaborted for client-side aborts, sess_fail_enomem for memory pressure, and several others. Reading only the aggregate counter is a common diagnostic mistake.

Varnish session close reasons: reading the sc_* counters

Mon, 01 Jan 0001 00:00:00 +0000

The MAIN.sc_* counters in Varnish record why every client session ended. Each close reason is a DIAG-level counter visible through varnishstat. The distribution across these counters is one of the fastest triage signals available: it tells you whether sessions are ending normally or whether the client, the network, or Varnish itself is causing problems.

The sc_* family has grown across versions. Some counters shifted accounting between releases, and several error counters are benign under certain traffic patterns. Treating every nonzero error counter as an incident leads to alert fatigue; treating them all as noise means missing real problems.

Varnish shared-memory log overruns: losing the evidence during an incident

Mon, 01 Jan 0001 00:00:00 +0000

Cache hit rate dropped, backend request rate spiked, users saw elevated 503s. You need to reconstruct what happened from Varnish request logs: which URLs triggered the problem, what backend errors were returned, how long fetches took. You reach for varnishncsa output or varnishlog traces and find gaps, partial records, or nothing at all.

Varnish served traffic throughout the incident. The logging subsystem quietly discarded data when you needed it most.

Varnish slow responses: hit latency, miss latency, and why varnishstat has no timing

Mon, 01 Jan 0001 00:00:00 +0000

Users report slow responses through Varnish. You open varnishstat and look for a latency counter. There isn’t one. Varnish’s counter subsystem maintains event counts and gauges: cache_hit, cache_miss, threads, backend_fail. None of them measure how long a request took.

This is architectural. Varnish writes per-request timing into the shared memory log (VSL) as Timestamp tags with per-phase resolution. The tools to read them are varnishlog, varnishncsa, and varnishhist. If you are diagnosing slow responses using varnishstat alone, you have no timing data.

Varnish SMA c_fail: allocation failures and malloc fragmentation

Mon, 01 Jan 0001 00:00:00 +0000

SMA.<name>.c_fail counts storage allocation failures after LRU eviction. Under normal conditions it stays at zero. When it climbs, memory management is wrong, and the cause is not always “the cache is full.”

The deceptive variant is the malloc stevedore. Over weeks of object churn, the underlying allocator (jemalloc or glibc malloc) fragments the heap. Varnish reports free bytes in SMA.<name>.g_space, but those bytes are scattered across many small free regions that cannot satisfy a contiguous allocation. Varnish evicts objects, retries, and fails. c_fail increments while g_space insists there is room.

Varnish storage full: the LRU nuke storm and cache thrashing

Mon, 01 Jan 0001 00:00:00 +0000

Varnish cache hit ratio is declining and backend request rate is climbing. Storage utilization is near 100%. MAIN.n_lru_nuked is incrementing steadily. You are in a storage exhaustion cascade: the cache is too small for the working set, and every new object forces Varnish to evict an existing one via LRU. Evicted objects get re-requested, miss the cache, trigger a backend fetch, and the cycle repeats.

As SMA.<name>.g_space approaches zero, the LRU eviction rate accelerates. With malloc storage, once free space is exhausted, allocation failures (SMA.<name>.c_fail) begin. With file storage, the SMF counters track the same fill dynamics; system-level impact may differ because the OS manages which mmap’d pages stay resident, but the cache behavior is the same: storage fills, nuking begins, hit ratio degrades.

Varnish storage sizing: malloc vs file, and why -s malloc,ALL-RAM kills you

Mon, 01 Jan 0001 00:00:00 +0000

The -s malloc startup parameter controls how much memory Varnish allocates for cached object storage. It does not control total process memory. That distinction is the single most expensive misunderstanding in Varnish operations.

When you set -s malloc,32G on a 32 GB machine, you have allocated 32 GB for object storage and left nothing for the operating system, the Varnish process itself, worker thread stacks, per-request workspace, transient storage, and allocator fragmentation. The kernel OOM killer sees a process consuming all available memory and terminates it. Varnish restarts, the management process reloads the child, the cache warms from zero, and the cycle repeats if the configuration has not changed.

Varnish thread pool exhaustion: workers all busy, queue full, sessions dropped

Mon, 01 Jan 0001 00:00:00 +0000

Varnish is dropping client connections and CPU looks idle. The process is running, the management CLI responds, and backends are technically up, but users see connection resets or stream resets. This is thread pool exhaustion: all worker threads are busy, the request queue is full, and Varnish has started refusing traffic.

Varnish’s concurrency model is thread-per-request with a bounded pool. Each pool holds between thread_pool_min and thread_pool_max threads, and Varnish runs thread_pools pools (default 2). When a request arrives and no idle worker is available, it enters a bounded queue (thread_queue_limit, default 20 per pool). When the queue is full, the session or stream is dropped.

Varnish thread pool tuning: thread_pool_min, thread_pool_max, and thread_pools

Mon, 01 Jan 0001 00:00:00 +0000

Varnish uses a thread-per-request concurrency model with a bounded pool. Every client request occupies a worker thread for its entire lifecycle, from accept through response delivery. When the pool is full and the overflow queue overflows, sessions are dropped.

The three parameters that control this behavior, thread_pools, thread_pool_min, and thread_pool_max, determine how Varnish responds to load spikes, slow backends, and idle periods. The dominant factor in thread pool sizing is backend response time, not request rate. A cache miss that holds a thread for 2 seconds consumes the same pool capacity as thousands of sub-millisecond cache hits. Slow backends are the single most common cause of thread pool exhaustion, and they make thread_pool_max the parameter that matters most under stress.

Varnish thread_queue_len above zero: requests waiting for a worker

Mon, 01 Jan 0001 00:00:00 +0000

MAIN.thread_queue_len is the point-in-time count of client sessions sitting in Varnish’s bounded worker queue, waiting for an idle thread. In normal operation it is zero. Any sustained nonzero value means every worker thread across all pools is busy and incoming requests are piling up in the last buffer before Varnish starts dropping them.

The thread pool has a cliff-edge failure curve: performance is fine until the pool is saturated, then requests queue, then the queue fills, then sessions are dropped. The distance between “queue length is 1” and “clients are getting connection resets” can be seconds if the queue limit is small (the default thread_queue_limit is 20 per pool).

Varnish threads_failed: the OS refusing to create worker threads

Mon, 01 Jan 0001 00:00:00 +0000

When MAIN.threads_failed is nonzero, the Linux kernel is refusing Varnish’s pthread_create() calls. This is an OS-level resource limit preventing the worker thread pool from growing, not a Varnish configuration problem.

Do not confuse this with MAIN.threads_limited, which increments when Varnish declines to create a thread because thread_pool_max has been reached. threads_limited means Varnish chose not to create the thread. threads_failed means Varnish tried and the OS said no. The remediation differs: threads_limited requires raising a Varnish parameter; threads_failed requires fixing OS-level limits.

Varnish threads_limited climbing: hitting thread_pool_max

Mon, 01 Jan 0001 00:00:00 +0000

MAIN.threads_limited increments each time Varnish wanted to create a new worker thread but thread_pool_max prevented it. Movement during a traffic burst is expected. A sustained nonzero rate during steady-state operation means the configured thread pool ceiling is a binding constraint on concurrency, and if the bounded queue behind the pool fills, Varnish will drop sessions.

The default thread_pool_max is 5000 threads per pool. With the default thread_pools value of 2, the absolute ceiling is 10,000 worker threads. For most workloads that ceiling is high enough that the counter stays at zero. The problem arises when backends are slow: each thread is held longer per request, effective concurrency drops, and the pool fills at a request rate that would be trivial if backends responded in single-digit milliseconds.

Varnish Too many open files: EMFILE, sess_fail_emfile, and the FD ceiling

Mon, 01 Jan 0001 00:00:00 +0000

sess_fail_emfile is climbing and new client connections are being refused. The Varnish child process has hit its file descriptor ceiling. This is a cliff-edge failure: once the FD limit is exhausted, Varnish cannot accept new connections, cannot open backend connections, and can panic if internal operations that require FDs fail. There is no graceful degradation and no queuing.

The default ulimit -n of 1024 on most Linux distributions is far too low for production Varnish. The Varnish package’s default systemd unit sets LimitNOFILE=131072, but inherited or misconfigured environments often leave the lower default in place.

Varnish transient storage OOM: the unbounded memory path that kills the process

Mon, 01 Jan 0001 00:00:00 +0000

Varnish dies. The OOM killer takes it. No panic, no crash, no VCL error in the logs. Process RSS was far above your configured -s malloc,8G storage, and the kernel reclaimed memory the only way it knows how. This is the transient storage OOM path: the number-one missed signal in Varnish operations.

The problem is structural. Varnish has two storage paths: your configured cache storage (malloc or file, capped at a known size) and transient storage. Transient storage holds objects that will never be cached: pass responses, hit-for-pass, hit-for-miss, and piped traffic. By default, transient storage uses malloc with no upper limit. A pass storm, a batch of large uncacheable responses, or a slow leak over weeks can push process RSS past the configured storage size and into system memory the kernel will not let you keep.

Varnish unauthorized PURGE/BAN: cache invalidation as an attack surface

Mon, 01 Jan 0001 00:00:00 +0000

Varnish treats PURGE and BAN as ordinary HTTP methods. It has no built-in VCL that handles, gates, or rejects them. If your custom VCL handles these methods but does not check an ACL first, anyone who can reach your Varnish listener can invalidate cached objects. Unauthenticated Varnish PURGE has been reported as a valid security finding against major platforms through bug bounty programs.

The consequences fall into two categories. Mass invalidation forces a cache stampede: concurrent requests miss simultaneously and hit backends sized for cached traffic. Selective invalidation creates a cache poisoning window: an attacker purges a specific URL, then races to have a malicious response cached before legitimate traffic refills it.

Varnish VCL compilation failed: reload rejected and the old VCL still live

Mon, 01 Jan 0001 00:00:00 +0000

You deployed a new VCL file, ran systemctl reload varnish or varnishadm vcl.load, and the reload was rejected. The VCC-compiler printed an error with a line and column number, and the management process refused to load the new configuration. The previous VCL is still serving traffic.

This is not an outage. Varnish reloads are hitless: the new VCL is compiled and loaded into memory before activation. When compilation fails, the old VCL stays in place and no client sees an interruption. But your change did not land. Until you fix the compilation error and retry, the running configuration is whatever was active before the failed deploy.

Varnish vcl_fail: VCL runtime errors during request processing

Mon, 01 Jan 0001 00:00:00 +0000

MAIN.vcl_fail is incrementing. The VCL loaded and compiled successfully at vcl.load time, but something is breaking at runtime while real requests flow through the VCL subroutines.

The vcl_fail counter (Varnish 6.0 and later) counts failures that prevented VCL from completing. Each increment means a request could not finish its normal VCL lifecycle. On the client side, the request is diverted to vcl_synth with a 503 status. On the backend side, the fetch fails. In both cases the user gets an error response instead of the content they asked for.

Varnish workspace overflow types: client, backend, thread, and session

Mon, 01 Jan 0001 00:00:00 +0000

Varnish allocates small, fixed-size memory regions called workspaces to hold HTTP headers, VCL string operations, and intermediate request state during processing. Each stage of the request lifecycle draws from a different workspace type, with its own size parameter and overflow counter. When any workspace runs out of room, Varnish cannot finish processing the request and returns HTTP 500 to the client. This is a 500, not a 503: the backend did not fail. Varnish itself ran out of working memory.

Varnish workspace_client overflow: 500 errors from oversized headers and cookies

Mon, 01 Jan 0001 00:00:00 +0000

HTTP 500 responses from Varnish when the backend is healthy, the object is in cache, the thread pool is not saturated, and the 503 backend-fetch path is not involved. The 500s come from Varnish itself, and they cluster on requests carrying large Cookie headers, long URLs, deep Via or X-Forwarded-For chains, or responses where VCL has added many synthetic headers.

The root cause is per-request workspace exhaustion. Varnish allocates a fixed block of memory (workspace_client) for each client request to hold the parsed HTTP object, headers, and VCL string operations. When request headers plus any data VCL pushes into that workspace exceed the allocation, the request fails with a 500.