$ guides / varnish / varnish-storage-full-lru-nuking ▌

Operations Guides

Varnish storage full: the LRU nuke storm and cache thrashing

Varnish cache hit ratio is declining and backend request rate is climbing. Storage utilization is near 100%. MAIN.n_lru_nuked is incrementing steadily. You are in a storage exhaustion cascade: the cache is too small for the working set, and every new object forces Varnish to evict an existing one via LRU. Evicted objects get re-requested, miss the cache, trigger a backend fetch, and the cycle repeats.

As SMA.<name>.g_space approaches zero, the LRU eviction rate accelerates. With malloc storage, once free space is exhausted, allocation failures (SMA.<name>.c_fail) begin. With file storage, the SMF counters track the same fill dynamics; system-level impact may differ because the OS manages which mmap’d pages stay resident, but the cache behavior is the same: storage fills, nuking begins, hit ratio degrades.

The critical diagnostic question is not “is storage full?” but “is nuking hurting the hit ratio?” Some nuking is healthy. The LRU is designed to evict cold objects from the tail of the popularity distribution. The problem starts when active, frequently-requested objects are being evicted and immediately re-fetched. Correlate n_lru_nuked rate with cache hit rate decline before acting.

What this means

Varnish stores cached objects in a storage backend (malloc, file, or deprecated persistent). Each storage segment tracks g_bytes (bytes in use), g_space (bytes available), and c_fail (allocation failure count). When g_space reaches zero, Varnish cannot store a new object without first evicting an existing one.

The eviction mechanism walks the LRU list and removes (“nukes”) the least recently accessed object whose TTL has not expired. This is distinct from natural TTL expiry, tracked by MAIN.n_expired. When nuking dominates over expiry, the cache is storage-bound rather than TTL-bound. The ratio n_lru_nuked / (n_lru_nuked + n_expired), computed on rates over a recent window, above 0.5 indicates storage pressure is the primary eviction driver.

The cascade unfolds as follows:

flowchart TD
    A["Storage fills, g_space near 0"] --> B["New objects force LRU eviction"]
    B --> C["n_lru_nuked spikes"]
    C --> D["Evicted objects get re-requested"]
    D --> E["cache_miss rises"]
    D --> F["backend_req rises"]
    E --> G["Hit ratio drops"]
    F --> H["Backend load increases"]
    H --> I["Thread pressure, possible drops"]

Cache thrashing occurs when the eviction rate exceeds the working set’s ability to stay resident. Objects are evicted before their next request arrives, so they miss, get re-fetched from the backend, get stored again, and then get evicted again. The cache is doing work but providing no acceleration. Backend request rate approaches client request rate, and the cache becomes an expensive pass-through.

A second failure path exists through transient storage. Objects marked uncacheable (beresp.uncacheable = true), pass transactions, and hit-for-miss objects consume SMA.Transient, which is unbounded by default. If transient storage grows without limit, it competes with the primary stevedore for system memory and can trigger OOM kills before the primary storage ever reports exhaustion.

Common causes

Cause	What it looks like	First thing to check
Working set larger than cache	`n_lru_nuked` sustained, hit ratio declining gradually, `g_space` near zero	Compare `n_lru_nuked` rate to `n_expired` rate
Large objects disproportionate to storage	Few objects but `g_bytes` near limit, specific URL patterns in `varnishtop`	`varnishtop -I ObjHeader:Content-Length`
TTLs too long for storage budget	High `n_object`, stable object count, slow fill then sudden thrashing	Review TTL distribution in VCL and backend headers
Malloc fragmentation	`g_space` shows available bytes but `c_fail` increments	Compare `g_space` to actual allocation success rate
Transient storage growth	Process RSS exceeds configured storage, `SMA.Transient.g_bytes` growing	`varnishstat -1 -f SMA.Transient.g_bytes`

Quick checks

All read-only and safe to run during an active incident.

# Storage utilization and allocation failures for all stevedores
varnishstat -1 -f 'SMA.*.g_bytes' -f 'SMA.*.g_space' -f 'SMA.*.c_fail'

# For file-backed storage, use SMF prefix instead
varnishstat -1 -f 'SMF.*.g_bytes' -f 'SMF.*.g_space' -f 'SMF.*.c_fail'

# LRU eviction rate vs natural TTL expiry
varnishstat -1 -f MAIN.n_lru_nuked -f MAIN.n_expired

# Hit ratio components
varnishstat -1 -f MAIN.cache_hit -f MAIN.cache_miss -f MAIN.cache_hitpass

# Backend request pressure vs client requests
varnishstat -1 -f MAIN.backend_req -f MAIN.client_req

# Content-Length distribution of cached objects (by frequency, not size)
varnishtop -I ObjHeader:Content-Length

# Transient storage growth (common OOM path)
varnishstat -1 -f 'SMA.Transient.*'

# Current object count
varnishstat -1 -f MAIN.n_object

How to diagnose it

Confirm storage is actually full. Check SMA.<name>.g_space. If it is near zero, storage is exhausted. Compute utilization as g_bytes / (g_bytes + g_space). Above 90% sustained warrants investigation. If g_space is substantial but c_fail is nonzero, suspect malloc fragmentation.
Determine whether nuking is harmful. Take two readings of MAIN.n_lru_nuked, MAIN.cache_hit, and MAIN.cache_miss ten seconds apart. Compute the rates. cache_hit and cache_miss are cumulative counters, so compare their delta rates, not absolute values. If n_lru_nuked is incrementing but the hit rate (cache_hit / (cache_hit + cache_miss)) is stable, the LRU is evicting cold objects. This is healthy. If n_lru_nuked is incrementing and hit rate is declining, hot objects are being evicted. This is the nuke storm.
Check the eviction ratio. Compare the n_lru_nuked rate to the n_expired rate over the same window. If n_lru_nuked / (n_lru_nuked + n_expired) exceeds 0.5, the cache is storage-bound, not TTL-bound. Compute this on deltas, not cumulative values.
Identify what is consuming storage. Run varnishtop -I ObjHeader:Content-Length to find the distribution of Content-Length values among cached objects. Look for large values appearing with any frequency. A handful of multi-megabyte responses can fill a cache that should hold hundreds of thousands of small objects. Check whether these objects should be cached at all.
Check transient storage. Run varnishstat -1 -f 'SMA.Transient.*'. If SMA.Transient.g_bytes is growing, uncacheable traffic (pass, hit-for-miss, hit-for-pass) is consuming memory outside the primary storage budget. This can cause OOM before primary storage reports exhaustion.
Check for allocation failures. If SMA.<name>.c_fail is nonzero, the allocator failed even after LRU eviction. This means fragmentation is preventing usable allocations, or insertion rate exceeds eviction rate.
Correlate with backend load. Check whether MAIN.backend_req is rising in proportion to MAIN.cache_miss. If backend request rate is climbing and backend response times are degrading, the nuke storm is cascading toward backend overload and potential thread pool exhaustion.

Metrics and signals to monitor

Signal	Why it matters	Alert threshold
`SMA.<name>.g_space`	Bytes available in primary storage	Sustained below 10% of total
`SMA.<name>.c_fail`	Allocation failures after eviction	Any nonzero value
`MAIN.n_lru_nuked` rate	Objects forcefully evicted for new ones	Rate sustained and exceeding `n_expired` rate
Nuke-to-expiry ratio	Whether storage or TTL drives eviction	Above 0.5 sustained indicates storage-bound
Cache hit rate	Cache effectiveness	Declining trend correlated with nuking rate
`MAIN.backend_req` rate	Load reaching backends	Rising proportionally to `cache_miss` rate
`SMA.Transient.g_bytes`	Memory for uncacheable objects	Monotonic growth
`MAIN.n_object`	Current cached object count	Sudden drop may indicate mass invalidation

Fixes

Increase storage allocation

The most direct fix. Increase the -s malloc,SIZE or -s file,SIZE parameter. This requires a Varnish restart, which empties the cache. Plan for a cold-cache warmup period where backend load will be elevated.

Reserve 20-30% of system RAM for non-storage use: the OS, Varnish process overhead, thread stacks, workspace memory per request, and transient storage. Setting -s malloc to all available RAM is a common cause of OOM kills.

Exclude large objects from caching

If varnishtop -I ObjHeader:Content-Length shows objects consuming disproportionate storage, exclude them in VCL. In vcl_backend_response, check beresp.http.Content-Length and return pass for objects above a threshold. This frees storage for the high-cardinality small objects that drive hit ratio.

Large media files, software downloads, and API responses with oversized payloads are common culprits. These often have low cache value because they are requested infrequently relative to their size.

Reduce TTLs

If TTLs are longer than necessary for the available storage budget, objects accumulate and fill the cache before expiring. Shorter TTLs mean faster natural expiry (n_expired), which frees space without nuking. Review beresp.ttl assignments in VCL and Cache-Control / Expires headers from the backend.

This is a tradeoff: shorter TTLs mean more frequent backend fetches for objects that would otherwise be served from cache. Tune toward the point where n_lru_nuked rate drops to near zero and n_expired dominates.

Address malloc fragmentation

With malloc storage, long-running Varnish processes can develop fragmentation. g_space reports available bytes, but the allocator cannot find contiguous blocks for new objects. c_fail increments even though g_space looks sufficient.

A restart defragments by rebuilding the cache from scratch, but this is disruptive and empties the cache. If fragmentation recurs regularly, consider switching to file storage, which handles allocation differently, or evaluate whether object size variance in your workload is the root cause.

Cap transient storage

By default, transient storage (SMA.Transient) is unbounded. If pass traffic, hit-for-miss objects, or streaming responses are consuming significant memory, configure a sized transient stevedore: -s Transient=malloc,SIZE. This prevents uncacheable traffic from competing with primary storage for system memory.

If SMA.Transient.g_bytes is the growth driver rather than primary storage, investigate why traffic is being passed. Check MAIN.cache_hitpass and MAIN.cache_hitmiss rates. Application changes that add Set-Cookie headers or Cache-Control: private to previously cacheable responses are a common root cause.

Prevention

Right-size storage from the start. Allocate enough storage to hold the working set with 10-20% headroom. Estimate the working set from traffic patterns and average object size, not from total system RAM.
Monitor the nuke-to-expiry ratio, not just utilization. Storage at 95% is normal for a well-utilized cache. Alert on nuke rate correlated with hit ratio decline, not on utilization percentage alone.
Alert on c_fail. Any nonzero value means the allocator failed after eviction. This is never healthy.
Monitor transient storage. Track SMA.Transient.g_bytes or compare process RSS to configured storage size. Monotonic growth is an OOM path.
Watch for large object drift. If average object size increases over time (application changes, new endpoints, uncompressed responses), storage fills faster. Monitor g_bytes / g_alloc for average object size trends.

How Netdata helps

Netdata provides per-second resolution on all Varnish counters, which matters because the transition from healthy nuking to cache thrashing can happen in under a minute with malloc storage.

Per-second SMA and SMF counters show g_space decline and c_fail increments in real time, rather than minutes after hit ratio has already collapsed.
Correlated dashboards display n_lru_nuked, cache_hit, cache_miss, and backend_req on the same timeline. Visual correlation between nuke rate spikes and hit ratio drops is the fastest way to confirm a storage-bound problem versus a TTL or VCL issue.
Transient storage monitoring through SMA.Transient.g_bytes catches the silent OOM path that primary storage metrics miss.
Composite alerting across storage, hit ratio, and backend request rate distinguishes “cache is full and working correctly” from “cache is thrashing and backend is at risk.”
Anomaly detection on storage utilization and nuke rate catches gradual trends (working set growth, large object drift) before they become incidents, without static thresholds that fire too early or too late.