$ guides / varnish / varnish-lru-nuked-vs-expired ▌

Operations Guides

Varnish n_lru_nuked vs n_expired: healthy eviction or an undersized cache

Every Varnish cache evicts objects. The operational question is whether those evictions are healthy housekeeping or evidence that storage is too small for the working set. Two counters hold the answer. MAIN.n_expired tracks objects that aged out past their TTL. MAIN.n_lru_nuked tracks objects forcibly evicted from storage because a new object needed the space.

The distinction matters because nuking is not inherently a problem. A right-sized cache continuously nukes the unpopular tail of its working set, which is correct LRU behavior. The problem arrives when nuking removes objects that would still serve hits, driving cache hit rate down and pushing more traffic to backends.

What the two counters mean

Both counters are available in varnishstat. On Varnish 4.x and later they carry the MAIN. prefix. On Varnish 3.x the prefix is absent.

Counter	Definition
`MAIN.n_expired`	Objects removed from cache because they reached their TTL and aged out naturally
`MAIN.n_lru_nuked`	Objects forcefully evicted from storage to make room for a new object
`MAIN.n_lru_limited`	Times more storage space was needed but the `nuke_limit` was reached before enough space was freed
`MAIN.n_lru_moved`	Move operations on the LRU list, where accessed objects are promoted toward the head

n_expired is the healthy baseline. Every object with a finite TTL eventually expires. If your cache only ever increments n_expired and never touches n_lru_nuked, storage is large enough to hold the entire working set through each object’s full TTL. That is ideal but not always cost-effective. For most workloads, some nuking is expected and normal.

n_lru_nuked increments only when storage is full and Varnish must evict a live, not-yet-expired object to insert a new one. The LRU algorithm selects the least recently used object from the tail of the list. This keeps the cache populated with the hottest content, but degrades hit rate when the working set exceeds the allocated storage.

n_lru_limited is the escalation signal. When storage is under enough pressure that Varnish would need to nuke more objects than the nuke_limit parameter allows (default 50 in Varnish 6.0 LTS), the fetch fails and the client receives a 503. This counter tells you nuking has progressed from a cache efficiency problem to an availability problem.

n_lru_moved is not an eviction counter. It tracks how often objects are repositioned on the LRU list because they were accessed. High values are normal cache activity.

How eviction works

When a client request results in a cache miss, Varnish fetches the object from the backend and stores it. The storage allocator checks whether enough free space exists in the configured stevedore (typically malloc or file).

If free space exists, the object is inserted. If not, Varnish walks the LRU list from the tail, evicting objects one by one until enough space is freed. Each evicted object increments n_lru_nuked. This happens synchronously during the fetch path: the worker thread inserting the new object performs the nuking.

Objects expire through a separate path. The expiry thread removes objects whose TTL, grace, and keep timers have all elapsed. Each removal increments n_expired. This happens asynchronously and does not block request processing.

These two counters measure fundamentally different exit paths from the cache:

flowchart TD
    A[Object in cache] --> B{Why removed?}
    B -->|TTL elapsed| C[n_expired]
    B -->|Storage full, new fetch| D[n_lru_nuked]
    C --> E[Healthy eviction]
    D --> F{Hit rate declining?}
    F -->|No| G[Healthy: LRU pruning tail]
    F -->|Yes| H[Undersized: working set exceeds storage]

A race condition in the nuking path is worth knowing about. A worker thread can request LRU nuking to free space, but a competing thread can claim the freed space before the first thread uses it. This means n_lru_nuked can increment even when the total free space across the stevedore would appear sufficient. The counter reflects nuking attempts, not a clean capacity calculation.

The discriminator ratio

A single counter value tells you nothing in isolation. The ratio that separates TTL-bound eviction from storage-bound eviction is:

n_lru_nuked / (n_lru_nuked + n_expired)

Below 0.5: the cache is TTL-bound. Most objects leave the cache by expiring naturally. Storage is adequate for the working set.
Above 0.5: the cache is storage-bound. More objects leave by forceful eviction than by natural expiry. The working set does not fit in the allocated storage.

# Read both counters in one shot
varnishstat -1 -f MAIN.n_lru_nuked -f MAIN.n_expired

Take two readings several seconds apart and compute the deltas. The ratio of the deltas is more informative than the ratio of cumulative totals, because cumulative totals include the initial cache warmup period where n_lru_nuked was zero.

Hit rate is the decisive signal

The ratio tells you whether the cache is storage-bound, but not whether that matters. A cache can be storage-bound and still perform well. The decisive signal is whether the cache hit rate is declining.

Moderate nuking with a stable hit rate is healthy. The LRU is evicting the unpopular tail: objects cached once, rarely re-requested, and unlikely to generate hits. The hot objects stay. This is the LRU doing exactly what it should.

Nuking with a declining hit rate is the problem. Objects that would still serve hits are evicted before they are re-requested. Each evicted popular object becomes a cache miss on its next request, generating a backend fetch. The backend fetch retrieves the same object that was just nuked, inserts it (nuking something else), and the cycle repeats. This is cache thrashing.

# Watch nuking and cache outcomes together
varnishstat -1 -f MAIN.n_lru_nuked -f MAIN.n_expired -f MAIN.cache_hit -f MAIN.cache_miss -f MAIN.cache_hitpass

If n_lru_nuked is climbing and cache_miss is climbing while cache_hit stays flat or drops, the cache is undersized. The working set is larger than what the stevedore can hold.

Another useful correlation: check whether SMA.{name}.g_space is near zero when nuking is active. For malloc storage, use SMA.s0.g_space. For file storage, use SMF.s0.g_space. If storage shows meaningful free space but nuking is still happening, the issue may be malloc fragmentation rather than genuine capacity shortage. Fragmentation causes g_space to report bytes that cannot be allocated as contiguous blocks.

When nuking escalates to 503s

Varnish caps the number of objects a single fetch can nuke via the nuke_limit parameter. The default is 50 in Varnish 6.0 LTS. If a fetch needs to free space and hits this limit before enough space is freed, the fetch fails.

Two consequences follow:

MAIN.n_lru_limited increments. This counter is the hard signal that storage pressure has progressed beyond cache inefficiency into request failure.
The client receives a 503 response. With streaming enabled (the default in Varnish 4.x+), Varnish may begin client delivery in parallel with the backend fetch, so the failure can manifest as a truncated response or a “transfer closed with outstanding read data remaining” error rather than a clean 503.

A version history note: in Varnish versions before 4.1.7, the nuke_limit parameter was not enforced. The fix introduced in 4.1.7-beta1 began honoring the limit, which caused new 503 errors on caches with heavy nuking. If you see n_lru_limited incrementing after an upgrade from an older version, the parameter change is likely the cause. The workaround is to raise nuke_limit, but the real fix is more storage or a smaller working set.

Where it shows up in production

Several real-world patterns produce nuking:

Traffic growth without storage resize. The working set grew as the site added content or traffic, but the -s malloc,SIZE allocation was not updated. Nuking begins gradually and accelerates as the working set exceeds capacity.
Large objects entering the cache. A few large responses such as video segments, API payloads, or PDFs can consume disproportionate storage. Use varnishtop -I ObjHeader:Content-Length to identify the heaviest objects by response size. Note that chunked responses do not carry a Content-Length header.
Object overhead with many small objects. Each cached object carries metadata (objecthead, objectcore, object structs) beyond its body bytes. A cache holding many small objects may nuke aggressively because per-object overhead consumes storage faster than body size alone would suggest.
Cache warmup after restart. After a child process restart, the cache is empty and fills rapidly. Nuking is zero initially while storage is empty, then spikes once storage fills. This is transient and should subside as the cache reaches steady state.
Shortlived objects and transient storage. Objects with TTL below the shortlived parameter threshold go to transient storage rather than the main stevedore. The default is 10 seconds. If many short-TTL objects are created, the main stevedore may show different nuking patterns than expected. Transient storage is unbounded by default and grows independently.

Signals to watch in production

Signal	Why it matters	Warning sign
`MAIN.n_lru_nuked` rate	Storage is full and objects are being forcibly evicted	Sustained nonzero rate exceeding `n_expired` rate
`MAIN.n_expired` rate	Healthy TTL-based eviction baseline	Near zero while `n_lru_nuked` climbs: almost no natural expiry, all removals are forced
`n_lru_nuked / (n_lru_nuked + n_expired)`	Discriminates TTL-bound from storage-bound eviction	Above 0.5 means storage-bound
`MAIN.cache_hit` rate	Whether evicted objects are being re-requested	Declining while `n_lru_nuked` climbs means cache thrashing
`SMA.{name}.g_space`	Free space in the stevedore	Near zero confirms storage is full
`SMA.{name}.c_fail`	Allocation failures after eviction	Nonzero means even nuking could not free usable space
`MAIN.n_lru_limited`	Fetch failures because `nuke_limit` was hit	Any increment means 503s from storage pressure
`MAIN.n_object`	Current object count in cache	Declining during nuke storms confirms active eviction

How Netdata helps

Netdata’s Varnish collector surfaces per-second rates for both n_lru_nuked and n_expired, so you can read the storage-bound ratio as a live signal rather than computing deltas manually.

Rate computation: Netdata derives rates from cumulative counters automatically. You see nukes-per-second and expirations-per-second without manual delta math.
Hit rate correlation: the same dashboard shows n_lru_nuked alongside cache_hit, cache_miss, and backend_req, confirming whether nuking is driving miss rate up or the LRU is cleanly pruning the tail.
Storage utilization context: SMA.*.g_bytes and g_space appear alongside eviction counters, showing whether the stevedore is genuinely full or fragmentation is the real issue.
n_lru_limited visibility: the 503-producing escalation path is monitored as a distinct signal. Alert on any increment before users see errors.