Varnish SMA c_fail: allocation failures and malloc fragmentation

SMA.<name>.c_fail counts storage allocation failures after LRU eviction. Under normal conditions it stays at zero. When it climbs, memory management is wrong, and the cause is not always “the cache is full.”

The deceptive variant is the malloc stevedore. Over weeks of object churn, the underlying allocator (jemalloc or glibc malloc) fragments the heap. Varnish reports free bytes in SMA.<name>.g_space, but those bytes are scattered across many small free regions that cannot satisfy a contiguous allocation. Varnish evicts objects, retries, and fails. c_fail increments while g_space insists there is room.

What this means

c_fail is a cumulative counter under the SMA (Storage Malloc) or SMF (Storage File) namespace. It increments each time the stevedore cannot satisfy an allocation request after attempting LRU eviction. Every increment is an object Varnish wanted to cache but could not store.

Two conditions produce c_fail:

1. Real storage exhaustion. g_space is near zero. The cache is genuinely full, and even after evicting, there is not enough contiguous room. Fix: more storage or a smaller working set.

2. Fragmentation-induced failure. g_space shows meaningful free bytes, but the allocator cannot return a contiguous block of the requested size. Varnish evicts, retries, and fails. The cache reports capacity it cannot deliver.

The second condition develops slowly over weeks, is invisible to Varnish’s own accounting, and produces symptoms that look like capacity problems but are allocator pathology.

When an allocation fails, Varnish has a safety valve: the nuke_limit parameter (default 50). If evicting more than nuke_limit objects is required to free enough space for one allocation, Varnish gives up rather than clearing the cache for a single object. MAIN.n_lru_limited counts these events. The object goes uncached and the request is treated as a miss.

flowchart TD
    A["c_fail incrementing"] --> B{"g_space near 0?"}
    B -- "Yes" --> C["Real storage exhaustion
Cache genuinely full"]
    B -- "No, free bytes reported" --> D["Fragmentation suspected"]
    C --> E["Increase storage size
or reduce working set"]
    D --> F{"Process RSS much larger
than configured storage?"}
    F -- "Yes" --> G["Check transient storage
and allocator overhead"]
    F -- "No" --> H["Heap fragmentation
Consider restart or
alternative stevedore"]

Common causes

Quick checks

These commands are read-only and safe for production.

# Current allocation failure count and storage state
varnishstat -1 -f 'SMA.s0.c_fail' -f 'SMA.s0.g_bytes' -f 'SMA.s0.g_space' -f 'SMA.s0.g_alloc'

# LRU nuking rate and expiry
varnishstat -1 -f MAIN.n_lru_nuked -f MAIN.n_expired

# Transient storage (unbounded by default)
varnishstat -1 -f 'SMA.Transient.*'

# Process RSS in KB (compare against configured -s size)
ps -p $(pgrep -n varnishd) -o rss=

# Allocator Varnish is linked against
ldd $(which varnishd) | grep -i 'jemalloc\|malloc'

# Transparent Huge Pages state (relevant for jemalloc)
cat /sys/kernel/mm/transparent_hugepage/enabled

How to diagnose it

1. Confirm c_fail is climbing. Take two readings 60 seconds apart and compute the delta. A one-time bump from process start is noise. A sustained rate is the problem.

2. Check g_space. If near zero, you have real storage exhaustion, not fragmentation. Skip to the cache-sizing fix below.

3. Check average object size. If g_space shows free bytes but c_fail is climbing, compute g_bytes / g_alloc. If this is reasonable (10-100 KB) but allocations still fail, the free space is fragmented.

4. Compare RSS to configured storage. RSS significantly larger than your -s malloc,SIZE means either transient storage is growing or the allocator is retaining freed memory. jemalloc in particular holds virtual memory and may not return it to the OS after free().

5. Check transient storage. SMA.Transient.g_bytes tracks memory for uncacheable responses (pass, hit-for-miss, pipe). Transient storage is unbounded by default and is a common OOM path. If it is growing, the problem may not be in your primary storage at all.

6. Check the allocator version. Note the library shown by ldd. jemalloc 5.3.0 changed large-allocation search and decay-based purging to reduce virtual-memory churn and fragmentation relative to earlier releases. It is not a universal cure, but upgrading from an older jemalloc can materially reduce monotonic RSS growth without any Varnish configuration change.

7. Check THP state. Transparent Huge Pages interact poorly with jemalloc, causing memory over-consumption. If THP is always or madvise, set MALLOC_CONF=thp:never in the Varnish process environment.

8. Correlate with hit rate and backend load. If c_fail is climbing, hit rate is declining, and backend_req increases, the allocation failures are causing real user impact: objects that should be cached are not being stored.

Metrics and signals to monitor

Fixes

Cache too small for the working set

If g_space is genuinely near zero, the cache cannot hold your active working set. Options:

• Increase -s malloc,SIZE. Reserve 20-30% of system RAM for the OS, Varnish process overhead, thread stacks, workspace, and transient storage. Allocating all system RAM to storage is the most common path to OOM.
• Reduce the working set. Exclude large or uncacheable objects. Identify them with varnishtop -I ObjHeader:Content-Length (or BerespHeader while sampling live fetches).
• Shorten TTLs for low-value content so objects expire naturally instead of occupying storage.

Malloc fragmentation confirmed

If g_space shows free bytes but allocations fail, the heap is fragmented. This is a property of the allocator, not Varnish’s accounting. Varnish tracks bytes in and out of malloc() and free(); it has no visibility into allocator-internal fragmentation.

Immediate fix: restart the child process to clear the heap and reset fragmentation state. Schedule during a low-traffic window. The cache will be cold afterward.

Medium-term mitigations:

• Switch to the file stevedore (-s file). File-backed storage uses memory-mapped pages and is not subject to the same fragmentation pattern. Performance is comparable until memory pressure forces paging.
• Tune jemalloc. For jemalloc-based builds, MALLOC_CONF parameters such as dirty_decay_ms (jemalloc 5.x) or lg_dirty_mult (older versions) can reduce fragmentation waste at a modest performance cost. Start conservatively, measure RSS and hit rate, and avoid copying values from another workload.
• Consider MSE. Massive Storage Engine is an advanced stevedore for Varnish Enterprise, not part of open-source Varnish Cache; its compact object and metadata design is intended to reduce storage/database fragmentation.

Transient storage growth

If SMA.Transient.g_bytes is growing without bound, uncacheable traffic is consuming memory.

• Identify what is being passed: varnishlog -q 'VCL_call eq "PASS"' to see URL and header patterns.
• Fix the VCL or application to cache content that should be cacheable.
• Cap transient storage with -s Transient=malloc,1G. Current Varnish supports named stevedores, so this is a supported way to prevent unbounded growth at the cost of hard-limiting transient capacity.

Oversized storage allocation

If Varnish is being OOM-killed, -s malloc,SIZE is likely too aggressive. The OS needs memory for page tables, kernel structures, other processes, and the Varnish management process. Reduce the storage size by 20-30% or add system RAM.

Prevention

• Size storage conservatively. Allocate at most 70-80% of available RAM to malloc storage. The remainder covers allocator overhead, process overhead, thread stacks, workspace, and transient storage.
• Monitor RSS against configured storage. RSS beyond the configured storage size is the leading indicator of fragmentation or transient growth. Alert on RSS exceeding 130% of the configured -s size as a sustained condition.
• Schedule periodic restarts for long-running malloc instances. If fragmentation accumulates over weeks, a planned restart during a maintenance window is less disruptive than an unplanned OOM. Monitor the fragmentation trend to determine cadence.
• Disable THP for the Varnish process. Set MALLOC_CONF=thp:never in the process environment.
• Upgrade jemalloc if on 5.2.1. Version 5.3.0 resolves the known allocation problem causing monotonic RSS growth.
• Cap transient storage. Use -s Transient=malloc,SIZE to prevent unbounded transient growth from consuming memory the primary cache needs.
• Track average object size. Monitor g_bytes / g_alloc over time. A sudden shift may indicate a change in what your application is caching, affecting storage sizing assumptions.

How Netdata helps

Netdata collects Varnish SMA counters at per-second resolution, making the correlation between c_fail, g_space, and process RSS visible without manual polling.

• Per-second c_fail rate. Netdata computes the rate from the cumulative counter automatically, showing allocation failures per second.
• g_space vs c_fail correlation. Displayed on the same timeline, the fragmentation signature is immediately visible: c_fail climbing while g_space remains nonzero.
• Process RSS alongside storage counters. RSS from /proc is visible next to Varnish’s own storage accounting, so growth beyond the configured -s size stands out.
• Eviction pressure and hit rate together. n_lru_nuked alongside cache_hit ratio on one timeline shows whether storage problems are affecting cache effectiveness.
• Transient storage tracking. Continuous collection of SMA.Transient.g_bytes catches slow growth before the OOM killer does.
• ML anomaly detection on RSS. Slow RSS growth over weeks is exactly the pattern that threshold-based alerts miss but anomaly detection catches.

Related guides

• Varnish Error 503 Backend fetch failed: what the error page actually means
• Varnish backend_fail, backend_unhealthy, and backend_busy: three different backend problems
• Varnish backend connection reuse low: keepalive not working and slow TTFB
• Varnish backend probe configuration: threshold, window, interval, and initial
• Varnish backend is sick: health probes, all-backends-sick, and grace
• Varnish cache hit ratio dropped: hit rate collapse and backend overload
• Varnish cache stampede: a popular object expires and the herd hits the backend
• Varnish ESI errors: broken pages and workspace pressure from Edge Side Includes
• Varnish fetch_failed: backend connected but the fetch broke
• Varnish grace masking a backend outage: the ticking-clock incident
• Varnish Guru Meditation: reading the XID and tracing the failing request
• Varnish cache_hitpass / cache_hitmiss climbing: uncacheable content bleeding to the backend