$ guides / varnish / varnish-storage-sizing-malloc-file ▌

Operations Guides

Varnish storage sizing: malloc vs file, and why -s malloc,ALL-RAM kills you

The -s malloc startup parameter controls how much memory Varnish allocates for cached object storage. It does not control total process memory. That distinction is the single most expensive misunderstanding in Varnish operations.

When you set -s malloc,32G on a 32 GB machine, you have allocated 32 GB for object storage and left nothing for the operating system, the Varnish process itself, worker thread stacks, per-request workspace, transient storage, and allocator fragmentation. The kernel OOM killer sees a process consuming all available memory and terminates it. Varnish restarts, the management process reloads the child, the cache warms from zero, and the cycle repeats if the configuration has not changed.

What it is and why it matters

Varnish stores cached objects in a storage backend called a stevedore. The startup parameter -s selects the backend type and size. Three backends exist, but only two are production-relevant:

malloc allocates object storage from the process heap via malloc(). It is the fastest backend: all objects live in RAM with no disk I/O on the cache path. It is volatile; a child restart or crash empties the cache. The size passed to -s malloc,SIZE is a hard cap on how much memory the storage engine will request for object bodies.

file stores objects in a memory-mapped file on disk. Capacity can exceed RAM because the kernel pages objects in and out via the OS page cache. Hot objects perform nearly as well as malloc because they reside in page cache. Cold objects incur disk reads on the cache path. Cache contents do not survive a child restart: Varnish reinitializes storage on startup regardless of whether the backing file remains on disk. The file backend adds page cache pressure and disk I/O as monitoring concerns.

persistent is deprecated. Do not use it in production.

The core sizing mistake is treating -s malloc,SIZE as “how much RAM Varnish is allowed to use.” It is how much RAM the storage engine is allowed to allocate for object bodies. Total process RSS will be higher because of per-object metadata overhead, thread stacks, workspace memory, transient storage, and allocator fragmentation.

How it works

Beyond storage: what else consumes RAM

A Varnish host has multiple memory consumers beyond the configured storage size. Understanding each one is necessary to size storage without OOM risk.

Consumer	Approximate cost	Notes
Object storage	Configured via `-s malloc,SIZE`	The cache body data itself
Per-object overhead	~1 KB per object outside the storage engine	Object metadata, headers, tracking structures
Worker thread stacks	`thread_pool_stack` (default 80 KB on 64-bit systems) per thread	Up to `thread_pool_max * thread_pools` threads. At 5000 threads per pool with 2 pools, that is 10,000 threads consuming ~800 MB of stack memory alone
Per-request workspace	`workspace_client` (default 96 KB) + `workspace_backend` (default 96 KB) per active request	Allocated for HTTP headers, VCL string operations, and request processing
Transient storage	Unbounded by default	Pass, pipe, hit-for-pass, and hit-for-miss object bodies. Uses system malloc with no size cap unless explicitly configured
VSM log segment	Configured via `-l` parameter	Shared memory log buffer for varnishlog, varnishstat, and varnishncsa
OS and page cache	Variable	Kernel needs memory. The file backend competes for page cache directly

On a 32 GB host with 10,000 peak threads and a working set of one million objects, non-storage memory demand can reach 3 to 5 GB before fragmentation. That is the budget you must leave empty when sizing -s malloc.

Fragmentation amplifies the problem

The -s malloc,SIZE parameter limits the net amount of memory the storage engine requests for objects. The gross memory footprint can be substantially higher due to allocator fragmentation.

The official Varnish documentation states that the gross memory used by the malloc implementation “might be substantially higher by a factor of typically two to four times.” Many Linux distribution packages link Varnish against jemalloc, which handles fragmentation more efficiently than the standard C library allocator.

The practical consequence: process RSS will exceed the configured -s malloc,SIZE value. If you sized storage to fill all available RAM, RSS will exceed physical memory and the OOM killer will intervene.

flowchart LR
    subgraph WRONG["32G box: -s malloc,32G"]
        W1["Storage cap 32G"] --> W2["OS, threads,
workspace, transient,
fragmentation: 0G left"]
        W2 --> W3["OOM killer
terminates varnishd"]
    end
    subgraph RIGHT["32G box: -s malloc,22G"]
        R1["Storage cap 22G"] --> R2["Reserve 10G
for OS, threads,
transient, fragmentation"]
        R2 --> R3["Stable operation"]
    end

Where this fails in production

The direct OOM kill

Varnish RSS grows to fill physical memory, the kernel OOM killer selects varnishd as the largest memory consumer, and the process is terminated. The management process restarts the child, cache warms from zero, and if traffic is high, the cycle repeats.

Confirm this path by checking kernel logs and Varnish child restart counters:

# Check for OOM killer activity targeting varnishd
dmesg -T | grep -i 'oom.*varnish\|killed process.*varnish'

# Check child process restart history
varnishstat -1 -f MGT.child_died -f MGT.child_panic -f MGT.child_start

If MGT.child_died is incrementing and dmesg shows OOM kills, storage sizing is the likely root cause.

Transient storage: the silent OOM path

Objects that will never be cached (pass, hit-for-pass, hit-for-miss, pipe) go to transient storage, which is an unbounded malloc backend by default. A pass storm from a VCL misconfiguration, large uncacheable responses, or accumulating hit-for-pass objects can cause transient storage to grow without limit and consume all remaining RAM.

Process RSS grows steadily while the SMA storage counters look fine because the configured storage backend is within its cap. The delta between process RSS and the configured -s malloc,SIZE is largely transient storage plus overhead. If that delta grows monotonically over hours or days, transient storage is the leak.

# Check transient storage usage
varnishstat -1 -f 'SMA.Transient.g_bytes' -f 'SMA.Transient.c_fail'

# Compare process RSS to configured storage.
# Note: pgrep -n returns the newest varnishd process, which may be the
# management process rather than the child. Identify the child (higher RSS)
# explicitly if the number looks wrong.
ps -p $(pgrep -n varnishd) -o rss=
# RSS in KB minus configured storage size = transient + overhead

To cap transient storage explicitly:

-s Transient=malloc,1G

This prevents unbounded transient growth from consuming the memory reserve.

Fragmentation making g_space lie

With malloc storage, fragmentation can make g_space report available bytes that cannot actually be allocated. The storage engine reports free space, but the allocator cannot find contiguous blocks for the requested object size. The result is c_fail incrementing even when g_space is nonzero. This is less common with jemalloc than with libc malloc but still possible after weeks of object churn with widely varying object sizes.

Sizing storage correctly

The reserve rule

Reserve 20-30% of system RAM for non-storage use: OS, page cache, thread stacks, workspace, transient storage, per-object overhead, and fragmentation headroom.

On a 32 GB host:

Reserve: 6.4 to 9.6 GB
Available for storage: 22.4 to 25.6 GB
Practical starting point: -s malloc,22G

On a 64 GB host:

Reserve: 12.8 to 19.2 GB
Available for storage: 44.8 to 51.2 GB
Practical starting point: -s malloc,46G

If your Varnish build uses jemalloc and fragmentation is low in your workload, push toward the lower end (20%). If you are unsure which allocator your build uses, or if RSS significantly exceeds the configured storage size, stay at the higher end (30%).

Sizing against the working set

The configured storage size should hold your working set with headroom. The working set is the collection of objects actively requested within a window roughly equal to your shortest meaningful TTL.

# Current storage usage and object count
varnishstat -1 -f 'SMA.s0.g_bytes' -f 'SMA.s0.g_space' -f MAIN.n_object

# Average object size (g_bytes / g_alloc)
varnishstat -1 -f 'SMA.s0.g_bytes' -f 'SMA.s0.g_alloc'

# Eviction pressure: are useful objects being nuked before TTL?
varnishstat -1 -f MAIN.n_lru_nuked -f MAIN.n_expired

If n_lru_nuked is sustained above zero and cache hit ratio is declining, storage is undersized for the working set. If n_lru_nuked is zero and g_space is large, storage may be over-provisioned.

The ratio n_lru_nuked / (n_lru_nuked + n_expired) tells you whether the cache is storage-bound (high ratio, objects evicted before TTL) or TTL-bound (low ratio, objects expiring naturally).

malloc vs file: when to choose which

Choose malloc when:

Your working set fits comfortably in RAM with the 20-30% reserve applied.
Latency is the top priority and you want zero disk I/O on the cache path.
You accept that a restart empties the cache and requires warmup time.

Choose file when:

Your working set exceeds available RAM after the reserve.
You can tolerate occasional disk reads for cold objects and have budget for page cache monitoring.

With the file backend, the OS page cache consumes additional RAM for hot pages. This is invisible to Varnish but counts against total system memory. The page cache and Varnish storage compete for the same RAM, so the reserve rule applies to file storage as well. The failure mode is paging (slow performance) rather than OOM kills.

Signals to watch in production

Signal	Why it matters	Warning sign
`SMA.{name}.g_bytes` and `g_space`	Shows how full the configured storage is and how much headroom remains	`g_space` approaching 0 means LRU eviction is active
`SMA.{name}.c_fail`	Allocation failures mean storage cannot satisfy requests even after eviction	Any nonzero value indicates fragmentation or exhaustion
`SMA.Transient.g_bytes`	Transient storage is unbounded by default and is a common silent OOM path	Monotonic growth over hours or days indicates a pass or pipe problem
Process RSS vs configured `-s malloc,SIZE`	The delta reveals overhead, transient, and fragmentation consumption	RSS approaching physical memory limit means OOM is imminent
`MAIN.n_lru_nuked` rate	Sustained nuking means storage is undersized for the working set	Rate sustained above zero with declining hit ratio
`MGT.child_died` or `MGT.child_panic`	Child crashes may indicate OOM kills or storage corruption	Repeated increments with low `MAIN.uptime` indicate a crash loop
`MAIN.uptime` vs `MGT.uptime`	Large discrepancy means the child has restarted recently	`MAIN.uptime` much smaller than `MGT.uptime` after initial startup

How Netdata helps

Per-second collection of SMA.*.g_bytes, g_space, and c_fail shows storage fill rates and allocation failures in real time, before the OOM happens.
Process RSS alongside configured storage size makes the delta (transient, overhead, fragmentation) visible as its own signal. Monotonic RSS growth with stable SMA storage reveals a transient leak.
SMA.Transient.g_bytes monitored separately distinguishes storage pressure from transient growth without manual ps and varnishstat cross-referencing.
MGT.child_died and MGT.child_panic correlated with kernel OOM kill events connects the Varnish crash to memory pressure in one timeline.
ML anomaly detection on RSS and storage utilization flags unusual growth before it reaches the OOM threshold.
n_lru_nuked rate and cache hit ratio tracked together show whether storage is undersized for the working set.