$ guides / varnish / varnish-transient-storage-oom ▌

Operations Guides

Varnish transient storage OOM: the unbounded memory path that kills the process

Varnish dies. The OOM killer takes it. No panic, no crash, no VCL error in the logs. Process RSS was far above your configured -s malloc,8G storage, and the kernel reclaimed memory the only way it knows how. This is the transient storage OOM path: the number-one missed signal in Varnish operations.

The problem is structural. Varnish has two storage paths: your configured cache storage (malloc or file, capped at a known size) and transient storage. Transient storage holds objects that will never be cached: pass responses, hit-for-pass, hit-for-miss, and piped traffic. By default, Varnish uses an unbounded malloc/default storage for Transient. A pass storm, a batch of large uncacheable responses, or a slow leak over weeks can push process RSS past the configured storage size and into system memory the kernel will not let you keep.

The primary storage counters (SMA.s0.g_bytes, SMA.s0.g_space) show a healthy cache. The transient path is a separate accounting that, on older versions, is invisible in varnishstat output entirely. The cache looks full but fine, and then the process is gone.

What this means

flowchart TD
    A["Client request"] --> B{"VCL: cacheable?"}
    B -->|Yes: hit or miss| C["Primary storage
-s malloc,SIZE"]
    B -->|No: pass or hit-for-miss| D["Transient storage
unbounded malloc"]
    C -->|Full: LRU evict| E["Hit rate drops
backend load up"]
    D -->|Grows without limit| F["RSS climbs
past configured size"]
    F --> G["OOM killer
terminates child"]
    G --> H["Management process
restarts child"]
    H -->|Cache lost| I["Cold cache
backend flood"]

Uncacheable traffic enters transient storage, which has no cap, and RSS climbs until the OOM killer fires. When the management process restarts the child, the cache is empty, sending a flood of misses to backends on top of whatever caused the pass storm.

Transient storage is not a bug. It is how Varnish handles objects that should not enter the normal cache: responses marked uncacheable by VCL (return(pass)), hit-for-pass and hit-for-miss objects (Varnish cached the decision not to cache), and short-lived objects whose TTL is below the shortlived runtime parameter threshold. Each consumes transient storage memory, and none of it counts against your primary storage allocation.

The default transient backend has no size limit. Current Varnish accepts a named Transient stevedore with an explicit size, and that is the clean way to cap it.

Common causes

Cause	What it looks like	First thing to check
Pass storm from VCL	`MAIN.cache_hitpass` rate high; RSS climbing steadily	`varnishlog -q 'VCL_call eq "PASS"'` to see which URLs are being passed
Application adding cookies to all responses	Hit-for-pass objects accumulating; content that used to cache is now uncacheable	Check `Set-Cookie` headers on responses that should be cacheable
Large uncacheable responses (file downloads, video)	RSS spikes sharply when large pass responses are served	`varnishtop -I ObjHeader:Content-Length` to find large objects being passed
Slow RSS creep over weeks	RSS grows monotonically but slowly; cache hit rate stable; no pass storm	Process RSS vs configured storage delta growing over time
Long-lived uncacheable decisions	RSS grows under steady uncacheable traffic	Inspect TTL/HitMiss/HitPass records and VCL for nonzero grace/keep on the uncacheable branch

Quick checks

# Get the child process PID (higher PID is typically the child)
# On most setups: management starts first, child is forked second
CHILD_PID=$(pgrep -n varnishd)

# Check child RSS in KB
ps -p "$CHILD_PID" -o rss=

# Check transient storage counters
varnishstat -1 -f 'SMA.Transient.*'

# Check pass rate and hit-for-pass/hit-for-miss activity
varnishstat -1 -f MAIN.cache_hitpass -f MAIN.cache_hitmiss -f MAIN.s_pass

# Check what traffic is being passed right now
varnishlog -q 'VCL_call eq "PASS"' -g request

# Check for OOM kills in system logs
dmesg | grep -i oom | grep -i varnish
journalctl -u varnish --since "1 hour ago" | grep -i "child"

# Check child process stability and restart history
varnishstat -1 -f MGT.uptime -f MAIN.uptime -f MGT.child_died -f MGT.child_panic

# Check storage configuration from process arguments
# Note: -P flag is GNU grep only; use -E on BSD/Alpine
grep -oE '\-s [[:space:]]*\S+' /proc/$CHILD_PID/cmdline 2>/dev/null || tr '\0' ' ' < /proc/$CHILD_PID/cmdline | grep -oE '\-s \S+'

RSS is the most important signal. If RSS is significantly above your configured storage size, the difference is transient storage plus overhead (thread stacks, workspace memory, VSM segment, allocator fragmentation). On a version that omits these counters, RSS is the direct window into transient consumption.

How to diagnose it

Confirm the OOM path. Check dmesg for OOM killer entries targeting the Varnish child process. Check MGT.child_died or MGT.child_panic counters in varnishstat. If MAIN.uptime is much smaller than MGT.uptime, the child has been restarting. An OOM kill shows up as MGT.child_died (child exited on a signal), not MGT.child_panic.
Measure RSS vs configured storage. Get the child process RSS and subtract your configured malloc storage size. The remainder is transient storage plus overhead. A well-tuned Varnish should have RSS at roughly configured storage plus 20-30% for thread stacks, workspaces, VSM, and allocator overhead. If the remainder is much larger or growing, transient storage is the problem.
Check transient counters. SMA.Transient.g_bytes shows bytes currently held in transient storage. If it is growing, the path is confirmed. SMA.Transient.g_space reads 0 when transient is unbounded, which is itself diagnostic: it confirms there is no cap.
Identify the pass traffic. Run varnishlog -q 'VCL_call eq "PASS"' to see which requests are being passed. Look for patterns: a specific URL prefix, requests with cookies, large Content-Length responses. Correlate with MAIN.cache_hitpass and MAIN.cache_hitmiss rates. If cache_hitpass is climbing, the application is emitting response headers that make content uncacheable.
Check for the slow leak. If there is no pass storm but RSS is still growing, the cause may be malloc fragmentation in transient storage or a slow accumulation of hit-for-miss objects. Compare RSS readings over days or weeks, and review the release notes for your exact Varnish 6.0 LTS patch version before attributing the growth to an already-fixed lifecycle bug.
Check VCL for accidental pass. A common mistake is leaving debugging VCL in production that passes traffic under certain conditions. Review vcl_recv and vcl_backend_response for return(pass) calls broader than intended. Also check whether Set-Cookie headers from the backend are causing Varnish to mark responses uncacheable.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Process RSS (OS-level)	The only reliable signal on all versions; captures transient and overhead that internal counters miss	RSS exceeding configured storage plus 25-30% overhead
`SMA.Transient.g_bytes`	Direct measure of transient storage bytes	Monotonic growth, or spike well above baseline
`MAIN.cache_hitpass` rate	Rate of hit-for-pass lookups; each one implies uncacheable content in transient	Sustained rate higher than baseline
`MAIN.cache_hitmiss` rate	Hit-for-miss traffic also uses transient storage	Same as above
`MGT.child_died` / `MGT.child_panic`	Child process crash or OOM kill detection	Any nonzero increment; repeated increments indicate a crash loop
`MAIN.uptime` vs `MGT.uptime`	Detects child restarts	`MAIN.uptime` much smaller than `MGT.uptime`
`SMA.Transient.c_fail`	Allocation failures in transient storage when capped	Any nonzero value means transient is full and allocations are failing

Fixes

Cap transient storage

The most direct fix. Configure transient storage with a hard limit at startup:

# Add to varnishd startup arguments
-s Transient=malloc,1G

This caps transient storage at 1GB. When transient is full, Varnish cannot allocate space for pass response bodies, producing fetch errors and potentially HTTP 503 responses for pass traffic. It prevents the OOM kill that takes down the entire process along with the full cache.

The tradeoff is explicit: bounded transient storage means pass traffic may fail when the cap is reached. The alternative is an unbounded path that kills the process. Choose the bounded failure mode.

Size the cap based on observed transient usage during normal operation plus headroom for spikes. Monitor SMA.Transient.g_bytes after capping to confirm the cap is adequate.

Reduce pass traffic

Fix the root cause: why is traffic being passed instead of cached?

Application cookies on cacheable content: Strip cookies in vcl_recv for cacheable URL patterns, or fix the application to not send Set-Cookie on public content. Most common cause of unexpected pass traffic.
Overly broad Vary headers: A Vary: Cookie or Vary: * header makes responses uncacheable under many conditions. Normalize or remove unnecessary Vary headers at the backend or in VCL.
Debugging VCL left in production: Remove return(pass) calls added for debugging. Audit every pass path in the VCL to confirm it is intentional and bounded.
Large objects that should not be cached: Consider whether these need to go through Varnish at all. Pipe mode (return(pipe)) bypasses the object store entirely, though large pipe bodies still consume socket buffer memory.

Address the slow RSS creep

If RSS grows slowly over weeks with no pass storm, the cause is likely malloc fragmentation in the transient allocator or gradual accumulation of short-lived objects not being freed efficiently. Options:

Cap transient storage as described above. A capped transient backend forces eviction rather than unbounded growth. This is the durable fix.
Upgrade to a supported release if running an older version. Review the exact release notes for known lifecycle/storage fixes; do not assume one patch resolves a workload-specific growth pattern.
Schedule periodic child restarts during low-traffic windows as a last-resort stopgap. This resets transient storage and defragments the allocator but loses the entire cache. Use only while working on the durable fix.

Prevention

Monitor process RSS as a primary metric. Alert when RSS exceeds configured storage size plus 25-30% overhead. This catches both acute pass storms and slow fragmentation-driven creep.
Monitor SMA.Transient.g_bytes. Alert on monotonic growth or sharp spikes above baseline.
Monitor MAIN.cache_hitpass and MAIN.cache_hitmiss rates independently. A gradual increase means more traffic is flowing through the transient path. Investigate before it becomes a storage problem.
Cap transient storage. Even if normal traffic does not cause OOM, a future traffic change or application deploy could trigger a pass storm. A cap converts an unbounded failure into a bounded one.
Audit VCL regularly for pass paths. Every return(pass) is a transient storage consumer. Ensure each one is intentional and that traffic volume through each pass path is understood.
Review beresp.grace and beresp.keep settings in vcl_backend_response. Settings inherited from an earlier branch can outlive the intended TTL on hit-for-miss objects. Inspect TTL/HitMiss/HitPass records; if an uncacheable branch retains objects longer than expected, set beresp.grace = 0s (and beresp.keep = 0s where applicable) on that branch.

How Netdata helps

Netdata’s per-second metrics collection makes the slow RSS creep visible long before it becomes an OOM kill. The signals worth correlating:

Process RSS collected per second against configured storage size. A monotonic RSS trend over hours or days is the leading indicator, and per-second resolution catches the inflection point where growth accelerates.
Varnish counters via the built-in collector: SMA.Transient.g_bytes, MAIN.cache_hitpass, MAIN.cache_hitmiss, and MGT.child_* are collected automatically. Correlating a hitpass rate spike with an RSS increase confirms the transient storage path within seconds rather than minutes.
ML-based anomaly detection on RSS and transient storage counters. The slow weeks-long fragmentation creep has a characteristic signature that static threshold alerts miss but anomaly detection flags early.
Child process restart detection via MGT.child_start, MGT.child_died, and MGT.child_panic. Correlating these with system-level OOM events and RSS trends distinguishes a transient storage OOM from a VCL panic or VMOD crash.
Composite dashboards that overlay cache hit rate, pass rate, backend request rate, and process RSS on the same timeline. The pattern is visually obvious: pass rate increases, RSS follows with a lag, then the child restarts and the cycle repeats.