$ guides / varnish / varnish-vcl-reload-accumulation ▌

Operations Guides

Varnish n_vcl accumulation: cold VCLs never discarded

MAIN.n_vcl is climbing steadily in varnishstat. Each deploy loads a new VCL, and the count never comes back down. varnishadm vcl.list shows dozens of VCLs in “available” state, most of them cold. Process RSS is creeping upward. This is VCL accumulation: automated reload pipelines that load new VCLs on every deploy but never run vcl.discard on old ones.

Cold VCLs (no longer serving traffic) have released some runtime resources per the VCL temperature system, but the compiled shared object remains loaded until explicitly discarded. A deployment pipeline that only runs vcl.load and vcl.use without vcl.discard leaves every old VCL resident indefinitely.

The telltale signal is MAIN.n_vcl steadily greater than 1 and growing. MAIN.n_vcl_discard should track n_vcl over time. If it stays at or near zero while n_vcl climbs, old VCLs are accumulating.

What this means

Each loaded VCL progresses through a temperature lifecycle controlled by the management process. vcl.load compiles the VCL and enters “available” state. vcl.use activates it. The previously active VCL transitions to “available” and begins cooling.

The vcl_cooldown parameter controls how long Varnish waits before transitioning a VCL from warm to cold. The default is 600 seconds (ten minutes) in every supported release. Once cold, Varnish releases resources that can be reacquired later, but the VCL itself remains loaded. It is not freed until vcl.discard is called and no in-flight request references remain.

A VCL with in-flight request references is in the “busy” temperature state. Varnish cannot cool or discard a busy VCL. Workers release VCL references at the beginning of each new transaction, so under high traffic or with parked worker threads, a VCL can remain “busy” long after it is no longer active.

flowchart TD
    A["Deploy triggers reload"] --> B["vcl.load compiles new VCL"]
    B --> C["vcl.use activates new VCL"]
    C --> D["Old VCL enters cooling"]
    D --> E{"vcl.discard called?"}
    E -->|"Yes"| F["VCL freed, memory reclaimed"]
    E -->|"No"| G["Cold VCL retained in memory"]
    G --> H["Next deploy repeats cycle"]
    H --> D
    G --> I["n_vcl grows steadily"]

The gap is between “cold” and “discarded.” Cold VCLs are safe to remove, but Varnish does not auto-discard them. Something must explicitly call vcl.discard.

Common causes

Cause	What it looks like	First thing to check
Reload script omits vcl.discard	n_vcl grows by 1 per deploy, never decreases	Inspect the reload mechanism or script
CI/CD pipeline calls vcl.load + vcl.use only	vcl.list shows many “available” VCLs with old timestamps	Review deploy pipeline VCL commands
Busy VCLs never releasing	vcl.list shows VCLs stuck in “busy” with nonzero busy count	Check for long-lived connections or parked threads holding references
vcl.discard fails silently	Discard attempted but VCL remains in list	Check for labels or dependencies preventing discard

Quick checks

# Check VCL counters: loaded, available, discarded, and failures
varnishstat -1 -f MAIN.n_vcl -f MAIN.n_vcl_avail -f MAIN.n_vcl_discard -f MAIN.vcl_fail

# List all loaded VCLs with status, state, temperature, busy count, and name
varnishadm vcl.list

# Check process RSS for memory growth correlation
# Note: multiple varnishd processes (parent + child) may be listed
ps -p $(pgrep varnishd) -o pid,rss,vsz,etime --no-headers

# Count VCLs by status to see the distribution
varnishadm vcl.list | awk '{print $1}' | sort | uniq -c

# Check vcl_cooldown setting
varnishadm param.show vcl_cooldown

# Identify the reload mechanism in use
which varnishreload 2>/dev/null; systemctl cat varnish 2>/dev/null | grep -i reload

How to diagnose it

Run varnishadm vcl.list and examine the output. Each line shows status (active/available), state, temperature (warm/cold/busy/cooling), busy count, and VCL name. A healthy system has one “active” VCL and at most a few “available” VCLs transitioning to cold.
Count the VCLs by status. If you see dozens of “available” VCLs, they are accumulating. Cross-reference with MAIN.n_vcl in varnishstat to confirm the counter matches.
Look at temperatures. Any VCL in “busy” state with a nonzero busy count is held by in-flight requests. These cannot be discarded until references are released. Under normal traffic, references drain within seconds to minutes.
Check MAIN.n_vcl_discard. If this counter is zero or far behind MAIN.n_vcl, nothing in your pipeline is discarding old VCLs. In a healthy reload cycle, every load is eventually followed by a discard.
Correlate n_vcl growth with process RSS. Each retained VCL adds memory for compiled code and associated state. If RSS climbs in step with n_vcl, the accumulation is consuming meaningful memory.
Identify your reload mechanism. Check whether deploys use varnishreload, a custom script, or direct varnishadm calls. The fix depends on what is running the reload sequence.
If vcl.list is slow to respond or returns no output, the VCL count may be high enough to cause CLI latency. With enough accumulated VCLs, vcl.list can take multiple seconds or appear to hang. This is itself a symptom of severe accumulation.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`MAIN.n_vcl`	Total loaded VCLs (active + available). Direct measure of accumulation.	Steadily growing, never decreasing
`MAIN.n_vcl_avail`	Available, non-active VCLs. Should be near zero in steady state.	Growing in lockstep with n_vcl
`MAIN.n_vcl_discard`	VCLs that have been discarded. Should track n_vcl over time.	Zero or far behind n_vcl
`MAIN.vcl_fail`	VCL execution failures during request processing.	Any nonzero rate indicates VCL runtime errors
Process RSS	Total memory consumed by the child process.	Growing in step with n_vcl
`vcl.list` busy count	In-flight request references holding a VCL.	Persistent nonzero busy count on old VCLs

Fixes

Manually discard cold VCLs

Identify cold VCLs from vcl.list output and discard them individually:

# List VCLs to identify candidates for discard
varnishadm vcl.list

# Discard a specific cold VCL by name
varnishadm vcl.discard <vcl_name>

VCLs in “busy” state cannot be discarded until in-flight references are released. If vcl.discard fails or the VCL remains in the list, check for a nonzero busy count or for labels pointing at the VCL.

Warning: by design, vcl.discard refuses to remove a VCL that has labels attached, and a label cannot be removed while any VCL still returns vcl(<label>) — so discard pipelines must handle labels explicitly. Crashes involving labeled-VCL discard have also been reported in the field against older releases; if you use labeled VCLs, test discard behavior in staging before automating it in production.

Switch to varnishreload

varnishreload, shipped with Varnish packages from Varnish 6 onward, handles load and activation. By default it keeps every reload: old VCLs are only discarded when you pass -m <max>, which leaves at most <max> available reloads behind the active one. -w <seconds> inserts a wait between load and use, and VCLs referenced by a label are never discarded.

# Reload VCL and keep at most 2 previous reloads
varnishreload -m 2

If your current reload mechanism is a custom script or direct varnishadm calls, replacing it with varnishreload is the simplest long-term fix. Verify that it also discards old VCLs in your version, or add explicit discard calls as described below.

Fix a custom reload pipeline

Add explicit vcl.discard calls to your deploy pipeline after each successful reload. The pattern:

vcl.load <new_name> <file>
Verify load succeeded by checking the exit code
vcl.use <new_name>
Wait briefly for in-flight requests on the old VCL to drain
vcl.discard <old_name>

Name VCLs with a deploy identifier (timestamp or build number) so old VCLs are easy to identify and discard programmatically.

Avoid rapid load-use-discard sequences with no delay between steps. Discarding the previous VCL before the new one’s directors have initialized can surface FetchError: Director <name> returned no backend for early requests — an operator-reported failure mode that is most common with VMOD-based directors. A short sleep between load and use gives the new VCL time to initialize backend probes.

Handle stuck busy VCLs

If a VCL remains in “busy” state with a nonzero busy count for an extended period, in-flight references are not being released. Long-lived connections such as WebSocket sessions, long-polling endpoints, or parked worker threads can hold references indefinitely.

A debug flag exists to force rapid VCL reference release:

# Force rapid VCL reference release (NOT for production)
varnishadm param.set debug +vclrel

This is explicitly not intended for production use. The correct fix is to identify and address the long-lived connections or thread parking behavior holding the references. Check for WebSocket or long-polling traffic with varnishlog and consider whether those sessions should be routed around Varnish.

Prevention

Use varnishreload instead of manual reload sequences. It handles load and activation in one operation, closing the gap where old VCLs accumulate. Verify discard behavior for your version.
Alert on n_vcl growth. Any sustained increase without corresponding n_vcl_discard growth indicates accumulation. A practical threshold is MAIN.n_vcl greater than 5 sustained over an hour, which catches accumulation before memory pressure becomes critical.
Track n_vcl_discard alongside n_vcl. In a healthy pipeline, the two counters should track each other over time. Divergence is the signal.
Audit every CI/CD path that triggers a VCL reload. Each path should also trigger discard of the previous VCL. This includes deploy scripts, config management runs, and manual operator actions.
Do not set vcl_cooldown to zero. A bug report against Varnish 6.0 (GitHub issue #3135) confirmed that setting vcl_cooldown to zero makes the management CLI unresponsive with the management process at 100% CPU, while the worker keeps serving traffic. Leave vcl_cooldown at its default value.

How Netdata helps

Netdata collects MAIN.n_vcl, MAIN.n_vcl_avail, and MAIN.n_vcl_discard per second. A steady upward trend in n_vcl with flat n_vcl_discard is the primary accumulation signal.
Correlating n_vcl growth with process RSS in the same dashboard makes the memory impact immediately visible. When both climb in lockstep, the root cause is clear.
ML anomaly detection flags unusual n_vcl growth patterns even when absolute values are still low, catching accumulation early.
MAIN.vcl_fail is collected alongside VCL counters. A spike in vcl_fail after a reload indicates the new VCL has runtime errors, a distinct failure mode worth distinguishing from accumulation.
The Varnish collector also surfaces backend health, thread pool saturation, and storage utilization, helping distinguish VCL accumulation from other causes of memory growth such as malloc fragmentation or storage leaks.