$ guides / varnish / varnish-shm-log-overrun ▌

Operations Guides

Varnish shared-memory log overruns: losing the evidence during an incident

Cache hit rate dropped, backend request rate spiked, users saw elevated 503s. You need to reconstruct what happened from Varnish request logs: which URLs triggered the problem, what backend errors were returned, how long fetches took. You reach for varnishncsa output or varnishlog traces and find gaps, partial records, or nothing at all.

Varnish served traffic throughout the incident. The logging subsystem quietly discarded data when you needed it most.

Varnish does not write request logs to disk. All request-level logging goes to a circular shared-memory buffer called the Varnish Shared-memory Log (VSL), inside the VSM (Varnish Shared Memory) region. When the buffer fills and wraps before a consumer reads the records, those records are permanently overwritten. There is no recovery.

This failure is self-amplifying during incidents: high traffic generates more log records per second, causing the buffer to wrap faster and overwrite exactly the forensic data you need to diagnose the incident.

What this means

The VSL is a fixed-size ring buffer in shared memory. Worker threads write transaction records (request details, backend fetch information, timestamps, VCL decisions) as they process requests. External tools (varnishlog, varnishncsa, custom VSL readers) attach to the shared memory segment and read records sequentially.

When a writer reaches the end of the buffer, it wraps to the beginning. If a consumer has not yet read the records at the beginning, those records are overwritten. The consumer’s read cursor jumps forward, and the skipped records are lost.

flowchart LR
    A["Worker threads
write VSL records"] --> B["Circular VSL buffer
default 80M"]
    B --> C{"Consumer attached
and keeping up?"}
    C -->|Yes| D["varnishncsa / varnishlog
captures complete records"]
    C -->|No, buffer wraps| E["shm_cycles increments
records overwritten"]
    C -->|No consumer at all| F["All records lost
buffer cycles silently"]
    E --> G["Forensic gaps during
peak traffic window"]
    F --> G

varnishstat reads live counters from a separate part of the VSM, not from the VSL log buffer. The counters that report log overrun health (shm_cycles, shm_flushes, shm_cont) remain available even when request-level log records are being actively destroyed. The evidence of data loss survives even when the data itself does not.

Varnish exposes several counters that describe VSL health:

MAIN.shm_cycles (diag): the definitive indicator of log data loss. Increments each time a write of log records would reach past the end of the shared memory log and cycle back. Any nonzero rate means records were overwritten before consumers could read them.
MAIN.shm_flushes (diag): in Varnish 6.3+, measures per-task buffer flushes triggered when adding a record to a batch would exceed vsl_buffer (default 16k in the 7.x/8.x series, 4k through the 6.x line). This is buffer management, not necessarily data loss. In Varnish 5.x and earlier, this counter was described more generically as “SHM flushes due to overflow.” Do not treat shm_flushes alone as evidence of log loss; use shm_cycles for that.
MAIN.shm_cont (diag): SHM lock contention. Increments when a write had to wait for the lock. High contention with rising shm_cycles means the buffer is under pressure from both write volume and consumer competition.

MAIN.vsm_overflowed tracked bytes that did not fit in the shared memory used to communicate with tools like varnishstat and varnishlog. This counter existed in Varnish 4.x through 5.1 and was removed in 5.2. The deprecated vsm_space parameter was removed in Varnish 6.0. Neither exists in supported releases today — rely on shm_cycles as the cross-version indicator of record loss.

Common causes

Cause	What it looks like	First thing to check
No persistent consumer running	All request-level history absent; `shm_cycles` rising continuously	`pgrep -a varnishncsa`
VSL buffer too small for log volume	`shm_cycles` rising during traffic peaks, stable during quiet periods	`varnishadm param.show vsl_space`
Consumer reading too slowly	`shm_cycles` rising; consumer process disk I/O or CPU saturated	`iostat -x 1` or `top` on the consumer process
Excessive log volume from VCL	`shm_records` rate very high relative to `client_req`; `shm_cycles` tracks request rate	`varnishlog -q 'VCL_Log'` for `std.log()` usage
Multiple broad-filter consumers	`shm_cont` elevated; multiple consumer processes competing	`pgrep -a varnish` to list all consumers

Quick checks

# Check SHM health counters
varnishstat -1 -f MAIN.shm_cycles -f MAIN.shm_flushes -f MAIN.shm_cont -f MAIN.shm_records

# Check for running log consumers
pgrep -a varnishncsa
pgrep -a varnishlog

# Check configured VSL space (default 80M, min 1M, max 4G)
varnishadm param.show vsl_space

# Check per-task VSL buffer size
varnishadm param.show vsl_buffer

# Check request rate for context on log volume
varnishstat -1 -f MAIN.client_req

# If running Varnish 4.x through 5.1, check the legacy overflow counter
varnishstat -1 -f MAIN.vsm_overflowed

Take two readings of the SHM counters several seconds apart. A rising shm_cycles value confirms active data loss. A stable shm_cycles with high shm_flushes means per-task buffer churn but not necessarily record loss.

How to diagnose it

Confirm data loss is occurring. shm_cycles is cumulative, so a single reading tells you nothing. Take two readings several seconds apart and compute the delta. Any nonzero delta per second means records are being overwritten right now.
Verify a consumer is running. The most common cause is simply that no varnishncsa or varnishlog process is attached to the VSM. Without a consumer, the buffer fills and wraps continuously, and every request-level record is lost. Check with pgrep -a varnishncsa and pgrep -a varnishlog.
If a consumer is running, check its throughput bottleneck. If varnishncsa is writing to disk, check disk I/O latency with iostat -x 1. A slow disk causes the consumer to fall behind the write rate. If the consumer is CPU-bound (complex format string, regex filtering), check its CPU usage with top or pidstat -p <pid> 1.
Check whether the VSL buffer is sized for peak load. The default vsl_space is 80M. During a traffic spike that doubles or triples request volume, the buffer wraps proportionally faster. If shm_cycles only rises during peaks and is stable during normal load, the buffer is undersized for your worst case.
Check for excessive log volume. VCL using std.log() for debugging, or verbose VMOD logging, increases the number of VSL records per request. Check shm_records rate relative to client_req rate. If records per request is abnormally high, investigate VCL logging.
Check for multiple competing consumers. Each consumer (varnishlog, varnishncsa, custom VSL readers, monitoring agents) reads from the same buffer. Multiple consumers with broad filters (no -q query) multiply read load and increase shm_cont contention, which slows writers and can accelerate cycling.
In containerized deployments, verify consumer-to-VSM connectivity. Varnish 7.6 introduced liveness checks for VSM consumers based on signaling the manager PID (kill(pid, 0)); the check does not exist before 7.6. If varnishlog or varnishncsa runs in a separate container with a different PID namespace, it may fail liveness checks silently and stop reading without an obvious error. Set VSM_NOPID=1 in the consumer’s environment to disable PID-based checks across namespaces.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`MAIN.shm_cycles` rate	Definitive indicator of log record overwrites (data loss)	Any nonzero sustained rate
`MAIN.shm_flushes` rate	Per-task buffer flushes from `vsl_buffer` pressure (not necessarily data loss in 6.3+)	Sudden increase correlating with traffic spikes
`MAIN.shm_cont` rate	Lock contention among writers and consumers	Rising alongside `shm_cycles`
`MAIN.shm_records` rate	Total VSL records being written	Records-per-request ratio abnormally high
`MAIN.client_req` rate	Context for log volume (more requests produce more records)	Spikes correlate with `shm_cycles` spikes
Consumer process CPU and disk I/O	Slow consumers cause buffer overruns	Disk wait time or CPU near saturation on consumer

Fixes

No consumer running

Without a persistent consumer, the VSL buffer cycles silently and all request-level data is lost.

Run varnishncsa writing to persistent storage with log rotation. This is a baseline operational requirement, not an optimization.

# Start varnishncsa writing to a file with Apache Combined format plus %D (request duration in microseconds)
# Adapt the path and format to your environment
varnishncsa -a -w /var/log/varnish/varnishncsa.log \
  -F '%h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i" %D'

In production, run varnishncsa as a managed service (systemd unit, container sidecar) with automatic restart on failure and log rotation via logrotate or an equivalent mechanism.

VSL buffer too small

If shm_cycles rises during traffic peaks but is stable during normal load, increase vsl_space.

The -l flag to varnishd at startup is shorthand for -p vsl_space=<size>. Valid range is 1M to 4G, default 80M. vsl_space cannot be changed at runtime: the parameter is flagged restart-only, so param.set requires a child restart to take effect.

Update the varnishd startup configuration to include -l 256M (or your chosen size), then restart the child process. Plan for cache loss on restart.

Increasing vsl_space costs memory. The buffer is allocated from system memory and stays resident. On a host with tight memory margins, weigh this against cache storage allocation (-s malloc,SIZE) and the memory needed for thread stacks, workspace, and transient storage.

Consumer too slow

If varnishncsa is the bottleneck:

Write to the fastest available disk. SSD or NVMe local storage, not network-attached storage.
Use log rotation with compression as a downstream step, not inline in the varnishncsa pipeline.
If the format string is complex (many %{}i header extractions, computed fields), simplify it. Apache Combined Log Format with %D for request duration is sufficient for most forensic needs.

If varnishlog is the bottleneck:

Use query filters (-q) to narrow the record set. A broad-filter consumer reads every record, maximizing CPU and I/O load.
Avoid running varnishlog interactively during incidents unless you need real-time transaction tracing. It competes with varnishncsa for buffer reads.

Excessive logging from VCL

If shm_records rate is disproportionately high relative to client_req, VCL may be generating excessive log output. Check for:

std.log() calls left from debugging sessions
Verbose VMOD logging
Unnecessary synthetic header operations that generate additional records

Remove or gate debugging log calls behind a header check or VMOD-controlled flag so they do not run in production traffic paths.

Prevention

Always run varnishncsa to persistent storage with rotation. Without it, all request-level forensic data is ephemeral and will be lost during the next traffic spike.
Monitor MAIN.shm_cycles continuously, not just during incidents. Alert on any nonzero sustained rate. The counter is cumulative, so monitor the rate (delta per second), not the absolute value.
Size the VSL buffer against peak log volume, not average. Incidents generate the most log volume. If the buffer survives average load but overruns during peaks, it fails exactly when you need it.
Avoid running multiple broad-filter consumers simultaneously. Each consumer adds read load and lock contention. Use targeted queries (-q) for ad-hoc investigation rather than unfiltered continuous consumption.
In containerized deployments, set VSM_NOPID=1 for consumers in separate PID namespaces. Without this, Varnish 7.6+ consumers may fail liveness checks and silently stop reading (the checks and the VSM_NOPID opt-out are both new in 7.6).
Verify mlock() succeeds for the VSM region. If the OS cannot lock the shared memory segment in RAM, it may be paged out under memory pressure, severely impacting consumer read performance. Varnish 7.6 emits a Warning: mlock() of VSM failed: <reason> (<errno>) message on stderr when this occurs (this warning is new in 7.6). In Docker, use --ulimit=memlock=-1. In Kubernetes, add CAP_IPC_LOCK to the security context.

How Netdata helps

Per-second shm_cycles rate detection. Netdata collects Varnish counters at one-second resolution, making incremental overruns visible immediately rather than after a longer polling interval has already lost more data.
Correlation between shm_cycles, request rate, and consumer process metrics. When log overruns occur, Netdata correlates the overrun timestamp against traffic spikes, consumer CPU usage, and disk I/O latency in a single view.
Anomaly detection on SHM counter rates. Netdata’s anomaly detection flags unusual shm_cycles or shm_cont patterns before they cross a static threshold, useful for catching slow consumer degradation.
Consumer process monitoring alongside Varnish metrics. If varnishncsa is disk-bound or CPU-saturated, Netdata surfaces the process-level resource pressure next to the Varnish-side overrun signal.
Alerting on MAIN.uptime resets. When the Varnish child restarts, all MAIN.* counters reset to zero. Netdata’s uptime monitoring prevents false-positive rate calculations during the reset window, so you do not mistake a counter reset for a sudden improvement.