Varnish child panic: Child died signal, core dumps, and the crash loop

Child (NNNN) died signal=N means the Varnish child (worker) process crashed. The management process supervises and restarts it automatically, so a single crash is self-recovering. But each restart empties the cache, and if the child keeps crashing, caching drops to zero and backends absorb full uncached traffic.

The immediate question: one-off crash or crash loop? If the child recovers and the cache warms back up, it is a TICKET. If MAIN.uptime never stabilizes and stays far below MGT.uptime, the child is dying before the cache can warm. That is a PAGE.

What this means

Varnish runs two processes. The management process (root-owned) handles VCL compilation, CLI access, and child supervision. The child process drops privileges and handles all cache operations: accepting connections, executing VCL, serving objects, fetching from backends.

When the child hits an unrecoverable error such as an assertion failure or a segfault, Varnish calls its internal panic handler. The child logs a panic message with a stack trace and terminates. The management process detects the death via a signal and restarts the child. The listening socket stays open during the restart window, but no requests are served until the child is back.

Each restart empties the cache completely. All MAIN.* counters reset to zero. Cache hit rate drops to zero until the cache warms back up, which takes minutes to hours depending on traffic volume and object TTLs.

The critical distinction:

• Single crash: MGT.child_panic or MGT.child_died increments once. The child restarts, MAIN.uptime eventually exceeds your warmup window and stabilizes. Cache hit rate recovers. This is a TICKET: investigate the root cause, but service has recovered.
• Crash loop: MGT.child_panic or MGT.child_died increments repeatedly. MAIN.uptime never stabilizes because the child keeps dying before the cache warms. MAIN.uptime stays far below MGT.uptime. Cache hit rate stays near zero. Backends receive full uncached traffic. This is a PAGE.

Management counters (MGT.*) do not reset on child restart. MGT.uptime continues counting through restarts, while MAIN.uptime resets each time. MAIN.uptime far below MGT.uptime is the definitive signal of recent or repeated restarts.

flowchart TD
    A["Child running\nMAIN.uptime climbing"] --> B["Child crashes\nsignal or panic"]
    B --> C["MGT detects death\nrestarts child"]
    C --> D["Cache emptied\nMAIN.uptime resets"]
    D --> E{"Child stays up?"}
    E -->|"Yes: uptime recovers"| F["Single crash\nself-recovered - TICKET"]
    E -->|"No: crashes again"| B

Common causes

Quick checks

# Management counters for crash, restart, and dump events
varnishstat -1 -f 'MGT.child_panic' -f 'MGT.child_died' -f 'MGT.child_dump' -f 'MGT.child_start'

# Compare child uptime to management uptime - if MAIN is far below MGT, recent restart(s)
varnishstat -1 -f MAIN.uptime -f MGT.uptime

# Last panic message with full stack trace
varnishadm panic.show
# JSON output: varnishadm panic.show -j

# System logs for Varnish restart messages
journalctl -u varnish --since "1 hour ago"

# Kernel OOM killer activity
dmesg | grep -i oom

# Panic files and core dumps in the working directory
ls -la /var/lib/varnish/*/

# Loaded VCL versions and timestamps
varnishadm vcl.list

# Child process memory usage (newest varnishd PID is the child)
ps -p $(pgrep -n varnishd) -o rss= -o vsz=

How to diagnose it

1. Confirm crash vs crash loop. Check MGT.child_panic, MGT.child_died, and MGT.child_start. A single increment of child_panic or child_died is one crash. Repeated increments mean a loop. Confirm by comparing MAIN.uptime to MGT.uptime: if MAIN.uptime is 30 seconds against MGT.uptime of several hours, the child restarted recently. If MAIN.uptime never climbs past your warmup window, the child is in a crash loop.

2. Read the panic trace. Run varnishadm panic.show. This returns the signal number, the assertion that failed (if any), and the C stack trace. If the CLI has no panic to show, check for _.panic files in the working directory: ls /var/lib/varnish/*/_.panic. No panic message but child_died present suggests an external kill (OOM killer) rather than an internal Varnish panic.

3. Check for OOM kills. Run dmesg | grep -i oom and look for entries referencing the varnishd child process. The OOM killer sends SIGKILL, which appears as child_died without a corresponding child_panic. Compare process RSS against system memory and configured storage size. Transient storage growth beyond configured limits is a common OOM path.

4. Correlate with recent changes. Run varnishadm vcl.list and check timestamps. A VCL reload that preceded the first crash is a strong suspect. If the crash started after a VMOD upgrade or a new VMOD was loaded, the VMOD’s C code is the likely culprit. Check journalctl -u varnish for the exact timing of the first crash relative to deployments.

5. Examine the stack trace for the failing subsystem. The panic trace shows the C call stack. Function names indicate the subsystem: vbf_fetch_thread points to fetch handling, VRE_match to regex or ban evaluation, HSH_Lookup to cache lookup, storage allocator functions to the object store. This narrows the root cause significantly.

6. Check for stack overflow. If the panic trace includes THIS PROBABLY IS A STACK OVERFLOW - check thread_pool_stack parameter, the crash was caused by thread stack exhaustion. A reasonable first step is adding 128k to the current thread_pool_stack value.

Metrics and signals to monitor

Fixes

OOM kill

If dmesg shows the OOM killer targeting the child, Varnish is consuming more memory than the system allows. The most common cause is allocating all system RAM to cache storage (for example, -s malloc,32G on a 32GB machine). The OS, Varnish process overhead, thread stacks, workspace memory, and transient storage all need memory beyond the configured cache size.

Reserve 20-30% of system RAM for non-storage use. Reduce -s malloc,SIZE if needed. Also check transient storage: SMA.Transient.g_bytes can grow without bound because transient objects have no size limit by default. Uncacheable responses, pass traffic, and hit-for-pass objects all consume transient storage.

If the OOM pattern is intermittent and correlates with traffic spikes, cap transient storage by assigning it to a sized storage segment.

VCL bug

If panic.show shows a stack trace through VCL subroutines or the VCL runtime, the compiled VCL triggered an assertion or memory error. Check varnishadm vcl.list for the most recent VCL change. If the crash started after a reload, roll back to the previous VCL:

varnishadm vcl.list

Identify the active VCL, then switch to a known-good version:

# Non-disruptive: switches active VCL immediately
varnishadm vcl.use <label>

If no previous version is available, load the last known-good VCL file and activate it. Review the panic stack trace for the specific subroutine and function that failed. Complex VCL with inline C, deeply nested ESI, or heavy regex operations in vcl_recv or vcl_hash are common triggers.

VMOD bug

If the panic trace references functions in a VMOD (custom C module), the VMOD’s code segfaulted or triggered an assertion. VMOD bugs are harder to diagnose because the stack trace points to compiled C, not VCL.

The immediate fix is to remove or replace the failing VMOD. If the VMOD provides critical functionality, check for an updated version or file a bug with the panic trace attached. If the crash correlates with specific request patterns (particular URLs, headers, or user agents), blocking that traffic temporarily can stop the crash loop while a fix is developed.

Stack overflow

If the panic message includes THIS PROBABLY IS A STACK OVERFLOW, increase the thread_pool_stack parameter:

# Check current value
varnishadm param.show thread_pool_stack

# Set new value (runtime change, affects newly created threads only)
varnishadm param.set thread_pool_stack <new_value>

Existing threads keep their old stack size. A Varnish restart applies the new value to all threads.

Version-specific bug

Some child panics are caused by bugs in specific Varnish versions. Known examples:

• Varnish 7.0.0 had a VRE_match() assertion failure during ban evaluation, triggered by specific request patterns hitting ban evaluation during cache lookup (GitHub issue #3714). The panic trace showed ban_evaluate through BAN_CheckObject through HSH_Lookup.
• Varnish 9.0.0 was affected by CVE-2026-40394, a workspace overflow panic in HTTP/2 session setup, fixed in 9.0.1.

If your panic trace matches a known version-specific bug, upgrade to the nearest patched release. Before upgrading, review the release notes for breaking changes, especially VCL syntax or parameter defaults.

Storage corruption

If the panic trace references storage allocator functions (malloc, SMA, or SMF internals), the storage backend may be corrupted. This is rare but can occur under severe memory fragmentation or filesystem issues with file-backed storage.

The immediate fix is to restart Varnish with a clean storage state. For malloc storage, this is automatic: a restart starts with an empty cache. For file-backed storage, check whether the underlying filesystem is healthy. If corruption recurs, switch to malloc storage temporarily to isolate whether the issue is storage-backend-specific.

Prevention

• Monitor MGT.child_panic and MGT.child_died. These counters do not reset on child restart, so any increment is a permanent record of a crash. Alert on any nonzero value.
• Monitor MAIN.uptime against MGT.uptime. A widening gap is the leading indicator of a crash loop. If MAIN.uptime resets while MGT.uptime keeps climbing, the child restarted.
• Enable core dumps. On systemd-managed deployments, add LimitCORE=infinity to the [Service] section of the Varnish unit file. Setting ulimit -c unlimited in a shell is not sufficient when systemd starts varnishd. Verify that the kernel’s core_pattern does not redirect core files to a pipeline that discards them.
• Reserve memory headroom. Do not allocate all system RAM to cache storage. Keep 20-30% for the OS, Varnish process overhead, thread stacks, workspace, and transient storage.
• Test VCL changes before production. Use varnishd -C -f <vcl_file> to compile-check VCL before loading it. Load new VCL alongside the existing one, verify it serves traffic correctly, then switch with vcl.use. Keep the previous VCL loaded for instant rollback.
• Track Varnish version against known issues. Subscribe to security advisories. Versions past end of life do not receive patches for crash-inducing bugs.

How Netdata helps

• Per-second MGT.child_panic and MGT.child_died collection shows the exact moment a crash occurred and whether the child recovered or looped, without waiting for a manual varnishstat poll.
• MAIN.uptime and MGT.uptime as continuous gauges make the crash loop pattern immediately visible: MAIN.uptime resetting to zero while MGT.uptime keeps climbing is definitive.
• Cache hit rate correlation shows the operational impact: hit rate dropping to zero on each restart and failing to recover in a loop confirms backends are taking full traffic.
• Process RSS alongside configured storage size provides early warning before the OOM killer fires, especially for unbounded transient storage growth.
• Correlating Varnish child crashes with system-level signals (memory pressure, dmesg OOM events, cgroup limits) in a single timeline eliminates the gap between “Varnish restarted” and “why did it restart.”

Related guides

• Varnish Error 503 Backend fetch failed: what the error page actually means
• Varnish backend_fail, backend_unhealthy, and backend_busy: three different backend problems
• Varnish backend connection reuse low: keepalive not working and slow TTFB
• Varnish backend probe configuration: threshold, window, interval, and initial
• Varnish backend is sick: health probes, all-backends-sick, and grace
• Varnish ban list growing: O(n) lookups and the lurker falling behind
• Varnish ban lurker not keeping up: contention and ban_lurker_sleep
• Varnish cache hit ratio dropped: hit rate collapse and backend overload
• Varnish cache stampede: a popular object expires and the herd hits the backend
• Varnish ESI errors: broken pages and workspace pressure from Edge Side Includes
• Varnish fetch_failed: backend connected but the fetch broke
• Varnish grace masking a backend outage: the ticking-clock incident