Varnish threads_failed: the OS refusing to create worker threads

When MAIN.threads_failed is nonzero, the Linux kernel is refusing Varnish’s pthread_create() calls. This is an OS-level resource limit preventing the worker thread pool from growing, not a Varnish configuration problem.

Do not confuse this with MAIN.threads_limited, which increments when Varnish declines to create a thread because thread_pool_max has been reached. threads_limited means Varnish chose not to create the thread. threads_failed means Varnish tried and the OS said no. The remediation differs: threads_limited requires raising a Varnish parameter; threads_failed requires fixing OS-level limits.

Any nonzero value indicates system misconfiguration. In a properly tuned deployment, this counter is always zero. When it increments, Varnish cannot scale its worker pool, requests queue, and sessions are eventually dropped.

What this means

Each worker thread handles one request at a time. When traffic increases and idle threads run out, Varnish creates new threads up to thread_pool_max per pool (default 5000). If pthread_create() fails during this ramp-up, threads_failed increments. The thread was not created, so the request that needed it either waits in the bounded queue or, if the queue is full, is dropped.

pthread_create() returns EAGAIN (errno 11, “Resource temporarily unavailable”) when the OS cannot allocate the thread. Three categories of OS limits cause this:

1. Process/thread count limit. RLIMIT_NPROC (visible as “Max processes” in /proc/<pid>/limits) or a cgroup pids controller limit caps the total threads the Varnish user can create.

2. Memory map limit. Each thread allocates memory maps for its stack. The kernel’s vm.max_map_count caps total maps per process, which indirectly caps total threads.

3. Insufficient memory for thread stacks. Each thread stack costs 64KB-512KB depending on thread_pool_stack. If the system cannot commit this memory (cgroup memory limits, strict overcommit, or actual exhaustion), thread creation fails.

flowchart TD
    A["threads_failed nonzero"] --> B{"dmesg pids controller?"}
    B -->|"Yes"| C["systemd TasksMax"]
    B -->|"No"| D{"RLIMIT_NPROC at ceiling?"}
    D -->|"Max processes hit"| E["ulimit -u too low"]
    D -->|"Headroom available"| F{"vm.max_map_count low?"}
    F -->|"Near limit"| G["kernel map count"]
    F -->|"Adequate"| H["memory or cgroup limit"]

The parameter thread_pool_fail_delay (default 0.200 seconds) controls how long Varnish pauses before retrying a failed thread creation. This prevents a tight retry loop but does not resolve the underlying OS limit.

Common causes

Quick checks

All commands are read-only and safe for production.

# Confirm threads_failed is actually incrementing (take two readings 10s apart)
varnishstat -1 -f MAIN.threads_failed

# See the diagnostic message Varnish logs when thread creation fails
varnishlog -g raw -I Debug:thread

# Check if systemd's pids controller is the limiting factor
systemctl show varnish -p TasksCurrent -p TasksMax

# Check the RLIMIT_NPROC (Max processes) for the Varnish child process
cat /proc/$(pgrep -n varnishd)/limits | grep "Max processes"

# Check current thread count vs configured maximum
varnishstat -1 -f MAIN.threads -f MAIN.pools
varnishadm param.show thread_pool_max

# Check if kernel memory map count is the bottleneck
cat /proc/$(pgrep -n varnishd)/maps | wc -l
sysctl vm.max_map_count

# Check memory overcommit status
grep -E 'Committed_AS|CommitLimit' /proc/meminfo
sysctl vm.overcommit_memory

# Check the thread pool stack size (each thread costs this much memory)
varnishadm param.show thread_pool_stack

# Check whether the queue is filling as a consequence
varnishstat -1 -f MAIN.thread_queue_len -f MAIN.sess_dropped -f MAIN.req_dropped

How to diagnose

1. Confirm the counter is live, not stale. Take two varnishstat readings 10 seconds apart. If MAIN.threads_failed is not changing, you may be looking at a historical value from before a previous fix. MAIN.* counters reset on child process restart.

2. Check for the pids controller. Run dmesg | grep -i pids and look for “fork rejected by pids controller.” If present, systemd’s TasksMax is the culprit. This is the most common cause on modern Linux running Varnish under systemd.

3. Check RLIMIT_NPROC. Read /proc/$(pgrep -n varnishd)/limits and compare “Max processes” to the current thread count from MAIN.threads. If they are close, the user-level process/thread limit is the constraint.

4. Check vm.max_map_count. Count the memory maps for the Varnish child process with cat /proc/$(pgrep -n varnishd)/maps | wc -l. Compare to sysctl vm.max_map_count. A rough rule of thumb is two maps per thread, so the default of 65530 limits you to approximately 32,000 threads.

5. Check memory availability. Compare Committed_AS to CommitLimit in /proc/meminfo. If vm.overcommit_memory=2 (strict no-overcommit), the kernel enforces CommitLimit as a hard ceiling. Each thread stack (controlled by thread_pool_stack) consumes committed memory.

6. Read the Varnish debug log. Run varnishlog -g raw -I Debug:thread to see the exact failure. The typical message is “Create worker thread failed 11 Resource temporarily unavailable,” confirming EAGAIN from pthread_create().

Metrics and signals to monitor

Fixes

Identify which OS limit is binding before making changes.

systemd TasksMax (cgroup pids controller)

This is the most common cause on modern Linux. systemd v236+ applies a default TasksMax to services (typically 4915 or derived from system RAM), which caps total threads in the service cgroup.

Create a systemd override:

systemctl edit varnish

Add:

[Service]
TasksMax=infinity

Then apply:

systemctl daemon-reload
systemctl restart varnish

Warning: restarting Varnish flushes the cache. Schedule during a maintenance window if possible. Newer Varnish packages from the official repositories already ship with TasksMax=infinity in the unit file. Check whether this override is already present before adding it.

RLIMIT_NPROC (ulimit -u)

If “Max processes” in /proc/<pid>/limits is the binding constraint, raise it in the systemd unit:

[Service]
LimitNPROC=infinity

systemd services use LimitNPROC in the unit file, not PAM limits from /etc/security/limits.conf. Editing limits.conf has no effect on a systemd-managed Varnish process.

vm.max_map_count

If the process is hitting the kernel’s memory map limit:

# Apply immediately (affects future allocations, not existing threads)
sysctl -w vm.max_map_count=1048576

# Persist across reboots
echo 'vm.max_map_count=1048576' >> /etc/sysctl.d/99-varnish.conf

No restart of Varnish is needed. New thread creation will succeed once the map ceiling is raised.

Memory exhaustion or overcommit

If Committed_AS is near or above CommitLimit:

• If vm.overcommit_memory=2: Consider switching to 0 (heuristic overcommit, the kernel default) or 1 (always overcommit). Changing overcommit mode is a system-wide decision with tradeoffs beyond Varnish.

• If cgroup memory limit is binding: Raise MemoryMax for the Varnish service, or ensure the machine has enough physical RAM for configured storage plus thread stacks plus overhead.

• If physical memory is genuinely exhausted: Reduce thread_pool_stack to lower per-thread cost, reduce thread_pool_max, or add memory. At 10,000 threads (5000 per pool, 2 pools), stack memory alone costs 640MB to 5GB.

thread_pool_add_delay tuning (mitigation, not fix)

If threads_failed occurs in bursts during rapid traffic spikes, Varnish may be creating threads faster than the OS can allocate them. Increasing thread_pool_add_delay smooths the creation rate:

# Live change, no restart needed
varnishadm param.set thread_pool_add_delay 0.002

This is a mitigation. If the OS limit is the binding constraint, smoothing the creation rate only delays the failure. Always address the OS limit first.

Prevention

• Audit systemd limits on every new deployment. Check TasksMax and LimitNPROC before going live. Default values suit typical services, not high-concurrency caching proxies that may need thousands of threads.

• Calculate thread stack memory budget. Multiply thread_pool_max x thread_pools x thread_pool_stack to estimate worst-case stack memory. Ensure the machine or cgroup has headroom for this plus cache storage plus overhead.

• Monitor vm.max_map_count utilization. If your thread count is high, verify the map count ceiling is well above threads x 2.

• Alert on threads_failed. This counter should always be zero. Any nonzero value warrants immediate investigation, not waiting for session drops.

How Netdata helps

• Per-second granularity on MAIN.threads_failed catches short bursts that 15-second or 60-second polling intervals miss. Thread creation failures during traffic spikes can come and go within a single minute.

• Correlation with MAIN.threads, MAIN.threads_limited, and MAIN.thread_queue_len in a single view distinguishes OS-level refusal from config-level ceiling without switching tools.

• System-level metrics alongside Varnish counters correlate threads_failed spikes with cgroup memory pressure, system-wide memory utilization, and process RSS in the same dashboard. This is what separates a Varnish problem from an OS constraint.

• Downstream impact visibility through MAIN.sess_dropped and MAIN.req_dropped confirms whether thread creation failures are actually affecting users or being absorbed by the queue.

Related guides

• How Varnish actually works in production: a mental model for operators
• Varnish monitoring checklist: the signals every production cache needs
• Varnish monitoring maturity model: from survival to expert
• Varnish thread pool exhaustion: workers all busy, queue full, sessions dropped