HAProxy PoolFailed: buffer starvation and memory-allocator failures

PoolFailed in show info counts failed internal memory-pool allocations since process start. It is normally zero. Any nonzero value means HAProxy tried to allocate a buffer, connection, or session object and could not get one. Work was dropped or stalled because of it.

The tricky part is how this surfaces. Buffer starvation rarely looks like a memory problem from the outside. It looks like mysterious latency: rtime and backend health are fine, CPU has headroom, the network is clean, and yet clients see stalls and sporadic failures. PoolFailed is one of the few direct signals that HAProxy itself is refusing work because it ran out of an internal resource.

What this means

HAProxy pre-allocates memory in pools for high-frequency objects: connection buffers (two per connection, request and response, each sized by tune.bufsize, default 16384 bytes), sessions, connections, and related metadata (roughly 20KB of overhead per connection on top of the buffers). These pools sit between HAProxy and the system allocator so the hot path does not hit malloc() on every event.

A pool allocation can fail for three broad reasons:

  1. The buffers are all checked out. Under extreme concurrency with large payloads, every buffer in the pool is in use. New or existing connections stall waiting for one to be released.
  2. A configured memory limit is reached. If Memmax_bytes (shown as Memmax_MB in show info) is set, HAProxy stops allocating at that ceiling, even if the host has free RAM.
  3. The system cannot give HAProxy more memory. Host-level memory pressure means the underlying allocation fails. Left unchecked, the kernel OOM killer is the next step. HAProxy has no built-in memory limit unless Memmax_bytes is set.

The failure cascade:

flowchart TD
  A[Traffic surge or large payloads] --> B[All pool buffers checked out]
  B --> C{More memory available?}
  C -->|No: Memmax reached or host OOM pressure| D[Allocation fails - PoolFailed increments]
  C -->|Yes, but slowly| E[Connections stall waiting for buffers]
  D --> F[Work dropped or sessions fail]
  E --> G[Mysterious latency, no CPU/network cause]
  F --> H[If unchecked: kernel OOM killer]

The latency-only path (E) is what makes this painful: nothing in the usual error counters moves. The dropped-work path (D) is where PoolFailed finally gives you a counter to look at.

Common causes

CauseWhat it looks likeFirst thing to check
Buffer starvation under extreme concurrencyLatency spikes with healthy backends and normal CPU; PoolFailed nonzero; CurrConns at or near peakshow pools output; CurrConns trend
Memmax_bytes ceiling reachedPoolUsed_MB pinned just under Memmax_MB; host has free RAMCompare PoolUsed_MB to Memmax_MB in show info
Host memory pressure / pre-OOMSystem free memory near zero, HAProxy RSS growing; dmesg may show OOM killer activityfree -m, HAProxy process RSS, dmesg
Reload storm inflating total memoryMultiple HAProxy processes, each holding its own pools; system memory climbs while per-process stats look normalpgrep -c haproxy, Stopping: 1 on old processes
Buffer size vs workload mismatchLarge headers or bodies; tune.bufsize raised without accounting for per-connection memory (2 x bufsize + ~20KB per connection)Current tune.bufsize; estimated memory per connection vs CurrConns
Counter left over from before a reloadPoolFailed nonzero but static; no current symptomsTake two samples; if the value is not moving, it is historical

Quick checks

All read-only. Adjust the socket path to your deployment.

# PoolFailed and the memory summary in one shot
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | \
  grep -E "^(PoolFailed|PoolAlloc_MB|PoolUsed_MB|Memmax_MB|CurrConns|Maxconn|Idle_pct|Stopping):"

# Per-pool breakdown: which object type is exhausted
echo "show pools" | socat unix-connect:/var/run/haproxy.sock stdio

What to look for:

  • PoolFailed: should be zero. Any nonzero value is real.
  • Memmax_MB: 0 means no configured limit; the ceiling is then host memory and the OOM killer.
  • PoolUsed_MB vs PoolAlloc_MB: used is what is actively checked out; allocated is what the pools have taken from the OS. Allocated can legitimately sit above used because pools cache freed objects.
  • show pools: per-pool size, count, and failure counts. The pool with failures is the resource that ran dry, which tells you whether you are out of buffers or out of session/connection objects.
# Host-level memory and OOM evidence
free -m
dmesg -T | grep -i -E "out of memory|oom-kill" | tail -20

# HAProxy resident memory and process count (reload storm check)
HPID=$(pgrep -x haproxy | head -1)
grep -E "VmRSS" /proc/$HPID/status
pgrep -c haproxy

A process count above 2-3 with Stopping: 1 on old processes means old workers are still draining and still holding their pools. Total HAProxy memory is the sum across all processes, and counters like PoolFailed reset on each reload, so a fresh process can show zero while the system is still tight.

How to diagnose it

  1. Confirm the failure is current. Sample PoolFailed twice, a minute apart. A nonzero but static value is history from before a reload. A moving value is an active allocation failure.

  2. Identify the failing pool. Run show pools and find the pool line with failures. Buffer-pool failures point at concurrency and payload size. Session or connection object failures point at connection count and per-connection metadata.

  3. Check the configured ceiling. If Memmax_MB is nonzero and PoolUsed_MB is riding just under it, the limit is the bottleneck regardless of free host RAM. If Memmax_MB is 0, the effective ceiling is host memory.

  4. Check host memory pressure. free -m and HAProxy RSS tell you whether the system can give HAProxy more. dmesg shows whether the OOM killer has already been involved. With Memmax_bytes unset, HAProxy allocates until the kernel intervenes.

  5. Estimate expected memory per connection. Each connection costs roughly 2 x tune.bufsize + ~20KB. Multiply by CurrConns (and by process count if old draining processes exist). If that estimate is close to Memmax_MB or to available host memory, you have a sizing problem, not a leak.

  6. Rule out the lookalikes. Check Idle_pct (unless busy-polling is enabled, in which case it is meaningless; use show activity instead) and backend rtime. Healthy CPU and backends plus spiking latency plus a moving PoolFailed points at buffer starvation.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
PoolFailed (show info)Direct count of failed internal allocationsAny nonzero value; a rate of increase during peaks
PoolUsed_MB / PoolAlloc_MBPool memory in use vs held from the OSPoolUsed_MB trending up without matching traffic growth
Memmax_MBThe configured memory ceiling (0 = unlimited)PoolUsed_MB approaching it
CurrConnsDrives buffer and connection-object demandPeak values near levels where PoolFailed last fired
Idle_pctSeparates CPU saturation from memory stallsHealthy idle while latency spikes: consistent with buffer starvation
HAProxy process RSS and process countCatches reload storms and system-level pressureRSS climbing across multiple lingering processes
System free memory / OOM killsThe final failure mode when no limit is setOOM killer events in dmesg

Fixes

Add memory headroom or raise the ceiling

If Memmax_bytes is set and the workload legitimately needs more, raise it. If the host is the constraint, add RAM or reduce co-located consumers. Setting Memmax_bytes on a shared host is still the right call: it makes HAProxy fail allocations (visible in PoolFailed) instead of growing until the OOM killer picks a victim, which may not be HAProxy.

Reduce per-connection memory

tune.bufsize (default 16384) is paid twice per connection. Lowering it reduces the footprint linearly but risks parse failures for large headers, which surface as ereq and 400s. Raising it for big-header workloads must be budgeted: 2 x bufsize x peak CurrConns plus metadata. Treat any bufsize change as a capacity decision, not just a protocol one. Very large SSL records may not fit in a 16KB buffer; some deployments raise it to 17-18KB for that reason, which raises per-connection cost further.

Control buffer pool growth explicitly

tune.buffers.limit caps the total number of buffers HAProxy will allocate globally, and tune.buffers.reserve keeps buffers aside for allocation-failure conditions so in-flight sessions can finish. Together they convert an unbounded failure mode (grow until OOM) into a bounded one (fail early, visibly, and recoverably).

Fix the reload storm

If multiple draining processes are the memory consumer, set hard-stop-after so old workers are force-killed after a bounded drain, reduce reload frequency, and use master-worker mode. Without hard-stop-after (default: no limit), a single long-lived connection can keep an old process and its pools alive indefinitely.

Do not flush pools on a live process as a fix

show pools is safe and does not flush anything. Sending SIGQUIT to force pool flushing on a busy production process is disruptive and has been associated with crashes on loaded instances. Diagnose with show pools; fix with sizing and limits, not with signals.

Allocator-level notes

On HAProxy 2.4 and later, periodic malloc_trim() behavior can be disabled with no-memory-trimming in the global section. Very recent versions have seen watchdog kills triggered by pool trimming under high load; if you hit that, no-memory-trimming is the documented workaround.

Prevention

  • Alert on any nonzero PoolFailed rate. It should always be zero. This is a Level 4 signal in the HAProxy monitoring maturity model for a reason: it is the earliest direct evidence of memory-allocator failure.
  • Budget memory before raising concurrency. Any increase to maxconn, tune.bufsize, or stick-table size is a memory commitment. Estimate CurrConns x (2 x bufsize + ~20KB) against Memmax_bytes and host RAM before deploying.
  • Set Memmax_bytes on shared hosts. Bounded, visible allocation failures beat an unbounded march to the OOM killer.
  • Keep draining-process headroom in mind during reloads. Total HAProxy memory during a reload window is old process plus new process.
  • Trend PoolUsed_MB against CurrConns. Divergence between the two over weeks is either a leak or an unbudgeted consumer such as growing stick tables.

How Netdata helps

  • Netdata collects the show info fields continuously, so PoolFailed, PoolUsed_MB, Memmax_MB, and CurrConns are graphed together at per-second resolution instead of sampled by hand during an incident.
  • Correlating PoolFailed increments against CurrConns peaks answers the first diagnostic question immediately: concurrency-driven starvation or a fixed ceiling?
  • System-level memory, per-process RSS, and OOM events are collected on the same host and timeline, so host pressure and HAProxy pool pressure appear in one view.
  • Reload detection via process uptime and counter resets prevents misreading a post-reload zero as “problem solved.”
  • Alerting on a nonzero PoolFailed delta catches buffer starvation during the latency-only phase, before sessions start failing.