HAProxy memory growth and OOM: connection buffers, pools, and memory ceilings

The symptom usually arrives as a dead process, not a slow one. HAProxy was proxying fine, then the PID changed, connections dropped for a few seconds, and dmesg shows the OOM killer picked haproxy as the victim. Or, before it gets that far, RSS climbs day over day and nobody knows whether it is a leak or just load.

HAProxy has no internal memory ceiling unless you set one. It allocates what the workload demands, holds most of it in internal pools for reuse, and keeps going until the kernel OOM killer intervenes. Memory is a cliff-edge resource: fine, fine, fine, dead.

The good news: HAProxy memory consumption is dominated by one predictable term, current connections, and the allocator exposes counters that warn you before the cliff.

What this means

Every established connection costs roughly 2 x tune.bufsize (16KB each by default, so about 32KB) for the request and response buffers, plus around 20KB of connection and session metadata. Call it about 50KB per connection. At 10,000 concurrent connections that is roughly 500MB before anything else. CurrConns from show info times the per-connection cost is the dominant term in any memory estimate.

On top of that:

  • Stick tables consume memory proportional to their configured size and entry data types. Tables without a sane expire fill with stale entries.
  • The SSL session cache and certificates add steady-state memory, larger with big cert bundles.
  • DNS cache and pool caching add smaller amounts.

The second thing to understand is pool behavior. HAProxy’s allocator keeps released objects in internal pools for reuse instead of returning them to the OS. After a traffic spike, RSS stays high: PoolAlloc_MB (allocated from the OS) sits above PoolUsed_MB (actively in use). This looks like a leak and usually is not. A real leak shows PoolUsed_MB itself climbing without a matching climb in CurrConns or table usage.

When memory actually runs out, the PoolFailed counter in show info increments: HAProxy tried to allocate a buffer or session and failed. Any nonzero PoolFailed means requests were dropped or stalled for lack of memory.

Common causes

CauseWhat it looks likeFirst thing to check
Connection count growthRSS tracks CurrConns; step changes with trafficCurrConns trend vs RSS trend
Long-lived connections (WebSocket, SSE, slow clients)High scur, low req_rate, low throughputshow sess for session age and state
Stick tables without expireMemory grows with unique clients, never shrinksshow table used vs size
Pool retention after a spikeRSS jumped once, now flat; PoolUsed_MB stableshow info pool fields; not a leak
Reload stormMultiple haproxy PIDs, each holding memorypgrep -c haproxy
Real leak or version bugPoolUsed_MB climbs with flat trafficshow pools to find the growing pool

Quick checks

All read-only and safe to run during an incident.

# Connections and memory in one shot
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | \
  grep -E "^(CurrConns|Maxconn|Memmax_MB|PoolAlloc_MB|PoolUsed_MB|PoolFailed|Uptime_sec|Stopping):"

# Per-pool breakdown: which pool is large
echo "show pools" | socat unix-connect:/var/run/haproxy.sock stdio

# Stick table usage
echo "show table" | socat unix-connect:/var/run/haproxy.sock stdio

# Process count (reload storm check)
pgrep -c haproxy

# RSS of the worker
HPID=$(pgrep -x haproxy | head -1)
grep VmRSS /proc/$HPID/status

# Kernel OOM evidence
dmesg -T | grep -iE "out of memory|killed process"

Two things to note in the show info output. Memmax_MB is the configured memory ceiling; 0 means unlimited. PoolFailed should always be zero.

How to diagnose it

  1. Confirm whether an OOM kill actually happened. Check dmesg for the OOM killer naming haproxy, and check Uptime_sec in show info: a recently restarted process with no deploy record is a crash or kill. If there is no OOM record, you are looking at growth, not a kill, and you have time.

  2. Compute the expected connection memory. Take CurrConns, multiply by roughly 50KB (2 x tune.bufsize plus about 20KB metadata; more if you raised tune.bufsize). Compare against RSS. If expected connection memory explains most of RSS, this is workload, not a leak. The question becomes why connection count is high, which is a different investigation: see scur approaching slim.

  3. Check for idle connection pile-up. High CurrConns with low req_rate means connections are being held, not worked. Look at timeout client, timeout http-request, and timeout http-keep-alive, and inspect show sess for long-lived or stuck sessions. WebSocket and streaming deployments should expect this shape; everyone else should not.

  4. Check the pool fields. If PoolAlloc_MB is high but PoolUsed_MB is low and stable, the allocator is holding memory from a past spike. That is normal retention. If PoolUsed_MB itself climbs while CurrConns is flat, run show pools twice a few minutes apart and find which pool is growing. That pool name points at the subsystem.

  5. Check stick tables. show table reports used vs size per table. A table pinned near 100% with no expire configured (or a very long one) is steady memory that grows with unique client keys. Also note: with default settings, full tables evict expired entries silently; with nopurge, new entries are rejected. Neither failure mode shows up in memory counters directly.

  6. Count processes. If pgrep -c haproxy returns more than 2 or 3, old soft-stopped workers are holding memory. Check Stopping: 1 on old processes and whether hard-stop-after is configured. During a reload storm, total HAProxy memory is the sum across all PIDs, and system-level RSS keeps climbing even though each process looks normal.

  7. Only then suspect a leak. Flat traffic, flat CurrConns, flat tables, single process, and still-climbing PoolUsed_MB is the real leak signature. Capture show pools output, note your HAProxy version, and check the changelog before upgrading in place.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
CurrConns (show info)Dominant memory term at ~50KB per connectionClimbing without a traffic explanation
RSS of worker PIDWhat the OOM killer actually seesSustained upward trend across days
PoolUsed_MBMemory actively in use by HAProxy objectsRising while CurrConns is flat
PoolAlloc_MB vs PoolUsed_MBSeparates pool retention from real growthAlloc » Used after a spike is normal; Used rising is not
PoolFailedAllocation failures, dropped or stalled workAny nonzero value
Memmax_MBThe configured ceiling, 0 means none0 on hosts running other services
show table used/sizeStick table memory and eviction riskused > 80% of size, no expire
Stopping + PID countReload storms holding memory in old workersMore than 2-3 PIDs, lingering soft-stop
dmesg OOM eventsConfirms the kill and the victimhaproxy named in OOM killer output

Fixes

Reduce the per-connection footprint

The per-connection cost scales directly with tune.bufsize. If you raised it for large headers or big cookies, you raised memory per connection proportionally: at tune.bufsize 32768, buffers alone are about 64KB per connection. Do not lower bufsize blindly; too small causes request parse failures on large headers (ereq, 400s). Set it for your actual header sizes and account for the memory cost in capacity math.

Reap idle connections

If CurrConns is inflated by idle sessions, tighten the timeouts that hold them: timeout http-keep-alive, timeout client, and timeout http-request for slow senders. This frees buffers and file descriptors without touching capacity. Tradeoff: aggressive keep-alive timeouts cost clients reconnections and TLS handshakes, so lower them stepwise and watch SslFrontendKeyRate and cli_abrt.

Give stick tables an expiry

Every stick table should have an expire that matches the tracking window. A rate-limit table tracking 10-second rates does not need hour-long entry lifetimes. Right-size size as well; a table sized for 1M entries reserves for 1M entries. Tradeoff: shorter expiry evicts entries that are still legitimate if your window was too tight, so watch dreq behavior after the change.

Set a memory ceiling

If HAProxy shares a host with anything else, give it a ceiling. The ceiling is set at startup with the -m <MB> flag and shows up as Memmax_MB in show info.

When the ceiling is hit, HAProxy fails allocations (PoolFailed increments) and stalls or drops work instead of growing until the OOM killer fires. That converts an unbounded cliff into bounded, observable degradation. Tradeoff: you are choosing “HAProxy degrades under memory pressure” over “HAProxy survives until the kernel kills something, possibly something else.” On a shared host that is usually the right choice; on a dedicated host you may prefer headroom and an alert instead. An OS-level cgroup or systemd memory limit is a viable alternative, but the OOM killer inside the cgroup will terminate the process rather than degrade it.

Size the ceiling from your own math: expected peak CurrConns x per-connection cost, plus stick table configured sizes, plus SSL cache, plus headroom for pool retention after spikes.

Stop reload storms

If old workers linger in soft-stop, set hard-stop-after so draining processes are force-killed after a bounded window, and reduce reload frequency by batching config changes. Without hard-stop-after, a single WebSocket connection can pin an old worker, and its memory, indefinitely. Tradeoff: clients still connected when the timer fires get a hard disconnect.

Handle the aftermath of an OOM kill

If the OOM killer already fired, the process is gone and connections dropped. Restart it, then treat it as a capacity incident. Pull the numbers from steps 2 through 6, because without a ceiling and without trending on CurrConns and RSS, it will happen again at the next traffic peak.

Prevention

  • Do the capacity math once, write it down. Peak expected CurrConns x ~50KB, plus stick tables, plus SSL cache, plus 30% headroom. Compare against host RAM. This is a 10-minute exercise that prevents the 3 a.m. version.
  • Set a ceiling on shared hosts. -m (visible as Memmax_MB) so HAProxy degrades observably instead of triggering the kernel OOM lottery.
  • Alert on PoolFailed > 0. It is normally zero and it is the earliest in-process signal that memory pressure is dropping work.
  • Trend PoolUsed_MB and RSS against CurrConns. Divergence between them is the leak detector.
  • Expire every stick table. No table without an expire matching its tracking purpose.
  • Bound reloads. hard-stop-after set, reload frequency kept low, PID count monitored.
  • Gate on connection saturation upstream. Memory follows connections, so the scur/slim ratio is a leading indicator for memory too.

How Netdata helps

  • Netdata collects the show info fields that matter here, including CurrConns, PoolAlloc_MB, PoolUsed_MB, and PoolFailed, so the connection-to-memory relationship is visible on one dashboard instead of in ad-hoc socket commands.
  • Per-second RSS and system memory charts let you overlay HAProxy process memory against CurrConns to spot the divergence that separates leaks from pool retention.
  • Alerting on PoolFailed going nonzero and on RSS trend deviations catches allocator pressure before the OOM killer does.
  • Reload storms show up as process count changes and Uptime_sec resets, which Netdata tracks alongside memory, making the multi-PID memory pile-up obvious.
  • Correlating HAProxy memory signals with system-level OOM events and stick table utilization shortens the path from “RSS is climbing” to the specific term driving it.