HAProxy show pools: finding which internal pool is consuming memory

HAProxy’s memory footprint is not one number. The process holds dozens of internal object pools: buffers, connections, sessions, tasks, stick-table entries, SSL state. When RSS climbs, when PoolFailed goes nonzero, or when you are sizing a memory budget for a new instance, the question is never “how much memory is HAProxy using” but “which pool is using it”. show pools is the runtime API command that answers that.

PoolFailed in show info tells you an allocation failed. It does not tell you what failed to allocate. show pools is the diagnostic companion: it lists every internal pool with used versus allocated counts, per-pool failure counters, and average demand, so you can distinguish a connection-buffer blowup from stick-table growth from SSL-context memory.

The command is available since HAProxy 1.9 and is safe to run on a production process: it is read-only and does not flush or reclaim anything.

What show pools gives you

Run it through the stats socket like any other runtime command:

# Dump all internal memory pools
echo "show pools" | socat unix-connect:/var/run/haproxy.sock stdio

Each line is one pool. The output includes the pool name, the item size in bytes, the allocated count and total allocated bytes, the used count, a needed_avg field, a failures counter, and a [SHARED] flag on pools shared across threads. Item sizes are rounded up to the nearest multiple of 16 bytes, so small objects look larger than their struct definitions suggest.

Two things in the output trip people up:

• Used includes thread caches. Since 2.9 the output annotates (~N by thread caches). Objects recently freed by a thread sit in a per-thread cache and still count as used. A pool can show a high used count with zero active connections. That is normal, not a leak.
• Allocated is high-water behavior, not current demand. Pools grow to satisfy peak demand and do not aggressively return memory to the OS. Allocated greater than used is expected.

Pair show pools with the summary fields from show info:

# Pool summary and memory limit
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | \
  grep -E "^(Memmax_MB|PoolAlloc_MB|PoolUsed_MB|PoolFailed|CurrConns):"

A diagnostic flow

flowchart TD
  A[RSS growth or PoolFailed nonzero] --> B[show info: PoolUsed_MB vs PoolAlloc_MB vs Memmax_MB]
  B --> C[show pools byusage]
  C --> D{Largest growing pool?}
  D --> E[Buffers: CurrConns x 2 x tune.bufsize]
  D --> F[Stick tables: show table used vs size]
  D --> G[SSL pools: handshake rate, Lua, cert storage]
  D --> H[Monotonic growth across reloads: suspect leak, check version]

Sorting and filtering the output

On a busy instance the unfiltered dump is long. Since HAProxy 2.7 the command accepts sorting and filtering options:

# Sort by total usage, largest consumers first (2.7+)
echo "show pools byusage" | socat unix-connect:/var/run/haproxy.sock stdio

# Sort by item size
echo "show pools bysize" | socat unix-connect:/var/run/haproxy.sock stdio

# Only pools whose name starts with a prefix, limit to 10 entries
echo "show pools match ssl 10" | socat unix-connect:/var/run/haproxy.sock stdio

byusage is the one you want during an incident: it sorts by total usage in reverse order and only shows pools with used entries, so the top of the list is your answer. byname sorts alphabetically, useful for diffing two captures.

Since HAProxy 3.2, show pools detailed shows extra per-pool information, including which pools have been merged. Pool merging combines pools whose item sizes are close together to reduce pool count and memory overhead, so a single reported pool may back more than one object type. If you are comparing output across versions, expect the pool list on 3.2 to be shorter than on older releases.

Reading the fields that matter

The exact relationship between needed_avg and when a pool grows is not formally documented. Treat it as a trend signal. If needed_avg sits far below allocated for weeks, the pool was sized for a peak that is not coming back. If it tracks close to allocated, the pool is under real pressure.

Mapping pools to subsystems

The pool names map onto HAProxy’s internal subsystems. Knowing which subsystem owns the growing pool is what turns show pools output into an action.

• Buffer pools. Two buffers per connection (request and response), each tune.bufsize bytes, 16384 by default. Add roughly 20 KB of connection metadata and the rule of thumb from the HAProxy documentation is about 33 kB per connection. If the buffer pool dominates, the driver is CurrConns. Reduce concurrency or reduce tune.bufsize; both trade against real capacity.
• Connection and session pools. Scale with concurrent sessions and with idle backend connections kept alive for reuse (http-reuse). Growth here tracks CurrConns and connection pool sizing.
• Task pools. Proportional to active connections plus health checks. Rarely the problem on their own; if tasks grow, connections grew first.
• Stick-table entries. Proportional to configured table sizes and entry data types. Cross-check with show table, which reports used versus size per table. A table near capacity is both a memory consumer and a functional risk: eviction is silent, and with nopurge new entries are rejected, which breaks rate limiting and persistence for new clients.
• SSL pools. Session cache, OCSP stapling state, certificates in memory, and per-connection SSL context. Large certificate bundles cost megabytes before a single connection arrives. High new-handshake rates (SslFrontendKeyRate) churn the context pools.

These have completely different fixes. Buffer pressure means tuning tune.bufsize or maxconn. Stick-table growth means resizing tables or shortening expire. SSL pool growth means reviewing session cache sizing, certificate count, or what is loaded into the process.

Distinguishing pressure from a leak

Most show pools investigations end in “this is demand, not a leak”. Before concluding otherwise, eliminate the normal explanations:

1. Thread caches. On multi-threaded instances (nbthread > 1), the used count includes per-thread caches. Quiet instance, high used count, large (~N by thread caches) annotation: normal.
2. High-water allocation. PoolAlloc_MB above PoolUsed_MB after a traffic peak is the allocator caching, not leaking.
3. Reload residue. After a reload, the old worker retains its pools until it finishes draining. Total HAProxy memory is the sum of all running processes. Check pgrep -c haproxy and Stopping in show info on the old process before blaming the new one. Frequent reloads with long-lived connections (WebSocket, streaming) turn this into real growth; hard-stop-after bounds it.
4. Connection growth. Recompute the expected buffer footprint: CurrConns x (2 x tune.bufsize + ~20 KB metadata). If the pool numbers match the formula, the “leak” is traffic.

A genuine leak has a specific signature: one or two pools growing monotonically across hours and across reloads of the same binary, uncorrelated with CurrConns, with needed_avg climbing alongside used. There is precedent. In HAProxy 2.4.x, a non-POSIX realloc behavior in the Lua allocator caused the h2s and ssl_sock_ct pools to grow unboundedly when Lua scripts were loaded (fixed in later 2.4.x patches). If you see that shape, check your version’s changelog before building a mitigation, and capture show pools twice with a known interval so the growth rate is in evidence.

One operational note: show pools does not flush pools. Per the management guide, sending SIGQUIT to a foreground process dumps and flushes pool state, but sending signals to a production worker is disruptive and can terminate it. For diagnosis, stick to the read-only socket command.

Sizing a memory budget

show pools is also the sizing tool. When you are setting a budget for a new instance or a new maxconn, work from the drivers:

• Connections. maxconn x ~33 kB at default tune.bufsize is the floor for buffer and connection memory at full saturation. Raising tune.bufsize to 32768 doubles the buffer term; lower maxconn proportionally or accept the larger footprint.
• Stick tables. Sum configured table sizes. Large tables with many stored data types can dominate pool memory on low-traffic instances.
• TLS. Session cache size plus in-memory certificates. Multiple large SAN bundles are not free.
• Headroom. Peaks, reload overlap (two processes briefly), and thread caches all sit on top.

Then enforce it. Memmax_MB in show info reports the per-process memory limit (0 means unlimited); the limit itself is set with the -m command-line option. Without a limit, HAProxy allocates until the OOM killer decides for you. That is a cliff edge, not a degradation curve. On a shared host, set the limit and alert on PoolUsed_MB / Memmax_MB crossing 80%.

Any nonzero PoolFailed means allocations have already failed and work has been dropped. Use show pools to find which pool’s failures counter is incrementing, then decide whether the fix is more budget, less demand, or a smaller tune.bufsize.

Signals to watch in production

Collect show pools byusage on a slow schedule (minutes, not seconds) so that when PoolFailed fires or RSS alarms, you already have the per-pool history to compare against.

How Netdata helps

• Netdata collects the show info memory fields (PoolAlloc_MB, PoolUsed_MB, PoolFailed) as time series, so the high-water-vs-used relationship and any failure events are visible without socket scripts.
• Plotting pool memory against CurrConns and session rate makes the demand-vs-leak distinction a visual one: correlated curves are traffic, diverging curves are a leak.
• Alerting on PoolFailed transitioning from zero catches allocation failures at the first occurrence rather than after the OOM killer acts.
• Per-backend and per-frontend session saturation metrics (scur/slim) explain sudden buffer-pool growth when a slow backend starts holding connections longer.
• Process-count and RSS tracking per HAProxy PID exposes reload residue, the most common false positive in pool-growth investigations.

Related guides

• HAProxy 502 Bad Gateway: backend connection and response failures
• HAProxy 503 Service Unavailable: no server is available to handle this request
• HAProxy 504 Gateway Timeout: timeout server and the backend timeout cascade
• HAProxy backend losing servers: active server count and cascade risk
• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy compression overhead: comp_byp, CPU cost, and when compression backfires
• HAProxy connect time (ctime) high: network latency and backend accept-queue overflow
• HAProxy connection errors (econ): backend connections refused, timed out, or reset
• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy 5xx delta: telling HAProxy-generated errors from backend errors
• HAProxy health checks green but the application is broken: when UP does not mean healthy
• HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable