HAProxy reload storm: lingering processes leaking file descriptors and memory

You log into the HAProxy host because memory and file descriptor usage have been climbing for days, and you find a dozen haproxy processes running. Only one is serving traffic. The rest are old workers stuck in soft-stop, each holding open connections, connection buffers, and file descriptors it will never release. Traffic looks fine, per-process metrics look fine, and the host is still slowly running out of resources.

Every config reload spawns a new HAProxy worker while the old one enters a graceful drain. If connections on the old worker never close, the old worker never exits. Reload often enough and the drain backlog grows faster than it clears. The signature: pgrep -c haproxy well above 2-3, Stopping: 1 on old workers, and system-level FD and memory usage rising while HAProxy’s own per-process metrics look normal.

This is most common in Kubernetes Ingress Controller deployments and other environments with automated config generation: every endpoint change, Ingress update, or service discovery event can trigger a reload. A few reloads per day is harmless. A few per minute, combined with long-lived connections, is a slow leak that ends in FD exhaustion or an OOM kill of the worker actually serving traffic.

What this means

HAProxy reloads are designed to be seamless. In master-worker mode, the master receives SIGUSR2 (for example via systemctl reload haproxy), spawns a new worker with the new configuration, and tells the old worker to soft-stop: stop accepting new connections, finish draining existing ones, then exit. The old worker sets Stopping: 1 in show info and waits.

The design assumption is that connections close. A short-lived HTTP request drains in milliseconds. WebSockets, Server-Sent Events, gRPC streams, long-polling, and keepalive connections with active traffic can keep the old worker alive indefinitely. There is no default time limit on the drain: hard-stop-after is unset by default, so a single idle WebSocket can pin an old process for days.

Each lingering worker holds roughly two file descriptors per proxied connection plus listener and health check sockets, and its full connection buffer and memory footprint. The new worker’s stats show a small, healthy process. The sum across all processes is what consumes the host.

flowchart TD
    A[Config change triggers reload] --> B[New worker starts with fresh counters]
    B --> C[Old worker receives soft-stop signal]
    C --> D{Connections drained?}
    D -->|yes| E[Old worker exits, resources freed]
    D -->|no: long-lived or idle connections| F[Old worker lingers with Stopping: 1]
    F --> G[Holds FDs, buffers, memory]
    A2[Next reload repeats the cycle] --> B
    G --> H[Accumulating processes exhaust host FDs or memory]

Two side effects make this worse. First, every reload resets all cumulative stats counters to zero, so rate-based dashboards show gaps or phantom drops. Second, lingering workers keep running health checks against backends, adding probe traffic that no longer corresponds to real routing decisions.

Common causes

CauseWhat it looks likeFirst thing to check
No hard-stop-after configuredOld processes linger for hours or days; PID count grows monotonicallygrep hard-stop-after /etc/haproxy/haproxy.cfg (check included files too)
Long-lived connections (WebSocket, SSE, gRPC, long-poll)Old workers stuck in soft-stop with a few connections that never closeshow sess on the old process for connection ages
Reloads too frequentReload timestamps in logs every few minutes; PID count tracks change ratejournalctl -u haproxy --since "1 hour ago" | grep -i reload or ingress controller logs
Kubernetes Ingress Controller churnReloads on every endpoint/Ingress/CRD eventController logs; reload events per hour
Not using master-worker modeEach reload forks a fully independent process; worse accumulationmaster-worker in global config or -W flag in the service unit
Keepalive-heavy clients (for example SSH over TCP proxy)Connections stay ESTABLISHED indefinitely due to keepalive packetsshow sess for session age and state on draining workers

Quick checks

All read-only and safe to run during an incident.

# 1. Count haproxy processes. More than 2-3 persistent PIDs is abnormal.
pgrep -c haproxy
ps -eo pid,etimes,rss,cmd | grep [h]aproxy

# 2. Check whether a worker is in soft-stop
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep -E "^(Stopping|Uptime_sec|CurrConns):"

# 3. FD usage per haproxy process
for p in $(pgrep -x haproxy); do
  echo "pid $p: $(ls /proc/$p/fd 2>/dev/null | wc -l) fds, limit: $(awk '/Max open files/{print $4}' /proc/$p/limits)"
done

# 4. Memory per haproxy process
ps -o pid,rss,cmd -C haproxy --sort=-rss

# 5. What is holding connections open on a draining worker
echo "show sess" | socat unix-connect:/var/run/haproxy.sock stdio | head -50

# 6. How often reloads are happening
journalctl -u haproxy --since "6 hours ago" | grep -ciE "reload|stopping|new worker"

Caveat on checks 2 and 5: after a reload, the stats socket is typically taken over by the new process, so if show info answers, it is the new worker answering. To inspect old workers, use /proc/<pid> data, ss -p filtered by PID, or per-process sockets if your deployment exposes them.

# Old-worker connection state via ss (no stats socket needed)
for p in $(pgrep -x haproxy); do
  echo "pid $p: $(ss -tanp 2>/dev/null | grep -c "pid=$p,") established sockets"
done

How to diagnose it

  1. Confirm the pattern. pgrep -c haproxy should be 1 (standalone) or 2 (master plus one active worker) in steady state. A count of 3+ that persists for more than a few minutes, or grows over hours, confirms lingering processes. Check etimes from ps: old workers with ages matching past reload times are the leak.

  2. Confirm the drain state. On any worker you can still query, Stopping: 1 confirms soft-stop. For old workers you cannot query, infer from age and process count.

  3. Identify what is pinning connections. Use show sess (if reachable) or ss -tanp against the old PID. Sessions in ESTABLISHED state with ages in the hours range, especially on ports serving WebSocket, streaming, or TCP-mode proxies, are the blockers. If sessions show regular traffic, client keepalives may be keeping them alive.

  4. Measure the reload rate. Count reload events in the last hour from journald or the ingress controller logs. Compare the reload interval to the longest observed drain time: if reloads arrive faster than drains complete, the PID count can only grow.

  5. Quantify the resource damage. Sum FDs across all haproxy PIDs against the per-process ulimit and the system limit (/proc/sys/fs/file-max). Sum RSS across PIDs against host memory. Note that during every reload, old and new process FDs coexist, so peak FD usage during a reload window is roughly double steady state even in a healthy setup.

  6. Check the monitoring side effects. Look at your time series for counter resets: sudden drops to zero in cumulative metrics like hrsp_5xx, econ, bin. If your monitoring does not handle resets, reload storms also show up as phantom rate spikes and data gaps. A drop in Uptime_sec is the reliable reload detector.

  7. Rule out the upstream trigger. In Kubernetes, correlate reloads with endpoint churn: frequent pod rescheduling, a flapping readiness probe, or a CRD/Ingress being updated in a loop can drive continuous reloads. Tuning HAProxy without fixing the trigger only slows the leak.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
HAProxy PID count (pgrep -c haproxy)Direct measure of lingering processes; the primary reload storm indicator> 2-3 persistent processes, or a rising trend
Stopping field in show infoConfirms a worker is drainingStopping: 1 lasting more than a few minutes
System FD usage across all haproxy PIDs vs ulimitFD exhaustion is a hard cliff; accept() fails immediately on the active workerTotal usage above ~60-80% of limit, or rising between reloads
Aggregate RSS across all haproxy PIDsOld workers hold their full memory footprint until exitGrowing sum while active-traffic metrics are flat
Uptime_sec dropsReliable reload detector; also explains counter resetsFrequent resets correlating with PID count growth
Reload events per hour (logs)The driver of the stormReload interval shorter than observed drain time
CurrConns on the active worker vs system-wide socket countLarge gap means connections live on old, uncounted workersSystem connection count much higher than HAProxy reports

Fixes

Bound the drain time with hard-stop-after

Set hard-stop-after in the global section. This is the definitive fix for indefinite lingering: after the configured time from receiving the soft-stop signal, the old process is force-killed regardless of remaining connections.

global
    hard-stop-after 60s

The value is a tradeoff: whatever you set is the maximum connection lifetime that survives a reload. HTTP workloads commonly use 30 seconds to 5 minutes. Workloads proxying legitimately long-lived connections need more: GitLab’s production infrastructure, where SSH keepalive connections blocked drains, settled on 30 minutes. If you serve WebSockets with multi-hour sessions, either the connections get cut at hard-stop-after or the processes linger; there is no third option.

The directive has existed since HAProxy 1.7. If you set it and old processes still linger past the window, check the version: a confirmed bug in 2.9.1 caused lingering processes with high CPU on reload when TLS traffic was present, fixed in 2.9.2.

Cap worker generations in master-worker mode

In master-worker mode, mworker-max-reloads limits how many reloads a worker can survive. Once a worker has lived through more reloads than the limit, it receives SIGTERM.

global
    master-worker
    mworker-max-reloads 10

This complements hard-stop-after: the reload-count cap catches workers that would otherwise survive across many generations even with modest per-reload drain times. If you are not running master-worker mode at all, enable it; without it, each reload forks an independent process tree and accumulation is worse.

Reduce reload frequency

This is the root-cause fix when the trigger is automated config management:

  • Batch changes. If config is regenerated on every service discovery event, debounce it: accumulate changes and reload at most once per N seconds.
  • Use the runtime API instead of reloads for dynamic backends. Server address changes, weight changes, and server add/remove can be applied at runtime through the stats socket (add server, set server addr, set server ... state) without a reload, without counter resets, and without new processes. In Kubernetes Ingress, prefer controller configurations that use dynamic server updates rather than full reloads.
  • Fix the churn source. Flapping readiness probes, rapidly rescheduling pods, or an Ingress object being updated in a loop will drive reload storms no matter how well HAProxy is configured.

Clear the existing backlog carefully

To reclaim resources now, kill old draining workers with SIGTERM. This drops every connection still held by that worker immediately; clients on those sessions see hard disconnects.

# DISRUPTIVE: drops all connections still held by old workers.
# Identify old workers first (all PIDs except the current active worker),
# then terminate them one at a time while watching client impact.
kill -TERM <old_worker_pid>

Do not kill the active worker or the master. If you are unsure which PID is active, the one bound to the listener ports (ss -tlnp | grep haproxy) and answering the stats socket is current.

If reloads stop working entirely

A known issue in HAProxy 3.2.x master-worker mode: if a new worker fails during startup (for example a missing map file), the master can stop processing subsequent SIGUSR2 reload signals, leaving reloads silently ineffective until a full restart. If systemctl reload seems to do nothing, check for failed worker startups in the logs and plan a restart at a safe window. A restart drops all connections, so schedule it deliberately.

Prevention

  • Set hard-stop-after everywhere. Treat it as a required global directive, not an optional tuning knob. Size it to the longest connection lifetime you are willing to protect.
  • Alert on process count, not just process existence. A PID count above 2-3 sustained for more than a few minutes is a ticket-level condition; catch it days before FD or memory exhaustion forces the issue.
  • Gate reload frequency. In automation that generates HAProxy configs, enforce a minimum reload interval and emit a metric for reloads per hour so the trigger is visible.
  • Prefer runtime API updates for dynamic state. Backend membership changes do not need process restarts; reserve reloads for structural config changes.
  • Handle counter resets in monitoring. Use Uptime_sec drops to detect reloads and make rate calculations reset-aware, so reload storms do not also corrupt your other alerts.
  • Capacity-plan for the reload window. Even in a healthy setup, FD and memory usage roughly double during a drain. Keep steady-state headroom above that: peak FD usage should stay under about 60% of ulimit.
  • Be aware of old-process health checks. Lingering workers continue probing backends. If backend health check load looks anomalous, correlate with PID count before blaming the backends.

How Netdata helps

  • Per-process visibility. Netdata charts per-process FD counts, memory, and CPU from /proc, which is exactly the view this problem needs: the sum across all haproxy PIDs, not just the active worker’s self-reported stats.
  • HAProxy stats collection with reload detection. Netdata collects the stats CSV and show info fields (including CurrConns, Maxsock, Stopping, Uptime_sec) at one-second granularity, so reloads show up as Uptime_sec drops you can correlate against PID count and system resource charts.
  • Correlation in one timeline. The diagnostic sequence above (PID count up, system FDs up, active-worker metrics flat, Uptime_sec resetting) crosses process-level, HAProxy-level, and system-level signals. Seeing them on the same timeline turns a multi-command investigation into a glance.
  • Alerting on the leading indicator. Alerting on haproxy process count and aggregate FD usage catches the storm while it is still a capacity trend, not an accept() failure cliff.
  • Post-fix verification. After setting hard-stop-after or reducing reload frequency, the PID count and FD trend lines are the direct proof the fix worked.