HAProxy stuck in soft-stop: draining connections and hard-stop-after

You reloaded HAProxy minutes ago, but pgrep haproxy still shows two, three, maybe a dozen PIDs. The old process is not serving new traffic, but it refuses to die. Memory and file descriptor usage on the host keep climbing with every reload, and your monitoring loses counter history each time it happens.

The old process is in soft-stop: it has unbound from its listeners and is waiting for its existing connections to close before exiting. A drain of a few seconds to a couple of minutes is normal. A process draining for more than 10 minutes is stuck, and the cause is almost always long-lived connections (WebSocket, Server-Sent Events, gRPC streaming, or plain TCP keepalive sessions) that have no reason to close on their own.

Without hard-stop-after, HAProxy waits forever. A single idle WebSocket can pin an old process, and its memory and file descriptors, indefinitely.

What this means

A graceful reload or shutdown sends the old HAProxy process a soft-stop signal (SIGUSR1). The process stops accepting new connections, unbinds from its listening sockets, and continues processing only the connections it already holds. When the last connection closes, the process exits cleanly.

The show info field Stopping is the canonical signal: Stopping: 0 means normal operation, Stopping: 1 means soft-stop in progress. Brief is expected. Lingering is the symptom.

stateDiagram-v2
  [*] --> Active : serving traffic
  Active --> SoftStop : SIGUSR1 (reload / graceful stop)
  SoftStop --> Draining : Stopping=1, no new connections
  Draining --> Exited : last connection closes
  Draining --> Stuck : long-lived connection never closes
  Stuck --> Killed : hard-stop-after expires
  Stuck --> Stuck : no hard-stop-after (default: forever)

A stuck process serves no new traffic but still costs you: it holds file descriptors (two per proxied connection, plus listeners and health check sockets), it holds memory (connection buffers, stick tables, SSL session cache), and the reload that orphaned it reset all cumulative stats counters on the new process, creating monitoring gaps. Stack several reloads on top of each other and you get the reload storm pattern: multiple draining processes accumulating until the host runs out of FDs or memory.

Common causes

CauseWhat it looks likeFirst thing to check
Long-lived HTTP connections (WebSocket, SSE)Process draining for hours; session count on the old process shrinks very slowlyshow sess on the old process, look for upgraded or streaming sessions
gRPC / HTTP2 streamingSame as above, but sessions show H2 streams that never endshow sess, inspect stream state
TCP-mode keepalive connectionsOld process holds a handful of TCP sessions with no trafficshow sess on the old process
Missing hard-stop-afterEvery reload leaves at least one drainer behindgrep hard-stop-after in the config; default is infinity
Frequent reloads (reload storm)Many haproxy PIDs, rising system FD and memory usagepgrep -c haproxy, check what triggers reloads
hard-stop-after too aggressiveClients hard-disconnected on every reload; reconnect spikes in app logsCorrelate reload timestamps with client disconnect reports

Quick checks

These are all read-only.

# How many HAProxy processes are running?
pgrep -c haproxy

# Is the process in soft-stop? Stopping: 1 = draining
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep "^Stopping:"

# How long has this process been up? A drainer with a large Uptime_sec
# relative to your last reload has been stuck since that reload.
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep "^Uptime_sec:"

# What connections is the draining process still holding?
echo "show sess" | socat unix-connect:/var/run/haproxy.sock stdio | head -50

# How many connections is it holding?
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep "^CurrConns:"

# System-level FD usage of the old process
HPID=<old-haproxy-pid>
ls /proc/$HPID/fd | wc -l

# Is hard-stop-after configured at all?
grep -n "hard-stop-after" /etc/haproxy/haproxy.cfg

After a reload, each process may have its own stats socket (for example, one per worker PID in master-worker mode). Make sure you are querying the draining process, not the new active worker. The new worker shows Stopping: 0 and near-zero counters, which can hide the problem.

How to diagnose it

  1. Confirm the state. Check Stopping: 1 in show info for the old process and note Uptime_sec. If the process has been in soft-stop for more than 10 minutes, treat it as stuck rather than draining.

  2. Count the processes. pgrep -c haproxy. More than two or three concurrent processes (one active worker plus one drainer) points to a reload storm: reloads happening faster than drains complete.

  3. Identify what is holding the drainer open. Run show sess against the old process’s stats socket and look at what remains. Long-lived sessions stand out: WebSocket upgrades, SSE endpoints, gRPC streams, or TCP-mode sessions with high age and no recent activity. If CurrConns on the drainer is small but nonzero and static, those are your pinning connections.

  4. Check the config. If hard-stop-after is absent from both the global and defaults sections, the drain timeout is infinity. That is the default, and it is the root configuration gap in nearly every stuck soft-stop incident.

  5. Check the reload trigger. If processes are accumulating, find what is reloading HAProxy: config management, service discovery, certificate rotation, an ingress controller. Reloads faster than drains will accumulate processes no matter what the drain timeout is.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Stopping (show info)Marks a process as draining1 for more than 10 minutes
HAProxy process countDetects reload storms and accumulating drainersMore than 2-3 PIDs
Drain duration (per process)The direct measure of this failure modeExceeds your hard-stop-after intent
CurrConns on the draining processShows how many connections still pin itNonzero and static over minutes
System FD usage vs ulimitDrainers hold FDs against system limitsRising across reloads
System memoryDrainers hold buffers, stick tables, SSL cacheStepwise growth correlated with reloads
Uptime_secDetects counter resets on the new processSudden drop = a reload happened

Fixes

Set hard-stop-after (the actual fix)

Add a drain bound in the global section:

global
    hard-stop-after 60s

After the timeout, the draining process is force-killed even if connections remain. Without this directive the timeout is infinite, which is exactly the bug you are hitting.

The tradeoff is the tuning knob: any client still connected when the timer fires gets a hard disconnect. For request/response HTTP traffic, drains finish in seconds and hard-stop-after 30s to 60s is conservative. For WebSocket, SSE, or gRPC streaming workloads, connections legitimately live for minutes or hours, so decide which is worse: killing those clients on every reload, or letting old processes linger. A common compromise is a larger window (5 to 15 minutes) combined with client-side reconnect logic, which well-behaved streaming clients should have anyway. Do not set this to a few seconds unless your workload is strictly short-lived requests.

Spread the close with close-spread-time

close-spread-time spreads the closing of idle and active connections over a time window during soft-stop instead of closing them all at once, which softens the reconnect thundering herd on the new process. It must be set lower than hard-stop-after, otherwise connections will not finish closing before the hard stop kills them.

Reduce reload frequency

If the trigger is service discovery or an ingress controller reacting to every pod event, batch or debounce updates. Every reload resets cumulative counters on the new process, invalidates the in-process SSL session cache, and cold-starts backend connection pools. Reloads are cheap, not free.

Use master-worker mode

In master-worker mode the master orchestrates worker lifecycle, and old workers enter soft-stop under its supervision. It does not remove the need for hard-stop-after, but it makes process accounting and signals (Stopping, per-worker stats sockets) easier to reason about than unmanaged fork-per-reload setups.

Do not SIGKILL as the routine answer

Killing the stuck process with kill -9 frees resources immediately but hard-drops every connection it holds and bypasses any cleanup. As a one-off during an incident it is acceptable when the process has been stuck for hours and the host is running out of FDs or memory. As a standing practice it just hides the missing hard-stop-after configuration.

Prevention

  • Always set hard-stop-after in global config. Treat the absence of this directive as a misconfiguration on any host that serves long-lived connections. The default (infinity) is only safe for pure short-request workloads.
  • Alert on drain duration. Alert when any process reports Stopping: 1 for more than 10 minutes. Brief drains are normal; lingering ones are not.
  • Alert on process count. pgrep -c haproxy above 2-3 sustained indicates reloads outpacing drains.
  • Track reload frequency. Correlate Uptime_sec resets with your deployment pipeline. If reloads happen more often than every few minutes, fix the trigger before tuning the drain.
  • Handle counter resets in monitoring. All cumulative counters reset on reload. Rate calculations must tolerate resets, or every reload will produce phantom metric drops.
  • Size client expectations. If you serve WebSocket or SSE, document that reloads will disconnect clients within the hard-stop-after window and verify clients reconnect cleanly.

How Netdata helps

  • Netdata collects show info fields including Stopping and Uptime_sec per process, so a draining process and its age are visible without SSHing in during an incident.
  • Process-level charts (FD count, memory) per HAProxy PID make a stuck drainer obvious: the new worker’s counters reset while the old process’s resource usage stays flat and nonzero.
  • Correlating drain duration with reload events (visible as Uptime_sec resets) tells you whether you have one pinned connection or a systemic reload storm.
  • Alerting on “process in soft-stop longer than N minutes” and on HAProxy process count turns this from a 3 a.m. discovery into a ticket during business hours.
  • Connection and session charts on the draining instance show whether CurrConns is shrinking (healthy drain) or static (pinned by long-lived connections), which decides whether you wait or tune hard-stop-after.