$ guides / haproxy ▌

HAPROXY · OPERATIONS PLAYBOOK

HAProxy fails at the edges: a connection ceiling, a health-check cascade, and errors that hide which side broke

An event-driven L4/L7 proxy where every proxied request crosses four independent connection limits, two file descriptors, a CPU-hungry TLS handshake, and a queue — and where a 503 can mean a saturated frontend, an all-DOWN backend, or a full stick table. We trace how that pipeline behaves under load, where throttling turns into rejection, and what to look at when it does.

> Start with the monitoring checklist → # Jump to the full guide list

HAProxy is rock-solid until it hits an edge, and its edges are cliffs — the process looks perfectly healthy right up to the moment it stops accepting work.

The defaults get you to production fast. Then one frontend crosses its maxconn and new connections queue at the kernel then get rejected. A backend loses enough servers that the survivors overload, fail their own health checks, and the whole backend goes DOWN — every request to it now a 503. A slow dependency holds every connection slot hostage until timeout server fires and 504s begin. The file-descriptor limit — auto-derived from ulimit at startup — is reached and accept() fails with Too many open files. A certificate expires and every TLS client is locked out at once. None of these are gradual.

These guides are written for engineers who already run HAProxy, not for people learning what a load balancer is. The goal is the mental model of how the proxy actually behaves under load, the failure patterns that keep recurring, the monitoring story that catches them before they page anyone, and the runbooks you wish someone had handed you before your last incident.

How HAProxy actually runs in production

HAProxy is not just a load balancer. It is an event loop that multiplexes tens of thousands of connections across a few threads, terminates TLS, evaluates ACLs, queues requests against four independent connection limits, and reuses pooled backend connections. Most production failures live between these layers, not inside any one of them.

clients + frontends

Connections land on a frontend listener socket and pass through the kernel accept queue (<code>net.core.somaxconn</code>). Each proxied connection costs two file descriptors and counts toward the frontend and global <code>maxconn</code>. Session rate is the raw traffic pulse; overflow here is silent — the kernel drops SYNs before HAProxy ever sees them.

TLS termination

If SSL is terminated, every new connection pays for a handshake — the single most CPU-intensive thing HAProxy does. A session cache lets clients resume cheaply; when it misses, CPU cost climbs with <code>SslFrontendKeyRate</code>. An expired certificate fails every client at once.

TLS

parsing + ACLs + routing

Request headers are read into a per-connection buffer (<code>tune.bufsize</code>, 16KB default); ACLs, regex, and stick-table lookups pick a backend. Headers larger than the buffer produce an HAProxy <code>400</code>. Topic-heavy ACL and regex chains burn event-loop CPU.

ROUTE

the queue

If the chosen server or backend is at its <code>maxconn</code>, the request waits in a per-server or per-backend queue (<code>qcur</code>). With <code>maxqueue</code> unset the queue is unbounded; once it fills or times out, HAProxy answers <code>503</code>. Queue time is the earliest saturation signal there is.

QUEUE

backend connection pool

HAProxy opens a new TCP connection or reuses an idle pooled one (<code>http-reuse</code>), consuming an ephemeral port per new connect. High reuse hides handshake and port cost; when reuse drops, <code>ctime</code> and ephemeral-port pressure spike together.

CONNECT

backend servers + health checks

A health-check engine probes each server independently of traffic (L4 TCP, L6 TLS, L7 HTTP) and drives UP/DOWN with <code>rise</code>/<code>fall</code>. A server can pass an L7 check while failing real requests. When enough go DOWN, traffic concentrates on survivors and the backend can collapse.

BACKEND

event loop + threads

An <code>epoll</code> loop per thread (<code>nbthread</code>) multiplexes everything cooperatively. <code>Idle_pct</code> is the CPU-headroom signal — until <code>busy-polling</code> pins it near zero, when the run queue in <code>show activity</code> becomes the truth. Average idle can look fine while one hot thread is saturated.

EVENT-LOOP

OS resources

File descriptors, ephemeral ports, kernel socket buffers, and memory pools are the hard floor. FD exhaustion is a cliff — <code>accept()</code> fails immediately. Reloads make it worse: the old draining worker holds FDs and memory alongside the new one.

KERNEL

Why this matters: 'HAProxy is slow' or 'we're getting 503s' can come from a saturated frontend at its maxconn, an all-DOWN backend, a slow dependency holding connection slots, an event loop drowning in TLS handshakes, file-descriptor exhaustion, kernel accept-queue drops the metrics never show, or a stale DNS resolution. The symptom rhymes but each layer has a different signal — and a different fix.

The failures you'll actually see

Most HAProxy incidents fall into a small set of recurring patterns. Recognise the shape, and triage gets dramatically faster.

CRITICAL

The 503 wall

Every request to a service starts returning 503 Service Unavailable. Either the backend has no UP server left, or connection admission is saturated and the queue overflowed. The tell is the frontend-versus-backend 5xx delta: a 503 is HAProxy-generated, so it shows on the frontend but not on backend rows. The process looks healthy — it is refusing work, not failing at it.

Frontend hrsp_5xx rising with little or no matching backend 5xx
Backend aggregate status DOWN, or active server count at zero
qcur climbing then connections rejected at maxqueue
scur pinned at slim on a frontend, backend, or server

Investigate →

CRITICAL

The backend collapse cascade

One server fails its health check with L4TOUT or L4CON and is marked DOWN. Its traffic redistributes to survivors, which slow down or hit their own maxconn, start failing checks, and drop out too. Fewer servers, more concentrated load, and the backend walks itself to zero capacity.

Multiple servers going DOWN in sequence, last_chk showing L4TOUT or L4CON
Active server count falling below half the pool
scur and rtime spiking on the remaining servers
econ and qcur rising as survivors saturate

Investigate →

CRITICAL

The backend timeout cascade

A backend dependency turns slow. Servers stay UP because TCP accepts still succeed, but requests hang: rtime climbs, connection slots fill, qcur rises, and once timeout server fires HAProxy emits 504 Gateway Timeout. Distinct from the 503 wall — here the servers are reachable, just not answering in time.

Frontend 504 rate rising while servers stay UP
rtime well above baseline across the backend
qcur and qtime climbing, econ staying low
cli_abrt rising as clients give up waiting

Investigate →

ACTIVE

Bad gateway from a broken backend

HAProxy answers 502 Bad Gateway because it could not connect to a server (econ) or the server sent a truncated or invalid response and closed mid-stream (eresp). A protocol-level failure, not a server-returned 500 — which is why econ versus eresp tells you whether the backend is unreachable or just broken.

Frontend 502s with econ rising on backend rows (cannot connect)
eresp and srv_abrt rising (server closed mid-response)
wretr / wredis climbing as HAProxy retries around it
Backend OOM or crash events lined up in time

Investigate →

CRITICAL

The certificate cliff

A frontend certificate expires and every TLS client is locked out at the same instant — one of the most common self-inflicted outages in the industry. Certificate expiry is not a live HAProxy metric, so nothing in the stats socket warns you; it needs an external check counting down the days before it happens.

All TLS clients failing at the handshake simultaneously
ereq rising on SSL frontends, handshake errors only in the logs
An ACME / Let's Encrypt renewal job that quietly stopped
Cert enddate inside the alert window (30d plan, 7d ticket, 24h page)

Investigate →

IMMINENT

The file-descriptor cliff

Each connection burns two file descriptors, plus listeners, stats, logs, health checks, and peers. At the ulimit ceiling accept() fails with Too many open files and new connections are dropped with no graceful degradation. Because HAProxy derives maxconn from the FD limit at startup, an undersized ulimit silently caps capacity long before you notice.

'Too many open files' in the HAProxy or system log
FD count above 80% of the process limit
New connections refused while CPU looks idle
Reload storm: old draining workers holding FDs alongside the new one

Investigate →

HAProxy monitoring maturity levels

HAProxy observability works in four practical levels. Each is a complete operation, not a stepping stone. Pick the level that matches how much your proxy matters. Most production deployments should land at the second level.

Level 1: Survival

Know that something is wrong

Survival monitoring is the floor. With these signals you can answer one question: is HAProxy alive and is traffic getting through? You will not learn what broke, but you will learn that something broke before users do. Survival is enough for dev proxies and non-critical paths.

Process / stats-socket liveness The stats socket answering proves the event loop runs, not just that a PID exists.
Backend aggregate status At least one server UP per backend, or every request to it is a 503.
Frontend accepting connections Sessions arriving means clients can actually connect right now.
HTTP 5xx rate on the frontend A wall of 5xx is the fastest 'something is broadly broken' signal.

Level 2: Operational

Diagnose most incidents on your own

Operational monitoring is what most production deployments should target. Survival tells you something is wrong; operational tells you what. With this coverage your team can usually diagnose an incident on its own: saturation, backend loss, error type, and where the errors originate.

Per-server UP / DOWN status Which servers left the rotation, and how many remain.
Current sessions vs maxconn (scur/slim) The primary saturation ratio — watch it as a percentage, never an absolute.
Backend aggregate status (UP/DOWN) Is any server available at all to serve this backend?
HTTP 5xx response rate Frontend 5xx is the user-facing error pulse.
Frontend vs backend 5xx delta Separates HAProxy-generated 503/504 from backend-returned 500s.
Bytes in / bytes out Bandwidth trend and anomalous transfers.
Session rate and request rate One session carries many requests under keep-alive — watch both.
File-descriptor headroom A cliff: at the ulimit, accept() fails immediately.

Level 3: Mature

Catch problems before they become incidents

Mature monitoring catches problems before they wake anyone up. Queue time creeping, retries quietly masking backend instability, connect and response time drifting apart, a backend losing servers one by one. None of these page you on day one. They become page-out incidents on day thirty.

Queue depth (qcur) and queue time (qtime) The earliest sign every server slot is full.
Connection errors (econ) vs response errors (eresp) Cannot connect versus connected-but-broken — different fixes.
Retries and redispatches (wretr/wredis) HAProxy masking backend instability; the canary most teams miss.
Connect time (ctime) and response time (rtime) Network/accept cost versus backend processing cost.
Active server count per backend Capacity headroom and cascade risk, not just UP/DOWN.
Health-check failure rate (chkfail) Rising chkfail with a still-UP server means it is flapping near threshold.
Idle_pct (event-loop CPU) HAProxy CPU headroom — meaningless under busy-polling, use show activity there.
Certificate expiry countdown Not a live metric; an external check is the only warning.

Level 4: Expert

Reactive instrumentation after real incidents

Expert signals enter your stack the day after a specific incident proved you needed them. Kernel accept-queue drops, per-thread skew, connection-reuse collapse, stick-table overflow, resolver staleness. Most teams never need every signal here. Add the ones your incident history says you do.

SSL key rate + session-cache effectiveness High key rate with a low cache hit ratio means resumption is broken.
Connection reuse (connect vs reuse) A reuse drop spikes ctime, ephemeral ports, and backend handshakes.
Kernel accept-queue overflows (ListenOverflows) Silent SYN drops the HAProxy metrics never show.
Per-thread show activity (thread skew) One hot thread caps throughput while average Idle_pct looks fine.
Stick-table utilisation (show table) A full table silently stops rate limiting and persistence.
Soft-stop / draining detection Old workers lingering past their drain window leak resources.
PoolFailed (allocator pressure) A direct signal HAProxy is dropping work it cannot allocate for.
DNS resolver and peer health Stale resolutions misroute traffic; broken peer sync diverges tables.

Operating mistakes worth avoiding

The traps HAProxy teams keep falling into. Each has a clear, well-known fix. Most teams only learn it after an incident.

Ignoring retries and redispatches

Most teams watch errors and latency but never look at <code>wretr</code> and <code>wredis</code>. These counters show HAProxy quietly masking backend instability — retrying and redispatching around failures. High retries with no user-visible errors means compensation is working; when it stops working the outage feels sudden, but the warning sat in the retry counters for hours or days.

Monitoring econ but not eresp (or vice versa)

<code>econ</code> means HAProxy could not connect — the server is down or unreachable. <code>eresp</code> means it connected but the response was invalid or truncated — the application crashed mid-response. Fundamentally different failure modes needing different responses, yet teams routinely monitor one and lump or ignore the other.

Using absolute thresholds instead of ratios

'Alert when scur > 1000' is meaningless without knowing <code>maxconn</code>; 'alert when scur > 80% of slim' works at any size. '5xx > 100/min' is noisy on high traffic and deaf on low. Express saturation and error thresholds as ratios so they scale with the instance and the workload.

Confusing session rate with request rate

In HTTP mode with keep-alive, one session (<code>rate</code>) carries many requests (<code>req_rate</code>), and HTTP/2 multiplexes further. Teams alert on a session-rate drop when request rate is fine (just better reuse), or miss a request-rate spike because session rate stayed flat. Watch both and the ratio between them.

Blind trust in health checks

A TCP check only proves the port is open; an L7 check to <code>/health</code> only proves that endpoint works. Neither proves real traffic works — an app can answer the check with 200 while returning 500s on every real request. Monitor per-server <code>hrsp_5xx</code> alongside health status, and make the check exercise the real dependency path.

Ignoring the maxconn hierarchy

HAProxy enforces <code>maxconn</code> at four independent levels: global, frontend, backend, and per-server. Teams typically watch only the global limit and miss a per-server maxconn bottlenecking a single backend while the global has plenty of headroom. An unset per-server limit is the opposite trap — unlimited concurrency onto a server that cannot take it.

Ignoring stick-table overflow

When a stick table fills, rate limiting and session persistence stop working <em>silently</em> — the default LRU-evicts expired entries, <code>nopurge</code> rejects new ones, and neither increments an error counter. Teams discover it only when a DDoS succeeds despite 'having rate limiting configured.' Monitor <code>used</code> versus <code>size</code> with show table.

Monitoring system CPU instead of Idle_pct

System CPU includes every process and kernel overhead. HAProxy's <code>Idle_pct</code> tells you specifically how much event-loop headroom remains — a box at 50% CPU with Idle_pct at 80% is fine, the same box with Idle_pct at 10% is saturated. The one caveat: with <code>busy-polling</code> enabled, Idle_pct sits near zero by design, so use show activity run-queue instead.

HAProxy runbooks in this section

Each guide is a focused runbook for one symptom or topic. Pick one when you have an incident, or use the categories to learn the area.

▸

Start here

▸

Connection saturation and maxconn

▸

Backend health and health checks

▸

Latency, queueing, and retries

▸

HTTP errors and error counters

▸

TLS/SSL and certificates

▸

CPU, the event loop, and threads

▸

Memory, buffers, and pools

▸

File descriptors, sockets, and reuse

▸

Reloads, draining, and lifecycle

▸

Stick tables, rate limiting, and abuse

▸

Service discovery and dynamic backends

WHERE TO GO NEXT

Setting up HAProxy monitoring, or putting out a fire?

If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, jump straight to the symptom that matches what you're seeing.

> Start with the checklist > Back to Operations Guides