HAProxy Operations Guides on Netdata

HAProxy 400 and 408 spikes: bad requests, request timeouts, and Slowloris

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy 400 and 408 spikes: bad requests, request timeouts, and Slowloris

Your frontend hrsp_4xx counter just doubled, and it is not 404s. The spike is 400s and 408s, which means HAProxy itself is rejecting or timing out requests before they reach a backend. Unlike backend-generated 4xx, these errors say something about the bytes arriving at your frontend: malformed requests, protocol mismatches, oversized headers, or clients that open connections and never finish the request.

HAProxy 502 Bad Gateway: backend connection and response failures

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy 502 Bad Gateway: backend connection and response failures

Clients are getting 502 Bad Gateway from HAProxy. The backend may look healthy, health checks may be green, and yet some fraction of requests comes back with a 502 the application never logged. That is the signature: a 502 from HAProxy is usually generated by HAProxy itself, not passed through from the backend.

HAProxy emits a 502 in two situations. First, it could not establish or use a connection to the backend server at all. Second, it connected fine, but the server sent an invalid, truncated, or incomplete response, or closed the connection mid-response. These are different failure modes with different fixes, and the fastest path to resolution is separating them before touching anything.

HAProxy 503 Service Unavailable: no server is available to handle this request

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy 503 Service Unavailable: no server is available to handle this request

Clients are getting 503 Service Unavailable with the message no server is available to handle this request. HAProxy generated this error. Unlike a 500 from your application or a 502 from a broken backend response, a 503 with this message never touched a backend server. HAProxy looked at its own state, concluded it had nowhere to send the request, and answered on its own.

HAProxy 504 Gateway Timeout: timeout server and the backend timeout cascade

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy 504 Gateway Timeout: timeout server and the backend timeout cascade

Your frontend error rate jumps. Clients report “504 Gateway Timeout.” HAProxy looks alive: the process is running, servers are UP, health checks are green. Yet requests are dying on a timer.

This is the backend timeout cascade, and the 504 is its final symptom. The servers are reachable (TCP accepts succeed), but the application behind them has slowed past the timeout server budget. HAProxy holds the client connection, waits, gives up, and generates a 504 itself.

HAProxy 5xx delta: telling HAProxy-generated errors from backend errors

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy 5xx delta: telling HAProxy-generated errors from backend errors

Your frontend hrsp_5xx counter is climbing and users are seeing errors. The first question that decides the next hour of your incident response: is HAProxy generating these errors, or is it passing through errors your backend servers produced?

HAProxy counts 5xx responses in both places. Frontend rows count every 5xx the client saw, including responses HAProxy fabricated itself. Backend and server rows count only 5xx responses that actually came back from a server. The difference between the two answers “is the backend failing or is HAProxy failing?” without walking every backend one by one.

HAProxy backend losing servers: active server count and cascade risk

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy backend losing servers: active server count and cascade risk

A backend can lose half its servers and HAProxy will still report the BACKEND aggregate row as UP. As long as one server is available, the rollup status says traffic is being served, and status-based dashboards stay green. Meanwhile the surviving servers are absorbing redistributed load, their session counts are climbing, and you are one health check failure away from a full 503 outage for that backend.

HAProxy backend queue building (qcur): requests waiting for a free server slot

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy backend queue building (qcur): requests waiting for a free server slot

You are looking at HAProxy stats and qcur is nonzero on a backend, or worse, climbing. Clients have not seen errors yet, but latency is drifting up. That instinct that something is wrong is correct: qcur is the earliest signal that backend capacity is exhausted, and it shows up before any error counter moves.

qcur is the instantaneous count of connections sitting in a queue because every server that could take them is already at its maxconn limit. Those requests are not being served. They are parked, consuming their client’s timeout budget, waiting for a slot to free up. If the queue grows faster than slots free up, the outcome is queue timeouts and 503s.

HAProxy bandwidth (bin/bout): bytes in, bytes out, and capacity planning

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy bandwidth (bin/bout): bytes in, bytes out, and capacity planning

Every row in HAProxy’s stats output carries two cumulative byte counters: bin (bytes in) and bout (bytes out). They are the ground truth for how much data actually moves through the proxy, per frontend, per backend, and per server. Most teams only look at them when a network link saturates. Used properly, they are a baseline signal, an anomaly detector, and the raw material for bandwidth capacity planning.

HAProxy compression overhead: comp_byp, CPU cost, and when compression backfires

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy compression overhead: comp_byp, CPU cost, and when compression backfires

HTTP compression in HAProxy is a trade: CPU cycles on the proxy in exchange for fewer bytes on the wire. When it works, text responses shrink by 50 to 80 percent and clients on slow links see faster page loads. When it backfires, the proxy burns event-loop CPU compressing content that does not compress, or it sheds compression work under load and you lose the bandwidth savings exactly when traffic is highest.

HAProxy connect time (ctime) high: network latency and backend accept-queue overflow

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy connect time (ctime) high: network latency and backend accept-queue overflow

Your HAProxy backend shows ctime climbing from a steady 1-2ms to tens or hundreds of milliseconds. Total request latency (ttime) rises with it, and if the trend continues, connect timeouts (econ increments, 502/504s on the frontend) are next. The connect phase of the HAProxy-to-backend path is degrading, and it almost always means one of three things: the network path is congested or losing packets, the backend’s kernel accept queue is overflowing, or the backend is too CPU-starved to answer the handshake promptly.

HAProxy connection errors (econ): backend connections refused, timed out, or reset

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy connection errors (econ): backend connections refused, timed out, or reset

Your HAProxy backend shows a rising econ counter. Clients may not see errors yet, because HAProxy retries failed connections and redispatches to other servers, but the counter means HAProxy is trying to open TCP connections to a backend server and failing. The connection is being refused, timing out, or getting reset.

econ is one of the highest-signal error counters HAProxy exposes, and one of the most misread. It is cumulative, it only increments on new connection attempts (so connection reuse can mask a real problem for hours), and it lumps three very different failure modes into one number. Telling “the server process is dead” apart from “a firewall is silently dropping SYNs” requires correlating econ with ctime and a few other signals.

HAProxy connection reuse dropping: http-reuse, pooling, and handshake overhead

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy connection reuse dropping: http-reuse, pooling, and handshake overhead

Backend latency crept up, CPU on the HAProxy host is climbing, and ctime went from zero to measurable milliseconds, but the backends are healthy and the network looks fine. Before you blame the application or the network, check the one ratio almost nobody graphs: connection reuse, reuse / (connect + reuse) on the backend. If it fell off a cliff, HAProxy stopped reusing pooled backend connections and is now paying a full TCP (and possibly TLS) handshake for every request.

HAProxy counter resets on reload: phantom drops and spikes in your metrics

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy counter resets on reload: phantom drops and spikes in your metrics

Your HAProxy 5xx dashboard just showed a cliff-edge drop to zero, then a vertical spike an hour later. No user complaints. No backend incidents. No deploys. If this pattern repeats on a schedule, or right after config changes, you are almost certainly looking at counter resets from HAProxy reloads, not real traffic events.

Every time HAProxy reloads its configuration, a new worker process starts. That new process has its own in-memory statistics, and every cumulative counter starts at zero: hrsp_5xx, econ, eresp, bin, bout, wretr, wredis, chkfail, the SSL counters, all of them. A reload is a new process for stats purposes, not a continuation of the old one.

HAProxy DNS resolver failure: stale IPs and silently misrouted traffic

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy DNS resolver failure: stale IPs and silently misrouted traffic

Your backend servers report UP. Health checks pass. The stats page is green. And traffic is going to IP addresses that belong to pods deleted an hour ago, or to a node that now runs something else entirely.

This is the HAProxy DNS resolver failure mode, and it only bites deployments that resolve backends dynamically: server-template with Consul DNS, Kubernetes headless services, SRV records, or any setup where server addresses come from DNS instead of static config lines. When the resolver times out or a TTL expires without a successful re-resolution, HAProxy keeps routing to the last IPs it knew about. There is no stats CSV counter for this. show resolvers is the only direct view into it, and most teams never look.

HAProxy DroppedLogs: losing log lines exactly when you need them

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy DroppedLogs: losing log lines exactly when you need them

You are mid-incident. Traffic is spiking, 5xx rates are climbing, and you go to the HAProxy logs for per-request timing and termination codes. The logs are not there. Whole seconds of traffic are missing, right at the peak. Then you notice DroppedLogs in show info has been incrementing for hours and nobody was watching it.

DroppedLogs counts log messages HAProxy tried to emit but could not deliver. The cause is almost never HAProxy itself: it is the log pipeline downstream. A syslog daemon that cannot keep up, a socket buffer that overflowed, a log target that stalled. There is no error in the HAProxy log, because the line that would have told you is one of the ones that got dropped.

HAProxy ephemeral port exhaustion: TIME_WAIT and backend connection churn

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy ephemeral port exhaustion: TIME_WAIT and backend connection churn

Your backends are healthy. Every server is UP, health checks pass, the application team sees nothing wrong on their side. Yet HAProxy is logging failed backend connections, econ counters are climbing, and clients see intermittent 502s or stalls. On the HAProxy host, tens of thousands of sockets sit in TIME_WAIT.

This is ephemeral port exhaustion on the backend-facing side of HAProxy. Every new TCP connection from HAProxy to a backend consumes one local ephemeral port. When the connection closes, that port is held in TIME_WAIT for 60 seconds before it can be reused. If the rate of new backend connections is high enough, the port range empties and connect() starts failing with EADDRNOTAVAIL. HAProxy reports these as connection errors even though nothing is wrong with the backend.

HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable

You look at the HAProxy stats page (or your monitoring) and a server row is red. The last_chk field says L4TOUT or L4CON, status says DOWN, and HAProxy has stopped routing traffic to that server. Both codes mean the failure happened before any HTTP, before any application code, at the plain TCP layer.

L4TOUT means HAProxy opened a socket and the TCP connect did not complete within the check timeout. L4CON means the connect attempt failed immediately with an error, typically “Connection refused” (TCP RST) or “No route to host” (ICMP). The server is unreachable at the TCP layer, and after fall consecutive failures (default 3), HAProxy marks it DOWN and redistributes its traffic.

HAProxy health check L7STS: server DOWN on the wrong HTTP status

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy health check L7STS: server DOWN on the wrong HTTP status

A backend server flips to DOWN in the stats page, last_chk shows L7STS, and the log line says something like Layer7 wrong status, code: 503, info: "Service Unavailable". The server process is running. The port is open. You can curl it by hand and get a response. But HAProxy has pulled it out of rotation and traffic is concentrating on the survivors.

HAProxy health checks green but the application is broken: when UP does not mean healthy

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy health checks green but the application is broken: when UP does not mean healthy

Every server in the backend shows UP. Health checks are passing. And users are getting 500s. This is one of the most common HAProxy postmortem findings: “health checks said everything was fine.”

The failure is conceptual, not a bug. HAProxy’s health check engine runs independently of traffic and only proves that the specific probe you configured succeeds. A TCP check proves the port accepts connections. An HTTP check to /health proves that one URL returns the expected status. Neither proves that real requests, which exercise the database, auth, and downstream dependencies, actually work.

HAProxy hot-thread skew: when average Idle_pct hides a saturated thread

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy hot-thread skew: when average Idle_pct hides a saturated thread

Latency is climbing, throughput has plateaued below what the hardware should deliver, and show info reports Idle_pct at a comfortable 60 or 70 percent. System CPU shows one core pegged while the others coast. The bottleneck is not HAProxy as a whole. It is one thread.

With nbthread > 1 (the default since HAProxy 2.0, where nbthread is set to the number of available CPUs), work is spread across per-thread event loops. Idle_pct is a process-level average across all of them. One thread pinned at 100% while the rest sit at 80% idle averages out to “looks fine,” but that hot thread caps overall throughput: every connection it owns gets processed late, and no other thread can take them over.

HAProxy Idle_pct dropping: event-loop CPU saturation and the busy-polling trap

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy Idle_pct dropping: event-loop CPU saturation and the busy-polling trap

Your HAProxy latency is climbing, nothing is queuing on the backends, and the one metric that looks wrong is Idle_pct sliding toward zero. Or worse: Idle_pct has been pinned at zero for weeks and you only just noticed because someone finally asked what it meant. Both situations are common, and they have very different explanations.

Idle_pct is HAProxy’s self-reported measure of event-loop headroom: the share of time the loop spends waiting in poll() versus processing events. It measures HAProxy CPU headroom, not system CPU. A host at 50% system CPU with Idle_pct at 80% is fine. A host at 50% system CPU with Idle_pct at 10% means HAProxy itself is saturated. That distinction is the entire point of the metric, and it is also where the traps live.

HAProxy in TCP mode: the HTTP signals you lose and what to watch instead

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy in TCP mode: the HTTP signals you lose and what to watch instead

When a frontend and its backend run with mode tcp, HAProxy stops parsing the protocol inside the connection. It forwards bytes and nothing more. That is often correct: databases, gRPC, Redis, TLS passthrough, or protocols HAProxy does not understand. But it amputates a large part of the monitoring surface most HAProxy alerting assumes exists.

The failure mode this creates is not an outage; it is a blind spot. Dashboards that worked in HTTP mode start reporting zeros that look like “no traffic” or “no errors.” Health checks stay green while the application behind them is broken. Latency alerts keyed on rtime never fire again because rtime is permanently zero. The proxy is fine; your visibility into what flows through it is not.

HAProxy kernel accept-queue overflow: silent SYN drops and somaxconn

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy kernel accept-queue overflow: silent SYN drops and somaxconn

Clients report connection timeouts. You open the HAProxy stats page and everything looks fine: session rate is normal, no 5xx spike, servers are UP, Idle_pct is healthy. The network team sees nothing. The clients insist the load balancer is dropping them.

Both sides are right. The kernel is silently dropping SYN packets in the accept queue before they ever reach HAProxy’s event loop. From HAProxy’s perspective, those connections never existed. There is no log line, no error counter, no stat field that increments. The only evidence lives in two kernel counters, ListenOverflows and ListenDrops, that almost nobody graphs.

HAProxy maxconn hierarchy: global, frontend, backend, and per-server limits

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy maxconn hierarchy: global, frontend, backend, and per-server limits

HAProxy does not have one connection limit. It has four, enforced independently at the global, frontend, backend, and per-server levels. Any one of them can throttle traffic while the other three show headroom. This is why “global maxconn is at 30%” tells you almost nothing during an incident: the request path can be blocked at a per-server limit you never set, or queuing behind a backend constraint you did not know existed.

HAProxy maxconn reached: new connections queued and rejected

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy maxconn reached: new connections queued and rejected

HAProxy is refusing work. Clients see connection timeouts or resets, load tests fail at a suspiciously round concurrency number, and yet the HAProxy process looks healthy: CPU is fine, memory is fine, the process is up, health checks pass. The stats tell the real story: CurrConns is flat at Maxconn, or scur is pinned at slim on a frontend, and it does not move.

HAProxy memory growth and OOM: connection buffers, pools, and memory ceilings

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy memory growth and OOM: connection buffers, pools, and memory ceilings

The symptom usually arrives as a dead process, not a slow one. HAProxy was proxying fine, then the PID changed, connections dropped for a few seconds, and dmesg shows the OOM killer picked haproxy as the victim. Or, before it gets that far, RSS climbs day over day and nobody knows whether it is a leak or just load.

HAProxy monitoring checklist: the signals every production proxy needs

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy monitoring checklist: the signals every production proxy needs

Most HAProxy outages are visible in its own stats long before users notice. The problem is that the stats socket exposes dozens of fields, and teams usually wire up the five metrics their dashboard template shipped with and stop there. Then a reload storm, a retry storm, or a silent stick-table overflow teaches them what they were missing.

This checklist organizes the signals that catch production failures into four maturity levels: survival, operational, mature, and expert. Each level builds on the previous one. You do not need to reach expert on day one, but you should know which level you are at, because that determines which failure modes are invisible to you.

HAProxy monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy monitoring maturity model: from survival to expert

Most HAProxy monitoring setups fall into one of two states: a process check and a prayer, or a wall of charts nobody reads. Neither answers the question that matters during an incident: is HAProxy the problem, the messenger, or the victim.

This article defines four maturity levels for HAProxy monitoring: Survival, Operational, Mature, and Expert. Each level adds signals that catch a class of failures the previous level cannot see. The goal is not to collect everything. The goal is to know, at any moment, which questions your monitoring can answer and which it cannot.

HAProxy peers sync broken: divergent stick tables across instances

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy peers sync broken: divergent stick tables across instances

You have two or more HAProxy instances fronting the same service, and their behavior no longer matches. Rate limiting triggers on one node but lets the same client through on another. A failover happens and every sticky session evaporates, even though you configured stick tables and a peers section specifically to survive that event. Or you compare show table output across instances and the entry counts are wildly different when they should be near-identical.

HAProxy per-source-IP rate limiting: stick tables, conn_rate, and http_req_rate

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy per-source-IP rate limiting: stick tables, conn_rate, and http_req_rate

Per-source-IP rate limiting in HAProxy is built on stick tables: in-memory key-value stores that track counters per client key (usually the source IP) and expose those counters to ACLs. When it breaks, it breaks silently. A full table, a mismatched expire, or an IPv6 client population can disable your limiting without a single error in the logs, and most teams find out only after an abuse incident.

HAProxy PoolFailed: buffer starvation and memory-allocator failures

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy PoolFailed: buffer starvation and memory-allocator failures

PoolFailed in show info counts failed internal memory-pool allocations since process start. It is normally zero. Any nonzero value means HAProxy tried to allocate a buffer, connection, or session object and could not get one. Work was dropped or stalled because of it.

The tricky part is how this surfaces. Buffer starvation rarely looks like a memory problem from the outside. It looks like mysterious latency: rtime and backend health are fine, CPU has headroom, the network is clean, and yet clients see stalls and sporadic failures. PoolFailed is one of the few direct signals that HAProxy itself is refusing work because it ran out of an internal resource.

HAProxy queue time (qtime) rising: the earliest backend-saturation signal

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy queue time (qtime) rising: the earliest backend-saturation signal

You are looking at a backend whose qtime has crept from zero to a few hundred milliseconds, or a few seconds, and nothing is “down” yet. Health checks are green, error rates are flat, and no alert has fired. That is the point of qtime: it moves before the failure signals.

Nonzero qtime means requests spent time in HAProxy’s queue because every usable server connection slot was busy when they arrived. It is usually the first measurable symptom of backend saturation: ahead of 503s, ahead of timeout server expiries, ahead of user complaints.

HAProxy reload storm: lingering processes leaking file descriptors and memory

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy reload storm: lingering processes leaking file descriptors and memory

You log into the HAProxy host because memory and file descriptor usage have been climbing for days, and you find a dozen haproxy processes running. Only one is serving traffic. The rest are old workers stuck in soft-stop, each holding open connections, connection buffers, and file descriptors it will never release. Traffic looks fine, per-process metrics look fine, and the host is still slowly running out of resources.

HAProxy response errors (eresp): truncated and invalid backend responses

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy response errors (eresp): truncated and invalid backend responses

The eresp counter is climbing on a backend, and clients are seeing broken pages, truncated downloads, or intermittent 502s. The backend servers report healthy. Health checks are green. The 5xx rate, if it moved at all, does not explain the volume of user complaints.

This is the failure mode eresp exists to capture: HAProxy connected to the backend successfully, sent the request, and then received a response it could not treat as valid. Either the bytes were not parseable HTTP, or the server closed the connection before the response finished. That is a protocol-level failure, not an application returning an error status code, and it needs a different investigation path than a 5xx spike.

HAProxy response time (rtime) climbing: slow backends behind the proxy

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy response time (rtime) climbing: slow backends behind the proxy

Your HAProxy dashboards show rtime creeping up on one or more backends. Clients have not started timing out yet, or maybe they just have. HAProxy itself looks fine: CPU is reasonable, no servers are DOWN, error rates are flat or only slightly elevated. But the average time between HAProxy forwarding a request and the backend’s first response byte is growing, and it is not coming back down.

HAProxy retries and redispatches (wretr/wredis): the backend instability nobody watches

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy retries and redispatches (wretr/wredis): the backend instability nobody watches

Your dashboards are clean. No 5xx spike, latency averages look normal, every backend server is UP. Then the backend falls over in what feels like seconds, and the postmortem shows the application was flapping for six hours before the outage. The warning was there the whole time, sitting in two counters almost nobody charts: wretr and wredis.

These counters record HAProxy’s built-in compensation mechanism at work. When a connection to a backend server fails, HAProxy does not immediately return an error to the client. It retries. If option redispatch is set, it can also give up on the original server and dispatch the request to a different one. From the client’s perspective the request succeeds, possibly a bit slower. From the proxy’s perspective, something in your backend just failed and got papered over.

HAProxy run queue rising: task backlog and scheduler pressure

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy run queue rising: task backlog and scheduler pressure

HAProxy’s event loop processes every connection, request, health check, and timer as tasks scheduled across its threads. When tasks arrive faster than threads can drain them, they pile up in the run queue. That backlog is the earliest internal signal that HAProxy is CPU-saturated: it appears before latency spikes, before backend queuing, and before client timeouts.

It is also the signal to use when Idle_pct is meaningless. With busy-polling enabled, the event loop never sleeps, so Idle_pct sits near zero by design. The Run_queue field from show info and the per-thread counters from show activity are the replacement.

HAProxy scur approaching slim: the concurrent-session saturation signal

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy scur approaching slim: the concurrent-session saturation signal

Your alert says scur is at 85% of slim on a frontend, or CurrConns is closing on Maxconn in show info. HAProxy is still up. Traffic is still flowing. But you are running out of concurrent-session headroom, and what happens next depends entirely on which of the four independent limits you are about to hit.

The danger is that the system still appears responsive: HAProxy is alive and healthy, but clients are about to see queuing delays or 503 rejections. The scur/slim ratio is the warning, if you read it correctly.

HAProxy server flapping: rise, fall, and health checks oscillating UP and DOWN

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy server flapping: rise, fall, and health checks oscillating UP and DOWN

A backend server that alternates between UP and DOWN is worse than one that is cleanly down. Every transition redistributes traffic: when the server goes DOWN its connections move to the survivors, when it comes back UP it absorbs a fresh share of load before it has warmed up, and every cycle costs retries, redispatches, and connection churn. If the flap rate is high enough, the oscillation itself becomes the incident.

HAProxy server-template and dynamic backends: monitoring service-discovery routing

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy server-template and dynamic backends: monitoring service-discovery routing

With a static HAProxy config, the server list lives in the config file. Capacity review is config review: count the server lines and you know what the backend can absorb. With server-template, the server list lives in DNS. HAProxy pre-provisions a pool of empty server slots, and its internal resolver fills and empties them as A or SRV records change. This is the mechanism behind HAProxy fronting Kubernetes Ingress, Consul, and most autoscaled service-discovery setups.

HAProxy show pools: finding which internal pool is consuming memory

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy show pools: finding which internal pool is consuming memory

HAProxy’s memory footprint is not one number. The process holds dozens of internal object pools: buffers, connections, sessions, tasks, stick-table entries, SSL state. When RSS climbs, when PoolFailed goes nonzero, or when you are sizing a memory budget for a new instance, the question is never “how much memory is HAProxy using” but “which pool is using it”. show pools is the runtime API command that answers that.

HAProxy Slowloris and idle-connection pile-up: high scur, low request rate

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy Slowloris and idle-connection pile-up: high scur, low request rate

Your HAProxy frontend shows scur pinned at or near slim, but req_rate is a fraction of what those connections should be producing. Throughput (bin/bout) is near zero, backend queue depth (qcur) is zero because backends are doing nothing, and ereq with 408 responses is climbing. HAProxy is not down. It is not slow. It is full: thousands of connections are holding session slots open and sending data so slowly they may as well be dead.

HAProxy SSL certificate expired: total TLS failure and how to catch it first

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy SSL certificate expired: total TLS failure and how to catch it first

Every client connecting to the affected frontend gets a TLS handshake failure. Not a percentage of clients, not a slow degradation: all of them, immediately, from the moment the certificate’s notAfter timestamp passes. Browsers show NET::ERR_CERT_DATE_INVALID, API clients throw certificate validation errors, and HAProxy itself is perfectly healthy, running, and passing every process-liveness check you have.

The mechanism is trivial, the blast radius is total, and HAProxy gives you no built-in metric, counter, or log warning that a certificate is about to expire. If you are reading this during an incident, skip to Quick checks and Fixes. If you are reading it afterward, the Prevention section is the part that matters.

HAProxy SSL session cache misses: full handshakes and lost resumption

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy SSL session cache misses: full handshakes and lost resumption

Your HAProxy CPU is climbing but connection counts look normal. Idle_pct is dropping, latency on new connections is rising, and existing sessions seem fine. When you pull show info, SslFrontendKeyRate is far above baseline and the SSL session cache numbers tell the story: almost every lookup is a miss.

A high cache miss rate means most TLS clients are paying for a full handshake instead of resuming a previous session. Full handshakes are the most CPU-intensive operation HAProxy performs; resumed handshakes skip most of that cost. When resumption breaks, the load balancer loses a large fraction of its capacity without any change in traffic volume.

HAProxy SslFrontendKeyRate: the handshake rate that drives CPU cost

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy SslFrontendKeyRate: the handshake rate that drives CPU cost

SslFrontendKeyRate is the number of new (non-resumed) TLS handshakes per second across your HAProxy frontends. Because full TLS handshakes are the most CPU-intensive operation HAProxy performs, this metric is effectively the CPU cost of your TLS traffic expressed as a rate.

Operators usually look for this metric in two situations: HAProxy CPU is climbing and they need to know whether TLS is the cause, or they are doing capacity planning and need a defensible number for “how many handshakes per second can this box absorb.” Both depend on understanding what the metric counts, where it comes from, and what breaks it.

HAProxy stick-table overflow: rate limiting that silently stops working

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy stick-table overflow: rate limiting that silently stops working

Your HAProxy config has rate limiting. It has session persistence. Both are enforced by stick tables, and both can stop working without an error counter, a log line, or a page.

Stick tables are fixed-size, in-memory key-value stores. When a table reaches its configured size, HAProxy has two possible behaviors and neither one alerts you. With the default configuration, HAProxy flushes expired entries to make room for new ones. If the table is full of entries that have not yet expired, new entries are refused. With the nopurge option, eviction is disabled entirely and new entries are silently rejected. In both paths, whatever the table was doing (rate limiting, sticky sessions, abuse tracking) degrades or stops for affected clients.

HAProxy stuck in soft-stop: draining connections and hard-stop-after

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy stuck in soft-stop: draining connections and hard-stop-after

You reloaded HAProxy minutes ago, but pgrep haproxy still shows two, three, maybe a dozen PIDs. The old process is not serving new traffic, but it refuses to die. Memory and file descriptor usage on the host keep climbing with every reload, and your monitoring loses counter history each time it happens.

The old process is in soft-stop: it has unbound from its listeners and is waiting for its existing connections to close before exiting. A drain of a few seconds to a couple of minutes is normal. A process draining for more than 10 minutes is stuck, and the cause is almost always long-lived connections (WebSocket, Server-Sent Events, gRPC streaming, or plain TCP keepalive sessions) that have no reason to close on their own.

HAProxy TLS handshake failures: broken clients after a cert or cipher change

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy TLS handshake failures: broken clients after a cert or cipher change

A certificate rotation or TLS policy change goes out, HAProxy reloads cleanly, health checks pass, traffic graphs look normal, and then the tickets start: a subset of clients can no longer connect. The load balancer itself is fine. The handshake rate even looks normal. What changed is the failure rate within handshakes, and HAProxy has no dedicated stats counter that shows it to you directly.

HAProxy TLS handshake storm: new-connection floods that saturate the event loop

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy TLS handshake storm: new-connection floods that saturate the event loop

Traffic looks normal at the TCP layer. Connections are arriving, the frontend is accepting them, but latency is climbing, some clients time out before ever sending a request, and backend servers look healthy. When you check show info, Idle_pct is near zero and SslFrontendKeyRate is far above anything you have tested. You are in a TLS handshake storm: HAProxy is burning every CPU cycle on cryptographic handshake work and the event loop has nothing left for the rest of the pipeline.

HAProxy Too many open files: file descriptor exhaustion and refused connections

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy Too many open files: file descriptor exhaustion and refused connections

HAProxy is refusing new connections. Clients see connection timeouts or resets, health checks start failing for no application reason, and the logs show Too many open files. The process is alive and CPU looks fine, which makes this confusing the first time you hit it: nothing is overloaded in the usual sense. The proxy has run out of file descriptors.

HAProxy tune.bufsize and 400 errors: buffers too small for large headers

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy tune.bufsize and 400 errors: buffers too small for large headers

Your frontend 400 rate just spiked. The backends insist they never saw the requests, and they are telling the truth: HAProxy generated these 400s itself, during header parsing, before any backend was selected. The usual trigger is a request header block that no longer fits in the per-connection buffer, and the usual suspects are large cookies, JWT bearer tokens, or a newly added header that pushed total header size past the limit.

How HAProxy actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How HAProxy actually works in production: a mental model for operators

Most HAProxy incidents come from an operator applying the wrong mental model: treating HAProxy like a web server, like nginx, or like a black box that either works or does not. HAProxy is an event-driven state machine that multiplexes hundreds of thousands of connections across a small number of threads, and almost every confusing symptom traces back to one of a handful of internal mechanisms: the accept pipeline, the maxconn hierarchy, the buffer and connection pools, the health check engine, or a subsystem you forgot was running.