HAProxy monitoring maturity model: from survival to expert

Most HAProxy monitoring setups fall into one of two states: a process check and a prayer, or a wall of charts nobody reads. Neither answers the question that matters during an incident: is HAProxy the problem, the messenger, or the victim.

This article defines four maturity levels for HAProxy monitoring: Survival, Operational, Mature, and Expert. Each level adds signals that catch a class of failures the previous level cannot see. The goal is not to collect everything. The goal is to know, at any moment, which questions your monitoring can answer and which it cannot.

Use this model two ways. First, as an audit: find the level you are at and look at what the next level buys you. Second, as a design checklist: when you build or rebuild HAProxy monitoring, work the levels in order. Do not jump to Expert signals before the Operational ones are solid, because the Expert signals are mostly meaningless without the context the lower levels provide.

flowchart TD
  L1["Level 1 - Survival
process alive, backend UP, frontend accepting"]
  L2["Level 2 - Operational
per-server status, scur/slim, 5xx, bytes"]
  L3["Level 3 - Mature
queue, econ/eresp, retries, latency, FD headroom"]
  L4["Level 4 - Expert
Idle_pct, TLS cost, reuse, stick tables, kernel drops"]
  L1 --> L2 --> L3 --> L4

All commands below assume a stats socket at /var/run/haproxy.sock exposed via the runtime API. Adjust the path to your deployment. Every command shown is read-only.

Level 1: Survival

Survival answers one question: is the building on fire? Three checks, all cheap, all suitable for paging.

Process and event loop liveness. A PID check is necessary but not sufficient. The process can exist while the event loop is stuck, or while it is an old worker draining after a reload. The stats socket check is strictly better because it proves the event loop is processing:

# Proves the event loop is responsive, not just that a PID exists
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | head -1

In master-worker mode, the master process stays alive even when workers are restarting. Query the worker’s stats socket, and gate the alert on Uptime_sec so restarts and reloads do not page you.

At least one server UP per backend. A backend whose aggregate status is DOWN returns 503 (or a TCP reset in TCP mode) for 100% of requests routed to it. Pull the BACKEND rows from the stats CSV:

echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 == "BACKEND" {print $1": "$18}'

A backend can report UP with only one of many servers available, so this check catches total outage, not capacity loss. Capacity loss is a Level 2 and Level 3 concern.

Frontend accepting connections. The frontend session rate (rate on FRONTEND rows, or SessRate in show info) should be nonzero during traffic hours. rate is an instantaneous last-second snapshot, not a counter, so do not compute deltas on it. Distinguishing “zero traffic because it is 3 AM” from “zero traffic because something broke” requires a baseline, which is why this stays a coarse check at Survival level.

With these three checks you will know within a minute or two of a total outage. You will not know about degradation, partial failures, or anything building toward an outage. That is fine for a first week. It is not fine for long.

Level 2: Operational

Operational adds the signals you need to detect user-impacting problems before the ticket queue does.

Per-server health status. The status field per server row (UP, DOWN, NOLB, MAINT, DRAIN, plus transitional states like UP 1/3) is HAProxy’s routing table. A single server DOWN is usually a ticket, not a page, because HAProxy redistributes automatically. What matters is the trend: how many are down, and whether the survivors are absorbing the load. Do not alert on MAINT or DRAIN since those are operator-intentional, but do track how long they persist.

Current sessions vs limits (scur/slim). This is the primary saturation signal, enforced independently at four levels: global, frontend, backend, and per-server. Check all of them, not just the global CurrConns/Maxconn from show info, because a per-server limit can bottleneck while the global has plenty of headroom:

echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '{if($7>0) print $1"/"$2": scur="$5" slim="$7" pct="int($5/$7*100)"%"}'

Alert on ratios (above 80% sustained is a reasonable warning), never absolute numbers. An absolute threshold is meaningless across instances of different sizes.

HTTP 5xx rate. Track hrsp_5xx on frontends and backends as a percentage of req_rate. It is a cumulative counter that resets on reload, so compute delta-based rates. At this level you are watching the total; the frontend-vs-backend split that tells you whether HAProxy or the application generated the errors is a Level 4 refinement.

Bytes in/out. bin and bout give you bandwidth for capacity planning and anomaly detection. The bout/bin ratio for a typical web app is above 1; a sudden inversion is worth a look. These are cumulative counters, so derive rates from deltas and handle reload resets.

Backend aggregate status. The rollup from Level 1 stays here, now with traffic context: a DOWN backend with zero traffic is different from a DOWN backend actively returning 503s.

Level 2 catches outages and gross saturation. It does not catch the slow-burn failures: backends degrading without errors, retries masking instability, queues forming before timeouts. Those are the failures that “felt sudden” in the postmortem but were visible for hours.

Level 3: Mature

Mature is where you start catching problems before they become incidents. The common thread: these signals expose compensation and backpressure, the places where HAProxy is working harder to hide a problem from users.

Queue depth and queue time (qcur/qtime). Nonzero queuing means every server connection slot is full and requests are waiting. qcur is the instantaneous depth; qtime is a rolling average over the last 1024 requests, so brief bursts are smoothed out of it. In a well-sized system both are zero in steady state. Any sustained nonzero value is your earliest warning of backend saturation, arriving before errors and before timeouts. With the default maxqueue of 0 the queue grows unbounded; with a positive maxqueue, overflow becomes 503s.

econ vs eresp. Teams routinely monitor one and not the other, but they are different failure modes. econ means HAProxy could not connect: server down, SYN queue full, network partition, ephemeral port exhaustion. eresp means it connected but the response was broken: crash mid-response, protocol mismatch, timeout server too low. Correlating econ with ctime distinguishes timeout from refusal: high ctime before errors means timeout; near-zero ctime with errors means immediate refusal.

Retries and redispatches (wretr/wredis). This is the canary most teams miss. High retries with zero user-visible errors means backend instability is currently being compensated. When the compensation runs out, the outage “feels sudden” even though the retry counters showed it for hours. Note that wredis requires option redispatch; without it HAProxy only retries the same server.

Session rate vs request rate (rate/req_rate). The ratio req_rate / rate measures connection reuse efficiency. Request rate dropping while session rate holds means keep-alive or multiplexing broke. High session rate with low request rate is the Slowloris signature. Both are instantaneous gauges, not counters.

Connect and response time (ctime/rtime). ctime is network health to the backends (same-datacenter should be low single-digit milliseconds). rtime is backend application health. Both are rolling averages over 1024 samples, and both have a trap: with http-reuse, ctime sits near zero, so a sudden appearance of nonzero ctime means reuse broke down. And a backend returning instant 500s shows low rtime, so always correlate rtime with error rates. There are no percentiles in the stats; tail latency requires parsing per-request timing fields from logs.

Active server count per backend. The act/bck fields on BACKEND rows count UP servers in active and backup roles. Below 50% active, one more failure risks a cascade, but correlate with scur on the survivors: an overprovisioned pool at half capacity with idle survivors is less urgent than a tight pool losing one server.

Health check failure rate (chkfail). chkfail rising while status stays UP means the server is flapping just under the fall threshold. Also watch check_duration approaching timeout check, which inflates detection delay.

File descriptor headroom. FD exhaustion is a hard cliff: accept() fails immediately. Each proxied connection costs roughly two FDs, plus listeners, stats, log, and health check sockets. Compare the /proc/<pid>/fd count against the limit in /proc/<pid>/limits, and remember that during reloads, old and new processes both hold FDs against the system limit.

Level 4: Expert

Expert signals are the ones experienced operators add after learning the hard way. Most are cheap to collect once you know they exist.

Idle_pct with busy-polling awareness. Idle_pct from show info is HAProxy’s own event loop saturation signal, far more specific than system CPU. Below 20% sustained, the event loop is under stress; below 5%, it is saturated. The critical caveat: with busy-polling enabled, the loop never sleeps and Idle_pct sits near zero by design, making the metric meaningless. In that mode, use show activity run-queue depth or system CPU instead. It is also an average across threads, so one hot thread can hide inside a healthy-looking number.

SSL key rate and session cache. TLS handshakes are the dominant CPU cost. SslFrontendKeyRate (from show info, not the CSV) tells you the cryptographic load. Pair it with cache effectiveness computed from SslCacheLookups and SslCacheMisses (there is no hits field; hits = lookups minus misses). A high miss rate means every connection pays full handshake cost. Expect a brief miss spike after every reload since the in-process session cache is invalidated.

Connection pool utilization. The connect vs reuse ratio on backend rows measures how well http-reuse is working. A sudden drop in reuse is a leading indicator of configuration drift: a backend changed HTTP version, disabled keepalive, or started sending Connection: close. The fallout shows up as ctime and SslBackendKeyRate increases, but the reuse ratio is where you see it first.

Stick-table utilization. show table exposes size and used per table. When a table fills, behavior depends on configuration: by default HAProxy evicts expired entries to make room; with nopurge, new entries are rejected. Either way, once entries are lost or refused, rate limiting and session persistence silently stop working for affected clients, with no error counter anywhere. Teams discover this when an attack succeeds despite “having rate limiting configured.”

Soft-stop detection. Stopping: 1 in show info marks a draining process. Brief is normal during reload; lingering beyond minutes means long-lived connections (WebSockets, streaming) are holding it alive, usually because hard-stop-after is not set. Count HAProxy PIDs alongside this: more than two or three suggests a reload storm, which also resets all cumulative counters on each cycle.

PoolFailed. Any nonzero PoolFailed in show info means the internal allocator failed to allocate buffers or connection objects, which means HAProxy dropped work. Normally zero. show pools breaks down which pool is under pressure.

Kernel accept queue overflows. ListenOverflows and ListenDrops in /proc/net/netstat (or nstat -az | grep -i listen) count SYNs dropped before they ever reached HAProxy. Clients time out; HAProxy sees nothing. This is the invisible failure that sends teams debugging the network when the real problem is net.core.somaxconn sized too small for connection bursts. These are system-wide counters, not HAProxy-specific. The ConnRate vs SessRate gap in show info is a secondary hint of pre-accept losses.

Per-thread skew via show activity. With nbthread > 1, show activity exposes per-thread event counters. Significant skew means one thread is the bottleneck while averages look fine. Counters are 32-bit and can wrap on long-running, high-traffic instances.

Resolver and peer health. show resolvers reveals DNS timeouts and errors. In server-template or service-discovery deployments, resolver failures silently route traffic to stale IPs while everything looks healthy. show peers shows stick-table replication state; broken peer sync means rate limits and session affinity diverge between nodes. Neither appears in the CSV stats.

DroppedLogs. DroppedLogs in show info counts log lines that could not be delivered. UDP syslog drops are silent at the sender, so this counter is the only signal. Logs tend to drop exactly when you need them most: during traffic spikes.

Frontend minus backend 5xx delta. Frontend hrsp_5xx includes HAProxy-generated errors (503 no server, 504 timeout); backend rows carry only what servers returned. The delta isolates HAProxy’s own failures from application failures and points the investigation in the right direction immediately: rising delta means look at HAProxy capacity and connectivity, not the app.

Moving up a level

Two traps show up repeatedly when teams work this model.

The first is skipping levels. Expert signals like Idle_pct and reuse ratio are diagnostic multipliers, not standalone alerts. An Idle_pct drop only tells you something is consuming the event loop; you need Level 3 signals (key rate, session rate, latency context) to know what.

The second is alert hygiene at every level. Express thresholds as ratios, not absolutes. Handle counter resets on reload using Uptime_sec as a reset detector. Gate pages on persistence and traffic floors so cold starts, reloads, and idle periods do not page you for non-events.

How Netdata helps

  • Netdata’s HAProxy collector pulls the stats CSV and show info continuously, so Level 1 through Level 3 signals (server status, scur/slim, hrsp_5xx, qcur/qtime, econ/eresp, wretr/wredis, ctime/rtime) are per-second time series rather than manual socket queries during an incident.
  • Cumulative counters like hrsp_5xx, econ, and bin are automatically converted to rates, and reload-induced resets are handled, which removes the most common source of phantom spikes and drops.
  • Correlating the frontend-minus-backend 5xx delta, retry counters, and per-server rtime on one dashboard is what turns “errors are up” into “HAProxy is masking a flapping backend” in one look.
  • Saturation signals (scur/slim ratios, FD usage, Idle_pct, PoolFailed) are collected alongside system-level data like ListenOverflows and per-core CPU, so the HAProxy-internal view and the kernel view line up on the same timeline.
  • ML-based anomaly detection on rate and latency signals helps separate a real deviation from normal time-of-day baseline drift, which is the part of Level 1 and 2 alerting that is hardest to hand-tune.