HAProxy health checks green but the application is broken: when UP does not mean healthy

Every server in the backend shows UP. Health checks are passing. And users are getting 500s. This is one of the most common HAProxy postmortem findings: “health checks said everything was fine.”

The failure is conceptual, not a bug. HAProxy’s health check engine runs independently of traffic and only proves that the specific probe you configured succeeds. A TCP check proves the port accepts connections. An HTTP check to /health proves that one URL returns the expected status. Neither proves that real requests, which exercise the database, auth, and downstream dependencies, actually work.

This guide covers how to confirm the pattern in minutes, how to identify which servers are lying, and how to make the health check exercise the real dependency path so UP starts meaning healthy again.

What this means

HAProxy decides where to route traffic based on health state: UP, DOWN, MAINT, DRAIN. That state comes from the health check engine, which periodically probes each server with whatever check you configured. The engine has no visibility into what happens to production requests after they leave HAProxy.

So the following situations all produce a green dashboard while the application is broken:

• TCP check only. The check proves the port is open. An application whose worker pool is deadlocked, or whose host went read-only after a disk failure, still accepts TCP connections. HAProxy sees UP; every real request hangs or errors.
• Stub HTTP check. The check hits /health, which returns 200 without touching the database, cache, auth service, or downstream APIs. Real requests under /api/* fail while /health stays green.
• Partial application failure. Some endpoints work, others do not. If the health endpoint is in the working set, the server stays UP while a meaningful fraction of traffic 500s.
• Agent check self-report. With agent-check, the application reports its own status. The application can lie, or simply not know it is broken.

The result: HAProxy keeps routing traffic to servers that return 5xx on real requests, and nothing in the status column tells you.

flowchart LR
  HC[Health check engine] -->|TCP connect or GET /health| SRV[Backend server]
  SRV -->|200 on stub endpoint| HC
  HC -->|status: UP| RT[Routing table]
  CL[Client request] --> RT
  RT -->|real request| APP[App code path]
  APP -->|DB / auth / downstream| DEP[Dependencies]
  DEP -.->|broken| APP
  APP -->|500| CL
  HC -.->|never exercises| DEP

The check path and the request path are disjoint. The fix is to overlap them.

Common causes

Quick checks

All commands are read-only against the runtime socket. Adjust the socket path if yours differs.

# 1. Confirm all servers really are UP
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$1 !~ /^#/ && $2 != "FRONTEND" && $2 != "BACKEND" {print $1"/"$2": status="$18}'

# 2. Per-server 5xx counters (cumulative; sample twice to get a rate)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$1 !~ /^#/ {print $1"/"$2": 5xx="$44}'

# 3. Frontend vs backend 5xx, to see who generates the errors
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$1 !~ /^#/ && ($2 == "FRONTEND" || $2 == "BACKEND") {print $1"/"$2": 5xx="$44}'

# 4. Response time per server (fast errors look healthy)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$1 !~ /^#/ && $2 != "FRONTEND" && $2 != "BACKEND" {print $1"/"$2": rtime="$61"ms"}'

# 5. What the health check engine last saw
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$1 !~ /^#/ && $2 != "FRONTEND" && $2 != "BACKEND" {print $1"/"$2": status="$18" chkfail="$22" check_status="$37}'

# 6. Retries and redispatches: is HAProxy silently compensating?
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$1 !~ /^#/ && $2 != "FRONTEND" {print $1"/"$2": wretr="$16" wredis="$17}'

Three things to note when reading this output. Field numbers in the CSV can shift across HAProxy versions; the first line of show stat is a header naming every column, so confirm the indices there before trusting them. hrsp_5xx is cumulative and resets on reload, so take two samples a known interval apart and compute a delta; Uptime_sec in show info tells you if a reload just zeroed everything. And rtime is a rolling average over the last 1024 requests, so a server that just started failing may not show it yet.

How to diagnose it

1. Confirm the signature. All server rows show UP while per-server hrsp_5xx is climbing. If servers are flapping or DOWN instead, this is a different problem.
2. Establish who generates the errors. Compare frontend hrsp_5xx against the backend row’s hrsp_5xx for the same window. If frontend 5xx is approximately equal to backend 5xx, the errors come from the application and HAProxy is just passing them through. If frontend is significantly higher, HAProxy itself is generating 503s (no available server) or 504s (timeouts), which points to a capacity or connectivity problem instead.
3. Identify the affected servers. Per-server hrsp_5xx rates tell you whether all servers fail equally (shared dependency, bad deploy) or one server is the outlier (host-level issue).
4. Reproduce outside HAProxy. Curl the health endpoint and a real application endpoint directly on a backend, bypassing the proxy. If /health returns 200 and the real route returns 500, you have confirmed the check does not exercise the real code path.
5. Read what the check actually tests. Look at the backend configuration: a bare check on the server line is TCP-only. option httpchk or http-check send defines the HTTP probe. Ask what that endpoint verifies. If the answer is “returns 200 unconditionally,” you have found the root cause.
6. Check wretr and wredis. Elevated retries with still-rising errors mean HAProxy is compensating for backend instability and the compensation is failing. Elevated retries with no visible errors mean this pattern was developing silently before users noticed.
7. Engage the application team with evidence. Per-server 5xx rates, the health-endpoint-vs-real-endpoint comparison, and the check_status values are usually enough to hand off a crisp bug report.

Metrics and signals to monitor

Fixes

Make the check exercise the real dependency path

The durable fix is a health endpoint that verifies what real requests need: database connectivity, cache, auth, critical downstream calls. Point the check at it.

On HAProxy 2.2 and later, prefer the explicit http-check send form. The legacy bare option httpchk sends an OPTIONS / request without a Host header, and the inline form sends a minimal request that many modern backends reject:

backend app_servers
  option httpchk
  http-check send meth GET uri /healthz ver HTTP/1.1 hdr Host app.internal
  http-check expect status 200
  server app1 10.0.0.11:8080 check
  server app2 10.0.0.12:8080 check

If your health endpoint can return 5xx when dependencies fail, make HAProxy treat that as DOWN rather than passing it:

  http-check expect ! rstatus ^5

Tradeoffs to think through:

• Deep checks can flap. If the check depends on the database and the database has a 2-second blip, every server goes DOWN at once and you convert a degradation into a total outage. Keep the deep check focused on the one or two dependencies that define “can serve traffic,” not every dependency the app has.
• Check cost. A deep check runs every inter interval against every server. Keep it cheap; it should touch the dependency, not run a real transaction.
• The endpoint must not be cacheable or static. If a CDN layer, framework shortcut, or web server in front of the app can answer the check without the app being involved, you are back to a stub.

Add passive checking on live traffic

Active checks probe a synthetic path. Passive checking watches real responses and marks a server down when live traffic errors:

  server app1 10.0.0.11:8080 check observe layer7 error-limit 10 on-error mark-down

This catches application-level failure that no synthetic probe can anticipate. Two caveats. Only certain response codes count as errors under observe layer7 (5xx codes except 501 and 505, by design, so clients cannot trivially DoS the mechanism). And an active check hitting a still-green stub endpoint will revive a server that passive checking just marked down, creating a mark-down/revive cycle. If you run both, make the active check exercise the same code path, or raise rise so revival is slower.

Interim mitigation during an incident

If servers are UP but broken right now and you cannot fix the check immediately:

• Put the worst offenders in MAINT via the runtime API (disable server / set server ... state maint) to stop the bleeding while you investigate. This is disruptive to in-flight traffic on that server; prefer drain if the servers are partially functional.
• Do not restart HAProxy. It is not the broken component, and a restart resets the counters you need for diagnosis.

Prevention

• Alert on the pattern, not just the status. “All servers UP AND backend hrsp_5xx rate elevated AND frontend-minus-backend 5xx delta near zero” is a pageable composite. Status alone will never catch this.
• Review health endpoints like code. During design review, ask what the health endpoint verifies. If the answer is “returns 200,” treat it as a finding.
• Watch wretr/wredis as a leading indicator. Retries climbing for hours with no visible errors is this pattern developing in slow motion.
• Correlate rtime with error rate. A server getting faster is only good news if errors are flat.
• Rehearse the question in postmortems. Any incident where “health checks were green” appears should produce a concrete change to what the check verifies.

How Netdata helps

Netdata collects the HAProxy stats CSV continuously, which is exactly what this pattern requires, because the signature only appears when you correlate fields that no single alert watches:

• Per-server hrsp_5xx alongside server status, so an UP server returning 500s shows up immediately instead of hiding behind a green status page.
• Frontend and backend 5xx on the same dashboard, making the “errors come from the app, not from HAProxy” determination a visual comparison instead of a manual socket session.
• Per-server rtime next to error rates, exposing the fast-failure pattern where latency improves while the backend breaks.
• wretr and wredis trends, surfacing the silent compensation phase before users see errors.
• chkfail, chkdown, and lastchg per server, so you can distinguish a check that is blind from a check that is flapping.
• Counter-reset handling across reloads, so the delta-based 5xx rates this diagnosis depends on stay accurate through config reloads.

Related guides

• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy scur approaching slim: the concurrent-session saturation signal
• How HAProxy actually works in production: a mental model for operators
• HAProxy maxconn hierarchy: global, frontend, backend, and per-server limits
• HAProxy maxconn reached: new connections queued and rejected
• HAProxy monitoring checklist: the signals every production proxy needs
• HAProxy monitoring maturity model: from survival to expert
• HAProxy Slowloris and idle-connection pile-up: high scur, low request rate