HAProxy 503 Service Unavailable: no server is available to handle this request

Clients are getting 503 Service Unavailable with the message no server is available to handle this request. HAProxy generated this error. Unlike a 500 from your application or a 502 from a broken backend response, a 503 with this message never touched a backend server. HAProxy looked at its own state, concluded it had nowhere to send the request, and answered on its own.

That narrows the diagnostic question to: why did HAProxy believe no server was available? The honest answers are a short list. Every server in the backend is DOWN or in MAINT. The request waited in queue too long and expired. The queue overflowed. The frontend had no matching backend at all. Or connection admission was saturated and the request was rejected before dispatch.

This page walks through distinguishing those causes using the stats socket and log termination codes, and separating a HAProxy-generated 503 from backend 5xx errors and from the neighboring 502 and 504 codes.

What this means

HAProxy returns a 503 to the client in these situations:

• No UP server exists in the backend. All servers are DOWN, in MAINT, or DRAIN. The backend’s aggregate status flips to DOWN, and 100% of requests routed to it get a 503. In TCP mode the equivalent is a connection reset, not a 503.
• The request expired in the queue. The chosen server and backend were at their connection limits, the request waited in queue, and timeout queue fired before a slot opened. The log shows the sQ termination state. If timeout queue is unset, it defaults to the value of timeout connect.
• The queue overflowed. If maxqueue is set on a server and the queue reaches that limit, further requests are refused rather than queued. With the default maxqueue 0, the queue is unlimited, so overflow-driven 503s only happen when an operator has set a bound.
• No backend was selected. ACLs did not match any use_backend rule and no default_backend exists, or the named backend does not exist. The log shows <NOSRV> as the server name.
• Admission was saturated. Global or per-frontend maxconn was reached, so the connection was rejected before a server was ever considered.

The signature that ties all of these together is the frontend-vs-backend 5xx delta. A HAProxy-generated 503 increments hrsp_5xx on the frontend row but not on any backend or server row, because no server responded. If frontend 5xx is climbing while backend 5xx stays flat, HAProxy is the source.

flowchart TD
  A[Client gets 503] --> B{Frontend 5xx rising,
backend 5xx flat?}
  B -- No --> C[Errors come from backends.
Not a HAProxy 503 problem.]
  B -- Yes --> D{Backend aggregate
status UP?}
  D -- DOWN --> E[All servers DOWN or MAINT.
Check per-server status and last_chk.]
  D -- UP --> F{qcur nonzero or
sQ in logs?}
  F -- Yes --> G[Queue expiry or overflow.
Check scur/slim, maxqueue, timeout queue.]
  F -- No --> H{<NOSRV> in logs?}
  H -- Yes --> I[Routing gap: no ACL match,
missing default_backend.]
  H -- No --> J[Admission saturation:
scur at slim, maxconn reached.]

Common causes

Quick checks

All read-only, run against the runtime socket. Adjust the socket path if yours differs.

# 1. Prove the event loop is alive before trusting any stats
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | head -1

# 2. Backend aggregate status: DOWN means every request gets a 503
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 == "BACKEND" {print $1": status="$18" act="$20" bck="$21}'

# 3. Per-server status and last health check result
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" && $2 != "BACKEND" {print $1"/"$2": status="$18" last_chk="$37}'

# 4. Queue depth and session saturation per backend/server
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": qcur="$3" scur="$5" slim="$7}'

# 5. Frontend vs backend 5xx: is HAProxy generating the errors?
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '{print $1"/"$2": 5xx="$44}'

# 6. Global connection saturation
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | \
  grep -E "^(CurrConns|Maxconn|Uptime_sec|Stopping):"

# 7. Termination codes and server names from the log
# Look for sQ (queue expiry), SC/sC (connect refused/timeout), <NOSRV> (no backend)
grep " 503 " /var/log/haproxy.log | tail -20

How to diagnose it

1. Confirm the 503 is HAProxy-generated. Compare hrsp_5xx on the frontend row against the backend and server rows for the same window. If the frontend count climbs and backend rows do not, HAProxy produced the errors. If they climb together, the backends are returning the 5xx and this is an application problem, not an availability problem. These counters reset on reload; use Uptime_sec to sanity-check your deltas.

2. Check backend aggregate status. A BACKEND row with status = DOWN means zero available servers and a guaranteed 503 for every routed request. If DOWN, move to per-server rows: status plus last_chk tells you why each server was marked out. L4TOUT and L4CON mean HAProxy cannot reach the server at all. L7STS means the health check got the wrong HTTP status. Also check act: a backend can show UP with one surviving server while capacity is critically low.

3. If the backend is UP, look at the queue. Nonzero qcur with sQ termination codes in the log means requests waited in queue past timeout queue and were expired with a 503. The upstream cause is server saturation: every server at scur == slim, so nothing frees a slot in time. Check per-server scur/slim and rtime. Slow responses hold slots longer and feed the queue.

4. Check for routing gaps. If the log shows <NOSRV> as the server name for the 503 requests, HAProxy never selected a backend. Typical reasons: an ACL did not match, default_backend is missing, or a use_backend rule names a backend that does not exist. This often appears after a config change and affects only specific URL patterns or hosts.

5. Check admission limits. CurrConns approaching Maxconn in show info, or a frontend scur at its slim, means connections were rejected before routing. A useful tell: Idle_pct may look healthy because HAProxy is not overloaded with work, it is simply full. High scur with low req_rate points at idle or long-lived connections holding slots.

6. Separate from 502 and 504. The termination code in the log disambiguates: SC means the connection to the server was actively refused (likely a 502 or 503), sC means timeout connect fired (503 or 504), sQ means queue expiry (503), and SH/sH with a full timeout points at timeout server (504). If you are seeing 504s with servers UP and rtime climbing, that is the timeout cascade, not a 503 availability problem.

Metrics and signals to monitor

Fixes

All servers DOWN

Fix the health check failure reason, not HAProxy. L4CON/L4TOUT means a network, port, or process problem on the server side. L7STS means the health check endpoint is returning an unexpected status, so verify the endpoint against the application. If several servers are flapping and redistributing load onto survivors, put the worst offenders into MAINT to stop the flap loop while you fix the root cause:

# Disruptive: takes the server out of rotation. Use deliberately.
echo "set server <backend>/<server> state maint" | socat unix-connect:/var/run/haproxy.sock stdio

Queue expiry and overflow

The 503s are a symptom of backend saturation, so the durable fix is capacity: more servers, higher per-server maxconn if the application can absorb it, or faster backend responses. Shorter term, raising timeout queue lets requests wait longer instead of failing, at the cost of longer client-visible latency under saturation. Removing a tight maxqueue converts instant 503s into queued waiting, which is usually better for users but delays the failure signal. Do not just raise every limit without watching qcur and qtime afterward; you can convert fast, visible failures into slow, invisible ones.

Routing gaps

Fix the config: add the missing default_backend, correct the ACL, or point use_backend at a backend that exists. Validate with haproxy -c -f /etc/haproxy/haproxy.cfg before reloading. Config validation catches a nonexistent backend reference, but it cannot catch an ACL that simply never matches your traffic.

maxconn saturation

Determine whether connections are busy or idle. High scur with high req_rate is a genuine traffic surge: raise maxconn (remember the file descriptor budget, roughly maxconn x 2 plus overhead) or scale horizontally. High scur with low req_rate is idle connections holding slots: tighten timeout client and timeout http-request, and investigate long-lived connections with show sess. Remember maxconn is enforced at four levels (global, frontend, backend, per-server); a per-server limit can cause 503s while the global limit has plenty of headroom.

Prevention

• Alert on backend aggregate DOWN with traffic confirmation. Page when a backend is DOWN and the frontend has req_rate > 0 or a rising 5xx rate, sustained past a short grace period, with an Uptime_sec gate to ignore cold start.
• Alert on the frontend-minus-backend 5xx delta. It catches every HAProxy-generated error class (503, 504, 502) faster than per-backend investigation.
• Watch act per backend, not just aggregate status. A backend at one surviving server is one failure from a 503 storm. Page-worthy when survivors are also loaded (scur/slim high or qcur > 0).
• Track qcur and qtime as leading indicators. Queuing appears before sQ expiries. Any sustained nonzero queue in a system that normally runs at zero deserves a ticket.
• Watch wretr/wredis. Rising retries mean HAProxy is compensating for instability that will become visible 503s when compensation runs out.
• Set timeout queue and per-server maxconn deliberately. Defaults are permissive in different ways: unlimited queue time falls back to timeout connect, and servers have no connection limit unless you set one. Both decisions shape how saturation surfaces.

How Netdata helps

• Netdata collects the HAProxy stats CSV continuously, so hrsp_5xx per frontend, backend, and server becomes a time series. The frontend-minus-backend delta that identifies a HAProxy-generated 503 is a direct chart correlation instead of a manual diff during an incident.
• Per-server status, act, and chkdown trends show a backend losing servers over hours, which is the most common path to the all-DOWN 503 storm.
• qcur, qtime, and per-server scur/slim on the same dashboard let you see queue-driven 503s forming before timeout queue starts expiring requests.
• Counter resets on reload are handled automatically, so reload storms do not produce phantom 5xx drops or spikes in the middle of your diagnosis.
• Correlating econ, wretr, and rtime per server alongside 5xx rates separates “server unreachable” from “server slow” from “server failing fast” without log diving as the first step.

Related guides

• HAProxy 504 Gateway Timeout: timeout server and the backend timeout cascade
• HAProxy backend losing servers: active server count and cascade risk
• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy health checks green but the application is broken: when UP does not mean healthy
• HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable
• HAProxy health check L7STS: server DOWN on the wrong HTTP status
• How HAProxy actually works in production: a mental model for operators
• HAProxy maxconn hierarchy: global, frontend, backend, and per-server limits
• HAProxy maxconn reached: new connections queued and rejected
• HAProxy monitoring checklist: the signals every production proxy needs
• HAProxy monitoring maturity model: from survival to expert