Traefik 503 Service Unavailable: no healthy backends left in the pool

Traefik is returning 503 Service Unavailable for every request to a service. Clients are down. Traefik itself looks fine: the process is up, /ping returns 200, other services route normally. This is the backend pool collapse failure mode: Traefik matched a router to a service, but every backend in that service’s load balancer pool has been marked down by health checks, so Traefik has nowhere to send the request.

A Traefik 503 is not a Traefik failure. It is Traefik correctly reporting that its upstream pool is empty. The investigation belongs one hop upstream, at the backends or at the health check configuration itself.

This article covers how to confirm the pool collapse, how to distinguish a genuine backend outage from the three common impostors (post-deploy warm-up, a service intentionally scaled to zero, and a misconfigured health check), and how to stop it from paging you for the wrong reason.

What this means

Traefik’s request pipeline ends at the service load balancer, which maintains a pool of backend servers. Health-checker goroutines probe each backend on a configurable interval (the Traefik default is 30 seconds, with a 5 second timeout; healthy means a 2XX or 3XX response, or whatever status you configured). When a backend fails its checks, Traefik removes it from rotation. When every backend for a service is out of rotation, Traefik short-circuits and returns 503 without proxying anything.

Three properties matter for diagnosis:

1. Traefik stays healthy throughout. /ping returns 200 because it only checks process liveness. It says nothing about backend health. See the health-check trap guide for why this bites operators.
2. The 503 appears at the service level, not the entrypoint level. You will see it in traefik_service_requests_total{code="503"} for the affected service, while entrypoint metrics keep counting the requests as arriving normally.
3. Do not confuse it with its neighbors. Per the Traefik FAQ: 404 means no router matched at all; 502 means Traefik contacted the backend and got an invalid response; 504 means the backend accepted the request but did not respond in time. 503 specifically means the pool is empty. Each of these has a different root cause and a different fix.

The defining metric is traefik_service_server_up{service, url}, a 1/0 gauge per backend URL. When it reads 0 for every URL in a service, the service is down. One critical caveat: this series only exists for services that have Traefik health checks enabled. If health checks are not configured, the series is absent, and absence means unmonitored, not healthy.

Common causes

Quick checks

All read-only. Run these before changing anything. The examples assume Traefik’s metrics and API are exposed on :8080; adjust to your metrics/API entrypoint.

# 1. Confirm the pool state: every backend URL at 0 for the service
curl -s http://localhost:8080/metrics | grep traefik_service_server_up

# 2. Confirm the 503s are service-level and which service
curl -s http://localhost:8080/metrics | grep 'traefik_service_requests_total' | grep 'code="503"'

# 3. Check retries in the lead-up (cascade pattern shows retries spiking first)
curl -s http://localhost:8080/metrics | grep traefik_service_retries_total

# 4. See what Traefik's API thinks the service looks like (backend URLs, status)
curl -s http://localhost:8080/api/http/services | head -100

# 5. Sanity check that Traefik itself is alive (expected: 200, tells you nothing about backends)
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8080/ping

Then probe a backend directly, the same way the health checker would:

# 6. Hit the configured health-check path on a backend, bypassing Traefik
curl -s -o /dev/null -w '%{http_code}\n' http://<backend-ip>:<port>/<healthcheck-path>

# 7. Hit the real application path on the same backend for comparison
curl -s -o /dev/null -w '%{http_code}\n' http://<backend-ip>:<port>/<real-path>

Checks 6 and 7 together separate “backend is dead” from “health check is lying.” If 6 returns 404 or 500 while 7 returns 200, the backends are fine and the health check is misconfigured.

How to diagnose it

Work through these in order. Each step eliminates a class of causes.

1. Confirm the pool collapse. From check 1: is traefik_service_server_up 0 for every URL in the service? If the series is entirely absent, the service has no health checks configured and the 503 is coming from somewhere else (for example, a service explicitly configured with no servers). Re-read the metric absence caveat before proceeding.

2. Establish the timeline. Did all backends go to 0 at the same instant, or one by one? Simultaneous failure points to a shared dependency (database, cache, DNS) or a config change. Gradual decline, especially with traefik_service_retries_total rising first, is the cascading failure pattern: fewer healthy backends take more load, fail, and concentrate load further.

3. Correlate with deployment activity. Was there a rollout, scale event, or config change in the minutes before? A 503 window of 10 to 30 seconds immediately after a deploy is the warm-up edge case: the backend passes readiness, Traefik adds it to the pool, but the application is still initializing connection pools and caches, and the health check flaps. This is expected behavior, not an incident.

4. Check for intentional scale-to-zero. If the service is supposed to be dormant (batch service, off-hours tool, preview environment), zero healthy backends may be the correct state. The alert condition needs a traffic floor, not just a health state.

5. Probe the backend directly (checks 6 and 7). Dead backend: fix the backend. Alive backend with a failing health-check path: fix the health check. This fork determines everything downstream.

6. If backends are alive but marked down, inspect the health check config. Common misconfigurations: wrong path (app has no /health, check returns 404), wrong port, expected status code not matching what the app returns, or a check path that depends on a downstream dependency that flaps independently of the app.

7. If backends are genuinely down, move upstream. Application logs, resource exhaustion, database connectivity. Traefik’s job is done at this point; it told you the truth.

flowchart TD
  A[503 on service] --> B{traefik_service_server_up
all URLs = 0?}
  B -->|series absent| C[No health checks configured
investigate service config]
  B -->|all zero| D{Backends alive
on direct probe?}
  D -->|no| E{Simultaneous
or gradual?}
  E -->|simultaneous| F[Shared dependency failure
DB / cache / DNS]
  E -->|gradual + retries rising| G[Cascading collapse
fix initial backend failure]
  D -->|yes| H{Health-check path
returns 2XX?}
  H -->|no| I[Misconfigured health check
path / port / expected status]
  H -->|yes| J[Check recent deploy or scale event
warm-up window or scale-to-zero]

Metrics and signals to monitor

On the first row: one backend down reduces capacity and resilience. All backends down is the outage. Alert differently on the two conditions, and add a traffic floor to the all-down alert so a legitimately idle or scaled-to-zero service does not page anyone.

The v3 issue that renders the service label as an opaque hash instead of service@provider remains open as traefik/traefik#10734. If you see hash labels, map them via the /api/http/services output rather than assuming the issue is fixed in your minor.

Fixes

Genuine backend outage

Fix the backend, not Traefik. Roll back the bad deploy, restore the shared dependency, or replace the crashed instances. Once backends pass health checks again, Traefik returns them to rotation immediately. There is no warm-up or ramp-up mechanism: a recovered backend gets its full traffic share instantly, which can re-kill a fragile recovery. If that happens, investigate the backend’s ability to handle cold load rather than blaming the proxy.

Post-deploy warm-up window

Do not “fix” this with alert tuning alone; fix the deploy. Give the backend health check an interval and threshold that matches the application’s real startup time, and use startup probes in Kubernetes so Traefik’s provider does not see the pod as ready before the app actually is. Brief 5xx spikes right after a rollout are expected; do not alert on them.

Scaled to zero

If the service is legitimately dormant, the 503 is correct behavior. Suppress the all-backends-down alert for that service, or gate it on observed request traffic so it only fires when someone is actually being served errors.

Misconfigured health check

Correct the path, port, scheme, or expected status so the check reflects the application. Longer term, make the health check meaningful: Traefik’s fallback behavior without an explicit check only detects connection-level failure, and a check that hits a trivial /healthz while real traffic needs a database will lie to you in the other direction (all backends “up”, real requests 5xx). Cross-referencing traefik_service_server_up against the actual 5xx rate catches both directions of lie.

Cascading collapse

Break the feedback loop first: reduce or disable the retry middleware on the affected service so surviving backends stop absorbing amplified traffic. Then address whatever degraded the first backend. Retries masking errors on the dashboard while tripling backend load is how a partial failure becomes a total one.

Prevention

• Configure explicit health checks on every production service. Without them, traefik_service_server_up does not exist for that service and you are blind to pool state until clients report errors.
• Make health checks representative. The check path should exercise what real traffic needs, or at least fail when the app cannot serve. Otherwise you trade 503s for silent 5xx.
• Align health-check timing with deploy behavior. Check interval and thresholds should tolerate real startup time so rollouts do not flap the pool.
• Alert on partial pool loss, not just total loss. One backend down in a two-backend pool is 50% capacity gone and zero redundancy. That is a ticket, not a page, but it must not be invisible.
• Watch retries as a leading indicator. Retry rate relative to request rate is the earliest signal of a cascade in progress.
• Put a traffic floor on the all-down alert. Scale-to-zero and dormant services are healthy at zero backends. Page only when zero backends coincides with real request traffic.

How Netdata helps

• Netdata charts traefik_service_server_up per backend URL, so you see the pool draining in real time and can tell gradual cascade from simultaneous failure at a glance.
• Per-service response code breakdown puts the 503 rate next to retries and latency on one dashboard, which is exactly the correlation that separates a cascade from a config mistake.
• Comparing service-level 5xx against entry-level metrics shows the errors are service-scoped, confirming Traefik itself is healthy and the problem is upstream.
• Netdata’s process and Go runtime collectors cover the Traefik side of the house (FDs, goroutines, memory), so you can rule out proxy-level resource issues quickly and focus on the backends.
• Anomaly detection on per-service request rates helps distinguish a real outage from a legitimately idle service, which is the difference between a page and a non-event.

Related guides

• How Traefik actually works in production: a mental model for operators
• Traefik monitoring checklist: the signals every production edge router needs
• Traefik monitoring maturity model: from survival to expert
• Traefik /ping returns 200 while everything is broken: the health-check trap