Traefik cascading backend failure: how a partial outage becomes a total one

A backend pool behind Traefik rarely fails all at once. It fails one server at a time, and each failure makes the next one more likely. Two of your eight backends go down, the remaining six absorb the redistributed traffic plus the retry traffic Traefik generates, one of the six starts timing out under the extra load, and now five are carrying the full weight. Within minutes the pool is empty and Traefik returns 503 to every client, even though Traefik itself is perfectly healthy.

This is composite failure pattern 2.4 in the Traefik playbook: cascading backend failure. It is a gradual collapse, which distinguishes it from provider desync (a configuration problem where Traefik runs stale routes) and file descriptor exhaustion (a Traefik resource problem that fails at a cliff edge). Because it is gradual, it is detectable early. A declining traefik_service_server_up count and a rising traefik_service_retries_total precede the 503 spike by minutes. That gap is your response window.

What this means

Traefik’s load balancer tracks each backend server independently. Health check goroutines probe backends on a configurable interval; when a server fails its checks, it is removed from rotation and traefik_service_server_up{service, url} drops to 0 for that URL. Traffic that would have gone to the failed server is redistributed across the survivors.

That redistribution is the engine of the cascade. If N backends share the load and M fail, the remaining N-M servers carry all of the original traffic. If retry middleware is configured, they also carry the retried requests from every failure, so each failed request can arrive two or three times. A pool sized with modest headroom can absorb one failure. It often cannot absorb one failure plus 2-3x retry amplification on top of a 25% capacity reduction.

The survivors then start failing their own health checks, not because they are broken, but because they are saturated. Health checks time out or return errors under load, Traefik marks them down, the pool shrinks further, and the loop accelerates. The end state is traefik_service_server_up == 0 for every URL in the service and a 503 for every request.

flowchart TD
  A[First backends fail] --> B[Traffic redistributed to survivors]
  B --> C[Retry middleware multiplies load]
  C --> D[Survivors saturate and fail health checks]
  D --> E{Pool still has healthy servers?}
  E -- yes --> B
  E -- no --> F[503 to every client]

The key early-warning property: the retry spike and the declining server count come before the 503 spike. If you only alert on 503s, you find out when the cascade is finished. If you alert on the retry-to-request ratio and on any backend dropping out of the pool, you find out while there is still time to act.

Common causes

Cause	What it looks like	First thing to check
Bad deployment rolled out gradually	Backends fail one by one as new pods or instances come up; failures correlate with a deploy timestamp	Backend application logs and deploy history for the affected service
Shared dependency failure (database, cache)	Multiple backends degrade simultaneously; latency rises before health checks fail	Dependency health: DB connections, query latency, cache availability
Autoscaler scale-down removed too many instances	Backend count drops sharply with no application errors; survivors saturate	Autoscaler events and current replica count vs. traffic
Backend resource exhaustion (connection pools, memory, CPU)	Latency climbs steadily, then health checks start failing; backends flap between up and down	Backend CPU, memory, and connection pool metrics
Retry middleware amplifying a partial failure	`traefik_service_retries_total` climbing faster than the underlying error rate; latency rising with retries	Retry-to-request ratio per service

Quick checks

All of these are read-only.

# Which backends are up or down, per service
curl -s http://localhost:8080/metrics | grep traefik_service_server_up

# Retry pressure per service (rate this counter over 60s)
curl -s http://localhost:8080/metrics | grep traefik_service_retries_total

# 503s per service: is the pool already empty for anything?
curl -s http://localhost:8080/metrics | grep 'traefik_service_requests_total' | grep 'code="503"'

# 5xx breakdown per service
curl -s http://localhost:8080/metrics | grep 'traefik_service_requests_total' | grep -E 'code="5[0-9][0-9]"'

# Service latency: rising p95/p99 on survivors confirms saturation
curl -s http://localhost:8080/metrics | grep traefik_service_request_duration_seconds

# Backend connection states from the OS (CLOSE_WAIT growth hints at leaks)
# needs sudo if Traefik runs as a different user, otherwise -p shows no process info
sudo ss -tnp | grep traefik | awk '{print $1}' | sort | uniq -c

Two caveats. First, traefik_service_server_up only exists for services with Traefik health checks configured. If the series is absent for a service, that means unmonitored, not healthy; fall back to the 503 rate for that service. Second, if all backends report up but 5xx is still climbing, you are likely in the “healthy yet failing” edge case: the health check path (for example /healthz) is fine while real request paths are broken. Cross-reference server_up with the actual error rate before trusting it.

How to diagnose it

Identify the affected service. Use traefik_service_server_up{service, url} to find which service has backends at 0, and traefik_service_requests_total{code="503"} to find which services are already returning 503. In Kubernetes, service label names follow the <namespace>-<ingressroute-name>-<hash> pattern, so map the label back to the actual workload before proceeding.
Establish the timeline. Plot three series over the last 30-60 minutes: the count of up backends per service, the retry rate, and the 503 rate. A cascade shows a characteristic order: backend count declines first, retries rise second, 503s spike last. If all three moved simultaneously, suspect a shared dependency failure or a sudden traffic event rather than a gradual cascade.
Check the retry ratio. Compute rate(traefik_service_retries_total) / rate(traefik_service_requests_total) for the affected service. If retries are climbing faster than the original failure rate, retry middleware is amplifying the collapse. Retries per second approaching or exceeding original requests per second means you are deep in amplification territory.
Find the root cause in the backends. Check backend application logs, resource utilization (CPU, memory, connection pools), and downstream dependencies. Traefik is reporting the failure, not causing it. The most common root causes are a bad deployment, a shared database or cache failure, or an autoscaler that scaled down too aggressively.
Rule out the lookalikes. Confirm this is not provider desync: check that traefik_config_last_reload_success is recent and config reloads are still happening, which means Traefik’s routing table is current and the backends really are failing. Confirm it is not FD exhaustion: check process_open_fds / process_max_fds on the Traefik process. Both produce 5xx at the edge but have completely different fixes.
Check whether health checks are lying. If backends show up but errors are high, compare what the health check path exercises versus real traffic. A health endpoint that does not touch the database will pass while every real request fails.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`traefik_service_server_up` per URL	Direct view of pool shrinkage; the earliest cascade signal	Any backend at 0 for more than 2 health check intervals; count of up servers declining over time
`traefik_service_retries_total`	Measures amplification; rises before 503s	Retry rate > 5% of request rate sustained; retries rising together with latency
`traefik_service_requests_total{code="503"}`	Tells you the pool has fully collapsed for a service	Any sustained 503s on a production service
`traefik_service_requests_total{code=~"5.."}`	Distinguishes 502 (backend unreachable/garbage) from 503 (no healthy backends) from 504 (backend too slow)	5xx > 1% of service requests for > 5 minutes
`traefik_service_request_duration_seconds`	Rising latency on survivors confirms saturation before they fail	p95 exceeding 2x baseline for > 5 minutes
`go_goroutines` and `process_resident_memory_bytes`	Stuck backend connections accumulate goroutines during slow collapses	Growth disconnected from traffic patterns

Fixes

Interrupt the amplification first

If retry middleware is in the chain and the retry ratio is climbing, reduce attempts or remove the retry middleware from the affected router via dynamic configuration. This stops Traefik from multiplying load on the surviving backends while you fix the root cause. Left in place, Traefik will effectively DDoS a degrading backend while trying to help. Do this before anything that takes longer, like a rollback.

Tradeoff: clients will now see the transient errors that retries were masking. That is the correct trade during a cascade, because a visible partial error rate beats an invisible path to total failure.

If the timeline shows failures correlating with a deploy, roll back the affected service. This removes the root cause and lets the pool recover. Watch traefik_service_server_up after the rollback: the load balancer has no recovered-backend warm-up option, so a healthy backend re-enters normal weighted rotation immediately and a fragile recovery can re-trigger failure. Traefik’s separate circuit-breaker recovering state does progressively increase traffic, but only while that middleware is configured.

If the cause is saturation or an autoscaler that removed too many replicas, scale the backend pool back up or fix the resource constraint (connection pool limits, memory, CPU). Adding capacity is only safe once retries are under control; otherwise the new capacity is immediately consumed by amplified traffic.

Fix the shared dependency

If a database or cache failure is degrading all backends at once, no amount of Traefik-side tuning helps. Shed load at the edge if you can (rate limiting on non-critical routes) and fix the dependency.

Prevention

Alert on the early signals, not just the end state. Alert when any backend sits at traefik_service_server_up == 0 for more than two health check intervals, and when the retry-to-request ratio exceeds 5% sustained. Both fire minutes before the 503 spike.
Enable health checks on every service. Without them, traefik_service_server_up is absent and you lose the earliest cascade signal entirely. Use a health check path that exercises real dependencies, or accept that it lies.
Treat retry middleware as a risk multiplier. Keep attempts low. Some retry behavior is expected during rolling deployments, but a retry rate that climbs with latency is the cascade signature. Pair retry with Traefik’s built-in circuitBreaker middleware: retry absorbs transient blips, while the breaker stops load to a systemically failing service before retries amplify it. Use an explicit expression such as NetworkErrorRatio() > 0.30 (no default expression); defaults are 100ms checks, 10s fallback, 10s recovery, and response code 503.
Size the pool for N-2, not N-1. The cascade math says losing one backend is rarely the problem; losing one backend while retries double the effective load is.
Cap autoscaler scale-down speed. A scale-down that removes a third of the pool during a traffic peak manufactures a cascade without any application bug.
Do not trust /ping. It returns 200 as long as the process is alive, throughout the entire cascade. It checks nothing about backend health.

How Netdata helps

Netdata charts traefik_service_server_up per service and per backend URL at per-second resolution, so pool shrinkage is visible as it happens rather than at the next scrape aggregate.
Retry amplification shows up as traefik_service_retries_total climbing in the same view as per-service request rates, so the retry-to-request ratio is readable without writing a query during an incident.
The 5xx breakdown by response code (502 vs 503 vs 504) is preserved per service, so you can tell “backends failing” apart from “pool empty” and “backends too slow” without relabeling anything.
Service latency, retry rate, and server_up on one dashboard surface the cascade signature (retries and latency rising together while the up count declines) without switching tools.
Go runtime metrics (go_goroutines, heap, GC pauses) sit alongside the Traefik service metrics, which helps rule out Traefik-side resource problems like goroutine leaks during slow collapses.
ML-based anomaly detection on the retry and error-rate series flags the deviation from baseline before a static threshold would, which is where the response window comes from.