$ guides / traefik / traefik-retry-amplification ▌

Operations Guides

Traefik retry amplification: how retries turn a slow backend into an outage

A backend service starts degrading. Responses get slow, some connections fail. Traefik’s retry middleware re-sends the failed requests, usually to other backends in the pool. Backend load doubles or triples. The struggling service falls further behind, which produces more failures, which produces more retries. Within minutes, a partial degradation becomes a complete outage, and Traefik is doing a significant share of the damage while trying to help.

The worst part: this loop can be invisible on the dashboards your team actually watches. If a request fails twice but succeeds on the third attempt, the client got a 200. Your success-rate graphs look fine while the backend absorbs 3x the real client traffic and slides toward collapse.

This guide covers how the loop forms, how to confirm you are in one, and how to break it safely.

What this means

The retry middleware exists to mask transient backend failures. A connection reset, a brief network blip, a backend restarting mid-request: retrying against another server in the load-balancer pool turns a client-visible error into a success. That is useful behavior when failures are rare and uncorrelated.

It becomes dangerous when failures are correlated with load. The retry budget is per-request: every failed attempt can be re-sent, and each re-send consumes backend capacity. When the backend’s problem is that it is overloaded (database lock contention, connection pool exhaustion, a slow downstream dependency), adding more requests makes the root cause worse. The retry mechanism assumes failures are random; under saturation they are causal, and every retry feeds the cause.

Two aggravating factors make this worse in practice:

Health checks may still pass. Health check paths often differ from real traffic paths. A backend can answer /health in 5ms while /api/v1/data times out. Traefik keeps the backend in rotation and keeps retrying into it. See Traefik health checks pass but requests fail.
The default retry triggers are transport failures, with optional status-code ranges. By default, status is empty and disableRetryOnNetworkError is false, so an unserved response can be retried; an ordinary HTTP 500 is not retried unless you add it to status. A degrading backend still produces timeouts, resets, and hung connections, so the default trigger overlaps the overload failure mode.

flowchart TD
  A[Backend degrades: slow responses, transport errors] --> B[Retry middleware re-sends failed requests]
  B --> C[Backend load doubles or triples]
  C --> D[More timeouts and connection failures]
  D --> B
  C --> E[Fewer effective backends: pool capacity drops]
  E --> F[Survivors get original traffic plus retry traffic]
  F --> D
  D --> G[All backends saturated: 502/503/504 to clients]

Common causes

The loop always needs two things: a backend that is degrading, and retries that multiply the load. The initial degradation is the root cause; the retry config determines how fast it escalates.

Cause	What it looks like	First thing to check
Backend resource saturation (DB locks, connection pool exhaustion)	Latency climbs steadily, then transport errors appear; retries and latency rise together	Backend CPU, memory, DB connection counts, slow query log
Downstream dependency failure cascading upstream	Multiple backends slow simultaneously because they share a database or cache	The shared dependency’s health, not the backends themselves
Retry attempts configured too high for the workload	`traefik_service_retries_total` rate approaching or exceeding request rate	The retry middleware config: attempts count and which routers use it
Traffic spike against insufficient headroom	Sudden request-rate increase preceding the latency climb	`traefik_service_requests_total` rate vs. baseline
Rolling deployment in progress	Retries elevated but latency normal; self-resolving	Whether a deployment is actually running (do not confuse this with amplification)
Non-idempotent requests being retried	Duplicate records, double charges, repeated side effects reported downstream	Whether `retryNonIdempotentMethod` is enabled and which methods the retried routes accept

The distinguishing test: if retries are high but latency is low, you are looking at instance flapping (common during rolling updates, usually not dangerous). If retries and latency are both rising together, you are in amplification territory.

Quick checks

All read-only. Run against Traefik’s metrics endpoint (adjust host and port to your deployment).

# Retry counters per service
curl -s http://localhost:8080/metrics | grep traefik_service_retries_total

# Request counters per service, with status codes
curl -s http://localhost:8080/metrics | grep traefik_service_requests_total

# Backend health status (only present for services with health checks enabled)
curl -s http://localhost:8080/metrics | grep traefik_service_server_up

# Service latency histograms
curl -s http://localhost:8080/metrics | grep traefik_service_request_duration_seconds

The critical computation is the retry-to-request ratio. Take two samples of traefik_service_retries_total and traefik_service_requests_total 60 seconds apart, compute the per-second rates for the affected service, and divide:

Retries/requests under 1%: occasional blips. Not an incident by itself.
Retries/requests over 5% sustained: backends are unhealthy and retries are amplifying load. Investigate now.
Retries/requests approaching or exceeding 1.0: every request is being retried at least once. You are deep in the loop and Traefik is sending roughly double the client traffic to the backends. Treat as urgent.

Also check whether traefik_service_server_up shows all backends as 1 while 5xx rates and retries climb. That combination means the health checks are passing but real traffic is failing, which is the classic setup for this incident.

How to diagnose it

Identify the affected service. Find which service label on traefik_service_retries_total is spiking. Do not start with aggregate dashboards; you need the per-service view.
Confirm the amplification signature. For that service, verify all three signals moving together: retry rate rising sharply, traefik_service_request_duration_seconds p95/p99 climbing, and 5xx rate moderate and rising. All three together is the pattern. Retries alone, with flat latency, is flapping, not amplification.
Compute the retry ratio as described above. This tells you severity and how much extra load Traefik is adding.
Check backend health signals. Look at traefik_service_server_up per backend URL. Declining or flapping values mean the pool is shrinking and survivors are absorbing original plus retry traffic. All values at 1 with high 5xx means health checks are misleading.
Find the original failure. The retry loop is a symptom. Check the backend’s own resources: CPU, memory, database connection pool, lock contention, downstream dependencies. A shared dependency (database, cache) failing is a common trigger because it degrades every backend at once.
Check what changed. A slow code deployment, a database query regression, an autoscaler scale-down, or a sudden traffic spike. The trigger explains why the backend degraded before the first retry was ever sent.
Check request methods on the retried routes. If the affected routes accept POST/PUT and retries are firing, assume duplicate side effects have occurred and plan reconciliation.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`traefik_service_retries_total` (rate)	The amplification mechanism itself; counts each retry per service	Rate >5% of request rate sustained; approaching request rate is severe
`traefik_service_request_duration_seconds` (p95/p99)	Shows the degradation the retries are responding to and worsening	Rising together with retries is the amplification signature
`traefik_service_requests_total{code=~"5.."}`	Distinguishes Traefik-generated errors (502/503/504) from backend pass-through	Moderate and rising alongside retries; 503 means pool collapse has started
`traefik_service_server_up`	Pool capacity; each backend dropping out concentrates load on survivors	Any backend at 0, flapping values, or declining count over time
Retry-to-request ratio	The single best top-level indicator for this failure mode	Sustained >5%; >50% means amplification is actively escalating

Note that traefik_service_server_up only exists for services with health checks configured. Absence of the series means unmonitored, not healthy.

Fixes

Break the loop first: reduce retry attempts

The first response is to cut retries on the affected service. Reduce the retry middleware’s attempts value (or remove the middleware from the router chain entirely) via dynamic configuration. This stops Traefik from multiplying load while you fix the actual problem.

Tradeoff: you lose masking of genuinely transient failures, so some clients will see errors they would not have seen before. During an active amplification incident, that is the right trade. A few visible errors are far better than a total outage.

Do not restart Traefik as a first move. It does not fix the backend, and it drops all in-flight connections.

Fix the root cause at the backend

Retries are never the root cause. Work the backend problem:

Database or connection pool exhaustion: kill blocking queries, raise pool limits, or shed load.
Bad deployment: roll back. This is the fastest fix when the degradation correlates with a release.
Shared dependency down: failing over or restoring the database/cache recovers all backends at once.
Insufficient capacity: scale the backend pool out. But note that adding backends during an active retry storm means new instances join a pool receiving amplified traffic; cut retries first, then scale.

Right-size the retry configuration

After the incident, revisit the retry policy rather than restoring it blindly:

Keep attempts low. Each additional attempt is another full request against the backend pool. For most services, one or two retries is the ceiling of what is safe under load.
Scope retries to idempotent routes. Retrying GET is usually safe. By default the middleware excludes non-idempotent POST, LOCK, and PATCH; only set retryNonIdempotentMethod: true deliberately, because it can duplicate side effects. Apply retry middlewares only to routers serving idempotent traffic.
Do not stack retries. If clients also retry aggressively on timeout, and the client timeout is shorter than Traefik’s full retry sequence, client-side retries multiply on top of Traefik’s. Coordinate timeouts so the client gives up after Traefik, not before.

Prevention

Alert on the retry-to-request ratio, not just error rates. Retries mask errors from clients, so a success-rate dashboard will not warn you. The ratio is the leading indicator: it rises before 5xx does.
Dashboard the amplification triad. Retry rate, service latency, and 5xx rate on the same per-service graph. The visual correlation is the fastest way to recognize the pattern at 3 a.m.
Make health checks meaningful. If health checks hit a lightweight path while real traffic stresses the database, Traefik will keep routing and retrying into functionally impaired backends. Health endpoints should exercise the dependencies that matter.
Document a per-service retry budget. Decide which services get retries at all, how many attempts, and which methods. Default-off for non-idempotent routes.
Load-test the failure mode. Deliberately degrade a backend in staging with retries enabled and watch the ratio. Teams that have seen the loop once in a controlled setting recognize it instantly in production.

How Netdata helps

Per-service retry rates out of the box: Netdata charts traefik_service_retries_total per service, so the spiking service is visible immediately rather than buried in an aggregate.
The amplification triad on one screen: retries, traefik_service_request_duration_seconds percentiles, and 5xx rates per service are correlated on the same dashboard, which is exactly the comparison this diagnosis requires.
Retry-to-request ratio alerting: alert on the ratio crossing thresholds (for example >5% sustained) so you are paged on the leading indicator, not on the 503s that arrive after pool collapse.
Backend pool visibility: traefik_service_server_up per backend URL shows the pool shrinking in real time as the cascade progresses.
ML anomaly detection on retry counters: catches retry rates deviating from baseline even when they have not yet crossed a static threshold, which matters because rolling deployments make naive thresholds noisy.