$ guides / traefik / traefik-service-retries-total ▌

Operations Guides

Traefik service retries climbing: reading the retry signal before it cascades

Treat traefik_service_retries_total as documented but not wired in current Traefik Proxy v2/v3: the retry middleware is constructed with an empty listener list, so no Prometheus samples are emitted. Your dashboards may show green: clients are getting 200s, the error rate looks flat, and /ping is happy. That is exactly why hidden retries are dangerous. Retries mask backend instability from clients while multiplying the load Traefik sends to the backends. By the time client-facing errors appear, the amplification loop may already be running.

This guide covers how to read the retry signal, how to tell a harmless burst (rolling update, instance flapping) from the early stage of a cascade, and what to do before the loop feeds itself.

What this means

The retry middleware sits in Traefik’s middleware chain, between the router and the service load balancer. When a request to a backend fails at the connection level, the middleware re-sends it, possibly to a different backend server. The documented traefik_service_retries_total (label: service) is intended to increment per retry attempt, but its listener is not instantiated; use access-log RetryAttempts as the practical per-request counter.

Three things follow:

The client may never know. A request that fails twice and succeeds on the third attempt returns 200. Your service-level 5xx rate stays low while the retry counter climbs. If you only alert on error rates, you are blind to this.
Backend load is amplified. Each retry is a full new request to the backend pool. At 2 retries per request, Traefik is sending up to 3x the client traffic to your backends. If the backends are degraded, that extra load makes the degradation worse.
Latency grows. Every retry adds a full request cycle. Rising retries and rising latency together are the signature of the amplification loop, not of harmless flapping.

By default, retries fire on connection-level failures only: no response at the TCP level. A backend that returns HTTP 500 has completed the request cycle from Traefik’s point of view and is not retried unless you explicitly configure status-based retries. This is the most common operator misunderstanding: if your backends are returning 500s and no retries appear in access logs, that is expected behavior. Also remember that an absent Prometheus retry series is the current wiring, not proof that retries are disabled.

flowchart TD
  A[Backend degrades: slow or dropping connections] --> B[Retry middleware re-sends failed requests]
  B --> C[Backend pool receives 2x-3x client traffic]
  C --> D[Remaining backends overload]
  D --> E[More timeouts and connection failures]
  E --> B
  D --> F[traefik_service_server_up declines]
  F --> G[503 to all clients]

Common causes

Cause	What it looks like	First thing to check
Rolling update or deployment	Retries spike, latency stays flat, pattern matches pod/container churn	Correlate retry onset with deployment events; check `traefik_service_server_up` for flapping backends
Backend partial degradation	Retries rising AND latency rising together	`traefik_service_request_duration_seconds` p95/p99 for the service; backend CPU/memory/DB pool
Network instability between Traefik and backends	Intermittent retry bursts across multiple services at once	Connection-level errors in Traefik logs; TIME_WAIT/CLOSE_WAIT socket counts
Aggressive retry config	High retry-to-request ratio even at low error rates	Retry middleware `attempts` value; whether retries are applied to non-idempotent methods
Health checks passing on a dead path	`traefik_service_server_up` all 1, retries and 5xx still climbing	Compare health check path with the real traffic path; see the false-positive health check guide
Autoscaler scale-down	Backends removed mid-request, retries spike, then settle	Correlation with scale-down events; watch for repeats at each scale event

The decisive split: high retries with flat latency is usually harmless flapping (rolling update, brief network blip). High retries with rising latency is danger. That combination means the retries are not absorbing the failures; they are feeding them.

Quick checks

All read-only. These assume Traefik’s Prometheus metrics endpoint is reachable (commonly on the Traefik dashboard/API port).

# Verify that the documented retry series is absent in current v2/v3
curl -s http://localhost:8080/metrics | grep traefik_service_retries_total || echo "not emitted"

# Request totals per service (for computing the ratio yourself)
curl -s http://localhost:8080/metrics | grep traefik_service_requests_total

# Backend health per server (only present if health checks are enabled)
curl -s http://localhost:8080/metrics | grep traefik_service_server_up

# Latency histogram for the affected service
curl -s http://localhost:8080/metrics | grep traefik_service_request_duration_seconds

Two samples 60 seconds apart give you rates: subtract, divide by 60. Compute the retry-to-request ratio per service from access-log RetryAttempts and ServiceName, not from the unwired Prometheus series. A service doing 10,000 req/s with 200 retries/s (2%) is in a different universe from one doing 50 req/s with 10 retries/s (20%).

In the access log, the RetryAttempts field shows retries per request. A request that succeeded only after retries shows RetryAttempts > 0 with a 200 response code:

# Requests that needed retries, from JSON access logs
jq 'select(.RetryAttempts > 0) | {RequestPath, RetryAttempts, DownstreamStatus, time}' /var/log/traefik/access.log | tail -50

This assumes JSON access log format and that the file path matches your deployment; adjust accordingly.

How to diagnose it

Compute the retry-to-request ratio per service. Group JSON access-log records with RetryAttempts > 0 by ServiceName, or use your log pipeline; do not use traefik_service_retries_total, which is not emitted. Under 1% is background noise. Over 5% sustained is a real problem. Approaching 100% means every request is being retried and you are in amplification territory.
Check the latency pair. Pull p95/p99 from traefik_service_request_duration_seconds for the same service and window. Retries up + latency flat = flapping, likely a deployment. Retries up + latency up = the loop is forming.
Check backend health. Look at traefik_service_server_up per URL for the service. A declining count means the healthy pool is shrinking and remaining backends are absorbing original plus retried traffic. If all servers show 1 but retries and 5xx climb, your health checks are lying: the probe path works, the real path does not.
Correlate with change. Did a deployment, scale event, or config reload (traefik_config_reloads_total incrementing) coincide with the retry onset? Rolling updates produce retry bursts that settle within minutes. A retry climb with no change event points at genuine backend or network degradation.
Confirm what is actually failing. Default retries fire on connection-level failure, not HTTP 5xx. If retries are firing, backends are refusing connections, resetting them, or not responding at all. Check backend logs for restarts, OOM kills, connection pool exhaustion, or accept-queue drops. If your 5xx rate is high but retries are flat, the backends are answering with errors and the retry middleware is not involved; that is a different incident (see the 5xx error rate guide).
Check the retry config itself. How many attempts are configured? Is the middleware attached to routes carrying POST/PUT traffic? Retrying non-idempotent methods can cause duplicate side effects downstream, which is a correctness problem on top of the load problem.

The v3 option is exactly retryNonIdempotentMethod; v3.7.0+ defaults to false and adds retries for POST, PATCH, and LOCK only when it is true. Traefik v2 does not expose this option.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Access-log `RetryAttempts` (rate, per service)	Leading indicator of backend instability, visible before client-facing errors	Retry attempts > 5% of service requests for 5m; > 50% means amplification
Retry-to-request ratio	Normalizes for traffic volume; raw counts mislead	Sustained climb without a deployment event
`traefik_service_request_duration_seconds`	Distinguishes flapping from cascade when paired with retries	p95/p99 rising while retries rise
`traefik_service_server_up`	Shows the healthy pool shrinking as the cascade progresses	Any URL at 0; count declining over time
`traefik_service_requests_total{code=~"5.."}`	Confirms whether failures are surfacing to clients	5xx rising alongside retries means retries are failing too
`RetryAttempts` in access logs	Per-request ground truth on which paths absorb retries	Successful 200s with `RetryAttempts > 0` clustering on one service
`traefik_config_reloads_total`	Retry bursts that track config reloads point at churn, not backend failure	Retry spikes tightly correlated with reload increments

Fixes

If it is a rolling update or known deployment

Do nothing to Traefik. Note the retry ceiling for the deployment so future alerts can suppress or tolerate that window. If the bursts are unreasonably large, the real fix is on the backend side: graceful shutdown (stop accepting, drain in-flight), startup/readiness probes that gate traffic until the app can actually serve, and health check intervals short enough that Traefik stops sending traffic to terminating backends quickly.

If the amplification loop is forming

Reduce retry pressure first. Lower attempts on the affected service’s retry middleware, or detach the middleware from the router, via dynamic configuration. Warning: this changes live routing behavior. It is a throttle, not a fix: it stops Traefik from multiplying load on your backend while you find the root cause. Tradeoff: clients now see the errors the retries were masking, so client-facing 5xx will rise. That is honest signal, and it is preferable to a total pool collapse.
Identify the degrading backend. Use traefik_service_server_up per URL plus backend-side metrics: CPU, memory, DB connection pool, downstream dependency latency.
Fix the root cause at the backend. Roll back the bad deploy, scale the pool, clear the DB lock contention. Retries are never the root cause; they are the accelerant.
Re-enable retries at a conservative level once the backend is stable.

If retries are flat but 5xx is high

The retry middleware is not firing because backends are answering with HTTP errors, not connection failures. On Traefik v3.7.0 and later, retry.status opts into status-based retries (for example ["500-599"]); retry.disableRetryOnNetworkError disables the default TCP/network retry and requires status to be set. Be careful: retrying 5xx from an overloaded backend amplifies load. Traefik v2 has no status-based retry; its middleware supports only attempts and initialInterval.

If health checks are lying

Fix the health check path so it exercises the same dependencies as real traffic, and cross-reference traefik_service_server_up against actual error rates from now on. A backend that passes /health while its database is down will absorb full traffic share, fail at the connection or application level, and drive retries against the rest of the pool.

Prevention

Alert on the ratio, not the count. Something like sum by (service) (rate(traefik_service_retries_total[5m])) / sum by (service) (rate(traefik_service_requests_total[5m])) > 0.1 for 5 minutes catches instability early without paging on deployment noise. Tune the threshold per service; batch-heavy or WebSocket-heavy services have different baselines.
Keep attempts low. Two to three attempts is the sane range for most services. Higher values convert every backend wobble into a load multiplier.
Do not retry non-idempotent methods unless you have application-level idempotency (idempotency keys, dedup). Duplicate charges and duplicate records are worse than a returned error.
Use initialInterval for backoff so retries do not land on the backend in a tight burst. Without backoff, N attempts arrive nearly simultaneously, which is the worst possible shape for an already-struggling backend.
Pair the retry alert with latency. A composite condition (retry ratio high AND service p95 rising) is far more page-worthy than either signal alone, and matches the amplification signature.
Size health checks honestly. Intervals and paths that reflect real dependencies shrink the window where Traefik sends traffic to doomed backends, which reduces retry volume at the source.

How Netdata helps

Netdata collects traefik_service_retries_total per service alongside request rates, so the retry-to-request ratio is one chart, not a hand-computed PromQL query during an incident.
The decisive pair for this symptom, retries rising versus latency rising, is visible side by side: traefik_service_request_duration_seconds percentiles next to retry rate per service.
Per-URL traefik_service_server_up shows the healthy pool shrinking in real time, which separates a cascade in progress from transient flapping.
ML anomaly detection on the retry counter flags a service whose retry behavior deviates from its own baseline, catching slow-building amplification that static thresholds miss.
Because retries hide errors from clients, correlating retry rate with 5xx rate and access-log RetryAttempts in one place shortens the path from “retries are up” to “backend X is the cause.”