Traefik 5xx error rate: telling Traefik-generated errors from backend errors

Your edge dashboard shows a 5xx spike on Traefik. Someone pages the backend team. The backend team finds nothing wrong in their application logs. An hour later it turns out Traefik had no healthy backends for one service, or a route disappeared after a config change, and the application was never in the request path at all.

Treating all 5xx responses as backend failures is one of the most common time-wasters in Traefik operations. A 502 generated by Traefik because it could not reach an upstream is a different incident from a 502 the backend produced itself. A 503 because every backend failed health checks is a different incident from a 503 your application returns under load. The root causes, the owners, and the fixes are all different.

Traefik already records the distinction. It instruments requests at two layers: at the entrypoint (where the client connection terminates) and at the service (where the request is forwarded to a backend). Comparing 5xx counts at these two layers tells you which layer produced the error before you open a single log file.

What this means

Every request through Traefik is counted at the entrypoint level in traefik_entrypoint_requests_total (labels: code, method, protocol, entrypoint). Requests that get routed and forwarded are counted again at the service level in traefik_service_requests_total (labels: code, method, protocol, service).

The operational meaning of the split:

• 5xx at the entrypoint but not at the service: Traefik generated the response itself. The request never successfully reached a backend. Causes: no router matched, no healthy backends in the pool, a middleware short-circuited the chain, or the connection to the upstream failed before a response existed.
• 5xx at both levels: the backend returned the error and Traefik passed it through. Traefik is a messenger here. The investigation belongs in the application, its database, or its downstream dependencies.

The same logic applies to 404s. An entrypoint-level 404 means no router matched the request: a Traefik configuration problem. A service-level 404 means the backend returned “not found”: an application problem. For the broader mental model of the routing pipeline, see How Traefik actually works in production.

flowchart TD
  A[5xx spike detected] --> B{5xx at entrypoint?}
  B -- yes --> C{5xx also at service level?}
  C -- no --> D[Traefik-generated error]
  C -- yes --> E[Backend-generated error]
  D --> F[Check server_up, routes, middleware, config freshness]
  E --> G[Check backend logs, dependencies, per-service 5xx]

Common causes

Quick checks

All of these are read-only. They assume the Prometheus metrics endpoint is reachable (typically on the Traefik admin port).

# 5xx at the entrypoint level
curl -s http://localhost:8080/metrics | grep 'traefik_entrypoint_requests_total' | grep 'code="5'

# 5xx at the service level
curl -s http://localhost:8080/metrics | grep 'traefik_service_requests_total' | grep 'code="5'

# Backend health per service (only present for services with health checks enabled)
curl -s http://localhost:8080/metrics | grep traefik_service_server_up

# Config freshness: is the routing table still updating?
curl -s http://localhost:8080/metrics | grep traefik_config_last_reload_success

# Retry activity per service
curl -s http://localhost:8080/metrics | grep traefik_service_retries_total

# Which routers does Traefik actually have loaded right now?
curl -s http://localhost:8080/api/http/routers

Two things to note before interpreting output. First, traefik_service_server_up only exists for services with health checks enabled; absence of the series means unmonitored, not healthy. Second, both metric families are emitted by default in Traefik v3 (addEntryPointsLabels and addServicesLabels default to true), but verify both are actually present in your scrape before you rely on the comparison. A missing service-level family makes “no 5xx at the service” an empty statement.

How to diagnose it

1. Confirm the split. Query the rate of 5xx at both layers over the same window. In PromQL, that is sum(rate(traefik_entrypoint_requests_total{code=~"5.."}[5m])) versus sum(rate(traefik_service_requests_total{code=~"5.."}[5m])). If the service-level rate roughly matches the entrypoint rate, the backend returned the errors. If the entrypoint rate is high and the service rate is near zero, Traefik generated them.

2. If Traefik-generated, branch on the code. A 503 pattern means Traefik had a matching router but no ready servers: check traefik_service_server_up for the affected service. A 502 pattern means Traefik tried an upstream and failed: check backend reachability, connection state, and whether health checks report up while real traffic fails (the classic “healthy yet failing” case where the health endpoint is fine but the real path is broken). Rising 404s mean routing: compare /api/http/routers output with what you expect to be loaded.

3. Check config freshness. If traefik_config_last_reload_success has stopped advancing while deployments are happening, Traefik is serving stale routes. Provider desync produces entrypoint-level errors (404s for new services, dead traffic for removed ones) while /ping keeps returning 200. See the /ping health-check trap.

4. If backend-generated, identify the service. Break the 5xx rate down by the service label. One service failing with everything else clean points at that application. Many services failing simultaneously points at a shared dependency: database, cache, or a downstream API.

5. Check for retry amplification. If traefik_service_retries_total is spiking alongside the 5xx rate and latency, Traefik may be multiplying load on an already-degraded backend. The final client response can still be 200 while retries triple backend traffic, so check this even when the client-visible error rate looks mild.

6. Correlate with deployments and config changes. A 5xx onset that lines up exactly with an increment in traefik_config_reloads_total or a deployment event narrows the cause to a routing or application change, not infrastructure.

One historical limitation: in early v2 releases, buffering middleware could make traefik_service_requests_total record code="0" while the entrypoint metric recorded the real status code. This was fixed by PR #9440 in Traefik v2.9.4, so current v2/v3 releases should not exhibit it; if the service-level codes look anomalous, verify against access logs before concluding anything.

Metrics and signals to monitor

Fixes

No healthy backends (503 at entrypoint)

Fix the backends, not Traefik. Check why health checks are failing: application crash, resource exhaustion, or a health-check path that does not reflect real application health. If the health endpoint passes while real traffic fails, align the health check with the actual traffic path. Do not alert only on traefik_service_server_up: services without health checks emit no series at all.

Upstream connection failures (502 at entrypoint)

Check network reachability, DNS resolution time for backend names, and connection pool behavior between Traefik and the backend. A frequent cause of intermittent 502s is a timeout mismatch across the chain: a cloud load balancer or backend with a longer idle timeout than Traefik sends requests on connections the other side already closed. Align the timeout chain so each layer’s timeout is longer than the one in front of it.

Middleware short-circuits

Traefik has no per-middleware Prometheus metrics, so you confirm these by elimination: entrypoint error present, service metric absent, and a middleware on that router that can produce that code (auth returning 401/403, rate limiting returning 429). Check the middleware configuration attached to the affected router and any recent changes to it.

Stale routes (404/5xx at entrypoint with frozen config timestamp)

Restore provider connectivity and authentication. Traefik retains its last-known configuration and recovers on its own once the provider is reachable again; there is no manual reload step. Then audit what changed during the desync window.

Backend application errors (5xx at both levels)

Traefik is healthy; do not spend time on the proxy. Take the failing service name from the service label and debug the application, its resource saturation, and its downstream dependencies. If retries are amplifying the load, reduce retry attempts on that service’s middleware while you fix the root cause.

Prevention

• Alert on the split, not just the total. A 5xx alert that includes the entrypoint-vs-service comparison routes the page to the right team immediately.
• Enable health checks on every production service. Without them you lose traefik_service_server_up and the earliest warning of backend pool collapse.
• Monitor config freshness. Alert on traefik_config_last_reload_success age in environments with active deployment churn, and compare timestamps across replicas in HA setups to catch per-instance drift.
• Track the retry ratio. Make traefik_service_retries_total relative to request rate a standing dashboard panel, not an incident-time query.
• Baseline per service. Error-rate thresholds that fit an API service do not fit a streaming service. Establish per-service baselines and alert on deviation, as described in the Traefik monitoring checklist.

How Netdata helps

• Netdata charts traefik_entrypoint_requests_total and traefik_service_requests_total broken down by status code, so the entrypoint-vs-service split is visible side by side without writing PromQL mid-incident.
• Per-second collection catches short 5xx bursts that a 30 or 60 second scrape interval averages away, which matters when the spike lasts less than a minute.
• traefik_service_server_up per backend URL is charted alongside error rates, making “503 with zero ready backends” a one-screen correlation.
• Retry counters, request duration percentiles, and config reload timestamps live on the same dashboard, so retry amplification and provider desync show up in the same view as the error rate.
• Anomaly detection on per-service error rates surfaces deviation from each service’s own baseline instead of relying on one global threshold.

Related guides

• How Traefik actually works in production: a mental model for operators
• Traefik monitoring checklist: the signals every production edge router needs
• Traefik monitoring maturity model: from survival to expert
• Traefik /ping returns 200 while everything is broken: the health-check trap