Traefik circuit breaker: shedding load from a failing backend

A backend is degrading. It is not fully down, so health checks still pass or flap, and Traefik keeps sending it full traffic. If you also have the retry middleware attached, every failed request comes back two or three more times, and the retry amplification loop finishes what the original failure started. The circuit breaker middleware exists for exactly this situation: it watches error and latency ratios on a router, and when they cross a threshold you define, it stops forwarding requests and answers with a fast 503 until the backend recovers.

The operational problem is that the breaker is nearly invisible. Traefik exposes no per-middleware Prometheus metrics, so there is no breaker-state gauge to alert on. An open breaker looks, at a glance, like a backend pool collapse: clients get 503s. And a badly tuned breaker is worse than none at all: it either never trips, or it flaps between open and closed and turns a steady degradation into an intermittent one that is much harder to debug.

What this means

The circuit breaker is a middleware. It sits in the middleware chain of a router and continuously evaluates an expression over the traffic it observes. The expression is built from three measurement functions:

• NetworkErrorRatio(): ratio of requests that failed at the network level (connection refused, reset, timeout before a response).
• ResponseCodeRatio(from, to, dividedByFrom, dividedByTo): ratio of responses in one status code range relative to another, for example ResponseCodeRatio(500, 600, 0, 600) for 5xx as a fraction of all responses.
• LatencyAtQuantileMS(quantile): request latency at a given quantile in milliseconds. The quantile must be a float with a trailing .0, so LatencyAtQuantileMS(50.0) is valid and LatencyAtQuantileMS(50) is not.

You combine these with comparison operators (>, >=, <, <=, ==, !=) and logical operators && and ||. The OR keyword is not supported; writing OR in an expression fails with a parse error at load time. Use ||.

The breaker has three states:

stateDiagram-v2
  [*] --> Closed
  Closed --> Open: expression true at checkPeriod
  Open --> Recovering: fallbackDuration elapsed
  Recovering --> Closed: recoveryDuration elapsed
  Recovering --> Open: expression still true
  Open: clients get fast 503
  Recovering: linear ramp of probes

• Closed: normal operation. All traffic flows. The expression is evaluated every checkPeriod (default 100ms).
• Open: the expression evaluated true. The breaker short-circuits the chain and returns 503 (configurable via responseCode) immediately, without touching the backend. This lasts fallbackDuration (default 10s).
• Recovering: after fallbackDuration, the breaker lets a linearly increasing number of probe requests through over recoveryDuration (default 10s). If the expression is satisfied again during recovery, it re-opens. If recovery completes, it closes.

Two properties of this design matter operationally. First, each router gets its own independent instance of the breaker, even if multiple routers reference the same middleware definition. One router can be open while another, using the “same” breaker, is closed. Second, the breaker only observes what happens after its own position in the middleware chain. Anything a preceding middleware short-circuits (auth rejections, rate limit 429s) is invisible to it.

The defaults are reasonable starting points, but the expression itself has no default. You must write it, and the quality of the whole mechanism depends on that expression matching your backend’s real failure signature.

Configuring it

A minimal dynamic configuration (file provider YAML):

http:
  middlewares:
    api-breaker:
      circuitBreaker:
        expression: "NetworkErrorRatio() > 0.30"
        checkPeriod: 100ms
        fallbackDuration: 10s
        recoveryDuration: 10s

Common expression patterns:

• Connection-level failure: NetworkErrorRatio() > 0.30. Trips when 30% of requests fail before a response.
• Application-level failure: ResponseCodeRatio(500, 600, 0, 600) > 0.30. Trips when 30% of all responses are 5xx.
• Latency degradation: LatencyAtQuantileMS(50.0) > 500. Trips when the median request takes over 500ms.
• Combined: NetworkErrorRatio() > 0.30 || ResponseCodeRatio(500, 600, 0, 600) > 0.50.

Ordering with retry. Retries add load; the breaker removes it. They complement each other only if ordered correctly. Place the retry middleware first and the circuit breaker after it in the chain. If the retry middleware sits after the breaker, the breaker can trip on the first failure before a retry has a chance to succeed, which defeats the retry entirely. With retry first, the breaker sees the final outcome of each request after retries are exhausted, which is the signal you actually want to trip on.

Known limitations to design around:

• There is no fallback service. An open breaker returns a static status code (503 by default). It cannot redirect traffic to a standby backend or a maintenance page upstream. Requests for fallback routing have been declined by the maintainers.
• In current stable releases, there is no minimum request count. A ratio-based expression can trip on the first handful of requests after a quiet period. A RequestThreshold() expression function that addresses this has been merged into the master branch but is not yet in a stable release. Until then, low-traffic routers need conservative ratios or latency-based expressions, which are less jumpy on small samples.
• The official documentation does not document the sliding-window size. Traefik’s circuit-breaker dependency implements 10 one-second rolling buckets for counters and a 10-second rolling histogram, so treat ratios as roughly the last 10 seconds and validate against your own traffic rate.

Common failure modes of the breaker itself

Quick checks

All read-only. These assume the metrics and API endpoints are reachable on the Traefik instance (typically port 8080 for the dashboard/API entrypoint; adjust for your deployment).

# 1. Is the 503 rate elevated on a specific service?
curl -s http://localhost:8080/metrics | grep 'traefik_service_requests_total' | grep 'code="503"'

# 2. Are the backends actually down, or is something upstream of them refusing traffic?
curl -s http://localhost:8080/metrics | grep traefik_service_server_up

# 3. Are retries running at the same time? (retry amplification signal)
curl -s http://localhost:8080/metrics | grep traefik_service_retries_total

# 4. Is latency rising on the affected service?
curl -s http://localhost:8080/metrics | grep traefik_service_request_duration_seconds

# 5. Is the breaker middleware actually loaded, and with what expression?
curl -s http://localhost:8080/api/rawdata | grep -i -A5 circuitbreaker

Check 2 is the discriminator. The breaker is a middleware; it does not change backend health state. If 503s are spiking while traefik_service_server_up still shows 1 for the service’s backends, the 503s are coming from the middleware layer, not from health-check eviction. That is either your circuit breaker working as intended or, if you have not configured one, another short-circuiting middleware. Note that traefik_service_server_up only exists for services with health checks enabled; absence of the series means unmonitored, not healthy.

How to diagnose a 503 spike

1. Confirm the 503s are real and scoped. Look at traefik_service_requests_total{code="503"} per service. A single service means a service-level cause; many services at once points at a shared dependency or at Traefik itself.

2. Check backend health state. If traefik_service_server_up is 0 for every URL in the service, this is backend pool collapse, not the breaker. Follow Traefik 503 Service Unavailable: no healthy backends left in the pool.

3. If backends are up, suspect the middleware layer. Verify a circuit breaker is attached to the router (check 5 above) and evaluate its expression by hand against recent traffic: what fraction of recent requests were network errors or 5xx?

4. Check the timing pattern. An open breaker produces a distinctive signature: a block of 503s lasting roughly fallbackDuration, then a recovery window, then either normal traffic or another block. Flapping at this cadence means the threshold is too close to the backend’s steady-state error rate.

5. Check retries. If traefik_service_retries_total is climbing in step with the 503s, you have both mechanisms active. Verify the middleware order (retry first) and consider whether retries are feeding the failure the breaker is trying to shed.

6. Check the logs. Breaker state transitions are not exposed as metrics. State changes and recovery decisions are emitted at DEBUG; fallback activation also emits a WARN. Traefik’s default log level is ERROR, so raise the level temporarily to see these transitions.

Metrics and signals to monitor

Because there is no breaker-state metric, breaker-aware alerting has to be composite: alert on 503 rate combined with backends still up. That combination almost always means intentional shedding (breaker, rate limiter, or another short-circuit) rather than infrastructure failure, and it should be routed and severity-tagged differently than a genuine pool collapse. Breaker 503s are the mechanism working; they deserve a ticket and a dashboard annotation, not a page.

Tuning guidance

Threshold never trips. The expression does not match the backend’s real failure mode. A backend that hangs responds too slowly rather than erroring, so NetworkErrorRatio() stays at zero and only a latency expression will catch it. Match the expression to the failure signature you actually see in an incident, not the one you expect.

Threshold flaps. The trip point sits inside the backend’s normal error band. Raise the ratio, lengthen fallbackDuration so the backend gets real time to recover, and lengthen recoveryDuration so the probe ramp is gentler. A breaker that re-opens on its first probe burst is telling you the backend needs minutes, not seconds.

Trips on low traffic. With no minimum request count in stable releases, small samples make ratios violent. For quiet routers, prefer a latency-based expression, or a higher ratio combined with && against a second condition so a single bad request cannot open the circuit.

503s surprise clients. The breaker returns a bare 503 by default. You can change responseCode, but you cannot return a body or redirect. If clients need a graceful degradation path, implement it at the client or in a higher layer, not in Traefik.

Prevention

• Write the expression from incident data. Take the error ratio and latency quantiles from your last real backend incident and set the threshold just outside that range. An expression invented in a vacuum will misfire.
• Fix the middleware order once, in review. Retry first, breaker after. Make it a lint or code-review rule for your dynamic config.
• Baseline per service. Universal thresholds across heterogeneous backends mask problems. A compute-heavy API and a static file service need different ratios and quantiles.
• Load-test the breaker before you need it. Trip it deliberately in staging with fault injection and confirm the 503 cadence, the recovery ramp, and that your dashboards show the composite signature.
• Revisit expressions after upgrades. The expression language is stable, but new functions land (for example RequestThreshold() on master). As of v3.7.12 it is not in a stable release; recheck the docs for your version before assuming a function exists.

How Netdata helps

• Netdata collects Traefik’s Prometheus endpoint and charts traefik_service_requests_total by response code per service, so a 503 spike from an open breaker is visible at per-second resolution instead of being averaged away.
• Plotting traefik_service_server_up alongside the 503 rate makes the key discriminator automatic: 503s with backends still up means middleware-level shedding; 503s with backends at 0 means pool collapse.
• Correlating traefik_service_retries_total with the 503 and latency charts shows whether retry amplification is feeding the failure the breaker is shedding, which is the escalation path in Traefik cascading backend failure.
• Latency histograms from traefik_service_request_duration_seconds let you watch the quantile your LatencyAtQuantileMS expression trips on, so you can see a threshold approaching before the breaker opens.
• Because Traefik exposes no breaker-state metric, anomaly detection on the 503 rate is the practical early warning: the periodic open/recover cadence of a flapping breaker shows up as a repeating anomaly pattern rather than a flat threshold breach.

Related guides

• Traefik 404 not found: requests arriving with no matching router
• Traefik 502 Bad Gateway: when the backend is unreachable or returns garbage
• Traefik 503 Service Unavailable: no healthy backends left in the pool
• Traefik 504 Gateway Timeout: the backend is alive but too slow
• Traefik 5xx error rate: telling Traefik-generated errors from backend errors
• Traefik cascading backend failure: how a partial outage becomes a total one
• Traefik config last reload success: monitoring configuration freshness
• Traefik dashboard returns 404: reaching the API and dashboard correctly
• Traefik health checks pass but requests fail: when the probe lies
• How Traefik actually works in production: a mental model for operators
• Traefik monitoring checklist: the signals every production edge router needs
• Traefik monitoring maturity model: from survival to expert