Traefik context canceled during reload: intermittent errors on config change

You are seeing small bursts of failed requests: a handful of 499s or 502s, lasting a second or two, then nothing. The backends are healthy, health checks pass, latency is normal, and the errors do not repeat on any schedule. When you overlay the error timestamps on Traefik’s metrics, each burst lines up exactly with an increment of traefik_config_reloads_total.

During a dynamic configuration rebuild, Traefik’s cancelPrevState() cancels the context attached to the previous router configuration. In-flight requests, middleware chains, or backend connections that still hold a reference to that context receive a context-canceled error mid-request and fail. It is rare, intermittent, and almost impossible to reproduce on demand, which is why it gets misdiagnosed as flaky backends or network blips.

The blast radius is small: only requests in flight at the exact moment of the rebuild are affected. The danger is that frequent config churn (common in Kubernetes environments with heavy pod churn) turns a rare event into a steady drip of user-visible failures, and the correlation with reloads is the only reliable way to identify it.

What this means

Traefik rebuilds its routing table every time a provider delivers a configuration change. The rebuild uses a handler switcher: in-flight requests continue on the old handler, new requests land on the new one after the swap. In theory this is seamless. The edge case is that during the rebuild, cancelPrevState() cancels the previous router’s context. Any middleware, backend connection, or handler still holding that context gets context canceled while it is mid-flight.

The client sees one of two outcomes:

• 499 (client closed request): Traefik’s context cancellation surfaces as a 499 in the access log. Note that Traefik reports 499 for server-side context cancellation even when the client did not close the connection, so the code is misleading.
• 502: the context cancellation interrupts the backend exchange and Traefik reports it as a bad gateway.

The defining characteristic is timing. These errors do not follow traffic patterns, backend health, or latency. They follow traefik_config_reloads_total increments one-for-one.

flowchart LR
  P[Provider change
pod churn, label edit] --> R[Config rebuild triggered
traefik_config_reloads_total +1]
  R --> S[Handler switcher swaps
old router -> new router]
  R --> C[cancelPrevState cancels
old router context]
  I[In-flight request on
old context] --> C
  C --> E[context canceled mid-request
intermittent 499 or 502]
  N[New requests] --> S

Common causes

The last two rows matter because “context canceled” appears in Traefik logs for reasons unrelated to reloads. The reload correlation test is the discriminator.

Quick checks

All read-only. Run these against the affected Traefik instance.

# Confirm reload activity and freshness
curl -s http://localhost:8080/metrics | grep -E 'traefik_config_reloads_total|traefik_config_last_reload_success'

# Count 499s and 502s in the access log (JSON format)
grep -c '"DownstreamStatus":499' /var/log/traefik/access.log
grep -c '"DownstreamStatus":502' /var/log/traefik/access.log

# Look for context-canceled errors in Traefik's own log
grep -i 'context canceled' /var/log/traefik/traefik.log | tail -20

For the correlation itself, Prometheus queries are the practical tool:

# Reload events per minute
increase(traefik_config_reloads_total[1m])

# 499 and 502 rate at the entrypoint
sum by (code) (rate(traefik_entrypoint_requests_total{code=~"499|502"}[1m]))

If you run multiple Traefik replicas, check per-instance reload timestamps:

# Compare last successful reload across replicas
for i in traefik-1 traefik-2 traefik-3; do
  echo "$i: $(curl -s http://$i:8080/metrics | grep traefik_config_last_reload_success)"
done

How to diagnose it

1. Establish the correlation first. Plot increase(traefik_config_reloads_total[1m]) against the 499/502 rate over a window with at least a few error events. If every error burst sits on a reload increment, you have this edge case. If errors occur without reloads, you have a different problem: check readTimeout (see below) and ForwardAuth latency.

2. Quantify the churn. Measure rate(traefik_config_reloads_total[5m]). Occasional reloads with occasional 499s are cosmetic. Sustained reload rates above roughly one every few seconds mean the edge case is firing constantly and you have a reload storm as the underlying condition. In Kubernetes, common churn sources are autoscalers, rolling deployments, and flapping readiness probes re-registering endpoints.

3. Identify which requests die. From the access log, pull the requests with 499/502 during a reload window and look at RequestPath, ServiceName, and RequestProtocol. Long-running requests (large uploads, streaming responses, WebSocket handshakes) are disproportionately affected because they are more likely to be in flight during any given rebuild. If the victims are all sub-100ms API calls, the volume of affected requests tracks total throughput instead.

4. Rule out HA drift. If Traefik runs behind a load balancer with multiple replicas, each replica reloads independently. A client whose requests alternate between replicas may see errors from only one. Compare traefik_config_last_reload_success and reload counts across replicas and check whether the errors cluster on one instance.

5. Rule out the readTimeout impostor. Since v2.11.2, entryPoints have a default readTimeout of 60 seconds, which produces “context canceled” 499s on slow uploads and long-lived requests with no reload involved. The test is simple: if the 499s hit the same long-running endpoints at a consistent ~60s duration and ignore reload timing, that is readTimeout, not cancelPrevState(). Also note that in v3.x the experimental fastProxy mode reportedly returns 504 instead of 499 on read timeout, which changes what you grep for.

6. Check the logs at the right level. The context-canceled errors from this edge case may only appear at DEBUG level in Traefik’s log, while the access log shows the 499/502 regardless. Do not conclude “no evidence” from an empty application log alone.

Metrics and signals to monitor

Fixes

Reduce config churn at the source

This is the only true fix. The edge case fires once per rebuild; fewer rebuilds means fewer killed requests. In Kubernetes, find what is generating provider events: autoscalers oscillating replica counts, CI/CD systems redeploying constantly, or probes flapping endpoints in and out of service. Fixing a flapping readiness probe often cuts reload volume by an order of magnitude.

Throttle provider updates

Traefik coalesces rapid provider events using providersThrottleDuration (2 seconds by default in current versions). Increasing it to 5-10 seconds in busy environments batches many small changes into a single rebuild, directly reducing both the reload rate and the number of context-cancellation events. The tradeoff: new services and removed backends take a few extra seconds to be reflected in the routing table. For most workloads that delay is invisible; for rapid failover scenarios it may not be.

Batch provider-side changes

If your deployment pipeline applies many Ingress, IngressRoute, or label changes in sequence, batch them. Ten changes applied one second apart produce ten rebuilds; the same ten changes in one apply produce one. With the file provider, write configuration atomically (write to a temp file, then rename) so a partially written file never triggers an extra rebuild.

Tune long-lived traffic away from the blast radius

Requests most likely to be caught mid-flight are the long ones: uploads, streams, WebSockets. You cannot make them immune, but you can make the damage visible and bounded by ensuring these routes have deliberate respondingTimeouts settings rather than inherited defaults, so a genuine timeout is distinguishable from a reload casualty.

What not to do

Do not restart Traefik. The mechanism is inherent to how the handler switcher and cancelPrevState() work; a restart changes nothing. Do not chase the backends: traefik_service_server_up staying at 1 throughout is the proof they are innocent. And do not disable health checks or retries in response; they are unrelated to this failure mode.

Prevention

• Alert on reload rate, not just reload failures. A sustained rate(traefik_config_reloads_total[5m]) above your environment’s norm is the leading indicator. There is no reload-failure counter in Traefik v3, so infer health from the success timestamp, but alert on the rate itself for this failure mode.
• Baseline your environment’s normal churn. A quiet VM-based deployment might reload a few times a day; a busy Kubernetes cluster might reload a few times a minute legitimately. The alert threshold must come from your baseline.
• Set providersThrottleDuration deliberately rather than inheriting the default, especially on clusters with heavy pod churn.
• Keep a 499 dashboard panel. Most teams never graph 499s because “the client went away.” In front of Traefik, a 499 rate that tracks reloads is a proxy-internal signal, not client behavior.
• Compare reload timestamps across HA replicas on a schedule. Divergence indicates one instance is churning or stale independently, which produces the intermittent per-client errors that are hardest to reproduce.

How Netdata helps

• Netdata charts traefik_config_reloads_total and traefik_config_last_reload_success per instance at per-second resolution, so reload bursts are visible at the exact second they happen rather than averaged away.
• Entrypoint request metrics broken down by response code let you overlay the 499/502 rate on reload events in the same dashboard, which is the single correlation that confirms this edge case.
• Per-service 5xx charts alongside traefik_service_server_up let you rule out backend health in one glance: errors with all backends up points at the proxy, not the application.
• Process CPU and Go runtime charts (goroutines, heap) expose the secondary cost of reload storms, where rebuild CPU starts competing with request serving.
• Anomaly detection on the reload rate flags churn spikes from flapping probes or misbehaving autoscalers before they generate enough 499s to be reported by users.

Related guides

• Traefik 404 not found: requests arriving with no matching router
• Traefik 502 Bad Gateway: when the backend is unreachable or returns garbage
• Traefik 503 Service Unavailable: no healthy backends left in the pool
• Traefik 504 Gateway Timeout: the backend is alive but too slow
• Traefik 5xx error rate: telling Traefik-generated errors from backend errors
• Traefik ACME challenge failed: HTTP-01, DNS-01, and TLS-ALPN-01 renewal errors
• Traefik acme.json permissions and corruption: renewal silently blocked
• Traefik ACME rate limit: too many certificates already issued for this domain
• Traefik backend connection pool: keep-alive, MaxIdleConnsPerHost, and reuse
• Traefik cannot assign requested address: ephemeral port exhaustion
• Traefik cascading backend failure: how a partial outage becomes a total one
• Traefik certificate expired: when ACME renewal has been failing silently