Traefik configuration reload storm: provider churn eating CPU

Traefik’s CPU is pinned, request latency is jittery, GC pauses are climbing, and traffic volume looks completely normal. The culprit is not the traffic. It is the control plane: rate(traefik_config_reloads_total[5m]) is sitting above 1 reload per second, and every reload rebuilds the entire routing table.

In busy Kubernetes and Docker environments, every pod event (creation, deletion, readiness change) can trigger a configuration rebuild. At high churn rates Traefik spends more CPU rebuilding the routing table than it spends routing requests. This is most visible during cluster-wide rollouts and autoscaler events, but it also appears when health checks flap and pods re-register in a loop.

The primary fix is a single configuration knob, providers.providersThrottleDuration, and the diagnostic path is short. This article walks through confirming the storm, classifying the churn as legitimate or pathological, and mitigating it without masking real problems.

What this means

Traefik is both a data-plane proxy and a control-plane configuration reconciler. Provider watchers (Kubernetes API, Docker socket, Consul, file) feed change events into an aggregator, which merges them and rebuilds the internal router tree. The rebuild allocates new router, middleware, and service objects, then swaps the handler: in-flight requests continue on the old configuration while new requests pick up the new one.

The swap itself is fast and safe. The cost is the rebuild, which grows with (routers x middlewares x services). With a large routing table, each rebuild consumes meaningful CPU and allocations, and the discarded objects feed straight into Go’s garbage collector. When events arrive faster than Traefik can digest them, you get a feedback loop: rebuilds consume CPU, CPU pressure slows request serving, latency jitter appears, and GC pauses spike from the churn of discarded configuration objects.

flowchart LR
  P[Provider events
pod churn, container events] --> A[Configuration
aggregator]
  A --> R[Routing table rebuild
CPU + allocations]
  R --> S[Handler swap
in-flight keep old config]
  R --> G[GC pressure
discarded objects]
  G --> L[Latency jitter
p99 spikes]
  R --> C[CPU saturation
rebuild vs routing]
  C --> L

Two characteristics make this failure mode easy to misdiagnose:

• Traffic metrics look fine. Request rate, error rate, and backend health are all normal. Only CPU, GC, and tail latency show the problem. If you alert only on 5xx and request volume, a reload storm is invisible.
• The trigger is external. Traefik is reacting correctly to what the provider tells it. The question is never “why is Traefik reloading” but “why is the provider sending this many events.”

Traefik v2.3 and later throttle provider-driven reloads with providers.providersThrottleDuration (default 2 seconds). Events arriving inside the throttle window are coalesced, so bursts become one rebuild instead of many. Many deployments leave this at the default, which is often not enough for clusters with thousands of churning pods.

Common causes

Quick checks

All of these are read-only and safe to run during an incident.

# Confirm the storm: reload rate from the metrics endpoint
curl -s http://localhost:8080/metrics | grep traefik_config_reloads_total

# Check last successful reload timestamp (is config fresh while churning?)
curl -s http://localhost:8080/metrics | grep traefik_config_last_reload_success

# CPU consumed by the Traefik process (rate this over 60s)
curl -s http://localhost:8080/metrics | grep process_cpu_seconds_total

# GC pressure: are pause durations climbing with the reload rate?
curl -s http://localhost:8080/metrics | grep go_gc_duration_seconds

# Memory: rebuild churn allocates new objects each time
curl -s http://localhost:8080/metrics | grep go_memstats_heap_inuse_bytes

# Kubernetes: what is the actual churn rate? Watch pod events live
kubectl get events -A --sort-by=.lastTimestamp | tail -30

# Which pods are flapping? High restart counts are the usual suspects
kubectl get pods -A --sort-by=.status.containerStatuses[0].restartCount | tail -20

# Is a rollout in progress right now?
kubectl get deployments -A

# How big is the routing table? (requires API enabled)
curl -s http://localhost:8080/api/http/routers | python3 -c "import json,sys; print(len(json.load(sys.stdin)))"
curl -s http://localhost:8080/api/http/services | python3 -c "import json,sys; print(len(json.load(sys.stdin)))"

# What throttle is configured? Check the static config (adjust path to yours)
grep -i throttle /etc/traefik/traefik.yml

If you have Prometheus, the single most useful query is rate(traefik_config_reloads_total[5m]). Sustained values above roughly 1 per second are the definition of this problem. Overlay it with rate(process_cpu_seconds_total[5m]) and the correlation is usually obvious.

How to diagnose it

1. Confirm the reload rate. Scrape traefik_config_reloads_total twice, 60 seconds apart. More than ~60 new reloads in that window confirms a storm. A burst right after Traefik starts is normal (every provider sends its initial configuration) and settles within seconds; ignore it.

2. Correlate reloads with CPU and GC. If process_cpu_seconds_total rate and GC pause durations rise and fall in lockstep with the reload rate, the rebuilds are the CPU consumer. If CPU is high but reload rate is low, you have a different problem (TLS handshake load, regex routing, compression).

3. Check tail latency, not averages. Reload storms produce jitter: intermittent p99 spikes aligned with rebuilds, not a sustained latency increase. Compare traefik_service_request_duration_seconds percentiles against the reload timeline. Also watch for rare 502 errors tightly correlated with reload increments: a rebuild can cancel the previous router’s context, which may surface as errors on in-flight requests in edge cases.

4. Classify the churn: legitimate or pathological. This is the decision that matters:

   • Legitimate: a mass deployment, a node drain, an autoscaler reacting to a real load change. The event rate reflects reality. You mitigate with throttling and possibly capacity, not by stopping the churn.
   • Pathological: flapping readiness probes, a crash-looping pod repeatedly re-registering, a misconfigured health check toggling a backend up and down. Here the throttle hides the symptom; the fix is to stop the flapping at the source.

5. Find the flapping source if pathological. Look for pods with rising restart counts, readiness probe failures in events, or backends oscillating in traefik_service_server_up. One flapping deployment can generate a continuous event stream.

6. Check the configured throttle. If providersThrottleDuration is absent from the static config, it is the default 2s. In a cluster with thousands of pods and continuous churn, 2 seconds of coalescing still permits up to 30 rebuilds per minute of expensive work.

7. Estimate rebuild cost. Fetch the router and service counts from the API. Rebuild cost scales with routers x middlewares x services, so 50 reloads per minute against 200 routers is a very different problem than against 10,000. Large tables also drive memory growth and GC pressure per rebuild.

Metrics and signals to monitor

Fixes

Raise the provider throttle

The direct mitigation. In the static configuration:

providers:
  providersThrottleDuration: 10s

The default is 2s. For busy Kubernetes environments, 5-10s is the recommended range. Events arriving within the window are coalesced into a single rebuild, so a 10s throttle caps rebuild cost at 6 per minute regardless of event volume.

Tradeoff, stated plainly: the throttle delays configuration convergence. A newly deployed service can take up to the throttle duration to become routable, and a removed backend can receive traffic for up to that long. For most workloads 5-10s of convergence lag is invisible. For environments where routing must converge in under a second (rare), you are trading rebuild CPU for staleness, and you should instead attack the churn source or the table size.

Stop pathological churn at the source

If the event stream comes from flapping, no throttle setting is the real fix:

• Fix readiness/liveness probes that toggle on tight timeouts or expensive checks. A probe that fails under brief load spikes causes the pod to drop out of endpoints and re-register, generating events both ways.
• Fix crash-looping pods. A pod restarting every 30 seconds is a reload generator. Its restart also signals a real application problem that deserves attention anyway.
• Batch mass deployments. If the storm coincides with your own CI/CD rolling out hundreds of services, stagger the rollout. The cluster events are legitimate, but nothing requires them all to land in the same 60 seconds.

Reduce rebuild cost per event

When the churn rate is legitimate and cannot be reduced, shrink what each rebuild costs:

• Reduce routing table size. Rebuild cost scales with routers x middlewares x services. Consolidate routers, remove dead Ingress/IngressRoute objects, and split very large configurations across multiple Traefik instances by namespace or ingress class if the table has grown into the thousands.
• Raise CPU limits for the Traefik pods as a stopgap. This does not fix the storm but converts “CPU-saturated with latency jitter” into “expensive but tolerable” while you work the real fixes.

Multi-instance note

Every Traefik replica watches the provider independently, so every replica rebuilds on the same event stream. In a 20-replica deployment, one pod event costs 20 rebuilds. Throttling helps each replica equally, but if rebuild cost is your bottleneck, fewer larger replicas burn less total CPU on rebuilds than many small ones watching the same cluster.

What not to do

• Do not restart Traefik as a fix. A restart clears nothing; the provider will immediately resend its full state and the churn resumes. The startup reload burst will make the first seconds worse, not better.
• Do not disable the provider watch (for example, switching to file provider) unless you are prepared to manage configuration statically. You trade a CPU problem for a stale-config problem, which fails more quietly and at a worse time.

Prevention

• Alert on the reload rate, not just its absence. Most teams that monitor configuration at all alert only on traefik_config_last_reload_success going stale. Add a low-severity alert on rate(traefik_config_reloads_total[5m]) sustained above ~1/s so the storm is caught before users feel the jitter.
• Set the throttle deliberately at install time. Treat providersThrottleDuration as a required sizing decision for any Kubernetes deployment, the same way you treat resource limits. Default 2s is a starting point, not a recommendation for large clusters.
• Baseline your cluster’s normal churn. Record the typical reload rate during business hours, during deploys, and overnight. Storm detection is deviation from that baseline; without it, every rollout looks like an incident or, worse, every incident looks like a rollout.
• Correlate deploy pipelines with reload spikes. If your rollout tooling stamps timestamps you can join against metrics, you can separate “expected rollout burst” from “unexpected storm” in one dashboard panel.
• Watch RSS during storm windows. Sustained reload churn against a large routing table drives memory growth that can end in an OOM kill. An alert on memory trend during high reload rates gives you runway before the cliff.

How Netdata helps

• Reload rate and CPU on one timeline. Netdata charts traefik_config_reloads_total rate alongside process CPU per second, so the correlation between rebuild frequency and CPU saturation is visible without exporting queries to another system.
• Go runtime visibility. GC pause durations, heap in use, and goroutine count are collected from the same metrics endpoint, letting you confirm that latency jitter is rebuild/GC-driven rather than backend-driven.
• Per-instance comparison. In multi-replica deployments, per-instance views make it easy to see all replicas rebuilding simultaneously, which distinguishes provider-driven churn from a single misbehaving instance.
• Latency jitter detection. Per-second service latency percentiles surface the intermittent p99 spikes that reload storms produce, which coarse 1-minute averages smooth away entirely.
• Anomaly flags on churn rate. ML-based anomaly detection on the reload counter flags deviations from your cluster’s learned baseline, catching pathological churn (flapping probes, crash loops) that static thresholds tuned for rollouts would miss.

Related guides

• Traefik 404 not found: requests arriving with no matching router
• Traefik 502 Bad Gateway: when the backend is unreachable or returns garbage
• Traefik 503 Service Unavailable: no healthy backends left in the pool
• Traefik 504 Gateway Timeout: the backend is alive but too slow
• Traefik 5xx error rate: telling Traefik-generated errors from backend errors
• Traefik ACME challenge failed: HTTP-01, DNS-01, and TLS-ALPN-01 renewal errors
• Traefik acme.json permissions and corruption: renewal silently blocked
• Traefik ACME rate limit: too many certificates already issued for this domain
• Traefik backend connection pool: keep-alive, MaxIdleConnsPerHost, and reuse
• Traefik cannot assign requested address: ephemeral port exhaustion
• Traefik cascading backend failure: how a partial outage becomes a total one
• Traefik certificate expired: when ACME renewal has been failing silently