Traefik serving stale configuration: the silent provider desync

A developer deploys a new service. The pods are running, the Ingress or IngressRoute exists, everything looks right in Kubernetes. But the service returns 404 from the edge. Meanwhile, a service deleted an hour ago is still receiving traffic and failing. You check Traefik: process is up, /ping returns 200, existing routes work. Nothing is alerting.

This is the silent provider desync, and it is the most commonly missed Traefik failure mode. The provider watch (Kubernetes API, Docker socket, Consul, file) has died or fallen behind. Traefik does not flush its routes when this happens. It keeps the last-known-good configuration and retries the provider connection in the background, with no loud failure signal. The routing table freezes at the moment of disconnection.

The blast radius grows with time. Every deployment after the disconnection is invisible. Every scale-down leaves dead backends in the pool. Every removed route keeps serving. In a static environment you might never notice. In an active Kubernetes cluster, the drift compounds hourly until someone reports their service is unreachable.

What this means

Traefik is both a data-plane proxy and a control-plane configuration reconciler. Provider watcher goroutines poll or watch configuration sources and feed updates into an internal aggregator, which rebuilds the routing table. When a provider loses connectivity, Traefik retries and keeps serving the configuration it already has. This is deliberate: a transient API blip should not wipe your routing table.

The tradeoff is that there is no failure counter to alert on. In Traefik v3, traefik_config_reloads_failure_total and traefik_config_last_reload_failure do not exist. You must infer the failure from absence: the reload timestamp stops advancing while the environment keeps changing.

flowchart TD
  K8s[Kubernetes API / Docker socket / Consul] -->|watch events| W[Provider watcher]
  W -->|config updates| A[Configuration aggregator]
  A -->|rebuild| R[Routing table]
  R -->|serve traffic| C[Clients]
  X[Watch dies: RBAC revoked, API unreachable, token rotated] -.->|watcher retries silently| W
  W -.->|no updates| A
  A -.->|frozen last-known-good| R
  R -.->|new services: 404; removed services: dead traffic| C

Two consequences follow:

• New services are invisible. Their routes were never loaded, so requests hit the entrypoint, match no router, and Traefik itself returns 404. This shows up in traefik_entrypoint_requests_total{code="404"}, not in any service metric, because no service was selected.
• Removed services keep getting traffic. Their routes still exist in the stale table. The backends behind them are gone or scaled down, so those requests fail, typically as 502s or connection errors against dead IPs.

Throughout all of this, /ping returns 200. The ping endpoint only proves the process is alive. It does not check provider connectivity, route freshness, or backend health. See the /ping health-check trap for why this check lies to you.

Common causes

A note on upgrades: moving between major versions or adding new resource types (for example Gateway API resources) can require additional RBAC rules. If the ClusterRole is not updated, the provider watch for those resources fails and the corresponding routers silently disappear from the loaded set while other routes keep working.

Quick checks

All read-only. These assume the API/dashboard and Prometheus metrics are enabled on port 8080; adjust for your deployment.

# 1. How old is the last successful config reload? Value is a Unix timestamp.
curl -s http://localhost:8080/metrics | grep traefik_config_last_reload_success

# 2. Is the reload counter still moving?
curl -s http://localhost:8080/metrics | grep traefik_config_reloads_total
# Take two samples 60s apart. Flat during active deployments = desync.

# 3. Are entrypoint-level 404s rising? (Traefik-generated, no matching router)
curl -s http://localhost:8080/metrics | grep 'traefik_entrypoint_requests_total{.*code="404"'

# 4. What routes does Traefik actually have loaded right now?
curl -s http://localhost:8080/api/http/routers | head -c 4000
# Compare against what should exist in the provider.

For a Kubernetes deployment:

# 5. Can Traefik's network path reach the API server?
# Test from a throwaway pod in the same namespace instead of the Traefik image;
# pin an approved curl image or use your organization's internal debug image.
kubectl run nettest --rm -i --image=curlimages/curl:8.5.0 --restart=Never -- \
  curl -sk --max-time 5 https://kubernetes.default.svc/healthz


# 6. Does the service account still have the permissions it needs?
kubectl auth can-i --as=system:serviceaccount:<ns>:<traefik-sa> list ingresses
kubectl auth can-i --as=system:serviceaccount:<ns>:<traefik-sa> watch ingressroutes.traefik.io

# 7. Look for provider errors in Traefik's logs
kubectl logs deploy/traefik --since=30m | grep -iE 'provider|watch|forbidden|unauthorized|reflector'

Note that check 5 only proves network and TLS reachability, not that Traefik’s specific service account is authorized; that is what check 6 covers. For a stronger signal, use kubectl debug to attach an ephemeral container to a Traefik pod so the test runs from the same network namespace.

For a Docker provider deployment:

# 8. Is the socket still mounted, and is the daemon responsive?
# Again, no shell inside the Traefik container. Inspect from the host.
docker inspect traefik --format '{{json .Mounts}}' | grep docker.sock
time curl --max-time 2 --unix-socket /var/run/docker.sock http://localhost/_ping

How to diagnose it

1. Confirm the freeze. Compare traefik_config_last_reload_success against wall clock and against deployment activity. If the timestamp is hours old and your cluster has had pod churn, deployments, or scaling events in that window, the provider watch is dead or lagging. A frozen timestamp alone proves nothing in a static environment; frozen timestamp plus known changes is the signal.

2. Check the counter, not just the timestamp. traefik_config_reloads_total flatlining while events are occurring means no reload attempts are happening at all. There is no failure to count because there are no attempts. Absence is the alarm.

3. Correlate with entrypoint 404s. A rising rate of traefik_entrypoint_requests_total{code="404"} on routes that should exist confirms the data-plane symptom. Distinguish these from service-level 404s, which are backend-generated and mean something entirely different. See Traefik 404 not found.

4. Diff the loaded config against the desired state. Pull /api/http/routers and /api/http/services from Traefik and compare with kubectl get ingressroutes,ingress -A (or your provider’s source of truth). Routers present in the provider but missing from Traefik’s view confirm desync rather than misconfiguration. If a route never existed in Traefik at all, also consider silent annotation or label rejection: an unknown annotation prefix or misspelled key is ignored without any error.

5. Find the provider-side cause. Work the cause table above: RBAC, API reachability, credentials, socket mount, Consul health. Traefik’s logs at the time the timestamp froze usually name the failing provider and the error.

6. In HA deployments, check each replica. Every Traefik instance watches the provider independently. One replica can lose connectivity while the others stay current, producing intermittent failures behind the load balancer. Compare traefik_config_last_reload_success across all instances. Significant divergence means the stale instance must be fixed or drained.

7. Restore and verify convergence. Once provider connectivity and permissions are fixed, Traefik re-syncs on its own; no manual reload is required. Watch the reload timestamp advance and the 404 rate fall. In large clusters the re-sync can take tens of seconds and produce a CPU spike as the routing table rebuilds. If connectivity is healthy but the timestamp remains frozen, use the reliable rolling-restart recovery path below.

Metrics and signals to monitor

The detection alert that works: (now() - traefik_config_last_reload_success) > threshold, combined with external evidence of change activity such as Kubernetes event rate or a deployment pipeline signal. The timestamp alone cannot distinguish “nothing changed” from “reload is broken.” This is a ticket-level signal, not a page, because existing routes keep working. Treat it as urgent in any environment with frequent deploys: drift compounds.

Fixes

RBAC or permissions restored

Re-apply the ClusterRole and ClusterRoleBinding the provider needs, including rules for any resource types added by a recent upgrade. Verify with kubectl auth can-i before assuming the fix landed. Traefik re-syncs automatically once the watch succeeds.

API server or network path restored

Fix the partition, DNS, or network policy blocking Traefik from the API server, Docker socket, or Consul cluster. Note the gap: events that occurred during a Docker daemon restart are lost, so after restoring the socket, verify the loaded config matches reality even if the watch reconnects cleanly.

Credentials rotated

Update the token, kubeconfig, or Consul ACL in Traefik’s configuration and restart only if the credential cannot be reloaded live. This is one case where a restart is the fix, not the workaround, because the watch may be holding a dead authenticated session.

Stale replica in an HA set

If one replica’s timestamp lags the others and does not recover after provider health is confirmed, remove it from the load balancer rotation and restart that instance. Restarting is safe here because the remaining replicas are current. Then investigate why that instance lost the watch.

Force convergence when the watch will not recover

If connectivity is confirmed healthy but the reload timestamp still does not advance, the watcher goroutine itself may be dead. There is no metric for “is the watcher actually watching.” A rolling restart of the affected instances is the reliable recovery. Audit what changed during the desync window afterward: routes that were deleted may have been serving dead traffic the whole time.

Prevention

• Alert on config freshness with context. Page nothing; ticket on traefik_config_last_reload_success older than a few minutes while deployment activity is detected. Without the activity correlation this alert false-fires on static setups.
• Baseline the reload counter. Know your normal traefik_config_reloads_total rate. A drop to zero in a busy cluster is as meaningful as a spike.
• Watch entrypoint 404 rate as a routing-freshness proxy. Sustained rate above roughly 5% of entrypoint requests, or 3x baseline, warrants a freshness check.
• Compare replicas. In HA deployments, alert on inter-instance divergence of the reload timestamp. Per-instance freshness comparison catches the intermittent-failure pattern before users do.
• Include RBAC in upgrade runbooks. Any Traefik or API-version upgrade should re-verify provider permissions as a step, not an afterthought.
• Throttling tradeoff. providersThrottleDuration batches provider events before applying a new config. Raising it reduces rebuild churn in noisy environments but delays config application, which can mask or mimic desync. If you raise it, tighten your freshness alert accordingly.

How Netdata helps

• Netdata charts traefik_config_last_reload_success and traefik_config_reloads_total per instance, so a frozen timestamp or flatlined counter is visible without writing PromQL, and replica divergence in HA setups shows up as per-instance drift.
• Entrypoint request rates broken down by response code put code="404" trends next to config freshness on the same dashboard, which is exactly the correlation that separates desync from backend trouble.
• Service-level 5xx and traefik_service_server_up per backend URL reveal the other half of the symptom: removed backends still receiving dead traffic from the stale pool.
• Anomaly detection on the 404 rate catches the slow, gradual rise that static thresholds miss, which matters because desync onset is rarely a step change.
• Per-second sampling and high-resolution history let you pinpoint the moment the timestamp froze and line it up with the deploy, RBAC change, or daemon restart that caused it.

Related guides

• Traefik 404 not found: requests arriving with no matching router
• Traefik 502 Bad Gateway: when the backend is unreachable or returns garbage
• Traefik 503 Service Unavailable: no healthy backends left in the pool
• Traefik 504 Gateway Timeout: the backend is alive but too slow
• Traefik 5xx error rate: telling Traefik-generated errors from backend errors
• Traefik cascading backend failure: how a partial outage becomes a total one
• Traefik health checks pass but requests fail: when the probe lies
• How Traefik actually works in production: a mental model for operators
• Traefik monitoring checklist: the signals every production edge router needs
• Traefik monitoring maturity model: from survival to expert
• Traefik /ping returns 200 while everything is broken: the health-check trap
• Traefik service server up at zero: backend health checks are failing