$ guides / traefik / traefik-ha-config-drift ▌

Operations Guides

Traefik HA config drift: replicas serving different routing tables

The symptom: a route works, then it doesn’t, then it does. Refreshing the page flips between a working service and a 404 or 502. Testing from one network works; from another it fails. Behind this randomness is a simple mechanism: you run multiple Traefik replicas for availability, one has gone stale, and the load balancer in front of them sends your requests to whichever replica it picks.

Each Traefik replica watches its configuration provider (Kubernetes API, Docker socket, Consul, file) independently. There is no shared state between replicas. If one replica loses provider connectivity, through an RBAC change, a network partition, or a dead Docker socket, it keeps serving its last-known routing table. It does not flush routes, it does not crash, and /ping still returns 200. The other replicas update normally. Your fleet is now serving two different views of the world, and the failure only appears when the load balancer picks the stale instance.

ACME certificates make this worse. Certificates stored in acme.json are local to each replica and are not shared. A replica that renewed successfully serves a valid cert while another serves one approaching expiry, so TLS errors can be intermittent in exactly the same way.

What this means

Config drift in an HA Traefik deployment is provider desync multiplied by the replica count. A single stale replica behind a load balancer with N replicas fails roughly 1/N of requests for any route that changed after the desync. That produces the classic “works sometimes” report: the same URL, the same client, different outcomes depending on connection hashing.

flowchart LR
  C[Clients] --> LB[Load balancer]
  LB --> R1[Traefik replica 1
config current]
  LB --> R2[Traefik replica 2
config STALE]
  LB --> R3[Traefik replica 3
config current]
  P[Provider
K8s API / Docker / Consul] -->|watch OK| R1
  P -.->|watch broken
RBAC / network| R2
  P -->|watch OK| R3
  R1 --> B[Backends]
  R2 --> B
  R3 --> B

The key properties of this failure:

The stale replica looks healthy. Process is up, /ping returns 200, existing routes keep serving, established connections work.
Only changed routes fail. Routes that existed before the desync still work on the stale replica. New routes 404 on it; removed backends keep receiving (failing) traffic.
The blast radius grows over time. Every deployment, scale event, or cert renewal after the desync increases divergence.

Common causes

Cause	What it looks like	First thing to check
RBAC revoked or reduced for one replica’s service account (Kubernetes)	One pod’s config timestamp frozen; others current	`kubectl auth can-i --as=system:serviceaccount:<ns>:<sa> list ingresses`
Network partition between one replica and the provider	One replica stale; provider reachable from other pods	Exec into the stale pod and curl the provider API
Docker socket unmounted or daemon restarted on one node	Replicas on that node stop receiving container events	Verify socket mount and Docker daemon state on the affected node
Provider credential rotation missed by one instance	One replica’s watch fails auth while others reconnected	Traefik logs on the stale replica for provider auth errors
ACME state divergence	TLS cert errors from some clients only	Compare `traefik_tls_certs_not_after` per instance
File provider watching a node-local path that differs	Each replica literally has different config files	Compare the watched files across replicas

Quick checks

Run these against every replica. The point is comparison, not absolute values.

# 1. List your replicas so you know what to compare
kubectl get pods -n <ns> -l app=traefik -o wide

# 2. Compare last successful config reload across replicas
# A frozen timestamp on one replica is the smoking gun
for p in $(kubectl get pods -n <ns> -l app=traefik -o name); do
  echo -n "$p: "
  kubectl exec -n <ns> $p -- wget -qO- http://localhost:8080/metrics \
    | grep traefik_config_last_reload_success
done

# 3. Compare the loaded routing tables across replicas
# Different hashes = different routing tables = drift confirmed
for p in $(kubectl get pods -n <ns> -l app=traefik -o name); do
  echo -n "$p: "
  kubectl exec -n <ns> $p -- wget -qO- http://localhost:8080/api/http/routers \
    | md5sum
done

# 4. Same for services (backend pools)
for p in $(kubectl get pods -n <ns> -l app=traefik -o name); do
  echo -n "$p: "
  kubectl exec -n <ns> $p -- wget -qO- http://localhost:8080/api/http/services \
    | md5sum
done

# 5. Check TLS cert expiry per instance (ACME state is local)
for p in $(kubectl get pods -n <ns> -l app=traefik -o name); do
  echo -n "$p: "
  kubectl exec -n <ns> $p -- wget -qO- http://localhost:8080/metrics \
    | grep traefik_tls_certs_not_after
done

# 6. From the stale replica, test provider reachability
kubectl exec -n <ns> <stale-pod> -- wget -qO- --no-check-certificate \
  https://kubernetes.default.svc 2>&1 | head -5

Check 6 only proves network and TLS to the API server. It does not prove the service account still has watch permissions; check RBAC separately.

On the hashes: /api/http/routers returns that replica’s live router set. The comparison works if output ordering is deterministic across replicas.

How to diagnose it

Confirm the intermittency is replica-dependent. Hit the failing URL repeatedly. If errors cluster on a subset of requests with no client-side pattern, suspect per-replica divergence rather than a backend problem. If you can address pods directly (bypassing the upstream LB), send the same request to each replica and compare responses; that turns “works sometimes” into “replica 2 always 404s this route.”
Compare traefik_config_last_reload_success across all replicas. If one timestamp is frozen while others advance during active deployments, that replica is stale. In v3 there is no reload-failure counter; absence of progress is the failure signal.
Hash the API output across replicas. If reload timestamps diverge, confirm with the router and service hashes from the quick checks. A hash mismatch tells you the drift has already affected routing behavior.
Identify what the stale replica cannot see. Diff its /api/http/routers output against a healthy replica. Missing routes explain 404s; extra routes pointing at removed backends explain 502s. Which deployments are missing also tells you roughly when the desync happened.
Find why the replica went stale. Check its logs for provider errors (watch failures, RBAC denials, connection resets). Verify provider reachability and auth from inside that pod. In Kubernetes, verify the service account permissions.
Check ACME divergence separately. Even if routing config converges, certs may not. Compare traefik_tls_certs_not_after per instance for the same CN.
Verify the upstream load balancer health checks. If the LB health-checks /ping, it will keep sending traffic to the stale replica indefinitely because /ping lies. That is why the drift persists instead of self-isolating.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`traefik_config_last_reload_success` per instance	The primary drift detector; per-instance timestamps must advance together	One instance’s timestamp frozen while others advance during deploy activity
`/api/http/routers` content hash per instance	Ground truth of what each replica is actually serving	Hash mismatch across replicas sustained > 5 minutes
`traefik_entrypoint_requests_total{code="404"}` per instance	Entrypoint 404s mean “no router matched” on that replica	404 rate elevated on one replica only
`traefik_service_server_up` per instance	Stale replicas may report removed backends as up	Backend up/down sets differ across replicas
`traefik_tls_certs_not_after` per instance	ACME state is local; certs diverge independently	Same CN showing different expiry across replicas
`traefik_config_reloads_total` per instance	A flat counter means the watcher has stopped entirely	Counter not incrementing while deployments occur

Brief divergence during rollouts is normal; convergence takes a moment. Only sustained divergence (minutes) is worth alerting on.

Fixes

Restore provider connectivity to the stale replica

Fix the root cause first: restore RBAC, repair the network path, remount the Docker socket, or rotate the credential into that instance. Once the provider is reachable and authorized, Traefik resyncs automatically. No manual reload is needed; the watcher reconnects and applies the current state.

Remove the stale replica from rotation

If you cannot fix connectivity quickly, pull the stale replica out of the load balancer or scale it down. This stops user-facing intermittency immediately at the cost of capacity. Deleting the pod (in Kubernetes) is acceptable here: the replacement comes up with a fresh watch and current config. This is disruptive to in-flight connections on that replica; expect a small connection reset blip.

Restart only as a last resort, and understand why it works

Restarting the stale replica clears the symptom because startup performs a full provider resync. But if the underlying cause (RBAC, network, socket) is still broken, the replica comes up and immediately starts going stale again. Restart without fixing connectivity buys you one config load, not a fix.

Handle ACME divergence

Do not share acme.json between replicas; Traefik does not support concurrent access to it. Options: designate a single instance to handle ACME, use distributed ACME storage with locking (KV store), or move certificate management to an external controller such as cert-manager that stores certs in Kubernetes Secrets readable by all replicas. If one replica is already holding an expiring cert others have renewed, force renewal on it or remove it from rotation until it converges.

Prevention

Alert on per-instance reload timestamp divergence. Alert when max(traefik_config_last_reload_success) - min(traefik_config_last_reload_success) across instances exceeds a threshold (5 minutes in an actively-managed environment), conditioned on deployment activity. A flat alert on “timestamp old” false-fires on static environments.
Alert on router-hash mismatch. Periodically hash /api/http/routers per replica and alert on sustained mismatch. This catches drift even when timestamps look plausible.
Health-check the LB on something real. /ping only proves the process is alive. Where possible, have the upstream load balancer check a signal that reflects config freshness, or accept that LB-level isolation will not save you and rely on alerting instead.
Monitor provider health from each replica’s perspective. RBAC changes and network policies are per-identity and per-pod-path. A cluster-wide “provider is up” check does not see one replica losing access.
Treat ACME as per-replica state. Track traefik_tls_certs_not_after per instance, not as a fleet aggregate. The fleet minimum and the fleet maximum tell you divergence directly.
Watch entrypoint 404 rate per instance. A replica-specific 404 elevation is often the first user-visible symptom of drift and shows up before hash-diff tooling notices.

How Netdata helps

Netdata charts traefik_config_last_reload_success per instance on one dashboard, so a frozen replica stands out visually against advancing peers without writing a PromQL diff.
Per-instance entrypoint 404 rate correlation: when one replica’s 404 line separates from the fleet, that is drift surfacing as user-facing failure, and Netdata’s per-instance breakdown makes the split obvious.
Comparing traefik_service_server_up sets across instances side by side reveals backend-pool divergence (stale replicas reporting removed backends as up) that aggregated fleet views hide.
traefik_tls_certs_not_after per instance exposes ACME divergence: the same certificate showing different expiry timestamps on different replicas is immediately visible.
ML-based anomaly detection on per-instance reload and request-rate series catches the “one replica behaving differently” pattern without you having to predefine the divergence threshold.
Correlating reload timestamps with deployment events (config reload counter, request-rate changes) shortens the path from “intermittent 404s” to “replica 2 lost its provider watch at 14:03”.