Traefik config last reload success: monitoring configuration freshness

Traefik’s most dangerous failure mode is the one that looks like nothing. The process is up, /ping returns 200, existing routes keep serving traffic, and every dashboard is green. Meanwhile the configuration provider disconnected two hours ago, the three services you deployed since then are invisible, and the service you scaled down is still getting traffic at dead addresses. This is provider desync, and the only Traefik-native signal that exposes it is configuration freshness.

Two metrics carry that signal: traefik_config_last_reload_success (a gauge holding a Unix timestamp of the last successful configuration reload) and traefik_config_reloads_total (a counter of reload events). This guide covers what these metrics actually mean in v3, why they are harder to use than they look, and how to build staleness alerting that catches real desync without paging you every weekend on a static file-based deployment.

What it is and why it matters

Traefik is both a data-plane proxy and a control-plane reconciler. Background provider watcher goroutines (Kubernetes API, Docker socket, file, Consul, etcd) feed configuration changes into an aggregator that rebuilds the routing table. Each successful rebuild advances traefik_config_last_reload_success to the current time and increments traefik_config_reloads_total.

When the provider link breaks, Traefik retains its last-known configuration and retries with backoff. It does not flush routes. That is a sensible safety property and a nasty observability trap: established traffic keeps flowing while the routing table drifts further from reality. New deployments get no routes, removed backends keep receiving requests, and the failure grows with time instead of failing fast. The timestamp and the counter are the only place this drift is visible.

This matters most in high-churn environments. In Kubernetes, every pod event can trigger a reload, so a healthy instance advances the timestamp constantly. A frozen timestamp there is a red flag within minutes. In a static file-provider setup, the timestamp may legitimately sit unchanged for weeks, which is why naive staleness alerts false-fire.

How the two metrics actually work in v3

The v3 situation is worse than most operators assume, for three reasons.

There is no failure counter. In v2 the documentation listed traefik_config_reloads_failure_total and traefik_config_last_reload_failure. In v3 these were removed. The only reload metrics exposed are traefik_config_reloads_total and traefik_config_last_reload_success. Any alert rule copied from older community rule sets that references traefik_config_reloads_failure_total will match no series and never fire. You have a blind spot that looks like a working alert.

The counter does not distinguish success from failure. traefik_config_reloads_total increments on reload events; there is no success/failure split in v3. Failure must be inferred: the counter moves but the success timestamp does not advance, or neither moves while the environment is changing.

Absence of activity is not evidence of health. The worst case is the watcher goroutine silently stopping. Then there are no reload attempts at all, so traefik_config_reloads_total simply goes flat. No error metric, no failure counter increment, nothing for a threshold alert to catch. This is the core trap: there is no failure signal, only absence of activity. A dead watcher and an idle static config look identical in these two metrics.

flowchart TD
    A[Provider: K8s API / Docker / file] -->|watch events| B[Provider watcher goroutine]
    B -->|config update| C[Aggregator rebuilds routing table]
    C -->|success| D[traefik_config_last_reload_success = now, reloads_total +1]
    B -.->|watcher dies silently| E[No events, no reloads]
    E --> F[Timestamp goes stale, counter flat]
    C -->|rebuild fails| G[reloads_total may move, timestamp frozen]
    F --> H[Stale routes, new services invisible]
    G --> H
    H --> I["Alert: time() - last_reload_success, gated on expected change activity"]

Two practical consequences follow. First, you cannot alert on “reload failed” because that state is not exported. Second, any staleness alert must answer a second question: was a reload expected? That requires context from outside Traefik.

Where staleness shows up in production

Provider disconnection. Kubernetes API unreachable, RBAC revoked, Docker socket unmounted, Consul cluster down. Traefik keeps serving old routes. The symptom that eventually surfaces is rising entrypoint-level 404s (traefik_entrypoint_requests_total{code="404"}) as clients hit routes that exist in the provider but not in Traefik’s frozen view. See Traefik 404 not found: requests arriving with no matching router.

Watcher goroutine death. The watcher stops consuming events without crashing the process. traefik_config_reloads_total goes flat and stays flat. Because nothing “failed,” only a staleness alert gated on environment activity catches this. There is no dedicated metric for “is the watcher actually watching.”

File provider with subdirectories. The file provider watches a single directory non-recursively. If you organize dynamic config into subdirectories (routers/, services/), edits there never trigger a reload. The timestamp freezes while the team believes config is live. This looks exactly like a dead watcher in the metrics; the fix is layout, not connectivity.

HA instance drift. Each replica watches the provider independently. If one replica loses connectivity, it serves a stale routing table behind the load balancer, producing intermittent failures that depend on which instance handled the request. Comparing traefik_config_last_reload_success across replicas is the detection mechanism: divergence between instances means one is behind.

Static configs and false positives. The mirror-image problem. A file-provider deployment with watch: true whose files genuinely never change will have a timestamp that never advances. Any alert on raw timestamp age will fire continuously on a perfectly healthy system.

Alerting on staleness without lying to yourself

The core expression is timestamp age:

# Raw staleness: seconds since last successful reload
time() - traefik_config_last_reload_success

Start with a threshold around 5 minutes for actively managed environments, routed as a TICKET-level signal, never a PAGE. It cannot be paged because the monitoring system cannot distinguish “nothing changed” from “reload failing” without external context, and stale config degrades gradually rather than catastrophically.

That external context is the whole game. A production-quality staleness alert combines timestamp age with evidence that a change was expected:

Kubernetes deployments: gate on deployment or pod-churn activity. If your CI/CD emits events or you track rollout timestamps (for example via a deployment-activity metric or alert annotations from your CD system), require “staleness > 5m AND a rollout occurred in the last N minutes.” If the environment changed and the timestamp did not, the watcher or provider link is broken.
Counter flatness: pair the timestamp with rate(traefik_config_reloads_total[15m]) == 0 in environments where the counter is normally non-zero. A busy cluster where the reload rate suddenly drops to zero is suspicious even before the age threshold trips, because reloads in Kubernetes are frequent background noise.
Per-instance comparison in HA: max(time() - traefik_config_last_reload_success) by (instance) diverging across replicas catches the single-stale-instance case that a global view averages away.
Static file configs: suppress or scope out. For file-provider deployments where config changes only through your config-management pipeline, gate the alert on pipeline runs (a deployment event, a configmap version bump, an Ansible run timestamp) rather than on wall-clock age. Without that gate, disable the alert for those instances entirely; a permanently-firing alert is worse than none.

One manual lever worth knowing: when the file provider is active, sending SIGHUP to Traefik makes that provider rebuild its file-derived configuration. It does not force Kubernetes, Docker, Consul, or another provider to resync, so use it only as a file-provider diagnostic; test it on a non-production instance before relying on it.

Also check Traefik’s own logs when the timestamp is frozen. Provider connection errors, RBAC denials, and watch failures are logged, and the logs carry the cause that the metrics cannot. A related gotcha: Traefik can tolerate missing configuration files with only a log entry and no metric movement at all. For a single missing file-provider filename, v3.3 emits the wrapped open ...: no such file or directory error under Error while building configuration (for the first time), so log review is part of triage, not optional.

Correlating freshness with traffic symptoms

Staleness alone is a control-plane signal. To judge impact, correlate it with data-plane signals:

Entrypoint 404s rising while the timestamp is frozen: the desync is already user-visible. Requests are arriving for routes Traefik has not loaded.
traefik_service_server_up showing backends as up that have actually been removed: stale config keeps routing to dead addresses, surfacing as 502s at the service level. See Traefik 502 Bad Gateway and Traefik service server up at zero.
/ping returning 200 throughout: expected, and precisely why /ping is not a health signal. See Traefik /ping returns 200 while everything is broken.

The provider-desync composite pattern (frozen timestamp + rising entrypoint 404s + healthy /ping) is one of the highest-value correlations you can build into a dashboard, because each individual signal looks benign.

Signals to watch in production

Signal	Why it matters	Warning sign
`time() - traefik_config_last_reload_success`	Age of the routing table	>5m in an actively managed environment, gated on deployment activity
`rate(traefik_config_reloads_total[15m])`	Reload activity; flatness indicates dead watcher or quiet provider	Drops to 0 in an environment where reloads are normally frequent
Per-instance timestamp spread (HA)	Detects single-replica desync behind a load balancer	One replica significantly older than peers
`traefik_entrypoint_requests_total{code="404"}` rate	Data-plane symptom of missing routes	Rising while config timestamp is frozen
Traefik logs: provider errors, RBAC denials	Cause detail that metrics do not carry	Watch connection failures, permission errors
`process_start_time_seconds`	Distinguishes “watcher dead” from “process restarted recently”	Recent restart resets the freshness baseline

How Netdata helps

Timestamp age as a first-class chart: Netdata collects traefik_config_last_reload_success and traefik_config_reloads_total from the Prometheus endpoint and can alert directly on computed staleness, so the frozen-timestamp condition is visible without hand-building recording rules.
Counter flatness detection: per-second collection of traefik_config_reloads_total makes “the counter stopped moving” obvious on a dashboard, which is the only visible trace of a silently dead watcher goroutine.
Correlation in one view: entrypoint 404 rate, service-level 5xx, traefik_service_server_up, and config freshness sit on the same dashboard, so the provider-desync pattern (frozen config + rising 404s + green health) is visible in one place instead of a cross-tool investigation.
HA divergence: per-instance freshness charts expose a single stale replica behind a load balancer, the case that fleet-wide averages hide.
Process context: correlating freshness with process restarts and Go runtime metrics (goroutines, memory) helps separate “watcher died” from “whole process is unhealthy.”