Traefik fails in ways that static proxies do not. It is both a data-plane proxy and a control-plane configuration reconciler, and its worst failure modes are silent: a dead provider connection, a stalled ACME renewal, a file descriptor limit creeping toward exhaustion. The process stays up, /ping returns 200, and traffic keeps flowing on stale configuration while new deployments get 404s.
This checklist organizes the signals that matter into four maturity levels: survival, operational, mature, and expert. Each level builds on the previous one. Every item names the exact metric, why it matters, and the severity it deserves. Use it to audit an existing monitoring setup or to build one from scratch.
One caveat up front: /ping is not on this list as a health signal. It tells you the process is alive and nothing else. It does not check provider connectivity, route validity, backend reachability, or certificate state. Treat it as a process check, never as a health assessment.
flowchart TD L1["Level 1 - Survival
process, traffic, FDs, fatal errors"] L2["Level 2 - Operational
5xx, backend health, latency,
config freshness, TLS expiry"] L3["Level 3 - Mature
retries, bytes, TLS versions,
CPU, goroutines, memory"] L4["Level 4 - Expert
per-router labels, GC pauses,
synthetic probes, HA drift"] L1 --> L2 --> L3 --> L4
Level 1: survival
Minimum viable monitoring. Catches “it is completely dead” and “it is about to hit a hard resource limit.”
- Scrape target up. Why: a dead proxy passes zero traffic. If the metrics endpoint stops responding, page. This is binary and unconditional; there is no legitimate reason for a production edge router to go silent.
- Process restarts. Why: watch
process_start_time_secondsfor recent start times. Frequent restarts mean crash loops (OOM kills, panics, port conflicts). In Kubernetes, checkrestartCountto distinguish a rolling update from a crash loop. Multiple restarts within 30 minutes is a ticket; scrape target down is a page. - Entrypoint request rate. Why:
rate(traefik_entrypoint_requests_total[5m])is the fundamental throughput signal. A drop to near-zero on a normally active entrypoint means traffic is not reaching Traefik (upstream load balancer, DNS, firewall) or Traefik cannot accept connections (FD exhaustion). Alert on a sustained drop greater than 50% from the same-time-of-day baseline for more than 5 minutes, without a known deployment cause. This is an anomaly signal, not a static threshold. - Fatal backend errors. Why: presence of
traefik_service_requests_total{code="503"}means a service has zero healthy backends and Traefik is returning 503 to every client. Any 503 on a production service warrants immediate investigation. - File descriptor ratio. Why:
process_open_fds / process_max_fdsis the one cliff-edge resource on this list. When the limit is hit, 100% of new connections fail instantly with no graceful degradation. A low inherited or misconfigured container limit of 1024 is catastrophically low for an edge proxy; modern daemon/runtime limits may be much larger. Ticket above 80%. Page above 95% sustained for more than 2 minutes while the ratio is still rising or not declining. The rising-trend condition prevents false pages on deployments that intentionally sit at high connection counts (heavy WebSocket or gRPC).
Level 2: operational
Covers the major failure modes: backend failures, stale configuration, certificate expiry, and routing problems.
- 5xx rate per service. Why:
rate(traefik_service_requests_total{code=~"5.."}[5m])by service tells you which backends are failing. Learn the distinction, because the causes are different: 502 means Traefik reached the backend and got garbage back, or the outbound transport hit a non-timeout network error; 503 means there are zero healthy backends; 504 means an outbound transport timeout—Traefik itself can emit it when the backend dial, TLS handshake, or configuredresponseHeaderTimeoutreturns a net.Error timeout. Ticket above 1% of total service requests sustained for 5 minutes; page above 5% sustained for 2 minutes. Do not alert on brief post-deployment spikes; backends warming connection pools after a rollout is expected behavior. - Backend health per server. Why:
traefik_service_server_up{service, url}drops to 0 when a backend fails health checks. Ticket when all URLs in a service are 0, and ticket when some are down (reduced pool capacity). Critical limitation: this metric only exists for services with Traefik health checks configured. Absence of the series means unmonitored, not healthy. For services without health checks, fall back to 503 rate monitoring. Also cross-reference with actual 5xx rates: a service can show all backends up while returning 502s because the health check path (/health) differs from the real traffic path. - Service latency. Why:
traefik_service_request_duration_seconds(histogram, default buckets[0.1, 0.3, 1.2, 5.0]) is end-to-end duration as Traefik sees it. Ticket when p95 exceeds the service-specific SLA for more than 5 minutes. Note the default buckets are coarse; configure sub-100ms buckets for low-latency services or your percentile estimates will be meaningless. - Configuration freshness. Why:
traefik_config_last_reload_successis a Unix timestamp of the last successful config apply. If it stops advancing while your environment is actively changing, Traefik is running on stale configuration: new services get no traffic, removed services keep receiving it. This is the single most missed signal in Traefik monitoring. Ticket when the timestamp is older than 5 minutes in an actively managed environment, correlated with evidence of deployment activity. Important:traefik_config_reloads_failure_totaldoes not exist in v3; you must infer failure from the timestamp not advancing. In static file-provider setups this alert does not apply, since config legitimately does not change for weeks. - Entrypoint 404 rate. Why:
traefik_entrypoint_requests_total{code="404"}means requests arrived that matched no router. That is a Traefik configuration problem, completely different from a service-level 404 (the backend returned it). A rising entrypoint 404 rate plus a frozen config timestamp is the signature of provider desync. Ticket on a sustained rate above 5% of entrypoint requests or a sudden increase beyond 3x baseline. - TLS certificate expiry. Why:
traefik_tls_certs_not_afteris a Unix timestamp of expiry per certificate (cn,sans,seriallabels). Ticket when any certificate has less than 7 days remaining; plan-level review at 30 days. Traefik manages default 90-day ACME certificates and starts renewal 30 days before expiry, so a Let’s Encrypt certificate 7 days from expiry means renewal attempts have been failing for about three weeks. There is no ACME failure metric; you can only detect the consequence, not the cause. The cause lives in Traefik logs: challenge failures, rate limits, lock contention, corruptedacme.json. Metrics alone cannot justify a page here because the store also holds dormant certificates no longer serving traffic. Pair with external synthetic TLS probes on real production hostnames to make this pageable. - Open connections. Why:
traefik_open_connections{entrypoint, protocol}(v3 name; in v2 this wastraefik_entrypoint_open_connections) tracks current connections per entrypoint. Ticket on sustained growth without a corresponding request rate increase, which indicates a connection leak. This is a supporting signal; FD-level alerting above is more comprehensive.
Level 3: mature
Catches subtle degradation patterns and feeds capacity forecasting.
- Retry rate. Why: retries mask backend instability from clients (the final response is a 200) while multiplying backend load. A high retry rate concurrent with rising latency is the retry amplification pattern: Traefik effectively DDoS-es a degrading backend while trying to help. Ticket when retries exceed 5% of requests sustained for 5 minutes. One hard caveat:
traefik_service_retries_totalappears in the documentation, but the retry middleware is still constructed with an empty listener list in v3.7.12, so it emits no Prometheus samples. This remains upstream issue 10928. Do not build dashboards or alerts on the absent series; infer amplification from latency, 5xx correlation, and access-logRetryAttempts. - Traffic volume in bytes. Why:
traefik_service_requests_bytes_totalandtraefik_service_responses_bytes_totalper service support capacity planning and anomaly detection (abnormal payload sizes, asymmetric responses). Ticket on sustained deviation beyond 2x baseline. - TLS version distribution. Why:
traefik_entrypoint_requests_tls_total{tls_version, tls_cipher}shows negotiated protocol versions. Any sustained TLS 1.0 or 1.1 traffic is a compliance and security ticket. A sudden appearance of unusual ciphers can indicate scanning or downgrade attempts. - CPU utilization. Why:
process_cpu_seconds_totalin Traefik is dominated by TLS handshakes, middleware processing (regex routing, gzip, rate-limit evaluation), and config rebuilds. A CPU spike correlated with risingtraefik_entrypoint_requests_tls_totalrate is a handshake storm; a spike correlated with risingtraefik_config_reloads_totalrate is a rebuild storm. Ticket on sustained high utilization relative to the container CPU limit. - Goroutine count. Why:
go_goroutinesshould track active connections plus a stable baseline of provider watchers and health checkers. Unbounded growth without a traffic increase is a leak, typically hung backend connections, and it ends in an OOM kill. Ticket on sustained growth beyond 2-3x baseline without corresponding traffic. - Memory trend. Why:
process_resident_memory_bytesandgo_memstats_heap_inuse_bytesgrowing over days without traffic growth means a leak. Go GC absorbs pressure gracefully until it cannot, then the OOM kill is instant. Ticket above 80% of the container limit sustained. Size container limits at roughly 2x observed stable peak heap for GC headroom. - Composite patterns. Why: at this level, correlate. If the retry metric ever ships, retries high plus latency rising is amplification, while retries high with latency flat is just instance flapping during rollouts. Today, the workable composites are: declining
traefik_service_server_upcount plus rising 5xx is a cascading backend failure, and theserver_updrop precedes the 503 spike, which makes it the early warning.
Level 4: expert
Full observability for root-causing subtle issues. Everything below has a real cost in cardinality, storage, or operational effort.
- Per-router metrics. Why:
addRoutersLabels: true(default is false) gives per-router visibility, at significant cardinality cost. Enable selectively. - GC pause analysis. Why:
go_gc_duration_secondsp99 above 100ms sustained causes latency spikes across all concurrent requests. Mostly diagnostic; matters at high allocation rates. - Entrypoint versus service latency split. Why: comparing
traefik_entrypoint_request_duration_secondswith the service-level histogram isolates Traefik’s own overhead (TLS, middleware, compression). Entrypoint high, service normal means the bottleneck is in the middleware stack, or, at extreme log volume, access-log buffer blocking. - Config churn correlation. Why:
rate(traefik_config_reloads_total[5m])correlated with error rates catches rebuild storms and the rare context-cancellation errors during rebuilds. In noisy Kubernetes environments,providersThrottleDuration(default 2 seconds) may need raising to 5-10 seconds. - Synthetic TLS probes per hostname. Why: closes the gap that makes certificate expiry pageable, by verifying actively served certificates rather than every cert in the store.
- Per-instance config freshness comparison. Why: in HA deployments each replica watches providers independently. Comparing
traefik_config_last_reload_successacross replicas catches a stale instance serving old routes behind a load balancer, which presents as intermittent, unreproducible failures. - Admin endpoint exposure checks. Why:
/api/*,/dashboard/,/debug/*reachable from public networks disclose your full routing table and backend topology. Probe externally on a schedule; this is not a Prometheus metric.
Version caveats that break checklists
Two traps silently invalidate Traefik monitoring during upgrades:
- Open connections metrics were renamed and reduced in v3.
traefik_entrypoint_open_connections,traefik_router_open_connections, andtraefik_service_open_connectionsare gone, replaced by a singletraefik_open_connectionslabeled only byentrypointandprotocol. Per-service open-connection visibility from v2 has no v3 replacement. Dashboards migrated without adjustment show zeros. - Reload failure metrics were suppressed in v3.
traefik_config_reloads_failure_totalandtraefik_config_last_reload_failureno longer exist. Freshness inference fromtraefik_config_last_reload_successis the only option.
Also note what the metric surface structurally cannot tell you: there are no per-middleware metrics (only entrypoint, router, and service level instrumentation), no ACME failure metrics, and no provider connectivity metrics. Those three gaps require logs, synthetic probes, and external checks respectively.
How Netdata helps
- Per-second scraping of the Prometheus endpoint catches the fast-moving signals on this list: FD ratio climbing toward the cliff, 5xx spikes during amplification, and connection count surges that minute-resolution scrapes average away.
- Correlating
traefik_config_last_reload_successage with entrypoint 404 rate surfaces provider desync, the failure mode where every individual health check looks fine. - Tracking
go_goroutinesandprocess_resident_memory_bytestrends over days catches leaks long before the OOM kill, while the process still looks healthy. - Per-service 5xx and latency dashboards with the 502/503/504 split preserved keep the three failure causes distinct instead of aggregating them into a useless “5xx” line.
- ML anomaly detection on entrypoint request rate handles the baseline problem directly, since “normal traffic” varies too much per deployment for static thresholds.
Related guides
The Traefik section is new; individual guide pages are still being added. See the Traefik guides hub for the full signal catalog, failure pattern catalogue, and maturity model this checklist is drawn from.