Traefik fails in ways that static proxies do not. It is both a data-plane proxy and a control-plane configuration reconciler, and its worst failure modes are silent: a dead provider connection, a stalled ACME renewal, a file descriptor limit creeping toward exhaustion. The process stays up, /ping returns 200, and traffic keeps flowing on stale configuration while new deployments get 404s.

This checklist organizes the signals that matter into four maturity levels: survival, operational, mature, and expert. Each level builds on the previous one. Every item names the exact metric, why it matters, and the severity it deserves. Use it to audit an existing monitoring setup or to build one from scratch.

One caveat up front: /ping is not on this list as a health signal. It tells you the process is alive and nothing else. It does not check provider connectivity, route validity, backend reachability, or certificate state. Treat it as a process check, never as a health assessment.

flowchart TD
  L1["Level 1 - Survival
process, traffic, FDs, fatal errors"]
  L2["Level 2 - Operational
5xx, backend health, latency,
config freshness, TLS expiry"]
  L3["Level 3 - Mature
retries, bytes, TLS versions,
CPU, goroutines, memory"]
  L4["Level 4 - Expert
per-router labels, GC pauses,
synthetic probes, HA drift"]
  L1 --> L2 --> L3 --> L4

Level 1: survival

Minimum viable monitoring. Catches “it is completely dead” and “it is about to hit a hard resource limit.”

  • Scrape target up. Why: a dead proxy passes zero traffic. If the metrics endpoint stops responding, page. This is binary and unconditional; there is no legitimate reason for a production edge router to go silent.
  • Process restarts. Why: watch process_start_time_seconds for recent start times. Frequent restarts mean crash loops (OOM kills, panics, port conflicts). In Kubernetes, check restartCount to distinguish a rolling update from a crash loop. Multiple restarts within 30 minutes is a ticket; scrape target down is a page.
  • Entrypoint request rate. Why: rate(traefik_entrypoint_requests_total[5m]) is the fundamental throughput signal. A drop to near-zero on a normally active entrypoint means traffic is not reaching Traefik (upstream load balancer, DNS, firewall) or Traefik cannot accept connections (FD exhaustion). Alert on a sustained drop greater than 50% from the same-time-of-day baseline for more than 5 minutes, without a known deployment cause. This is an anomaly signal, not a static threshold.
  • Fatal backend errors. Why: presence of traefik_service_requests_total{code="503"} means a service has zero healthy backends and Traefik is returning 503 to every client. Any 503 on a production service warrants immediate investigation.
  • File descriptor ratio. Why: process_open_fds / process_max_fds is the one cliff-edge resource on this list. When the limit is hit, 100% of new connections fail instantly with no graceful degradation. Default container limits of 1024 are catastrophically low for an edge proxy. Ticket above 80%. Page above 95% sustained for more than 2 minutes while the ratio is still rising or not declining. The rising-trend condition prevents false pages on deployments that intentionally sit at high connection counts (heavy WebSocket or gRPC).

Level 2: operational

Covers the major failure modes: backend failures, stale configuration, certificate expiry, and routing problems.

  • 5xx rate per service. Why: rate(traefik_service_requests_total{code=~"5.."}[5m]) by service tells you which backends are failing. Learn the distinction, because the causes are different: 502 means Traefik reached the backend and got garbage back; 503 means there are zero healthy backends; 504 means a timeout somewhere in the chain. Ticket above 1% of total service requests sustained for 5 minutes; page above 5% sustained for 2 minutes. Do not alert on brief post-deployment spikes; backends warming connection pools after a rollout is expected behavior.
  • Backend health per server. Why: traefik_service_server_up{service, url} drops to 0 when a backend fails health checks. Ticket when all URLs in a service are 0, and ticket when some are down (reduced pool capacity). Critical limitation: this metric only exists for services with Traefik health checks configured. Absence of the series means unmonitored, not healthy. For services without health checks, fall back to 503 rate monitoring. Also cross-reference with actual 5xx rates: a service can show all backends up while returning 502s because the health check path (/health) differs from the real traffic path.
  • Service latency. Why: traefik_service_request_duration_seconds (histogram, default buckets [0.1, 0.3, 1.2, 5.0]) is end-to-end duration as Traefik sees it. Ticket when p95 exceeds the service-specific SLA for more than 5 minutes. Note the default buckets are coarse; configure sub-100ms buckets for low-latency services or your percentile estimates will be meaningless.
  • Configuration freshness. Why: traefik_config_last_reload_success is a Unix timestamp of the last successful config apply. If it stops advancing while your environment is actively changing, Traefik is running on stale configuration: new services get no traffic, removed services keep receiving it. This is the single most missed signal in Traefik monitoring. Ticket when the timestamp is older than 5 minutes in an actively managed environment, correlated with evidence of deployment activity. Important: traefik_config_reloads_failure_total does not exist in v3; you must infer failure from the timestamp not advancing. In static file-provider setups this alert does not apply, since config legitimately does not change for weeks.
  • Entrypoint 404 rate. Why: traefik_entrypoint_requests_total{code="404"} means requests arrived that matched no router. That is a Traefik configuration problem, completely different from a service-level 404 (the backend returned it). A rising entrypoint 404 rate plus a frozen config timestamp is the signature of provider desync. Ticket on a sustained rate above 5% of entrypoint requests or a sudden increase beyond 3x baseline.
  • TLS certificate expiry. Why: traefik_tls_certs_not_after is a Unix timestamp of expiry per certificate (cn, sans, serial labels). Ticket when any certificate has less than 7 days remaining; plan-level review at 30 days. A Let’s Encrypt certificate 7 days from expiry means renewal attempts have been failing for weeks. There is no ACME failure metric; you can only detect the consequence, not the cause. The cause lives in Traefik logs: challenge failures, rate limits, lock contention, corrupted acme.json. Metrics alone cannot justify a page here because the store also holds dormant certificates no longer serving traffic. Pair with external synthetic TLS probes on real production hostnames to make this pageable.
  • Open connections. Why: traefik_open_connections{entrypoint, protocol} (v3 name; in v2 this was traefik_entrypoint_open_connections) tracks current connections per entrypoint. Ticket on sustained growth without a corresponding request rate increase, which indicates a connection leak. This is a supporting signal; FD-level alerting above is more comprehensive.

Level 3: mature

Catches subtle degradation patterns and feeds capacity forecasting.

  • Retry rate. Why: retries mask backend instability from clients (the final response is a 200) while multiplying backend load. A high retry rate concurrent with rising latency is the retry amplification pattern: Traefik effectively DDoS-es a degrading backend while trying to help. Ticket when retries exceed 5% of requests sustained for 5 minutes. One hard caveat: traefik_service_retries_total appears in the documentation, but Traefik maintainers have confirmed it was never actually implemented and emits nothing (upstream issue 10928). Do not build dashboards or alerts on a permanently zero series and assume you have coverage. Track the retry-to-request ratio once the metric actually ships; until then, infer amplification from latency and 5xx correlation.
  • Traffic volume in bytes. Why: traefik_service_requests_bytes_total and traefik_service_responses_bytes_total per service support capacity planning and anomaly detection (abnormal payload sizes, asymmetric responses). Ticket on sustained deviation beyond 2x baseline.
  • TLS version distribution. Why: traefik_entrypoint_requests_tls_total{tls_version, tls_cipher} shows negotiated protocol versions. Any sustained TLS 1.0 or 1.1 traffic is a compliance and security ticket. A sudden appearance of unusual ciphers can indicate scanning or downgrade attempts.
  • CPU utilization. Why: process_cpu_seconds_total in Traefik is dominated by TLS handshakes, middleware processing (regex routing, gzip, rate-limit evaluation), and config rebuilds. A CPU spike correlated with rising traefik_entrypoint_requests_tls_total rate is a handshake storm; a spike correlated with rising traefik_config_reloads_total rate is a rebuild storm. Ticket on sustained high utilization relative to the container CPU limit.
  • Goroutine count. Why: go_goroutines should track active connections plus a stable baseline of provider watchers and health checkers. Unbounded growth without a traffic increase is a leak, typically hung backend connections, and it ends in an OOM kill. Ticket on sustained growth beyond 2-3x baseline without corresponding traffic.
  • Memory trend. Why: process_resident_memory_bytes and go_memstats_heap_inuse_bytes growing over days without traffic growth means a leak. Go GC absorbs pressure gracefully until it cannot, then the OOM kill is instant. Ticket above 80% of the container limit sustained. Size container limits at roughly 2x observed stable peak heap for GC headroom.
  • Composite patterns. Why: at this level, correlate. If the retry metric ever ships, retries high plus latency rising is amplification, while retries high with latency flat is just instance flapping during rollouts. Today, the workable composites are: declining traefik_service_server_up count plus rising 5xx is a cascading backend failure, and the server_up drop precedes the 503 spike, which makes it the early warning.

Level 4: expert

Full observability for root-causing subtle issues. Everything below has a real cost in cardinality, storage, or operational effort.

  • Per-router metrics. Why: addRoutersLabels: true (default is false) gives per-router visibility, at significant cardinality cost. Enable selectively.
  • GC pause analysis. Why: go_gc_duration_seconds p99 above 100ms sustained causes latency spikes across all concurrent requests. Mostly diagnostic; matters at high allocation rates.
  • Entrypoint versus service latency split. Why: comparing traefik_entrypoint_request_duration_seconds with the service-level histogram isolates Traefik’s own overhead (TLS, middleware, compression). Entrypoint high, service normal means the bottleneck is in the middleware stack, or, at extreme log volume, access-log buffer blocking.
  • Config churn correlation. Why: rate(traefik_config_reloads_total[5m]) correlated with error rates catches rebuild storms and the rare context-cancellation errors during rebuilds. In noisy Kubernetes environments, providersThrottleDuration (default 2 seconds) may need raising to 5-10 seconds.
  • Synthetic TLS probes per hostname. Why: closes the gap that makes certificate expiry pageable, by verifying actively served certificates rather than every cert in the store.
  • Per-instance config freshness comparison. Why: in HA deployments each replica watches providers independently. Comparing traefik_config_last_reload_success across replicas catches a stale instance serving old routes behind a load balancer, which presents as intermittent, unreproducible failures.
  • Admin endpoint exposure checks. Why: /api/*, /dashboard/, /debug/* reachable from public networks disclose your full routing table and backend topology. Probe externally on a schedule; this is not a Prometheus metric.

Version caveats that break checklists

Two traps silently invalidate Traefik monitoring during upgrades:

  • Open connections metrics were renamed and reduced in v3. traefik_entrypoint_open_connections, traefik_router_open_connections, and traefik_service_open_connections are gone, replaced by a single traefik_open_connections labeled only by entrypoint and protocol. Per-service open-connection visibility from v2 has no v3 replacement. Dashboards migrated without adjustment show zeros.
  • Reload failure metrics were suppressed in v3. traefik_config_reloads_failure_total and traefik_config_last_reload_failure no longer exist. Freshness inference from traefik_config_last_reload_success is the only option.

Also note what the metric surface structurally cannot tell you: there are no per-middleware metrics (only entrypoint, router, and service level instrumentation), no ACME failure metrics, and no provider connectivity metrics. Those three gaps require logs, synthetic probes, and external checks respectively.

How Netdata helps

  • Per-second scraping of the Prometheus endpoint catches the fast-moving signals on this list: FD ratio climbing toward the cliff, 5xx spikes during amplification, and connection count surges that minute-resolution scrapes average away.
  • Correlating traefik_config_last_reload_success age with entrypoint 404 rate surfaces provider desync, the failure mode where every individual health check looks fine.
  • Tracking go_goroutines and process_resident_memory_bytes trends over days catches leaks long before the OOM kill, while the process still looks healthy.
  • Per-service 5xx and latency dashboards with the 502/503/504 split preserved keep the three failure causes distinct instead of aggregating them into a useless “5xx” line.
  • ML anomaly detection on entrypoint request rate handles the baseline problem directly, since “normal traffic” varies too much per deployment for static thresholds.

The Traefik section is new; individual guide pages are still being added. See the Traefik guides hub for the full signal catalog, failure pattern catalogue, and maturity model this checklist is drawn from.