Traefik monitoring maturity model: from survival to expert

Most Traefik outages are not exotic monitoring gaps. They are the same six failures repeating across teams: the process died, the FD limit was 1024, the cert expired, the provider went silent, the backend pool collapsed, or retries amplified a partial failure into a total one. A maturity model helps because it sequences coverage against the failures you are actually going to have, in the order you are going to have them.

This article defines four levels of Traefik monitoring: Survival, Operational, Mature, and Expert. Each level builds on the previous one. The levels are not about how many dashboards you have. They are about which failure classes you can detect before users do, and how quickly you can isolate the layer at fault.

Use this as a gap assessment. Find the highest level where you genuinely have every signal covered and alerting in place. Everything above that level is your backlog.

flowchart TD
  L1["Level 1: Survival
process, traffic, FDs, certs"]
  L2["Level 2: Operational
per-service errors, health,
latency, config freshness"]
  L3["Level 3: Mature
retries, Go runtime, TLS versions,
composite patterns"]
  L4["Level 4: Expert
per-router labels, HA drift,
ACME forensics, synthetics"]
  L1 --> L2 --> L3 --> L4

How to read the levels

Each level lists signals, what they catch, and what still gets through. A few rules before you start:

Metric names below assume Traefik v3 with Prometheus metrics enabled. Some names differ from v2. Where a metric is version-sensitive, that is called out.
A signal you collect but never alert on does not count. Coverage means the alert fires and someone acts on it.
The /ping endpoint is not a health signal. It returns 200 whenever the process is alive, regardless of provider connectivity, route validity, backend health, or certificate state. It appears nowhere in these levels as a primary signal for that reason.

Level 1: Survival

Minimum viable monitoring. This level answers two questions: is Traefik alive, and is it about to hit a wall. If you have nothing else, have this.

Process and scrape liveness. The Prometheus scrape target for Traefik is reachable and process_start_time_seconds is not resetting frequently. A dead proxy passes zero traffic, so target-down is a page. Repeated restarts within 30 minutes indicate a crash loop: OOM kills, panics, or port conflicts. In Kubernetes, distinguish pod replacement during a rollout from the same pod dying repeatedly by checking restart counts.
Entrypoint request rate. traefik_entrypoint_requests_total is above zero on entrypoints that normally serve traffic. A drop to near-zero means traffic is not reaching Traefik (upstream DNS, load balancer, firewall) or Traefik cannot accept connections (FD exhaustion, bind failure). This is an anomaly signal, not a static threshold: baseline it per entrypoint per time of day.
Presence of 503s. Any traefik_service_requests_total{code="503"} on a production service. A 503 means every backend for that service failed health checks and Traefik has nowhere to send the request. Traefik itself is healthy; the pool is not.
File descriptor ratio. process_open_fds / process_max_fds. Page at 95% sustained for 2 minutes and still rising. Ticket at 80%. FD exhaustion is a cliff edge: existing connections keep working, new connections fail instantly, and there is no graceful degradation. If process_max_fds reports 1024, that is the default container limit and it is too low for a production edge proxy. Fix the limit before you tune the alert.
Certificate expiry. traefik_tls_certs_not_after per certificate. Ticket under 7 days, plan under 30. Let’s Encrypt certificates renew 30 days before expiry, so a cert at 7 days has had renewal failing for roughly 53 days. There is no ACME failure metric; approaching expiry is how you find out.
Memory versus limit. process_resident_memory_bytes against the container memory limit. Go’s garbage collector absorbs pressure until it cannot, and then the OOM kill is instant. Ticket at 80% of limit sustained.

What this level misses: which backend is failing and why, latency degradation, stale configuration, retry amplification, and every slow-burn failure. Level 1 tells you the building is on fire. It does not tell you which floor.

Level 2: Operational

This is the level where you stop learning about incidents from user reports. Everything from Level 1, plus the signals that localize failures to a service, a provider, or a route.

Per-service error rate by code. traefik_service_requests_total split by code and service. Do not lump 5xx together. A 502 means Traefik reached the backend and got garbage back. A 503 means health checks removed every backend. A 504 means the backend is alive but too slow. These have different root causes and different fixes. Ticket at 5xx above 1% of service requests for 5 minutes.
Backend health. traefik_service_server_up{service, url} is 0 or 1 per backend. All zeros for a service means 503s for every request. One important limitation: this metric only exists for services with Traefik health checks configured. If health checks are not configured, the series is absent, and absence means unmonitored, not healthy. For those services, fall back to 503 rate or external probes.
Service latency. traefik_service_request_duration_seconds histograms, alerting on p95 against a per-service baseline. The default buckets (0.1, 0.3, 1.2, 5.0 seconds) are coarse; if your SLA lives between them, adjust buckets or accept imprecise percentiles. Set thresholds per service. 200ms p95 is excellent for a compute-heavy API and terrible for static files.
Config reload freshness. traefik_config_last_reload_success is a Unix timestamp of the last successful reload. In an actively changing environment, alert when the timestamp stops advancing. Traefik retains its last-known configuration when a provider disconnects; it does not flush routes. Existing routes keep working, new services get nothing, removed services keep receiving traffic, and /ping stays green the entire time. There is no reload failure counter in v3: traefik_config_reloads_failure_total does not exist. You infer failure from the timestamp not advancing.
Open connections. traefik_open_connections{entrypoint, protocol} per entrypoint. Sustained growth without a matching request-rate increase suggests a connection leak. In v3 this is a global gauge with only entrypoint and protocol labels; the old per-service and per-router open-connection metrics from v2 were removed because they were counting HTTP requests, not TCP connections. For saturation alerting, the FD ratio from Level 1 is the comprehensive signal; this one is for diagnosis and capacity planning.
Entrypoint 404s. traefik_entrypoint_requests_total{code="404"} measures requests that matched no router. This is a Traefik configuration signal: stale provider config, misconfigured annotations or labels silently ignored, or scanning. Do not confuse it with service-level 404s, which the backend generated and mean something entirely different. A rising entrypoint 404 rate combined with a frozen traefik_config_last_reload_success is the signature of provider desync.

What this level misses: degradation that never becomes an error, resource trends that precede saturation, and the composite patterns where two or three signals moving together are the real alarm. Most teams should treat Level 2 as the minimum bar for production.

Level 3: Mature

Level 3 adds leading indicators and pattern detection. The goal shifts from “detect failures” to “see failures forming.”

Retry ratio. traefik_service_retries_total relative to traefik_service_requests_total per service. Retries mask failures from clients (the final response is a 200) while doubling or tripling backend load. When a backend degrades, retries can push it over the edge: Traefik effectively DDoS-es the backend while trying to help. Alert on retries above 5% of requests sustained. The dangerous pattern is retries and latency rising together; retries alone with flat latency is usually instance flapping during a rollout. One caveat: early v3 releases lacked this metric; current v3 source and reference docs wire it, but scrape your endpoint to confirm the series before trusting an alert.
Goroutine count. go_goroutines against a baseline. Each active connection spawns goroutines; the count should track traffic. Monotonic growth disconnected from traffic is a leak, usually from hung backend connections with missing or overly long timeouts. Leaked goroutines hold memory, and the end state is OOM. Ticket at 3x baseline without a matching traffic increase.
Memory and GC. go_memstats_heap_inuse_bytes trending and go_gc_duration_seconds p99. Heap growing without traffic growth points to a leak or to routing-table growth. GC pauses above 100ms p99 cause latency spikes across all in-flight requests. These are the early chapters of the OOM story that Level 1 only reads at the last page.
CPU with attribution. process_cpu_seconds_total rate, correlated against traefik_entrypoint_requests_tls_total (TLS handshake load) and traefik_config_reloads_total (rebuild cost). CPU in Traefik is dominated by TLS handshakes, middleware processing, and config rebuilds. High CPU with low request rate and high reload rate is a rebuild storm, common in clusters with heavy pod churn.
TLS version and cipher distribution. traefik_entrypoint_requests_tls_total{tls_version, tls_cipher}. Any sustained TLS 1.0 or 1.1 traffic is a compliance finding and worth a ticket. Sudden cipher distribution shifts can indicate scanning or downgrade attempts.
Traffic volume. traefik_service_requests_bytes_total and traefik_service_responses_bytes_total per service, for capacity planning and payload anomaly detection.
Composite patterns. Wire alerts for the multi-signal failures: retry amplification (retries, latency, and 5xx rising together), cascading backend failure (server_up declining gradually, retries spiking before the 503 spike), and provider desync (frozen config timestamp plus rising entrypoint 404s while deployments are happening). Single-signal alerts catch simple failures; these catch the ones that escalate.

What this level misses: per-route attribution, cross-instance consistency, and the signals that require instrumentation outside Traefik itself.

Level 4: Expert

Expert-level monitoring is where you answer questions the metrics endpoint alone cannot. Some of these require configuration changes, external probes, or cross-instance comparison.

Per-router metrics. Enable addRoutersLabels: true in the Prometheus metrics configuration. It defaults to false in v3, so if you upgraded from v2 and lost router-level visibility, this is why. The cost is cardinality: one series set per router per metric. Enable it deliberately, and watch your metrics storage after you do.
Entrypoint versus service latency split. Compare traefik_entrypoint_request_duration_seconds with traefik_service_request_duration_seconds. The difference is Traefik’s own overhead: TLS, middleware chain, compression, and internal blocking. Entrypoint latency high with service latency normal means the problem is inside Traefik, not the backend. This split is also how you catch access-log buffer blocking, where log volume stalls request handling without any upstream slowness.
Per-instance config freshness comparison. In multi-replica deployments, each instance watches providers independently. One instance can silently lose provider connectivity while the others stay current, and the load balancer then spreads requests across divergent routing tables. The symptom is intermittent, unreproducible 404s or stale routes. Compare traefik_config_last_reload_success across all replicas; sustained divergence means one instance is running on stale config.
DNS resolution time for backends. In Docker and Kubernetes, backend addresses are service names. Slow DNS adds latency to every new backend connection and shows up nowhere in Traefik’s own metrics. Measure it at the resolver or node level and correlate with service latency.
Provider watcher liveness. Traefik retains last-known config when a provider dies, and there is no metric that says “the watcher is actively watching.” The proxy signal is the config reload timestamp, but at this level you also watch provider health directly: API server reachability and RBAC validity in Kubernetes, socket accessibility for Docker, cluster health for Consul or etcd.
ACME forensics. Beyond expiry countdowns: track certificate serial churn in traefik_tls_certs_not_after, alert on ACME errors in logs (there is no ACME failure metric), and monitor acme.json integrity. Corrupt acme.json is a hard startup failure and a silent mid-operation one. A stale entry for an old serial also lingers after renewal: current Traefik adds new label combinations on config updates but does not reset previously created certificate-gauge children, so validate the metric against the actually served certificate.
External synthetic TLS probes. The expiry metric tracks every cert in the store, including dormant ones, and cannot tell you which cert serves which production hostname. Synthetic probes per production hostname close that gap and are what makes a cert alert page-worthy.
Admin and API exposure checks. Probe /api/http/routers, /dashboard/, and /debug/pprof/ from outside the intended network. The API exposes your complete routing table, backend addresses, and health state. This is not a Prometheus signal; it is a recurring external check.

Moving between levels

The jump from Level 1 to Level 2 is mostly configuration: enable the metrics, build the per-service views, write the alerts. The jump from Level 2 to Level 3 is baselining: none of the Mature signals work as static thresholds, so you need enough history to know what normal looks like per service. The jump from Level 3 to Level 4 is organizational: synthetic probes, cross-instance comparison, and security checks span teams, not just the proxy.

Two traps to avoid at any level. First, do not alert on brief post-deployment 5xx spikes or startup reload bursts; backends warm up and providers push initial config, and alerting on expected noise trains people to ignore the alerts that matter. Second, do not let “healthy” health checks override error rates. If traefik_service_server_up is all 1s but the 5xx rate is high, the health check path does not exercise the real request path, and the checks are lying to you.

How Netdata helps

Netdata collects the signals this model is built on and keeps them correlated on one timeline, which matters most for the composite patterns:

Per-second collection of Traefik’s Prometheus endpoint, covering entrypoint, service, and config reload metrics alongside Go runtime metrics like goroutines, heap, and GC pauses, so retry spikes, latency, and memory growth line up in one view.
Process-level signals (process_open_fds, process_max_fds, RSS, CPU) collected from the host or cgroup, so the FD cliff and OOM trajectory are visible even where Traefik’s own metrics are thin.
ML-based anomaly detection on request and error rates, which fits the baseline-dependent signals (entrypoint request rate, per-service 5xx) better than static thresholds.
Config reload freshness and cert expiry surfaced as time-based gauges, making it straightforward to alert on a frozen traefik_config_last_reload_success or an approaching traefik_tls_certs_not_after.
System-level network state (TCP connection states like TIME_WAIT and CLOSE_WAIT) on the same host view, which supports the Mature and Expert diagnosis of connection pool and churn problems.