Most Traefik incidents are confusing because operators reason about it like nginx: a static proxy with a config file that gets reloaded on change. Traefik is not that. It is two systems sharing one process: a data-plane reverse proxy that terminates and forwards connections, and a control-plane reconciler that continuously watches configuration providers and rebuilds the routing table without a restart.

Almost every characteristic Traefik failure, from silent config drift to retry amplification to the FD cliff, follows from that dual nature. If you hold the right mental model, the runbooks make sense. If you hold the wrong one, you will keep being surprised by a proxy that reports healthy while routing to dead backends with stale routes and expiring certificates.

What it is and why it matters

Traefik is a Go-based dynamic edge router. Unlike static proxies (nginx, HAProxy) that load configuration files and reload on change, Traefik watches configuration providers (Kubernetes API, Docker socket, Consul, etcd, files) and rebuilds its routing table in real time.

This has three operational consequences:

  1. The data plane and control plane fail independently. Traefik can serve traffic perfectly on a configuration that is hours stale, because a dead provider does not stop the proxy. Conversely, a config rebuild storm can eat CPU while every route is technically correct.
  2. “Healthy” is ambiguous. Process liveness, config freshness, backend health, and certificate validity are four separate axes. Traefik’s /ping endpoint measures only the first.
  3. There is no config file to diff. The effective configuration is whatever the providers last delivered. Debugging “why is this route missing” means debugging a watch stream, not a file.

If you remember one thing: Traefik’s worst failures are the ones where every health check passes.

How it works

The request pipeline (data plane)

Every request walks the same five-stage pipeline:

  1. Entrypoints. TCP/UDP listeners bound to ports. Each entrypoint accepts connections and spawns a goroutine per connection (Go’s net/http model). TLS termination happens here. Each entrypoint tracks open connections via traefik_open_connections{entrypoint, protocol}.
  2. Routers. Rule-based matchers evaluating Host, Path, Headers, and SNI. Priority ordering decides which router wins when rules overlap. If no router matches, Traefik itself returns 404 at the entrypoint level. That 404 appears in traefik_entrypoint_requests_total{code="404"}, never in service metrics, because no service was selected.
  3. Middleware chain. Ordered request/response transformers: auth, rate limiting, compression, buffering, retries, circuit breakers. Any middleware can short-circuit the chain. There are no per-middleware Prometheus metrics; instrumentation exists only at entrypoint, router, and service level.
  4. Service / load balancer. Upstream definitions with health checks and load balancing (round-robin, weighted round-robin, sticky sessions). Each backend server is tracked via traefik_service_server_up{service, url} (0 or 1), but only when health checks are enabled. Without health checks, the series is absent, and absence means unmonitored, not healthy.
  5. Backend. The actual upstream, reached through pooled net/http transports.
flowchart LR
  client[Client] --> ep[Entrypoint
port listener]
  ep --> rt[Router
rule match]
  rt -->|no match| nf[404 at entrypoint]
  rt --> mw[Middleware chain]
  mw --> svc[Service /
load balancer]
  svc --> be[Backend server]
  prov[Providers
K8s API, Docker, file] --> agg[Config aggregator]
  agg -->|atomic swap| rt
  hc[Health checkers] -.->|mark up/down| svc

The configuration machinery (control plane)

This is the half operators underestimate:

  • Provider watchers. Background goroutines that poll or watch each configuration source. Each provider runs independently and feeds updates into an internal aggregator. Critical behavior: when a provider loses connectivity, Traefik retains its last-known configuration and retries with backoff. It does not flush routes. A provider outage causes zero immediate traffic failure but growing configuration drift: new services are invisible, removed services keep receiving traffic.
  • Configuration aggregator. Merges configuration from all providers, resolves conflicts via priority, and rebuilds the routing table. The rebuild swaps handlers behind a lock: in-flight requests finish on the old handler, new requests take the new one. There is no restart, no dropped listener, no downtime.
  • Hot-reload boundary. Only routing (dynamic) configuration is hot-reloaded: routers, middlewares, services, TLS certs. Install (static) configuration, meaning entrypoints, providers, metrics, and logging settings, is loaded once at startup. Changing an entrypoint port requires a process restart. Operators who assume everything is dynamic learn this the hard way.

There is a subtle edge case in the swap: during a rebuild, the previous router’s context can be cancelled, and any in-flight request holding a reference to that context may see a cancellation error mid-request. It shows up as rare 499/502 spikes tightly correlated with traefik_config_reloads_total increments.

One goroutine per connection

Traefik inherits Go’s concurrency model: one goroutine per active connection, plus health-check goroutines, plus provider watcher goroutines. Each proxied connection holds two file descriptors (client-side and backend-side).

Two consequences dominate production behavior:

  • Goroutine count is your best concurrency proxy. Baseline goroutines are proportional to active connections plus watchers. Monotonic growth without traffic growth means a leak, usually a backend that accepted the TCP connection and never responded, with no timeout configured to reap it. Leaked goroutines pin memory until OOM.
  • File descriptors are a cliff, not a slope. Two FDs per connection, plus provider connections, log files, and sockets. Default container ulimits are often 1024, which is catastrophically low for an edge proxy. At the limit, 100% of new connections fail instantly with zero graceful degradation. Alert on process_open_fds / process_max_fds, not on traefik_open_connections (which only counts entrypoint connections, a subset).

The two throttles nobody expects

Throttle 1: provider event throttling. Traefik does not apply every provider event immediately. It batches them behind providersThrottleDuration, default 2 seconds. During a rolling update that cycles many pods, Traefik intentionally lags up to 2s (or your configured value) behind reality. This is usually invisible, but in high-churn clusters it both protects you (fewer rebuilds) and confuses you (“the pod is ready, why is it 404?”). Raising it to 5-10s is a standard fix for config rebuild storms; lowering it makes routing converge faster at the cost of rebuild CPU.

Throttle 2: there is no global connection limit. Traefik has no built-in cap on concurrent connections or requests. Every accepted connection gets a goroutine, with no upper bound, until FDs or memory run out. The only throttling mechanisms are middlewares you opt into: InFlightReq for HTTP and InFlightConn for TCP. If you have not configured them, your only backpressure is the kernel and your ulimits. Most teams discover this during their first connection-flood incident.

The /ping trap

Traefik’s /ping endpoint returns 200 if the process is alive. It does not check provider connectivity, route validity, backend reachability, TLS certificate validity, or configuration freshness.

Two traps compound this:

  • /ping is not enabled by default. You must set ping: {} (or --ping) in static configuration. Deployments that skip this have no health endpoint at all, and traefik healthcheck fails asking you to enable it.
  • It lies by omission. A Traefik instance can return 200 on /ping while all backends are down, the provider has been disconnected for six hours, and the certificate expires tomorrow. In Kubernetes, also note the shutdown behavior: during graceful shutdown /ping returns 503 by default, which a LivenessProbe will interpret as “restart this pod” mid-drain. The terminatingStatusCode option exists to change that code.

Never use /ping as your only health signal. It answers exactly one question: “is the process running?”

Where this shows up in production

The mental model pays off when you map it to the failure archetypes:

  • Provider desync. The watcher died or lost connectivity. Traefik serves stale config happily. New services get no traffic; removed services get dead traffic. /ping returns 200. The only signals are a frozen traefik_config_last_reload_success timestamp and rising entrypoint 404s on routes that should exist.
  • Backend pool collapse. All backends fail health checks. Traefik returns 503 to every client while being perfectly healthy itself. traefik_service_server_up reads 0 for every URL in the service.
  • Retry amplification. A backend degrades, the retry middleware re-sends failed requests, doubled traffic worsens the backend, more retries follow. Traefik DDoS-es your backend while trying to help. Visible as traefik_service_retries_total spiking together with latency.
  • FD cliff. FD limit hit. Existing connections keep working; new ones fail instantly. No warning gradient.
  • Config rebuild storm. High-churn providers trigger near-continuous rebuilds. CPU goes to rebuilding routing tables instead of serving requests; you get latency jitter, not errors.
  • HA instance drift. Each replica watches providers independently. One stale replica behind a load balancer produces intermittent, unreproducible failures that depend on which instance served the request. Compare traefik_config_last_reload_success across replicas.

Common misuses of the model

These are the reasoning errors this mental model corrects:

  • “Traefik is up, so routing is fine.” Liveness, config freshness, and backend health are independent. Check all three axes.
  • “The config reloaded, so the change is live.” Static config (entrypoints, providers, logging) never hot-reloads. If you changed an entrypoint and did not restart, nothing changed.
  • “No 5xx, so the backend is healthy.” Health checks can pass on /health while the real path fails. Cross-reference traefik_service_server_up with actual 5xx rates.
  • “Retries are a safety net.” Retries multiply load on the weakest component. Monitor the retry-to-request ratio, not just final response codes.
  • “All 5xx are the same.” 502 means Traefik got garbage from the backend, 503 means all backends are marked down, 504 means the backend was too slow. Different causes, different fixes.
  • “404 is an application problem.” Entrypoint 404 means no router matched (a Traefik config issue). Service-level 404 means the backend returned it (an application issue). Different investigation paths entirely.

Signals to watch in production

The minimum set that maps to this mental model:

SignalWhy it mattersWarning sign
process_open_fds / process_max_fdsFD exhaustion is a cliff-edge failure of the goroutine-per-connection modelRatio above 80%; above 95% and rising is page-worthy
traefik_config_last_reload_successOnly window into control-plane health; frozen timestamp means provider desyncTimestamp older than ~5 min in an actively changing environment
traefik_entrypoint_requests_total{code="404"}Unmatched routes: stale config or broken route definitionsSustained rate above 5% of entrypoint traffic, or 3x baseline
traefik_service_server_upPer-backend health; all-zero means 503 for every requestAny URL at 0; all URLs at 0 for a service
traefik_service_retries_totalRetry amplification hides behind successful final responsesRetry rate above 5% of request rate, rising with latency
traefik_open_connectionsConnection load per entrypoint; leak detectionGrowth without corresponding request-rate growth
go_goroutinesDirect view of the goroutine-per-connection model; earliest leak indicatorSustained 3x baseline without traffic growth
traefik_tls_certs_not_afterOnly certificate signal; there is no ACME failure metricAny cert under 7 days (renewal has been failing for weeks)
traefik_entrypoint_request_duration_seconds vs traefik_service_request_duration_secondsThe gap isolates Traefik’s own overhead (TLS, middleware, log buffering) from backend slownessEntrypoint latency high while service latency normal

How Netdata helps

The mental model above is only useful if you can observe both halves of Traefik at once:

  • Control-plane freshness next to data-plane traffic. Netdata charts traefik_config_last_reload_success age alongside entrypoint request rates, so a frozen config timestamp is visible in the same view as the 404s it causes.
  • The three-way health split. Process liveness, backend health (traefik_service_server_up), and 5xx rates by service are correlated on one dashboard, which is exactly the distinction /ping refuses to make.
  • FD and goroutine tracking. process_open_fds vs process_max_fds and go_goroutines are charted continuously, so the cliff and the leak are visible as trends hours before they become incidents.
  • Retry amplification as a ratio. Plotting traefik_service_retries_total against request rate and latency makes the feedback loop obvious instead of hiding behind successful responses.
  • Per-second granularity during rebuild storms. Config reload events and CPU spikes correlate at per-second resolution, which is what you need to catch rebuild-induced latency jitter that 30-second scrape intervals average away.