$ guides / traefik ▌

TRAEFIK · OPERATIONS PLAYBOOK

Traefik is a proxy and a config reconciler at once — and it fails silently at the seam between them

A Go edge router that rebuilds its routing table live from providers it watches, terminates TLS, load-balances to backends it health-checks, and renews its own certificates. When a provider watch dies, a backend pool collapses, a file-descriptor limit is hit, or an ACME renewal quietly fails, Traefik itself stays up and /ping keeps returning 200. We trace how that design behaves under load and what to do when it breaks.

> Start with the monitoring checklist → # Jump to the full guide list

Traefik's dynamic configuration gets you to production in minutes, then hands you a set of failure modes that stay invisible precisely because the process never crashes.

The defaults work. Until a provider watch (the Kubernetes API, the Docker socket, Consul, a file) dies and Traefik silently keeps serving its last-known config — new services never appear, removed backends keep taking traffic, and /ping still returns 200. Until every backend for a service fails its health check and each request comes back 503 Service Unavailable. Until the file-descriptor limit — often the container default of 1024 — is hit and new connections are refused instantly with too many open files. Until an ACME renewal quietly fails for 53 days and the certificate expires under live traffic. Until a slow backend triggers retries that amplify the load and finish it off.

These guides are written for engineers who already run Traefik, not for people learning what a reverse proxy is. The goal is the mental model of how the router actually behaves under load, the failure patterns that keep recurring, the monitoring story that catches them before they page anyone, and the runbooks you wish someone had handed you before your last incident.

How Traefik actually runs in production

Traefik is not just a proxy. It is a request pipeline — entrypoint, router, middleware chain, service load-balancer, backend — fed by provider watchers that continuously rebuild the routing table, wrapped in a Go runtime that terminates TLS and manages certificates. Most production failures live between these layers, not inside any one of them.

entrypoints / clients

TCP/UDP listeners bound to ports. Each accepts connections and spawns a goroutine per connection; TLS is terminated here. Every client, backend, and provider connection costs a file descriptor. <code>traefik_open_connections</code> tracks the entrypoint count; <code>process_open_fds</code> is the real ceiling.

ENTRY

routers

Rule-based matchers evaluating Host, Path, Headers, and SNI in priority order. A request that matches no router returns a 404 at the entrypoint level (<code>traefik_entrypoint_requests_total{code="404"}</code>) — a routing fault, not a backend one. Routers are rebuilt on every configuration change.

ROUTE

middleware chain

An ordered pipeline: auth, rate limiting, compression, buffering, circuit breakers, retries. Each middleware can short-circuit the chain and each adds CPU. There are no per-middleware metrics — you infer their cost from the gap between entrypoint and service latency.

MIDDLEWARE

services / load balancer

Backend pools with weighted round-robin and health checks. <code>traefik_service_server_up</code> is 0/1 per backend — but only exists when health checks are configured. When every backend is down, the service returns <code>503</code> while Traefik itself stays perfectly healthy.

SERVICE

backends + connection pool

Go <code>net/http</code> transports pool connections to each backend. Slow or hung backends leave connections in ESTABLISHED, CLOSE_WAIT, or TIME_WAIT; poor reuse burns ephemeral ports. This layer produces most 502/504 errors — the backend, not Traefik, is the fault.

BACKEND

providers + config aggregator

Background watchers poll Kubernetes, Docker, Consul, etcd, or files and feed a configuration aggregator that atomically swaps the routing table. On disconnect Traefik retains the old config and retries — it does not flush routes. <code>traefik_config_last_reload_success</code> is the freshness signal.

PROVIDER

ACME / certificate resolver

A background process issues and renews Let's Encrypt certificates via HTTP-01, TLS-ALPN-01, or DNS-01 challenges, storing them in <code>acme.json</code> or a KV store. There is no renewal-failure metric — <code>traefik_tls_certs_not_after</code> approaching now is the only in-band signal that renewal has broken.

TLS

Go runtime (goroutines, heap, GC)

One goroutine per connection plus watchers and health checkers; the heap holds the routing table and buffers. A goroutine leak from hung backends grows <code>go_goroutines</code> and RSS in lockstep until an instant OOM kill. TLS handshakes dominate CPU.

RUNTIME

Why this matters: 'Traefik is returning errors' or 'my service isn't reachable' can come from a backend that is unreachable (502), a service with no healthy backends (503), a backend that is merely slow (504), a request that matched no router (404), a frozen provider serving stale config, an expired certificate, a file-descriptor cliff, a goroutine leak, or retry amplification. The symptom rhymes but each layer has a different signal — and a different fix.

The failures you'll actually see

Most Traefik incidents fall into a small set of recurring patterns. Recognise the shape, and triage gets dramatically faster.

CRITICAL

The backend blackout

Every backend for a service fails its health check, Traefik has nowhere to send the request, and each one comes back 503 Service Unavailable. traefik_service_server_up reads 0 for every URL in the service. Traefik itself is perfectly healthy — /ping returns 200, the process is fine — which is exactly why teams look in the wrong place first. Common after a bad deploy, a shared-dependency failure, or a health-check path that doesn't match reality.

503 Service Unavailable on every request to the service
traefik_service_server_up = 0 for all URLs in the service
Traefik process healthy, /ping returning 200
Backends may pass their own health endpoint while the app path fails

Investigate →

CRITICAL

The bad-gateway storm

Traefik reaches the backend but gets an invalid response or a reset connection, returning 502 Bad Gateway. If retry middleware is enabled, each failure is re-sent to other backends, doubling or tripling load on an already-degraded pool — Traefik effectively DDoSes the backend while trying to help. Retries and latency climb together; when retries per second exceed requests per second, a partial failure is turning into a total one.

502 Bad Gateway at the service level
traefik_service_retries_total spiking with rising latency
Backends flapping (server_up alternating 0/1)
CLOSE_WAIT sockets or ephemeral ports climbing

Investigate →

CRITICAL

The file-descriptor cliff

Traefik sits at the edge and every connection costs a file descriptor. When process_open_fds reaches process_max_fds — often the container default of 1024 — new connections are refused instantly with too many open files. Existing connections keep working, so throughput looks fine while new users are dropped. This is a cliff-edge with zero graceful degradation, and the single most common cause of 'mysterious' production outages.

accept4: too many open files in the logs
process_open_fds / process_max_fds above 95% and rising
New connections refused while existing ones continue
process_max_fds stuck at the 1024 default

Investigate →

IMMINENT

The silent config drift

A provider watch loses connectivity — the Kubernetes API becomes unreachable, the Docker socket is unmounted, an RBAC token expires — and Traefik keeps serving its last-known-good configuration. Established routes work, so nothing fires. But new services are never routed and removed backends keep receiving traffic. The blast radius grows with every deploy, and it's usually discovered hours later when a developer reports 'my new service isn't reachable'.

traefik_config_last_reload_success frozen while deploys happen
traefik_config_reloads_total flat (or reloads failing)
Entrypoint 404s rising on routes that exist in the provider
/ping healthy, existing routes unaffected

Investigate →

IMMINENT

The certificate cliff

ACME renewal has been failing silently — a blocked HTTP-01 port, a rotated DNS token, a hit rate limit, or an acme.json permission change. Traefik keeps serving the current certificate until it expires, then every HTTPS client is rejected. There is no renewal-failure metric; by the time a certificate is 7 days from expiry, renewal has been failing for roughly 53 days. The failure is invisible until users report TLS errors.

traefik_tls_certs_not_after approaching the current time
ACME errors in the logs (challenge failed, rate limited, lock)
Certificate serial not changing across days
A trickle of TLS errors from strict clients before full expiry

Investigate →

ACTIVE

Routing into the void

Requests arrive for a hostname or path that matches no router and Traefik returns a 404 at the entrypoint level — its own 404, not the backend's. Either the route was never created (a silently-ignored annotation or a wrong prefix), or the config went stale, or someone is scanning. Teams waste time investigating the application when the fault is in Traefik's routing table; the two 404s look identical in a log but live in different layers.

traefik_entrypoint_requests_total{code="404"} elevated or rising
Routes missing from /api/http/routers that exist in the provider
404s appearing in step with a deployment
Annotations or labels present but silently not applied

Investigate →

Choosing a tool

Best Traefik Monitoring Tools (10 Ranked for 2026)

A ranked review of the tools teams actually shortlist here, what each one is genuinely good at, and how the pricing behaves as you scale.

Read the buyer's guide →

Traefik monitoring maturity levels

Traefik observability works in four practical levels. Each is a complete operation, not a stepping stone. Pick the level that matches how much your edge matters. Most production deployments should land at the second level.

Level 1: Survival

Know that something is wrong

Survival monitoring is the floor. With these signals you can answer one question: is Traefik alive and passing traffic? You will not learn what broke, but you will learn that something broke before users do. Survival is enough for dev instances and non-critical edges.

Process alive / scrape reachable If the metrics target is down, the proxy is down — no traffic passes.
Entrypoint request rate > 0 traefik_entrypoint_requests_total flatlining means traffic isn't arriving or can't be accepted.
5xx / 503 presence Any 503 means a service has no healthy backends left.
File descriptor ratio process_open_fds / process_max_fds — the cliff-edge that drops new connections.
TLS certificate expiry traefik_tls_certs_not_after within 14 days means renewal has likely already failed.
Process memory (RSS) vs limit Go GC hides pressure until the OOM kill is instant.

Level 2: Operational

Diagnose most incidents on your own

Operational monitoring is what most production deployments should target. Survival tells you something is wrong; operational tells you what. With this coverage your team can usually diagnose an incident on its own: backend failures, stale config, cert expiry, resource saturation, routing gaps.

Per-service 5xx by code 502 vs 503 vs 504 have different root causes; never lump them.
Backend server up per service traefik_service_server_up = 0 across a service is an all-backends-down 503.
Per-service latency p95 / p99 Establish per-service baselines; one threshold cannot fit every backend.
Config reload freshness traefik_config_last_reload_success age is the only signal for silent provider desync.
Open connections per entrypoint Rising without traffic growth is a leak or backends backing up.
File descriptor ratio Alert at 80%, page at 95% and still rising.
Goroutine count (baseline deviation) go_goroutines climbing off traffic is the earliest leak signal.
Retry rate traefik_service_retries_total ratio to requests predicts an amplification cascade.
Entrypoint 404 rate No router matched — a routing fault, distinct from a backend 404.

Level 3: Mature

Catch problems before they become incidents

Mature monitoring catches problems before they wake anyone up. Memory trending toward OOM, connection pools degrading, retries starting to amplify, a certificate quietly failing to renew, config churn eating CPU. None of these page you on day one. They become page-out incidents on day thirty.

Go heap and GC pause times go_memstats_heap_inuse_bytes trend and go_gc_duration_seconds p99.
CPU utilisation Attribute it: TLS handshakes, compression, or config rebuilds.
Retry-to-request ratio The amplification early warning, not the raw retry count.
TLS version distribution Sustained TLS 1.0/1.1 is legacy clients or a downgrade attack.
TIME_WAIT / CLOSE_WAIT counts Ephemeral-port exhaustion and connection leaks at the OS layer.
Config reload rate Above ~1/s is a rebuild storm burning CPU on churn.
Entrypoint vs service latency The gap is Traefik's own middleware/TLS overhead.
ACME renewal tracking Alert when renewal has failed for 2 days, not at 7 days to expiry.

Level 4: Expert

Reactive instrumentation after real incidents

Expert signals enter your stack the day after a specific incident proved you needed them. Per-router cardinality, provider-watcher liveness, ACME rate-limit consumption, per-instance config drift, kernel accept-queue drops. Most teams never need every signal here. Add the ones your incident history says you do.

Per-router metrics (addRoutersLabels) High-cardinality; only where you need per-route attribution.
Provider watcher liveness A watcher can panic and stop while the process stays up — absence, not failure.
DNS resolution time for backends Slow service-name resolution adds invisible latency in K8s/Docker.
ACME rate-limit consumption Certs issued this week vs the Let's Encrypt weekly ceiling.
Per-instance config consistency Compare reload timestamps and router hashes across replicas.
Connection reuse ratio Reused vs total connections — pool efficiency and port pressure.
Dashboard / API exposure probes Is /api/rawdata reachable from outside the intended network?
acme.json integrity + kernel accept queue Storage corruption and SYN/accept-queue drops under load.

Operating mistakes worth avoiding

The traps Traefik teams keep falling into. Each has a clear, well-known fix. Most teams only learn it after an incident.

Trusting /ping as a health indicator

<code>/ping</code> returns 200 if the process is alive — period. It does not check provider connectivity, route validity, backend reachability, or certificate state. A Traefik instance can be 'healthy' by <code>/ping</code> while routing to dead backends with expired certificates on stale configuration. Always supplement liveness with metric-based signals: backend health, config freshness, and certificate expiry.

Not monitoring configuration provider health

The single most common gap. Teams watch request handling obsessively but never check whether the provider is still delivering updates. When the Docker socket goes away or the Kubernetes API becomes unreachable, Traefik serves stale config for hours and <code>/ping</code> stays green. Alert on <code>traefik_config_last_reload_success</code> age — and remember a dead watcher stops incrementing <code>traefik_config_reloads_total</code>, so absence, not failure, is the signal.

Running with the default file descriptor limit

Traefik handles every incoming connection, yet containers often inherit the default 1024 FD limit. That is fine for staging and catastrophic under production load — the failure is a sudden cliff of <code>too many open files</code> with new connections refused and existing ones fine, which is why it's so often misdiagnosed. Raise both the OS <code>ulimit</code> and the container/systemd limit to 65536 or higher.

Enabling retries without watching the retry rate

Retry middleware masks backend instability — a request that failed twice and succeeded on the third try shows as 'success' — while quietly tripling backend load. When a backend is already degrading, that extra load pushes it over. Make the ratio of <code>traefik_service_retries_total</code> to request rate a top-level metric, and know that non-idempotent retries (POST, PUT) can cause duplicate side effects.

Lumping 502, 503, and 504 together as '5xx'

A 502 means Traefik reached the backend but got garbage; a 503 means no backends are available at all; a 504 means the backend was reachable but too slow. They have completely different root causes and fixes. Aggregating them as 'errors' leads straight to the wrong diagnosis — and comparing entrypoint-level with service-level 5xx tells you whether Traefik or the backend generated them.

Checking certificate expiry but not renewal health

Most teams alert when a certificate is 7 days from expiry. By then ACME renewal has been failing for roughly 53 days (a 90-day Let's Encrypt cert renews at 30 days out). Alert when renewal has failed for two consecutive days, not when expiry is imminent — and pair the <code>traefik_tls_certs_not_after</code> metric with an external synthetic probe of the real production hostname, since the metric tracks dormant and staging certs too.

Confusing entrypoint 404s with backend 404s

An entrypoint-level 404 means 'no router matched' — a Traefik configuration fault. A service-level 404 means 'the backend returned 404' — an application concern. They look identical in a raw log but require completely different investigation paths. Teams that don't split <code>traefik_entrypoint_requests_total{code="404"}</code> from service 404s waste an incident looking at the wrong layer.

Uncoordinated timeouts across the stack

The timeout chain client → load balancer → Traefik → backend must have each layer's timeout longer than the one behind it. The classic break is a cloud LB idle timeout longer than Traefik's <code>respondingTimeouts.idleTimeout</code>: Traefik closes an idle keep-alive connection the LB still reuses, producing intermittent 502s that are notoriously hard to reproduce. Coordinate <code>respondingTimeouts</code>, <code>forwardingTimeouts</code>, and transport settings with the rest of the stack.

Traefik runbooks in this section

Each guide is a focused runbook for one symptom or topic. Pick one when you have an incident, or use the categories to learn the area.

▸

Start here

▸

Backend errors and health

▸

Routing, providers, and stale config

▸

Retries and resilience

▸

Connections, file descriptors, and ports

▸

TLS certificates and ACME

▸

Go runtime: memory, goroutines, CPU

▸

Latency and performance

▸

Configuration reloads and churn

▸

HA, load balancers, and timeouts

▸

Security: exposure, auth, and TLS compliance

▸

Traffic, throughput, and capacity

WHERE TO GO NEXT

Setting up Traefik monitoring, or putting out a fire?

If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, jump straight to the symptom that matches what you're seeing.

> Start with the checklist > Back to Operations Guides