Traefik Operations Guides on Netdata

How Traefik actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

Most Traefik incidents are confusing because operators reason about it like nginx: a static proxy with a config file that gets reloaded on change. Traefik is not that. It is two systems sharing one process: a data-plane reverse proxy that terminates and forwards connections, and a control-plane reconciler that continuously watches configuration providers and rebuilds the routing table without a restart.

Almost every characteristic Traefik failure, from silent config drift to retry amplification to the FD cliff, follows from that dual nature. If you hold the right mental model, the runbooks make sense. If you hold the wrong one, you will keep being surprised by a proxy that reports healthy while routing to dead backends with stale routes and expiring certificates.

Traefik /ping returns 200 while everything is broken: the health-check trap

Mon, 01 Jan 0001 00:00:00 +0000

Traefik’s /ping endpoint answers one narrow question: “Is the Traefik process alive enough to answer this request?” It does not answer “Can Traefik correctly route production traffic?”

That distinction matters because Traefik is both a data-plane proxy and a control-plane configuration reconciler. The process can stay alive while its Docker socket or Kubernetes API watch has failed, its routing table is stale, every backend is failing health checks, or a certificate is approaching expiry. In all of those states, /ping can still return 200 OK.

Traefik 401/403 spike: auth failures, ForwardAuth outages, and brute force

Mon, 01 Jan 0001 00:00:00 +0000

Your dashboard shows traefik_service_requests_total{code="401"} climbing steeply, or the entrypoint-level 4xx panel has turned red. A 401/403 surge is one of the most ambiguous signals a reverse proxy produces, because the same symptom maps to three different situations: your auth layer correctly rejecting bad credentials, your auth layer incorrectly rejecting everyone, or an attacker hammering a login endpoint.

The trap: a ForwardAuth service that is up but degraded (session store exhausted, internal rate limit hit, broken ACL rule) rejects all requests with 401/403. To Traefik this looks like “auth working as designed.” To your users it is a full outage. Conversely, a credential-stuffing run produces the same metric shape but demands the opposite response: block, don’t fix.

Traefik 404 not found: requests arriving with no matching router

Mon, 01 Jan 0001 00:00:00 +0000

A client reports that your service returns “404 page not found”. The backend is healthy, its logs show zero traffic, and the 404 body is a bare 19-byte plain-text response, not your application’s error page. That 404 did not come from your backend. It came from Traefik itself.

When a request arrives at a Traefik entrypoint and matches no router, Traefik answers it directly with a 404. The request never touches a service, never runs through a middleware chain, and never reaches an upstream. This is a routing-layer problem with a completely different investigation path from a backend-originated 404, and mixing them up is one of the most common time-wasters in Traefik operations.

Traefik 502 Bad Gateway: when the backend is unreachable or returns garbage

Mon, 01 Jan 0001 00:00:00 +0000

A 502 from Traefik means the request got past routing and middleware, Traefik selected a service and tried to talk to a backend, and the backend hop failed. Traefik connected (or tried to connect) and got an invalid response, a reset connection, or nothing usable back. The usual mechanics: the backend crashed mid-response, sent something that is not valid HTTP for the negotiated protocol, or closed a connection Traefik wanted to reuse.

Traefik 503 Service Unavailable: no healthy backends left in the pool

Mon, 01 Jan 0001 00:00:00 +0000

Traefik is returning 503 Service Unavailable for every request to a service. Clients are down. Traefik itself looks fine: the process is up, /ping returns 200, other services route normally. This is the backend pool collapse failure mode: Traefik matched a router to a service, but every backend in that service’s load balancer pool has been marked down by health checks, so Traefik has nowhere to send the request.

A Traefik 503 is not a Traefik failure. It is Traefik correctly reporting that its upstream pool is empty. The investigation belongs one hop upstream, at the backends or at the health check configuration itself.

Traefik 504 Gateway Timeout: the backend is alive but too slow

Mon, 01 Jan 0001 00:00:00 +0000

Your Traefik instance is up. /ping returns 200. The backend service is running, its health checks pass, and clients are getting 504 Gateway Timeout. The connection to the backend was established, but the response never arrived within the configured timeout.

A 504 in Traefik means: the backend exists, Traefik reached it, the TCP connection (and usually the request) succeeded, but the response took too long. That makes a 504 fundamentally different from a 502 (invalid response or connection error from the backend) or a 503 (no healthy backends at all). Lumping all 5xx together leads directly to investigating the wrong layer.

Traefik 5xx error rate: telling Traefik-generated errors from backend errors

Mon, 01 Jan 0001 00:00:00 +0000

Your edge dashboard shows a 5xx spike on Traefik. Someone pages the backend team. The backend team finds nothing wrong in their application logs. An hour later it turns out Traefik had no healthy backends for one service, or a route disappeared after a config change, and the application was never in the request path at all.

Treating all 5xx responses as backend failures is one of the most common time-wasters in Traefik operations. A 502 generated by Traefik because it could not reach an upstream is a different incident from a 502 the backend produced itself. A 503 because every backend failed health checks is a different incident from a 503 your application returns under load. The root causes, the owners, and the fixes are all different.

Traefik access log blocking: when logging stalls request handling

Mon, 01 Jan 0001 00:00:00 +0000

Latency spikes across every route on a Traefik instance. Backends report nothing wrong. Service-level latency looks normal, but entrypoint latency is elevated. Health checks pass. If you see this combination and access logging is enabled at high verbosity or high request rate, the log writer is a prime suspect.

Traefik’s access log sits in the request path. With the default configuration (bufferingSize: 0), the access log line for a request is written synchronously from the request-handling goroutine before the request fully completes. When log volume is extreme (thousands of requests per second with verbose field selection), or when the write destination stops draining (full filesystem, blocked pipe), the log write stalls and the goroutine handling that request stalls with it. Requests do not fail; they get slow. That distinction is what makes this failure mode easy to miss.

Traefik ACME challenge failed: HTTP-01, DNS-01, and TLS-ALPN-01 renewal errors

Mon, 01 Jan 0001 00:00:00 +0000

Your certificates are approaching expiry and Traefik’s logs contain lines like “unable to obtain ACME certificate” or “Error renewing certificate”. HTTPS still works for now because Traefik keeps serving the old certificate, but the clock is running: Let’s Encrypt certificates live 90 days, Traefik attempts renewal 30 days before expiry, and if you are seeing renewal errors today, they have likely been failing silently for weeks.

The uncomfortable part: Traefik exposes no Prometheus metric for ACME renewal failures. The only metric-adjacent signal is traefik_tls_certs_not_after counting down toward expiry. The actual failure details exist only in the logs. That means this failure mode is invisible until someone either reads the logs, alerts on certificate expiry, or gets a browser warning.

Traefik ACME lock contention: stuck distributed locks blocking renewal

Mon, 01 Jan 0001 00:00:00 +0000

Every Traefik instance in your HA deployment is healthy. /ping returns 200, traffic flows, backends are up. Yet traefik_tls_certs_not_after keeps creeping toward the current time, and the logs on each instance repeat some variant of “unable to acquire lock”. Certificates are drifting toward expiry and nobody is renewing them.

This is the multi-instance ACME failure mode: Traefik uses a distributed lock in a KV store (Consul or etcd) so that exactly one instance performs ACME issuance and renewal at a time. When the lock holder crashes mid-renewal, is force-killed, or loses its session to a network partition, the lock can be left behind. Every remaining instance refuses to renew because the lock appears held, and the cluster slides toward certificate expiry while looking healthy on every conventional health signal.

Traefik ACME rate limit: too many certificates already issued for this domain

Mon, 01 Jan 0001 00:00:00 +0000

You find it in the Traefik log, usually right after a deploy, a restart storm, or a cluster rebuild: an ACME order rejected with an HTTP 429 and the message “too many certificates already issued” for your domain. Traefik cannot get a new certificate, and it will keep retrying and keep getting rejected.

Two very different situations sit behind this error. If your existing certificates are still valid and stored in acme.json, traffic keeps flowing and you have days or weeks to fix the renewal path before expiry. If acme.json was lost along with the certificates, you are rate limited and serving nothing valid on HTTPS, which is an outage on a timer.

Traefik acme.json permissions and corruption: renewal silently blocked

Mon, 01 Jan 0001 00:00:00 +0000

Your certificates are approaching expiry and Traefik has not renewed them. There is no alert, no error metric, and depending on how the failure happened, possibly nothing useful in the logs either. The root cause in a large share of these incidents is the ACME storage file itself: acme.json has the wrong permissions, or it has been corrupted by a partial write, a restore, or an out-of-memory event during a write.

Traefik backend connection pool: keep-alive, MaxIdleConnsPerHost, and reuse

Mon, 01 Jan 0001 00:00:00 +0000

Every request Traefik proxies needs a TCP connection to a backend. Whether that connection is freshly dialed or reused from a pool determines how much per-request overhead you pay, and it is one of the least monitored parts of the proxy. A misconfigured pool shows up as elevated latency on every request (TCP and possibly TLS handshake costs per request), as growing TIME_WAIT socket counts marching toward ephemeral port exhaustion, or as sporadic 502s with no backend outage to explain them.

Traefik cannot assign requested address: ephemeral port exhaustion

Mon, 01 Jan 0001 00:00:00 +0000

Traefik starts returning 502 Bad Gateway for some or all proxied requests. Backend health checks are green. Backends respond fine when you hit them directly. The Traefik logs show the real error: dial tcp <backend-ip>:<port>: connect: cannot assign requested address.

This is ephemeral port exhaustion: the kernel has no free local ports left for Traefik to open a new outbound TCP connection to a backend. Almost every port in the ephemeral range is pinned by a socket in TIME_WAIT, left behind by a connection that closed up to a minute ago.

Traefik cascading backend failure: how a partial outage becomes a total one

Mon, 01 Jan 0001 00:00:00 +0000

A backend pool behind Traefik rarely fails all at once. It fails one server at a time, and each failure makes the next one more likely. Two of your eight backends go down, the remaining six absorb the redistributed traffic plus the retry traffic Traefik generates, one of the six starts timing out under the extra load, and now five are carrying the full weight. Within minutes the pool is empty and Traefik returns 503 to every client, even though Traefik itself is perfectly healthy.

Traefik certificate expired: when ACME renewal has been failing silently

Mon, 01 Jan 0001 00:00:00 +0000

Browsers are rejecting your site with a certificate error, and Traefik is the TLS terminator. The process is up, /ping returns 200, traffic flows on port 80, and every HTTPS client reports an expired or expiring certificate. This is the signature of silent ACME renewal failure.

The nasty part is the timeline. Traefik attempts renewal 30 days before expiry. For a 90-day Let’s Encrypt certificate, that means renewal has been attempted since day 60. If a certificate is 7 days from expiry, renewal has already been failing for roughly 53 days. There is no Prometheus metric for ACME failure. The only metric you get is the consequence: traefik_tls_certs_not_after sliding toward the current time while nothing alerts.

Traefik circuit breaker: shedding load from a failing backend

Mon, 01 Jan 0001 00:00:00 +0000

A backend is degrading. It is not fully down, so health checks still pass or flap, and Traefik keeps sending it full traffic. If you also have the retry middleware attached, every failed request comes back two or three more times, and the retry amplification loop finishes what the original failure started. The circuit breaker middleware exists for exactly this situation: it watches error and latency ratios on a router, and when they cross a threshold you define, it stops forwarding requests and answers with a fast 503 until the backend recovers.

Traefik CLOSE_WAIT pile-up: backend connections that never close

Mon, 01 Jan 0001 00:00:00 +0000

You run ss -s or glance at process_open_fds and notice the Traefik process is holding thousands of sockets in CLOSE_WAIT. Traffic is still flowing, health checks pass, and /ping returns 200, but the count grows every hour. Left alone, the leak consumes file descriptors until the process hits its limit, at which point every new connection fails and clients start seeing 502s.

CLOSE_WAIT means the remote side sent a FIN and the kernel acknowledged it, but the local application never called close() on the socket. In Traefik’s case, the backend (or an intermediary) closed its end of a backend connection, and Traefik’s side is still open, holding a file descriptor. There is no kernel timer that reaps CLOSE_WAIT. These sockets persist until the process closes them or dies.

Traefik config last reload success: monitoring configuration freshness

Mon, 01 Jan 0001 00:00:00 +0000

Traefik’s most dangerous failure mode is the one that looks like nothing. The process is up, /ping returns 200, existing routes keep serving traffic, and every dashboard is green. Meanwhile the configuration provider disconnected two hours ago, the three services you deployed since then are invisible, and the service you scaled down is still getting traffic at dead addresses. This is provider desync, and the only Traefik-native signal that exposes it is configuration freshness.

Traefik configuration reload storm: provider churn eating CPU

Mon, 01 Jan 0001 00:00:00 +0000

Traefik’s CPU is pinned, request latency is jittery, GC pauses are climbing, and traffic volume looks completely normal. The culprit is not the traffic. It is the control plane: rate(traefik_config_reloads_total[5m]) is sitting above 1 reload per second, and every reload rebuilds the entire routing table.

In busy Kubernetes and Docker environments, every pod event (creation, deletion, readiness change) can trigger a configuration rebuild. At high churn rates Traefik spends more CPU rebuilding the routing table than it spends routing requests. This is most visible during cluster-wide rollouts and autoscaler events, but it also appears when health checks flap and pods re-register in a loop.

Traefik context canceled during reload: intermittent errors on config change

Mon, 01 Jan 0001 00:00:00 +0000

You are seeing small bursts of failed requests: a handful of 499s or 502s, lasting a second or two, then nothing. The backends are healthy, health checks pass, latency is normal, and the errors do not repeat on any schedule. When you overlay the error timestamps on Traefik’s metrics, each burst lines up exactly with an increment of traefik_config_reloads_total.

During a dynamic configuration rebuild, Traefik’s cancelPrevState() cancels the context attached to the previous router configuration. In-flight requests, middleware chains, or backend connections that still hold a reference to that context receive a context-canceled error mid-request and fail. It is rare, intermittent, and almost impossible to reproduce on demand, which is why it gets misdiagnosed as flaky backends or network blips.

Traefik dashboard and API exposed: your routing table on the public internet

Mon, 01 Jan 0001 00:00:00 +0000

If Traefik’s dashboard, API endpoints (/api/rawdata, /api/http/services, /api/http/routers), or /debug/pprof answer requests from the public internet, you are publishing a map of your infrastructure. The API returns every router rule, backend server URL, middleware configuration, and upstream health status. /debug/pprof adds Go runtime profiling data. Automated scanners fingerprint Traefik continuously, and the dashboard API has a history of information-disclosure issues. The fix is usually a ten-minute configuration change, but only if you know the exposure exists. Many teams never test from outside their own network.

Traefik dashboard returns 404: reaching the API and dashboard correctly

Mon, 01 Jan 0001 00:00:00 +0000

You opened a browser to what you think is the dashboard address and got a bare “404 page not found”. Or the dashboard HTML loads but every panel is empty because the underlying /api calls all return 404. This almost always comes down to one of three things: the API is not enabled, you are hitting the wrong port or path, or secure mode is on but no router was ever defined for the internal API service.

Traefik entrypoint vs service latency: measuring middleware overhead

Mon, 01 Jan 0001 00:00:00 +0000

A client reports that requests through Traefik are slow. You check traefik_service_request_duration_seconds and the backends look fine: p95 is 80ms, well inside the SLO. Yet clients measure 400ms end to end. The missing time is being spent inside Traefik itself, and the service-level metric cannot see it.

Traefik exposes request duration at two points in its pipeline: at the entrypoint, where the request arrives from the client, and at the service, where the request is handed to a backend. The difference between the two is Traefik’s own overhead: TLS termination, router matching, the middleware chain, compression, and request/response buffering. Because Traefik ships no per-middleware Prometheus metrics, this subtraction is the only metrics-based way to localise middleware cost without capturing Go profiles or traces.

Traefik file descriptor monitoring: process_open_fds, limits, and headroom

Mon, 01 Jan 0001 00:00:00 +0000

Traefik file descriptor exhaustion is a cliff-edge failure. The proxy works until it hits its OS limit on open files, then 100% of new connections fail instantly with “too many open files” errors. Existing connections keep working, so dashboards can look calm while every new client is dropped. There is no graceful degradation.

The most obvious Traefik connection metric, traefik_open_connections, does not measure the thing that kills you. It tracks only entrypoint connections, a subset of total FD usage. The comprehensive signal is the ratio process_open_fds / process_max_fds from the Go process collector, which counts everything the process holds open: client sockets, backend sockets, provider connections, log files, ACME storage, and pipes.

Traefik GC pauses: when Go garbage collection shows up in request latency

Mon, 01 Jan 0001 00:00:00 +0000

Your p99 latency graph has periodic spikes that do not line up with traffic, deployments, or backend slowness. The spikes hit every service at once, last tens to hundreds of milliseconds, and then everything goes back to normal. When a latency event affects all concurrent requests simultaneously and leaves backends untouched, the suspect list gets short, and Go garbage collection is near the top of it.

Traefik is a Go process. Every request it proxies runs on goroutines inside one shared heap, and when the runtime stops the world to collect garbage, every in-flight request waits. Most of the time these pauses are sub-millisecond and invisible. Under high heap size or a high allocation rate, they become long enough to show up in your request duration histograms.

Traefik goroutine leak: go_goroutines climbing toward OOM

Mon, 01 Jan 0001 00:00:00 +0000

Your Traefik dashboard shows go_goroutines climbing steadily over hours or days. Traffic is flat. Latency and error rates look normal. Yet the goroutine count keeps rising, RSS grows in lockstep, and you can extrapolate a straight line from today’s memory usage to the container limit and read off the day Traefik gets OOM-killed, dropping every in-flight connection with it.

In Traefik this almost always has the same shape: a backend accepts the TCP connection but never sends response headers, and the forwarding transport has no timeout (or a very long one) on that wait. Each stuck request leaves a goroutine blocked indefinitely. The goroutine count is the earliest visible signal, long before users feel anything.

Traefik HA config drift: replicas serving different routing tables

Mon, 01 Jan 0001 00:00:00 +0000

The symptom: a route works, then it doesn’t, then it does. Refreshing the page flips between a working service and a 404 or 502. Testing from one network works; from another it fails. Behind this randomness is a simple mechanism: you run multiple Traefik replicas for availability, one has gone stale, and the load balancer in front of them sends your requests to whichever replica it picks.

Each Traefik replica watches its configuration provider (Kubernetes API, Docker socket, Consul, file) independently. There is no shared state between replicas. If one replica loses provider connectivity, through an RBAC change, a network partition, or a dead Docker socket, it keeps serving its last-known routing table. It does not flush routes, it does not crash, and /ping still returns 200. The other replicas update normally. Your fleet is now serving two different views of the world, and the failure only appears when the load balancer picks the stale instance.

Traefik health checks pass but requests fail: when the probe lies

Mon, 01 Jan 0001 00:00:00 +0000

Every backend for the service shows traefik_service_server_up = 1. The dashboard is green. And clients are getting 502s and 504s on real requests. Traefik is not malfunctioning: it is faithfully reporting the results of the probe you configured. The problem is that the probe and the production traffic path are testing different things.

This failure mode inverts your usual triage instinct. Normally server_up = 1 means “rule out the backend.” Here it means nothing of the sort: the health check answered a question, just not the one your users are asking. Three distinct mechanisms produce this state, and they have different fixes.

Traefik high CPU usage: TLS handshakes, regex rules, and config rebuilds

Mon, 01 Jan 0001 00:00:00 +0000

Traefik is pegging cores and request latency is climbing. The process is alive, /ping returns 200, and traffic is still flowing, but p99 latency is creeping up and TLS connections are slower to establish. You need to know which of the three usual suspects is burning the CPU: cryptographic work, middleware processing, or configuration rebuilds.

Traefik CPU is dominated by TLS handshake computation (especially with RSA certificates), then middleware chain processing (gzip compression, JWT validation, regex-based routing rules), then configuration rebuilds whose cost scales with routers times middlewares times services. The diagnosis is almost entirely correlation: match the CPU curve against the TLS handshake rate, the config reload rate, and the request rate, and the cause usually names itself.

Traefik high request latency: isolating Traefik overhead from backend slowness

Mon, 01 Jan 0001 00:00:00 +0000

Latency dashboards are red, p95 is climbing, and the first question from the incident channel is “is it the proxy or the app?” With Traefik in the path, that question has a precise answer if you compare the right two metrics and read the JSON access-log fields most operators skip.

Traefik measures request duration at two points in its pipeline. traefik_entrypoint_request_duration_seconds measures the request at the edge, including TLS, middleware processing, and backend time. traefik_service_request_duration_seconds measures only the leg from Traefik to the backend and back. The difference between the two is Traefik’s own overhead. If entrypoint latency is high but service latency is normal, the time is being burned inside Traefik itself: TLS handshakes, middleware chain processing, gzip compression, response buffering, or an access-log write that is blocking request goroutines. If both are high, the backend is slow and Traefik is just the messenger.

Traefik intermittent 502 behind a load balancer: the timeout chain mismatch

Mon, 01 Jan 0001 00:00:00 +0000

Some percentage of requests behind your cloud load balancer return 502. Not all of them, just a low-grade drip: one in a few hundred, sometimes one in a few thousand. Backend logs show nothing. Traefik logs show a burst of 502s with no corresponding backend error. Load balancer health checks are green. Everything looks healthy, and users keep hitting errors.

This is the classic signature of a keep-alive idle timeout mismatch in the chain client -> load balancer -> Traefik -> backend. Somewhere in that chain, a component is reusing a connection that the next hop has already closed. The request is written into a half-dead socket, the other side answers with a TCP reset, and the proxy reports 502 Bad Gateway. Because it only happens when a request lands on an idle connection at the wrong moment, it is intermittent, load-dependent, and hard to reproduce.

Traefik memory usage growing: routing table size and heap trends

Mon, 01 Jan 0001 00:00:00 +0000

process_resident_memory_bytes on your Traefik instance has been climbing for days. Request rates look normal, error rates are flat, and /ping returns 200. The question is whether the proxy legitimately needs more memory because its routing table grew, or whether it is leaking and will get OOM-killed at 3 a.m. with zero warning.

The failure mode is asymmetric. Go’s garbage collector absorbs growing allocation pressure gracefully, running more often and burning more CPU, until it cannot. Then the kernel OOM killer terminates the process instantly, dropping every connection in flight. There is no graceful degradation phase.

Traefik monitoring checklist: the signals every production edge router needs

Mon, 01 Jan 0001 00:00:00 +0000

Traefik fails in ways that static proxies do not. It is both a data-plane proxy and a control-plane configuration reconciler, and its worst failure modes are silent: a dead provider connection, a stalled ACME renewal, a file descriptor limit creeping toward exhaustion. The process stays up, /ping returns 200, and traffic keeps flowing on stale configuration while new deployments get 404s.

This checklist organizes the signals that matter into four maturity levels: survival, operational, mature, and expert. Each level builds on the previous one. Every item names the exact metric, why it matters, and the severity it deserves. Use it to audit an existing monitoring setup or to build one from scratch.

Traefik monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Most Traefik outages are not exotic monitoring gaps. They are the same six failures repeating across teams: the process died, the FD limit was 1024, the cert expired, the provider went silent, the backend pool collapsed, or retries amplified a partial failure into a total one. A maturity model helps because it sequences coverage against the failures you are actually going to have, in the order you are going to have them.

Traefik open connections growing: spotting a connection leak

Mon, 01 Jan 0001 00:00:00 +0000

The traefik_open_connections gauge is climbing. Request rate is flat. Every hour the number is higher than the last, and it never comes back down, even after the traffic peak passes. That divergence is the signature of a connection leak: connections are opened and never closed, and each one holds a file descriptor, a goroutine, and some memory.

Left alone, this ends one way. Traefik hits its file descriptor limit and stops accepting new connections instantly. Existing connections keep working, so some health checks still pass, but every new client is refused. It is a cliff-edge failure with no graceful degradation, and the climb gives you hours or days of warning if you are watching the right signal.

Traefik out of memory: OOM kills and the crash loop that follows

Mon, 01 Jan 0001 00:00:00 +0000

Traefik was proxying traffic normally. Then the process vanished. In Kubernetes you see OOMKilled in the pod’s last terminated state and a restart count that keeps climbing. On bare metal or Docker you see exit code 137 and a container that came back up, ran for a while, and died again.

An OOM kill has no graceful degradation phase. Go’s garbage collector absorbs growing memory pressure by running more often, right up until the moment it cannot. Then the kernel OOM killer terminates the process instantly. Every in-flight connection drops at once: client connections, backend connections, WebSockets, gRPC streams. From the outside it looks like a total, simultaneous outage of everything behind the proxy.

Traefik request rate monitoring: entrypoint and service throughput baselines

Mon, 01 Jan 0001 00:00:00 +0000

Traefik exposes two request counters that look similar but measure different things. traefik_entrypoint_requests_total counts every request that arrives at a listener, whether or not Traefik finds a route for it. traefik_service_requests_total counts only requests that matched a router, survived the middleware chain, and were proxied to a backend. The difference between the two is where most routing and middleware problems show up first.

Static thresholds (“page if requests/sec drops below 50”) fail on a reverse proxy fronting heterogeneous workloads. A static asset service and a compute-heavy API differ by orders of magnitude, and the same entrypoint differs by time of day. This article covers what each counter counts, how to read the gap between them, and how to build per-entrypoint, per-service, per-time-of-day baselines that catch real faults without paging on legitimate traffic dips.

Traefik retry amplification: how retries turn a slow backend into an outage

Mon, 01 Jan 0001 00:00:00 +0000

A backend service starts degrading. Responses get slow, some connections fail. Traefik’s retry middleware re-sends the failed requests, usually to other backends in the pool. Backend load doubles or triples. The struggling service falls further behind, which produces more failures, which produces more retries. Within minutes, a partial degradation becomes a complete outage, and Traefik is doing a significant share of the damage while trying to help.

The worst part: this loop can be invisible on the dashboards your team actually watches. If a request fails twice but succeeds on the third attempt, the client got a 200. Your success-rate graphs look fine while the backend absorbs 3x the real client traffic and slides toward collapse.

Traefik route not loading: silently ignored annotations and labels

Mon, 01 Jan 0001 00:00:00 +0000

You deployed a service, added the Traefik annotations or labels, and the route does not work. Requests to the hostname return 404. Traefik is up, other routes work, the provider is connected, and there is no error anywhere in the logs. The service simply does not exist as far as Traefik is concerned.

This is silent rejection: Traefik ignores any annotation or label key it does not recognize. A wrong prefix, a misspelled key, a label at the wrong YAML level, or a value mangled by YAML parsing all produce the same outcome: the route is never created, no error metric is emitted, and nothing is logged at the default log level. The configuration you wrote and the configuration Traefik loaded diverge without either side complaining.

Traefik router priority conflicts: when overlapping rules match the wrong service

Mon, 01 Jan 0001 00:00:00 +0000

Traffic for api.example.com is landing on your generic dashboard service instead of the API backend. Or you deployed a new router with what looks like a correct rule, and it never matches a single request. Traefik is up, /ping returns 200, there are no errors in the logs, and the config reload metric keeps advancing. Everything looks healthy except the routing decision itself.

This is almost always a router priority conflict. When two routers have overlapping rules, Traefik does not pick the most specific match by semantics. It sorts routers by a numeric priority, and the first one whose rule matches wins. The default priority is the length of the rule string, not the specificity of the rule. A longer but broader rule will shadow a shorter, more specific one.

Traefik routing table explosion: thousands of routes and slow rebuilds

Mon, 01 Jan 0001 00:00:00 +0000

Your Traefik instance is still passing traffic, but something is off. Memory climbs week over week. Every Ingress change causes a CPU blip and a small latency spike. Restart counts are creeping up, and the last restart was an OOM kill. When you look at the provider, you find thousands of Ingress or IngressRoute objects that nobody ever pruned.

This is the routing table explosion failure mode. Traefik rebuilds its entire routing table on every configuration change, holds the whole thing in memory, and pays a per-request matching cost for every router it knows about. The failure is slow and linear right up until the OOM killer makes it instant.

Traefik service retries climbing: reading the retry signal before it cascades

Mon, 01 Jan 0001 00:00:00 +0000

traefik_service_retries_total is climbing for one of your services. Your dashboards show green: clients are getting 200s, the error rate looks flat, and /ping is happy. That is exactly what makes this signal dangerous. Retries mask backend instability from clients while multiplying the load Traefik sends to the backends. By the time client-facing errors appear, the amplification loop may already be running.

This guide covers how to read the retry signal, how to tell a harmless burst (rolling update, instance flapping) from the early stage of a cascade, and what to do before the loop feeds itself.

Traefik service server up at zero: backend health checks are failing

Mon, 01 Jan 0001 00:00:00 +0000

traefik_service_server_up is 0 for one or more URLs in a service. Traefik’s health checker has pulled those backends out of the load balancer rotation. If every URL in the service is at 0, Traefik has nowhere to send requests and is returning 503 to clients right now.

This is a backend-side symptom, not a Traefik-side one. Traefik is doing what you configured it to do: stop sending traffic to servers that fail its checks. The useful question is whether the checks are telling the truth about the backends, and if so, why the backends are failing.

Traefik serving stale configuration: the silent provider desync

Mon, 01 Jan 0001 00:00:00 +0000

A developer deploys a new service. The pods are running, the Ingress or IngressRoute exists, everything looks right in Kubernetes. But the service returns 404 from the edge. Meanwhile, a service deleted an hour ago is still receiving traffic and failing. You check Traefik: process is up, /ping returns 200, existing routes work. Nothing is alerting.

This is the silent provider desync, and it is the most commonly missed Traefik failure mode. The provider watch (Kubernetes API, Docker socket, Consul, file) has died or fallen behind. Traefik does not flush its routes when this happens. It keeps the last-known-good configuration and retries the provider connection in the background, with no loud failure signal. The routing table freezes at the moment of disconnection.

Traefik serving the default certificate: SNI and certificate selection

Mon, 01 Jan 0001 00:00:00 +0000

A client hits your site and gets a certificate warning. The certificate presented is a self-signed cert with CN “TRAEFIK DEFAULT CERT” instead of the certificate you configured. Everything else looks fine: Traefik is up, /ping returns 200, backends are healthy, routes work. The proxy is completely healthy and the only broken thing is which certificate it picked during the TLS handshake.

This failure is invisible to most Traefik monitoring. There is no error counter, no 5xx, no failed health check. The TLS handshake succeeds; it just succeeds with the wrong certificate. Unless you probe TLS externally, you find out from a user’s screenshot.

Traefik TLS certificate expiry monitoring: reading traefik_tls_certs_not_after

Mon, 01 Jan 0001 00:00:00 +0000

traefik_tls_certs_not_after is the only signal Traefik gives you about certificate expiry. There is no metric for ACME renewal failures, no counter for rate limit rejections, no gauge for “the challenge did not complete.” You get a Unix timestamp per certificate, and everything else has to be inferred from it or pulled from logs.

Most teams wire it to a pager with a threshold like “alert at 7 days.” Then the first ACME renewal happens, the old certificate series stays in the output, and the pager fires for a certificate that is no longer serving anything. Or the alert fires for a staging certificate, or for a dormant cert left in the store from a decommissioned hostname, and the on-call learns to ignore it. This guide covers reading the metric correctly and building an alert chain that only wakes someone up when a certificate that is actually serving production traffic is about to expire.

Traefik TLS handshake latency: RSA cost, session resumption, and CPU

Mon, 01 Jan 0001 00:00:00 +0000

Clients report slow first page loads or high connect times, but backend latency looks fine and error rates are flat. When the delay shows up before the first byte of the request is processed, the TLS handshake at the Traefik entrypoint is the prime suspect.

Handshakes are the largest per-connection CPU cost for a terminating proxy, and they add latency to every new connection before any routing, middleware, or backend work happens. Traefik exposes no dedicated handshake-latency metric, so you have to infer it from latency decomposition and process CPU.

Traefik TLS version monitoring: catching TLS 1.0/1.1 legacy traffic

Mon, 01 Jan 0001 00:00:00 +0000

TLS 1.0 and 1.1 have been deprecated for years, and compliance regimes like PCI DSS require TLS 1.2 as a minimum. The risky part of setting a minimum TLS version is not the config change. It is finding out, before you enforce the floor, which clients still negotiate old protocol versions. Flip minVersion blind and you will discover the legacy client population from user complaints.

Traefik already counts every request by negotiated TLS version and cipher suite. That counter is the inventory tool you need: watch it, identify the holdouts, remediate or accept them, then enforce the floor with confidence.

Traefik too many open files: file descriptor exhaustion at the edge

Mon, 01 Jan 0001 00:00:00 +0000

Your Traefik logs are filling with accept4: too many open files errors. New clients cannot connect. Requests that do get through may return 502 or 503. Meanwhile, some clients insist everything works fine, because their existing connections are still alive.

This is file descriptor (FD) exhaustion. Traefik sits at the edge and holds at least two FDs per proxied connection: one for the client side, one for the backend side. Add provider connections (Docker socket, Kubernetes API watches), log files, and ACME storage handles, and the count climbs fast. When it hits the OS limit, every new accept4() call fails instantly.

Traefik traffic dropped to zero: entrypoint request rate flatlined

Mon, 01 Jan 0001 00:00:00 +0000

The request rate on an entrypoint that normally serves traffic has fallen to zero, or close enough. Clients are timing out or getting connection errors. Sometimes Traefik’s process is still running and /ping still returns 200, which makes this worse: your health checks are green while no traffic is flowing.

There are two fundamentally different families of root cause, and telling them apart early is the whole game. Either traffic is not reaching Traefik at all (DNS, cloud load balancer, firewall, network partition upstream of the proxy), or traffic is arriving at the host but Traefik cannot accept it (a dead listener, file descriptor exhaustion, a port-bind failure after restart). The checks below are ordered to split those two families within the first few minutes.

Traefik traffic volume: request and response bytes for capacity planning

Mon, 01 Jan 0001 00:00:00 +0000

Request rate and error rate tell you how much traffic Traefik is handling and whether it is succeeding. They do not tell you how much data is moving. A proxy serving ten thousand small API calls per second and a proxy serving ten thousand large file downloads per second look identical on a requests-per-second dashboard, but they have completely different bandwidth, memory, and buffering profiles.

Traefik’s bytes counters close that gap. traefik_service_requests_bytes_total and traefik_service_responses_bytes_total measure the volume of data flowing in each direction per service, with entrypoint-level and router-level equivalents for different aggregation needs. This article covers what these counters measure, how to turn them into capacity-planning inputs, which deviations are worth alerting on, and the instrumentation gaps that can mislead you.

Traefik under scanning: exploit-path probing and request smuggling signals

Mon, 01 Jan 0001 00:00:00 +0000

Any Traefik instance with a public entrypoint is being scanned right now. Requests for /.env, /.git/config, /wp-admin, /phpMyAdmin, and /actuator arrive continuously from botnets and research scanners, and on a correctly configured instance they all get the same answer: a 404 generated by Traefik itself, because no router matches. That is normal background radiation, and paging on it will burn out your on-call in a week.

The operator problem is not “are we being scanned” but “did anything change that makes this scan dangerous.” The three states that matter are: a probe for a sensitive path returns 200 with content, scanning volume or targeting shifts from generic to focused, or request patterns show smuggling or SSRF indicators that Go’s strict parser does not fully neutralize because of what sits behind Traefik. This guide covers how to separate those states from noise, what to check first, and which signals are worth alerting on.