Traefik CLOSE_WAIT pile-up: backend connections that never close

You run ss -s or glance at process_open_fds and notice the Traefik process is holding thousands of sockets in CLOSE_WAIT. Traffic is still flowing, health checks pass, and /ping returns 200, but the count grows every hour. Left alone, the leak consumes file descriptors until the process hits its limit, at which point every new connection fails and clients start seeing 502s.

CLOSE_WAIT means the remote side sent a FIN and the kernel acknowledged it, but the local application never called close() on the socket. In Traefik’s case, the backend (or an intermediary) closed its end of a backend connection, and Traefik’s side is still open, holding a file descriptor. There is no kernel timer that reaps CLOSE_WAIT. These sockets persist until the process closes them or dies.

This is different from TIME_WAIT, which is normal churn after an orderly close and expires on a kernel timer. A growing TIME_WAIT count points at connection pool inefficiency. A growing CLOSE_WAIT count points at a leak: something is closing connections that Traefik never finishes closing, and each one costs an FD forever.

What this means

A TCP connection between Traefik and a backend goes through a close handshake. When the backend closes first, the socket on Traefik’s side moves to CLOSE_WAIT and waits for the application to close it. If Traefik’s HTTP transport never does, the socket sits there indefinitely.

stateDiagram-v2
    ESTABLISHED --> CLOSE_WAIT: backend sends FIN
    CLOSE_WAIT --> LAST_ACK: Traefik calls close()
    LAST_ACK --> CLOSED: final ACK
    CLOSE_WAIT --> CLOSE_WAIT: Traefik never calls close() - leak

Normal traffic produces a small, stable CLOSE_WAIT population because there is always a brief window between the FIN arriving and the application closing. That transient population is fine. The failure mode is a CLOSE_WAIT count that trends upward over hours or days without returning to baseline.

The consequences arrive in a fixed order:

1. File descriptor consumption. Each leaked socket holds one FD. FDs are shared across everything Traefik does: client connections, backend connections, provider watches, log files.
2. FD limit approached. process_open_fds climbs toward process_max_fds. Default container limits (often 1024) make this arrive fast.
3. Cliff-edge failure. At the limit, new connections fail with “too many open files”. Traefik cannot accept new client connections or open new backend connections. Existing connections may still work, but new clients get errors, typically surfacing as 502s at the edge. There is no graceful degradation.

Common causes

Keep-alive mismatch. Traefik’s default forwardingTimeouts.idleConnTimeout is 90 seconds. If the backend’s keep-alive timeout is shorter, the backend closes idle connections first. If an intermediary’s idle timeout is shorter than both, it kills the connection state silently and Traefik keeps a socket it believes is usable. The rule: each layer’s idle timeout must be longer than the layer in front of it.

Health checks. Traefik’s health checker makes periodic requests to every backend. There is a long-standing report (filed against v2.2.1) that health check responses are closed without being read to completion, which prevents connection reuse and churns backend connections on every check cycle.

Hung backends with no timeout. A backend that accepts a connection and never responds leaves Traefik’s goroutine waiting. If responseHeaderTimeout is unset (default 0, unbounded) and the backend eventually dies, the cleanup path can leave sockets behind. Goroutine count rising in lockstep with CLOSE_WAIT is the tell.

Quick checks

All read-only. Run on the host or in the container’s network namespace.

# Count CLOSE_WAIT sockets held by the Traefik process
ss -tnp state close-wait | grep traefik | wc -l

# Show the CLOSE_WAIT sockets with process and peer info
ss -tnp state close-wait | grep traefik | head -20

# Group CLOSE_WAIT sockets by peer (destination) to find the leaking backend
ss -tn state close-wait | awk 'NR>1 {print $5}' | sort | uniq -c | sort -rn | head

# Socket state histogram for Traefik's network namespace
cat /proc/$(pgrep traefik)/net/tcp | awk 'NR>1 {print $4}' | sort | uniq -c
# State codes: 01=ESTABLISHED, 06=TIME_WAIT, 08=CLOSE_WAIT, 0A=LISTEN

# FD usage and limit
ls /proc/$(pgrep traefik)/fd | wc -l
grep 'Max open files' /proc/$(pgrep traefik)/limits

# Goroutine count (leak corroboration); adjust to your metrics entrypoint
curl -s http://localhost:8080/metrics | grep '^go_goroutines'

Two sampling passes a few minutes apart matter more than one reading. A stable CLOSE_WAIT count is background noise. A count that only goes up is the leak.

How to diagnose it

1. Confirm the trend. Take the CLOSE_WAIT count now and again in 10 minutes. If it grew without a matching traffic increase, you have a leak, not churn. Compare with the traefik_entrypoint_requests_total rate over the same window.

2. Identify the peer. Group the CLOSE_WAIT sockets by peer address and port. If they converge on one backend or one subnet, that backend (or something in front of it) is closing connections Traefik never finishes closing. If they are spread evenly, suspect a systemic cause like health checks or an intermediary idle timeout.

3. Check for an intermediary. Map the path from Traefik to the leaking backend: cloud load balancer, NAT gateway, stateful firewall, service mesh sidecar. Any stateful device with an idle timeout shorter than Traefik’s idleConnTimeout (default 90s) will silently drop connection state, and Traefik’s socket lingers. Some managed load balancers have fixed idle timeouts far longer than 90s (one major cloud HTTPS LB uses 600s), which inverts the problem: Traefik must raise idleConnTimeout above the intermediary’s value.

4. Check the keep-alive mismatch direction. Find the backend’s keep-alive timeout from its configuration (for example, nginx keepalive_timeout, or your app’s server setting). If the backend closes idle connections sooner than Traefik’s 90s idle timeout, the backend initiates the close and the mismatch is yours to fix on the Traefik side.

5. Correlate with health checks. If the service has active health checks, compute the leak rate: sockets per hour divided by backends. If it tracks the health check interval closely (roughly one churned connection per check cycle per backend), the health checker is implicated. Test by temporarily lengthening the check interval and watching whether the leak rate drops.

6. Corroborate with goroutines and FDs. A backend-connection leak usually shows go_goroutines and process_open_fds rising on the same curve as the CLOSE_WAIT count. If FDs rise but traefik_open_connections (entrypoint side) is flat, the leak is on the backend side, which matches the CLOSE_WAIT evidence.

7. Check how close the cliff is. Compute process_open_fds / process_max_fds. Above 80% you are in the danger zone; above 95% and rising, treat it as page-worthy because exhaustion is imminent.

Metrics and signals to monitor

Traefik does not expose backend connection pool state as a metric. CLOSE_WAIT counting is an OS-level signal, which is why the ss and /proc checks above are the primary diagnostic and worth turning into a collected metric.

Fixes

Align the idle timeouts

Set forwardingTimeouts.idleConnTimeout on the serversTransport to be shorter than the backend’s keep-alive timeout, so Traefik closes idle connections before the backend does. When an intermediary sits in the path, Traefik’s idle timeout must also be shorter than the intermediary’s. The general rule for the whole chain (client, load balancer, Traefik, backend): each layer’s idle timeout exceeds the layer in front of it.

Tradeoff: a shorter idle timeout means more connection establishment (TCP and possibly TLS handshakes to backends), which costs latency and CPU on the first request after idle periods.

Disable keep-alive as a stopgap

Setting maxIdleConnsPerHost: -1 disables connection reuse to backends entirely: one fresh connection per request, closed by Traefik afterward. This stops CLOSE_WAIT accumulation from idle-connection causes immediately.

Tradeoff: significant. You lose connection reuse, pay full connection setup per request, and increase TIME_WAIT churn on the backend side. Treat this as a mitigation while you find the real mismatch, not as the final state. On some older versions, negative values were rejected by CRD validation; verify your version accepts -1 before relying on it in a Kubernetes CRD.

Bound backend response time

Set responseHeaderTimeout (and review dialTimeout) in the serversTransport so a backend that accepts a connection and hangs cannot hold a goroutine and socket forever. Unbounded waits turn a sick backend into a socket and goroutine leak inside Traefik.

Tradeoff: legitimate long-running requests (large report generation, slow streaming endpoints) will be cut off. Set the timeout per service, not globally, if your backends have heterogeneous response profiles.

Address health check churn

If diagnosis implicates the health checker, lengthen the check interval or reduce the number of health-checked backends per service where safe. Confirm the actual behavior on your version before assuming the historical issue applies to you.

Tradeoff: longer intervals slow down failure detection. Do not stretch health checks past the point where a dead backend serves real traffic for an unacceptable window.

Recover the leaked FDs

Restarting the Traefik process clears all leaked sockets. This is a blunt instrument: it drops every in-flight connection. Use it when the FD ratio is approaching the cliff and you need headroom while the real fix rolls out, not as the fix itself. In HA deployments, restart replicas one at a time.

Prevention

• Track CLOSE_WAIT as a first-class signal. Scrape the count per Traefik instance (a small exporter running the ss or /proc parse works) and alert on sustained growth, not on absolute value.
• Set explicit timeouts everywhere. Do not rely on defaults silently. Document the intended idle-timeout ordering across client, LB, Traefik, and backend, and verify it when any layer changes.
• Alert on the FD ratio with a derivative. process_open_fds / process_max_fds above 80% is a ticket; above 95% sustained and rising is a page. The rising condition separates a leak from a stable high-connection workload.
• Baseline goroutines. go_goroutines growing without a traffic increase is the earliest warning of the hung-connection variant.
• Size FD limits for production. Default container limits around 1024 are not viable for an edge proxy. Raise them deliberately and keep steady-state usage below 70% of the limit.
• Test intermediary idle behavior. After introducing or reconfiguring any NAT, firewall, or cloud LB between Traefik and backends, watch CLOSE_WAIT for a day. This is the change class that most often introduces the leak.

How Netdata helps

• FD headroom per process. Netdata charts open file descriptors against the process limit per application, so the slow climb toward exhaustion is visible days before the cliff, with alerts on the ratio.
• Socket state visibility. Netdata’s network and systemd-service charts surface TCP connection states on the host, letting you watch CLOSE_WAIT as a trend rather than a snapshot from a manual ss run during the incident.
• Go runtime correlation. Goroutine count, heap usage, and GC behavior charted next to connection counts confirm whether a leak is stuck sockets, stuck goroutines, or both.
• Cross-layer correlation. When 502s appear, per-second granularity lets you line up the error burst with idle-period gaps in traffic, the signature of reusing half-dead keep-alive connections.
• Per-instance comparison. In HA deployments, comparing FD and socket trends across replicas shows whether the leak is systemic or tied to one instance’s path to the backends.

Related guides

• Traefik 502 Bad Gateway: when the backend is unreachable or returns garbage
• Traefik 503 Service Unavailable: no healthy backends left in the pool
• Traefik 504 Gateway Timeout: the backend is alive but too slow
• Traefik 5xx error rate: telling Traefik-generated errors from backend errors
• Traefik file descriptor monitoring: process_open_fds, limits, and headroom
• Traefik cascading backend failure: how a partial outage becomes a total one
• Traefik circuit breaker: shedding load from a failing backend
• Traefik health checks pass but requests fail: when the probe lies
• Traefik 404 not found: requests arriving with no matching router
• Traefik config last reload success: monitoring configuration freshness
• Traefik dashboard returns 404: reaching the API and dashboard correctly
• How Traefik actually works in production: a mental model for operators