Traefik open connections growing: spotting a connection leak

The traefik_open_connections gauge is climbing. Request rate is flat. Every hour the number is higher than the last, and it never comes back down, even after the traffic peak passes. That divergence is the signature of a connection leak: connections are opened and never closed, and each one holds a file descriptor, a goroutine, and some memory.

Left alone, this ends one way. Traefik hits its file descriptor limit and stops accepting new connections instantly. Existing connections keep working, so some health checks still pass, but every new client is refused. It is a cliff-edge failure with no graceful degradation, and the climb gives you hours or days of warning if you are watching the right signal.

This guide covers how to confirm a real leak versus legitimate long-lived traffic, how to find where the connections are piling up, and how to stop the growth before the FD limit does it for you.

What this means

Traefik spawns a goroutine per accepted connection on each entrypoint, and tracks the current count in traefik_open_connections with entrypoint and protocol labels. On Traefik v2 the equivalent series is traefik_entrypoint_open_connections; in v3 the per-router and per-service variants were removed and a single global gauge per entrypoint and protocol remains.

A rising count is not automatically a problem. Three protocol behaviors distort this metric, and you have to account for them before declaring a leak:

• HTTP/2 and gRPC multiplex many streams over one connection. Under heavy HTTP/2 throughput, the connection count can look deceptively low while request rate is high. The inverse matters too: a modest connection count can carry enormous traffic, so “low connections” does not mean “low load.”
• WebSockets are long-lived by design. Every WebSocket session holds a connection (and an FD) for its entire lifetime. If you front WebSocket traffic, the gauge inflates permanently and must be baselined separately.
• Keep-alive keeps idle connections open. Clients and load balancers hold connections open for reuse. A plateau after a traffic peak that decays slowly is often just keep-alive expiry, not a leak.

The actual leak test is comparative: is the connection trend diverging from the request-rate trend? Genuine load moves both together. Slow backends pile connections up while request rate stays flat or falls. A true leak grows the count monotonically regardless of traffic shape.

flowchart TD
    A[traefik_open_connections rising] --> B{Request rate also rising?}
    B -- Yes --> C[Genuine traffic growth]
    C --> C1[Capacity planning, not a leak]
    B -- No, flat or falling --> D{Service latency rising too?}
    D -- Yes --> E[Backends too slow: connections piling up]
    E --> E1[Check backend latency, timeouts, retries]
    D -- No --> F{WebSocket or gRPC traffic present?}
    F -- Yes --> G[Long-lived sessions: baseline separately]
    F -- No --> H[Suspected connection leak]
    H --> H1[Check CLOSE_WAIT, goroutines, FD trend]

Common causes

Quick checks

All read-only. Run on the Traefik host or against its metrics endpoint (typically port 8080; adjust for your deployment).

# 1. Current open connections per entrypoint and protocol
curl -s http://localhost:8080/metrics | grep -E 'traefik.*open_connections'

# 2. Request rate to compare against (take two samples 60s apart)
curl -s http://localhost:8080/metrics | grep traefik_entrypoint_requests_total

# 3. File descriptor usage and limit - the hard backstop
curl -s http://localhost:8080/metrics | grep -E 'process_open_fds|process_max_fds'

# 4. FD usage from the kernel's view
ls /proc/$(pgrep -f traefik | head -1)/fd | wc -l
grep 'Max open files' /proc/$(pgrep -f traefik | head -1)/limits

# 5. Socket states for the Traefik process
ss -tnp | grep traefik | awk '{print $1}' | sort | uniq -c

# 6. CLOSE_WAIT specifically - backends closed, Traefik did not
ss -tan state close-wait | wc -l

# 7. Goroutines - should track connections; divergence means trouble
curl -s http://localhost:8080/metrics | grep go_goroutines

# 8. Which service is slow right now
curl -s http://localhost:8080/metrics | grep traefik_service_request_duration_seconds

Interpretation shortcuts:

• Check 1 growing, check 2 flat: the leak test is positive. Move to diagnosis.
• Check 5/6 showing hundreds of CLOSE_WAIT: the far end closed and your side did not. That is a leak pattern, not load.
• Check 3 above 80%: you are out of runway regardless of root cause. Treat FD pressure as its own incident.
• Check 7 climbing with check 1: consistent with stuck connections. If goroutines climb but connections do not, suspect hung middleware or provider watchers instead.

How to diagnose it

1. Confirm divergence over a real window. Plot traefik_open_connections and rate(traefik_entrypoint_requests_total[5m]) over at least 24 hours. A leak is monotonic growth with no decay after traffic peaks. Keep-alive decay produces a plateau that falls within the keep-alive timeout window. One noisy hour proves nothing.

2. Rule out protocol effects. If the entrypoint carries WebSocket traffic, compare the gauge against an independent session count from your application. If it carries HTTP/2 or gRPC, one connection can hide thousands of streams; use stream-level signals from the backend if you have them. A count that matches active WebSocket sessions plus a small HTTP baseline is healthy, not leaking.

3. Check whether backends are the cause. Correlate the connection climb with traefik_service_request_duration_seconds per service. If one service’s latency rose at the same time the connections started piling up, you do not have a leak in Traefik; you have a backend that stopped finishing requests, and Traefik is correctly holding connections open waiting for it. Also check traefik_service_retries_total for retry amplification making the pile-up worse.

4. Inspect socket states. A large and growing CLOSE_WAIT count (check 6) means remote ends finished and the local side never called close. A large ESTABLISHED count to one backend IP points at that upstream holding connections. A large TIME_WAIT count is a different problem entirely: connection churn and possible ephemeral port exhaustion, not a leak.

5. Cross-check goroutines and memory. go_goroutines and process_resident_memory_bytes growing in lockstep with connections confirms stuck handlers holding resources. If you have the debug API enabled, a goroutine dump shows what the stuck goroutines are waiting on: curl http://localhost:8080/debug/pprof/goroutine?debug=1. Treat this endpoint as sensitive; do not expose it beyond localhost or an authenticated network.

6. Verify the metric itself. On Traefik v3.0.x, traefik_open_connections has a confirmed bug where the gauge drifts and can go negative because connection removal can fire more than once per connection. If your values look impossible (negative, or falling while FDs rise), check your version before trusting the trend.

7. Measure FD runway. Compute (process_max_fds - process_open_fds) / fd_growth_rate. If the trend says you hit the limit in 48 hours, that is your real deadline, and it overrides everything else on this list.

Metrics and signals to monitor

Alerting priority: the FD ratio is the PAGE-level signal because it is comprehensive and the failure is binary. The connection-gauge divergence is a TICKET-level early warning that should fire days before the FD alert ever does.

Fixes

Backends too slow (connections piling up, not leaking)

Fix the backend, not the proxy. While you do, bound the damage:

• Set responseHeaderTimeout in the serversTransport so a backend that accepts a connection but never answers gets cut loose instead of holding a goroutine forever. Without it, the default is no timeout.
• Review idleConnTimeout (default 90s) so idle backend connections are reaped on a sane schedule.
• Reduce retry attempts if traefik_service_retries_total shows amplification; retries multiply the connection pile on an already slow backend.

True connection leak (CLOSE_WAIT growth, stuck goroutines)

• Identify and remove the offending backend from rotation if one upstream is responsible, using the socket-state evidence from step 4.
• Restart Traefik to reclaim FDs when the FD ratio is already critical. This is disruptive: it drops all live connections. It buys time but fixes nothing permanently; the leak will regrow on the same schedule.
• Upgrade Traefik if you are on a version with a confirmed connection-accounting or WebSocket cleanup bug. Check the changelog for your minor version before scheduling anything else.

Clients holding connections forever

• Set keepAliveMaxRequests and keepAliveMaxTime on the entrypoint transport (available since v2.11). Both default to 0, meaning unlimited. When the limit is hit, Traefik sends Connection: close for HTTP/1.1 or a GOAWAY for HTTP/2, forcing well-behaved clients to reconnect and preventing indefinite accumulation.
• Check upstream load balancer idle timeouts. If a cloud LB in front of Traefik has a longer idle timeout than Traefik’s own respondingTimeouts.idleTimeout, the LB reuses connections Traefik already closed, which surfaces as resets and churn. Order the timeouts so the client side of each hop expires first.

FD limit too low

• Raise the limit. Default container ulimits (often 1024) are dangerously low for an edge proxy. Production Traefik should run with at least 65536. Set it in the container spec, compose ulimits, or systemd unit, and verify with process_max_fds after restart.
• Keep steady state below 70% of the limit so reconnection storms and WebSocket bursts have headroom.

Prevention

• Baseline per entrypoint, per protocol. WebSocket-heavy entrypoints need their own baseline and thresholds. A single global threshold on a mixed deployment will either page on healthy WebSocket growth or miss a real HTTP leak.
• Alert on divergence, not absolute values. The useful alert is “connection count growth without request-rate growth, sustained.” Absolute thresholds break every time traffic grows legitimately.
• Always pair the gauge with the FD ratio alert. traefik_open_connections covers entrypoint connections only; FDs cover backend connections, provider sockets, and log files too. The FD ratio is the alert that cannot be fooled.
• Set timeouts deliberately. responseHeaderTimeout, idleConnTimeout, keepAliveMaxTime, and keepAliveMaxRequests should be conscious choices reviewed per environment, not inherited defaults discovered during an incident.
• Track goroutines as a first-class signal. For a reverse proxy, go_goroutines is one of the earliest leak indicators available. Trend it and alert on sustained deviation from baseline.

How Netdata helps

• Netdata charts traefik_open_connections per entrypoint and protocol at per-second resolution, so the slow monotonic climb that defines a leak is visible on a single dashboard instead of being reconstructed from scattered scrape data.
• Plotting connection count next to entrypoint request rate makes the core divergence test (connections up, requests flat) a visual check rather than a PromQL exercise.
• Netdata collects process_open_fds and process_max_fds alongside Traefik metrics, so the FD backstop ratio is monitored on the same host view, with alerts as it approaches the cliff.
• Goroutine count, resident memory, and GC behavior from the Go runtime are correlated on the same host, which is exactly the confirmation chain (connections, goroutines, memory) that separates a stuck-backend pile-up from a true leak.
• Anomaly detection on the connection gauge flags growth that deviates from the learned traffic pattern, catching leaks that grow slowly enough to slip under static thresholds.

Related guides

• Traefik 404 not found: requests arriving with no matching router
• Traefik 502 Bad Gateway: when the backend is unreachable or returns garbage
• Traefik 503 Service Unavailable: no healthy backends left in the pool
• Traefik 504 Gateway Timeout: the backend is alive but too slow
• Traefik 5xx error rate: telling Traefik-generated errors from backend errors
• Traefik cascading backend failure: how a partial outage becomes a total one
• Traefik circuit breaker: shedding load from a failing backend
• Traefik config last reload success: monitoring configuration freshness
• Traefik dashboard returns 404: reaching the API and dashboard correctly
• Traefik health checks pass but requests fail: when the probe lies
• How Traefik actually works in production: a mental model for operators
• Traefik monitoring checklist: the signals every production edge router needs