Traefik cannot assign requested address: ephemeral port exhaustion

Traefik starts returning 502 Bad Gateway for some or all proxied requests. Backend health checks are green. Backends respond fine when you hit them directly. The Traefik logs show the real error: dial tcp <backend-ip>:<port>: connect: cannot assign requested address.

This is ephemeral port exhaustion: the kernel has no free local ports left for Traefik to open a new outbound TCP connection to a backend. Almost every port in the ephemeral range is pinned by a socket in TIME_WAIT, left behind by a connection that closed up to a minute ago.

The failure shape is nasty: sudden onset, the busiest backends hit first, and health checks usually keep passing because they run at low rate and rarely need a fresh port at the exact moment the range is dry. Everything looks healthy until the cliff.

What this means

Every outbound TCP connection needs a unique (source IP, source port, destination IP, destination port) tuple. The kernel allocates source ports from the ephemeral range, which on most Linux systems defaults to net.ipv4.ip_local_port_range = 32768 60999, roughly 28,000 ports.

When a connection closes, the side that initiated the close keeps the socket in TIME_WAIT for about 60 seconds to absorb straggler packets. During that time the port is unusable for a new connection to the same destination.

Now do the arithmetic. If Traefik creates C new backend connections per second and does not reuse them, steady-state TIME_WAIT occupancy is roughly 60 * C. At about 470 brand-new connections per second to a single backend IP:port pair, you consume the entire default ephemeral range for that destination. Past that point, connect() fails with EADDRNOTAVAIL, surfaced in Go as “cannot assign requested address”, and Traefik returns 502.

flowchart TD
  A[High request rate to backends] --> B[Connections not reused: pool too small or Connection: close]
  B --> C[New socket per request]
  C --> D[Closed sockets linger in TIME_WAIT for ~60s]
  D --> E{Ephemeral ports left?}
  E -->|yes| C
  E -->|no| F[connect fails: cannot assign requested address]
  F --> G[Traefik returns 502 to clients]
  H[Low-rate health checks] -.->|mostly still pass| G

The root cause is almost never the kernel defaults. It is connection churn: Traefik is creating and destroying backend connections instead of pooling and reusing them.

Common causes

Quick checks

All read-only. Run these on the host (or in the network namespace) where Traefik runs.

# 1. Socket summary: is TIME_WAIT large?
ss -s

# 2. Count TIME_WAIT sockets overall
ss -tn state time-wait | wc -l

# 3. TIME_WAIT sockets to a specific backend port
ss -tn state time-wait | grep :<backend_port> | wc -l

# 4. Which destinations are consuming ports (top offenders)
ss -tn state time-wait | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head

# 5. Connection states for the Traefik process specifically
ss -tnp | grep traefik | awk '{print $1}' | sort | uniq -c

# 6. Current ephemeral port range
sysctl net.ipv4.ip_local_port_range

# 7. Confirm the error in Traefik logs. Traefik logs to stdout by default,
#    so use journalctl -u traefik or your configured log file path.
grep "cannot assign requested address" /var/log/traefik/traefik.log | tail

Interpretation anchors: TIME_WAIT above 10K means connection churn worth investigating; above 20K you are inside most of the default ephemeral range and should treat it as an active risk. If check 4 shows the TIME_WAIT mass concentrated on one or two backend IP:port pairs, the constraint is per-destination tuple exhaustion, and connection reuse (not a wider port range) is the real fix.

Also watch CLOSE_WAIT while you are here: a CLOSE_WAIT count that grows over time is a different bug (Traefik not closing sockets the backend already closed, i.e. a connection leak), not ephemeral port exhaustion.

How to diagnose it

1. Confirm the error string. Find cannot assign requested address in Traefik’s logs correlated with the 502 window. This distinguishes port exhaustion from generic 502 causes (backend crash, protocol error, timeout misconfiguration).
2. Quantify TIME_WAIT. Run the ss checks above during the incident. If TIME_WAIT is a large fraction of the ephemeral range (check 6), the diagnosis is confirmed.
3. Identify the destination concentration. If most TIME_WAIT sockets point at one backend IP:port, you are exhausting the tuple space to that destination. This is the classic shape: one hot service behind Traefik, everything else fine.
4. Check the pool configuration. Look at your serversTransport for the affected service. If maxIdleConnsPerHost is unset, you are on the default of 2 (Go’s http.DefaultMaxIdleConnsPerHost), which is far too low for a proxy fronting real traffic. Note that setting it to 0 also falls back to the default; it does not mean unlimited.
5. Check whether backends force close. Send a request through to the backend and inspect response headers. A Connection: close header (or an HTTP/1.0-only backend) defeats pooling entirely: one fresh socket per request no matter what Traefik is configured to do.
6. Rule out the intermediary. If there is a load balancer, NAT gateway, or stateful firewall between Traefik and the backends with an idle timeout shorter than Traefik’s idle connection timeout, it silently kills pooled connections. Traefik then discovers the dead socket on reuse, retries on a fresh connection, and churn climbs. Align Traefik’s idleConnTimeout below the intermediary’s idle timeout.
7. Correlate with traffic. Compare the 502 onset with request rate (traefik_entrypoint_requests_total) and with any recent change: a deployment that disabled keep-alive on the backend, a config change to serversTransport, a traffic spike, or a new service consolidating traffic onto fewer backend IPs.

Metrics and signals to monitor

Traefik does not expose backend connection pool state directly, so some of these are OS-level signals collected alongside the Traefik process.

The distinguishing feature against other 502 causes: sudden onset, all backends behind the same hot destination affected simultaneously, health checks green, and “cannot assign requested address” in the logs. Compare with Traefik 502 Bad Gateway: when the backend is unreachable or returns garbage for the broader 502 differential.

Fixes

Apply these in order of operational safety. The first two are immediate kernel-level relief; the third is the actual fix.

Immediate relief: widen the ephemeral range

# Widen the source port range (takes effect immediately, no restart)
sysctl net.ipv4.ip_local_port_range="1024 65535"

This multiplies available ports roughly 2.3x and buys time. Ports below 32768 can collide with well-known service ports on some systems, but on a dedicated proxy host the risk is low because the allocator skips ports that are already bound. Persist it in /etc/sysctl.conf or a drop-in under /etc/sysctl.d/ so it survives reboot.

Immediate relief: enable TIME_WAIT reuse

# Allow safe reuse of TIME_WAIT sockets for new outbound connections
sysctl net.ipv4.tcp_tw_reuse=1

tcp_tw_reuse lets the kernel recycle a TIME_WAIT socket for a new outgoing connection when it is protocol-safe. It applies to outbound connections, which is exactly Traefik’s backend side. Persist it the same way.

Do not follow old blog posts recommending net.ipv4.tcp_tw_recycle. That sysctl was removed from the kernel in 4.12 and does not exist on any modern distribution. It was also dangerous behind NAT, which is why it was removed.

Root cause fix: connection pooling in serversTransport

Raise the idle connection pool so Traefik reuses backend connections instead of opening new ones:

# Dynamic configuration: file provider example
http:
  serversTransports:
    pooled:
      maxIdleConnsPerHost: 100

Then attach that transport to the affected services. The right value depends on concurrency to a single backend host: size it so steady-state in-flight requests to one backend can be served from the pool. For a service doing a few hundred concurrent requests per backend pod, values in the 64-256 range are typical starting points.

Version notes worth knowing:

• The default is 2, and 0 is not “unlimited”: it falls back to the default of 2. Many operators have set 0 believing it disabled the limit and changed nothing.
• Traefik v3.5.3+ accepts -1, which disables connection reuse entirely (new connection per request). This is valid in the global static serverstransport path and in dynamic HTTP serversTransports/the ServersTransport CRD. That is the opposite of what you want here; it exists for correctness edge cases, not for this incident.
• In Traefik v1.x the equivalent option defaulted to 200. If you migrated from v1 to v2/v3 and never set it, you silently dropped to 2. That migration alone can turn a healthy system into this incident.

Root cause fix: stop backends forcing close

If the backend sends Connection: close, pooling cannot help. Fix the backend to support keep-alive (most app servers and frameworks do by default; check for reverse proxies or frameworks in front of the app that strip or force the header). If the backend speaks only HTTP/1.0, put something HTTP/1.1-capable in front of it or accept that it needs its own capacity plan.

Align idle timeouts

Set Traefik’s idleConnTimeout for the transport below the idle timeout of any intermediary (cloud LB, NAT gateway, firewall) between Traefik and the backends. If the intermediary kills idle sockets first, Traefik reuses dead connections, retries, and churn climbs even with a large pool.

Prevention

• Set maxIdleConnsPerHost deliberately for every production serversTransport. Never ship the default of 2 behind real traffic. Include it in configuration review checklists.
• Alert on TIME_WAIT fraction. Track TIME_WAIT count against the configured ephemeral range and ticket above 50%. The degradation curve is cliff-edge; the leading indicator is the trend, not the failure.
• Track connection creation rate per destination. Steady-state TIME_WAIT is roughly 60 times the new-connection rate. If creation rate times 60 approaches the port range for any single backend IP:port, pooling is not working.
• Load test with realistic keep-alive. Port exhaustion only appears at sustained connection churn. A load test that hammers one endpoint at production-plus RPS for several minutes will reproduce it before production does.
• Keep the kernel sysctls in configuration management. ip_local_port_range and tcp_tw_reuse set once by hand disappear on the next reprovision.
• Watch CLOSE_WAIT separately. A monotonically growing CLOSE_WAIT count is a connection leak, a different bug with the same eventual symptoms, and it will not be fixed by pooling or sysctls.

How Netdata helps

• Netdata’s per-second TCP stack charts show TIME_WAIT, ESTABLISHED, and CLOSE_WAIT socket counts on the Traefik host, so you see the port-pressure trend hours before the 502 cliff.
• Correlating host-level TIME_WAIT growth with Traefik’s per-service 502 rate confirms the signature in one view: error rate spiking while traefik_service_server_up stays flat at 1.
• Anomaly detection on socket counts flags a change in connection churn rate (the leading indicator) without hand-tuning a static threshold for every host.
• Because the same agent collects Traefik metrics, host network stack metrics, and process metrics, you can rule out the lookalikes (FD exhaustion, backend failure, config staleness) in the same dashboard.
• Alerts on TIME_WAIT as a fraction of the ephemeral range give you a ticket-level early warning while the pooling fix is still a change request, not an incident.

Related guides

• Traefik 404 not found: requests arriving with no matching router
• Traefik 502 Bad Gateway: when the backend is unreachable or returns garbage
• Traefik 503 Service Unavailable: no healthy backends left in the pool
• Traefik 504 Gateway Timeout: the backend is alive but too slow
• Traefik 5xx error rate: telling Traefik-generated errors from backend errors
• Traefik cascading backend failure: how a partial outage becomes a total one
• Traefik circuit breaker: shedding load from a failing backend
• Traefik config last reload success: monitoring configuration freshness
• Traefik dashboard returns 404: reaching the API and dashboard correctly
• Traefik health checks pass but requests fail: when the probe lies
• How Traefik actually works in production: a mental model for operators
• Traefik monitoring checklist: the signals every production edge router needs