Traefik backend connection pool: keep-alive, MaxIdleConnsPerHost, and reuse

Every request Traefik proxies needs a TCP connection to a backend. Whether that connection is freshly dialed or reused from a pool determines how much per-request overhead you pay, and it is one of the least monitored parts of the proxy. A misconfigured pool shows up as elevated latency on every request (TCP and possibly TLS handshake costs per request), as growing TIME_WAIT socket counts marching toward ephemeral port exhaustion, or as sporadic 502s with no backend outage to explain them.

The core problem is a default. Traefik’s backend transports are built on Go’s http.Transport, and the Go standard library defaults MaxIdleConnsPerHost to 2. That value was chosen for HTTP clients, not for a reverse proxy fanning thousands of requests per second into a handful of backends. Unless you have explicitly tuned serversTransport, your Traefik is likely running with that default today.

What the pool is and why it matters

Traefik’s load balancer holds a pool of net/http transports, one per backend target, and each transport maintains its own pool of idle keep-alive connections. When a request arrives for a service, the load balancer picks a backend server, then asks that server’s transport for a connection. If an idle connection exists in the pool, it is reused immediately. If not, a new connection is dialed.

The economics:

• Reused connection: near-zero setup cost. The request goes out on an established TCP stream.
• New connection: full TCP handshake (one round trip minimum), plus a TLS handshake if the backend scheme is HTTPS, plus ephemeral port allocation on the Traefik side.

At low traffic, this barely matters. At production rates, a pool that fails to reuse connections turns every request into a connection-establishment exercise. Each closed connection then sits in TIME_WAIT for roughly 60 seconds, holding an ephemeral port. This is the classic failure: everything looks fine on dashboards, then Traefik starts returning 502s because the kernel cannot assign a local port for a new backend connection.

How the pool works

flowchart LR
  REQ[Request matched to service] --> LB[Load balancer picks backend]
  LB --> POOL{Idle connection in pool?}
  POOL -->|yes| REUSE[Reuse connection]
  POOL -->|no| DIAL[Dial new connection]
  REUSE --> RESP[Response received]
  DIAL --> RESP
  RESP --> CLOSEHDR{Connection close requested?}
  CLOSEHDR -->|no| IDLE[Return to idle pool]
  CLOSEHDR -->|yes| CLOSE[Close socket - TIME_WAIT]
  IDLE --> EVICT{idleConnTimeout expired?}
  EVICT -->|yes| CLOSE
  EVICT -->|no| IDLE

A connection leaves the idle pool in one of three ways:

1. Reuse: a new request takes it.
2. Local eviction: Traefik’s forwardingTimeouts.idleConnTimeout expires and closes it.
3. Remote death: the backend, a NAT gateway, or a stateful firewall closes it first. Traefik does not learn about this until it tries to reuse the connection, at which point the write fails.

Case 3 is the dangerous one. The connection looks healthy in the pool, gets handed to a real request, and the request dies.

The defaults and where they bite

Traefik exposes the pool through serversTransport, configured globally in static config or per-service via a dynamic ServersTransport resource.

Three behaviors trip operators up:

• Zero is not unlimited. Setting maxIdleConnsPerHost: 0 does not mean “no cap”. It falls back to the default of 2. To disable connection reuse entirely, set it to -1. Note that -1 also makes idleConnTimeout meaningless, because there are no idle connections left to time out, and it maximizes churn: expect a large TIME_WAIT population and higher per-request latency.
• The default of 2 is per host, per transport, and only caps retention. Under concurrent load, Go’s transport opens as many connections as in-flight requests demand; maxIdleConnsPerHost only governs how many are kept afterward. With the default of 2, a burst of 50 concurrent requests to one backend opens 50 connections, then closes 48 of them when the burst drains. The next burst pays full dial cost again and adds 48 more sockets to TIME_WAIT.
• Kubernetes CRD users: on some Traefik versions the ServersTransport CRD validation set the minimum for maxIdleConnsPerHost to 0, which made -1 impossible to express via CRD. This was corrected in later releases.

Migration history matters here too: in Traefik v1.x the global default was 200 idle connections per host. The v2 redesign moved the setting into serversTransport and reset the effective default to Go’s stdlib value of 2. Teams that migrated without touching serversTransport silently lost two orders of magnitude of idle-connection retention.

Where pool problems show up in production

Ephemeral port exhaustion and TIME_WAIT growth

When connections are created and destroyed per request instead of reused, each one leaves a socket in TIME_WAIT for about 60 seconds. The steady-state TIME_WAIT population is roughly 60 times the new-connection rate. Common drivers:

• maxIdleConnsPerHost left at 2 on a high-throughput service.
• Backends responding with Connection: close, which Traefik honors. No pool setting overrides this; the backend forces one connection per request.
• maxIdleConnsPerHost: -1 set deliberately (for example, to work around dead-connection 502s) without accounting for the churn cost.

The symptom sequence is distinctive: backends pass health checks, but proxied requests start failing with 502s, and the kernel logs report “cannot assign requested address”. For the full diagnostic path on that symptom, see Traefik 502 Bad Gateway.

Dead pooled connections and sporadic 502s

NAT gateways and stateful firewalls between Traefik and its backends track connection state and silently expire idle entries. The backend itself may also close keep-alive connections on its own idle timer. Traefik’s pool still holds the socket, considers it reusable, writes a request into it, and gets a reset. The result is a low-rate, seemingly random 502 pattern that no health check catches.

The governing rule: Traefik’s idleConnTimeout must be shorter than every idle timeout in the path. That includes the backend’s own keep-alive timeout and any intermediary NAT or firewall idle timeout. If the backend closes connections at 60s and Traefik keeps them for 90s, every connection aged 60-90s in the pool is a 502 waiting to happen. If an intermediary expires state at 75s, same story. When you cannot learn or change the intermediary’s timeout, shortening idleConnTimeout below it is the fix. A retry middleware can mask the residual race, but timeout ordering is the actual repair.

The same mismatch exists on the client side: if a cloud load balancer in front of Traefik has a longer idle timeout than Traefik’s respondingTimeouts.idleTimeout, the LB reuses connections Traefik already closed, producing the same class of intermittent errors. Order the whole chain, not just one link.

File descriptors

Every pooled idle connection holds a file descriptor on the Traefik process, in addition to the two FDs per active proxied request. Raising maxIdleConnsPerHost raises steady-state FD usage. That is usually the right trade, but it interacts with the process FD limit: if you are running with the container default of 1024, aggressive pooling and moderate concurrency will collide. Watch process_open_fds / process_max_fds when changing pool sizes. See Traefik file descriptor monitoring for the headroom math.

Checking pool behavior from the OS

Traefik does not expose connection pool metrics directly, so pool state has to be observed at the socket layer. These checks are read-only and safe to run on a live host.

# Connection states for the Traefik process
ss -tnp | grep traefik | awk '{print $1}' | sort | uniq -c

# TIME_WAIT sockets toward a specific backend port
ss -tn state time-wait | grep :<backend_port> | wc -l

# System-wide socket summary
ss -s

# Connection states from /proc (01=ESTABLISHED, 06=TIME_WAIT, 08=CLOSE_WAIT, 0A=LISTEN)
cat /proc/$(pgrep traefik)/net/tcp | awk 'NR>1 {print $4}' | sort | uniq -c

What you are looking for:

• High TIME_WAIT with high request rate: connections are churning instead of being reused. Check maxIdleConnsPerHost and whether backends send Connection: close.
• Growing CLOSE_WAIT: the backend closed the connection and Traefik has not. A growing CLOSE_WAIT count is a leak indicator.
• TIME_WAIT approaching the ephemeral port range: exhaustion is near. The short-term mitigations are widening net.ipv4.ip_local_port_range and enabling net.ipv4.tcp_tw_reuse (which only applies to outbound connections, so it does help the proxy-to-backend direction). Both are kernel-level changes: review them against your environment’s networking constraints before applying, and treat them as buying time while you fix the pool.

Tuning guidance

• Set maxIdleConnsPerHost deliberately. For a service with meaningful concurrency, 2 is wrong. Size it to roughly the steady-state concurrency per backend so connections survive between bursts instead of being torn down. The cost is FDs and backend-side connection slots, not meaningful memory or CPU.
• Order the idle timeouts. idleConnTimeout on Traefik must be strictly lower than the backend’s keep-alive timeout and any NAT/firewall idle timeout in between. If you cannot verify the intermediary, shorten Traefik’s side.
• Fix backends that send Connection: close. That header defeats pooling entirely. If it comes from a legacy application or an intermediary, that is the thing to change; no Traefik setting compensates for it.
• Avoid -1 as a first resort. Disabling reuse silences dead-connection 502s by removing reuse, but it converts them into churn, latency, and port pressure. Prefer timeout ordering plus retries.
• Do not forget cold start. Traefik has no pool warmup. After a restart, every backend connection is new, so the first wave of traffic pays full dial and TLS cost and can hammer fragile backends. Plan backend capacity for post-restart connection storms.

Signals to watch in production

How Netdata helps

• Per-second socket and TCP-state visibility at the host level, so TIME_WAIT and CLOSE_WAIT growth is visible as a trend, not discovered at port exhaustion.
• Correlation of Traefik’s service-level 5xx breakdown (traefik_service_requests_total by code) with socket churn, which separates “backend is failing” from “pool is failing”: healthy backends plus rising TIME_WAIT plus sporadic 502s points at reuse, not the application.
• process_open_fds against process_max_fds tracking, so raising maxIdleConnsPerHost does not quietly trade a churn problem for an FD cliff.
• Service latency histograms (traefik_service_request_duration_seconds) per second, making the latency floor shift from lost reuse measurable against the pre-change baseline.
• Retry rate (traefik_service_retries_total) alongside error rate, surfacing the amplification that dead pooled connections cause when a retry middleware is in the chain.

Related guides

• Traefik 404 not found: requests arriving with no matching router
• Traefik 502 Bad Gateway: when the backend is unreachable or returns garbage
• Traefik 503 Service Unavailable: no healthy backends left in the pool
• Traefik 504 Gateway Timeout: the backend is alive but too slow
• Traefik 5xx error rate: telling Traefik-generated errors from backend errors
• Traefik cascading backend failure: how a partial outage becomes a total one
• Traefik circuit breaker: shedding load from a failing backend
• Traefik config last reload success: monitoring configuration freshness
• Traefik dashboard returns 404: reaching the API and dashboard correctly
• Traefik file descriptor monitoring: process_open_fds, limits, and headroom
• Traefik health checks pass but requests fail: when the probe lies
• How Traefik actually works in production: a mental model for operators