Traefik intermittent 502 behind a load balancer: the timeout chain mismatch

Some percentage of requests behind your cloud load balancer return 502. Not all of them, just a low-grade drip: one in a few hundred, sometimes one in a few thousand. Backend logs show nothing. Traefik logs show a burst of 502s with no corresponding backend error. Load balancer health checks are green. Everything looks healthy, and users keep hitting errors.

This is the classic signature of a keep-alive idle timeout mismatch in the chain client -> load balancer -> Traefik -> backend. Somewhere in that chain, a component is reusing a connection that the next hop has already closed. The request is written into a half-dead socket, the other side answers with a TCP reset, and the proxy reports 502 Bad Gateway. Because it only happens when a request lands on an idle connection at the wrong moment, it is intermittent, load-dependent, and hard to reproduce.

The fix is not a flag you flip in one place. It is making the timeout chain coherent: at every hop, the side that reuses pooled connections must close them before the peer does. This article walks through the mechanism, how to confirm it, and how to set Traefik’s respondingTimeouts and forwardingTimeouts so the resets stop.

What this means

Reverse proxies reuse backend and frontend connections via HTTP keep-alive. That reuse is what makes proxying cheap: no TCP handshake, no TLS handshake, no slow start per request. But keep-alive creates a race. Each side of a connection has its own idle timeout, and neither side tells the other when it is about to close. If side A’s timeout is shorter than side B’s, there is a window where side B still considers the connection usable while side A has already closed it. When side B reuses that connection, it gets a reset.

With Traefik behind a cloud load balancer, there are two such races:

1. LB to Traefik. The cloud LB holds idle keep-alive connections to Traefik. If the LB’s idle timeout is longer than Traefik’s entryPoints.<name>.transport.respondingTimeouts.idleTimeout, Traefik closes the connection first. The LB then sends a client request down the dead connection and gets a reset. Most LBs translate that into a 502 to the client. This is the most common break, and it produces 502s at the load balancer that never appear in Traefik’s own access logs.

2. Traefik to backend. Traefik pools backend connections. If the backend’s keep-alive timeout is shorter than or equal to Traefik’s serversTransport idle connection timeout, the backend closes first and Traefik writes into the dead socket. This time Traefik itself returns the 502, and it does show up in Traefik’s metrics and logs.

In both cases the application is fine. The failure is entirely at the transport layer, which is why application logs are empty.

flowchart LR
  C[Client] -->|"keep-alive"| LB[Cloud LB
idle timeout 60s]
  LB -->|"keep-alive"| T[Traefik
idleTimeout 180s]
  T -->|"pooled conns"| B[Backend
keep-alive timeout]
  T -.->|"idleTimeout shorter than LB idle timeout:
Traefik closes first, LB reuses dead conn, RST -> 502"| X[Intermittent 502]
  B -.->|"backend timeout shorter than Traefik idleConnTimeout:
Traefik reuses dead conn, RST -> 502"| X

The rule that keeps the chain safe: at each hop, the side that initiates and reuses connections must have the shorter idle timeout, so it always retires connections on its own timer instead of discovering the peer already closed them. Concretely:

• LB idle timeout < Traefik respondingTimeouts.idleTimeout
• Traefik forwardingTimeouts.idleConnTimeout < backend keep-alive timeout

This guarantees the side holding pooled connections (LB toward Traefik, Traefik toward backend) always sees the connection closed by its own timer, never by the peer mid-reuse.

Common causes

Quick checks

All read-only.

# 1. Pull Traefik's current timeout configuration from the API
curl -s http://localhost:8080/api/rawdata | python3 -m json.tool | grep -iA5 -e respondingTimeouts -e forwardingTimeouts

# 2. Entrypoint-level vs service-level 5xx: are the 502s even reaching Traefik?
curl -s http://localhost:8080/metrics | grep 'traefik_entrypoint_requests_total' | grep 'code="5'
curl -s http://localhost:8080/metrics | grep 'traefik_service_requests_total' | grep 'code="5'

# 3. Check retry activity: retries masking connection resets
curl -s http://localhost:8080/metrics | grep traefik_service_retries_total

# 4. Inspect TCP connection states around Traefik (CLOSE_WAIT buildup hints at peer-closed sockets)
#    Needs root (or sudo) for the -p process column
sudo ss -tnp | grep traefik | awk '{print $1}' | sort | uniq -c

# 5. Cloud LB error metrics: check the LB's own target-response/5xx counters
#    (AWS: CloudWatch HTTPCode_ELB_5XX_Count vs HTTPCode_Target_5XX_Count;
#     a high ELB_5XX with low Target_5XX points at the LB-to-Traefik hop)

Check 1 assumes the API is enabled on the dashboard entrypoint. Adjust the port and address to your deployment. If the API is not enabled, read the static configuration file or the startup flags instead.

How to diagnose it

1. Locate which hop generates the 502. Compare 502 counts at three points: the load balancer’s own metrics, Traefik’s entrypoint metrics, and Traefik’s service metrics. If the LB reports 502s that never appear in Traefik’s entrypoint counters, the failure is on the LB-to-Traefik hop. If Traefik’s service metrics show the 502s, the failure is on the Traefik-to-backend hop. If both, you may have both races. See telling Traefik-generated errors from backend errors for the general method.

2. Check the pattern against idle behavior. The timeout-mismatch 502 has a distinctive fingerprint: it hits requests that follow an idle gap on a pooled connection. At high sustained QPS the rate drops toward zero because connections never sit idle long enough to be closed. At low or bursty QPS the rate climbs. If your 502 rate is inversely correlated with throughput, you are almost certainly looking at an idle-timeout race. Constant-rate 502s that track throughput point elsewhere: see Traefik 502 Bad Gateway.

3. Read the actual timeout values. From the LB side: AWS ALB defaults to a 60-second idle timeout (configurable up to 4000 seconds). GCP’s external HTTPS load balancer uses a fixed 600-second keep-alive timeout to backends that cannot be changed. Azure Front Door documents its own backend idle timeout behavior. Verify against your provider’s current documentation, since these are platform behaviors that change.

4. Read Traefik’s values. In Traefik v3, entryPoints.<name>.transport.respondingTimeouts.idleTimeout defaults to 180s, and http.serversTransports.<name>.forwardingTimeouts.idleConnTimeout defaults to 90s. Other forwardingTimeouts defaults: dialTimeout 30s, responseHeaderTimeout 0s (disabled). Defaults have shifted across releases, so never assume them: dump the running config (check 1 above) and read the actual values. If you run Traefik v2, verify against the v2 docs; the field layout is similar but not identical.

5. Compare the pair at each hop. For the LB hop: LB idle timeout must be less than Traefik’s idleTimeout. For the backend hop: Traefik’s idleConnTimeout must be less than the backend’s keep-alive timeout (for example nginx keepalive_timeout, or your application server’s equivalent). Flag any hop where the front side is longer than, or equal to, the back side. Equal values are a violation too: two timers expiring at the same moment still race.

6. Rule out lookalikes. If 502s correlate with deploys or with traefik_config_reloads_total increments rather than with idle gaps, you are looking at backend churn or config-reload context cancellation, not a timeout mismatch. If backends flap on health checks, see Traefik 503 Service Unavailable. If the backend is alive but slow, see Traefik 504 Gateway Timeout.

Metrics and signals to monitor

Fixes

Fix the LB-to-Traefik hop

Make Traefik’s idleTimeout comfortably longer than the LB’s idle timeout. For an AWS ALB at its 60-second default, the Traefik v3 default of 180s is already safe. The break appears when someone lowered Traefik’s idleTimeout below the LB’s, or when the LB idle timeout was raised (ALBs can go to 4000s).

# static config: entrypoint respondingTimeouts
entryPoints:
  websecure:
    address: ":443"
    transport:
      respondingTimeouts:
        idleTimeout: 660s   # longer than the LB idle timeout, with margin

For GCP’s HTTPS load balancer with its fixed 600-second keep-alive, you cannot lower the LB side, so raise Traefik’s idleTimeout above 600s. Operators report 650s as a working value, with the caveat that some still saw residual 502s and needed the backend-hop fix below as well.

Fix the Traefik-to-backend hop

Make Traefik’s idleConnTimeout shorter than the backend’s keep-alive timeout, with real margin (tens of seconds, not one second).

# dynamic config: serversTransport
http:
  serversTransports:
    default:
      forwardingTimeouts:
        idleConnTimeout: 45s   # shorter than backend keep-alive timeout

If the backend’s keep-alive timeout is 60s, a 45s idleConnTimeout means Traefik always discards the pooled connection before the backend would close it. The cost is a slightly higher rate of new backend connections, which is negligible compared to the cost of 502s.

Retry middleware as a partial mitigation

Traefik’s retry middleware reissues a request when the backend connection fails, which covers reset-induced races where the pooled connection dies before a response arrives. A small retry count (attempts: 3) on idempotent routes will mask most residual races by re-sending the request over a fresh connection. Treat this as a safety net, not the fix: it hides the symptom and adds backend load. Also note the risk profile described in Traefik cascading backend failure: retries amplify load on a struggling backend.

What not to do

• Do not disable keep-alive globally (for example maxIdleConnsPerHost: -1). That disables connection reuse entirely, makes idleConnTimeout irrelevant, and replaces cheap 502s with expensive per-request connection setup plus TIME_WAIT accumulation. At high throughput that can trade this problem for ephemeral port exhaustion; see Traefik cannot assign requested address.
• Do not set both sides of a hop to the same value. Matching timers still race.
• Do not raise timeouts without checking the whole chain. Raising Traefik’s idleTimeout above the LB’s while leaving the backend hop mismatched just moves the 502s to the other hop.

Prevention

• Document the chain. Write down the idle timeout at every hop (client-side CDN, LB, Traefik entrypoint, Traefik serversTransport, backend server) in one place, in order, and enforce the strictly-decreasing-from-client rule in review.
• Verify after upgrades and LB changes. Timeout defaults have changed across Traefik minor releases, and cloud LB defaults differ by product. Any upgrade of Traefik, the LB configuration, or the backend server is a reason to re-check the chain.
• Alert on the ratio, not just the count. A low absolute 502 count hides at high traffic. Alert on 502s as a fraction of requests, and separately compare LB 5xx to Traefik entrypoint 5xx so a regression at either hop pages the right team.
• Baseline the inverse-load fingerprint. If your overnight 502 ratio creeps up over weeks, a timeout somewhere has drifted. Catch it in the trend before a config change makes it obvious at the worst time.

How Netdata helps

• Error-code breakdown per entrypoint and per service. Netdata charts Traefik’s Prometheus counters split by response code, so the LB-side vs Traefik-side 502 comparison from the diagnosis steps is a two-chart glance instead of a PromQL session.
• Load-correlation view. Plotting 502 rate against request rate on one dashboard makes the inverse-load fingerprint of an idle-timeout race visible immediately, which is the fastest single discriminator for this failure mode.
• Retry visibility. Netdata surfaces traefik_service_retries_total alongside request and error rates, so you can see retries silently absorbing resets before the client-visible 502 rate moves.
• Connection population. Per-entrypoint open connections and process-level socket states show the sawtooth expiry pattern of a too-short idle timer.
• Post-fix verification. After changing idleTimeout or idleConnTimeout, the 502 ratio and retry rate charts give you a clear before/after without waiting for user reports.

Related guides

• Traefik 404 not found: requests arriving with no matching router
• Traefik 502 Bad Gateway: when the backend is unreachable or returns garbage
• Traefik 503 Service Unavailable: no healthy backends left in the pool
• Traefik 504 Gateway Timeout: the backend is alive but too slow
• Traefik 5xx error rate: telling Traefik-generated errors from backend errors
• Traefik access log blocking: when logging stalls request handling
• Traefik ACME challenge failed: HTTP-01, DNS-01, and TLS-ALPN-01 renewal errors
• Traefik acme.json permissions and corruption: renewal silently blocked
• Traefik ACME rate limit: too many certificates already issued for this domain
• Traefik backend connection pool: keep-alive, MaxIdleConnsPerHost, and reuse
• Traefik cannot assign requested address: ephemeral port exhaustion
• Traefik cascading backend failure: how a partial outage becomes a total one