Traefik 502 Bad Gateway: when the backend is unreachable or returns garbage

A 502 from Traefik means the request got past routing and middleware, Traefik selected a service and tried to talk to a backend, and the backend hop failed. Traefik connected (or tried to connect) and got an invalid response, a reset connection, or nothing usable back. The usual mechanics: the backend crashed mid-response, sent something that is not valid HTTP for the negotiated protocol, or closed a connection Traefik wanted to reuse.

The first diagnostic question is not “why 502” but “whose 502.” Traefik generates a 502 itself when the backend hop fails. But a backend can also return its own 502, produced by some upstream it depends on, and Traefik will pass that through untouched. These two cases look identical to the client and have completely different owners. The fix for the first lives in your proxy/network layer; the fix for the second lives in someone else’s application.

Do not confuse 502 with its neighbors. A 503 means Traefik has no healthy backends for the service at all (health check failures, empty pool). A 504 means the backend exists and is reachable but did not respond within the configured timeout. A 502 means contact happened and the answer was garbage or the connection died. Different code, different layer, different runbook. Teams that aggregate all 5xx into one alert consistently investigate the wrong component.

What this means

Traefik’s request pipeline is: entrypoint, router match, middleware chain, service load balancer, backend server. A 502 originates at the last hop. Everything before it worked: the port accepted the connection, a router matched, middlewares passed, a server URL was selected.

That gives you a strong negative result immediately: a 502 is not a routing problem (that is a 404 at the entrypoint), not a pool-exhaustion problem (that is a 503), and not Traefik being down. Traefik is alive and proxying. The failure is between Traefik and one specific backend, or inside the backend’s own response.

To locate the failing service, compare the entrypoint-level and service-level counters. traefik_entrypoint_requests_total{code="502"} tells you 502s are leaving the edge. traefik_service_requests_total{code="502"} per service tells you which backend hop is failing. If the service-level counter for your suspected service increments in lockstep with the client-visible errors, the failure is on that service’s backend connection. Whether the 502 body was generated by Traefik or passed through from the backend is a second step: check the response body (Traefik’s generated error page looks different from an application error page) and check the backend’s own logs for the same timestamps.

Common causes

Quick checks

All read-only. Run from a host that can reach Traefik’s metrics endpoint and the Traefik process.

# Which services are producing 502s
curl -s http://localhost:8080/metrics | grep 'traefik_service_requests_total' | grep 'code="502"'

# Entrypoint-level 502s for comparison
curl -s http://localhost:8080/metrics | grep 'traefik_entrypoint_requests_total' | grep 'code="502"'

# Backend health per server URL (only present if health checks are enabled)
curl -s http://localhost:8080/metrics | grep traefik_service_server_up

# Retries masking intermittent backend failure
curl -s http://localhost:8080/metrics | grep traefik_service_retries_total

# TIME_WAIT accumulation (ephemeral port pressure)
ss -tn state time-wait | wc -l

# Socket states for the Traefik process (watch CLOSE_WAIT)
ss -tnp | grep traefik | awk '{print $1}' | sort | uniq -c

# File descriptor headroom
ls /proc/$(pgrep traefik)/fd | wc -l
grep 'Max open files' /proc/$(pgrep traefik)/limits

# Config reload activity (correlate with 502 timestamps)
curl -s http://localhost:8080/metrics | grep traefik_config_reloads_total

Three notes. First, traefik_service_server_up only exists for services with health checks configured; absence of the series means unmonitored, not healthy. Second, if retry middleware is enabled, traefik_service_requests_total counts each attempt, so the service-level rate can be higher than the real client request rate. Third, if more than one Traefik process runs on the host, pgrep traefik returns multiple PIDs and the /proc one-liners break; pin the PID explicitly.

How to diagnose it

1. Confirm the 502 is Traefik-adjacent. Reproduce with curl -v against the entrypoint and inspect the response body and headers. A backend pass-through 502 usually carries the backend application’s own error page or headers. A Traefik-generated 502 is a bare error response.
2. Locate the failing service. Compare traefik_entrypoint_requests_total{code="502"} with traefik_service_requests_total{code="502"} per service. The service whose counter rises in step with client errors owns the failing hop.
3. Check backend liveness and restarts. If the backend restarted or crashed at the same timestamps, the 502 is a mid-response crash. This is a backend bug, not a proxy problem.
4. Check socket states. Rising CLOSE_WAIT means backends closed connections Traefik has not cleaned up. TIME_WAIT climbing toward the ephemeral port range means connection churn is about to block new backend connections entirely.
5. Check the keep-alive race. If Traefik’s serversTransport.forwardingTimeouts.idleConnTimeout is longer than the backend’s own keep-alive timeout, the backend closes idle connections that Traefik then tries to reuse, producing a reset and a 502. Defaults from the Traefik docs: idleConnTimeout 90s, dialTimeout 30s, responseHeaderTimeout 0s (disabled).
6. Check the upstream timeout chain. If Traefik sits behind a cloud load balancer, each layer’s idle timeout must be longer than the layer in front of it. A common failure: the LB’s idle timeout exceeds Traefik’s respondingTimeouts.idleTimeout (default 180s), so the LB sends a request on a connection Traefik already closed. The result is intermittent 502s that no backend log will explain.
7. Correlate with deployments and reloads. Plot the 502 counter against traefik_config_reloads_total and your deploy pipeline. Brief spikes during rolling updates are terminating backends still receiving traffic. Rare spikes exactly on reload increments point to the context-cancellation edge case.

flowchart TD
  A[Client sees 502] --> B{Service-level 502 counter rising?}
  B -->|no| C[Edge issue: check routing, middleware, config freshness]
  B -->|yes| D{Backend restarts or crash logs at same time?}
  D -->|yes| E[Backend crashed mid-response: fix the backend]
  D -->|no| F{Correlates with deploys or reloads?}
  F -->|yes| G[Transient: rolling update or reload race - bounded retries]
  F -->|no| H{CLOSE_WAIT or TIME_WAIT climbing?}
  H -->|yes| I[Connection pool or ephemeral port exhaustion]
  H -->|no| J[Keep-alive or LB timeout-chain mismatch]

Metrics and signals to monitor

Fixes

Backend crashing mid-response

Fix the backend. Traefik is correctly reporting that the upstream died during the response. Check the backend’s panic logs, OOM kills, and deploy history. While you fix it, a bounded retry middleware (for example, 3 attempts) absorbs the failures for idempotent requests, at the cost of extra backend load. Watch traefik_service_retries_total so the retries do not amplify an already-degrading backend into a full outage.

Keep-alive race on reused connections

Make Traefik’s serversTransport.forwardingTimeouts.idleConnTimeout shorter than the backend’s keep-alive timeout, so Traefik is always the side that closes idle connections first. The same logic applies to anything stateful between Traefik and the backend (NAT gateways, firewalls): intermediary idle timeouts silently kill pooled connections, and Traefik discovers the corpse on the next request. Do not disable keep-alive entirely as a fix; it trades a small intermittent error for a large permanent performance cost.

Timeout-chain mismatch behind a load balancer

Audit the full chain: client, cloud LB, Traefik entrypoint (respondingTimeouts), Traefik to backend (forwardingTimeouts), backend server. Each layer should have a longer idle/read timeout than the layer in front of it. Traefik’s entrypoint defaults are readTimeout 60s and idleTimeout 180s. Long uploads that exceed readTimeout surface as 502s with I/O timeout errors; raise it for the affected entrypoint rather than globally.

Ephemeral port exhaustion and CLOSE_WAIT leaks

Short term, widen the port range and enable reuse:

sysctl -w net.ipv4.ip_local_port_range="1024 65535"
sysctl -w net.ipv4.tcp_tw_reuse=1

These apply host-wide and immediately, and they do not survive a reboot unless persisted to sysctl.conf. Treat them as mitigation, not the fix. The real fix is connection reuse: raise maxIdleConnsPerHost in the serversTransport so Traefik stops creating a new backend connection per request, and find out who is sending Connection: close. CLOSE_WAIT growth means Traefik is not reaping connections the backend closed; treat it as a leak and track it to zero after fixing the pooling config.

Transient 502s during rolling updates

Terminating backends keep receiving traffic for a window. Options: configure retry middleware with a small attempt count so a failed connection is retried against a different server, and make sure your platform’s shutdown hooks (preStop, readiness gates) give Traefik time to observe termination. If rare 502s align exactly with config reloads rather than deployments, reduce provider churn or raise providersThrottleDuration to batch updates.

Prevention

• Document the timeout chain. Write down every idle/read/write timeout from client to backend and verify the ordering after any infra change. Most “mystery intermittent 502” incidents are this.
• Enable health checks per service. Without them, traefik_service_server_up does not exist and dead backends stay in rotation until the first failed request tells you.
• Alert on the service-level 502 rate, not the aggregate 5xx rate. Per-service, per-code alerting is what routes the page to the right team.
• Track retry ratio. traefik_service_retries_total divided by service request rate is the earliest warning of backend instability, before clients see errors.
• Baseline per service. A static threshold across heterogeneous backends (API, WebSocket, streaming) hides slow-building 502 patterns. Alert on deviation from per-service baselines.
• Watch socket states. CLOSE_WAIT and TIME_WAIT trends are leading indicators; the 502 spike is the lagging one.

How Netdata helps

• Netdata charts traefik_service_requests_total broken down by response code and service, so a 502 burst is attributed to a specific backend hop in seconds rather than by grepping counters.
• Correlating service-level 502s with traefik_service_retries_total and traefik_service_request_duration_seconds on one dashboard shows the degradation sequence (latency climb, retry climb, then errors) instead of a flat error count.
• Netdata collects process-level signals alongside Traefik metrics: open FDs versus limits and TCP socket states including TIME_WAIT and CLOSE_WAIT, so the connection-exhaustion causes of 502s are visible in the same view.
• Per-second collection catches the brief 502 spikes during rolling updates that per-minute scraping averages away.
• Anomaly detection on per-service error rates surfaces deviation from baseline without hand-tuned thresholds per backend.

Related guides

• How Traefik actually works in production: a mental model for operators
• Traefik monitoring checklist: the signals every production edge router needs
• Traefik monitoring maturity model: from survival to expert
• Traefik /ping returns 200 while everything is broken: the health-check trap