Traefik 504 Gateway Timeout: the backend is alive but too slow

Your Traefik instance is up. /ping returns 200. The backend service is running, its health checks pass, and clients are getting 504 Gateway Timeout. The connection to the backend was established, but the response never arrived within the configured timeout.

A 504 in Traefik means: the backend exists, Traefik reached it, the TCP connection (and usually the request) succeeded, but the response took too long. That makes a 504 fundamentally different from a 502 (invalid response or connection error from the backend) or a 503 (no healthy backends at all). Lumping all 5xx together leads directly to investigating the wrong layer.

There are only two directions: the backend genuinely is slow (database queries, resource contention, a stalling downstream dependency), or the timeout on the relevant path is too aggressive for the workload. Determine which, and on which leg of the request the stall happens.

What this means

A Traefik 504 is generated by Traefik itself, not passed through from the backend. Traefik opened a connection to a backend server, sent the request, and gave up waiting. Two timeout families are involved:

• respondingTimeouts (per entrypoint, static config): readTimeout (default 60s), writeTimeout (default 0s, unbounded), idleTimeout (default 180s). These govern the client-facing side. readTimeout covers reading the entire request body, not just headers. A slow upload that exceeds 60 seconds is therefore a supported upgrade-compatibility risk since v2.11.2 (previously 0s, no timeout); whether the failure is logged as 504 or 499 is protocol/version-dependent.
• forwardingTimeouts (per ServersTransport, dynamic config): dialTimeout (default 30s), responseHeaderTimeout (default 0s, unbounded), idleConnTimeout (default 90s). These govern the Traefik-to-backend side. responseHeaderTimeout is the one most directly tied to backend slowness: how long Traefik waits for the backend to start responding. The 0s default means unbounded, so a backend that accepts a connection and stalls holds a Traefik goroutine indefinitely.

Corollary: if you have never set responseHeaderTimeout, Traefik does not time out waiting for response headers at all. In that configuration, a client-reported 504 is often the client-facing readTimeout firing, a load balancer in front of Traefik timing out, or the client itself giving up. Verify which component actually generated the 504 before tuning anything.

Common causes

Quick checks

These are read-only and safe to run during an incident.

# Which service is producing 504s
curl -s http://localhost:8080/metrics | grep 'traefik_service_requests_total' | grep 'code="504"'

# Backend latency as Traefik sees it (histogram buckets and counts)
curl -s http://localhost:8080/metrics | grep traefik_service_request_duration_seconds

# Entrypoint latency for comparison (isolates Traefik overhead vs backend time)
curl -s http://localhost:8080/metrics | grep traefik_entrypoint_request_duration_seconds

# Are backends passing health checks?
curl -s http://localhost:8080/metrics | grep traefik_service_server_up

# Retry activity on the affected service
curl -s http://localhost:8080/metrics | grep traefik_service_retries_total

# Goroutines and open connections piling up (stuck backend connections)
curl -s http://localhost:8080/metrics | grep -E 'go_goroutines|traefik_open_connections'

Then check the backend itself. A backend that passes /health can still stall on real paths if the health endpoint does not exercise the database or downstream dependencies. Hit the actual slow endpoint directly, bypassing Traefik, and time it. Also search Traefik’s debug logs for the timeout error string: 504s from the forwarding side typically log net/http: timeout awaiting response headers or a read i/o timeout against the backend address.

How to diagnose it

Work the legs of the request in order. Each step eliminates one segment of the path.

1. Identify the affected service. Group 504s by the service label on traefik_service_requests_total. If 504s span many unrelated services simultaneously, suspect Traefik-side resource pressure (CPU saturation, access log buffer blocking, config rebuild storms) rather than a single slow backend.
2. Confirm the backend is reachable and selected. Check traefik_service_server_up. If all backends are 0, you are looking at a 503 problem, not a 504 problem. Note that this metric only exists for services with health checks enabled; absence means unmonitored, not healthy.
3. Compare entrypoint latency to service latency. If traefik_entrypoint_request_duration_seconds is high but the service-level histogram is normal, the time is being spent inside Traefik (TLS, middleware chain, compression, access log buffer blocking). If both are high, the backend is slow. This single comparison splits the search space in half.
4. Bypass Traefik and time the backend directly. Send the same request to a backend pod or container directly, and time it. If it is slow without Traefik in the path, the problem is the backend: database queries, connection pool exhaustion, a stalled downstream dependency. Go investigate the backend; Traefik is just the messenger.
5. Match failed durations against configured timeouts. If the direct backend call is fast but Traefik 504s, look at the timeouts on the path. For request bodies (uploads), check the entrypoint readTimeout. For slow first-byte, check responseHeaderTimeout. Failed requests that die at almost exactly 60 seconds are a strong readTimeout signature; check whether Traefik logged the terminal status as 504 or 499.
6. Check for retry amplification. A rising traefik_service_retries_total concurrent with rising latency is a positive feedback loop: a slow backend gets multiplied traffic, gets slower, and 504s more. Traefik is contributing to the failure it is reporting.
7. Check goroutine and connection accumulation. If go_goroutines and traefik_open_connections climb while request rate is flat, backends are accepting connections and stalling. This is the goroutine leak pattern that eventually OOM-kills Traefik, and it usually means responseHeaderTimeout is unset (0s, unbounded).

flowchart TD
  A[504 Gateway Timeout] --> B{Entrypoint latency high, service latency normal?}
  B -- Yes --> C[Traefik-side: readTimeout, middleware, log buffer]
  B -- No --> D{Backend slow when called directly?}
  D -- Yes --> E[Backend problem: DB, CPU, downstream dependency]
  D -- No --> F{Timeout config on the path}
  F --> G[readTimeout too low for uploads]
  F --> H[responseHeaderTimeout too low for slow endpoints]
  D -- Stalls, never responds --> I[Unset responseHeaderTimeout: goroutine accumulation]

Metrics and signals to monitor

Caveat on the latency histograms: the default buckets [0.1, 0.3, 1.2, 5.0] are coarse. If your timeout is 30s or 60s, everything above 5 seconds lands in the top bucket, and percentile estimates will not warn you how close requests are running to the timeout before 504s start.

Fixes

The backend is genuinely slow

Fix the backend. Common root causes visible from the Traefik side: database lock contention or connection pool exhaustion, a degraded downstream service cascading upstream, CPU or memory saturation. Traefik cannot fix this; it can only report it. If the slowness is legitimate and unavoidable for some endpoints (large report generation, exports), those endpoints need a timeout budget that reflects reality rather than a generic default.

The timeout is too aggressive

Raise the timeout on the correct leg, deliberately:

• For slow uploads dying at 60s, raise or disable readTimeout on the entrypoint that serves them. Disabling (0s) removes the protection entirely, so prefer a value sized to your realistic worst case.
• For slow first-byte from backends, responseHeaderTimeout in the ServersTransport governs the wait. respondingTimeouts is per entrypoint and applies to every router on it; there is no per-router override. If one route needs a much longer budget, the standard approach is a dedicated entrypoint for that traffic class.

Tradeoff: every timeout you raise is a resource commitment. A 120s timeout means a stuck request holds a goroutine, a connection, and memory for up to 120 seconds. Size timeouts to the workload, not to silence the alert.

The timeout was never set

If responseHeaderTimeout is 0s (unbounded), a backend that accepts connections and stalls will accumulate Traefik goroutines until memory exhaustion. Set a finite responseHeaderTimeout on your ServersTransports. For typical request/response APIs this is cheap insurance; for streaming or long-poll endpoints, set it explicitly high rather than leaving it unbounded.

Retry amplification

If retries are amplifying a slow-backend incident, reduce or disable the retry middleware on the affected service while you fix the root cause. Retries against a latency-degraded backend convert a brownout into an outage.

Timeout chain coordination

Traefik sits in a chain: client, possibly a cloud load balancer, Traefik, backend. Each layer’s timeout should be longer than the one in front of it. When this ordering is violated, intermediate components close connections that downstream components are still using, producing intermittent errors that resist diagnosis. Audit the whole chain when you change any one value.

Prevention

• Set responseHeaderTimeout explicitly on every ServersTransport. The unbounded default is a goroutine leak waiting for a stalled backend.
• Alert on the latency-to-timeout gap. Track p95/p99 of traefik_service_request_duration_seconds per service against the configured timeout, not against an arbitrary absolute threshold. A service running at 80 percent of its timeout budget is a 504 incident in rehearsal.
• Monitor retries as a ratio. traefik_service_retries_total over traefik_service_requests_total per service. Retries during a brownout are how slow backends die completely.
• Split 5xx alerts by code. 502, 503, and 504 have different root causes, different owners, and different runbooks. Aggregate 5xx alerting guarantees slow, wrong triage.
• Cross-check health checks against reality. If traefik_service_server_up is all-green while 504s flow, your health endpoint does not exercise the real dependency chain. Make health checks meaningful or distrust them.
• Baseline per service. A 200ms p95 is excellent for a compute-heavy API and terrible for static file serving. Alert on deviation from per-service baselines, not universal thresholds.

How Netdata helps

• Netdata charts traefik_service_requests_total split by response code, so 504s are visible per service as a distinct series rather than buried in a 5xx aggregate.
• The traefik_service_request_duration_seconds histogram is rendered per service, letting you watch latency climb toward the configured timeout before 504s start.
• Correlating service latency with entrypoint latency on one dashboard answers the “Traefik overhead or backend slowness” question in seconds.
• Goroutine count and open connections alongside the 504 rate surface the stuck-backend-connection pattern early, before it becomes an OOM kill.
• Retry rate plotted against request rate makes amplification loops obvious while they are still recoverable.
• ML-based anomaly detection on per-service latency catches the “slow but not yet timing out” state that static thresholds miss.

Related guides

• How Traefik actually works in production: a mental model for operators
• Traefik monitoring checklist: the signals every production edge router needs
• Traefik monitoring maturity model: from survival to expert
• Traefik /ping returns 200 while everything is broken: the health-check trap