Envoy 504 upstream timeout: upstream_rq_timeout, per-try timeouts, and the UT flag

A 504 from Envoy with response flag UT means the upstream request timeout fired: Envoy sent the request to a backend and did not receive a complete response within the configured budget. The backend may be genuinely slow, the timeout may be too tight for the workload, or a filter may be interfering with the timeout clock. Each case needs a different fix.

One diagnostic trap: Envoy records a timed-out request in the upstream_rq_time histogram at the timeout duration, not at the actual response time. This creates an artificial ceiling in the latency distribution that looks like a real latency wall. Without cross-referencing the upstream_rq_timeout counter against the histogram, you cannot distinguish genuine backend slowness from the timeout itself shaping the data.

What this means

Envoy applies two layers of upstream timeout:

• Route timeout (overall budget): the total time allowed for the entire request, including all retry attempts. Tracked by the upstream_rq_timeout counter when it fires. Configured via the route’s timeout field or the x-envoy-upstream-rq-timeout-ms request header. If not set, the default is 15 seconds. This timer does not start until the entire downstream request has been received, which matters for slow-upload clients and streaming requests.
• Per-try timeout (per attempt): the time allowed for a single attempt to an upstream host. Tracked by the upstream_rq_per_try_timeout counter. Configured via retry_policy.per_try_timeout or the x-envoy-upstream-rq-per-try-timeout-ms header. The per-try timeout must be less than the route timeout or it has no practical effect.

When the route timeout fires, Envoy returns 504 with response flag UT. When the per-try timeout fires, Envoy retries the request (if retries are configured and the retry budget allows) and only returns a 504 if all retries are exhausted or the route timeout fires first. Envoy does not retry when the route timeout itself is exceeded. If you want retry-on-slow, you must use the per-try timeout.

For long-lived streams (gRPC, SSE, WebSocket), the route timeout is the wrong tool. Set timeout: 0 on the route and use max_stream_duration and idle_timeout instead. When the max stream duration is reached, the upstream_rq_max_duration_reached counter increments.

flowchart TD
    A[Downstream request fully received] --> B[Route timeout timer starts]
    B --> C[Upstream attempt begins]
    C --> D{Per-try timeout fires?}
    D -->|No, response received| E[Forward response - success]
    D -->|Yes| F{Retries remain and route budget left?}
    F -->|Yes| C
    F -->|No| G[Return 504 with UT flag]
    B --> H{Route timeout fires first?}
    H -->|Yes| G

Common causes

Quick checks

# Overall timeout counter for the cluster
curl -s http://localhost:9901/stats | grep 'upstream_rq_timeout'

# Per-try timeout counter
curl -s http://localhost:9901/stats | grep 'upstream_rq_per_try_timeout'

# Max stream duration counter (streaming workloads)
curl -s http://localhost:9901/stats | grep 'upstream_rq_max_duration_reached'

# Artificial ceiling in the latency histogram
curl -s http://localhost:9901/stats/prometheus | grep 'envoy_cluster_upstream_rq_time'

# Retry behavior alongside timeouts
curl -s http://localhost:9901/stats | grep -E 'upstream_rq_retry'

# Upstream host health (slow backend vs. dead backend)
curl -s http://localhost:9901/stats | grep 'membership_healthy'

# Circuit breaker state (saturation can look like slowness)
curl -s http://localhost:9901/stats | grep 'circuit_breakers'

# Connection establishment time (network vs. app latency)
curl -s http://localhost:9901/stats | grep 'upstream_cx_connect_ms'

If running in Istio sidecar mode, replace port 9901 with 15000.

How to diagnose it

1. Confirm UT is the response flag. Check access logs for the %RESPONSE_FLAGS% field. A 504 without UT may be forwarded from the upstream (the backend itself returned 504), which is a different problem. Only UT means Envoy generated the timeout.

2. Identify which timeout fired. Compare upstream_rq_timeout (route budget) against upstream_rq_per_try_timeout (per attempt). If only upstream_rq_per_try_timeout is incrementing and upstream_rq_retry is also elevated, the per-try timeout is the active constraint and retries are masking individual slow attempts.

3. Look for the artificial ceiling in upstream_rq_time. If the histogram shows a sharp spike at a bucket boundary matching your configured timeout, you are seeing the ceiling artifact, not real latency. Cross-reference: if upstream_rq_timeout increments at the same rate as the ceiling bucket fills, the timeout is shaping the distribution.

4. Determine if the backend is genuinely slow. Compare upstream_rq_time P50/P99 against the configured timeout. If P99 is well below the timeout and only a small fraction times out, the backend is mostly healthy with a slow tail. If P50 approaches the timeout, the entire distribution has shifted and the backend is in distress.

5. Check for blocking filters. If timeouts fire even when the backend is fast (test with a direct curl to the upstream), a blocking filter may be consuming wall-clock time that counts against the route timeout budget. Known case: a Lua filter with a blocking sleep blocks the worker thread, preventing timely processing of upstream responses. This is a documented limitation, not a bug.

6. Check for streaming requests on routes with a non-zero timeout. If the affected traffic is gRPC, SSE, or WebSocket, the route timeout should be 0. A non-zero route timeout on a streaming route causes premature 504 UT because the timer starts after the request is received but the stream may legitimately run for minutes or hours.

7. Check the UT with 200 status edge case. If a timeout fires after response headers have been written downstream, Envoy sets response_flag=UT but the status remains 200. Once headers are sent, the status code cannot be changed. If you see UT in logs with status 200, the timeout fired mid-response after headers were committed.

Metrics and signals to monitor

Fixes

Genuinely slow backend

If upstream_rq_time shows the backend is slow (P50/P99 trending up before hitting the timeout), the fix is in the backend, not Envoy. Common causes: database contention, GC pauses, resource saturation, or a scaling event that left too few endpoints handling the load.

Do not raise the timeout to mask the slowness. This delays the 504 but holds connections longer, which can trigger connection pool exhaustion and circuit breaker trips. See Envoy connection pool exhaustion: a slow upstream that fills the pool for the cascade pattern.

If the backend has a known bimodal latency distribution (cache hit vs. cache miss), tune the timeout to accommodate the slow path or split the route so each path has an appropriate budget.

Timeout too tight for the workload

If the backend is healthy (membership_healthy stable, backend P99 within normal range) but upstream_rq_timeout is incrementing and the histogram shows a ceiling at the timeout value, the timeout is too low.

Before raising it, measure the actual backend latency distribution. Set the route timeout to at least 2x the backend P99 under normal load to accommodate variance. The default 15-second timeout is a reasonable starting point for request-response APIs but is wrong for:

• APIs with heavy computation (may need 30-60s)
• File upload/download endpoints (dependent on client speed and body size)
• Streaming endpoints (set to 0, use max_stream_duration)

Per-try timeout and retry configuration

If upstream_rq_per_try_timeout is the active constraint and retries are firing, check:

• The per-try timeout must be less than the route timeout or it has no practical effect.
• Each retry consumes part of the overall route timeout budget. If per-try is 5s and route is 15s, you get at most 3 attempts.
• If retries rarely succeed (upstream_rq_retry_success / upstream_rq_retry < 0.5), retries are adding load without helping. Reduce the retry count or disable retries for this route.
• Retrying on non-idempotent requests (POST, PUT) can cause duplicate operations. Review the retry_on policy.

Blocking filter interference

If timeouts fire even when the backend is fast, inspect the filter chain for blocking operations. The known case is a Lua filter with a long sleep that blocks the worker thread. Any wall-clock time consumed by the blocking filter counts against the route timeout budget, leaving less time for the upstream to respond.

Remove the blocking operation. Envoy filters must be non-blocking. If you need to call an external service, use the async API or a filter designed for async I/O.

Streaming requests

For gRPC, SSE, WebSocket, or any long-lived stream, set timeout: 0 on the route and configure max_stream_duration and idle_timeout. The route timeout is designed for request-response semantics and starts after the full downstream request is received, which is incorrect for streaming where the “request” may be a single frame opening a long-lived stream.

If upstream_rq_max_duration_reached is incrementing on streaming routes, the max_stream_duration is too low for the stream lifetime. Raise it or set it with margin for the maximum expected stream duration.

Prevention

• Set explicit timeouts on every route. Do not rely on the 15-second default. Different endpoints have different latency profiles.
• Set timeout: 0 on streaming routes. Use max_stream_duration and idle_timeout for streams.
• Monitor the upstream_rq_timeout rate as a ratio of total requests. A baseline timeout rate of 0.01% may be normal for a service with a long tail. A sudden increase signals a change in backend behavior or traffic patterns.
• Track the upstream_rq_time histogram shape. A ceiling artifact appearing where none existed before means either the timeout was lowered or the backend latency distribution shifted to overlap the timeout.
• Configure custom histogram buckets aligned with your SLOs. The default buckets span 0.5ms to 3600000ms (1 hour) and are too wide to distinguish between, say, a 2s and a 3s timeout without custom boundaries.
• Review retry policies for amplification risk. If per-try timeouts cause retries, the upstream sees 2-3x load during degradation. See Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests for how saturation interacts with retries.

How Netdata helps

• Per-second resolution on upstream_rq_timeout and upstream_rq_per_try_timeout pinpoints the exact second timeouts began, narrowing the incident window for correlation with backend deploys, traffic spikes, or config changes.
• The upstream_rq_time histogram with the artificial ceiling is visible alongside the timeout counters on the same dashboard, so you can distinguish a genuine latency shift from the timeout shaping the distribution without manual cross-referencing.
• ML anomaly detection on latency percentiles flags a shifting P50 or P99 before it reaches the timeout, giving early warning of backend degradation.
• Correlation with membership_healthy, circuit breaker state, and upstream_cx_connect_ms on a single timeline separates “backend is slow” from “backend is unreachable” from “network is slow” without switching tools.

Related guides

• Envoy 502 and upstream resets: rx_reset, tx_reset, and mid-response failures
• Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend
• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy connection pool exhaustion: a slow upstream that fills the pool
• How Envoy actually works in production: a mental model for operators
• Envoy monitoring checklist: the signals every production proxy needs