Envoy URX upstream retry limit exceeded: all retries used, last error returned

When you see URX in Envoy access logs, every configured retry attempt has fired and failed, and the last upstream error is what the client receives. This is distinct from retry_overflow, where retries were never attempted because the retry budget or circuit breaker was full. The distinction matters because the two conditions have opposite root causes and opposite fixes.

URX means the retry mechanism is doing its job mechanically: it tried, it retried, and the upstream kept failing. The question is not “why did Envoy give up” but “why does the upstream keep failing on retried requests.” The answer is usually one of three things: the upstream error is genuinely non-retriable (a 501 Not Implemented will never succeed on retry), the retry window is too small for the upstream to recover, or the retry policy is amplifying a partial failure into a full one.

What this means

The URX response flag is set when Envoy exhausts its configured retry budget for a request. The flag covers both HTTP retry limits and TCP maximum connect attempts. The counter cluster.<name>.upstream_rq_retry_limit_exceeded tracks the total number of requests that reached this state.

The important semantic point: URX does not mean retries were attempted but throttled. It means retries were attempted, completed, and failed. Compare this with upstream_rq_retry_overflow, which counts requests that were never retried because the retry circuit breaker or retry budget was full. A request can contribute to one counter or the other in a given window, not both for the same attempt sequence.

The downstream response code attached to a URX request is whatever the last upstream attempt returned. If the upstream returned 503 on the final retry, the client sees 503. If the upstream reset the connection on the final retry, the client may see 503 with an accompanying flag like UF or UC. Multiple flags can be set simultaneously, so a single access log line might read UC,URX (upstream connection termination, retries exhausted).

flowchart TD
    A[Request fails] --> B{Retry policy matches?}
    B -->|No| C[Return upstream error]
    B -->|Yes| D{Retry budget available?}
    D -->|No, budget full| E[retry_overflow
Never retried]
    D -->|Yes| F[Attempt retry]
    F --> G{Retry succeeded?}
    G -->|Yes| H[Return success]
    G -->|No| I{Retry limit reached?}
    I -->|No| D
    I -->|Yes| J[URX
Last error returned]

Common causes

Quick checks

# Check retry exhaustion counters for a specific cluster
curl -s http://localhost:9901/stats | grep 'cluster.my_cluster.upstream_rq_retry'

# Distinguish URX from retry_overflow
curl -s http://localhost:9901/stats | grep -E 'retry_limit_exceeded|retry_overflow'

# Retry effectiveness ratio
curl -s http://localhost:9901/stats | grep -E 'upstream_rq_retry$|upstream_rq_retry_success'

# Upstream error breakdown by response code
curl -s http://localhost:9901/stats | grep -E 'cluster.my_cluster.upstream_rq_[45]'

# Per-try vs overall timeout counters
curl -s http://localhost:9901/stats | grep -E 'upstream_rq_per_try_timeout|upstream_rq_timeout'

# Connection failure context
curl -s http://localhost:9901/stats | grep -E 'upstream_cx_connect_fail|upstream_rq_rx_reset'

# Retry circuit breaker state
curl -s http://localhost:9901/stats | grep 'circuit_breakers.*rq_retry_open'

# Cluster membership context
curl -s http://localhost:9901/stats | grep 'cluster.my_cluster.membership'

# Config dump for retry policy on the affected route
curl -s http://localhost:9901/config_dump | jq '[.. | objects | select(has("retry_policy"))]'

The admin port is 9901 in standard deployments and 15000 in Istio sidecar mode. Adjust the URL accordingly.

How to diagnose it

1. Confirm URX is the right flag. Pull a sample of access log lines for affected requests and verify the %RESPONSE_FLAGS% field. URX may coexist with UF, UC, or UT. The co-occurring flag tells you what the final retry attempt actually returned.

2. Separate URX from retry_overflow. Compare upstream_rq_retry_limit_exceeded against upstream_rq_retry_overflow. If overflow is climbing faster than limit_exceeded, the retry budget is the binding constraint, not the retry count. That points to retry storm amplification, not a non-retriable error.

3. Compute retry effectiveness. Compare upstream_rq_retry_success against upstream_rq_retry. If retries almost never succeed, the retry policy is wasting upstream capacity. A success ratio below 0.5 during an incident means retries are adding load without recovering requests.

4. Inspect the retry_on policy for the affected route. Pull the route configuration from /config_dump and look at the retry_policy block. The conditions listed in retry_on determine which upstream responses trigger a retry. A broad policy like retry_on: 5xx retries on most 5xx responses. Envoy excludes 501 Not Implemented from the 5xx condition by default, but codes like 502 and 503 that reflect a persistent upstream condition will still burn through the retry budget without success.

5. Correlate with upstream error codes. Use cluster.<name>.retry.upstream_rq_<*xx> counters to see what response codes retries are actually receiving. If retries consistently receive 501, 502, or 503 from the same host, the error is non-retriable.

6. Check timeout configuration. The overall route timeout (set via the route’s timeout field or x-envoy-upstream-rq-timeout-ms) includes all retry attempts. If the first attempt consumes most of the budget, subsequent retries have very little time. Compare upstream_rq_per_try_timeout against upstream_rq_timeout to see whether per-try limits are configured and firing.

7. Check upstream host health. URX with UF or UC flags means the upstream connections themselves are failing. Look at membership_healthy, upstream_cx_connect_fail, and outlier_detection.ejections_active to understand whether the upstream cluster is degraded.

Metrics and signals to monitor

Fixes

Retrying non-retriable status codes

If retry_on: 5xx is configured and the upstream returns 502 Bad Gateway, 503 Service Unavailable, or another 5xx that will not recover on retry, every retry produces the same error. The retry policy is mechanically correct but semantically wrong.

Narrow the retry policy. Use retriable_status_codes to list only the codes that are genuinely transient for your workload, or use retry_on conditions that exclude permanent failures. For example, if the upstream only returns 503 under transient load but 502 due to a persistent configuration error, configure retriable_status_codes: [503] instead of relying on the broad 5xx condition.

The tradeoff: narrower retry policies recover fewer transient failures. Measure retry_success / retry before and after the change to confirm the narrowed policy still catches the errors that matter.

Retry window too short

The overall route timeout includes all retry attempts. If the route timeout is 3 seconds and the first attempt takes 2.7 seconds, retries have only 0.3 seconds to complete. URX appears with the UT flag, and upstream_rq_per_try_timeout increments.

Configure per-try timeout using x-envoy-upstream-rq-per-try-timeout-ms or the route’s retry_policy.per_try_timeout. This gives each attempt a fixed budget independent of the overall timeout. The overall timeout still caps the total, but each retry gets a fair window.

The tradeoff: per-try timeouts that are too long delay failure detection. Align per-try timeout with the upstream’s expected response time distribution, not with a generic default.

Retry storm amplification

If URX appears alongside climbing retry_overflow, the retry system is both exhausting its budget and hitting its concurrency limit. This is the retry storm pattern: the upstream is partially failing, retries multiply load, and the multiplied load causes more failures.

The immediate fix during an incident is to reduce retry aggressiveness. Lower the retry count, narrow retry_on conditions, or temporarily disable retries for the affected cluster. The retry budget defaults to 20% of active plus pending requests with a minimum concurrency of 3, which may be too permissive during partial failures.

The long-term fix is to address why the original requests are failing. Retries mask transient errors; they do not fix persistent ones. If the upstream error rate is consistently above a few percent, retries are amplifying a problem rather than recovering from noise.

Upstream connection failures during retry

URX with UF means the retry attempts are failing at the connection layer, not the response layer. The upstream hosts are refusing or resetting connections. Check upstream_cx_connect_fail, upstream_rq_rx_reset, and membership_healthy to understand the upstream state.

This is not a retry policy problem. The retry policy is correct, but the upstream is unreachable. Fix the upstream: scale out, resolve network partitions, or address host-level failures. See the related guides on connection termination and circuit breakers for the upstream diagnosis path.

gRPC ResourceExhausted

If retry_on: resource-exhausted is configured for gRPC routes, the upstream returning ResourceExhausted (gRPC code 8) triggers retries. ResourceExhausted is often a business-level rate limit signal, not a transient system failure. Retrying it produces more load on an already rate-limited upstream and eventually hits URX.

Distinguish between rate limiting that is transient (the upstream will recover capacity) and rate limiting that is policy-driven (the upstream is intentionally rejecting the request). For policy-driven rate limits, remove resource-exhausted from the retry policy or handle ResourceExhausted at the client layer with backoff, not at the proxy layer with blind retries.

Prevention

• Monitor the retry amplification ratio. Track upstream_rq_retry / upstream_rq_total as a standard dashboard metric. Sustained values above 0.1 warrant investigation.
• Monitor retry effectiveness. Track upstream_rq_retry_success / upstream_rq_retry. If retries rarely succeed during normal operation, the retry policy is misconfigured.
• Prefer retry budgets over static max_retries. Retry budgets scale with active request volume, which bounds amplification during traffic spikes. If a retry budget is configured, it overrides the static max_retries circuit breaker.
• Audit retry_on policies during config review. Confirm that every condition in the policy corresponds to a genuinely transient failure mode for the upstream.
• Configure per-try timeouts explicitly. Relying on the overall route timeout to cover all retries creates the short-window problem.
• Disable retries for non-idempotent endpoints. Restrict retries to connection-level failures (UF) only, or disable them entirely. Retrying a POST that timed out risks duplicate side effects.
• Track URX and retry_overflow as separate alerts. They indicate different problems and require different responses.

How Netdata helps

• Netdata surfaces upstream_rq_retry_limit_exceeded and upstream_rq_retry_overflow as per-second counters, so you can distinguish retry exhaustion from retry budget throttling in real time during an incident.
• The retry amplification ratio (upstream_rq_retry / upstream_rq_total) and effectiveness ratio (upstream_rq_retry_success / upstream_rq_retry) are directly visible on the same cluster dashboard.
• ML anomaly detection flags sudden changes in retry rates and upstream error rates, which often precede a URX spike by minutes.
• Correlating URX counters with upstream latency (upstream_rq_time), connection failures (upstream_cx_connect_fail), and cluster membership (membership_healthy) on a single timeline shortens the path from symptom to root cause.
• Per-second granularity matters for retry storms, where the difference between a transient blip and a cascading failure is visible in the rate of change of retry counters over seconds, not minutes.

Related guides

• Envoy 502 and upstream resets: rx_reset, tx_reset, and mid-response failures
• Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend
• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy connection pool exhaustion: a slow upstream that fills the pool
• Envoy health checks vs outlier detection: two systems that eject hosts differently
• How Envoy actually works in production: a mental model for operators
• Envoy membership_healthy dropping: reading the single most important cluster signal
• Envoy monitoring checklist: the signals every production proxy needs
• Envoy monitoring maturity model: from survival to expert
• Envoy no healthy upstream: the 503 when a cluster has no host to route to
• Envoy outlier detection mass ejection: when passive health checks empty a cluster
• Envoy panic threshold: why traffic routes to unhealthy hosts at 50%