$ guides / envoy / envoy-upstream-rq-retry-overflow ▌

Operations Guides

Envoy upstream_rq_retry_overflow: the retry budget exhausted and retries dropped

upstream_rq_retry_overflow is climbing on a cluster. Error rates are elevated, and retries should absorb the failures. But upstream_rq_retry is not growing proportionally, and the system looks like it stopped retrying. It did: the retry circuit breaker or retry budget is full, and Envoy is dropping retries to protect the upstream.

The retry system is working as designed, but legitimate retries are being dropped at the moment they are needed most. A budget that is too small loses the resilience retries provide. A budget that is too large risks 2x-3x traffic amplification during partial failures.

What this means

upstream_rq_retry_overflow is a per-cluster counter that increments every time Envoy declines to attempt a retry because the cluster-level retry circuit breaker or retry budget is at capacity. This is distinct from upstream_rq_retry_limit_exceeded, which increments when a single request has exhausted its configured per-request retry count (num_retries in the retry policy, or the x-envoy-max-retries header). Both prevent a retry, but for different reasons with different operational implications.

Two mechanisms can drive upstream_rq_retry_overflow:

Static max_retries circuit breaker. The default is 3 concurrent retries per cluster per priority. When 3 retries are in flight, additional retries are dropped. This is the classic circuit breaker pattern applied to retries.
RetryBudget configuration. When a RetryBudget is defined on the cluster’s circuit breaker thresholds, it overrides the static max_retries. The budget is calculated as a percentage of active plus pending requests, with defaults of budget_percent = 20% and min_retry_concurrency = 3.

When the retry circuit breaker is at capacity, the gauge circuit_breakers.<priority>.rq_retry_open is set to 1. This is the binary signal that retries are currently being dropped.

flowchart TD
    A[Request fails] --> B{Retry policy
configured?}
    B -->|No| C[Return error to client]
    B -->|Yes| D{Per-request retry
limit reached?}
    D -->|Yes| E[retry_limit_exceeded++]
    D -->|No| F{Retry budget or
max_retries full?}
    F -->|Yes| G[retry_overflow++
rq_retry_open = 1]
    F -->|No| H[Execute retry
upstream_rq_retry++]
    H --> I{Retry succeeded?}
    I -->|Yes| J[retry_success++]
    I -->|No| D

The system can look like it stopped retrying while it is actually out of retry capacity. If you only monitor upstream_rq_retry without watching upstream_rq_retry_overflow, you will see retries flatline during a failure and conclude the retry policy is broken. In reality, the budget is exhausted and retries are being dropped silently.

Common causes

Cause	What it looks like	First thing to check
Retry budget too small for error rate	`retry_overflow` climbing, `retry` rate low, upstream errors still high	Compare `retry_overflow` rate against `upstream_rq_5xx` rate
Upstream partial failure consuming budget	`retry` and `retry_overflow` both climbing, `retry_success` low	Check `retry_success / retry` ratio and upstream health
`RetryBudget` with `min_retry_concurrency` set to 0	`retry_overflow` incrementing even with zero active requests	Verify `min_retry_concurrency` is at least 1
Route timeout consuming retry window	Retries not attempted, no overflow, request times out	Check `upstream_rq_timeout` and per-try timeout config

Quick checks

# Check all retry-related counters for a specific cluster.
# Replace my_cluster; admin port is 9901 standalone, 15000 in Istio sidecar.
curl -s http://localhost:9901/stats | grep 'cluster.my_cluster.upstream_rq_retry'

# Check if the retry circuit breaker is currently open.
curl -s http://localhost:9901/stats | grep 'circuit_breakers.*rq_retry_open'

# Check remaining retry headroom (requires track_remaining: true in circuit breaker config).
curl -s http://localhost:9901/stats | grep 'remaining_retries'

# Check retry effectiveness: success vs total retries.
curl -s http://localhost:9901/stats | grep -E 'retry_success|upstream_rq_retry[^_]'

# Check upstream error rate on the affected cluster.
curl -s http://localhost:9901/stats | grep 'cluster.my_cluster.upstream_rq_5'

# Check upstream host health to confirm whether the upstream is degraded.
curl -s http://localhost:9901/stats | grep 'cluster.my_cluster.membership'

The remaining_retries gauge is only available if track_remaining: true is set in the circuit breaker configuration. This is disabled by default.

How to diagnose it

Confirm the retry circuit breaker is open. Check circuit_breakers.<priority>.rq_retry_open. If it is 1, retries are being actively dropped. If it is 0 but retry_overflow is still incrementing, the gauge may be oscillating (opening and closing rapidly as retries complete and new ones are attempted).
Separate overflow from limit-exceeded. Compare upstream_rq_retry_overflow against upstream_rq_retry_limit_exceeded. If retry_limit_exceeded dominates, the problem is per-request retry policy (num_retries too low or x-envoy-max-retries header), not the cluster budget. If retry_overflow dominates, the cluster-level budget is the bottleneck.
Assess retry effectiveness. Calculate retry_success / retry. If this ratio is below 0.5, retries are being attempted but rarely succeeding. The upstream is too degraded for retries to help. Increasing the budget in this situation amplifies load on a failing upstream without improving outcomes.
Check retry amplification. Compare upstream_rq_total against downstream_rq_total. A ratio above 1.3 indicates meaningful retry activity. Above 2.0 indicates a retry storm. If amplification is already high and retry_overflow is climbing, the budget exhaustion is protecting the upstream from worse amplification.
Identify which mechanism is active. Determine whether the cluster uses a static max_retries or a RetryBudget. A RetryBudget overrides max_retries when configured. Check the cluster’s circuit breaker configuration:
```
# Dump circuit breaker config. Output can be verbose; filter by cluster name if needed.
curl -s http://localhost:9901/config_dump | jq '.configs[] | .. | .circuit_breakers? // empty'
```
Check for the min_retry_concurrency edge case. If min_retry_concurrency is explicitly set to 0 and there are zero active and pending requests, the retry budget limit calculates to 0. Note that the default min_retry_concurrency is 3, so this only occurs with an explicit misconfiguration. Set min_retry_concurrency to at least 1.
Check upstream protocol. If you see inconsistent retry_overflow behavior across clusters with identical budget configuration, check the upstream protocol as a possible factor.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`upstream_rq_retry_overflow`	Directly counts dropped retries	Any sustained nonzero rate
`circuit_breakers.<priority>.rq_retry_open`	Binary gauge: is the budget full right now	Value of 1 sustained
`upstream_rq_retry`	Total retries attempted	Rate disproportionate to error rate
`upstream_rq_retry_success`	Retries that actually succeeded	`success / retry` ratio below 0.5
`upstream_rq_retry_limit_exceeded`	Per-request retry limit hit (not budget)	Dominates over `retry_overflow`
`remaining_retries`	Headroom in the retry budget	Trending toward 0 (requires `track_remaining: true`)
`upstream_rq_total / downstream_rq_total`	Retry amplification ratio	Above 1.3 meaningful retry load; above 2.0 is a storm
`upstream_rq_5xx`	Upstream error rate driving retries	Elevated rate is the root cause

Fixes

Retry budget too small for legitimate error rate

The default static max_retries is 3 concurrent retries per cluster. At any meaningful request rate, this is almost always too low. If 1000 requests per second are flowing and 5% are failing, 50 retries per second need to happen, but only 3 can be in flight at any moment.

Switch to a RetryBudget configuration. A budget of 20% allows retries proportional to current load rather than a fixed concurrency cap.

If you are already using a RetryBudget, increase budget_percent cautiously. Going from 20% to 40% doubles the potential retry amplification. Pair any budget increase with monitoring of the upstream_rq_total / downstream_rq_total ratio.

Upstream partial failure consuming budget

If retry_success / retry is low, retries are not helping. Increasing the budget sends more load to a degraded upstream. Address the upstream root cause instead. If the upstream is partially failing due to overload, additional retries accelerate the failure.

Consider temporarily reducing retry aggressiveness for the affected cluster. This can be done via runtime configuration or an xDS config push without restarting Envoy. Reducing the retry budget during an active incident is safer than increasing it.

The min_retry_concurrency edge case

If min_retry_concurrency is explicitly set to 0 and there are zero active and pending requests, the retry budget limit calculates to 0, and any retry attempt overflows immediately. The default min_retry_concurrency is 3, so this only happens with an explicit misconfiguration. Set it to at least 1 to guarantee a floor on the budget.

Route timeout consuming retry window

The route timeout includes all retry attempts and their backoff delays. If the route timeout is 3 seconds and the first attempt takes 2.7 seconds, the retry has only 0.3 seconds to complete, including backoff. Retries may not be attempted at all because there is insufficient time left in the request budget. This shows up as no overflow and no retry, just upstream_rq_timeout. Increase the overall route timeout, reduce the per-try timeout, or reduce retry backoff intervals.

Prevention

Enable track_remaining: true on circuit breaker thresholds. This exposes remaining_retries as a gauge, giving headroom visibility before the budget is exhausted. Without it, you only know the budget is full when retries are already being dropped.
Use RetryBudget instead of static max_retries. The static limit of 3 is too low for most production workloads.
Monitor retry amplification. Track upstream_rq_total / downstream_rq_total as a standard dashboard metric. A ratio above 1.3 warrants investigation. Above 2.0, retries are amplifying the failure they are supposed to mask.
Monitor retry effectiveness. Track retry_success / retry. If retries rarely succeed, they are adding load without providing resilience. This is a signal to investigate the upstream, not to increase the retry budget.
Set min_retry_concurrency to at least 1 when configuring a RetryBudget. This prevents the edge case where the budget evaluates to 0 capacity with zero active requests.
Audit retry policies for non-idempotent endpoints. Envoy does not know whether a request is idempotent. Retrying a POST that charges a credit card or writes a database row can cause duplicate operations. Restrict retries to idempotent methods and retriable status codes.
Scope retry_on triggers. retry_on: 5xx retries on all 5xx responses including 501 (Not Implemented), which will never succeed on retry. Scope retry triggers to codes that are genuinely transient (502, 503, 504).

How Netdata helps

Per-second granularity on retry counters. Netdata collects upstream_rq_retry, retry_success, retry_overflow, and retry_limit_exceeded at per-second resolution. Retry budget exhaustion can be transient, and 15-30 second scrape intervals can miss the window entirely.
Correlation between retry overflow and upstream health. When retry_overflow climbs, Netdata dashboards correlate it with membership_healthy, upstream_rq_5xx, and upstream_rq_time on the same timeline.
Retry amplification visibility. Netdata surfaces the relationship between upstream_rq_total and downstream_rq_total in real time. When retries inflate upstream load 2x-3x, the anomaly is visible without manual rate calculations.
Circuit breaker state alongside retry metrics. The rq_retry_open gauge sits next to cx_open and rq_pending_open in circuit breaker views, showing whether retry exhaustion is isolated or part of a broader saturation cascade.
Anomaly detection on retry rates. Netdata’s anomaly engine learns the normal retry rate for each cluster. A sudden spike in retry_overflow triggers an anomaly alert even if the absolute value is below a fixed threshold.