Envoy retry storm: when the upstream-to-downstream request ratio climbs

A retry storm is one of the most dangerous failure modes in an Envoy-based service mesh or edge proxy. An upstream service starts failing a fraction of requests. Envoy’s retry policy, configured to mask transient errors, fires retries on the failures. The extra load lands on an already-degraded upstream. More requests fail under the additional load. More retries fire. Within minutes, the upstream receives two to three times its normal traffic, mostly retries, and collapses under amplification it cannot escape.

The pattern is hard to see in the metrics most teams watch. Error rates climb, latency grows, and circuit breakers trip, but each individual signal looks like ordinary upstream degradation. The signature that distinguishes a retry storm from a simple upstream outage is the relationship between the volume of traffic Envoy sends upstream and the volume clients send to Envoy.

What this means

Under normal conditions, the number of upstream requests Envoy sends should be approximately equal to the number of downstream requests it receives. Every downstream request maps to roughly one upstream request. The ratio cluster.<name>.upstream_rq_total / http.<stat_prefix>.downstream_rq_total should hover around 1.0.

Retries push the ratio above 1.0 because each retried request is an additional upstream request. As the ratio climbs, retries consume an increasing fraction of upstream capacity. Thresholds:

• Around 1.3: meaningful retry activity. Investigate.
• Above 2.0: a retry storm. Retries are amplifying load to the point where they accelerate the failure they were meant to mask.

A ratio above 1.0 is not always pathological. Traffic mirroring or shadow traffic legitimately sends additional upstream requests without downstream volume. Hedging (speculative retries fired before the original request completes) also inflates the ratio. The diagnostic question is whether retries are inflating it.

flowchart TD
    A[Upstream partial failure] --> B[Some requests fail]
    B --> C[Envoy fires retries]
    C --> D[Extra load on struggling upstream]
    D --> E[More requests fail]
    E --> C
    D --> F[upstream_rq_total / downstream_rq_total climbs]
    F --> G[Retry storm]

The cascade feeds itself. Extra retry load makes the upstream slower and more error-prone, which triggers more retries, which adds more load. Without intervention, the upstream collapses entirely and every proxy retrying to it amplifies the cascade.

Common causes

Quick checks

These commands are read-only. Substitute your cluster name and stat prefix.

# Check the upstream-to-downstream ratio for a cluster
curl -s http://localhost:9901/stats | grep -E 'cluster.my_cluster.upstream_rq_total|http.my_hcm.downstream_rq_total'

# Inspect retry-specific stats
curl -s http://localhost:9901/stats | grep 'upstream_rq_retry'

# Check whether the retry circuit breaker is open
curl -s http://localhost:9901/stats | grep 'circuit_breakers.*rq_retry_open'

# Check for retry budget exhaustion
curl -s http://localhost:9901/stats | grep -E 'upstream_rq_retry_overflow|upstream_rq_retry_limit_exceeded'

# Confirm upstream error rate
curl -s http://localhost:9901/stats | grep -E 'cluster.my_cluster.upstream_rq_(2xx|5xx)'

# Confirm upstream latency trend
curl -s http://localhost:9901/stats | grep 'cluster.my_cluster.upstream_rq_time'

In Istio sidecar mode, the admin port is typically 15000 and the health endpoint is 15021. Adjust accordingly.

The two derived ratios that matter most:

# Retry fraction of upstream traffic:
#   rate(upstream_rq_retry) / rate(upstream_rq_total)
# Above 0.1 warrants investigation; above 0.3 is a retry storm.

# Retry effectiveness:
#   rate(upstream_rq_retry_success) / rate(upstream_rq_retry)
# Below 0.5 means retries are adding load without helping.

How to diagnose it

1. Confirm the upstream-to-downstream ratio is climbing. Pull both counters, compute the delta over a fixed window, and divide. A climbing ratio with retries as the only plausible source confirms amplification.

2. Check the retry stats. upstream_rq_retry gives you the absolute retry volume. upstream_rq_retry_success tells you whether retries are accomplishing anything. If retry volume is high and retry success is low, retries are pure overhead on a failing upstream.

3. Check the retry budget and circuit breaker state. circuit_breakers.<priority>.rq_retry_open is a binary gauge: 1 means the retry budget is at capacity and additional retries are being dropped via upstream_rq_retry_overflow. A tripped retry breaker means the system has already exhausted its retry headroom.

4. Distinguish retry storm from mirroring or hedging. If the ratio is above 1.0 but upstream_rq_retry is low and upstream_rq_5xx is flat, look for shadow traffic or hedging configuration. Both legitimately inflate the upstream rate without retries.

5. Identify the original failure. The retry storm is the amplifier, not the cause. Look at upstream_rq_5xx, upstream_rq_rx_reset, upstream_rq_tx_reset, upstream_rq_timeout, and upstream_rq_per_try_timeout to find what is failing on the upstream side. Response flags in access logs (UF, UC, UT, UO) tell you why Envoy is generating retries.

6. Confirm with URX in access logs. The URX response flag means the upstream retry limit was exceeded. Combined with a climbing retry ratio, it confirms the policy is firing and exhausting retries against an upstream that is not recovering.

Metrics and signals to monitor

Fixes

Stop the cascade mid-incident

When the ratio is above 2.0 and the upstream is collapsing, the fastest mitigation is to stop adding load. Disable retries for the affected cluster.

Envoy exposes a runtime kill switch, upstream.use_retry, that disables retries across the proxy via a percentage-based runtime setting. Pushing that to zero stops new retries immediately.

If you cannot push a runtime override, an xDS config push that removes or tightens the retry policy for the affected route achieves the same effect. The tradeoff is direct: clients will see the original errors instead of masked retries, but the upstream stops receiving amplified load and has a chance to recover.

Disabling retries is a temporary measure. Once the upstream recovers, restore retries with a budget.

Fix the original failure

A retry storm is a symptom. The root cause is whatever made the upstream start failing. Common patterns:

• Upstream saturation cascade: backend is slow, connections are held longer, the pool fills, the pending queue overflows, 503s with UO start. Fix the upstream latency.
• Connection pool exhaustion: too few connections for the workload, or the upstream is too slow to recycle them. See Envoy connection pool exhaustion: a slow upstream that fills the pool.
• Outlier detection mass ejection: too-aggressive outlier detection has emptied the cluster, leaving few hosts to absorb load. See Envoy outlier detection mass ejection: when passive health checks empty a cluster.
• No healthy upstream: the cluster has no host to route to, so retries cannot succeed. See Envoy no healthy upstream: the 503 when a cluster has no host to route to.

Tune the retry policy

After the incident, review the retry configuration that allowed amplification:

• Retry only on retriable conditions. retry_on: 5xx retries on all 5xx, including codes like 501 that will never succeed on retry. Prefer scoping to gateway-error,connect-failure,refused-stream,reset, or to specific retriable codes.
• Cap retries per request. Three or more retries per request allows 3x amplification under failure. One or two is usually enough.
• Keep jittered exponential backoff. Ensure retry_back_off is explicitly configured with a reasonable base and max interval. A route-level policy that omits backoff lets retries fire with no delay between attempts.
• Do not retry non-idempotent methods. Envoy does not know whether a request is idempotent. Restrict retry_on to safe methods or endpoints.

Prevention

Use retry budgets instead of static limits

Envoy recommends retry budgets over a static max_retries circuit breaker. A retry budget scales the allowed concurrent retries as a percentage of active request volume rather than a fixed count. The defaults are budget_percent of 20.0 and min_retry_concurrency of 3.

A retry budget caps the worst-case amplification. With a 20% budget, even under total failure the proxy cannot send more than 1.2x its active request volume as retries. Compare that to a static max_retries of, say, 1000, which allows 1000 concurrent retries regardless of how much real traffic is flowing.

There is a known subtlety. Retry budget accounting is not perfectly consistent across protocols: retries in backoff are counted as active but are not always included in the limit calculation, and HTTP/1, HTTP/2, and HTTP/3 upstreams count the just-failed request differently when the retry decision is made. The practical effect is that even with a high budget, retries can overflow in low-concurrency situations. Set min_retry_concurrency explicitly to avoid impractically low limits on low-traffic clusters.

Monitor the ratio as a first-class signal

Treat upstream_rq_total / downstream_rq_total as a standard dashboard metric for every cluster with retries enabled. Alert on the thresholds: investigate above 1.3, page above 2.0. Pair it with upstream_rq_retry rate and upstream_rq_retry_success ratio to distinguish amplification from benign mirroring or hedging.

Pair retries with circuit breakers and outlier detection

Retries are not a substitute for circuit breaking. The full defensive stack is:

• Per-cluster circuit breakers (max_connections, max_pending_requests, max_requests) to fast-fail when the upstream cannot absorb more load. See Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests.
• Outlier detection to eject hosts that are failing real traffic. See Envoy health checks vs outlier detection: two systems that eject hosts differently.
• Retry budgets to cap amplification when retries do fire.

Without the first two, retries have nothing to bound them. With all three, a partial upstream failure ejects bad hosts, fast-fails overload, and retries stay within a bounded budget.

How Netdata helps

Netdata surfaces the signals that distinguish a retry storm from ordinary upstream degradation at per-second resolution:

• The cluster.<name>.upstream_rq_total and http.<stat_prefix>.downstream_rq_total counters, tracked together, make the upstream-to-downstream ratio visible without manual sampling from the admin endpoint.
• upstream_rq_retry, upstream_rq_retry_success, upstream_rq_retry_overflow, and upstream_rq_retry_limit_exceeded expose retry volume, effectiveness, and budget exhaustion side by side.
• circuit_breakers.<priority>.rq_retry_open shows when the retry breaker has tripped, correlating with the moment retries stop helping.
• Anomaly detection flags unexpected changes in retry rate or ratio before alert thresholds trip, which is useful when the storm builds gradually from a small upstream regression.
• Correlated views across clusters let you see whether a retry storm is localized to one upstream or spreading across multiple clusters sharing a dependency.

Related guides

• Envoy 502 and upstream resets: rx_reset, tx_reset, and mid-response failures
• Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend
• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy connection pool exhaustion: a slow upstream that fills the pool
• Envoy health checks vs outlier detection: two systems that eject hosts differently
• How Envoy actually works in production: a mental model for operators
• Envoy membership_healthy dropping: reading the single most important cluster signal
• Envoy monitoring checklist: the signals every production proxy needs
• Envoy monitoring maturity model: from survival to expert
• Envoy no healthy upstream: the 503 when a cluster has no host to route to
• Envoy outlier detection mass ejection: when passive health checks empty a cluster
• Envoy panic threshold: why traffic routes to unhealthy hosts at 50%