Envoy retry policy tuning: retry_on, budgets, idempotency, and hedging

Retries are one of the few Envoy features that can make an outage measurably worse. A correctly tuned retry policy absorbs transient failures cleanly. A poorly tuned one multiplies upstream load by 2x-3x during a partial failure and accelerates the collapse it was supposed to mask. This article covers the knobs that matter operationally: retry_on conditions, retry budgets, per_try_timeout, and hedging. It also covers the one thing Envoy cannot decide for you: whether a request is safe to repeat.

Envoy does not retry by default. No retry_on, no retries. Retrying is a contract between you and your upstreams about what is safe to repeat, and Envoy cannot infer that contract from the wire. Everything below assumes you have explicitly chosen retry_on conditions and understand the idempotency implications.

A baseline retry rate under roughly 1% of requests is normal and healthy. If retries become a significant fraction of traffic, the policy is either compensating for a sick upstream or amplifying one. The two situations look similar in dashboards and have opposite fixes.

What it is and why it matters

Retry policy is route-level (or virtual-host-level) configuration that tells Envoy when to re-attempt a failed request, how many times, and how to space those attempts. It interacts with three other systems that each have their own limits:

per_try_timeout, which bounds a single attempt.
The global route timeout, which bounds the entire request including all retries.
The retry budget (or the legacy max_retries circuit breaker), which bounds concurrent retries across the whole cluster.

These interact in non-obvious ways. The global route timeout includes all retry attempts. If the global timeout is 3 seconds and the first attempt takes 2.7 seconds, the retry has 0.3 seconds to complete or fail. per_try_timeout, if set longer than the global timeout, is effectively dead because the overall timeout wins first.

The retry budget is the only thing preventing a retry storm from consuming the cluster. Without a budget, num_retries is per-request, and 1000 concurrent failing requests each allowed 3 retries produces 3000 retry attempts hitting an already-degraded upstream. The budget converts that into a cluster-wide concurrency limit.

How it works

flowchart TD
    A[Request enters retry_policy] --> B[Attempt to upstream]
    B --> C{Outcome}
    C -->|Success| D[Return to client]
    C -->|No retry_on match| E[Return response as-is]
    C -->|Matches retry_on| F{Within num_retries?}
    F -->|No| G[Return last error, URX flag]
    F -->|Yes| H{Retry budget available?}
    H -->|No| I[Drop retry, increment retry_overflow]
    H -->|Yes| J[Exponential backoff with jitter]
    J --> B
    I --> G

retry_on accepts a comma-separated list of conditions. The common ones:

5xx: retry on any 5xx response. Implicitly includes connect-failure and refused-stream, so listing them separately is redundant.
gateway-error: retry on 502, 503, 504. Narrower than 5xx.
connect-failure: TCP connect failed. Already a subset of 5xx.
retriable-status-codes: retry on specific status codes listed in retriable_status_codes. More precise than 5xx.
retriable-headers: retry when response headers match configured matchers.
reset: retry when the upstream sent a reset (RST_STREAM or connection close).
reset-before-request: retry only when the upstream reset before any request headers were sent. This is the safe-to-retry variant of reset because the upstream never saw the request body.
refused-stream: upstream refused the stream (HTTP/2 GOAWAY or REFUSED_STREAM).
http3-post-connect-failure: retry HTTP/3 connections that fail after the QUIC handshake.

One subtlety operators miss: a 504 produced by the global route timeout is NOT retried by the 5xx policy. The router filter docs state this explicitly. If you want to retry on overall timeout, set per_try_timeout shorter than the global timeout so individual attempts can fail and retry within the global budget.

num_retries defaults to 1, meaning one retry attempt after the original. Many operators assume the default is higher and are surprised by how few retries actually fire.

Retry backoff is exponential with full jitter. The default base interval is 25ms and the max interval is 10x the base (250ms). The base is configurable via the upstream.base_retry_backoff_ms runtime parameter. Without backoff, retries fire as fast as Envoy can send them, which is exactly what you do not want during a partial failure.

The retry budget lives on the cluster’s circuit breaker configuration, not the route. When configured, it overrides the max_retries circuit breaker entirely. The defaults when a budget is explicitly configured:

budget_percent: 20% of active requests may be retries.
min_retry_concurrency: 3, so small clusters still get a floor of concurrent retries.

If you do not configure a retry budget, the max_retries circuit breaker defaults to 3 concurrent retries for the whole cluster. For any non-trivial cluster, 3 is far too low and legitimate retries get dropped (visible as upstream_rq_retry_overflow). The retry budget is the recommended approach.

Historical note: in versions before v1.18.0, setting retry_budget: {} as an empty message did not activate the budget with defaults. This was fixed in v1.18.0. On older versions you must set at least one sub-field explicitly.

Where it shows up in production

The retry storm is the canonical failure mode. It follows a specific shape:

An upstream has a partial failure (20% of requests failing).
retry_on: 5xx fires on those failures, with num_retries: 3.
Retried requests add load to an already-struggling upstream.
The added load causes more failures.
More failures cause more retries.
Within minutes the upstream sees 2x-3x normal traffic, mostly retries, and collapses entirely.

The distinguishing signal is the ratio upstream_rq_total / downstream_rq_total climbing above 1.5. In a pure upstream failure with no retries, the upstream rate stays flat or drops. It only climbs when retries are adding load.

Hedging makes this worse. The hedge policy only supports hedge_on_per_try_timeout, and it defaults to false. When enabled, a per-try timeout triggers a second attempt in parallel rather than canceling the first. The original timed-out attempt stays alive. The first good response wins. This is good for tail latency in the steady state and bad for load during degradation, because now one logical request can consume two upstream slots indefinitely.

The other production trap is retrying non-idempotent requests. Envoy cannot tell whether a POST to /charge is safe to repeat. If retry_on fires on a 503 returned after the upstream committed a side effect, the client gets a duplicate operation. This is the operator’s responsibility: only enable retries on routes whose upstreams tolerate repetition, or scope retry_on to conditions that guarantee the upstream never saw the request. reset-before-request is the strongest such condition because it only fires when the reset happened before headers were sent upstream.

One more production gotcha: the x-envoy-retry-on request header can override the route’s retry_on value. When honored, a client (or a misconfigured upstream proxy) can inject retries you did not plan for. If you see retry volume you cannot explain from your route config, check whether downstream callers are sending retry headers.

For gRPC workloads, retries are configured via retry_on plus x-envoy-retry-grpc-on semantics.

Tradeoffs and when to use it

Decision	Tradeoff
`retry_on: 5xx` vs `gateway-error`	`5xx` is broader and catches more, including 500s from application bugs that will never succeed on retry. `gateway-error` limits to 502/503/504, which are more likely infrastructure-transient.
`num_retries: 1` vs 3	More retries absorbs more transient failures but multiplies load during partial failures. 1 is the safe default. 3 is aggressive.
Retry budget vs `max_retries`	Budget scales with traffic (20% of 10000 rps is 2000 retry slots). `max_retries` is a fixed cap that is either too low at peak or too high at idle. Budget is the recommended approach.
`per_try_timeout` set vs unset	Unset means each attempt can run up to the global timeout, leaving no room for retries. Set it shorter than the global timeout so retries actually have time to complete.
Hedging on vs off	Hedging improves P99 in the steady state by racing a slow attempt against a fresh one. It inflates upstream load during degradation because the original attempt is not canceled. Use only on idempotent reads.
`retriable-status-codes` on 503	Lets you retry specifically on 503 (often infrastructure-originated) without retrying 500/501 (often application bugs). More precise than `5xx`.

The unifying principle: retry budgets make most other knobs safer. With a tight budget, even an aggressive num_retries cannot amplify load beyond the budget cap. Without a budget, every per-request setting is a potential amplifier.

A reasonable production starting point for an idempotent read path: retry_on: "gateway-error,reset,connect-failure", num_retries: 1, a per_try_timeout at roughly half the global route timeout, and a retry budget of 20%. This absorbs transient infrastructure failures without giving a partial outage room to amplify.

Signals to watch in production

Signal	Why it matters	Warning sign
`upstream_rq_retry / upstream_rq_total`	Fraction of requests that needed at least one retry. Healthy baseline is under 1%.	Sustained above 10% indicates either a sick upstream or an over-aggressive policy.
`upstream_rq_retry_success / upstream_rq_retry`	Retry effectiveness. Are retries actually recovering requests?	Below 50% means retries are adding load without helping.
`upstream_rq_retry_overflow`	Retries dropped because the budget or `max_retries` circuit breaker is full.	Any sustained nonzero value means the system wanted to retry but could not.
`upstream_rq_total / downstream_rq_total`	Retry amplification ratio. Should be near 1.0.	Above 1.5 is meaningful retry activity. Above 2.0 is a retry storm.
`upstream_rq_per_try_timeout`	Individual attempts hitting `per_try_timeout`.	Correlate with `hedge_on_per_try_timeout` to understand hedging load.
`circuit_breakers.default.rq_retry_open`	Retry circuit breaker is open.	1 means the cluster is rejecting retries entirely.

The upstream_rq_retry_backoff_exponential and upstream_rq_retry_backoff_ratelimited counters show which backoff strategy is in use, which is useful when verifying config changes landed.

Response flags in access logs add the “why” behind these counters. The URX flag means upstream retry limit was exceeded. The UT flag means upstream request timeout. These are access-log only, not aggregate stats, so you need a log pipeline to alert on them.

How Netdata helps

Correlate upstream_rq_retry with upstream_rq_5xx on the same cluster. If retry rate tracks error rate with a lag, retries are reacting to failures. If retry rate climbs independently, the policy is mis-scoped or a client is injecting retry headers.
Track the upstream_rq_total to downstream_rq_total ratio as a single derived signal. This is the cleanest retry-amplification indicator and it is not a native Envoy stat.
Watch upstream_rq_retry_overflow alongside circuit_breakers.default.rq_retry_open. The first shows retries being dropped, the second confirms the budget is the cause.
Pair upstream_rq_retry_success with upstream_rq_retry to surface retry effectiveness. Low effectiveness during an incident is a signal to reduce num_retries or tighten retry_on, not increase them.
Monitor upstream_rq_per_try_timeout when hedging is enabled. Per-try timeouts are the only hedging trigger, so this stat directly predicts hedge-induced load.
Use per-second granularity to catch the leading edge of a retry storm before the budget saturates. Retry storms develop over tens of seconds, not minutes.