Envoy retry policy tuning: retry_on, budgets, idempotency, and hedging

Retries are one of the few Envoy features that can make an outage measurably worse. A correctly tuned retry policy absorbs transient failures cleanly. A poorly tuned one multiplies upstream load by 2x-3x during a partial failure and accelerates the collapse it was supposed to mask. This article covers the knobs that matter operationally: retry_on conditions, retry budgets, per_try_timeout, and hedging. It also covers the one thing Envoy cannot decide for you: whether a request is safe to repeat.

Envoy does not retry by default. No retry_on, no retries. Retrying is a contract between you and your upstreams about what is safe to repeat, and Envoy cannot infer that contract from the wire. Everything below assumes you have explicitly chosen retry_on conditions and understand the idempotency implications.

A baseline retry rate under roughly 1% of requests is normal and healthy. If retries become a significant fraction of traffic, the policy is either compensating for a sick upstream or amplifying one. The two situations look similar in dashboards and have opposite fixes.

What it is and why it matters

Retry policy is route-level (or virtual-host-level) configuration that tells Envoy when to re-attempt a failed request, how many times, and how to space those attempts. It interacts with three other systems that each have their own limits:

1. per_try_timeout, which bounds a single attempt.
2. The global route timeout, which bounds the entire request including all retries.
3. The retry budget (or the legacy max_retries circuit breaker), which bounds concurrent retries across the whole cluster.

These interact in non-obvious ways. The global route timeout includes all retry attempts. If the global timeout is 3 seconds and the first attempt takes 2.7 seconds, the retry has 0.3 seconds to complete or fail. per_try_timeout, if set longer than the global timeout, is effectively dead because the overall timeout wins first.

The retry budget is the only thing preventing a retry storm from consuming the cluster. Without a budget, num_retries is per-request, and 1000 concurrent failing requests each allowed 3 retries produces 3000 retry attempts hitting an already-degraded upstream. The budget converts that into a cluster-wide concurrency limit.

How it works

flowchart TD
    A[Request enters retry_policy] --> B[Attempt to upstream]
    B --> C{Outcome}
    C -->|Success| D[Return to client]
    C -->|No retry_on match| E[Return response as-is]
    C -->|Matches retry_on| F{Within num_retries?}
    F -->|No| G[Return last error, URX flag]
    F -->|Yes| H{Retry budget available?}
    H -->|No| I[Drop retry, increment retry_overflow]
    H -->|Yes| J[Exponential backoff with jitter]
    J --> B
    I --> G

retry_on accepts a comma-separated list of conditions. The common ones:

• 5xx: retry on any 5xx response. Implicitly includes connect-failure and refused-stream, so listing them separately is redundant.
• gateway-error: retry on 502, 503, 504. Narrower than 5xx.
• connect-failure: TCP connect failed. Already a subset of 5xx.
• retriable-status-codes: retry on specific status codes listed in retriable_status_codes. More precise than 5xx.
• retriable-headers: retry when response headers match configured matchers.
• reset: retry when the upstream sent a reset (RST_STREAM or connection close).
• reset-before-request: retry only when the upstream reset before any request headers were sent (added in 1.31.0). This is the safe-to-retry variant of reset because the upstream never saw the request body.
• refused-stream: upstream refused the stream (HTTP/2 GOAWAY or REFUSED_STREAM).
• http3-post-connect-failure: retry HTTP/3 connections that fail after the QUIC handshake.

One subtlety operators miss: a 504 produced by the global route timeout is NOT retried by the 5xx policy. The router filter docs state this explicitly. If you want to retry on overall timeout, set per_try_timeout shorter than the global timeout so individual attempts can fail and retry within the global budget.

num_retries defaults to 1, meaning one retry attempt after the original. Many operators assume the default is higher and are surprised by how few retries actually fire.

Retry backoff is exponential with full jitter. The default base interval is 25ms and the max interval is 10x the base (250ms). The base is configurable via the upstream.base_retry_backoff_ms runtime parameter. Without backoff, retries fire as fast as Envoy can send them, which is exactly what you do not want during a partial failure.

The retry budget lives on the cluster’s circuit breaker configuration, not the route. When configured, it overrides the max_retries circuit breaker entirely. The defaults when a budget is explicitly configured:

• budget_percent: 20% of active requests may be retries.
• min_retry_concurrency: 3, so small clusters still get a floor of concurrent retries.

If you do not configure a retry budget, the max_retries circuit breaker defaults to 3 concurrent retries for the whole cluster. For any non-trivial cluster, 3 is far too low and legitimate retries get dropped (visible as upstream_rq_retry_overflow). The retry budget is the recommended approach.

Historical note: in versions before 1.18.0, setting retry_budget: {} as an empty message did not activate the budget with defaults (the 1.18.0 changelog notes “retry budgets will now set default values for xDS configurations”). On older versions you must set at least one sub-field explicitly.

Where it shows up in production

The retry storm is the canonical failure mode. It follows a specific shape:

1. An upstream has a partial failure (20% of requests failing).
2. retry_on: 5xx fires on those failures, with num_retries: 3.
3. Retried requests add load to an already-struggling upstream.
4. The added load causes more failures.
5. More failures cause more retries.
6. Within minutes the upstream sees 2x-3x normal traffic, mostly retries, and collapses entirely.

The distinguishing signal is the ratio upstream_rq_total / downstream_rq_total climbing above 1.5. In a pure upstream failure with no retries, the upstream rate stays flat or drops. It only climbs when retries are adding load.

Hedging makes this worse. The hedge policy only supports hedge_on_per_try_timeout, and it defaults to false. When enabled, a per-try timeout triggers a second attempt in parallel rather than canceling the first. The original timed-out attempt stays alive. The first good response wins. This is good for tail latency in the steady state and bad for load during degradation, because now one logical request can consume two upstream slots indefinitely.

The other production trap is retrying non-idempotent requests. Envoy cannot tell whether a POST to /charge is safe to repeat. If retry_on fires on a 503 returned after the upstream committed a side effect, the client gets a duplicate operation. This is the operator’s responsibility: only enable retries on routes whose upstreams tolerate repetition, or scope retry_on to conditions that guarantee the upstream never saw the request. reset-before-request is the strongest such condition because it only fires when the reset happened before headers were sent upstream.

One more production gotcha: the x-envoy-retry-on request header can override the route’s retry_on value. The header defines its own retry policy and is honored even when the route has no retry_policy; all retries are additionally gated by the runtime key upstream.use_retry (enabled by default). When honored, a client (or a misconfigured upstream proxy) can inject retries you did not plan for. If you see retry volume you cannot explain from your route config, check whether downstream callers are sending retry headers.

For gRPC workloads, retries are configured via retry_on plus x-envoy-retry-grpc-on semantics. The gRPC retry decision is made in the response-headers phase only: gRPC status codes carried in trailers never trigger retry logic (router filter docs state this explicitly), so a trailers-only grpc-status failure is counted as an error but is not retried.

Tradeoffs and when to use it

The unifying principle: retry budgets make most other knobs safer. With a tight budget, even an aggressive num_retries cannot amplify load beyond the budget cap. Without a budget, every per-request setting is a potential amplifier.

A reasonable production starting point for an idempotent read path: retry_on: "gateway-error,reset,connect-failure", num_retries: 1, a per_try_timeout at roughly half the global route timeout, and a retry budget of 20%. This absorbs transient infrastructure failures without giving a partial outage room to amplify.

Signals to watch in production

The upstream_rq_retry_backoff_exponential and upstream_rq_retry_backoff_ratelimited counters show which backoff strategy is in use, which is useful when verifying config changes landed.

Response flags in access logs add the “why” behind these counters. The URX flag means upstream retry limit was exceeded. The UT flag means upstream request timeout. These are access-log only, not aggregate stats, so you need a log pipeline to alert on them.

How Netdata helps

• Correlate upstream_rq_retry with upstream_rq_5xx on the same cluster. If retry rate tracks error rate with a lag, retries are reacting to failures. If retry rate climbs independently, the policy is mis-scoped or a client is injecting retry headers.
• Track the upstream_rq_total to downstream_rq_total ratio as a single derived signal. This is the cleanest retry-amplification indicator and it is not a native Envoy stat.
• Watch upstream_rq_retry_overflow alongside circuit_breakers.default.rq_retry_open. The first shows retries being dropped, the second confirms the budget is the cause.
• Pair upstream_rq_retry_success with upstream_rq_retry to surface retry effectiveness. Low effectiveness during an incident is a signal to reduce num_retries or tighten retry_on, not increase them.
• Monitor upstream_rq_per_try_timeout when hedging is enabled. Per-try timeouts are the only hedging trigger, so this stat directly predicts hedge-induced load.
• Use per-second granularity to catch the leading edge of a retry storm before the budget saturates. Retry storms develop over tens of seconds, not minutes.

Related guides

• Envoy 502 and upstream resets: rx_reset, tx_reset, and mid-response failures
• Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend
• Envoy 504 upstream timeout: upstream_rq_timeout, per-try timeouts, and the UT flag
• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy connection pool exhaustion: a slow upstream that fills the pool
• Envoy health checks vs outlier detection: two systems that eject hosts differently
• How Envoy actually works in production: a mental model for operators
• Envoy membership_healthy dropping: reading the single most important cluster signal
• Envoy monitoring checklist: the signals every production proxy needs
• Envoy monitoring maturity model: from survival to expert
• Envoy no healthy upstream: the 503 when a cluster has no host to route to
• Envoy outlier detection mass ejection: when passive health checks empty a cluster