$ guides / envoy / envoy-503-uo-circuit-breaker ▌

Operations Guides

Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend

You see 503 responses in access logs with the UO response flag. The flag is what pins the cause: UO means upstream overflow. A circuit breaker fast-failed the request locally before forwarding. The backend did not refuse the connection, did not time out, and may not be aware the request existed. Envoy decided the cluster was at capacity and returned an immediate 503.

This is the most commonly misdiagnosed Envoy 503. Operators chase the backend, restart pods, scale the upstream, and the errors persist because the upstream is not the thing that is broken. The circuit breaker is working correctly. It is protecting an upstream that is either slow, under-provisioned, or receiving more concurrent load than its pool can absorb. The fix is to address the upstream saturation or right-size the breaker limit. Restarting Envoy will not help.

What this means

UO is set when Envoy’s router filter or connection pool rejects a request because a circuit breaker threshold was exceeded. When this happens, Envoy may set the x-envoy-overloaded: true response header on the 503, which is the most reliable client-visible signal that a breaker tripped.

Circuit breakers are per-cluster and per-priority (default and high). Envoy tracks four breaker types, each with its own gauge and counter:

Breaker	Gauge (0 or 1)	Counter
Connections	`circuit_breakers.<priority>.cx_open`	`upstream_cx_overflow`
Pending requests	`circuit_breakers.<priority>.rq_pending_open`	`upstream_rq_pending_overflow`
Active requests (HTTP/2 `max_requests`)	`circuit_breakers.<priority>.rq_open`	`upstream_rq_active_overflow` (Envoy 1.38.0+)
Retries	`circuit_breakers.<priority>.rq_retry_open`	`upstream_rq_retry_overflow`

The default thresholds are max_connections=1024, max_pending_requests=1024, max_requests=1024, and max_retries=3. These are per-cluster, per-priority. Hitting them during normal traffic is unusual and usually means the upstream is slow or the workload outgrew the configured limits.

The trap: response flags are access-log only. They are not exposed as aggregate Prometheus stats. If your alerting is based purely on counters and gauges, you will see a rising 503 rate without knowing why. The %RESPONSE_FLAGS% field in the access log is the only thing that distinguishes a circuit-breaker 503 from a no-healthy-upstream 503, a no-route 503 (NR), or a timeout (UT).

The failure sequence is the upstream saturation cascade:

flowchart TD
    A[Upstream responds slowly] --> B[Connections held longer]
    B --> C[Connection pool fills]
    C --> D[New requests queue in pending buffer]
    D --> E[Pending buffer hits max_pending_requests]
    E --> F[Circuit breaker trips]
    F --> G[New requests get 503 UO]

Envoy looks broken from the outside, but it is protecting itself and the upstream from collapse.

Common causes

Cause	What it looks like	First thing to check
Slow upstream saturating the pool	`upstream_rq_time` P99 climbing before the 503s start; `upstream_cx_active` near `max_connections`; `pending_active` growing	`upstream_rq_time` histogram trend
Breaker limits too low for the workload	Breaker trips during normal traffic; no latency increase preceding it; `cx_open` or `rq_pending_open` flaps	Compare configured limits against observed peak `upstream_cx_active`
Retry amplification	`upstream_rq_retry` rate climbing; `upstream_rq_total` significantly exceeds `downstream_rq_total`; `retry_overflow` incrementing	`upstream_rq_retry / upstream_rq_total` ratio
HTTP/1.1 pending queue behavior	Pending overflow with low `upstream_cx_active` on HTTP/1.1 backends; HTTP/2 multiplexes, so it rarely fills the pending queue	`upstream_cx_http1_total` vs `upstream_cx_http2_total`
Traffic spike exceeding capacity	`downstream_rq_total` jumps; breaker trips within seconds; recovers when traffic subsides	`downstream_rq_total` rate vs baseline

Quick checks

# Confirm the flag is UO in access logs (pattern depends on your log format; Istio sidecar logs to stdout)
grep ' 503 UO ' /var/log/envoy/access.log | tail -20

# Check which breaker gauges are open (0=closed, 1=open)
curl -s http://localhost:9901/stats | grep 'circuit_breakers.*_open'

# Check pending overflow counter (requests rejected by the pending breaker)
curl -s http://localhost:9901/stats | grep 'upstream_rq_pending_overflow'

# Check pending queue depth (leading indicator before overflow)
curl -s http://localhost:9901/stats | grep 'upstream_rq_pending_active'

# Check active connections against max_connections
curl -s http://localhost:9901/stats | grep 'upstream_cx_active'

# Check upstream latency (is the backend slow?)
curl -s http://localhost:9901/stats | grep 'upstream_rq_time'

# Verify upstream hosts are still healthy (UO with stable membership = saturation, not dead backend)
curl -s http://localhost:9901/stats | grep -E 'membership_healthy|membership_total'

# Check retry activity (retries add load and can trip the retry breaker)
curl -s http://localhost:9901/stats | grep 'upstream_rq_retry'

In Istio sidecar mode, the admin port is 15000, not 9901, and the health endpoint is 15021.

How to diagnose it

Confirm the flag. Grep access logs for UO alongside 503. If you see 503 without UO, or with UF, NR, or UT, this is a different failure mode.
Identify which breaker tripped. Check the circuit_breakers gauges. cx_open=1 means the connection breaker tripped (too many concurrent connections). rq_pending_open=1 means the pending-request breaker tripped (queue full). rq_open=1 means the max_requests breaker tripped (HTTP/2 concurrent stream limit). Each points to a different saturation mode.
Confirm upstream hosts are healthy. Check membership_healthy and membership_total. If the ratio is stable and healthy hosts exist, the backend is up. Envoy is protecting it from overload, not reporting it as dead. This is the key distinction from a genuine upstream outage.
Check upstream latency. Look at upstream_rq_time. If P50 or P99 is elevated relative to baseline, the backend is slow. This is the most common root cause: slow responses hold connections longer, the pool fills, the pending queue grows, the breaker trips.
Check pool saturation. Compare upstream_cx_active against max_connections. If active connections are near the limit, the pool is exhausted. Check upstream_rq_pending_active. Any sustained nonzero value means requests are queuing, which is the precursor to overflow.
Check for retry amplification. If upstream_rq_retry is a significant fraction of upstream_rq_total (above 0.1 is notable, above 0.3 is a storm), retries are inflating the load that trips the breaker.
Verify the version-specific counter. On Envoy 1.38.0 and later, max_requests overflow may be counted in upstream_rq_active_overflow, not upstream_rq_pending_overflow. If you are on a recent version, pending_overflow is zero, and you still see UO with rq_open=1, check upstream_rq_active_overflow.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`upstream_rq_pending_active`	Leading indicator before overflow	Any sustained nonzero value
`upstream_rq_pending_overflow`	Direct count of requests rejected by the pending breaker	Rate above 0
`circuit_breakers.<priority>.cx_open`	Connection breaker is tripped, pool is exhausted	Gauge = 1
`circuit_breakers.<priority>.rq_pending_open`	Pending-request breaker is tripped, queue is full	Gauge = 1
`circuit_breakers.<priority>.rq_open`	`max_requests` breaker is tripped (HTTP/2)	Gauge = 1
`upstream_cx_active` vs `max_connections`	How close the pool is to the connection limit	Ratio above 0.8
`upstream_rq_time`	Backend latency; slow backends are the most common root cause	P99 above 2x baseline
`membership_healthy / membership_total`	Distinguishes saturation from dead-backend outages	Stable ratio = saturation; dropping = outage
`upstream_rq_retry / upstream_rq_total`	Retry amplification inflating load	Ratio above 0.1

Fixes

Address the upstream first

The circuit breaker is a symptom, not the bug. If upstream_rq_time is elevated, the backend is slow. Common causes: database contention, GC pauses, disk I/O, downstream dependency saturation in the backend itself, or insufficient backend capacity. Fix the backend and the breaker stops tripping. Increasing the breaker limit without fixing the backend removes the protection and lets the upstream collapse under unbounded load.

Right-size the circuit breaker limits

If the breaker trips during normal traffic with no latency increase (upstream_rq_time is flat), the limits are too low for the workload. Compare the configured max_connections against observed peak upstream_cx_active during healthy operation. Set limits to at least 2x the observed peak to absorb bursts. This is the one case where raising the limit is the correct fix. Do not raise limits reactively during an incident unless you have confirmed the upstream is healthy and the traffic is legitimate.

Tradeoff: higher limits mean Envoy will pile more load onto a struggling upstream before it starts protecting itself. If you raise max_connections from 1024 to 4096 and the upstream is slow, you give the upstream 4x the concurrent load before Envoy intervenes. Only do this if the upstream can handle it.

Reduce retry aggressiveness

If retry amplification is the driver, the retries themselves are causing the load that trips the breaker. Reduce the retry count, narrow the retry_on conditions (avoid retrying on all 5xx), or add a retry budget. The retry breaker (max_retries, default 3) is deliberately low. Raising it allows retry storms.

Note: max_retries limits concurrent retries per cluster, not per-request retry count. With high traffic, 3 concurrent retries is often too low and trips the retry breaker (upstream_rq_retry_overflow), which also produces UO. Check whether the breaker is tripping on the request path or the retry path.

Check HTTP/1.1 vs HTTP/2 behavior

For HTTP/1.1 upstreams, pending requests accumulate when there are not enough upstream connections. The pending queue fills and overflows. For HTTP/2 upstreams, multiplexing means a single connection handles many concurrent streams, so the pending queue only fills when no connection can be established at all. UO from pending overflow is far more common with HTTP/1.1 backends. If your backend is HTTP/1.1, ensure keepalive is enabled and the connection pool is sized appropriately.

Prevention

Alert on pending queue growth, not overflow. By the time upstream_rq_pending_overflow increments, users are already getting 503s. Alerting on upstream_rq_pending_active growth gives advance warning.
Enable track_remaining on circuit breakers. Set track_remaining: true to expose remaining_cx, remaining_pending, and similar headroom gauges. These are disabled by default but give advance warning before the breaker trips.
Right-size limits based on observed peaks. Set limits to at least 2x the peak upstream_cx_active observed during healthy operation. Re-evaluate after traffic growth or architecture changes.
Monitor the retry amplification ratio. Track upstream_rq_retry / upstream_rq_total as a standard dashboard metric. A rising ratio predicts breaker trips.
Capture response flags in your log pipeline. Response flags are access-log only. Without log processing, you cannot distinguish UO from UF, NR, or UT in aggregate dashboards.

How Netdata helps

Per-second granularity on circuit_breakers gauges and upstream_rq_pending_overflow catches breaker trips as they happen, not on the next scrape interval.
Correlate upstream_rq_pending_active growth against upstream_rq_time to see the saturation cascade form before overflow starts.
membership_healthy alongside circuit breaker state distinguishes a saturated-but-alive upstream from a dead backend in a single view.
The retry amplification ratio (upstream_rq_retry / upstream_rq_total) is visible alongside breaker state, so retry-driven trips are obvious without a separate dashboard.