Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests

You see cx_open=1 or rq_pending_open=1 on a production cluster. Access logs show 503 responses tagged with the UO response flag. Clients receive fast-failed requests, sometimes with an x-envoy-overloaded header. The circuit breaker gauges are binary: 0 means the breaker has headroom and can admit more work, 1 means it is at capacity and rejecting. Each gauge is scoped per-cluster and per-priority (default or high), so you need the right cluster and priority combination to read the signal correctly.

The common reflex is to raise the limit. This is almost always wrong. When a circuit breaker opens, Envoy is protecting the upstream from load it cannot handle. Raising the limit removes the protection without addressing the cause, and the next failure involves an unprotected upstream collapsing completely. The breaker is a symptom. The upstream is the cause.

What this means

Envoy exposes five circuit breaker open gauges per cluster, per priority level:

When any gauge reads 1, Envoy has hit the configured limit for that resource type and is fast-failing new requests locally instead of forwarding them upstream. The client receives HTTP 503, the access log records response flag UO (UpstreamOverflow), and Envoy sets the x-envoy-overloaded response header.

For gRPC requests, the x-envoy-overloaded HTTP response header is set alongside the gRPC status headers, so gRPC clients can also detect circuit breaker rejections by inspecting this header. The gRPC status is derived from the HTTP code (503 maps to UNAVAILABLE).

The gauge naming tripped up operators for years. Older docs described the value as “closed (0) or open (1),” which implied traditional circuit breaker semantics. The corrected meaning: 0 means “has capacity, can admit more,” 1 means “at capacity, will reject.” Operators sometimes read cx_open=0 as “tripped” when it means the opposite.

The cascade from upstream slowness to breaker trip follows a predictable path:

flowchart TD
    A["Upstream latency rises"] -->|holds connections longer| B["upstream_cx_active climbs"]
    B -->|approaches max_connections| C["cx_open = 1"]
    A -->|no free connection| D["pending_active grows"]
    D -->|hits max_pending_requests| E["rq_pending_open = 1"]
    C -->|fast-fail 503| F["flag UO + x-envoy-overloaded"]
    E -->|fast-fail 503| F

There is no graceful degradation between “full” and “rejecting.” Once the limit is hit, the transition to 503 is immediate. The pending queue (upstream_rq_pending_active) is the critical leading indicator: it gives you minutes of warning before overflow starts.

Common causes

Quick checks

Run these read-only commands to inspect the current state. The admin port is 9901 for standalone Envoy and 15000 for Istio sidecar.

# Check all circuit breaker gauges for every cluster and priority
curl -s http://localhost:9901/stats | grep 'circuit_breakers'

# Check pending queue depth and overflow counter
curl -s http://localhost:9901/stats | grep -E 'upstream_rq_pending_(active|overflow)'

# Check upstream active connections against max_connections
curl -s http://localhost:9901/stats | grep 'upstream_cx_active'

# Check upstream latency histogram for the affected cluster
curl -s http://localhost:9901/stats | grep 'upstream_rq_time'

# Check cluster membership (is this also a host health issue?)
curl -s http://localhost:9901/stats | grep -E 'membership_(healthy|total)'

# Check retry volume
curl -s http://localhost:9901/stats | grep -E 'upstream_rq_retry$|upstream_rq_retry_overflow'

# Confirm UO flag in access logs (path varies by deployment)
# Tail first to avoid scanning the entire file
tail -5000 /var/log/envoy/access.log | grep 'UO' | tail -20

# For JSON access logs (common in Kubernetes, output to stdout):
# tail -5000 /var/log/envoy/access.log | jq 'select(.response_flags | test("UO"))'

How to diagnose it

1. Identify which breaker is open and on which priority. The circuit_breakers stat output includes both default and high priority. Most traffic uses default. If high priority traffic is tripping, check whether you have priority-specific configuration or a retry policy using high priority.

2. Check upstream latency before the trip. Pull upstream_rq_time and look at the trend. If P99 was climbing before cx_open or rq_pending_open flipped to 1, the upstream is slow and the breaker is doing its job.

3. Compare upstream_cx_active to max_connections. If upstream_cx_active is at or near max_connections, the connection pool is exhausted. Note that upstream_cx_active can briefly exceed max_connections during the window between a new connection being initiated and the breaker checking the gauge. This is expected behavior, not a leak.

4. Check upstream_rq_pending_active growth. This is the leading indicator. If pending_active is growing before overflow starts, the connection pool is becoming a bottleneck. The queue fills linearly until max_pending_requests, then overflow begins immediately.

5. Confirm via access logs. Look for the UO response flag in the %RESPONSE_FLAGS% field. Multiple flags can appear simultaneously (for example, UC,URX).

6. Check retry amplification. If upstream_rq_total is significantly higher than downstream_rq_total, retries are inflating load. A retry-to-total ratio above 0.1 warrants investigation.

7. Rule out version-specific behavior. Two version changes are known to cause confusion:

• Envoy 1.14.x: The HTTP/1 and HTTP/2 connection pool code was merged. Clusters that only set max_connections but relied on the default max_requests=1024 saw sudden upstream_rq_pending_overflow because max_requests enforcement was activated for HTTP/1.1 pools. The fix is to explicitly set max_requests and max_pending_requests to values appropriate for the workload.

• Envoy 1.38.0: A new upstream_rq_active_overflow counter was added. Previously, when the max_requests circuit breaker was exhausted, the condition incorrectly incremented upstream_rq_pending_overflow. If you are on 1.38 or later, check upstream_rq_active_overflow for max_requests attribution. You can preserve legacy behavior with the runtime flag envoy.reloadable_features.skip_pending_overflow_count_on_active_rq set to false.

8. Check for overflow without open gauges. On some Envoy versions, upstream_rq_pending_overflow can increment and 503 UO responses appear while cx_open, rq_open, rq_pending_open, and cx_pool_open all remain 0. This happens because the counter increments at two code paths: the cluster-level pending limit and the per-connection stream limit. The per-connection stream limit can be hit before the cluster-level circuit breaker threshold. The open gauges only flip when the cluster-level limit is reached. This is by design, not a bug.

Metrics and signals to monitor

If you enable track_remaining: true in the circuit breaker thresholds configuration, Envoy exposes remaining_cx, remaining_pending, remaining_rq, and remaining_retries gauges that show headroom before the breaker trips. These are off by default and most teams never enable them.

Fixes

When the upstream is slow (the common case)

Address the root cause. If the upstream has a database contention issue, GC pause problem, or resource saturation, fix that. Scale the upstream horizontally if possible. Do not raise the circuit breaker limit as the primary response. The breaker opened because the upstream cannot handle more concurrent work. Raising the limit lets more requests pile up against an already-saturated backend, which makes the eventual failure worse.

When limits are genuinely too low

If upstream_cx_active is consistently near max_connections during normal traffic with no latency spike, the limit is too low for the workload. In this case, raising the limit is correct. Set circuit breaker limits to at least 2x the peak upstream_cx_active observed during normal operation. This accommodates latency spikes without tripping on transient events.

Also check whether a recent Envoy upgrade changed enforcement behavior (see the 1.14.x note above). A sudden breaker trip after an upgrade with no traffic change often points to a version behavior shift.

When retry amplification is the cause

If the retry ratio is high, the retries themselves may be driving the load that trips the breaker. Reduce retry aggressiveness (fewer retry attempts, narrower retry_on conditions), check the retry budget, and consider whether retries on non-idempotent endpoints are appropriate.

Temporary mitigation during an incident

If you need to buy time during an active incident, the least-bad temporary measure is to scale the upstream, not to raise breaker limits. If scaling is not immediately possible, raising max_connections and max_pending_requests can absorb a transient spike, but you must investigate the root cause before the next cycle.

Prevention

• Enable track_remaining: true on production clusters. The remaining_cx and remaining_pending gauges give you headroom visibility before the breaker trips. Without them, you only learn about saturation when 503s start.
• Alert on upstream_rq_pending_active growth, not just overflow. The pending queue is the leading indicator. By the time upstream_rq_pending_overflow increments, users are already failing.
• Set breaker limits based on observed baselines, not defaults. The defaults (1024 for connections, 1024 for pending requests, 3 for retries) are starting points. Measure your steady-state upstream_cx_active and upstream_rq_pending_active and set limits with 2x headroom.
• Monitor retry ratios. A retry-to-total ratio above 0.1 sustained is an early warning of retry amplification that can cascade into breaker trips.
• Watch for version-specific behavior changes during upgrades. The 1.14.x max_requests enforcement change and the 1.38.0 counter split both caused confusion in production. Review the Envoy version history for circuit breaker changes before upgrading.

How Netdata helps

• Per-second granularity on cx_open and rq_pending_open gauge transitions catches breaker state changes that 15-30s scrape intervals miss entirely. Correlating breaker state with upstream_rq_time latency histograms in the same view shows immediately whether the upstream slowed before the breaker opened.
• upstream_rq_pending_active as a leading indicator, monitored with ML anomaly detection, surfaces queue growth before overflow produces user-visible 503s.
• Correlating breaker trips with membership_healthy drops and retry ratios distinguishes upstream saturation (hosts healthy but slow) from upstream failure (hosts being ejected) or retry amplification, each of which needs a different response.

Related guides

• Envoy upstream_rq_pending_overflow: the pending queue fills and 503s begin
• Envoy membership_healthy dropping: reading the single most important cluster signal
• Envoy upstream_cx_connect_fail: failed TCP connections to upstream hosts
• Envoy no healthy upstream: the 503 when a cluster has no host to route to
• Envoy outlier detection mass ejection: when passive health checks empty a cluster
• Envoy panic threshold: why traffic routes to unhealthy hosts at 50%
• How Envoy actually works in production: a mental model for operators
• Envoy monitoring checklist: the signals every production proxy needs