Envoy upstream_rq_pending_overflow: the pending queue fills and 503s begin

You see cluster.<name>.upstream_rq_pending_overflow incrementing on one or more clusters. Access logs show 503 with response flag UO. Clients get fast-fail 503s, not timeouts. The curve is cliff-edge: requests were flowing fine, and now a slice of them are rejected with no upstream attempt at all.

This is Envoy’s pending request circuit breaker doing exactly what it was configured to do. Requests arrive, there is no available upstream connection to attach them to, so they wait in upstream_rq_pending_active. When that queue hits max_pending_requests, Envoy stops queueing and starts rejecting. The overflow counter is the rejected count.

The trap is alerting on the 503s. By the time overflow is non-zero, the queue is already full and users are already failing. The leading indicator is upstream_rq_pending_active growth. Alert on that and you get minutes of warning before the first 503 UO.

What this means

upstream_rq_pending_overflow is a per-cluster counter. It increments each time Envoy cannot queue a request because the pending queue is at max_pending_requests. The rejected request returns immediately as 503 with the UO response flag in the access log.

This is distinct from a timeout. Overflow means Envoy did not even try to send the request upstream. There was no connection slot and no queue slot. A timeout (UT flag) means Envoy tried, the upstream was too slow, and the request budget expired. The two failure modes have different root causes and different fixes.

The failure curve is cliff-edge, not gradual. upstream_rq_pending_active fills linearly toward max_pending_requests. Once the limit is hit, every additional request that cannot be attached to a connection is rejected immediately. There is no soft degradation between “queue is full” and “requests are being dropped”.

flowchart TD
    A[Upstream responds slowly] --> B[Connections held longer]
    B --> C[upstream_cx_active approaches max_connections]
    C --> D[New requests wait in upstream_rq_pending_active]
    D --> E{pending_active at max_pending_requests?}
    E -->|No| D
    E -->|Yes| F[upstream_rq_pending_overflow increments]
    F --> G[503 with response flag UO]

The cascade is always the same shape: upstream latency rises, connections are held longer, the pool fills, requests queue, the queue fills, overflow starts. If you see overflow without a preceding latency increase, the circuit breaker limits are too low for normal traffic.

For HTTP/2 upstreams, streams multiplex over established connections, so the pending queue is mainly hit during pool warm-up or when the upstream’s max_concurrent_streams is exhausted. In practice, upstream_rq_pending_overflow fires most often on HTTP/1.1 clusters where each request needs its own connection slot.

Default circuit breaker values are max_connections=1024, max_pending_requests=1024, max_requests=1024, max_retries=3. These are per-cluster, per-priority (default and high). Worker threads share these limits with eventual consistency, so under concurrent load the breaker behavior is non-deterministic. Sending the same request volume can sometimes trip the breaker and sometimes not.

Common causes

Quick checks

All commands assume the admin interface on localhost:9901 (Istio sidecar: localhost:15000). Replace <cluster> with your cluster name.

# Confirm overflow is actively incrementing (run twice, 10s apart)
curl -s http://localhost:9901/stats | grep 'upstream_rq_pending_overflow'

# Leading indicator: queue depth
curl -s http://localhost:9901/stats | grep 'upstream_rq_pending_active'

# Whether the pending circuit breaker is currently open
curl -s http://localhost:9901/stats | grep 'circuit_breakers.default.rq_pending_open'

# Connection pool saturation
curl -s http://localhost:9901/stats | grep -E 'upstream_cx_active|cx_open'

# Upstream latency (is the backend actually slow?)
curl -s http://localhost:9901/stats/prometheus | grep 'envoy_cluster_upstream_rq_time'

# Retry amplification
curl -s http://localhost:9901/stats | grep -E 'upstream_rq_retry|upstream_rq_total|downstream_rq_total'

# Confirm response flags in access logs show UO (not UF, UT, or NR)
# Location varies; in Istio sidecar mode, check the sidecar container's stdout

The single most important check is the pair upstream_rq_pending_active and upstream_rq_time on the same cluster. If pending_active is growing and upstream_rq_time is elevated, the upstream is slow and the queue is the symptom. If pending_active is growing and latency is normal, the limits are wrong.

How to diagnose it

1. Confirm the flag is UO. Pull access logs for the affected cluster and verify the response flag. UO is circuit breaker overflow. UF is connection failure, UT is timeout, NR is no route. Each has a different fix.

2. Pull the saturation sequence. On the affected cluster, look at upstream_rq_time, upstream_cx_active, upstream_rq_pending_active, and upstream_rq_pending_overflow over the same time window. The order tells you the root cause:

   • Latency rises first, then connections, then pending, then overflow: the upstream is slow.
   • Connections hit the ceiling with normal latency: max_connections is too low.
   • Pending fills with very few connections: max_pending_requests is too low.

3. Check circuit breaker state gauges. circuit_breakers.default.cx_open=1 means the connection breaker is open. circuit_breakers.default.rq_pending_open=1 means the pending breaker is open. These confirm which limit is the binding constraint.

4. Check retry ratio. Compute upstream_rq_total / downstream_rq_total over a 5-minute window. Above 1.3, retries are adding meaningful load. Above 2.0, you are in a retry storm pushing the queue over the edge.

5. Check membership health. If membership_healthy is dropping alongside the overflow, the real problem is upstream host loss, not pool sizing. See the related guide on membership health.

6. Check the Envoy version. If you recently upgraded to v1.14.0 or later, max_requests is now enforced on HTTP/1.1 clusters where it previously was not. On v1.38.0 or later, upstream_rq_active_overflow is reportedly the new authoritative counter for the max_requests code path, and upstream_rq_pending_overflow no longer increments on that path by default.

Metrics and signals to monitor

If you have track_remaining: true enabled on the circuit breaker config, also watch circuit_breakers.default.remaining_pending and remaining_cx. These give direct headroom visibility. The setting is disabled by default.

Fixes

The fixes split by root cause. The wrong reflex is to increase the limits first. The breaker is a symptom. The upstream is usually the cause.

The upstream is slow

This is the most common root cause. Address the backend: database contention, GC pauses, disk I/O, a slow downstream dependency. Use upstream_rq_time and your backend’s own metrics to find the bottleneck.

Do not raise max_pending_requests to mask this. A larger queue means more requests piled up, more memory consumed, and longer effective latency for requests that do eventually get a connection. The clients are already timing out on their side. Raising the queue moves the failure from Envoy’s 503 to the client’s timeout, which is usually worse.

If you need a temporary bridge during an upstream incident, raising max_connections gives you real additional concurrency. A deeper pending queue gives you neither: the same number of connections are still processing at the same speed, and the queue just grows behind them.

max_connections is too low

Compare steady-state upstream_cx_active during normal peak traffic against the configured max_connections. If upstream_cx_active is regularly above 80% of the limit, raise the limit. Keep at least 20% headroom for bursts.

In Istio, this is set in the DestinationRule under trafficPolicy.connectionPool.tcp.maxConnections. A common pitfall is copy-pasting a small value like 10 from a tutorial. That is almost always wrong for production traffic.

max_pending_requests is too low

Same approach. Compare steady-state upstream_rq_pending_active against max_pending_requests. Under normal operation pending_active should be near zero. If it is regularly non-zero and you are not in an upstream-slow incident, the limit is too low.

In Istio, the relevant fields are trafficPolicy.connectionPool.http.http1MaxPendingRequests for HTTP/1.1 and http2MaxRequests for HTTP/2. Check the protocol your cluster actually uses.

Retry amplification

If the retry ratio is the driver, the queue is a victim, not the cause. Reduce retry aggressiveness: lower the retry count, narrow retry_on to genuinely retriable conditions, cap the retry budget via max_retries. During an active incident, consider disabling retries for the affected cluster entirely via a runtime flag or xDS config push. This stops the amplification immediately and lets the upstream recover.

HTTP/1.1 without keepalive

If upstream_cx_total (new connections per second) is high and the reuse ratio upstream_rq_total / upstream_cx_total is near 1.0, every request is opening a new connection. Each connection holds a pool slot for the full TCP and TLS handshake duration. Fix the keepalive policy on the upstream or the cluster config. This is a configuration issue, not a capacity issue.

v1.14.0 behavior change

If you upgraded from a pre-1.14.0 Envoy and suddenly see upstream_rq_pending_overflow on HTTP/1.1 clusters that were previously fine, the cause is the connection pool code merge. max_requests is now enforced on HTTP/1.1 clusters. If you had only set max_connections and left max_requests at the default 1024, that default may now be binding. Either raise max_requests or set it explicitly to match your workload.

Prevention

• Alert on upstream_rq_pending_active growth, not on overflow. The queue depth is the leading indicator. Alerting on overflow means alerting on user-visible failure that is already happening.
• Set circuit breaker limits to at least 2x the peak observed values. This accommodates latency spikes without tripping on transient events.
• Enable track_remaining: true on circuit breakers if you want headroom visibility directly in metrics rather than inferring it from gauges.
• Monitor the retry ratio as a standard dashboard metric. Retries are the most common amplifier of pending queue pressure during partial upstream failures.
• Review every Istio DestinationRule connectionPool setting. Treat any value below a few hundred with suspicion unless the workload is genuinely tiny.
• After any Envoy upgrade, check overflow rates. The v1.14.0 change altered which counter increments on which code path. A previously clean cluster can start showing overflow after an upgrade with no traffic change.

How Netdata helps

• Per-second resolution on upstream_rq_pending_active and upstream_rq_pending_overflow lets you see the queue fill before overflow starts. The cliff edge often plays out in seconds; minute-level scraping misses it entirely.
• Correlating upstream_rq_time, upstream_cx_active, and upstream_rq_pending_active on a single timeline makes the saturation cascade visible. You can see whether latency rose before the queue filled, which tells you whether the root cause is the upstream or the limits.
• Circuit breaker gauges (cx_open, rq_pending_open) appear alongside the counters, so you can confirm which breaker is binding without querying the admin interface.
• Retry stats (upstream_rq_retry, retry_overflow) on the same dashboard distinguish a retry-driven queue spike from an upstream-slow spike in seconds.

Related guides

• How Envoy actually works in production: a mental model for operators
• Envoy membership_healthy dropping: reading the single most important cluster signal
• Envoy monitoring checklist: the signals every production proxy needs
• Envoy monitoring maturity model: from survival to expert
• Envoy no healthy upstream: the 503 when a cluster has no host to route to
• Envoy outlier detection mass ejection: when passive health checks empty a cluster