$ guides / envoy / envoy-connection-pool-exhaustion-slow-upstream ▌

Operations Guides

Envoy connection pool exhaustion: a slow upstream that fills the pool

Clients are seeing 503 responses. Access logs show response flag UO on those requests. But upstream health looks fine: membership_healthy is stable and hosts are passing active health checks. The upstream is not down, yet Envoy is refusing to forward new requests to it.

This is connection pool exhaustion. An upstream that was previously fast has become slow. Each request now holds a connection longer, so the same request rate fills more connection slots. The per-worker connection pool saturates, new requests queue in the pending buffer, the buffer overflows, and Envoy fast-fails those requests with 503 UO rather than piling on more load.

The trap: Envoy looks like the culprit. CPU and memory look fine, the upstream is healthy, and the error is a 503 generated locally by Envoy’s circuit breaker. The fix is almost never in Envoy. The fix is upstream.

What this means

Envoy maintains separate connection pools per worker thread, per upstream host, per protocol. A cluster with 10 hosts and 4 worker threads has up to 40 independent connection pools. Circuit breaker limits (max_connections, max_pending_requests, max_requests, max_retries) apply per cluster per priority, shared across workers.

When an upstream slows down, the effect on the pool is multiplicative. If average response time doubles from 50ms to 100ms at a constant request rate, connections are held twice as long, so steady-state connection count roughly doubles. The same traffic that fit comfortably now pushes upstream_cx_active toward max_connections.

The cascade follows a strict sequence:

flowchart TD
    A[Upstream slows
upstream_rq_time rises] --> B[Connections held longer
upstream_cx_active climbs]
    B --> C[Pool saturates
new requests queue]
    C --> D[upstream_rq_pending_active grows]
    D --> E[max_pending_requests hit
upstream_rq_pending_overflow fires]
    E --> F[503 UO returned
circuit breaker rq_pending_open]

The critical distinguishing feature: membership_healthy stays stable throughout. If hosts were crashing or being ejected, you would see membership_healthy drop and a mix of 502 and 504 responses (connection failures and timeouts), not clean 503 UO. The UO flag confirms this is Envoy’s own circuit breaker rejecting the request, not an upstream-originated error.

A second distinguishing feature is ordering. If the circuit breaker trips without a preceding latency rise, the limits are simply too low for normal traffic. The cascade is defined by latency leading the chain. No latency rise, no upstream problem; the breaker configuration is the issue.

Common causes

Cause	What it looks like	First thing to check
Upstream latency regression	`upstream_rq_time` P50 and P99 rising before overflow starts	Check the upstream service: database queries, GC logs, disk I/O
Scaling event not complete	`membership_total` recently dropped, fewer hosts handling same load	Confirm new endpoints have joined and are passing health checks
Network latency increase	`upstream_cx_connect_ms` rising alongside `upstream_rq_time`	Check for cross-AZ routing, VPN path changes, or network congestion
HTTP/1.1 without keep-alive	`upstream_cx_total` rate high relative to `upstream_rq_total`, connections churned per request	Verify keep-alive is enabled upstream; HTTP/1.1 pools queue per connection so pending fills fast
Circuit breaker limits too low	Breaker trips with no preceding latency increase, `upstream_rq_time` flat	Compare `max_connections` and `max_pending_requests` against steady-state `upstream_cx_active`
Retry amplification	`upstream_rq_total` significantly higher than `downstream_rq_total`, ratio above 1.5	Check retry policy; retries multiply pool pressure on an already slow upstream

Quick checks

All read-only and safe during an incident. The admin port is 9901 in standard deployments and 15000 in Istio sidecar mode.

# Confirm the cascade order: latency should have risen first
curl -s http://localhost:9901/stats | grep 'upstream_rq_time'

# Check current pool depth and overflow
curl -s http://localhost:9901/stats | grep -E 'upstream_rq_pending_(active|overflow)'

# Check active connections against circuit breaker limits
curl -s http://localhost:9901/stats | grep -E 'upstream_cx_active|cx_open|rq_pending_open'

# Confirm upstream hosts are healthy (rules out crash or ejection)
curl -s http://localhost:9901/stats | grep -E 'membership_healthy|membership_total'

# Check for retry amplification
curl -s http://localhost:9901/stats | grep -E 'upstream_rq_retry|downstream_rq_total'

# Check connection establishment time (network latency indicator)
curl -s http://localhost:9901/stats | grep 'upstream_cx_connect_ms'

# Per-host health detail during the incident
curl -s http://localhost:9901/clusters?format=json | jq '.cluster_statuses[].host_statuses[].health_status'

Response flags are not exposed as aggregate stats. Inspect access logs for the %RESPONSE_FLAGS% field to confirm UO.

How to diagnose it

Confirm the cascade ordering. Pull upstream_rq_time and confirm latency rose before upstream_rq_pending_overflow started incrementing. If latency is flat and the breaker is tripping, skip to the “limits too low” cause in the table above.
Verify upstream health is stable. Check membership_healthy and membership_total. If the healthy count is dropping, you are looking at a different failure mode (host crash or outlier detection ejection), not pure pool exhaustion.
Measure pool saturation. Compare upstream_cx_active against your configured max_connections. If active connections are pegged at the limit and cx_open or rq_pending_open is 1, the circuit breaker is open.
Check the pending queue. upstream_rq_pending_active is the gauge of queued requests. Any sustained nonzero value means requests are waiting. When it hits max_pending_requests, upstream_rq_pending_overflow starts counting rejected requests.
Confirm the UO flag. Aggregate stats cannot tell you the response flag. Inspect access logs for the %RESPONSE_FLAGS% field. UO confirms circuit breaker origin. If you see UF or UT instead, the failure is upstream connection failure or timeout, a different problem.
Rule out retry amplification. Calculate upstream_rq_total / downstream_rq_total. If the ratio is above 1.5, retries are inflating load on the slow upstream, accelerating the failure they are trying to mask.
Identify the upstream root cause. Once you confirm this is pool exhaustion driven by upstream latency, investigate the backend: database lock contention, GC pauses, disk I/O saturation, or a dependent service that is itself slow.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`upstream_rq_time`	Confirms upstream latency is the root cause	P99 above 2x rolling baseline before overflow starts
`upstream_cx_active`	Shows how close the pool is to max_connections	Trending toward the configured limit with no corresponding traffic increase
`upstream_rq_pending_active`	Leading indicator before overflow begins	Any sustained nonzero value means requests are queuing
`upstream_rq_pending_overflow`	Direct count of rejected requests	Nonzero rate means users are getting 503 UO right now
`circuit_breakers.default.cx_open`	Connection circuit breaker state	Transitions from 0 to 1 when max_connections is hit
`circuit_breakers.default.rq_pending_open`	Pending queue circuit breaker state	Transitions from 0 to 1 when max_pending_requests is hit
`membership_healthy`	Rules out host crash or ejection	Stable count confirms this is latency-driven, not health-driven
`upstream_rq_retry` as ratio of total	Detects retry amplification	Ratio above 0.3 suggests retries are worsening the problem

Fixes

Address the upstream root cause

This is almost always the correct fix. The circuit breaker is working as designed. Increasing limits without fixing the upstream removes the protection and the next failure involves an unprotected upstream collapsing completely.

Investigate the backend: slow queries, GC pauses, disk contention, connection pool limits on the upstream itself. Once upstream latency returns to baseline, upstream_cx_active drops, the pending queue drains, and the breaker closes on its own.

Temporarily increase circuit breaker limits

Use this only during an incident when you have confirmed the upstream is recovering but the breaker is preventing throughput from recovering. Increasing max_connections and max_pending_requests gives the upstream room to drain its backlog. This is a stopgap. If the upstream does not recover, higher limits just delay the inevitable and consume more memory and file descriptors.

Default circuit breaker values are max_connections=1024, max_pending_requests=1024, max_requests=1024, max_retries=3. If steady-state upstream_cx_active is already near these defaults during normal operation, the limits are too low and should be raised to at least 2x the observed peak.

Reduce retry aggressiveness

If retry amplification is contributing (upstream-to-downstream ratio above 1.5), reduce or disable retries for the affected cluster during the incident. Retries multiply pool pressure. One client request becoming three upstream attempts triples the load on an already struggling backend. When the upstream recovers, re-enable retries with a conservative budget.

Check for HTTP/1.1 keep-alive issues

HTTP/1.1 pools queue requests per connection because there is no multiplexing. A slow upstream fills the pending queue quickly under HTTP/1.1. HTTP/2 multiplexes streams over fewer connections, so pending overflow fires less readily. If your upstream supports HTTP/2, switching the cluster protocol can reduce pool pressure. If you must use HTTP/1.1, ensure keep-alive is enabled so connections are reused rather than churned.

Prevention

Alert on upstream_rq_pending_active growth, not just overflow. By the time upstream_rq_pending_overflow increments, users are already getting 503s. The pending queue depth is a leading indicator that gives minutes of warning.
Set circuit breaker limits with headroom. Limits should be at least 2x the peak upstream_cx_active observed during normal operation.
Monitor the upstream-to-downstream request ratio. A ratio above 1.3 indicates meaningful retry activity. Above 2.0 is a retry storm that will accelerate any upstream degradation.
Track upstream_rq_time against baseline, not fixed thresholds. Different services have different latency profiles. Use deviation from rolling baseline (P99 above 2x rolling average) rather than absolute values.
Enable track_remaining on circuit breakers. Exposes headroom gauges before the breaker trips. Requires track_remaining: true in the circuit breaker config, which is disabled by default.
Monitor per-host, not just per-cluster. Aggregate cluster stats hide single-host problems. A cluster-level upstream_rq_time P99 of 200ms might mean every host is at 200ms, or one host is at 2000ms while the rest are at 50ms. Use GET /clusters?format=json during incidents to drill into per-host detail.

How Netdata helps

Per-second metric collection reveals the cascade ordering in real time. The latency rise, connection count climb, pending queue growth, and overflow all appear as distinct steps in the same dashboard, confirming that latency led the chain.
Cross-signal correlation lets you confirm the distinguishing feature in seconds. If membership_healthy is flat while upstream_rq_pending_overflow spikes, the visual correlation rules out a host crash and points to upstream latency.
Anomaly detection on upstream_rq_time can flag the latency regression before the pending queue fills, surfacing the slowdown before users see 503s.
Circuit breaker state visibility shows cx_open and rq_pending_open transitions alongside connection and pending counts, so you can see exactly when and why the breaker tripped.