Envoy upstream_cx_active near max_connections: the pool filling up

The cluster.<name>.upstream_cx_active gauge reports active connections across all hosts in a cluster. When it trends toward the circuit-breaker max_connections limit, the cluster is approaching saturation. Past the limit, Envoy stops opening new connections, queues new requests in upstream_rq_pending_active, and once max_pending_requests is hit, returns 503s with response flag UO.

Envoy is fast-failing locally to protect an upstream that cannot absorb more concurrent work. Raising max_connections without addressing the upstream removes that protection. The operator’s job is to find what is shrinking effective pool capacity.

For HTTP/2 or gRPC upstreams, upstream_cx_active is the wrong saturation signal. HTTP/2 multiplexes many streams over a single connection, so connection count stays low even when concurrent request count (upstream_rq_active) is saturated. Always correlate with the protocol mix (upstream_cx_http1_total, upstream_cx_http2_total) before treating upstream_cx_active as the bottleneck.

What this means

Envoy’s upstream connection pool is per-worker, per-host, per-protocol. A cluster with N hosts, W worker threads, and P protocol variants can have up to N * W * P independent pools. The cluster.<name>.upstream_cx_active gauge aggregates across all of them, while the circuit breaker applies to the aggregate per-priority budget.

Two consequences matter for diagnosis:

1. upstream_cx_active can legitimately exceed max_connections by a small amount. Envoy guarantees at least one connection per host per pool even after overflow trips, and per-worker counters are eventually consistent, so brief races push the aggregate slightly over the configured limit. Alerting on upstream_cx_active > max_connections produces false positives. The authoritative overflow signal is the upstream_cx_overflow counter incrementing and circuit_breakers.default.cx_open flipping to 1.

2. The default max_connections is 1024 per cluster per priority. If you have never set this explicitly and the gauge is approaching 1024, you are operating against Envoy’s shipped defaults. The number alone tells you nothing without context.

The failure pattern is consistent: a slow upstream holds each connection longer, the pool fills under a flat request rate, the pending queue grows, the breaker trips, and 503s with UO begin. The diagnostic question is which input is shrinking effective capacity.

flowchart TD
    A[Slow upstream or fewer hosts] --> B[Each request holds connection longer]
    B --> C[upstream_cx_active climbs]
    C --> D[max_connections reached]
    D --> E[upstream_rq_pending_active grows]
    E --> F[max_pending_requests hit]
    F --> G[upstream_rq_pending_overflow increments]
    G --> H[503 with response flag UO]

Common causes

Quick checks

All commands below are read-only. Substitute your cluster name for <name>. The admin port is 9901 in standalone deployments and 15000 in Istio sidecars.

# Watch active connections and circuit breaker state for one cluster
curl -s http://localhost:9901/stats | grep -E 'cluster.<name>.(upstream_cx_active|circuit_breakers)'

# Confirm whether the breaker has actually tripped
curl -s http://localhost:9901/stats | grep -E 'circuit_breakers.*(_open|remaining_cx)'

# Pending queue depth and overflow (the cliff-edge signal)
curl -s http://localhost:9901/stats | grep -E 'cluster.<name>.upstream_rq_pending_(active|overflow|total)'

# Upstream latency and connect time (root cause vs churn)
curl -s http://localhost:9901/stats | grep -E 'cluster.<name>.(upstream_rq_time|upstream_cx_connect_time)'

# Protocol mix and concurrent requests (HTTP/2 saturation hides here)
curl -s http://localhost:9901/stats | grep -E 'cluster.<name>.(upstream_cx_http[12]_total|upstream_rq_active)'

# Cluster membership context
curl -s http://localhost:9901/stats | grep -E 'cluster.<name>.membership'

# Confirm UO origin in access logs (path and format vary by deployment)
# grep -F '" 503 UO ' /var/log/envoy/access.log | tail

How to diagnose it

1. Confirm the breaker actually tripped. Look for circuit_breakers.default.cx_open = 1 and a non-zero rate on upstream_cx_overflow. If neither is true, upstream_cx_active is high but not yet saturated; treat it as a leading indicator.

2. Pull the protocol mix. If upstream_cx_http2_total dominates, switch focus to upstream_rq_active and circuit_breakers.default.rq_open. The connection breaker is not your problem.

3. Pull upstream_rq_time. If P50 or P99 is climbing, the upstream is the root cause. A slow upstream holds each connection longer, shrinking effective capacity without any change in request rate.

4. Pull upstream_cx_total as a rate and compare to upstream_rq_total. If the connection reuse ratio (upstream_rq_total / upstream_cx_total) is near 1.0 for HTTP/1.1, connections are not being reused and the pool is churning through TCP and TLS handshakes.

5. Check membership_healthy and membership_total. If hosts were ejected by outlier detection or scaled down, the same request rate now distributes over fewer hosts, inflating per-host connection counts.

6. Pull retry stats. If upstream_rq_retry / upstream_rq_total is greater than 0.1, retry amplification is consuming pool capacity. A retry is a new upstream request that holds a connection.

7. Inspect per-host state via the admin endpoint. Aggregate cluster stats hide single slow hosts. GET /clusters?format=json shows per-host health flags and connection counts, which is the only way to distinguish “all hosts are uniformly slow” from “one host is stuck and skewing the aggregate”.

Metrics and signals to monitor

Fixes

If the upstream is slow (the usual case)

Do not raise max_connections first. The breaker is protecting the upstream from further overload. Address the upstream root cause: database contention, GC pauses, downstream dependency latency, disk I/O. Reducing request rate from upstream callers (load shedding at the edge, disabling non-critical workloads) is often the fastest path to relief during an active incident.

If you need to absorb a transient slowdown without 503s, raise max_pending_requests rather than max_connections. The pending queue absorbs latency spikes at the cost of memory, not at the cost of additional upstream pressure.

If max_connections is genuinely too low

Set it to roughly 2x the peak upstream_cx_active observed during normal operation. Apply the change via xDS so Envoy reloads without dropping connections. Track the new peak over the next week to validate the setting; if you are still approaching the limit, the bottleneck is elsewhere.

If HTTP/1.1 connections are churning

Verify keepalive is enabled on the upstream cluster. Check max_connection_duration and idle_timeout; if either is very short, healthy connections are being torn down before being reused. Each new connection pays a TCP (and often TLS) handshake cost that shows up as elevated upstream_cx_connect_time and inflates the active connection count.

If HTTP/2 stream exhaustion is the real issue

The breaker you want is max_requests, not max_connections. Confirm by checking circuit_breakers.default.rq_open and upstream_rq_active. If the upstream advertises a low SETTINGS_MAX_CONCURRENT_STREAMS and you have configured nothing to override it, the per-connection stream limit may be the real ceiling. Each Envoy worker maintains its own pool, so the cluster-level aggregate obscures whether one worker is hitting the limit.

If retry amplification is consuming pool capacity

Reduce retry aggressiveness: lower num_retries, narrow retry_on to idempotent cases, and check the retry budget. If upstream_rq_retry_overflow is incrementing, the retry circuit breaker is already exhausted, which means retries are being dropped silently. Reducing retry count also reduces upstream load, which often resolves the upstream latency that triggered the retries in the first place.

Emergency load-shed during an active incident

Temporarily raising max_connections and max_pending_requests via xDS can absorb a spike and stop the 503s, but it is a stopgap. The upstream root cause still needs addressing. Watch server.memory_allocated while limits are raised: you are trading 503s for memory pressure, and an unconfigured overload manager offers no protection if memory runs away.

Prevention

• Enable track_remaining: true on circuit breakers. It surfaces remaining_cx and remaining_pending so headroom is visible before the breaker opens.
• Alert on upstream_rq_pending_active, not just 503s. Pending queue growth gives minutes of warning before overflow begins.
• Set max_connections to roughly 2x the peak upstream_cx_active observed in normal operation. Conservative headroom absorbs spikes without tripping.
• For HTTP/2 clusters, alert on upstream_rq_active / max_requests, not upstream_cx_active. Connection count is misleading under multiplexing.
• Keep roughly 20% headroom on the pool in steady state. Any sustained trend toward the limit is a leading indicator, not noise.
• Track the connection reuse ratio (upstream_rq_total / upstream_cx_total). A falling ratio means connections are not being reused and the pool is churning.

How Netdata helps

• Per-second collection of upstream_cx_active, upstream_rq_pending_active, and circuit_breakers.*.open shows the cliff approaching before cx_open actually flips.
• ML anomaly detection on upstream_rq_time flags slow-upstream conditions that drive pool saturation, even when the breaker has not tripped yet.
• Correlating upstream_cx_active against membership_healthy on the same chart makes it obvious whether pool fill is driven by host loss, ejection, or upstream latency.
• Anomaly flags on the upstream_rq_retry / upstream_rq_total ratio catch retry amplification before it consumes pool capacity.
• Per-host drilldown exposes single slow hosts that aggregate cluster stats hide.

Related guides

• Envoy health checks vs outlier detection: two systems that eject hosts differently
• How Envoy actually works in production: a mental model for operators
• Envoy membership_healthy dropping: reading the single most important cluster signal
• Envoy monitoring checklist: the signals every production proxy needs
• Envoy monitoring maturity model: from survival to expert
• Envoy no healthy upstream: the 503 when a cluster has no host to route to
• Envoy outlier detection mass ejection: when passive health checks empty a cluster
• Envoy panic threshold: why traffic routes to unhealthy hosts at 50%
• Envoy upstream_cx_connect_fail: failed TCP connections to upstream hosts
• Envoy upstream_rq_pending_overflow: the pending queue fills and 503s begin