Envoy downstream_cx_active growing: connection leaks and idle-timeout gaps

listener.<address>.downstream_cx_active is a gauge counting the downstream (client-facing) connections currently held open by a listener. When it climbs without a matching rise in request rate, the usual first call is “connection leak”. That diagnosis is sometimes correct and sometimes wrong, and the fixes are completely different.

The gauge reflects three independent inputs: how fast new connections arrive (downstream_cx_total), how fast old connections close (downstream_cx_destroy), and how long each connection is allowed to live once idle (idle, stream, drain, and connection-duration timers). A leak is only one explanation. Others include clients that legitimately hold connections open (HTTP/2 and gRPC multiplexing), loopback sidecar traffic, an idle timeout set far longer than the workload needs, or a slow upstream that holds streams, and therefore connections, open past the point the idle timer could reclaim them.

This article covers how to read the gauge correctly, distinguish a real leak from a configuration or workload-shape issue, and scope a fix without breaking long-lived streams.

What this means

downstream_cx_active is a listener-scoped gauge. It does not measure requests, streams, or bytes. It measures open sockets on the downstream side. Two implications follow.

In HTTP/1.1 with keep-alive, one connection usually carries one request at a time, so connection count is a reasonable proxy for concurrency. In HTTP/2 or gRPC, a single downstream connection multiplexes many concurrent streams. Connection count can be flat, low, or shrinking while request rate explodes. The reverse is also true: a growing connection count with flat request rate is suspicious in HTTP/1.1 but expected for some HTTP/2 clients that open one connection per priority bucket and hold them.

The gauge is directly governed by the connection lifecycle timers configured on the HTTP connection manager (HCM). The downstream idle_timeout is the dominant one. When it is set long, or set to 0 (which disables it), connections with no active streams can still sit in downstream_cx_active for a long time. Disabling the idle timeout is known to produce leaks via lost TCP FIN packets, so a value of 0 is a red flag on its own.

The diagnostic question is not “is downstream_cx_active growing?” but “is the close rate keeping up with the open rate, given the configured lifecycle timers and the actual traffic shape?”

flowchart TD
    A[downstream_cx_active climbing] --> B{downstream_cx_total rising too?}
    B -- Yes, proportional --> C[Traffic-driven: check downstream_rq_total]
    B -- No, flat/low --> D{downstream_cx_destroy keeping up?}
    D -- No --> E[Connections not closing]
    D -- Yes, but gauge still climbs --> F[HTTP/2 multiplexing or long-lived streams]
    E --> G1[Idle timeout 0 or very long?]
    E --> G2[Clients not sending FIN?]
    E --> G3[Drain/delayed-close stuck?]
    C --> H[Real load: capacity plan, not a leak]
    F --> I[Expected: switch to rq_active view]
    G1 --> J[Tighten idle_timeout, keep > 0]
    G2 --> K[connection_keepalive for HTTP/2 dead peers]
    G3 --> L[Check drain_timeout, delayed_close_timeout]

Common causes

Quick checks

# Current gauge per listener
curl -s http://localhost:9901/stats | grep 'downstream_cx_active'

# New vs closed rates - sample twice, ~10s apart, and compute deltas
curl -s http://localhost:9901/stats | grep -E 'downstream_cx_(total|destroy)'

# Idle-timeout driven closes - should be non-zero if idle_timeout is doing work
curl -s http://localhost:9901/stats | grep -E 'downstream_cx_idle_timeout|downstream_cx_destroy_active_rq'

# Protocol mix - is HTTP/2 hiding concurrency?
curl -s http://localhost:9901/stats | grep -E 'downstream_cx_http(1|2|3)_'

# Concurrent streams vs connections
curl -s http://localhost:9901/stats | grep -E 'downstream_rq_active|downstream_rq_total'

# HCM lifecycle timers actually in effect
curl -s http://localhost:9901/config_dump | jq '[.configs[] | .. | objects | select(has("idle_timeout") or has("drain_timeout") or has("stream_idle_timeout")) | {idle_timeout, drain_timeout, stream_idle_timeout, max_connection_duration}]'

# FD pressure check (each downstream conn is at least 1 FD, upstream is another)
ENVOY_PID=$(pgrep -x envoy | head -1); ls /proc/$ENVOY_PID/fd | wc -l; grep 'Max open files' /proc/$ENVOY_PID/limits

# Hot restart epoch - rule out gauge carry-over
curl -s http://localhost:9901/stats | grep -E 'hot_restart_epoch|server.uptime|server.state'

Replace 9901 with 15000 in Istio sidecar mode. All checks above are read-only.

How to diagnose it

1. Normalize against traffic. Pull downstream_cx_total and downstream_rq_total over the same window. If both climb together and downstream_cx_destroy is keeping up proportionally, the gauge is telling you about load, not a leak. Capacity-plan, do not chase a leak.

2. Compare open rate to close rate. Sample downstream_cx_total and downstream_cx_destroy ten seconds apart. If delta(total) is consistently larger than delta(destroy) over a sustained window, connections are accumulating. This is the strongest single signal of a real leak or a too-long idle timeout.

3. Resolve the protocol. Check downstream_cx_http1_active, downstream_cx_http2_active, and downstream_cx_http3_active. If HTTP/2 or HTTP/3 dominates, judge concurrency with downstream_rq_active and per-protocol stream counts, not raw connection count. A “high” gauge in HTTP/2 may simply mean many clients each opened one connection, which is normal.

4. Read the HCM timers from config_dump. The values that matter are idle_timeout (default 1 hour), stream_idle_timeout (default 5 minutes), drain_timeout (default 5 seconds), delayed_close_timeout (default 1 second), and max_connection_duration (default 0, unlimited). A downstream idle timeout of 0 or several hours, combined with bursty short-lived clients, will inflate the gauge even when nothing is broken.

5. Check whether idle closes are happening. If downstream_cx_idle_timeout is non-zero and incrementing, the timer is doing work. If it is zero while the gauge climbs, either no connections are reaching idle (streams are still active) or the timer is disabled. Both are diagnostic.

6. Look for streams that never go idle. The downstream idle timeout only fires when there are no active streams on the connection. A slow upstream that holds requests open prevents the timer from ever firing. Correlate with upstream_rq_time, upstream_rq_active, and response flag UT. This is the upstream-slow variant of the same symptom.

7. Rule out HTTP/2 dead peers. HTTP/2 PING frames do not reset the idle timeout. A stream created on a dead HTTP/2 connection will hang until a configured keepalive detects the failure, or indefinitely if no keepalive is configured. If the gauge holds flat at a level that no longer corresponds to live traffic, suspect dead HTTP/2 peers and check whether http2_protocol_options.connection_keepalive is set.

8. Scope sidecar loopback. In a sidecar, the inbound listener’s downstream_cx_active includes loopback connections from the local application. A misbehaving app client (opening connections without pooling, never closing them) shows up here. Cross-reference with the app’s own outbound connection metrics, not with external client traffic.

9. Rule out restart carry-over. After a hot restart, gauges can read inconsistently while old and new processes share the stats region. Check server.hot_restart_epoch and server.uptime. If the symptom appeared immediately after a restart and is not growing monotonically, wait one full drain cycle before treating it as a leak.

Metrics and signals to monitor

Fixes

Idle timeout disabled or set too long

If idle_timeout is 0 or multiple hours and your clients are bursty and short-lived, set it to a value matched to your workload (minutes, not hours). Do not set it to 0. Disabling it is likely to produce leaks via lost TCP FIN packets. Keep stream_idle_timeout and max_connection_duration consistent: max_connection_duration can close connections even when streams are active, so use it deliberately if you need a hard cap.

Tradeoff: lowering idle_timeout increases connection churn (more TLS handshakes, more TCP setup). On TLS-heavy edge deployments, measure the handshake cost before tightening aggressively.

HTTP/2 dead-peer gaps

For HTTP/2 and gRPC clients, configure http2_protocol_options.connection_keepalive with both interval and timeout. Without it, a stream placed on a dead connection will hang until the peer’s TCP stack gives up, which on some networks is effectively never. The keepalive PING does not reset the downstream idle timeout; the two mechanisms solve different problems.

Tradeoff: keepalive PINGs add a small amount of traffic and CPU. The cost is negligible compared to the cost of a stuck connection.

Clients that do not close

If clients open connections and never send FIN, the only defense is server-side lifecycle timers. Confirm idle_timeout is set to a finite value and that downstream_cx_idle_timeout is incrementing. If clients are inside your org, push a client-side fix (proper connection pooling, closing on idle). If they are external, the server-side timers are the only lever.

Tradeoff: aggressive server-side idle timeouts can interrupt legitimate long-poll or SSE workloads. Use stream_idle_timeout to differentiate “no streams” from “stream but no bytes”.

Slow upstream holding streams open

If upstream_rq_time is climbing and the downstream gauge tracks it, the leak is upstream, not downstream. Pursue the upstream latency root cause (database, GC, dependency saturation). The upstream HTTP idle timeout uses the same default (1 hour) as the downstream, but an explicit upstream override can be smaller than the downstream value, producing destroy-with-active-request spikes when the upstream closes first.

Tradeoff: aligning upstream and downstream timeouts reduces asymmetry surprises but does not fix the slow upstream.

Sidecar loopback accumulation

In a sidecar, work with the application team to use HTTP/2 or a properly pooled HTTP/1.1 client. The inbound downstream_cx_active follows the app’s outbound behavior, so the fix lives in the app, not in Envoy.

Hot restart carry-over

If the gauge misbehaves only after a restart, ensure the old process drains fully before the new one is considered healthy. Check server.parent_connections draining to zero, and confirm the hot restart shared memory is not corrupted by a version mismatch between old and new Envoy.

Prevention

• Alert on the open/close ratio, not the absolute gauge. A sustained delta(downstream_cx_total) > delta(downstream_cx_destroy) window is a better signal than a fixed threshold on the gauge.
• Never set idle_timeout to 0. Pick a finite value matched to your client population.
• Configure HTTP/2 connection_keepalive on any listener serving HTTP/2 or gRPC clients, especially external ones.
• Track downstream_cx_idle_timeout as a health signal. If it is zero for hours, either your clients never idle (rare) or your timer is misconfigured.
• Monitor file descriptors alongside the gauge. Each downstream connection is at least one FD. The gauge and FD count should move together. If FDs climb faster than the gauge, suspect an access log leak or health check FD leak.
• Track server.memory_allocated alongside the gauge. Per-connection buffers mean memory should scale roughly linearly with downstream_cx_active. Decoupled growth suggests a different memory issue (stats cardinality, buffering filter).
• Watch for the Rapid Reset pattern. A flat or low downstream_cx_active while downstream_rq_http2_total spikes is a red flag for a different class of problem (CVE-2023-44487-style behavior) rather than a leak.

How Netdata helps

• Per-second collection of downstream_cx_active, downstream_cx_total, and downstream_cx_destroy lets you compute open/close ratios on short windows where a 30-second scrape interval is too coarse to catch a burst.
• Correlating the gauge against downstream_rq_active and per-protocol counters (downstream_cx_http1_active, downstream_cx_http2_active) makes HTTP/2 multiplexing visible without manual cross-referencing.
• ML anomaly detection on the open/close delta surfaces accumulating-connection behavior before the gauge crosses a fixed threshold.
• Layering server.memory_allocated, FD usage, and upstream_rq_time against the downstream gauge separates a real leak from a slow-upstream or memory-buffering problem.
• Anomaly flags on downstream_cx_idle_timeout going to zero catch a timer misconfiguration or a workload shift that stops producing idle connections.
• Per-pod sidecar dashboards distinguish inbound loopback accumulation from outbound client behavior across a fleet.

Related guides

• Envoy 502 and upstream resets: rx_reset, tx_reset, and mid-response failures
• Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend
• Envoy 504 upstream timeout: upstream_rq_timeout, per-try timeouts, and the UT flag
• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy clusters stuck warming: warming_clusters non-zero and routes returning 503
• Envoy connection pool exhaustion: a slow upstream that fills the pool
• Envoy control_plane.connected_state = 0: running on stale xDS config
• Envoy downstream_rq_time high: client-observed latency and proxy overhead
• Envoy file descriptor exhaustion: the FD cliff that refuses every new connection
• Envoy health checks vs outlier detection: two systems that eject hosts differently
• How Envoy actually works in production: a mental model for operators
• Envoy listener_create_failure: a listener config Envoy could not apply