Envoy connection churn: a low reuse ratio and keepalive misconfiguration

Envoy’s upstream connection pool amortizes TCP (and TLS) handshakes across many requests. When it stops doing that, you have connection churn: the pool establishes a fresh connection for nearly every request, and the reuse ratio collapses toward 1.0.

The headline signal is the ratio upstream_rq_total / upstream_cx_total. For HTTP/1.1 with keepalive working, expect a number well above 1, often in the tens. For HTTP/2 with stream multiplexing, expect far higher still. When the ratio sits near 1.0, every request pays the full handshake tax.

Churn rarely pages anyone. The cluster looks healthy: membership_healthy is stable, 503s are not firing, error rates are flat. What you see instead is an elevated upstream_cx_connect_ms rate, extra worker CPU burned on TLS, more file descriptors cycling through the process, and a tail-latency bump on upstream_rq_time for every request that lands on a cold connection. This article covers how to compute the ratio correctly, identify the specific close-cause from Envoy’s per-connection counters, and fix the three common root causes: keepalive disabled, max_connection_duration recycling, and DNS TTL churn on STRICT_DNS clusters.

What this means

The reuse ratio compares two monotonic counters on the cluster stat prefix:

• cluster.<name>.upstream_rq_total: total requests (including retries) sent upstream.
• cluster.<name>.upstream_cx_total: total upstream connections ever established.

ratio = upstream_rq_total / upstream_cx_total. For a clean operational read, compute it from per-window rate deltas rather than cumulative counters since process start, which are dominated by cold-start noise.

Envoy also exposes cluster.<name>.upstream_rq_per_cx, a histogram that directly measures the number of requests handled per upstream connection across all HTTP protocols. This is the official, version-stable way to read reuse quality. A healthy distribution is centered well above 1; a distribution pinned at 1 means the pool is one-shotting connections.

A low ratio is expensive in three ways. First, every churned connection adds a TCP handshake (and, when upstream TLS is configured, a TLS handshake) to per-request latency, which shows up as an elevated upstream_cx_connect_ms rate. Second, TLS handshakes are typically the dominant CPU consumer in Envoy, so churn translates directly into worker CPU. Third, each connection consumes two file descriptors (downstream plus upstream) plus kernel socket overhead, so churn inflates FD cycling and can interact with FD exhaustion during spikes.

Envoy tells you exactly why each connection closed. The cluster stats include dedicated close-cause counters that decompose upstream_cx_total into its contributors. The diagnostic flow is to find which counter is consuming your pool.

flowchart TD
  A["Low reuse ratio
rq_total / cx_total near 1"] --> B["Inspect per-connection close counters"]
  B --> C{"Which counter is rising?"}
  C -->|upstream_cx_max_requests| D["max_requests_per_connection = 1
keepalive effectively disabled"]
  C -->|upstream_cx_max_duration_reached| E["max_connection_duration recycling pool"]
  C -->|upstream_cx_close_notify| F["Idle timeout or upstream GOAWAY"]
  C -->|none obvious| G["STRICT_DNS churn on short TTL"]

Common causes

Quick checks

The admin port is 9901 in standalone Envoy and 15000 in Istio sidecar mode. Adjust accordingly.

# Reuse ratio inputs for a specific cluster (lifetime counters)
curl -s http://localhost:9901/stats | grep -E 'cluster\.my_cluster\.(upstream_rq_total|upstream_cx_total)'

# Per-connection request histogram (the official reuse signal)
curl -s http://localhost:9901/stats | grep 'upstream_rq_per_cx'

# Close-cause counters - which one is consuming your pool?
curl -s http://localhost:9901/stats | grep -E 'upstream_cx_(max_requests|max_duration_reached|idle_timeout|close_notify)'

# Rate of new connection establishment (should be low at steady state)
curl -s http://localhost:9901/stats | grep 'upstream_cx_total'

# TCP (and TLS) connect time - elevated when churn forces fresh handshakes
curl -s http://localhost:9901/stats | grep 'upstream_cx_connect_ms'

# DNS update cadence for STRICT_DNS / LOGICAL_DNS clusters
curl -s http://localhost:9901/stats | grep -E 'cluster\.my_cluster\.update_(success|failure)'

# Active pool size and pending requests
curl -s http://localhost:9901/stats | grep -E 'cluster\.my_cluster\.(upstream_cx_active|upstream_rq_pending_active)'

For high-traffic proxies, prefer /stats/prometheus with the ?usedonly filter and a 30s+ scrape interval. Scraping the plain /stats endpoint every 10-15s blocks worker threads and can itself cause the latency symptoms you are investigating.

How to diagnose it

1. Compute the ratio over a clean window. Sample upstream_rq_total and upstream_cx_total 60 seconds apart and divide the deltas. Do not rely on cumulative counters, which are dominated by cold-start behavior.
2. Confirm with upstream_rq_per_cx. If the histogram is pinned at 1, the pool is one-shotting connections. If it is centered at, say, 5, reuse is happening but is weaker than expected.
3. Find the close-cause. Sum the close-cause counters (upstream_cx_max_requests, upstream_cx_max_duration_reached, upstream_cx_idle_timeout, upstream_cx_close_notify) and find the dominant contributor. The largest one names your failure mode.
4. Correlate with DNS. If no single close-cause dominates but upstream_cx_total spikes in lockstep with update_success on a STRICT_DNS cluster, DNS churn is resetting the pool. Check dns.cares.resolve_total for the resolution rate.
5. Check per-worker state. Connection pools are per-worker and per-host. A cluster with 10 hosts across 8 workers has up to 80 independent pools. Aggregate stats can hide a single host or worker that is churning while the rest are healthy. Drill into /clusters?format=json for per-host detail during incidents.

Metrics and signals to monitor

Fixes

Keepalive disabled (max_requests_per_connection: 1)

Per Envoy’s cluster proto documentation, setting max_requests_per_connection to 1 “will effectively disable keep alive.” This is the single most common cause of a reuse ratio pinned at 1.0. It is usually a copy-paste from a tuning guide or a leftover from debugging.

Fix: remove the field entirely (the default is unlimited) or set it to a large value. A finite cap can help spread load across connections and bound the blast radius of a single connection, but the cap should be in the hundreds or thousands, not 1.

max_connection_duration recycling

max_connection_duration defaults to 0 (unlimited). The Envoy timeouts FAQ notes that setting it can help with DNS-based clusters where resolved addresses may change even when upstreams stay healthy. But if it is set low (for example, 30 seconds), every connection in the pool cycles on that interval, and the reuse ratio floors at roughly duration / mean_interarrival_time.

Fix: raise it, remove it, or pair it with LOGICAL_DNS so the pool does not need to recycle to pick up DNS changes. If you genuinely need cycling for DNS reasons, measure the resulting reuse ratio and confirm the latency cost is acceptable.

STRICT_DNS churn

Per the service discovery documentation, STRICT_DNS clusters drain and recreate connection pools whenever DNS resolution returns a different IP set. The docs explicitly warn that with changing DNS, STRICT_DNS “would lead to draining connection pools, connection cycling, etc.” If respect_dns_ttl is enabled and upstream DNS records have short TTLs (common with cloud load balancers and service discovery systems), the pool resets on every refresh.

The recommended alternative is LOGICAL_DNS. Per the docs, with LOGICAL_DNS, “connections stay alive until they get cycled.” LOGICAL_DNS treats the resolved address set as a single logical host, so connection pools survive DNS refreshes as long as the resolved IP does not fully disappear.

Tradeoff: LOGICAL_DNS does not load balance across multiple resolved IPs the way STRICT_DNS does. Use it when you are pointing at a single logical upstream (a load balancer, a service VIP) rather than a pool of discrete endpoints. For real endpoint pools, prefer EDS via the control plane, which pushes endpoint updates without churning connections the way DNS does.

Upstream-driven close (GOAWAY / idle timeout)

If upstream_cx_close_notify is the dominant close-cause, the upstream is closing connections. For HTTP/2 and HTTP/3 this is a GOAWAY; for HTTP/1.1 it is a Connection: close header. The upstream server may be applying its own idle timeout or its own max-requests-per-connection policy.

Envoy does not propagate downstream Connection: close headers to upstream. The upstream connection lifecycle is decoupled from the downstream. So if you see upstream_cx_close_notify rising on the upstream side, it is the upstream server doing it, not your downstream clients.

Fix: align Envoy’s idle_timeout (default 1 hour) with the upstream’s idle behavior. If the upstream forces closes earlier than Envoy’s timeout, lower Envoy’s idle timeout to match so you do not hold dead connections. Check the upstream’s own keepalive and max-requests settings as well.

Prevention

• Track the reuse ratio per cluster. The playbook’s expert monitoring tier lists upstream_rq_total / upstream_cx_total as a Level 4 signal. Treat it as a baseline metric, not an incident-only check.
• Alert on close-cause balance. In a healthy pool, no single close-cause counter should dominate. Alert when any one counter exceeds the others by a meaningful margin.
• Default to LOGICAL_DNS or EDS for short-TTL upstreams. Reserve STRICT_DNS for cases where the resolved IP set is genuinely stable.
• Pair max_connection_duration changes with a reuse-ratio measurement. Any change to connection lifetime policy should be validated against the ratio.
• Audit cluster templates for max_requests_per_connection. A single misconfigured cluster template can churn across every service that imports it.

How Netdata helps

• Per-second collection of upstream_rq_total and upstream_cx_total lets you compute the reuse ratio on a tight window without waiting for a slow scrape interval, which matters when churn starts mid-incident.
• ML anomaly detection flags sudden drops in the reuse ratio even when no static threshold would catch it, for example when a config push silently sets max_requests_per_connection: 1.
• Correlating upstream_cx_connect_ms with the close-cause counters in a single view shortens the path from “latency is up” to “the pool is churning because of max_connection_duration.”
• Worker CPU and TLS handshake rate sit alongside the cluster stats, so the CPU cost of churn is visible in real time rather than inferred.
• DNS update counters for STRICT_DNS and LOGICAL_DNS clusters pair with the reuse ratio to confirm DNS-driven resets without a separate dashboard.

Related guides

• Envoy health checks vs outlier detection: two systems that eject hosts differently
• How Envoy actually works in production: a mental model for operators
• Envoy membership_healthy dropping: reading the single most important cluster signal
• Envoy monitoring checklist: the signals every production proxy needs
• Envoy monitoring maturity model: from survival to expert
• Envoy no healthy upstream: the 503 when a cluster has no host to route to
• Envoy outlier detection mass ejection: when passive health checks empty a cluster
• Envoy panic threshold: why traffic routes to unhealthy hosts at 50%
• Envoy upstream_cx_connect_fail: failed TCP connections to upstream hosts
• Envoy upstream_rq_pending_overflow: the pending queue fills and 503s begin