Envoy upstream_cx_connect_ms high: slow TCP connects to upstream hosts

When cluster.<name>.upstream_cx_connect_ms starts climbing, understand what this histogram actually measures: the time Envoy spent establishing a TCP connection to an upstream host. When upstream TLS is configured, the TLS handshake is rolled into the same number. It is not request latency, not application processing time, and not pure network RTT once TLS is in the path.

Same-zone connections typically sit under 2ms at P99. The alert threshold is a sudden 5x rise over rolling baseline. Anything beyond that means the network path has changed, the upstream kernel cannot accept connections fast enough, or something is forcing Envoy to open far more new connections than usual.

The trap: this metric is a histogram of a relatively rare event in steady state. HTTP/2 pools reuse connections aggressively, and HTTP/1.1 with keep-alive does too. So when upstream_cx_connect_ms looks bad, the more interesting question is often not “why is each connect slow” but “why are we doing so many connects.” An elevated connect rate is itself a pool-churn signal.

What this means

The cluster.<name>.upstream_cx_connect_ms histogram records milliseconds from the moment Envoy starts the upstream TCP connect to the moment the connection is usable. For plaintext upstreams, that is the TCP three-way handshake. For TLS upstreams, the TLS handshake is included in the same measurement, which is why a healthy mTLS mesh can show connect times an order of magnitude higher than a plaintext proxy.

Read it alongside two companion counters:

• cluster.<name>.upstream_cx_connect_fail - total TCP connection failures (immediate RST, port not open, refused).
• cluster.<name>.upstream_cx_connect_timeout - total connections that hit the cluster connect_timeout without completing.

A failure is a hard reject from the upstream. A timeout is a silent network path: SYN sent, nothing came back. They are different failure modes. A high upstream_cx_connect_ms without either counter moving usually means connects are succeeding but slowly. A high upstream_cx_connect_ms paired with rising connect_timeout means some connects are not succeeding at all.

This histogram is aggregated at the cluster level only. There is no per-host breakdown of upstream_cx_connect_ms, so you cannot point at a single slow upstream from this metric alone. Correlate with GET /clusters?format=json and per-host health flags to find the bad host.

flowchart TD
  A[upstream_cx_connect_ms high] --> B{connect_fail rising?}
  B -- Yes --> C[Upstream refusing
or backlog full]
  B -- No --> D{connect_timeout rising?}
  D -- Yes --> E[Network path lossy
or upstream saturated]
  D -- No --> F{connect rate elevated?}
  F -- Yes --> G[Pool churn:
idle timeout, GOAWAY,
max_connection_duration]
  F -- No --> H[TLS handshake cost
or slow DNS in path]

Common causes

Quick checks

All read-only and safe to run during an incident.

# Confirm the metric is actually elevated on a specific cluster
curl -s http://localhost:9901/stats | grep 'cluster.my_cluster.upstream_cx_connect_ms'

# Pull the histogram in Prometheus format to see bucket distribution
curl -s http://localhost:9901/stats/prometheus | grep 'envoy_cluster_upstream_cx_connect_ms.*my_cluster'

# Look for hard connection failures and timeouts on the same cluster
curl -s http://localhost:9901/stats | grep -E 'cluster.my_cluster.upstream_cx_connect_(fail|timeout)'

# Check whether the elevated connect time is really a churn problem
curl -s http://localhost:9901/stats | grep -E 'cluster.my_cluster.upstream_cx_(total|destroy|active)'

# Per-host view: which upstreams are failing or marked unhealthy
curl -s http://localhost:9901/clusters?format=json | jq '.cluster_statuses[] | select(.name=="my_cluster") | .host_statuses[] | {hostname, weight, health_flags}'

# Look for outlier ejections that may be related to slow connects
curl -s http://localhost:9901/stats | grep 'my_cluster.*outlier_detection.ejections'

# DNS health, only relevant for STRICT_DNS or LOGICAL_DNS clusters
curl -s http://localhost:9901/stats | grep -E 'dns\.|my_cluster.update_(success|failure)'

# Upstream TLS handshake stats, if upstream TLS is configured
curl -s http://localhost:9901/stats | grep 'cluster.my_cluster.ssl'

# Worker saturation, which can manifest as connect-level delays
curl -s http://localhost:9901/stats | grep -E 'watchdog_miss|watchdog_mega_miss'

If admin is on port 15000 (Istio sidecar default), substitute that for 9901. In Istio, the health endpoint is 15020 .

How to diagnose it

1. Establish the baseline. Pull the histogram from before the incident window. The relevant threshold is “sudden 5x rise over rolling baseline”, not a fixed number. A 3ms P99 is fine if baseline was 1ms. It is a problem if baseline was 0.2ms.
2. Separate slow from failing. If upstream_cx_connect_fail is also rising, you are looking at refused connections, not slow ones. Pivot to response flag UF in access logs. If upstream_cx_connect_timeout is rising, the network path is dropping SYNs.
3. Compute the connect rate. Sample cluster.<name>.upstream_cx_total twice, 10 seconds apart, and divide by the interval. Compare to the request rate over the same window. If connects per second is high relative to requests per second, you are in pool-churn territory, not network-latency territory.
4. Identify whether TLS is in the path. If upstream TLS is configured, the handshake is bundled into upstream_cx_connect_ms. Check cluster.<name>.ssl.handshake rate against the connect rate. If handshake rate matches connect rate one-to-one, every new connect is paying the full TLS cost. Look at cluster.<name>.ssl.connection_error and ssl.fail_verify_error for handshake failures that may be masquerading as slow connects.
5. Localize to a host or topology. Use GET /clusters?format=json to look for hosts with non-empty health_flags. A single bad host can drag the cluster histogram up because Envoy keeps trying and the slow connect is recorded before outlier detection ejects it.
6. Check for kernel-level refusal at the upstream. On the upstream host, ss -lnt shows the listen queue and netstat -s | grep -i listen shows overflow counters. A non-zero “times the listen queue overflowed” means the upstream kernel is dropping SYNs under burst, which surfaces as retransmits and elevated connect time at Envoy.
7. Verify the network path. From the Envoy host, run tcptraceroute or a one-off TCP SYN probe to a known upstream port. Look for loss or unexpected hops. Recent peering changes, MTU mismatches, and security-group edits all show up here.
8. Rule out DNS. For STRICT_DNS or LOGICAL_DNS clusters, check cluster.<name>.update_failure and dns.cares.pending_resolutions. DNS does not normally show up in upstream_cx_connect_ms because resolution happens at cluster-update time, but a slow resolver can delay endpoint availability enough that the first connects to new endpoints look slow. EDS-based clusters (Istio) do not use DNS for endpoint resolution and this check can be skipped.

Metrics and signals to monitor

Fixes

Network congestion or packet loss

Almost never fixable inside Envoy. Confirm the topology change with the network team, then either shift traffic back to the previous path or accept the higher baseline and adjust the alert. If the loss is due to cross-AZ traffic that should be same-zone, check that locality-aware load balancing is configured and that endpoint zones are being correctly advertised in EDS.

Do not raise connect_timeout to mask packet loss. The default of 5 seconds in current Envoy is already generous, and raising it lets Envoy hold pending requests longer, which feeds the connection pool exhaustion cascade. If connect_timeout is set aggressively low (sub-second), the more common problem is spurious upstream_cx_connect_timeout on otherwise healthy upstreams, particularly in local or containerized environments.

Upstream SYN backlog full

The upstream kernel cannot accept connections as fast as Envoy is opening them. Fixes live on the upstream side:

• Raise somaxconn and the application listen backlog so the kernel can queue more pending connections during bursts.
• Confirm the application calls accept() promptly. A slow accept loop fills the backlog even when somaxconn is high.
• Add upstream capacity. If a single host cannot keep up, the correct Envoy-side response is more upstream hosts, not more aggressive connecting.

On the Envoy side, let outlier detection eject the slow host. Do not disable outlier detection to paper over this. You will spread the slowness across more connections and accelerate the failure you were trying to hide.

Firewall or security-group change

upstream_cx_connect_fail jumps at the moment of the change. Find the change in the cloud audit log, revert it, and add a regression test. If the change was intentional, update Envoy’s cluster configuration to point at the now-correct destination.

A common variant: the security group allows traffic from the old Envoy subnet but not the new one after a migration. Symptoms are sudden, total, and accompanied by UF flags in access logs.

DNS resolution delay

Only relevant for STRICT_DNS and LOGICAL_DNS clusters. EDS-based clusters skip this. Fixes:

• Verify the DNS server. dns.cares.pending_resolutions > 100 indicates resolver backlog.
• Tune dns_refresh_rate. Very short refresh intervals overload a slow DNS server.
• Set respect_dns_ttl if your DNS infrastructure relies on TTL for caching.
• Fail over to a different resolver if the DNS server itself is the problem.

Connection pool churn

The cause most often misread as a network problem. The histogram is high because Envoy is doing many connects, each of which pays full TCP and possibly TLS cost. Pool churn has three common drivers:

• Idle timeout too low. common_http_protocol_options.idle_timeout closes connections that would otherwise be reused. Check whether a recent config push lowered it.
• max_connection_duration set. This intentionally cycles connections and shows up as churn. Verify it is set to the value you expect.
• Upstream sending GOAWAY. HTTP/2 upstreams that send GOAWAY force Envoy to open new connections. On older versions, GOAWAY churn may also inflate protocol error counters.

The fix is to stop the churn, not to make each connect faster. Correlate upstream_cx_total, upstream_cx_destroy, and upstream_cx_destroy_local to confirm which side is closing.

Prevention

• Alert on baseline deviation, not fixed thresholds. Same-zone P99 under 2ms is a guideline, not an SLO. The 5x rolling-baseline rule catches real regressions without paging on cross-region traffic that legitimately has higher connect time.
• Track connect rate alongside connect latency. A separate alert on upstream_cx_total rate spiking without a traffic increase catches pool churn before it shows up as user-facing latency.
• Monitor upstream TLS handshake rate. If cluster.<name>.ssl.handshake rate tracks upstream_cx_total rate, every new connect is paying TLS cost. Expected during cold start, pathological during steady state.
• Keep per-host visibility open. The cluster histogram hides bad hosts. During incidents, GET /clusters?format=json is the only way to find them.
• Set connect_timeout explicitly. The default is 5 seconds in current releases. If you depend on a specific value for your failure budget, set it in the cluster config rather than relying on the default.
• Watch kernel-level signals on upstreams. SYN backlog overflows are invisible from Envoy’s stats. Export netstat -s overflow counters from upstream hosts.

How Netdata helps

• Per-second histogram tracking. Netdata collects cluster.<name>.upstream_cx_connect_ms at one-second resolution, so a sudden 5x baseline shift is visible in the same window it happens, not at the next 30-second scrape.
• Anomaly detection on the histogram. A slow drift in connect time, or a step change that does not trip a fixed threshold, still surfaces as an anomaly without manual baseline tuning.
• Correlation with companion counters. upstream_cx_connect_fail, upstream_cx_connect_timeout, upstream_cx_total, and upstream_cx_destroy appear alongside the histogram in the same view, so the slow-vs-failing-vs-churning distinction is a visual one.
• Per-host drill-down. When the cluster aggregate moves, the related signals on outlier ejections, circuit breakers, and upstream health narrow the blast radius to a specific host.
• Worker and TLS signals in the same dashboard. watchdog_miss, ssl.connection_error, and ssl.fail_verify_error sit next to the connect histogram, which shortens the path from “connect time is high” to “TLS handshakes are failing” or “a worker is blocked”.

Related guides

• Envoy 502 and upstream resets: rx_reset, tx_reset, and mid-response failures
• Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend
• Envoy 504 upstream timeout: upstream_rq_timeout, per-try timeouts, and the UT flag
• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy connection pool exhaustion: a slow upstream that fills the pool
• Envoy health checks vs outlier detection: two systems that eject hosts differently
• How Envoy actually works in production: a mental model for operators
• Envoy membership_healthy dropping: reading the single most important cluster signal
• Envoy monitoring checklist: the signals every production proxy needs
• Envoy monitoring maturity model: from survival to expert
• Envoy no healthy upstream: the 503 when a cluster has no host to route to
• Envoy outlier detection mass ejection: when passive health checks empty a cluster