Envoy upstream_cx_connect_fail: failed TCP connections to upstream hosts

cluster.<name>.upstream_cx_connect_fail is a per-cluster counter that increments each time Envoy fails to establish a TCP connection to an upstream host. On a healthy cluster the rate is flat. The threshold for concern is connect_fail / connect_total > 0.05 , meaning more than 5% of connection attempts are failing. Sustained nonzero rates point at one of four root causes: the upstream process is down, the upstream is out of file descriptors, a firewall or ACL changed, or the upstream listen backlog is overflowing.

The critical first distinction is between upstream_cx_connect_fail and upstream_cx_connect_timeout. A connect failure is an immediate rejection: Envoy sent SYN and received RST. The port is closed, nothing is listening, the kernel returned ECONNREFUSED. A connect timeout is silence: SYN went out, no SYN-ACK came back. That points at a network partition, a firewall drop, or a saturated listen backlog. These two counters look similar in aggregate dashboards but they point at different layers of the stack.

When a connection fails, the request surfaces to the client as a 503 with response flag UF in the access log. Envoy sets the response code detail to upstream_reset_before_response_started{connection_failure}. If outlier detection is configured with split_external_local_origin_errors: true, repeated local-origin failures increment consecutive_local_origin_failure (default 5) and the host is ejected from the load balancing pool. A single host being ejected is normal resilience. A cluster-wide spike is an active incident.

What this means

The counter is exposed at cluster.<name>.upstream_cx_connect_fail via the admin endpoint (/stats or /stats/prometheus). In Istio sidecar mode, the admin port is typically 15000, not 9901. Because it is a counter, you care about the rate of change, not the absolute value. A cluster that has been running for weeks will have accumulated nonzero value from transient failures during deploys and restarts.

A cluster doing 10,000 new connections per second with a 3% failure rate has a bigger problem than a cluster doing 50 connections per second with a 20% failure rate, because the absolute failure volume is 30x higher. This is why ratio-based alerting is more portable than absolute thresholds.

Connection failures feed into two downstream effects:

• Client-visible errors. Each failed connection that cannot be retried produces a 503 with response flag UF. If retries are configured, Envoy may try another host. If all hosts fail, the client sees the 503.
• Outlier detection ejection. With split_external_local_origin_errors: true, connect failures count as local-origin failures. After consecutive_local_origin_failure (default 5) consecutive failures, the host is ejected. Check cluster.<name>.outlier_detection.ejections_active to see whether Envoy has already removed the failing host from rotation.

Common causes

Quick checks

All commands below are read-only. Adjust the admin port (9901 default, 15000 in Istio sidecar) and cluster name to match your deployment.

# Check connect_fail for a specific cluster
curl -s http://localhost:9901/stats | grep 'cluster.my_cluster.upstream_cx_connect_fail'

# Get connect_total to compute the failure ratio
curl -s http://localhost:9901/stats | grep 'cluster.my_cluster.upstream_cx_total'

# Distinguish from connect_timeout (different root cause)
curl -s http://localhost:9901/stats | grep 'cluster.my_cluster.upstream_cx_connect_timeout'

# Check active upstream host health
curl -s http://localhost:9901/stats | grep 'cluster.my_cluster.membership_healthy'

# Check whether outlier detection has already ejected failing hosts
curl -s http://localhost:9901/stats | grep 'cluster.my_cluster.outlier_detection.ejections_active'

# Per-host health status for the cluster
curl -s http://localhost:9901/clusters?format=json | jq '.cluster_statuses[].host_statuses[].health_status'

# Check upstream TLS errors (if mTLS is in use)
curl -s http://localhost:9901/stats | grep 'cluster.my_cluster.ssl.connection_error'

The UF response flag confirms connection failure as the source of 503s, but it is only available in access logs, not as an aggregate stat. Grep your access log pipeline for the flag if you have one.

How to diagnose it

flowchart TD
    A["upstream_cx_connect_fail rising"] --> B{"connect_timeout also rising?"}
    B -- "Yes" --> C["Network path: firewall drop,
partition, or backlog overflow"]
    B -- "No" --> D{"Scope?"}
    D -- "Single host" --> E["Process down or FD exhaustion"]
    D -- "Cluster-wide" --> F["ACL change or conntrack exhaustion"]
    C --> G["Check connect_ms and network path"]
    E --> H["Check process liveness and FD count"]
    F --> I["Check security groups and conntrack table"]

1. Confirm the signal is real. Sample upstream_cx_connect_fail and upstream_cx_total at two points 10 to 30 seconds apart. Compute the delta and the ratio. A single nonzero sample from hours ago is not an incident. A rising rate is.

2. Distinguish from connect_timeout. If upstream_cx_connect_timeout is also rising, the problem is in the network path or the listen backlog, not the upstream process. A pure connect_fail with zero timeout means the upstream is actively refusing connections via RST.

3. Determine scope. Use /clusters?format=json to check per-host health status. If only one or two hosts are failing, the problem is host-local (process crash, FD exhaustion). If all hosts in a subnet or AZ are failing simultaneously, the problem is in the network path.

4. Check upstream process liveness. On the failing host, verify the process is running and listening on the expected port. A process that crashed, was OOM-killed, or is still initializing will produce immediate RST on connection attempts.

5. Check upstream OS-level constraints. On the failing host, check FD utilization (ls /proc/<pid>/fd | wc -l versus the ulimit) and listen backlog settings (net.core.somaxconn, net.ipv4.tcp_max_syn_backlog). FD exhaustion produces intermittent failures that worsen under load. Backlog overflow produces failures and timeouts together during traffic spikes.

6. Check the network path. If the scope is cluster-wide or subnet-wide, check for recent firewall, security group, or network policy changes. In Kubernetes, check conntrack table utilization. Conntrack exhaustion drops new connections silently and manifests as intermittent upstream_cx_connect_fail with no clear cause in Envoy stats.

7. Verify outlier detection state. Check outlier_detection.ejections_active. If Envoy has already ejected the failing hosts, the cluster may be operating with reduced capacity. If ejections are climbing toward a majority of the cluster, you are at risk of a mass ejection cascade. See Envoy outlier detection mass ejection for that pattern.

Metrics and signals to monitor

Fixes

Upstream process not listening

Confirm the process is running and bound to the expected port. If it crashed, check its logs for the crash cause (OOM, panic, config error). Restart it, but investigate why it crashed. A process that crashes repeatedly under load has a resource exhaustion or bug problem that a restart will not fix.

Upstream FD exhaustion

Check FD usage on the upstream host. If it is at or near the ulimit, the process cannot accept new connections. Raise the ulimit (ulimit -n or the systemd unit’s LimitNOFILE) to give headroom. Also investigate whether the upstream has a connection leak: a process that accumulates open FDs without releasing them will hit any limit eventually. In Kubernetes, verify the actual FD limit inside the container, which may be restricted by securityContext or LimitRange below the node default.

Firewall or ACL change

Identify the change window and revert the rule. If the change was intentional (security hardening), add an exception for the Envoy-to-upstream path. Test connectivity directly from the Envoy host to confirm the path is open before declaring the incident resolved.

Listen backlog overflow

The upstream process is running but its kernel cannot queue incoming SYNs fast enough. Check net.core.somaxconn and net.ipv4.tcp_max_syn_backlog on the upstream host. Both should be set to match the expected peak connection rate. The application’s listen() call must also specify a backlog at least as large, or the kernel setting has no effect.

Conntrack exhaustion

In Kubernetes, conntrack table exhaustion on the node drops new connections silently. Check cat /proc/sys/net/netfilter/nf_conntrack_count versus cat /proc/sys/net/netfilter/nf_conntrack_max. If the table is full, raise net.netfilter.nf_conntrack_max at the node level. This is a node-level sysctl, not a pod-level setting.

Prevention

• Alert on the ratio, not the absolute counter. A fixed threshold on upstream_cx_connect_fail fires differently on quiet and busy clusters. The ratio connect_fail / connect_total > 0.05 is portable across traffic patterns.
• Monitor upstream FD usage as a capacity signal. FD exhaustion is a cliff edge with no graceful degradation. Track the ratio of used to limit on upstream hosts so you have runway before failures start.
• Use both active health checks and outlier detection. Health checks detect hosts that are fully down. Outlier detection catches hosts that are partially degraded. They are complementary, not redundant.
• Monitor conntrack utilization in Kubernetes. Node-level conntrack exhaustion produces symptoms that look like upstream failure but have no signal in Envoy stats.
• Track upstream_cx_connect_ms as a leading indicator. Rising connect latency often precedes outright failures when the upstream SYN backlog is filling.
• Keep access logs with %RESPONSE_FLAGS% enabled. Response flags are the only way to confirm the origin of a 503. They are not available as aggregate stats.

How Netdata helps

• Per-second granularity on upstream_cx_connect_fail rate catches spikes that longer scrape intervals miss, especially during short-lived deployment windows or traffic bursts.
• Correlate connect_fail with connect_timeout in a single view to distinguish immediate RST from network silence without switching between dashboards.
• Cross-reference with membership_healthy and outlier_detection.ejections_active to determine whether Envoy has already ejected failing hosts or whether the cluster is at risk of a mass ejection cascade.
• Track upstream_cx_connect_ms alongside connect_fail to separate slow-but-succeeding connections from hard failures.
• ML anomaly detection on the failure ratio surfaces abnormal patterns before they cross a static threshold, which matters because normal failure rates vary with traffic volume and deploy cadence.

Related guides

• How Envoy actually works in production: a mental model for operators
• Envoy membership_healthy dropping: reading the single most important cluster signal
• Envoy monitoring checklist: the signals every production proxy needs
• Envoy monitoring maturity model: from survival to expert
• Envoy no healthy upstream: the 503 when a cluster has no host to route to
• Envoy outlier detection mass ejection: when passive health checks empty a cluster