Envoy control_plane.connected_state = 0: running on stale xDS config

control_plane.connected_state is a gauge that flips to 0 the moment Envoy’s gRPC stream to its xDS management server drops. The flip produces no user-visible symptom: existing traffic keeps flowing, listener sockets stay open, and Envoy keeps serving its last-known-good configuration. There is no default expiry on that cached config.

That silence is why this signal is under-monitored. By the time anyone notices, new endpoints have been invisible for hours, removed endpoints have been sending traffic to dead or reassigned IPs, routes never updated, and SDS-managed certificates stopped rotating. The disconnect happened long before the visible incident, which makes root-cause correlation non-obvious.

This article covers what the stat means, where staleness bites, how to confirm the disconnect end to end, and how to scope recovery without making it worse. Treat any sustained connected_state = 0 as a deferred failure: page at >5 minutes disconnected, and assume that any routing change, 503, or TLS failure that began after a deploy is a stale-config symptom until proven otherwise.

What this means

control_plane.connected_state is a per-Envoy gauge emitted by the GrpcMux layer that backs ADS and per-type xDS subscriptions. Value 1 means an active xDS stream. Value 0 means Envoy is disconnected from its management server and is not receiving CDS, EDS, LDS, RDS, or SDS updates.

When the gauge is 0, Envoy latches onto its previous configuration and retries the stream in the background. There is no default TTL on those latched resources. The xDS protocol does support a resource TTL feature, but it is opt in: the management server must set TTLs on resources AND the Envoy client must advertise the xds.config.supports-resource-ttl client feature. Without TTL in play, every dynamic resource Envoy has ever received persists indefinitely after disconnect.

A few details that change how you read the signal:

• With ADS (Aggregated Discovery Service), all xDS types share one gRPC stream, so the top-level control_plane.connected_state covers everything.
• Without ADS, each xDS type runs its own stream and the stat is reported under that subscription’s prefix, for example cluster_manager.cds.control_plane.connected_state or listener_manager.lds.control_plane.connected_state. In non-ADS mode a CDS stream can be healthy while LDS is not.
• connected_state = 1 does not mean config is being applied. Envoy can be connected and silently NACKing every update. connected_state only proves the transport is up; it says nothing about whether updates are landing. For that, watch update_rejected.
• control_plane.pending_requests is rate-limit-specific. It tracks pending management-server requests when the control plane is rate-limiting Envoy. It is NOT a general lag indicator and NOT a NACK counter. Do not substitute it for connected_state.
• The signal is irrelevant for static config deployments. Static clusters, static listeners, and file-based certs never depended on a control plane connection.

flowchart TD
  A["xDS stream drops
connected_state = 0"] --> B["Envoy latches config"]
  B --> C["Plateau: traffic looks healthy"]
  C --> D["Hours later:
deploy, scale, or cert rotation"]
  D --> E["New endpoints invisible"]
  D --> F["Dead endpoints still routed"]
  D --> G["SDS certs stop rotating"]
  E --> H["User-visible incident"]
  F --> H
  G --> H

The plateau is what makes this signal easy to miss. Nobody pages during the plateau, and the eventual 503s and TLS failures get triaged as ordinary backend or certificate problems instead of stale config.

Common causes

Quick checks

The commands below are read only. Replace the admin port (9901 here) with 15000 in Istio sidecar mode.

# Confirm the disconnect on a single Envoy
curl -s http://localhost:9901/stats | grep 'control_plane.connected_state'

# Look for per-xDS-type variants in non-ADS deployments
curl -s http://localhost:9901/stats | grep 'connected_state'

# Confirm the xDS upstream cluster has no active connection
curl -s http://localhost:9901/stats | grep -E 'xds.*upstream_cx_active|xds.*upstream_cx_connect_fail'

# Look for NACKs that would explain silent non-application of config
curl -s http://localhost:9901/stats | grep -E 'update_failure|update_rejected|listener_create_failure'

# Check whether new resources are stuck in warming
curl -s http://localhost:9901/stats | grep -E 'warming_clusters|listeners_warming'

# Confirm membership_total is frozen (no scaling events reaching Envoy)
curl -s http://localhost:9901/stats | grep -E 'membership_total|update_success'

# Inspect mTLS health on the xDS cluster
curl -s http://localhost:9901/stats | grep -E 'ssl\.(fail_verify_error|connection_error)'

# Show the version of config Envoy is actually running
curl -s http://localhost:9901/config_dump | jq '.configs[].version_info' 2>/dev/null

# Check TLS cert runway (SDS staleness is the delayed-action failure)
curl -s http://localhost:9901/certs | jq '.certificates[] | {subject: .cert_chain[].subject, days_until_expiration}'

# Cross-check from the control plane side (Istio)
istioctl proxy-status

How to diagnose it

1. Confirm the symptom end to end. Cross-check connected_state = 0 against cluster.<xds_cluster>.upstream_cx_active (zero is consistent with disconnect) and against config_dump showing a version_info that has not moved despite known control plane pushes. In Istio, istioctl proxy-status reports STALE for proxies in this state.

2. Decide whether the problem is Envoy-side or control-plane-side. If every Envoy served by a control plane flipped to 0 in the same window, the control plane is the source. If only some Envoys dropped, look at network reachability, node-level firewall, or per-Envoy auth state. Fleet-wide versus subset is the most useful triage axis.

3. Check transport health to the xDS endpoint. From an affected Envoy, verify TCP reachability to the xDS cluster endpoint. Look at upstream_cx_connect_fail on the xDS cluster, and at ssl.fail_verify_error and ssl.connection_error on the same cluster if mTLS is used. mTLS failures are common after a CA root rotation that has not propagated.

4. Look for NACKs alongside the disconnect. Even after the stream comes back, every update can be silently rejected. Check update_rejected, update_failure, and listener_create_failure. If they are nonzero and climbing, the disconnect may be a side effect of Envoy refusing config rather than a transport problem. NACK reasons are in Envoy’s stderr or process logs, not in stats.

5. Quantify staleness before declaring severity. Compare the version_info in config_dump against the version the control plane thinks it has pushed. Compare membership_total against the orchestrator’s current endpoint count. Compare /certs against the cert rotation schedule. Danger is proportional to how stale the config is, not to the binary fact of disconnection.

6. Verify the reconnect path actually works. Envoy is supposed to retry in the background and reconnect automatically when the control plane is restored. If it does not, an Envoy restart is required. Confirm this is the failure mode before restarting, so you do not restart into the same broken reconnect path.

Metrics and signals to monitor

Fixes

The fixes are grouped by cause. Do not restart Envoy as a first move in any of these branches: restarting throws away the cached config and forces Envoy to wait for the initial xDS push, which makes things strictly worse if the control plane is still degraded.

Control plane is down or overloaded

Restore the control plane first. Envoy reconnects on its own when the gRPC stream is re-established. If the control plane is overloaded (too many sidecars to push to, slow EDS responses), the fix is capacity or push-scoping on the control plane side, not on Envoy. Restarting Envoy into a slow control plane trades cached config for an empty config.

Network partition or firewall change

Restore reachability. Common surprises: a security group change that allows the data-plane subnet but blocks the control-plane subnet, conntrack table exhaustion on the node where Envoy runs, or a kube-proxy / CNI churn that breaks the cluster IP for the xDS service. Verify with a TCP connection test from the affected Envoy to the xDS endpoint, not just a control plane health check.

mTLS cert failure on the xDS cluster

If ssl.fail_verify_error is ticking on the xDS cluster, the cert chain Envoy presents or expects has changed. Push the rotated cert via SDS, restore the previous CA root temporarily, or correct the SAN mismatch. Restarting Envoy will not help until the underlying cert problem is fixed: Envoy will fail the same handshake again.

Connected but NACKing (update_rejected climbing)

This is the variant where connected_state reads 1 but config is still frozen. Find the NACK reason in Envoy’s logs and fix the config on the control plane side. The blast radius is the same as a full disconnect, so the severity response should be the same. A common cause is an incompatible config push: a field the running Envoy version does not understand, a duplicate listener name, or a route referencing a missing cluster.

Envoy won’t reconnect after a hard control-plane reboot

If the control plane has been restored and verified, the network path is clean, and connected_state still stays at 0 on specific Envoys, this matches a known reconnect-handling failure after a hard xDS server reboot. The recovery is an Envoy restart, but confirm the failure mode first. Roll the restart gradually so you do not lose the cached config across the whole fleet at once, and only after the control plane is confirmed healthy.

Restoring SDS after a long disconnect

After reconnect, SDS resumes pushing certificates, but any certificate that should have rotated during the disconnect is now overdue. Check /certs and trigger a rotation cycle if your SDS pipeline supports it. Certificates that expired during the disconnect window are the delayed-action failure: they will start failing TLS handshakes at expiry regardless of the reconnect.

Prevention

• Page on connected_state = 0 sustained for more than 5 minutes. Envoy keeps serving, but stale config is a deferred failure. Treat the disconnect itself as the incident, not the eventual misrouting.
• Monitor disconnect and rejection together. Add update_rejected, update_failure, and listener_create_failure to the same alert. Connected-but-NACKing has the same impact as a full disconnect.
• In non-ADS deployments, monitor per-xDS-type connected_state. A single top-level stat hides partial disconnects where one stream is down and another is up.
• Configure TCP keepalives or HTTP/2 keepalives on the cluster that connects to the management server. This is how Envoy detects a half-open or silently broken connection promptly instead of waiting for the next push.
• Opt into xDS TTL if your control plane supports it. With TTL configured, stale resources expire on a schedule instead of persisting forever. The Envoy client must advertise xds.config.supports-resource-ttl and the management server must set per-resource TTLs.
• Track config version across the fleet. Flag any Envoy whose config_dump version diverges from the version the control plane last pushed. In Istio, istioctl proxy-status surfaces this as STALE.
• Track TLS cert runway via /certs. SDS disconnect is the delayed-action failure mode. The disconnect itself is recoverable; the expired cert is not.
• Do not use control_plane.pending_requests as a general lag indicator. It is rate-limit-specific. Substituting it for connected_state produces false negatives.
• Suppress the alert for static config. The signal is irrelevant when the deployment has no control plane.

How Netdata helps

• Per-second collection of control_plane.connected_state exposes the exact 1-to-0 transition, which matters when correlating the disconnect with a control-plane deploy, a cert rotation, or a node-level event that happened minutes earlier.
• Correlating connected_state with update_rejected, update_failure, and listener_create_failure on a single timeline separates “transport is down” from “transport is up but Envoy is NACKing”, which determines whether the fix is on the network path or on the config side.
• ML-based anomaly detection on per-cluster membership_total surfaces frozen endpoint counts even when the disconnect alert has not fired, catching the connected-but-stale variant.
• Per-cluster ssl.fail_verify_error and ssl.connection_error plotted next to connected_state localizes disconnects to mTLS failures on the xDS cluster.
• Alerting on the duration of connected_state = 0 rather than the binary value produces the >5 minute page without noisy flapping during expected control-plane restarts.

Related guides

• Envoy 502 and upstream resets: rx_reset, tx_reset, and mid-response failures
• Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend
• Envoy 504 upstream timeout: upstream_rq_timeout, per-try timeouts, and the UT flag
• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy connection pool exhaustion: a slow upstream that fills the pool
• Envoy downstream_rq_time high: client-observed latency and proxy overhead
• Envoy health checks vs outlier detection: two systems that eject hosts differently
• How Envoy actually works in production: a mental model for operators
• Envoy membership_healthy dropping: reading the single most important cluster signal
• Envoy monitoring checklist: the signals every production proxy needs
• Envoy monitoring maturity model: from survival to expert
• Envoy no healthy upstream: the 503 when a cluster has no host to route to