Consul xDS stream churn: Envoy sidecars running on stale configuration

Each Envoy sidecar in a Consul Connect mesh holds a long-lived gRPC xDS stream to a Consul server. That stream is the control plane: endpoint lists, intentions, and mTLS certificate rotations all flow across it. When the stream is healthy, the data plane converges within seconds of a catalog change. When it breaks or churns, the sidecar falls back to whatever Envoy cached last.

The dangerous part is not the disconnect. Envoy keeps its last-known-good configuration when the management server disappears. Traffic flows, health checks pass, and nothing in the data plane alarms. What you get is silent staleness: routes to endpoints removed minutes ago, certificates that have already rotated on the server, intentions that no longer match policy. The first visible symptom is usually a customer-facing incident, not a Consul alert.

This article covers the pattern where consul.xds.server.streams is below the sidecar count, consul.xds.server.streamDrained is climbing, or both. The cause is almost always server-side or in the network path between sidecar and server, not in Envoy itself.

In dot notation the metrics are consul.xds.server.streams (gauge), consul.xds.server.streamDrained (counter), and consul.xds.server.idealStreamsMax (gauge); Prometheus-style names replace dots with underscores.

What this means

xDS is the gRPC protocol Envoy uses to receive dynamic configuration. In Consul Connect, each sidecar opens long-lived streams to a Consul server for CDS, EDS, LDS, and SDS resources and holds them for the life of the proxy. On the Consul side, every stream holds a goroutine and a file descriptor. Server footprint scales with sidecar count, not service count.

When the stream count drops below the sidecar count, some proxies are disconnected. When consul.xds.server.streamDrained increments, the server has told a client to reconnect elsewhere, which only happens when enable_xds_load_balancing is enabled. When streams flap, reconnect storms compete with steady-state work for the same goroutine and FD budget. Disconnected proxies run on cached state until a new stream brings updates.

flowchart TD
    A[Sidecar opens xDS stream] --> B{Stream healthy?}
    B -- yes --> C[Envoy receives updates]
    B -- no --> D[Envoy keeps last-known-good]
    D --> E[Stale routes / endpoints / certs]
    E --> F[Silent customer impact]
    C --> G{Stream flaps?}
    G -- yes --> H[Reconnect storm on server]
    G -- no --> I[Steady state]
    H --> J[FD / goroutine pressure]

Common causes

The 1.14 limiter counted sessions for locally-registered proxies on server agents inconsistently. PR #15789 excluded locally-registered proxies from load balancing, and the fix shipped in Consul 1.15.0.

Quick checks

Run these read-only. None touch the data plane.

# Check daemon responsiveness
curl -s http://127.0.0.1:8500/v1/status/leader

# Active xDS streams on this server
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -E "xds.server.streams|xds.server.streamDrained|xds.server.idealStreamsMax"

# Open file descriptors vs. limit on the server
PID=$(pgrep -x consul); echo "FDs: $(ls /proc/$PID/fd | wc -l)"; grep "Max open files" /proc/$PID/limits

# Server goroutine and memory pressure
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -E "runtime.num_goroutines|runtime.alloc_bytes"

# Stream distribution across servers (run on each server, or query each)
for s in server1 server2 server3; do echo "== $s =="; curl -s http://$s:8500/v1/agent/metrics | grep xds.server.streams; done

# Recent xDS errors on the server (systemd-based hosts)
journalctl -u consul --since '15 min ago' | grep -iE "xds|resource_exhausted|too many streams"

# CA root and leaf status
curl -s http://127.0.0.1:8500/v1/connect/ca/roots | jq '.[] | {Active, Name, NotAfter}'

Compare the active stream count on each server against the expected sidecar count. In a healthy mesh with enable_xds_load_balancing = false, streams distribute according to whichever server address each sidecar resolved first. With load balancing enabled, they should be roughly even up to each server’s idealStreamsMax.

How to diagnose it

1. Confirm the sidecar count. List Connect-enabled services and count proxy instances. The sum is your expected stream count. Compare against the sum of consul.xds.server.streams across servers.

2. Check per-server distribution. If one server holds 80% of streams and the rest hold almost none, you have a load balancing or address configuration problem, not a server health problem. This is common when all sidecars use a single static server address or DNS record that resolves to one IP.

3. Check for RESOURCE_EXHAUSTED. With enable_xds_load_balancing = true, each server computes a stream limit from total mesh proxies, healthy server count, and a buffer. Over the limit, the server returns RESOURCE_EXHAUSTED and asks the client to retry elsewhere. If you see this on Consul 1.14.x at modest scale, upgrade.

The current formula is (proxies / healthy servers) * (1 + 0.1), rounded up; the 10% error margin is the hard-coded errorMargin.

4. Look at server resource pressure. xDS streams each hold a goroutine and an FD. Check consul.runtime.num_goroutines, FD count vs. limit, and memory. A server near its FD limit will refuse new streams before anything else.

5. Check the network path. gRPC is long-lived TCP. Stateful firewalls, conntrack table exhaustion, MTU mismatches, and load balancers with short idle timeouts all break long-lived streams. Look for streams that reconnect on a fixed cadence (commonly 60s or 350s), which points at an idle timeout in the path.

6. Verify TLS. On Kubernetes with external servers, consul-dataplane uses SNI per server. The Helm reference requires tlsServerName when TLS is enabled and hosts is not a DNS name, and defines it as the SNI host header and server certificate hostname. Set it to a DNS SAN common to every server; avoid relying on IP-specific certificates when server addresses can change.

7. Correlate with certificate rotation. SDS pushes new leaf certs over xDS. A rotation storm, often driven by short leaf TTL with mass renewal, can churn streams at the same cadence as the cert lifetime.

Metrics and signals to monitor

Fixes

Server resource exhaustion

The xDS footprint scales with sidecar count, not service count. If your mesh has grown past the server budget, the durable fix is more or larger servers. Short-term: raise the FD ulimit (Consul documentation recommends a minimum of 65536 for servers), add CPU, or reduce concurrent load by disabling non-critical health checks that drive churn. Do not restart servers as a first move. Reconnect storms will hit every surviving server.

RESOURCE_EXHAUSTED with enable_xds_load_balancing = true

Verify you are on Consul 1.15.0 or later if you saw this on 1.14.x. If you are on a fixed version and still see it, the cluster has more mesh proxies than the server fleet can absorb. Either add server capacity or front the servers with an external layer-4 load balancer and disable xDS load balancing so the LB distributes streams.

Single static server address

If every sidecar resolves the same DNS record to the same IP, enable_xds_load_balancing cannot help. Once that server hits its limit, new dataplanes cannot establish streams. Provide multiple server addresses or use server discovery so clients can retry to a different server.

TLS SAN mismatch on Kubernetes external servers

Set tlsServerName in the externalServers stanza to a DNS SAN name that all servers present in their certificate. Without a common DNS SAN, TLS verification can fail as server addresses change or requests target individual server IPs, which prevents streams from spreading normally.

External layer-4 load balancer

If you front the Consul servers with an NLB, HAProxy, or Envoy Gateway, keep enable_xds_load_balancing = false and let the LB distribute connections. Running both creates conflicting redistribution logic.

Network path issues

Look for idle timeouts on any stateful device in the path. gRPC keepalives must be shorter than the shortest idle timeout. Common offenders include cloud NLBs (AWS NLB defaults to 350 seconds), corporate firewalls, and conntrack with nf_conntrack_tcp_timeout_established set low. Raise the timeouts or shorten keepalives to stay under them.

Stuck sidecars that never recover

There is a class of issue where an Envoy sidecar stops receiving xDS updates and the only recovery is restarting the sidecar, the Nomad allocation, or the Kubernetes pod. If a small population of sidecars never re-establishes their stream after a known server event, plan a controlled restart of those specific proxies. Track them by missing stream count per node.

Prevention

• Track sidecar count as a capacity input. Server sizing must account for sidecar count, not just service count. Each sidecar is a goroutine and an FD for the life of the proxy.
• Alert on consul.xds.server.streams below expected. The expected value is your sidecar count. A sustained gap means proxies running on cached state.
• Alert on consul.xds.server.streamDrained rate. Non-zero in steady state means redistribution is active, which means something is pushing streams around.
• Decide your load balancing strategy once. Either enable_xds_load_balancing = true with multiple server addresses, or false with an external LB. Mixing them causes churn.
• Set tlsServerName on Kubernetes external servers. A one-line fix that prevents a class of stream-concentration bugs.
• Monitor FD and goroutine trends. Both are leading indicators before xDS streams start getting refused.
• Keep Consul current. If you are on 1.14.x with Connect at scale, treat upgrading as preventive maintenance.

How Netdata helps

• Per-second consul.xds.server.streams and streamDrained. Stream churn often happens between minute-scale samples. Per-second resolution catches reconnect bursts that aggregated metrics miss.
• Correlate stream count with sidecar count. Netdata’s labels let you compare the sum of xDS streams across servers against the expected proxy count from the catalog. A persistent gap is the earliest reliable signal of stale configuration.
• Server resource saturation alongside xDS. FD utilization, goroutine count, and memory appear on the same time axis as xDS metrics. The cause of stream refusal is usually visible in one of these signals.
• Anomaly detection on streamDrained. Drain rate is near-zero in steady state. Netdata’s anomaly advisor flags the moment that pattern breaks, before you have a customer-facing routing incident.
• CA and certificate signals. Leaf cert errors and CA rotation events correlate with SDS churn. Netdata surfaces consul.connect.ca metrics next to xDS streams so rotation-driven churn is obvious.

Related guides

• Consul blocking query accumulation: leaked watches that pile up goroutines
• Consul catalog bloat: too many services and checks slowing everything down
• Consul registration storm: catalog churn overwhelming Raft
• Consul anti-entropy not syncing: local agent state and the catalog drifting apart
• Consul client rpc failed: agents alive but the catalog is going stale
• Consul DeregisterCriticalServiceAfter: instances vanishing from the catalog
• Consul DNS latency high: slow lookups stalling connections and failovers
• Consul DNS SERVFAIL: service discovery is broken for your applications
• Consul stale DNS queries: the agent is answering from cache
• Consul on EBS: burst-credit exhaustion and the sudden latency cliff
• Consul Go GC pauses: stop-the-world stalls that disturb Raft timing
• Consul goroutine count climbing: the leak behind slow resource exhaustion