$ guides / envoy / envoy-cluster-warming-stuck ▌

Operations Guides

Envoy clusters stuck warming: warming_clusters non-zero and routes returning 503

cluster_manager.warming_clusters reading non-zero during steady state is a config convergence problem. Envoy has accepted a new cluster (or a cluster update) but cannot activate it because a dependency has not resolved: DNS, an SDS secret, an EDS endpoint set, or an active health-check initialization. Until warming completes, the cluster is invisible to routing for new additions, or held in swap-for-update for modifications. Routes targeting a freshly added but still-warming cluster return 503 with response flag NC (no cluster), or 404 with NR (no route), depending on whether the route table entry has been pushed yet.

A brief non-zero warming count during a config push is normal. It becomes actionable when it persists past the push window (typically beyond five minutes) with no corresponding xDS activity.

What this means

A warming cluster is functionally non-existent for routing. For newly added clusters, route resolution cannot find a target: requests return 503 with NC, or 404 with NR if the route entry has not been pushed yet. For updates to an existing cluster, traffic keeps flowing to the old version until the new one warms and is atomically swapped, so user impact is delayed convergence rather than immediate 503s.

Envoy holds cluster updates while they warm: a new or updated cluster does not become active until initial service discovery load completes and, if configured, an initial health-check pass has run. During initialization, CDS/LDS updates do not take effect until the required EDS/RDS responses are supplied. If a warming cluster never resolves, this gating is what blocks the control plane from converging, and downstream resources referencing the warming cluster (listeners, routes) are rejected.

Compare cluster_manager.warming_clusters against cluster_manager.active_clusters. A long-lived mismatch (warming > 0, active < expected) is the signature of a stuck cluster.

Common causes

Cause	What it looks like	First thing to check
DNS resolution stall (STRICT_DNS, LOGICAL_DNS)	`update_failure` incrementing on the cluster; `dns.cares.pending_resolutions` elevated	DNS server reachability from the Envoy pod
SDS secret not delivered	mTLS or TLS cluster; secret in `dynamic_warming_secrets`	`GET /config_dump`, look for `dynamic_warming_secrets`
EDS endpoint set never received	Cluster added but `update_success` never increments	Control plane EDS push logs
xDS sequencing violation	`listener_create_failure` nonzero; LDS rejected	Listener manager stats and Envoy stderr
Slow active health check init	Warming completes only after first probes return	Health-check configuration and target responsiveness
`wait_for_warm_on_init` blocking init	STRICT_DNS clusters take 60s+ to warm on startup	Cluster `wait_for_warm_on_init` field

Quick checks

Read-only and safe to run on a production Envoy.

# Confirm warming state outside a push window
curl -s http://localhost:9901/stats | grep -E '(warming_clusters|active_clusters|total_listeners_warming|total_listeners_active)'

# xDS connectivity - if 0, Envoy is on stale config and cannot resolve new clusters
curl -s http://localhost:9901/stats | grep 'control_plane.connected_state'

# Config rejections and listener create failures
curl -s http://localhost:9901/stats | grep -E '(update_rejected|update_failure|listener_create_failure)'

# DNS resolver state (only relevant for STRICT_DNS / LOGICAL_DNS clusters)
curl -s http://localhost:9901/stats | grep -E 'dns\.'

# Active clusters and their host counts. Warming clusters will NOT appear here.
curl -s http://localhost:9901/clusters?format=json | jq '.cluster_statuses[] | {name, success_rate}' 2>/dev/null || curl -s http://localhost:9901/clusters | head -40

# Full config dump including dynamic_warming_clusters and dynamic_warming_secrets
curl -s http://localhost:9901/config_dump > /tmp/envoy_config_dump.json

# Process state and uptime
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:9901/ready
curl -s http://localhost:9901/stats | grep -E 'server\.(state|uptime)'

In Istio sidecar deployments the admin port is 15000 and the readiness endpoint is 15021 /healthz/ready. Substitute accordingly.

How to diagnose it

Confirm warming is stuck, not transient. Sample cluster_manager.warming_clusters twice, 60 to 120 seconds apart, outside a known config push. If the value is non-zero and stable, the cluster is stuck.
Check control plane connectivity. If control_plane.connected_state is 0, the warming stall is downstream of an xDS disconnect. Resolve the control plane connection before chasing per-cluster causes. Envoy cannot fetch the resources it needs to warm.
Identify which cluster is warming. /clusters?format=json shows only active clusters, so a missing cluster there is a clue. Use /config_dump to enumerate dynamic_active_clusters versus dynamic_warming_clusters. Compare against your expected cluster inventory.
Classify the cluster’s discovery type. The fix path differs by type. STRICT_DNS and LOGICAL_DNS depend on DNS; check dns.cares.resolve_total, dns.cares.pending_resolutions, and cluster.<name>.update_failure. EDS (typical in Istio) depends on the control plane pushing endpoints; check update_success and update_empty. STATIC or ORIGINAL_DST clusters usually stall on health-check initialization or SDS secrets.
Check for SDS dependency. If the cluster has upstream TLS configured, look in /config_dump for dynamic_warming_secrets. A secret with version_info: "uninitialized" indicates SDS has not delivered the cert.
Check for sequencing rejection. If the control plane pushed LDS before CDS ACKed, listeners referencing the warming cluster will be rejected. Look for listener_manager.listener_create_failure increments and “unknown cluster” errors in Envoy stderr.
Check for health-check initialization stalls. If the cluster has active health checks configured and targets are not responding to the probe path, warming cannot complete. Inspect cluster.<name>.health_check.* stats.
Validate DNS resolver health directly. From inside the Envoy pod or host, confirm the resolver Envoy is configured to use actually answers for the cluster’s DNS name.

flowchart TD
    A[warming_clusters > 0] --> B{connected_state = 0?}
    B -- Yes --> C[Fix xDS connection first]
    B -- No --> D{Discovery type}
    D -- DNS-based --> E[Check update_failure and dns.cares]
    D -- EDS --> F[Check update_success and control plane]
    D -- STATIC --> G[Check SDS and health checks]
    E --> H[Fix DNS path or upgrade Envoy]
    F --> I[Verify EDS push from control plane]
    G --> J[Inspect dynamic_warming_secrets]

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`cluster_manager.warming_clusters`	Direct count of clusters stuck pre-activation	Non-zero for >5 min outside a push
`cluster_manager.active_clusters`	Expected active count for comparison	Lower than expected inventory
`listener_manager.total_listeners_warming`	Listener-side convergence state	Non-zero paired with warming clusters
`control_plane.connected_state`	Envoy cannot fetch dependencies if disconnected	0 sustained
`cluster.<name>.update_success` / `update_failure`	Per-cluster discovery outcome	`update_failure` increasing
`cluster.<name>.update_rejected`	Config NACK	Any non-zero value
`listener_manager.listener_create_failure`	LDS rejected	Any non-zero value
`dns.cares.pending_resolutions`	DNS resolver backlog	Sustained > 100
Response flags `NR`, `NC` in access logs	Symptom for routes hitting warming clusters	Non-zero rate outside push

Fixes

DNS resolution stalls (STRICT_DNS, LOGICAL_DNS)

Confirm the configured resolver is reachable from the Envoy network namespace. Common causes are dead CoreDNS pods (Envoy keeps querying the cached resolver IP), NXDOMAIN for the cluster name, or c-ares timeouts.

For Envoy v1.32.1, a c-ares upgrade regression reportedly caused DNS resolution timers to decay toward zero, producing persistent timeouts and stuck warming (Envoy issue 37676). Upgrade to a patched release if you are on that line.

If DNS is genuinely slow but eventually resolves, evaluate wait_for_warm_on_init. The default is true for STRICT_DNS, LOGICAL_DNS, and Redis cluster types. Setting it to false allows initialization to complete without waiting on warm-up, at the cost of brief 503s for routes targeting the not-yet-resolved cluster. Use this only when you understand the tradeoff.

The c-ares shared-channel cookie-state bug (Envoy issue 42836) affects Envoy 1.34+ when a DNS forwarder routes domains to upstreams that disagree on EDNS cookies; it was closed without a fix. The workaround is to configure typed_dns_resolver_config with explicit query_timeout_seconds and query_tries rather than relying on defaults, which isolates cookie state per cluster.

SDS secrets missing

If /config_dump shows a secret in dynamic_warming_secrets with version_info: "uninitialized", SDS has not delivered it. Verify the SDS server is reachable, the resource name in the cluster’s transport_socket matches an SDS-provided secret, and that the SDS connection has not silently dropped.

In some Envoy versions, an SDS server that goes down and returns will not trigger a re-fetch for an already-pending warming secret, leaving the cluster stuck until Envoy is restarted. If you observe this pattern, restarting Envoy is a workaround but will drop all in-flight requests. Track the upstream issue for a proper fix.

EDS endpoint set never received

For EDS clusters (typical in Istio), the control plane must push endpoints before warming can complete. Confirm with the control plane operator that EDS for the cluster’s resource name is being pushed. Check update_empty to see whether the control plane is sending an empty endpoint set, which can happen during scaling events but should not pin warming indefinitely.

initial_fetch_timeout controls how long Envoy waits for the first config response on an xDS subscription. A timeout of 0s means no timeout: Envoy will wait indefinitely. The field default is 15s. Confirm this is intentional for your deployment. Istio’s generated bootstrap sets initial_fetch_timeout: 0s explicitly.

xDS sequencing violations

The required order is CDS, then EDS, then LDS, then RDS. If the control plane sends LDS before CDS has ACKed, Envoy rejects listener updates that reference still-warming clusters. The control plane must wait for the CDS ACK before progressing.

If you operate the control plane, ensure LDS is gated on CDS ACK. With a managed control plane (Istio, Gloo, and similar), this is generally handled correctly, so a sequencing violation suggests a control plane bug rather than a configuration issue.

Health check initialization stalls

If the cluster has active health checks configured, warming does not complete until the first round of probes finishes. If targets are not responding to the configured probe path, port, or expected response, warming hangs.

Inspect cluster.<name>.health_check.* stats to see whether probes are being sent and what response they are getting. A health check endpoint that has been moved or renamed will cause this pattern silently.

Prevention

Alert on warming_clusters > 0 for >5 minutes outside a known push window. Pair with cluster_manager.active_clusters to catch partial convergence.
Alert on listener_create_failure and update_rejected. These surface the upstream cause of warming stalls before user impact.
Track control_plane.connected_state independently. A warming stall downstream of an xDS disconnect is a different incident than a DNS stall.
For STRICT_DNS clusters, monitor cluster.<name>.update_failure and dns.cares.pending_resolutions. DNS stalls are the most common warming cause.
Pin Envoy to versions with known DNS fixes. c-ares regressions produce silent warming stalls.
In service mesh, verify initial_fetch_timeout is set to a finite value. Infinite waits produce indefinite warming.
Validate SDS pipeline health. Cert rotation failures manifest as SDS-not-delivered, which pins any TLS-upstream cluster in warming.

How Netdata helps

Netdata exposes cluster_manager.warming_clusters and active_clusters as per-second gauges, making it easy to distinguish a brief push-driven spike from a sustained plateau.
Correlating warming state with control_plane.connected_state separates “warming because xDS is disconnected” from “warming because a dependency is unresolved”.
Per-cluster update_success, update_failure, and update_rejected breakdowns surface whether the stall is DNS, EDS, or a NACK without manual curl invocations.