cluster_manager.warming_clusters reading non-zero during steady state is a config convergence problem. Envoy has accepted a new cluster (or a cluster update) but cannot activate it because a dependency has not resolved: DNS, an SDS secret, an EDS endpoint set, or an active health-check initialization. Until warming completes, the cluster is invisible to routing for new additions, or held in swap-for-update for modifications. Routes targeting a freshly added but still-warming cluster return 503 with response flag NC (no cluster), or 404 with NR (no route), depending on whether the route table entry has been pushed yet.

A brief non-zero warming count during a config push is normal. It becomes actionable when it persists past the push window (typically beyond five minutes) with no corresponding xDS activity.

What this means

A warming cluster is functionally non-existent for routing. For newly added clusters, route resolution cannot find a target: requests return 503 with NC, or 404 with NR if the route entry has not been pushed yet. For updates to an existing cluster, traffic keeps flowing to the old version until the new one warms and is atomically swapped, so user impact is delayed convergence rather than immediate 503s.

Envoy is documented to pause CDS ACKs while any cluster is warming, resuming only once all warming clusters have initialized. If a warming cluster never resolves, the ACK never goes out, the control plane cannot progress the xDS sequence, and downstream resources referencing the warming cluster (listeners, routes) are rejected.

Compare cluster_manager.warming_clusters against cluster_manager.active_clusters. A long-lived mismatch (warming > 0, active < expected) is the signature of a stuck cluster.

Common causes

CauseWhat it looks likeFirst thing to check
DNS resolution stall (STRICT_DNS, LOGICAL_DNS)update_failure incrementing on the cluster; dns.cares.pending_resolutions elevatedDNS server reachability from the Envoy pod
SDS secret not deliveredmTLS or TLS cluster; secret in dynamic_warming_secretsGET /config_dump, look for dynamic_warming_secrets
EDS endpoint set never receivedCluster added but update_success never incrementsControl plane EDS push logs
xDS sequencing violationlistener_create_failure nonzero; LDS rejectedListener manager stats and Envoy stderr
Slow active health check initWarming completes only after first probes returnHealth-check configuration and target responsiveness
wait_for_warm_on_init blocking initSTRICT_DNS clusters take 60s+ to warm on startupCluster wait_for_warm_on_init field

Quick checks

Read-only and safe to run on a production Envoy.

# Confirm warming state outside a push window
curl -s http://localhost:9901/stats | grep -E '(warming_clusters|active_clusters|total_listeners_warming|total_listeners_active)'

# xDS connectivity - if 0, Envoy is on stale config and cannot resolve new clusters
curl -s http://localhost:9901/stats | grep 'control_plane.connected_state'

# Config rejections and listener create failures
curl -s http://localhost:9901/stats | grep -E '(update_rejected|update_failure|listener_create_failure)'

# DNS resolver state (only relevant for STRICT_DNS / LOGICAL_DNS clusters)
curl -s http://localhost:9901/stats | grep -E 'dns\.'

# Active clusters and their host counts. Warming clusters will NOT appear here.
curl -s http://localhost:9901/clusters?format=json | jq '.cluster_statuses[] | {name, success_rate}' 2>/dev/null || curl -s http://localhost:9901/clusters | head -40

# Full config dump including dynamic_warming_clusters and dynamic_warming_secrets
curl -s http://localhost:9901/config_dump > /tmp/envoy_config_dump.json

# Process state and uptime
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:9901/ready
curl -s http://localhost:9901/stats | grep -E 'server\.(state|uptime)'

In Istio sidecar deployments the admin port is 15000 and the readiness endpoint is 15021 /healthz/ready. Substitute accordingly.

How to diagnose it

  1. Confirm warming is stuck, not transient. Sample cluster_manager.warming_clusters twice, 60 to 120 seconds apart, outside a known config push. If the value is non-zero and stable, the cluster is stuck.

  2. Check control plane connectivity. If control_plane.connected_state is 0, the warming stall is downstream of an xDS disconnect. Resolve the control plane connection before chasing per-cluster causes. Envoy cannot fetch the resources it needs to warm.

  3. Identify which cluster is warming. /clusters?format=json shows only active clusters, so a missing cluster there is a clue. Use /config_dump to enumerate dynamic_active_clusters versus dynamic_warming_clusters. Compare against your expected cluster inventory.

  4. Classify the cluster’s discovery type. The fix path differs by type. STRICT_DNS and LOGICAL_DNS depend on DNS; check dns.cares.resolve_total, dns.cares.pending_resolutions, and cluster.<name>.update_failure. EDS (typical in Istio) depends on the control plane pushing endpoints; check update_success and update_empty. STATIC or ORIGINAL_DST clusters usually stall on health-check initialization or SDS secrets.

  5. Check for SDS dependency. If the cluster has upstream TLS configured, look in /config_dump for dynamic_warming_secrets. A secret with version_info: "uninitialized" indicates SDS has not delivered the cert.

  6. Check for sequencing rejection. If the control plane pushed LDS before CDS ACKed, listeners referencing the warming cluster will be rejected. Look for listener_manager.listener_create_failure increments and “unknown cluster” errors in Envoy stderr.

  7. Check for health-check initialization stalls. If the cluster has active health checks configured and targets are not responding to the probe path, warming cannot complete. Inspect cluster.<name>.health_check.* stats.

  8. Validate DNS resolver health directly. From inside the Envoy pod or host, confirm the resolver Envoy is configured to use actually answers for the cluster’s DNS name.

flowchart TD
    A[warming_clusters > 0] --> B{connected_state = 0?}
    B -- Yes --> C[Fix xDS connection first]
    B -- No --> D{Discovery type}
    D -- DNS-based --> E[Check update_failure and dns.cares]
    D -- EDS --> F[Check update_success and control plane]
    D -- STATIC --> G[Check SDS and health checks]
    E --> H[Fix DNS path or upgrade Envoy]
    F --> I[Verify EDS push from control plane]
    G --> J[Inspect dynamic_warming_secrets]

Metrics and signals to monitor

SignalWhy it mattersWarning sign
cluster_manager.warming_clustersDirect count of clusters stuck pre-activationNon-zero for >5 min outside a push
cluster_manager.active_clustersExpected active count for comparisonLower than expected inventory
listener_manager.total_listeners_warmingListener-side convergence stateNon-zero paired with warming clusters
control_plane.connected_stateEnvoy cannot fetch dependencies if disconnected0 sustained
cluster.<name>.update_success / update_failurePer-cluster discovery outcomeupdate_failure increasing
cluster.<name>.update_rejectedConfig NACKAny non-zero value
listener_manager.listener_create_failureLDS rejectedAny non-zero value
dns.cares.pending_resolutionsDNS resolver backlogSustained > 100
Response flags NR, NC in access logsSymptom for routes hitting warming clustersNon-zero rate outside push

Fixes

DNS resolution stalls (STRICT_DNS, LOGICAL_DNS)

Confirm the configured resolver is reachable from the Envoy network namespace. Common causes are dead CoreDNS pods (Envoy keeps querying the cached resolver IP), NXDOMAIN for the cluster name, or c-ares timeouts.

For Envoy v1.32.1, a c-ares upgrade regression reportedly caused DNS resolution timers to decay toward zero, producing persistent timeouts and stuck warming. Upgrade to a patched release if you are on that line.

If DNS is genuinely slow but eventually resolves, evaluate wait_for_warm_on_init. The default is true for STRICT_DNS, LOGICAL_DNS, and Redis cluster types. Setting it to false allows initialization to complete without waiting on warm-up, at the cost of brief 503s for routes targeting the not-yet-resolved cluster. Use this only when you understand the tradeoff.

SDS secrets missing

If /config_dump shows a secret in dynamic_warming_secrets with version_info: "uninitialized", SDS has not delivered it. Verify the SDS server is reachable, the resource name in the cluster’s transport_socket matches an SDS-provided secret, and that the SDS connection has not silently dropped.

In some Envoy versions, an SDS server that goes down and returns will not trigger a re-fetch for an already-pending warming secret, leaving the cluster stuck until Envoy is restarted. If you observe this pattern, restarting Envoy is a workaround but will drop all in-flight requests. Track the upstream issue for a proper fix.

EDS endpoint set never received

For EDS clusters (typical in Istio), the control plane must push endpoints before warming can complete. Confirm with the control plane operator that EDS for the cluster’s resource name is being pushed. Check update_empty to see whether the control plane is sending an empty endpoint set, which can happen during scaling events but should not pin warming indefinitely.

initial_fetch_timeout controls how long Envoy waits for the initial EDS response. A timeout of 0s means Envoy will wait indefinitely. Confirm this is intentional for your deployment.

xDS sequencing violations

The required order is CDS, then EDS, then LDS, then RDS. If the control plane sends LDS before CDS has ACKed, Envoy rejects listener updates that reference still-warming clusters. The control plane must wait for the CDS ACK before progressing.

If you operate the control plane, ensure LDS is gated on CDS ACK. With a managed control plane (Istio, Gloo, and similar), this is generally handled correctly, so a sequencing violation suggests a control plane bug rather than a configuration issue.

Health check initialization stalls

If the cluster has active health checks configured, warming does not complete until the first round of probes finishes. If targets are not responding to the configured probe path, port, or expected response, warming hangs.

Inspect cluster.<name>.health_check.* stats to see whether probes are being sent and what response they are getting. A health check endpoint that has been moved or renamed will cause this pattern silently.

Prevention

  • Alert on warming_clusters > 0 for >5 minutes outside a known push window. Pair with cluster_manager.active_clusters to catch partial convergence.
  • Alert on listener_create_failure and update_rejected. These surface the upstream cause of warming stalls before user impact.
  • Track control_plane.connected_state independently. A warming stall downstream of an xDS disconnect is a different incident than a DNS stall.
  • For STRICT_DNS clusters, monitor cluster.<name>.update_failure and dns.cares.pending_resolutions. DNS stalls are the most common warming cause.
  • Pin Envoy to versions with known DNS fixes. c-ares regressions produce silent warming stalls.
  • In service mesh, verify initial_fetch_timeout is set to a finite value. Infinite waits produce indefinite warming.
  • Validate SDS pipeline health. Cert rotation failures manifest as SDS-not-delivered, which pins any TLS-upstream cluster in warming.

How Netdata helps

  • Netdata exposes cluster_manager.warming_clusters and active_clusters as per-second gauges, making it easy to distinguish a brief push-driven spike from a sustained plateau.
  • Correlating warming state with control_plane.connected_state separates “warming because xDS is disconnected” from “warming because a dependency is unresolved”.
  • Per-cluster update_success, update_failure, and update_rejected breakdowns surface whether the stall is DNS, EDS, or a NACK without manual curl invocations.