$ guides / envoy / envoy-nc-no-cluster ▌

Operations Guides

Envoy NC no cluster: the cluster vanished mid-request

A request lands in Envoy, the route matches, the router filter looks up the cluster the route points at, and the cluster manager has no cluster by that name. Envoy fast-fails the request with the NC (No Cluster) response flag. This is not an upstream health problem or a transient network issue. Any non-zero rate of NC in production is a configuration or timing bug.

Unlike UO (circuit breaker) or UF (upstream connection failure), which point at the upstream, NC points at the configuration layer: a route exists in RDS, but the cluster it names is absent from CDS at the instant the request was routed.

The “cluster vanished mid-request” framing describes the operationally interesting case, where a CDS update removed the cluster while in-flight traffic was still being routed to it. But NC also covers the simpler and more common case: a route references a cluster that was never pushed at all.

What this means

NC means: route resolution succeeded, cluster lookup failed. The router filter matched a virtual host and route action, read the cluster name from that action, and asked the cluster manager for a cluster by that name. The cluster manager returned nothing. Envoy has no upstream to forward to, so it fast-fails the request locally.

The critical distinction operators miss:

NR (No Route): the route table had no match for the request. The request never reached cluster selection.
NC (No Cluster): the route matched, but the cluster it references does not exist.

Both surface as a 503 in aggregate counters. The access log response flag is the only signal that distinguishes them. If your monitoring only looks at downstream_rq_503 or upstream_rq_5xx, you cannot tell NC from NR, and you cannot tell either from a circuit-breaker 503 (UO).

The status code for NC is 503 by default, controlled by cluster_not_found_response_code on RouteAction (the API defines it simply as “the HTTP status code to use when configured cluster is not found”, with no restriction to cluster-header selection, so it applies to direct cluster references as well).

flowchart TD
    A["Request returns 503"] --> B{"Response flag in access log?"}
    B -->|NR| C["No route matched - RDS issue"]
    B -->|NC| D["Route matched, cluster missing"]
    B -->|UO| E["Circuit breaker tripped"]
    B -->|UF| F["Upstream connect failed"]
    D --> G{"Cluster in /clusters now?"}
    G -->|"No, never present"| H["Route refs cluster never created"]
    G -->|"Was present, now gone"| I["CDS removal race"]
    G -->|"Present, but NACK"| J["update_rejected climbing"]

Common causes

Cause	What it looks like	First thing to check
CDS removal race	Brief burst of `NC` exactly when `cluster_manager.cluster_removed` increments; resolves once workers converge	Correlate `NC` timestamps with `cluster_removed` and RDS version changes
Route references a cluster never created	Persistent `NC` for one route; the named cluster is absent from `/clusters`	`config_dump`: confirm the route action’s cluster exists in the active CDS dump
CDS update NACKed while RDS applied	`NC` starts after a config push and never recovers; `update_rejected` increments on the cluster	Check `update_rejected` and Envoy stderr for the NACK reason
xDS delivery ordering (RDS before CDS)	`NC` during convergence, then self-resolves as CDS catches up; `warming_clusters` nonzero briefly	`warming_clusters` and `active_clusters` during the window
Cluster removed, route retained	Persistent `NC` after an intentional cluster deletion; route was not updated	Compare current RDS and CDS in `config_dump`
Stale Envoy, new route expected	`control_plane.connected_state = 0`; route references a cluster the stale config never received	Check `connected_state` and config version

Quick checks

These are read-only and safe during an incident. The admin port is 9901 by default, 15000 in Istio sidecar mode.

# Check daemon responsiveness
time curl --max-time 2 http://localhost:9901/ready

# Is the named cluster present at all?
curl -s http://localhost:9901/clusters | grep -i '<cluster_name>'

# Has any cluster been removed recently?
curl -s http://localhost:9901/stats | grep 'cluster_manager.cluster_removed'

# Clusters warming right now?
curl -s http://localhost:9901/stats | grep -E 'warming_clusters|active_clusters'

# Connected to the control plane?
curl -s http://localhost:9901/stats | grep 'control_plane.connected_state'

# Any config rejections?
curl -s http://localhost:9901/stats | grep -E 'update_rejected|listener_create_failure'

# Full current config (large output, redirect to file)
curl -s http://localhost:9901/config_dump > /tmp/envoy_config_dump.json

On the access log side, count NC specifically and confirm you are not confusing it with NR. Access log location varies: in Istio it is the sidecar stdout, in standalone Envoy it is wherever your access log configuration points.

# Count NC vs NR (adjust path and flag column to your log format)
grep -c ' NC ' /var/log/envoy/access.log
grep -c ' NR ' /var/log/envoy/access.log

How to diagnose it

Confirm the flag is actually NC, not NR. They share a status code and both look like a generic 503 in aggregate counters. The access log is the only place they are distinguished. If you have no response flags in your logs, fix the access log pipeline first.
Determine whether the NC is persistent or bursty. Persistent NC on one route means the cluster is genuinely missing from the active config. A burst that aligns with a config push and then clears is a convergence race.
For persistent NC, pull the config dump and verify the route-to-cluster binding. Find the route configuration in config_dump, read the cluster name from the route action, then confirm that exact name exists in the dynamic active clusters section. A typo, a stale reference, or a missing CDS resource all surface here.
For bursty NC, correlate timestamps. Line up the NC burst against cluster_manager.cluster_removed, cluster_manager.cluster_added, and any RDS version change. If a cluster removal and the NC burst coincide, traffic was still being routed to a cluster that was being deleted.
Check for NACKs. If update_rejected is incrementing on the cluster, Envoy is refusing the CDS update and keeping the old config. If RDS was applied separately and references the new cluster name, NC is the expected result. The NACK reason is in Envoy’s stderr log, not in stats.
Check control_plane.connected_state. A disconnected Envoy runs stale config and will never receive the new cluster. If a route was baked into a newer image while the running instance is stale, NC appears because the expected cluster was never delivered.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Response flag `NC` in access logs	The only direct signal that a route matched a missing cluster	Any non-zero rate in production
`cluster_manager.cluster_removed`	Tells you a cluster was deleted from active config	Spike correlating with an `NC` burst indicates a removal race
`cluster_manager.cluster_added`	Confirms new clusters are arriving	Cluster the route expects never appears here
`cluster_manager.warming_clusters`	Clusters received but not yet active	Non-zero during convergence can coincide with transient `NC`
`update_rejected` (per cluster)	Envoy NACKed a CDS update	Non-zero means intended config is not active
`control_plane.connected_state`	Whether Envoy is receiving config at all	0 means stale config; new clusters will never arrive
`listener_create_failure`	Listener config rejected (related pipeline health)	Non-zero indicates broader config push problems
`version_info` from `config_dump`	Confirms which config generation Envoy is on	Version mismatch across instances during a rollout

A hard constraint: response flags are access-log only. They are not exposed as aggregate Prometheus stats. Monitoring NC requires either a log pipeline that counts the flag, or a custom Lua or Wasm filter that increments a counter. A team that alerts on 503 rates but has no response flag visibility will see NC incidents as unexplained 503 spikes.

Fixes

CDS removal race

This is the canonical “cluster vanished mid-request” case. The cluster was being deleted while in-flight requests were still being routed to it.

If the removal was intentional, stop routing new requests to the cluster first, let in-flight traffic complete, then remove the cluster. The control plane should update RDS to drop the route before CDS removes the cluster.
If the removal was a side effect of an endpoint sweep that escalated into a CDS removal, review the control plane logic. It should not remove a cluster while routes still reference it.
Brief bursts that self-resolve are usually tolerable. Persistent NC after a removal is not.

Route references a cluster that was never created

Fix the config: add the cluster to CDS, or fix the route to point at a cluster that exists. Verify with config_dump after the push.
If the cluster name is correct but CDS is not delivering it, check update_rejected and the NACK reason in stderr.
A route that statically references a cluster missing at bootstrap fails initialization outright, confirmed in envoyproxy/envoy#16850.

CDS update NACKed

Read the NACK reason from Envoy stderr. Common causes: schema validation failure, duplicate resource names, unsupported filter config, version skew between control plane and Envoy.
Fix the invalid config on the control plane side and re-push. Envoy keeps the last-known-good config until a valid one arrives.
Do not paper over this with a restart. A restart with no valid config can leave the proxy worse off, and it destroys the diagnostic evidence of which config was rejected.

xDS delivery ordering

Transient NC during convergence usually self-resolves once CDS catches up with RDS. If it does not, the cluster is genuinely missing from the control plane’s intended config.
The control plane should push the cluster before the route that references it. RDS-before-CDS is the most common NC trigger during rollouts.
On-demand CDS can produce request failures that look similar, but issue 20873 (still open) describes them as requests failing because hosts were not yet added to the returned cluster — closer to UF/no-healthy-upstream territory than NC. Treat any NC during on-demand CDS with that caveat.

Prevention

Never deploy route and cluster changes as uncoordinated independent pushes. The control plane should order-guarantee the cluster before the route.
Drain before removing. When removing a cluster, update routes to stop referencing it first, let in-flight traffic complete, then remove the cluster.
Alert on any NC. Unlike UO or DC, NC has no benign baseline in production. Any occurrence is a config or timing bug worth investigating.
Monitor update_rejected alongside connected_state. A connected Envoy that is NACKing every CDS push is functionally as stale as a disconnected one, and the operator usually does not realize it.
Include %RESPONSE_FLAGS% in your access log format. Without it, NC and NR are indistinguishable from aggregate 5xx counters. The Istio default format includes it, but custom formats sometimes strip it.
Treat NC and NR as separate incident classes. NR is a route table problem. NC is a cluster inventory problem. They have different owners and different fixes.

How Netdata helps

Per-second granularity on cluster_manager.cluster_removed, cluster_added, and warming_clusters lets you align a config push against an NC burst within the window needed to confirm or rule out a removal race.
control_plane.connected_state and per-cluster update_rejected on the same dashboard as cluster manager stats separates stale config from NACKed pushes from genuinely missing clusters.
membership_healthy and circuit breaker state alongside cluster manager stats lets you confirm NC is not UO or UF wearing the same 5xx mask.
Response flag counters derived from access logs (via a stats sink or Lua filter) overlay on per-second cluster manager metrics at the granularity where the race actually happens.