A request lands in Envoy, the route matches, the router filter looks up the cluster the route points at, and the cluster manager has no cluster by that name. Envoy fast-fails the request with the NC (No Cluster) response flag. This is not an upstream health problem or a transient network issue. Any non-zero rate of NC in production is a configuration or timing bug.

Unlike UO (circuit breaker) or UF (upstream connection failure), which point at the upstream, NC points at the configuration layer: a route exists in RDS, but the cluster it names is absent from CDS at the instant the request was routed.

The “cluster vanished mid-request” framing describes the operationally interesting case, where a CDS update removed the cluster while in-flight traffic was still being routed to it. But NC also covers the simpler and more common case: a route references a cluster that was never pushed at all.

What this means

NC means: route resolution succeeded, cluster lookup failed. The router filter matched a virtual host and route action, read the cluster name from that action, and asked the cluster manager for a cluster by that name. The cluster manager returned nothing. Envoy has no upstream to forward to, so it fast-fails the request locally.

The critical distinction operators miss:

  • NR (No Route): the route table had no match for the request. The request never reached cluster selection.
  • NC (No Cluster): the route matched, but the cluster it references does not exist.

Both surface as a 503 in aggregate counters. The access log response flag is the only signal that distinguishes them. If your monitoring only looks at downstream_rq_503 or upstream_rq_5xx, you cannot tell NC from NR, and you cannot tell either from a circuit-breaker 503 (UO).

The status code for NC is 503 by default, controlled by cluster_not_found_response_code on RouteAction.

flowchart TD
    A["Request returns 503"] --> B{"Response flag in access log?"}
    B -->|NR| C["No route matched - RDS issue"]
    B -->|NC| D["Route matched, cluster missing"]
    B -->|UO| E["Circuit breaker tripped"]
    B -->|UF| F["Upstream connect failed"]
    D --> G{"Cluster in /clusters now?"}
    G -->|"No, never present"| H["Route refs cluster never created"]
    G -->|"Was present, now gone"| I["CDS removal race"]
    G -->|"Present, but NACK"| J["update_rejected climbing"]

Common causes

CauseWhat it looks likeFirst thing to check
CDS removal raceBrief burst of NC exactly when cluster_manager.cluster_removed increments; resolves once workers convergeCorrelate NC timestamps with cluster_removed and RDS version changes
Route references a cluster never createdPersistent NC for one route; the named cluster is absent from /clustersconfig_dump: confirm the route action’s cluster exists in the active CDS dump
CDS update NACKed while RDS appliedNC starts after a config push and never recovers; update_rejected increments on the clusterCheck update_rejected and Envoy stderr for the NACK reason
xDS delivery ordering (RDS before CDS)NC during convergence, then self-resolves as CDS catches up; warming_clusters nonzero brieflywarming_clusters and active_clusters during the window
Cluster removed, route retainedPersistent NC after an intentional cluster deletion; route was not updatedCompare current RDS and CDS in config_dump
Stale Envoy, new route expectedcontrol_plane.connected_state = 0; route references a cluster the stale config never receivedCheck connected_state and config version

Quick checks

These are read-only and safe during an incident. The admin port is 9901 by default, 15000 in Istio sidecar mode.

# Check daemon responsiveness
time curl --max-time 2 http://localhost:9901/ready

# Is the named cluster present at all?
curl -s http://localhost:9901/clusters | grep -i '<cluster_name>'

# Has any cluster been removed recently?
curl -s http://localhost:9901/stats | grep 'cluster_manager.cluster_removed'

# Clusters warming right now?
curl -s http://localhost:9901/stats | grep -E 'warming_clusters|active_clusters'

# Connected to the control plane?
curl -s http://localhost:9901/stats | grep 'control_plane.connected_state'

# Any config rejections?
curl -s http://localhost:9901/stats | grep -E 'update_rejected|listener_create_failure'

# Full current config (large output, redirect to file)
curl -s http://localhost:9901/config_dump > /tmp/envoy_config_dump.json

On the access log side, count NC specifically and confirm you are not confusing it with NR. Access log location varies: in Istio it is the sidecar stdout, in standalone Envoy it is wherever your access log configuration points.

# Count NC vs NR (adjust path and flag column to your log format)
grep -c ' NC ' /var/log/envoy/access.log
grep -c ' NR ' /var/log/envoy/access.log

How to diagnose it

  1. Confirm the flag is actually NC, not NR. They share a status code and both look like a generic 503 in aggregate counters. The access log is the only place they are distinguished. If you have no response flags in your logs, fix the access log pipeline first.

  2. Determine whether the NC is persistent or bursty. Persistent NC on one route means the cluster is genuinely missing from the active config. A burst that aligns with a config push and then clears is a convergence race.

  3. For persistent NC, pull the config dump and verify the route-to-cluster binding. Find the route configuration in config_dump, read the cluster name from the route action, then confirm that exact name exists in the dynamic active clusters section. A typo, a stale reference, or a missing CDS resource all surface here.

  4. For bursty NC, correlate timestamps. Line up the NC burst against cluster_manager.cluster_removed, cluster_manager.cluster_added, and any RDS version change. If a cluster removal and the NC burst coincide, traffic was still being routed to a cluster that was being deleted.

  5. Check for NACKs. If update_rejected is incrementing on the cluster, Envoy is refusing the CDS update and keeping the old config. If RDS was applied separately and references the new cluster name, NC is the expected result. The NACK reason is in Envoy’s stderr log, not in stats.

  6. Check control_plane.connected_state. A disconnected Envoy runs stale config and will never receive the new cluster. If a route was baked into a newer image while the running instance is stale, NC appears because the expected cluster was never delivered.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Response flag NC in access logsThe only direct signal that a route matched a missing clusterAny non-zero rate in production
cluster_manager.cluster_removedTells you a cluster was deleted from active configSpike correlating with an NC burst indicates a removal race
cluster_manager.cluster_addedConfirms new clusters are arrivingCluster the route expects never appears here
cluster_manager.warming_clustersClusters received but not yet activeNon-zero during convergence can coincide with transient NC
update_rejected (per cluster)Envoy NACKed a CDS updateNon-zero means intended config is not active
control_plane.connected_stateWhether Envoy is receiving config at all0 means stale config; new clusters will never arrive
listener_create_failureListener config rejected (related pipeline health)Non-zero indicates broader config push problems
version_info from config_dumpConfirms which config generation Envoy is onVersion mismatch across instances during a rollout

A hard constraint: response flags are access-log only. They are not exposed as aggregate Prometheus stats. Monitoring NC requires either a log pipeline that counts the flag, or a custom Lua or Wasm filter that increments a counter. A team that alerts on 503 rates but has no response flag visibility will see NC incidents as unexplained 503 spikes.

Fixes

CDS removal race

This is the canonical “cluster vanished mid-request” case. The cluster was being deleted while in-flight requests were still being routed to it.

  • If the removal was intentional, stop routing new requests to the cluster first, let in-flight traffic complete, then remove the cluster. The control plane should update RDS to drop the route before CDS removes the cluster.
  • If the removal was a side effect of an endpoint sweep that escalated into a CDS removal, review the control plane logic. It should not remove a cluster while routes still reference it.
  • Brief bursts that self-resolve are usually tolerable. Persistent NC after a removal is not.

Route references a cluster that was never created

  • Fix the config: add the cluster to CDS, or fix the route to point at a cluster that exists. Verify with config_dump after the push.
  • If the cluster name is correct but CDS is not delivering it, check update_rejected and the NACK reason in stderr.

CDS update NACKed

  • Read the NACK reason from Envoy stderr. Common causes: schema validation failure, duplicate resource names, unsupported filter config, version skew between control plane and Envoy.
  • Fix the invalid config on the control plane side and re-push. Envoy keeps the last-known-good config until a valid one arrives.
  • Do not paper over this with a restart. A restart with no valid config can leave the proxy worse off, and it destroys the diagnostic evidence of which config was rejected.

xDS delivery ordering

  • Transient NC during convergence usually self-resolves once CDS catches up with RDS. If it does not, the cluster is genuinely missing from the control plane’s intended config.
  • The control plane should push the cluster before the route that references it. RDS-before-CDS is the most common NC trigger during rollouts.

Prevention

  • Never deploy route and cluster changes as uncoordinated independent pushes. The control plane should order-guarantee the cluster before the route.
  • Drain before removing. When removing a cluster, update routes to stop referencing it first, let in-flight traffic complete, then remove the cluster.
  • Alert on any NC. Unlike UO or DC, NC has no benign baseline in production. Any occurrence is a config or timing bug worth investigating.
  • Monitor update_rejected alongside connected_state. A connected Envoy that is NACKing every CDS push is functionally as stale as a disconnected one, and the operator usually does not realize it.
  • Include %RESPONSE_FLAGS% in your access log format. Without it, NC and NR are indistinguishable from aggregate 5xx counters. The Istio default format includes it, but custom formats sometimes strip it.
  • Treat NC and NR as separate incident classes. NR is a route table problem. NC is a cluster inventory problem. They have different owners and different fixes.

How Netdata helps

  • Per-second granularity on cluster_manager.cluster_removed, cluster_added, and warming_clusters lets you align a config push against an NC burst within the window needed to confirm or rule out a removal race.
  • control_plane.connected_state and per-cluster update_rejected on the same dashboard as cluster manager stats separates stale config from NACKed pushes from genuinely missing clusters.
  • membership_healthy and circuit breaker state alongside cluster manager stats lets you confirm NC is not UO or UF wearing the same 5xx mask.
  • Response flag counters derived from access logs (via a stats sink or Lua filter) overlay on per-second cluster manager metrics at the granularity where the race actually happens.