Envoy NR no route: 404s and 503s after a bad xDS route push

The NR response flag in Envoy access logs means no route matched the request. The HTTP connection manager walked its route table (delivered via RDS or static config) and found no match for the incoming Host header, path, or other match criteria. In production, sustained NR is a configuration error. A spike right after an xDS route push points to a bad or rejected config update.

NR is usually paired with HTTP 404. Operators sometimes see it paired with 503, which is a different root cause: on Envoy versions before 1.18.0, NR was also set when a route matched but the referenced upstream cluster did not exist. Envoy 1.18.0 (April 2021) introduced the NC (No Cluster) flag for that case, so on modern Envoy, NR with 503 suggests you are either on an older version or actually looking at NC.

The response is an Envoy-generated local reply. No upstream connection is attempted, so cluster health, connection pools, and circuit breakers are irrelevant for NR-flagged requests. The problem is entirely in the route configuration layer.

What this means

NR fires at the router filter level, before cluster selection. The HTTP connection manager receives the request, walks the configured route table, and finds no match. Envoy generates a local response and writes the access log entry with the NR flag.

Three failure modes produce this:

1. Route configuration not delivered: RDS failed to push the route table. The HCM has active listeners but no route configuration attached. Every request to that listener gets NR. Check control_plane.connected_state and whether the route configuration appears in config_dump.

2. Route exists but does not match the request: The route table is present, but the virtual host domains or route matchers do not align with the incoming request’s Host header, path prefix, or headers. The most common variant is a host:port mismatch where the route matches example.com but the client sends Host: example.com:443.

3. Route matches but cluster is missing (pre-1.18.0 behavior): The route entry exists and matches, but the cluster it references is not in the cluster manager. On Envoy before 1.18.0, this produces NR with a 503 status code. On Envoy 1.18.0 and later, this produces NC instead. If you see NR with 503 on modern Envoy, verify you are not confusing this with NC.

The response_code_details access log field provides finer-grained information. The key values are route_not_found (no route matched), route_configuration_not_found (no RDS config at all), and cluster_not_found (route matched but cluster absent).

Verify your access log format includes %RESPONSE_CODE_DETAILS%.

flowchart TD
    A["Access log: NR flag"] --> B{"Route config present
in config_dump?"}
    B -->|"No"| C["RDS not delivered
check connected_state
check update_rejected"]
    B -->|"Yes"| D{"Route domains
match Host header?"}
    D -->|"No"| E["Host:port or domain
mismatch in route config"]
    D -->|"Yes"| F{"Envoy >= 1.18.0
and status is 503?"}
    F -->|"Yes"| G["Check for NC flag
instead of NR
cluster may be missing"]
    F -->|"No"| H["Route matcher
too narrow
path or header criteria"]

Common causes

Quick checks

# Check if control plane is connected
curl -s http://localhost:9901/stats | grep 'control_plane.connected_state'

# Check for rejected config updates (NACKs)
curl -s http://localhost:9901/stats | grep -E '(update_rejected|update_failure|listener_create_failure)'

# Check for clusters or listeners stuck in warming
curl -s http://localhost:9901/stats | grep -E '(warming_clusters|total_listeners_warming)'

# Dump the active route configuration
curl -s http://localhost:9901/config_dump | jq '.configs[] | select(.["@type"]=="type.googleapis.com/envoy.admin.v3.RoutesConfigDump")'

# Verify a specific cluster exists in the active config
curl -s http://localhost:9901/config_dump | jq '.configs[] | select(.["@type"]=="type.googleapis.com/envoy.admin.v3.ClustersConfigDump") | .dynamic_active_clusters[]?.cluster.name'

# List active cluster names and health status
curl -s http://localhost:9901/clusters?format=json | jq '.cluster_statuses[].name'

# Confirm Envoy version (determines NR vs NC split behavior)
curl -s http://localhost:9901/server_info | jq '.version'

All commands are read-only and safe during incidents. In Istio sidecar mode, replace port 9901 with 15000.

How to diagnose it

1. Confirm NR in access logs. Verify your access log format includes %RESPONSE_FLAGS%. In Istio, this is in the default format but may be stripped in custom configurations. Look for lines where the flag field contains NR.

2. Check the HTTP status code paired with NR. If it is 404, the route table has no match for the request. If it is 503, verify whether you are actually seeing NC instead of NR (Envoy 1.18.0 and later), or whether you are on an older Envoy version where NR covered both cases.

3. Check response_code_details if your access log format includes it. route_configuration_not_found means RDS did not deliver the route config at all. route_not_found means the route table exists but no route matched. cluster_not_found means the route matched but the cluster is absent.

4. Correlate with the config push timeline. Compare the timestamp of the NR spike with update_success and update_rejected counters. If update_rejected incremented at the same time, the control plane pushed invalid config and Envoy NACKed it. Envoy keeps the old config, which may be missing routes for newly deployed services. The operator sees “deployment completed” on the control plane side while Envoy silently ignored it.

5. Dump the route configuration. Use the /config_dump admin endpoint to verify the route table is present and matches what the control plane intended to push. Check that the virtual host domains include the Host header values clients are sending.

6. Check for warming or stuck resources. Non-zero cluster_manager.warming_clusters or listener_manager.total_listeners_warming during steady state means Envoy received config but cannot activate it. A cluster stuck in warming does not receive traffic, and routes referencing it may fail.

7. Verify the cluster exists. If the route matched but produced NR with 503 (or NC), check /clusters?format=json for the cluster name the route references. The cluster may have been removed by EDS or never created by CDS.

Metrics and signals to monitor

Response flags are access-log only. Envoy does not expose aggregate NR or NC counters via the stats endpoint or Prometheus. Monitoring them requires access log processing, either through a log pipeline with counters or custom stats via a Lua or Wasm filter.

Fixes

Route config NACKed (update_rejected incrementing)

If Envoy rejected the new route config, the old config remains active. The fix is on the control plane side. Identify why the config was rejected by checking Envoy’s stderr logs for the NACK reason, fix the invalid configuration, and re-push.

Do not restart Envoy as a first fix. A restart triggers a fresh RDS request, and if the control plane sends the same invalid config, Envoy will NACK it again and continue running with stale routes.

If the NR impact is severe and the control plane fix will take time, roll back the control plane to the last-known-good configuration version. This restores the previous route tables.

Host header or domain mismatch

If response_code_details shows route_not_found and the route table exists in config_dump, the issue is in domain matching. Compare the Host header in the access log for NR-flagged requests against the domains list in the virtual host configuration.

Common fixes:

• Add port variants to the virtual host domains list (for example, both example.com and example.com:443).
• Configure strip_any_host_port or strip_matching_host_port on the HTTP connection manager to normalize the Host header before route matching.
• Verify that the path prefix or regex matcher in the route entry covers the request paths that are failing.

Route config not delivered (RDS failure)

If config_dump shows no route configuration for the affected listener, the RDS subscription failed. Check that control_plane.connected_state is 1, that the route configuration resource name in the listener matches what the control plane expects to push, and Envoy’s logs for RDS subscription errors or timeouts.

If the control plane is connected but not delivering the route config, the issue is in the control plane’s resource selection logic. Restarting Envoy will trigger a fresh RDS request but does not fix a control plane that is not responding to it.

Cluster referenced by route is missing

If the route matched but the cluster does not exist, you will see NR with 503 on Envoy before 1.18.0, or NC on Envoy 1.18.0 and later. Check /clusters?format=json for the cluster name. If the cluster was removed via CDS or never created, ensure the control plane pushes the cluster that the route references. If the cluster was removed intentionally, update the route to reference the correct cluster or remove the route entirely.

For incremental xDS (Delta xDS) deployments, clusters can disappear transiently due to resource-tracking eventual consistency during incremental updates. Sustained NR or NC after the config should have converged suggests a subscription or resource-tracking problem in the control plane.

Prevention

Canary config validation. Before a full fleet rollout of route changes, validate the new config statically with envoy --mode validate -c /path/to/config.yaml and push to a small subset of instances first. Watch their access logs for NR or NC before proceeding to the full fleet. Exit code 0 means the config is valid; non-zero means it is not.

Alert on NR and NC in access logs. Since these are not aggregate stats, instrument your log pipeline to count response flags. Any non-zero NR or NC rate should page. These flags indicate configuration errors, not transient failures.

Monitor update_rejected and listener_create_failure. These counters are zero in healthy operation. Non-zero values mean Envoy silently rejected config from the control plane. Alert on any increment.

Include response_code_details in access logs. The default access log format may not include %RESPONSE_CODE_DETAILS%. Adding it gives you immediate root-cause granularity for NR events without requiring a config_dump at incident time.

Review route config changes. Require that route additions and modifications include the full set of domains, path matchers, and cluster references. A mismatched domain or a typo in a cluster name produces NR or NC that is invisible until clients hit the broken route.

How Netdata helps

• Per-second metric resolution lets you pinpoint the exact second update_rejected or listener_create_failure increments, which you can align with the access log timestamp for the first NR-flagged request.
• Anomaly detection on control_plane.connected_state, warming_clusters, and update counters surfaces silent NACKs that would otherwise go unnoticed until clients report failures.
• Correlation across signals lets you overlay xDS update health with downstream 5xx rates, confirming whether a config push caused the error spike without jumping between tools.
• Access log processing through log-based collectors can surface response flag distributions as metrics, giving you NR and NC visibility that Envoy’s native stats endpoint does not provide.
• Composite dashboards showing Envoy, the control plane, and upstream services together help distinguish route-not-found from upstream-unhealthy within the same incident view.

Related guides

• Envoy 502 and upstream resets: rx_reset, tx_reset, and mid-response failures
• Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend
• Envoy 504 upstream timeout: upstream_rq_timeout, per-try timeouts, and the UT flag
• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy connection pool exhaustion: a slow upstream that fills the pool
• Envoy downstream_rq_time high: client-observed latency and proxy overhead
• Envoy health checks vs outlier detection: two systems that eject hosts differently
• How Envoy actually works in production: a mental model for operators
• Envoy membership_healthy dropping: reading the single most important cluster signal
• Envoy monitoring checklist: the signals every production proxy needs
• Envoy monitoring maturity model: from survival to expert
• Envoy no healthy upstream: the 503 when a cluster has no host to route to