Envoy update_rejected (xDS NACK): the config the control plane pushed and Envoy quietly refused

You deploy a config change through your control plane. The deployment reports success. Traffic flows, error rates are flat, control_plane.connected_state stays at 1. But the change never took effect: the new route is missing, the timeout override is gone, the certificate rotation did not happen.

Envoy received the new config, validated it, found it invalid, and rejected it. It kept the previous config and kept routing. The rejection is recorded in update_rejected and, for listeners, in listener_manager.listener_create_failure. The reason itself is only in the Envoy process log, not in stats. The control plane reports success because the gRPC stream is healthy, not because Envoy accepted the config. Nothing fails visibly. The only symptoms are the things that should have changed but did not.

What this means

When Envoy receives a config update via xDS (CDS, EDS, LDS, RDS, SDS), it validates the new resources before applying them. Validation covers protoc-gen-validate field constraints and internal consistency checks: whether referenced clusters exist, whether referenced secrets are available, whether resource names are unique within a response. If validation fails, Envoy rejects the update. In xDS protocol terms, this is a NACK: Envoy sends back a DiscoveryRequest with the rejection detail and continues serving with the last known good config.

A NACK does not break the data plane. Envoy does not crash, does not drop listeners, does not return errors. It keeps the previous config intact. This protects the data plane from a bad control plane push, but it also makes the rejection invisible unless you are actively looking for it.

The counters that track rejections:

• cluster.<name>.update_rejected: increments when Envoy NACKs a cluster config update (CDS or EDS). Primary NACK signal for cluster resources.
• listener_manager.listener_create_failure: increments when a listener fails to create (LDS). This covers both validation rejection and bind failures such as a port already in use.
• cluster.<name>.update_failure: a different signal. Counts delivery or processing failures (network issues, control plane errors), not validation rejections. Do not confuse it with update_rejected.

Both update_rejected and listener_create_failure should be zero in steady state. Any nonzero value warrants investigation.

The rejection reason is not in stats. Envoy logs it at warning level to its process log. Stats tell you a rejection happened. Logs tell you why.

Do not use control_plane.pending_requests to diagnose NACKs. That stat tracks pending management-server requests when rate limiting is enforced, not general control plane lag or config rejection.

flowchart TD
    A[Control plane pushes config] --> B{Envoy validates}
    B -->|Valid| C[Apply new config]
    B -->|Invalid| D[NACK the update]
    D --> E[Keep previous config]
    E --> F[Data plane unaffected]
    D --> G[update_rejected increments]
    D --> H[Reason logged at warning level]
    F --> I[connected_state stays 1]
    I --> J[Control plane reports success]
    G --> K[Silent unless monitored]
    H --> K

Common causes

Quick checks

The admin port is 9901 by default and 15000 in Istio sidecar mode. Adjust the examples below to match your deployment.

# Check for NACK counters across clusters and listeners
curl -s http://localhost:9901/stats | grep -E '(update_rejected|listener_create_failure|update_failure)'

# Confirm the control plane connection is up (it will be, even during NACKs)
curl -s http://localhost:9901/stats | grep 'control_plane.connected_state'

# Check for resources stuck in warming (secondary symptom of rejected dependencies)
curl -s http://localhost:9901/stats | grep -E '(warming_clusters|total_listeners_warming)'

# Show version_info for each dynamic active cluster, compare to what the control plane expects
curl -s http://localhost:9901/config_dump | python3 -c "
import sys, json
d = json.load(sys.stdin)
for c in d.get('configs', []):
    for r in c.get('dynamic_active_clusters', []):
        print(r.get('cluster', {}).get('name'), r.get('version_info'))
"

# Find the rejection reason in the Envoy process log (path varies by deployment)
# In Kubernetes with Istio, search recent istio-proxy logs:
kubectl logs <pod> -c istio-proxy --since=5m | grep -i 'reject'

How to diagnose it

1. Confirm the rejection. Check update_rejected and listener_create_failure. If both are zero and not increasing, the config was accepted. Your issue is elsewhere (wrong control plane, wrong Envoy instance, stale client cache).

2. Confirm connected_state is 1. A NACK with connected_state = 1 is the classic silent rejection pattern. If connected_state is 0, you have a connectivity problem, not a NACK.

3. Find the rejection reason in the Envoy log. The stat tells you a rejection happened. The log tells you why. Look for warning-level lines containing the xDS type URL and the validation error string. The exact format varies by Envoy version, but the pattern includes the resource type and a human-readable error.

4. Compare the active config with what the control plane thinks it pushed. Use config_dump and check the version_info of each dynamic resource. If the version does not match what the control plane reports as current, that resource was rejected or never delivered.

5. Check the control plane side. If you run Istio, pilot_xds_push_errors on istiod shows push failures from the control plane’s perspective. Cross-reference this with Envoy’s update_rejected to confirm you are looking at the same rejection event.

6. Check warming state as a secondary indicator. cluster_manager.warming_clusters and listener_manager.total_listeners_warming nonzero during steady state can indicate config convergence problems. A cluster stuck warming because its referenced SDS secret was rejected is a common pattern. Warming alone does not confirm a NACK, but it narrows the investigation.

7. Note the subscription type. For filesystem-based xDS subscriptions, there is no ACK or NACK at the protocol level. Only stats counters and logs exist, and the last valid configuration continues to apply. The rejection still shows up in update_rejected and logs, but there is no protocol-level feedback because there is no control plane stream.

Metrics and signals to monitor

Fixes

Fix the config that caused the NACK

The rejection reason in the Envoy log tells you what is wrong. Match the error to the cause:

• Unknown cluster reference: ensure the cluster exists in the same or a prior xDS response before the route that references it. CDS and RDS ordering matters. If the control plane pushes RDS before CDS, Envoy cannot validate the route.
• Duplicate resource names: fix the control plane resource generation to produce unique names within a single DiscoveryResponse.
• Schema validation failure: the PGV error string identifies the offending field. Fix the config to satisfy the constraint.
• Static config conflict: you cannot modify or remove statically-defined resources via xDS. Either remove them from the bootstrap config or stop pushing updates for them dynamically.
• Unknown filter or extension: ensure the extension is compiled into your Envoy build. Not all Envoy distributions include all extensions. Check the build flags or use GET /server_info to inspect compiled-in extensions.
• Transport version mismatch: the v2 xDS transport API has been removed in current Envoy versions. Ensure both the control plane and Envoy are configured for v3.

Verify the fix took effect

After pushing the corrected config:

1. Check that update_rejected stops incrementing for the affected resource.
2. Check that cluster.<name>.update_success (clusters) or listener_manager.listener_added / listener_manager.listener_modified (listeners) increments.
3. Confirm warming_clusters and total_listeners_warming return to zero.
4. Confirm via config_dump that the version_info matches what the control plane reports as current.

Do not restart Envoy to force the config

Restarting Envoy will not help if the config is invalid. On restart, Envoy requests the config again, receives the same invalid config, and NACKs it again. Fix the config first.

If the control plane is sending different config to different Envoys (version drift across the fleet), restarting a specific instance may cause it to receive a different response, but this is coincidental. The root cause is control plane inconsistency.

Prevention

• Alert on update_rejected and listener_create_failure. Any nonzero value is a config bug. Alert on the rate of change, not just the absolute counter.
• Do not trust connected_state alone. A connected Envoy can be NACKing every update. The pair connected_state = 1 plus update_rejected > 0 is the silent NACK signature.
• Ship the rejection reason to your log pipeline. The reason is only in Envoy’s process log. If you do not collect and index it, you will know a NACK happened but not why.
• Verify config convergence after every control plane deployment. Check config_dump version_info on a sample of instances. If versions diverge after a push, some instances are rejecting.
• Track config version consistency across the fleet. If 100 sidecars are on version X and 3 are on version Y, those 3 are likely rejecting updates or failed to receive them.
• Test config validation in CI. Run the same protoc-gen-validate checks the control plane uses before pushing to production. Catch invalid configs before they reach any Envoy.
• Order your xDS pushes correctly. Push CDS before RDS that references those clusters. Push SDS secrets before the listeners that reference them. Ordering violations are a frequent cause of transient NACKs during rollouts.

How Netdata helps

Netdata collects update_rejected, listener_create_failure, update_success, and warming state at per-second resolution on each node, alongside control_plane.connected_state. This lets you confirm the silent NACK signature (connected_state = 1 with update_rejected > 0) in a single view, and catch config version drift across the fleet as diverging metric patterns on different nodes before stale config causes a user-facing incident.

Related guides

• How Envoy actually works in production: a mental model for operators
• Envoy monitoring checklist: the signals every production proxy needs
• Envoy monitoring maturity model: from survival to expert
• Envoy no healthy upstream: the 503 when a cluster has no host to route to
• Envoy membership_healthy dropping: reading the single most important cluster signal
• Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend