Envoy DNS resolution failures: STRICT_DNS clusters serving stale endpoints

A STRICT_DNS cluster stops picking up new endpoints, and endpoints you removed from DNS hours ago are still receiving traffic. Envoy’s own dashboard shows the cluster as healthy, error rates look flat, and the upstream service reports traffic to IPs that no longer exist. The first concrete evidence is often a wave of 502s or connection failures when a stale IP gets reassigned to an unrelated workload.

This failure mode is specific to STRICT_DNS, and to a lesser extent LOGICAL_DNS, clusters. In these cluster types, DNS is the endpoint-discovery mechanism: Envoy resolves the configured DNS target on a timer, treats each returned A/AAAA record as an upstream host, and drains connections to hosts that disappear from the result. When resolution fails or returns an unexpected result, no membership update happens, and Envoy keeps load-balancing across the previous endpoint set.

This is not the failure pattern you see with EDS clusters. If your clusters receive endpoints via xDS (Istio, a custom control plane, Consul), DNS resolution stats are irrelevant. The control plane pushes endpoints directly and Envoy never resolves DNS for those clusters. Confirm the cluster type before debugging DNS.

What this means

For a STRICT_DNS cluster, Envoy asynchronously resolves the DNS target on every dns_refresh_rate tick. The default is 5000ms. If respect_dns_ttl is enabled, the effective rate is the minimum of the record TTL and dns_refresh_rate. Each successful resolution produces the authoritative endpoint set: every returned IP is an explicit host, and any host no longer in the result gets its connection pools drained.

When a resolution attempt fails, Envoy does not remove endpoints. It keeps the previous membership intact and retries on the next refresh. This is correct for transient failures, but it means a sustained DNS problem silently freezes the cluster’s view. New endpoints never appear. Removed endpoints keep taking traffic until they fail at the TCP layer and get ejected by outlier detection or health checks.

A particularly nasty variant is the NOERROR-with-zero-records case. Per the documented STRICT_DNS contract, a successful resolution that returns zero hosts should empty the cluster. In practice, a long-standing bug (Envoy issue #20890) means the c-ares library tracks NOERROR-with-zero-records as a failed resolution internally, which blocks host removal until a subsequent resolution returns at least one record. If your DNS provider intermittently returns empty answer sections, you can get stuck serving stale endpoints with no update_failure signal at all.

flowchart TD
  A["dns_refresh_rate fires"] --> B{"DNS resolution result"}
  B -->|"success, N records"| C["membership updated, old hosts drained"]
  B -->|"timeout or server down"| D["update_failure++"]
  B -->|"NOERROR, 0 records"| E["c-ares treats as failed (issue #20890)"]
  D --> F["no membership change"]
  E --> F
  F --> G["stale endpoints keep traffic"]
  G --> H{"stale IP still live?"}
  H -->|"yes"| I["silent drift, no errors"]
  H -->|"no, recycled"| J["502 / UF / connect_fail"]

Common causes

Quick checks

All read-only. The admin port is 9901 by default, or 15000 in Istio sidecar mode.

# 1. Confirm cluster type. STRICT_DNS or LOGICAL_DNS, not EDS.
curl -s http://localhost:9901/config_dump | grep -i 'STRICT_DNS\|LOGICAL_DNS\|EDS'

# 2. Update success vs failure on the cluster
curl -s http://localhost:9901/stats | grep -E 'cluster\.<name>\.(update_success|update_failure|update_attempt|update_empty)'

# 3. c-ares resolver stats
curl -s http://localhost:9901/stats | grep 'dns\.cares\.'

# 4. Current membership. Is it changing at all?
curl -s http://localhost:9901/stats | grep -E 'cluster\.<name>\.membership_(total|healthy)'

# 5. Resolve the FQDN from the Envoy host using the same resolver Envoy uses
dig +short <fqdn> @<resolver>
dig +short AAAA <fqdn> @<resolver>

# 6. Compare Envoy's endpoint list against live DNS
curl -s http://localhost:9901/clusters?format=json | jq '.cluster_statuses[] | select(.name=="<name>") | .host_statuses[].address'

# 7. Warming state. STRICT_DNS clusters blocked on first resolution pin warming above zero.
curl -s http://localhost:9901/stats | grep -E 'cluster_manager\.warming_clusters|listener_manager\.total_listeners_warming'

If the cluster is EDS, stop here. The DNS path is not in play.

How to diagnose it

1. Confirm the cluster is DNS-based. If discovery_type is EDS, this article is not your failure. EDS clusters get endpoints from the control plane and never resolve DNS themselves.

2. Read the failure ratio. Compute update_failure / (update_success + update_failure). Anything sustained above 0.1 is concerning. Near 1.0 means DNS is effectively down for this cluster.

3. Check pending_resolutions. A sustained value above 100 indicates the resolver is backed up. Either the upstream DNS server is slow, or Envoy has too many concurrent resolutions in flight. The latter is common with hundreds or thousands of STRICT_DNS clusters resolving at once.

4. Reproduce from the Envoy host. Use dig against the exact resolver configured in typed_dns_resolver_config (or dns_resolvers on older configs). If dig also fails, the problem is upstream of Envoy. If dig succeeds but Envoy’s update_failure keeps climbing, the problem is in Envoy’s resolver.

5. Compare Envoy’s view to live DNS. Pull host addresses from /clusters?format=json and compare to dig +short. Any IP in Envoy’s list that is not in the current DNS result is a stale endpoint.

6. Look at the time series, not the snapshot. A single failed resolution is normal. What matters is whether update_success is incrementing at all over a window of several refresh intervals. If it is flat across five or more intervals while DNS is healthy from the host, the resolver is wedged.

7. Check the config API generation. Several DNS fields are deprecated in the cluster proto: dns_refresh_rate, dns_failure_refresh_rate, respect_dns_ttl, dns_lookup_family, dns_resolvers, dns_resolution_config, and typed_dns_resolver_config are all replaced by the cluster_type extension with DnsCluster. If your config mixes old and new fields, the old ones may be silently ignored when cluster_type is set.

8. Check for the NOERROR-zero-records trap. Run dig <fqdn> without +short and inspect the answer section. If you see NOERROR with zero A records intermittently, you are likely hitting issue #20890. c-ares treats this as failure and Envoy never empties the cluster.

Metrics and signals to monitor

Fixes

DNS server unreachable or slow

Fix the resolver. This is rarely an Envoy problem. Verify the configured resolver list in typed_dns_resolver_config resolves to IPs the Envoy host can actually reach. If you point Envoy at a cluster-internal DNS service, check that service is healthy and not rate-limiting Envoy. Increasing query_tries (default 4) or query_timeout_seconds (default 5) only masks the underlying issue.

NOERROR with zero records (issue #20890)

There is no clean runtime workaround. The cluster keeps its stale membership until a resolution returns at least one record. Options:

• Fix the DNS provider so it stops returning empty answer sections for live services.
• Add a low-TTL secondary record so resolutions are less likely to hit the empty-result window.
• If you control the upstream, prefer EDS over STRICT_DNS so endpoint lifecycle does not depend on DNS semantic edge cases.

c-ares channel wedged

A c-ares channel that has become unusable after a network restart or a run of timeouts will not recover without a process restart. The reinit_channel_on_timeout option (default false) reinitializes the c-ares channel when a query fails with ARES_ETIMEOUT, which addresses the most common wedge. If you are seeing this repeatedly, enable it via typed_dns_resolver_config. For older Envoy versions without this option, a rolling restart is the only recovery.

Startup DNS burst

A large number of STRICT_DNS clusters all resolve at startup. With hundreds or thousands of clusters, the configured DNS server can be overwhelmed and return mass timeouts for up to a minute. Envoy marks itself ready before resolutions land, producing UH / no-healthy-upstream responses. There is no built-in batching. Mitigations:

• Stagger cluster creation across the startup window if your control plane supports it.
• Point clusters at a DNS resolver with adequate query capacity (a local caching resolver, not the cluster DNS directly).
• Prefer EDS for high-cluster-count deployments. DNS is the wrong discovery mechanism at that scale.

EDNS cookie mismatch through a shared forwarder (Envoy 1.34+)

When c-ares queries a DNS forwarder that routes different domains to upstream servers with varying EDNS cookie support, c-ares tracks cookie state per server IP:port. If one domain returns a server cookie, c-ares expects all domains through that forwarder to return cookies, and drops legitimate responses that do not. The result is ARES_ETIMEOUT for affected domains only. The workaround is to provide an explicit typed_dns_resolver_config with c-ares config per cluster, which creates a dedicated DNS channel instead of sharing one global channel.

Prevention

• Alert on the update_failure ratio per STRICT_DNS cluster. A ratio above 0.1 sustained over several refresh intervals is the earliest signal that the cluster’s view has stopped updating.
• Prefer EDS where you have a control plane. If endpoints come from an orchestrator, let the control plane push them. STRICT_DNS is appropriate for external services, cloud load balancers, and legacy backends where you genuinely need DNS-based discovery.
• Enable reinit_channel_on_timeout if you have ever seen a c-ares wedge. The cost is negligible and it removes the restart-only recovery path for the most common wedge cause.
• Cap STRICT_DNS cluster counts. Hundreds of STRICT_DNS clusters pointing at cloud load balancers is a known startup-time footgun. Monitor dns.cares.pending_resolutions during boot.
• Track membership drift. Periodically compare cluster.<name>.membership_total against what DNS actually returns. Drift is the silent failure mode that produces no error until a stale IP is recycled.
• Tune dns_refresh_rate and respect_dns_ttl together. A very low refresh rate against a low-TTL record produces excessive resolver load. A very high refresh rate against a high-TTL record produces stale endpoints. Neither is obviously wrong from the stats alone.

How Netdata helps

• Per-second cluster.<name>.update_failure and update_success let you see the failure ratio form in real time rather than discovering it from a 30-second scrape.
• dns.cares.pending_resolutions as a gauge surfaces resolver backlog the moment it starts building, before it translates into stale membership.
• Correlating update_failure with membership_total, upstream_cx_connect_fail, and upstream_rq_502 in one view distinguishes “DNS is glitchy but membership is current” from “membership is frozen and traffic is hitting recycled IPs”.
• ML anomaly detection on the update_failure ratio catches slow drift where the ratio never crosses a hard threshold but climbs steadily over hours.
• cluster_manager.warming_clusters alongside control_plane.connected_state separates DNS-wedge-induced warming stalls from control-plane-disconnect-induced warming stalls.

Related guides

• Envoy server.state not LIVE: draining, initializing, and the /ready probe
• Envoy stuck initializing: waiting on xDS while the pod never turns ready
• Envoy hot restart races: draining, epoch churn, and dropped connections
• Envoy symptoms from the kernel: conntrack exhaustion and CFS throttling
• Envoy monitoring checklist: the signals every production proxy needs
• How Envoy actually works in production: a mental model for operators
• Envoy monitoring maturity model: from survival to expert
• Envoy no healthy upstream: the 503 when a cluster has no host to route to
• Envoy membership_healthy dropping: reading the single most important cluster signal
• Envoy outlier detection mass ejection: when passive health checks empty a cluster
• Envoy panic threshold: why traffic routes to unhealthy hosts at 50%
• Envoy upstream_cx_connect_fail: failed TCP connections to upstream hosts