Consul Connect "no healthy upstream": Envoy has no endpoints to route to

When an application in a Consul Connect mesh receives HTTP 503 with response_flags=UH in the Envoy access log, Envoy’s router rejected the request because it had zero eligible endpoints for the upstream cluster. The downstream service is fine, the network is fine, and the request never left the sidecar. Somewhere between the Envoy admin port, the local Consul agent, the Consul servers, and the upstream instances, the endpoint set has gone empty, all-unhealthy, or silently ejected.

“No healthy upstream” tells you Envoy could not route. It does not tell you whether the upstream is genuinely down, whether Consul delivered an empty endpoint list, whether an outlier detection circuit breaker ejected the only healthy host, whether a closed xDS stream is serving stale cluster configuration, or whether a certificate expiry is causing mTLS handshake failures and triggering ejection. In a healthy mesh this string should never appear in access logs.

What it means

The UH response flag fires when, for the matched route, the load balancer found no host it considered eligible. Eligibility combines three things:

• Cluster membership. The cluster must have at least one host; an empty or all-unhealthy cluster produces UH, not UF.
• Per-host health. Either EDS-reported health (Consul pushes per-endpoint health flags) or Envoy’s own active health checks, depending on the upstream configuration.
• Outlier detection state. Hosts ejected by passive health checks (consecutive 5xx, gateway failure, local origin failures) are temporarily removed from the load balancing set.

In a Consul Connect mesh, the upstream cluster is populated by Consul through xDS. The endpoint list, per-endpoint health flags, certificate material, and intention-derived RBAC filters all flow from Consul servers to Envoy over a gRPC xDS stream. Any break in that pipeline can produce a UH 503 even when the actual upstream service is healthy.

flowchart TD
    A["503 UH in Envoy access log"] --> B{"Envoy /clusters
membership_healthy?"}
    B -->|"0 of 0, empty"| C["xDS not delivering endpoints"]
    B -->|"0 of N, all unhealthy"| D["All upstream instances
failing checks"]
    B -->|"N>0 but ejected"| E["Outlier detection
ejected hosts"]
    C --> F{"gRPC xDS stream alive?"}
    F -->|"closed, code 14"| G["ACL token, cert,
or network to Consul"]
    F -->|"alive but stale"| H["Catalog empty or stale
for the upstream"]
    D --> I["Real upstream outage"]
    E --> J["Single-instance upstream
or passive check tuning"]

Common causes

Quick checks

Run these on the host serving the failing sidecar. All are read-only.

# Confirm UH is the actual response flag in the access log
grep "response_flags=UH" /var/log/envoy/access.log | tail -20

# Cluster membership from Envoy admin (port 19000 by default)
curl -s http://localhost:19000/clusters | grep -E "membership_(healthy|total)"

# 503 counters and outlier detection ejections
curl -s http://localhost:19000/stats | grep -E "upstream_rq_503|outlier_detection.ejections"

# xDS state: version_info and token presence
curl -s http://localhost:19000/config_dump | jq '.configs[] | select(.type_url|test("EDS|CDS")) | .'

# Leaf certificate expiry
curl -s http://localhost:19000/certs | jq '.certificates[].cert_chain[].expiration_time'

# Consul view of upstream health (replace <upstream> with the service name)
curl -s "http://localhost:8500/v1/health/service/<upstream>?passing=true" | jq length

# All instances regardless of health
curl -s "http://localhost:8500/v1/catalog/service/<upstream>" | jq length

# Consul agent xDS and Connect metrics
curl -s http://localhost:8500/v1/agent/metrics | grep -E "consul_xds|consul_proxy|consul_intention|consul_connect_ca"

If /v1/health/service/<upstream>?passing=true returns a non-zero count but Envoy /clusters shows membership_healthy=0, the problem is between Consul and Envoy. If Consul also shows zero, the problem is upstream of Consul.

How to diagnose it

Work outward from the sidecar. Each step localizes the failure or eliminates a layer.

1. Read the Envoy access log response flag first. UH means no healthy upstream. UF means upstream failure (connection refused, timeout, reset). UH with response_code=503 and no via_upstream is the canonical “no endpoints” signature. RBAC denials from intentions show up as connection resets or -- flags, not UH.

2. Diff Envoy’s cluster view against Consul’s catalog. For the failing upstream cluster name in /clusters, extract membership_total and membership_healthy. Then call /v1/health/service/<upstream> on the Consul agent. Three cases:

   • Consul shows healthy instances, Envoy does not. Stale xDS, malformed endpoints, or outlier ejection.
   • Consul shows instances but all critical. Real upstream outage.
   • Consul shows zero instances. Registration problem, anti-entropy failure, or wrong upstream name.

3. Check the xDS stream. The most common Consul-side failure is a stream that looks alive but is not delivering updates, or a stream that has closed. In Envoy logs, gRPC config stream closed: 14 indicates an unavailable stream, typically caused by a missing ACL token, a network break to the Consul gRPC port, or a TLS handshake failure. In config_dump, the x-consul-token field should be present. An empty or missing token almost always explains a closed stream.

4. Inspect outlier detection. If the cluster has one instance, even a small burst of 5xx from the upstream will cause Envoy to eject the only host. The default is 5 consecutive 5xx errors followed by a 30-second ejection window. Watch outlier_detection.ejections_consecutive_5xx and ejections_active in /stats. A single-instance upstream that flaps under load produces periodic UH windows that line up with ejection intervals.

5. Verify certificates. Open /certs and check days_until_expiration on the leaf. Connect leaf certificates have short TTLs (commonly 72 hours). If the Consul client cannot rotate the leaf into Envoy in time, mTLS handshakes to the upstream start failing, which surfaces as ejection-then-UH cycles. Cross-check consul_connect_ca metrics and /v1/agent/connect/ca/roots for an active, non-expired root.

6. Rule out intentions. Intentions are enforced at the destination service’s sidecar via Envoy RBAC filters. A denied intention produces a connection reset or 403-style response at the destination, not a UH at the source. However, repeated connection resets can trigger outlier detection ejection at the source sidecar, which then produces UH once every host has been ejected. Check consul_intention deny counters and grep Envoy logs for rbac denials to separate intention-driven ejections from genuine upstream failures.

7. Look for stale endpoints. delayed connect error: 113 (EHOSTUNREACH) or failed_eds_health flags in /clusters indicate Envoy is still trying to dial IPs that no longer exist. This usually means a mesh gateway sidecar is carrying a stale load balancer DNS name pushed as an endpoint IP, or an upstream was deregistered in Consul but the EDS push did not arrive. Restarting the sidecar clears the cache; the underlying issue is the missing push.

8. Check version-specific bugs. Several Consul releases have known “no healthy upstream” or related regressions:

   • 1.16.0 and 1.16.1: snapshot restore while servers host xDS streams can cause Envoy to receive incorrectly populated upstream endpoints in agent-less service mesh. Upgrade to 1.16.2 or later.
   • 1.15.0 and 1.15.1: a race condition breaks leaf certificate rotation after roughly 72 hours, which surfaces as mTLS failures and downstream ejection cascades. Upgrade to 1.15.2 or later.
   • 1.11.0: xDS v2 support was removed. Sidecars still pinned to v2 stop receiving configuration and serve stale config until upgraded.

Metrics and signals to monitor

Fixes

All upstream instances are critical

Genuine upstream outage. Fix the upstream service, not the mesh. Temporarily widen DeregisterCriticalServiceAfter if you are at risk of mass deregistration during recovery; instances vanishing from the catalog will make recovery harder. Verify the Consul agents hosting the upstream can still RPC to servers. If consul.client.rpc.failed is elevated on those agents, the catalog may simply be stale rather than the services actually down.

Envoy cluster is empty (membership_total = 0)

Consul is not delivering any endpoints. Confirm the upstream service name in the proxy configuration matches the registered service name exactly (Enterprise namespaces and admin partitions change this). Confirm anti-entropy is succeeding on the upstream’s agents. If the upstream is registered through an external mechanism (consul-k8s catalog sync, consul services register), check that integration separately.

Outlier detection ejecting the only host

For single-instance upstreams, Envoy’s default passive health check is too aggressive. Disable consecutive-5xx ejection in a service-defaults config entry for that service by setting PassiveHealthCheck.EnforcingConsecutive5xx = 0 (Consul maps this to Envoy’s enforcing_consecutive_5xx).

The tradeoff is losing passive circuit breaking for that upstream. With only one instance, the circuit breaker is mostly noise. For multi-instance upstreams, prefer raising the threshold (PassiveHealthCheck.Consecutive5xx; the legacy MaxFailures field also maps to it) rather than disabling enforcement entirely.

Stale xDS stream

If Envoy has a working gRPC stream to Consul but is not receiving updates, the cheapest immediate action is to restart the sidecar, which forces a fresh xDS subscription and full config redelivery. Capture config_dump before restarting so you can compare version_info before and after. If streams repeatedly go stale, look at Consul server xDS load. Large meshes have hit a “too many xDS streams open” condition that starves updates. A sidecar restart does not fix the underlying capacity problem.

gRPC config stream closed (code 14)

Connectivity or authentication failure between Envoy and Consul. Common causes: ACL token missing from the bootstrap config, expired sidecar token, TLS SAN mismatch on the Consul gRPC listener, or a firewall change blocking the gRPC port. Inspect x-consul-token in config_dump. If you recently enabled ACLs or rotated the sidecar token, the bootstrap config generated for the sidecar is stale and must be regenerated.

Leaf certificate expiry

If /certs shows days_until_expiration: 0 or close to it, the Consul client is not rotating the leaf into Envoy. Restart the sidecar to force an immediate CSR. If the problem recurs, the cause is upstream in the CA pipeline: Vault backend unreachable, CA root expired, or the Consul server cannot sign.

Malformed endpoint IPs

If /clusters shows endpoints resolving to hostnames (ELB DNS names) instead of IPs, or failed_eds_health flags persist after the upstream recovers, the mesh gateway sidecar is carrying stale EDS state. Restart the mesh gateway sidecar to clear it. The root cause is that the upstream changed shape (ELB rebalanced, instances replaced) and the EDS push did not reflect the new IPs.

Prevention

• Scrape Envoy admin, not just Envoy process health. Per-cluster membership_healthy divergence from Consul’s view is the earliest sign of an xDS problem.
• Tune outlier detection for small upstreams. The defaults assume multiple instances. Single-instance upstreams need enforcement disabled or a higher consecutive_5xx threshold.
• Track leaf certificate rotation, not just expiry. Expiry time is a countdown clock; renewal success rate tells you whether the rotation pipeline works.
• Alert on UH response flags directly. A small sustained rate of UH in Envoy access logs is always abnormal. Page on any non-zero rate that persists beyond a single outlier ejection window.
• Avoid known-bad Consul releases. Specifically avoid 1.16.0, 1.16.1, 1.15.0, and 1.15.1 for any deployment where Connect is in the critical path; release notes mark the 1.16.0/1.16.1 agent-less endpoint defect fixed in 1.16.2 and the 1.15 leaf-certificate race fixed in 1.15.2.
• Watch xDS stream count against sidecar count. They should match. Drift means sidecars are disconnecting silently and serving stale configuration.

How Netdata helps

• Per-second Envoy admin scraping surfaces membership_healthy divergence and upstream_rq_503 rate changes before they accumulate into user-visible failures. The 1s resolution matters because outlier ejection windows and CDS warm-up pauses are short.
• Correlating Envoy sidecar metrics with Consul agent metrics on the same host separates an Envoy-local problem (ejections, stale config) from a Consul-side problem (catalog empty, CA errors, xDS stream drain). Same dashboard, same second.
• ML anomaly detection on consul_xds stream count and drain rate flags xDS instability early. Stream churn almost always precedes a stale-config UH incident.
• Certificate expiry tracking across the mesh with both leaf and root CA signals gives lead time on rotation failures rather than a 3 a.m. mTLS outage.
• Composite dashboards joining Consul catalog health, Connect CA state, and Envoy cluster state turn the “is it Consul, is it Envoy, or is it the upstream” decision into a quick glance instead of a multi-tool investigation.

Related guides

• Consul Connect CA rotation failure: a root roll that never finished
• Consul Connect certificate expired: mTLS handshakes failing across the mesh
• Consul DeregisterCriticalServiceAfter: instances vanishing from the catalog
• Consul client rpc failed: agents alive but the catalog is going stale
• Consul catalog bloat: too many services and checks slowing everything down
• Consul DNS SERVFAIL: service discovery is broken for your applications