Envoy no healthy upstream: the 503 when a cluster has no host to route to

When Envoy returns a 503 with body “no healthy upstream” and access log flag UH, the cluster has zero hosts available for load balancing. The cluster exists, routes point to it, traffic is flowing, but the load balancer cannot select a host.

The response body is literal. The flag UH appears in the %RESPONSE_FLAGS% access log field and means “No healthy upstream hosts in upstream cluster in addition to 503 response code.” It has been present since the earliest Envoy releases (v1.0.0). It is distinct from UO (circuit breaker overflow), UF (upstream connection failure), and NR (no route).

The cause is rarely a bug in Envoy itself. It is one of: a genuine upstream outage, a health-check misconfiguration that fails healthy hosts, an EDS scale-down that removed all endpoints, or an outlier-detection cascade that ejected every host. The diagnostic path and the fix differ for each.

The critical subtlety is the panic threshold. The default is 50%. When the healthy host percentage in a priority drops below this, Envoy load-balances across ALL hosts including unhealthy ones rather than overloading the few that remain. The observed error rate can understate the damage: Envoy is deliberately sending traffic to hosts it knows are bad. A cluster that looks 10% unhealthy can be 90% unhealthy with panic mode masking the true severity.

What this means

Envoy tracks cluster membership in health states. Per the HealthStatus enum in the admin API (UNKNOWN, HEALTHY, UNHEALTHY, DRAINING, TIMEOUT, DEGRADED), the load balancer selects from healthy hosts; degraded hosts receive reduced traffic, and “excluded” is a separate flag referring to hosts excluded from panic-threshold calculations. The load balancer selects from healthy hosts. When membership_healthy reaches zero while membership_total is greater than zero, every routed request gets a local 503 with body “no healthy upstream” and flag UH.

The panic threshold changes this behavior. With the default 50% threshold:

• If 50% or more of hosts are healthy: traffic goes to healthy hosts only.
• If fewer than 50% are healthy: panic mode engages, and traffic goes to ALL hosts regardless of health status.
• If membership_healthy == 0 AND membership_total > 0: no host qualifies even in panic mode, and you get UH.

The lb_healthy_panic counter increments when panic mode is active. If you see UH errors, panic mode has already failed to help because every host is unhealthy. If you see elevated error rates without UH, check whether lb_healthy_panic is incrementing: Envoy may be routing to known-bad hosts, and the error rate understates the damage.

The error rate is a lagging and understated signal below the panic threshold. Alerting on UH alone catches the cliff, not the slide toward it.

flowchart TD
    A["503 UH on cluster"] --> B{"membership_total > 0?"}
    B -- No --> C["EDS scale-down
DNS failure
control plane disconnect"]
    B -- Yes --> D{"ejections_active
high vs total?"}
    D -- Yes --> E["Outlier-detection
cascade"]
    D -- No --> F{"connect_fail
increasing?"}
    F -- Yes --> G["Genuine outage
or network partition"]
    F -- No --> H{"Correlates with
config push?"}
    H -- Yes --> I["CDS update
transient"]
    H -- No --> J["Health-check
misconfiguration"]

Common causes

Quick checks

# Check membership state for the affected cluster
curl -s http://localhost:9901/stats | grep 'cluster.<name>.membership'

# Check for panic mode activity
curl -s http://localhost:9901/stats | grep 'lb_healthy_panic'

# Check upstream connection failures (genuine outage indicator)
curl -s http://localhost:9901/stats | grep 'cluster.<name>.upstream_cx_connect_fail'

# Check outlier detection ejections
curl -s http://localhost:9901/stats | grep 'cluster.<name>.outlier_detection'

# Check EDS update health
curl -s http://localhost:9901/stats | grep -E 'cluster.<name>.(update_success|update_failure|update_empty|update_rejected)'

# Per-host health detail from the cluster
curl -s http://localhost:9901/clusters?format=json | jq '.cluster_statuses[] | select(.name == "<name>") | .host_statuses[] | {address: .address, health_status: .health_status}'

# Check control plane connectivity
curl -s http://localhost:9901/stats | grep 'control_plane.connected_state'

# Verify Envoy is LIVE (not stuck initializing)
curl -s -o /dev/null -w "%{http_code}" http://localhost:9901/ready

In Istio sidecar mode, use port 15000 for the Envoy admin interface. The readiness check is on port 15021 (/healthz/ready).

How to diagnose it

1. Confirm the cluster has traffic flowing. A cluster with membership_healthy == 0 but zero routed traffic is not user-impacting. Check cluster.<name>.upstream_rq_total is non-zero. If traffic is not flowing, the unhealthy cluster is inert.

2. Check membership_total. This splits the diagnosis in half. If membership_total > 0, hosts are known but unhealthy: the problem is health state, not discovery. If membership_total == 0, Envoy has no hosts at all: the problem is discovery (EDS, DNS, or control plane).

3. If membership_total dropped, check EDS and the control plane. Look at update_success, update_failure, update_empty, and update_rejected for the cluster. A spike in update_empty means the control plane sent an update with no endpoints. Check control_plane.connected_state is 1. If the control plane is disconnected, Envoy is running stale config and the scale-down never reached it, or endpoints were removed upstream but Envoy still holds old ones.

4. If membership_total is stable but membership_healthy is 0, check outlier detection. Pull outlier_detection.ejections_active and ejections_overflow. If ejections_active is high relative to membership_total, hosts are being ejected by passive health detection. Look at the per-type counters (ejections_enforced_consecutive_5xx, ejections_enforced_success_rate, ejections_enforced_failure_percentage) to identify the trigger.

5. If ejections are low or zero, check active health checks. Look at the health check failure stats for the cluster. If health_check.failure is increasing but the upstream is actually serving traffic, the health check is misconfigured. Verify the path, port, expected status, and timeout against the actual upstream behavior by hitting the endpoint directly from the Envoy host.

6. If health checks are passing but membership_healthy is still 0, check for a cold start or CDS transient. On startup, Envoy excludes hosts from load-balancing calculations until their first health check completes. A cluster that just received a CDS update can briefly show membership_healthy == 0 even with healthy backends. If the failure resolves within seconds and correlates with a config push, this is the likely cause.

7. Distinguish genuine outage from network partition. If upstream_cx_connect_fail is increasing alongside the health check failures, hosts are not accepting connections. This is either a real outage or a network or ACL problem between Envoy and the upstream. Test connectivity directly from the Envoy host to one of the upstream endpoints using the same port.

Metrics and signals to monitor

Fixes

Genuine upstream outage

The fix is upstream, not Envoy. Verify the upstream processes are running, listening on the expected port, and not resource-exhausted. If connect_fail is high, the hosts are not accepting connections. Check FD exhaustion, SYN backlog, and process health on the upstream side. Envoy is correctly reporting the failure.

If the outage is partial, the panic threshold will keep traffic flowing to the remaining hosts plus the unhealthy ones. Do not disable the panic threshold during an active outage: that forces Envoy to return UH immediately instead of attempting degraded service.

Health-check misconfiguration

If hosts are serving traffic but Envoy reports them unhealthy, the health check is wrong. Common causes: wrong path (health check hits an endpoint that returns 404), wrong port, expected response code mismatch (check expects 200, endpoint returns 204), or timeout too short for the upstream’s health response.

Verify by hitting the health check endpoint directly from the Envoy host using the same path, port, and headers Envoy uses. The /clusters admin endpoint shows the configured health check and per-host health status. Fix the configuration via xDS or static config and push the update.

EDS scale-down

If membership_total dropped, the control plane removed endpoints. This may be intended (a deploy, a scale-down) or unintended (a bug in the service registry, a label selector change). Check the control plane side. In Istio, verify the ServiceEntry and DestinationRule. Check update_empty to confirm the control plane sent an empty endpoint set. If unintended, fix the control plane configuration. If intended, the cluster should recover as new endpoints register.

Outlier-detection cascade

If ejections_active is high and ejections_overflow is incrementing, outlier detection is ejecting hosts faster than they recover. The root cause is usually a shared dependency failure (all hosts slow because of a database) or overly sensitive thresholds.

Short-term: the panic threshold is already doing its job by routing to ejected hosts. Do not fight it. Long-term: tune the outlier detection thresholds (consecutive_5xx, success_rate_request_volume, failure_percentage_threshold) and verify max_ejection_percent is appropriate for your cluster size.

CDS update transient

Brief UH errors during a CDS update are a known issue. The new cluster’s load balancer is used before its host set is initialized. This resolves in seconds and does not indicate a persistent problem. Upstream, Envoy issue #13070 (a race between CDS update and thread-local cluster initialization that produced transient UH responses) was confirmed and fixed; recent Envoy versions include the mitigation. If the errors are sustained, look for a different cause.

STRICT_DNS resolution failure

For DNS-based clusters, if update_failure is increasing and hosts never resolve, DNS is the bottleneck. This is especially common at startup with many STRICT_DNS clusters. Envoy does not retry DNS aggressively enough on initial resolution. Workarounds include increasing the DNS resolver timeout or ensuring DNS is reachable before Envoy starts.

TLS/SDS health-check failure

If health checks use upstream TLS with a client certificate delivered via SDS, health checks fail when the SDS secret is not ready at cluster creation. This causes a window of 503s after restart proportional to healthy_threshold * no_traffic_interval. The fix is ordering: ensure SDS secrets are delivered before or simultaneously with CDS/EDS resources.

Prevention

• Alert on membership_healthy / membership_total ratio, not just UH. Catch the slide before the cliff. Page when the ratio drops below 50% with traffic flowing and the cluster has been live for more than 600 seconds.
• Monitor lb_healthy_panic. Any increment means the cluster is in worse shape than the error rate suggests, because traffic is going to known-unhealthy hosts.
• Track both ejections_active and ejections_overflow. Outlier-detection cascades are preventable with proper thresholds. ejections_overflow means the cluster is worse than detection can address.
• Alert on control_plane.connected_state == 0 sustained. Stale config is a delayed-action failure. Also track update_rejected: Envoy can be connected but silently NACKing every update.
• Track update_empty and update_rejected. Silent discovery failures and rejected configs are the most missed signals in Envoy operations.
• Test health check endpoints independently. A health check that does not match actual upstream behavior will fail healthy hosts under load. Verify path, port, expected status, and timeout against the real endpoint.
• Know your Envoy version. Behavior around health-check initialization and panic calculations has changed across versions. Hosts may be excluded from load-balancing calculations until their first health check completes.

How Netdata helps

Netdata collects Envoy admin stats at per-second resolution, which matters when membership changes happen in seconds:

• membership_healthy and membership_total per cluster at per-second granularity show whether membership_total fell (discovery) or membership_healthy fell (health state), and exactly when.
• outlier_detection.ejections_active and ejections_overflow on the same timeline as membership drops separate passive-detection cascades from active health-check failures.
• upstream_cx_connect_fail alongside membership changes distinguishes connection-refused (real outage) from health-check misconfiguration.
• control_plane.connected_state and cluster update_* counters reveal whether an EDS scale-down or disconnect preceded the failure.
• ML anomaly detection on the membership_healthy / membership_total ratio can flag gradual declines before UH triggers.

Related guides

• Envoy membership_healthy dropping: reading the single most important cluster signal
• Envoy outlier detection mass ejection: when passive health checks empty a cluster
• Envoy panic threshold: why traffic routes to unhealthy hosts at 50%
• Envoy upstream_cx_connect_fail: failed TCP connections to upstream hosts
• Envoy health checks vs outlier detection: two systems that eject hosts differently
• Envoy monitoring checklist: the signals every production proxy needs
• How Envoy actually works in production: a mental model for operators
• Envoy monitoring maturity model: from survival to expert
• Envoy upstream_rq_pending_overflow: the pending queue fills and 503s begin
• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy upstream_cx_active near max_connections: the pool filling up
• Envoy connection pool exhaustion: a slow upstream that fills the pool