Envoy membership_healthy dropping: reading the single most important cluster signal

If you watch only one availability metric per Envoy cluster, watch the ratio of cluster.<name>.membership_healthy to cluster.<name>.membership_total. When the ratio collapses, the remaining healthy hosts take proportionally more load, and the cluster is one or two failures away from panic mode. Upstream 5xx rate, latency, circuit breaker state, and retries are all downstream of host availability.

The hard part is reading the gauge correctly, not collecting it. A drop can mean the upstream is genuinely broken, the control plane removed endpoints, or outlier detection ejected hosts that are still passing active health checks. Each case has a different response.

What the signal actually means

cluster.<name>.membership_healthy is a gauge. It reports the current count of hosts in the cluster that Envoy considers available for load balancing, after both active health checking and outlier detection have applied their logic. It is inclusive of both subsystems, which is the first thing operators get wrong.

The full membership family for a cluster:

What makes membership_healthy the right anchor is that it already incorporates the two subsystems you would otherwise have to reason about separately. A host that passes active health checks but has been ejected by outlier detection does not count as healthy. A host that is healthy by both subsystems counts as healthy. You do not have to do that math yourself.

The number to compute and alert on is the ratio membership_healthy / membership_total, not the absolute count. A cluster with 9 of 10 hosts healthy (0.90) is in far better shape than one with 2 of 3 (0.67), even though both have only one unhealthy host. The ratio tells you how concentrated the remaining load is about to become.

How to read it correctly

• Ratio below 0.5 is critical. Below this point the cluster is one failed host away from panic mode, and the remaining healthy hosts are absorbing at least double their normal share.
• Any non-zero membership_degraded warrants investigation. Degraded is not healthy. It is Envoy saying “this host is partially broken, route it less traffic.” A cluster with zero unhealthy hosts but several degraded hosts is trending the wrong way.
• membership_healthy == 0 with membership_total > 0 is total upstream failure for that cluster, but only if traffic is actually flowing to it. Gate the alert: require upstream_rq_total > 0 to rule out idle clusters, and server.live == 1 to rule out draining. Check Envoy uptime via the /server_info admin endpoint to rule out cold start and warming. Sustain the check across at least two health-check intervals to avoid paging on flaps.
• Clusters with zero routed traffic can sit unhealthy with no user impact. This is the false positive you must gate out. The traffic floor (upstream_rq_total > 0) is not optional.

Why a drop is not always a failure

This is the section that saves you from paging at 3 a.m. on a deployment.

In EDS-based deployments (Istio service mesh, any xDS control plane that pushes endpoints), membership_total is not a constant. It shifts as the control plane adds and removes endpoints. A scale-down event reduces membership_total. A rolling deploy churns it. If membership_healthy drops in lockstep with membership_total and the ratio stays roughly constant, the cluster is not getting less healthy. It is getting smaller, by design.

The failure case is the opposite: membership_total is flat or growing while membership_healthy drops. That is hosts failing in place. The ratio is the truth, and you confirm it by watching whether the two gauges move together or apart.

flowchart TD
    A["membership_healthy drops"] --> B{"membership_total also dropping?"}
    B -- "yes, in step" --> C["Likely control-plane churn
check update_success / update_empty"]
    B -- "no, flat or growing" --> D["Hosts failing in place
this is the real signal"]
    C --> E{"Ratio stable?"}
    E -- "yes" --> F["Scale event or deploy. Not an outage."]
    E -- "no" --> D
    D --> G["Correlate: upstream_cx_connect_fail,
health_check.failure,
outlier_detection.ejections_active"]

Two adjacent stats help you tell control-plane churn from real loss:

• cluster.<name>.update_success and cluster.<name>.update_empty should be incrementing if the membership change came from EDS. update_empty in particular means the control plane pushed an update with zero endpoints, which is expected during scale-to-zero or a deploy lull.
• cluster.<name>.update_failure means the update failed or Envoy refused the new config. Envoy keeps the old config and the membership numbers do not move. This is a control-plane problem, not a host-health problem.

What to correlate when the drop is real

Once you have confirmed the ratio is genuinely falling (hosts failing in place, not being removed by the control plane), three signals tell you why.

upstream_cx_connect_fail. This counter fires when the TCP connection attempt to an upstream host fails: process crash, port not listening, FD exhaustion, listen backlog overflow, firewall blocking. A rising connect_fail rate that tracks the falling membership_healthy is a host-availability problem, not a host-performance problem. Connect failures also feed outlier detection, so a host failing to connect will often be ejected on top of failing its health check.

health_check.failure and the health-check family. Active health checks run on the main thread, not worker threads. A rising failure rate here means the host is accepting the connection but not responding correctly to the probe: slow app, wrong response, timeout. The health-check interval bounds your detection latency (see the next section), so the rate of failure is the leading indicator and the count of failed hosts is the lagging one.

outlier_detection.ejections_active. Outlier detection is passive, based on real traffic, and operates independently from active health checks. A host can be passing its health checks and still be ejected by outlier detection because real requests are failing on it. membership_healthy reflects both. If you see the gauge dropping but health_check.failure is flat, check ejections_active and the per-type counters (ejections_enforced_consecutive_5xx, ejections_enforced_success_rate, ejections_enforced_failure_percentage) to find out which outlier rule is firing.

The other direction matters too. A mass-ejection event where ejections_active approaches membership_total is often a correlated infrastructure failure (network, AZ, shared dependency) rather than independent host failures. Hosts get ejected, load concentrates on the survivors, the survivors slow down, they get ejected, and the cluster falls through the panic threshold.

Detection latency is bounded by the health-check interval

Active health checks detect a failed host only as fast as the configured interval allows. A 30-second health-check interval means up to 30 seconds between a host going dark and Envoy marking it unhealthy. During that window, requests are still being routed to the dead host and failing.

The interval is the dominant term. If your SLO for upstream failure detection is faster than your health-check interval, the interval is the bug. Tighten the interval, or add passive outlier detection to catch failures that active checks would otherwise wait out.

Two practical implications:

• Sustain the alert across at least two health-check intervals. A single failed check is a flap. Two in a row is a host.
• Outlier detection does not save you for cold hosts. Outlier detection only acts on hosts that are receiving traffic. A host that just failed and has not received a request yet will not be ejected by outlier detection. Active health checks are how you detect a newly-failed host that has no traffic to fail. The two subsystems are complementary, not redundant.

Panic threshold changes the meaning of the ratio

When the healthy percentage drops below the panic threshold (default 50%, configurable per cluster), Envoy stops honoring health status and load-balances across all hosts, including unhealthy ones. This is intentional: it prevents the few remaining healthy hosts from being overloaded to death.

It also changes how you read the gauge. Once panic mode is active, membership_healthy is no longer the count of hosts receiving traffic. Envoy is routing to ejected and unhealthy hosts because the alternative is worse. Increased error rates during panic mode are the designed behavior, not a second failure layered on top of the first.

If you alert on upstream_rq_5xx and the cluster is below panic threshold, the 5xx rate is expected to be bad. The actionable signal is the ratio climbing back above the threshold, not the error rate. Do not treat panic-mode error rates as a new incident.

A stats-matcher gotcha

The HTTP health check filter, when run in “computed from upstream cluster health” mode, does not probe a backend. It reads the membership stats and returns 200 or 503 based on them. If you have configured a stats_matcher with reject_all: true or an exclusion list that drops the membership stats, the filter may have nothing to read and could return 503 regardless of actual cluster health.

The fix is to allowlist the membership stats explicitly in your stats_matcher:

stats_config:
  stats_matcher:
    inclusion_list:
      patterns:
      - suffix: membership_healthy
      - suffix: membership_degraded
      - suffix: membership_total
      - suffix: live

This is an easy mistake to make when pruning stats for cardinality reasons. The symptom is a health check endpoint that is permanently unhappy even though /clusters?format=json shows healthy hosts.

Signals to watch in production

How Netdata helps

• Per-second collection of the membership_healthy, membership_degraded, membership_excluded, and membership_total gauges means you see the ratio move as it happens. Detection latency in your monitoring should not be the bottleneck when Envoy’s own health-check interval already sets a floor.
• ML anomaly detection on the ratio catches gradual drift (one host every few minutes) that fixed thresholds miss, even when the absolute number is still above a static threshold.
• Correlating membership_healthy with upstream_cx_connect_fail, health_check.failure, and outlier_detection.ejections_active on a single timeline lets you distinguish host-level failure from control-plane churn from passive ejection in seconds.
• The same correlation works for ruling out false positives: membership_total moving in lockstep with membership_healthy, plus update_empty incrementing, shows up as a deploy or scale event rather than an outage.
• Alerting on the ratio with a traffic floor (upstream_rq_total > 0) and a sustained-duration window (at least two health-check intervals) keeps you from paging on cold-start, warming, or idle clusters.

Related guides

• Envoy monitoring checklist: the signals every production proxy needs
• Envoy no healthy upstream: the 503 when a cluster has no host to route to