Envoy outlier detection mass ejection: when passive health checks empty a cluster

membership_healthy is collapsing. outlier_detection.ejections_active is climbing toward membership_total, and ejections_overflow is ticking up because Envoy wanted to eject more hosts than max_ejection_percent allows. This is the outlier detection mass ejection pattern: one of the few Envoy failure modes that can amplify a partial upstream degradation into a cluster-wide outage.

The mechanism is subtle because outlier detection is doing exactly what it was configured to do. It is a passive health check: it ejects hosts based on the actual traffic they are serving, not on synthetic probes. When one host starts returning 5xx or its success rate drops, ejecting it is correct. The problem is what happens next. Load concentrates on the survivors, they get slower, their success rates drop, and they get ejected too. The cascade feeds itself.

This article covers how to recognize the cascade in real time, distinguish a genuine systemic upstream problem from too-sensitive thresholds, and stop the cascade without making things worse. It assumes you already understand the broad failure pattern catalogue in the Envoy operations hub.

What this means

Outlier detection is independent from active health checks. A host can pass active health checks (synthetic probes succeed) and still be ejected by outlier detection because real traffic is failing. The reverse is also true. When you see membership_healthy falling while membership_total stays constant, the most likely cause is outlier detection, not EDS-driven membership changes.

The cascade has a defined end state. Once the healthy host percentage drops below the panic threshold (default 50%), Envoy stops protecting the survivors and starts load balancing across all hosts, including ejected ones. This is intentional: it prevents the few remaining healthy hosts from being overloaded to death. The tradeoff is that traffic now flows to known-bad endpoints. If panic_threshold is set to 0%, Envoy instead returns 503 “no healthy upstream” when all hosts are ejected. Either way, users see degraded or failed responses, and the operator sees a system that looks broken even though Envoy is behaving as designed.

The critical question during the incident is not “why is Envoy ejecting hosts” but “are these ejections correct, or is outlier detection reacting to a self-inflicted load spike?”

flowchart TD
    A[Some hosts 5xx or
success_rate drops] --> B[Outlier detection
ejects a host]
    B --> C[Load concentrates
on remaining hosts]
    C --> D[Survivors slow down,
success_rate falls]
    D --> E{Threshold trips
again?}
    E -->|yes| B
    E -->|max_ejection_percent hit| F[ejections_overflow climbs
ejections_active plateaus]
    F --> G{healthy below
panic_threshold 50%?}
    G -->|yes| H[Panic mode: route to ALL hosts
including ejected]
    G -->|panic_threshold = 0| I[503 no healthy upstream]

Common causes

Quick checks

Run these against the Envoy admin interface. Default port is 9901; Istio sidecars use 15000 with /healthz/ready on 15021.

# Check current ejection volume per cluster
curl -s http://localhost:9901/stats | grep 'outlier_detection.ejections_active'

# Confirm the cap is being hit (Envoy wanted to eject more than max_ejection_percent allows)
curl -s http://localhost:9901/stats | grep 'ejections_overflow'

# Break down which detector is firing
curl -s http://localhost:9901/stats | grep -E 'ejections_enforced_(consecutive_5xx|success_rate|consecutive_gateway_failure|failure_percentage|local_origin_success_rate)'

# Compare detected vs enforced (detected fires even when capped by max_ejection_percent)
curl -s http://localhost:9901/stats | grep -E 'ejections_detected_|ejections_enforced_total'

# See the membership picture
curl -s http://localhost:9901/stats | grep -E 'membership_(healthy|degraded|excluded|total)'

# Per-host health status to spot correlated vs independent failures
curl -s http://localhost:9901/clusters?format=json | jq '.cluster_statuses[].host_statuses[].health_status'

# Latency on the survivors - rising P99 confirms load concentration
curl -s http://localhost:9901/stats/prometheus | grep 'envoy_cluster_upstream_rq_time'

# Confirm outlier detection config (consecutive_5xx, success_rate settings, max_ejection_percent)
curl -s http://localhost:9901/config_dump | jq '[.. | objects | select(has("outlier_detection")) | .outlier_detection]'

All of these are read-only. None of them touch the data plane.

How to diagnose it

1. Confirm it is actually a mass ejection, not an EDS membership change. If membership_total is stable while membership_healthy drops and ejections_active rises in lockstep, outlier detection is the cause. If membership_total is also dropping, the control plane is removing endpoints, which is a different incident.

2. Quantify the ejection rate. Sample ejections_active twice, ten seconds apart. A flat value means the cascade has saturated at the cap. A rising value means hosts are still being ejected. ejections_overflow increasing confirms Envoy wanted to eject more hosts than max_ejection_percent allows, which is a strong signal the cluster is in worse shape than the cap can express.

3. Identify the detector type. The ejections_enforced_* counters tell you which rule is firing. A dominant ejections_enforced_consecutive_5xx points to error-burst sensitivity. A dominant ejections_enforced_success_rate points to slow-host cascades where survivors degrade under concentrated load. ejections_enforced_failure_percentage fires on aggregate failure-rate thresholds.

4. Separate detected from enforced. ejections_detected_* counters increment whenever the detector trips, even if the ejection was suppressed by max_ejection_percent. A large gap between ejections_detected_total and ejections_enforced_total means many hosts are failing the detector but only some are being ejected. This is the cap working as intended, but it also means the upstream is systemically degraded.

5. Distinguish systemic upstream failure from threshold sensitivity. Pull per-host health from /clusters?format=json. If all hosts are failing independently with similar error patterns, the upstream has a real problem: shared database, AZ network, config deploy. If only a few hosts are failing and the cascade is driven by load concentration on survivors, the thresholds are too sensitive for the cluster size.

6. Verify observability is intact. In mesh environments with a custom stats matcher, the ejections_active counter may be filtered out. Without it, you cannot observe the cascade through stats even if the cap is still enforcing internally. Confirm ejections_active is exposed and non-zero.

7. Check whether panic mode is already active. If membership_healthy / membership_total is below 50% (the default panic threshold), Envoy is routing to all hosts including ejected ones. The elevated error rate you are seeing is panic-mode behavior, not an additional failure. Do not treat it as a new incident.

Metrics and signals to monitor

Fixes

Too-sensitive thresholds

The most common cause and the easiest to fix. consecutive_5xx (default 5) is aggressive for some workloads, and success_rate ejection on small clusters is fragile because the sample size is too small to distinguish noise from real degradation.

• Raise consecutive_5xx if deploys, GC pauses, or brief upstream restarts are triggering ejections.
• Raise success_rate_minimum_hosts (default 5) so the detector has enough samples. On a five-host cluster, success-rate ejection is at the statistical boundary.
• Review split_external_local_origin_errors. If local-origin errors (connect failures, resets) are lumped with upstream 5xx, a network blip can look like an upstream outage.

Changes to outlier detection config are applied via xDS. Verify with update_success and check update_rejected to confirm Envoy accepted the new config.

Genuine systemic upstream degradation

If all hosts are failing independently, disabling outlier detection is the wrong move. The ejections are correct; the upstream is broken. Address the root cause: shared dependency, AZ network issue, bad deploy.

During the incident, consider whether outlier detection is making things worse. If the upstream is slowly recovering and outlier detection keeps ejecting hosts that are nearly healthy again, the cascade extends the outage. Temporarily lowering max_ejection_percent can help: a tighter cap forces Envoy into panic mode sooner, which spreads load across all hosts including the recovering ones. This trades targeted ejection for broad degradation, which is preferable when no host is genuinely healthy.

max_ejection_percent misconfiguration

Default is 10%. Setting it to 100% lets outlier detection empty the cluster. Setting it too low forces panic mode on minor degradation. The right value depends on cluster size: on a three-host cluster, 10% rounds to zero ejections allowed, while on a fifty-host cluster, 10% allows five. Tune to your operational reality.

The always_eject_one_host option overrides max_ejection_percent to guarantee at least one bad host is ejected even in small clusters. Enable it deliberately; it can defeat the cap on tiny clusters.

Stats matcher hiding the ejection gauge

If a stats matcher filters out ejections_active, you lose visibility into the cascade. Confirm whether the cap is still enforcing internally by comparing ejections_enforced_total against cluster membership and the configured max_ejection_percent. Adjust the stats matcher to include ejections_active so the gauge is exported.

Premature unejection

successful_active_health_check_uneject_host defaults to true: a single successful active health check unejects a host that outlier detection ejected. During a marginal-host cascade, this causes flapping. The host is unejected, receives load, fails again, and is re-ejected. Setting it to false forces the host to serve its full ejection time before re-admission, which stabilizes flapping at the cost of slower recovery.

Prevention

• Do not treat outlier detection as a replacement for active health checks. Outlier detection only acts on hosts that are receiving traffic. Active health checks catch newly-failed hosts that have not yet been routed to. Use both.
• Tune thresholds to cluster size. success_rate ejection on a cluster below 5 hosts does not fire by default; at exactly 5 it is at the statistical boundary. Either raise success_rate_minimum_hosts or disable success-rate ejection for small clusters.
• Alert on ejections_overflow, not just ejections_active. ejections_overflow means the cluster is in worse shape than the cap can express. It is the leading indicator that a cascade is being artificially limited.
• Account for panic threshold behavior in your dashboards. When membership_healthy / membership_total crosses 50%, the meaning of “healthy” changes. Annotate this transition or your incident timeline will be misleading.
• Verify ejections_active is exposed in mesh environments. A stats matcher that filters it out makes the cascade invisible through stats.
• Set max_ejection_percent deliberately. The default is reasonable for most clusters, but small clusters and large clusters need different values. Document the reasoning.

How Netdata helps

• Per-second collection of outlier_detection.ejections_active and ejections_overflow makes the cascade visible in real time, including the moment ejections_overflow starts climbing. Alerting on sustained ejections_overflow gives warning that the cap is being hit before the cluster enters panic mode.
• Correlating ejections_active with upstream_rq_time P99 and membership_healthy in a single view confirms whether load concentration is driving the cascade. The ejections_enforced_* breakdown by detector type is collected separately, so you can see immediately whether consecutive_5xx or success_rate is the dominant cause.
• Per-cluster dashboards let you compare ejection behavior across clusters of different sizes, which is essential for tuning thresholds.

Related guides

• Envoy monitoring checklist: the signals every production proxy needs
• Envoy no healthy upstream: the 503 when a cluster has no host to route to