Envoy health checks vs outlier detection: two systems that eject hosts differently

Envoy has two independent mechanisms for removing unhealthy upstream hosts from load balancing. Active health checks send synthetic probes on a schedule. Outlier detection watches real request traffic and ejects hosts based on observed errors. They are not redundant, and they do not always agree.

A host can pass every health check and still be ejected by outlier detection. The two systems run on different threads, observe different signals, and act on different timelines. Operators who watch only membership_healthy or only outlier_detection.ejections_active miss half the picture.

What it is and why it matters

Active health checking is Envoy’s synthetic probe system. On a configured interval, Envoy opens a connection to each upstream host and sends a probe: HTTP, gRPC, L3/L4 (TCP), Redis, or Thrift. After unhealthy_threshold consecutive failures, the host is marked unhealthy and removed from load balancing. Health checks run on the main thread, not on the worker threads that carry production traffic.

Outlier detection is Envoy’s passive system. It watches the outcomes of real requests that workers already send to upstream hosts. A host is ejected when it trips one of the detection rules: consecutive 5xx, consecutive gateway failure (502, 503, 504), consecutive local origin failure, statistical success rate, or failure percentage. After ejection, the host is kept out of rotation for base_ejection_time (default 30 seconds), with re-admission on a timer.

Health checks answer “is this host dead?” Outlier detection answers “is this host degrading under real load?” A host that passes a synthetic probe but returns 503s for application traffic is invisible to health checks but caught by outlier detection. A host that has crashed and accepts no connections is caught immediately by health checks, while outlier detection only notices once real traffic has been sent and failed.

How it works

Both systems feed into the same load balancer decision, but through different paths.

flowchart LR
  MT[Main thread] -->|scheduled probe| HC[Active health check]
  HC -->|unhealthy_threshold fails| UNH[host unhealthy]
  WT[Worker threads] -->|request outcomes| OD[Outlier detection]
  OD -->|consecutive 5xx, success rate, failure pct| EJC[host ejected]
  UNH --> LB[Load balancer]
  EJC --> LB
  HC -.->|successful check unejects| EJC

Active health checks run on the main thread’s event loop. Each cluster with health checking configured gets a per-host probe on the configured interval. A host that fails unhealthy_threshold consecutive checks is marked unhealthy. A host that passes healthy_threshold consecutive checks is marked healthy again. The result is reflected in membership_healthy and health_check.healthy.

Outlier detection runs against real traffic on the worker threads. Each worker tracks per-host error statistics. When a host crosses a configured threshold, it is ejected for base_ejection_time. Ejection does not mark the host unhealthy in the health-check sense; it removes it from load balancing consideration. The stat outlier_detection.ejections_active is the count of currently ejected hosts.

The two timelines diverge in ways that matter operationally:

• Detection latency. Health check detection latency is bounded by the check interval. A 30-second interval means up to 30 seconds before a failed host is noticed. Outlier detection latency is bounded by real traffic: a host receiving 100 requests per second will trip consecutive_5xx: 5 in roughly 50 milliseconds.
• Coverage. Outlier detection only sees hosts that receive traffic. A host that has been ejected and receives no new requests cannot accumulate new errors, so it cannot be re-ejected by a different rule. A newly added host that has not yet received traffic is invisible to outlier detection until the load balancer sends it a request. Health checks cover every host in the cluster regardless of traffic.
• Recovery. A health-check-unhealthy host recovers when it passes healthy_threshold checks. An outlier-ejected host recovers when base_ejection_time expires and it is re-admitted to rotation. These are independent clocks.

Where they interact: the uneject behavior

When both systems are enabled, a successful active health check unejects a host that was ejected by outlier detection. This is controlled by successful_active_health_check_uneject_host, which defaults to true. The behavior also clears all outlier detection counters for that host.

The intent is reasonable: if the host is healthy enough to pass a probe, let it back in. In practice it produces two classes of confusion.

First, the timelines fight each other. Outlier detection ejects a host for 30 seconds based on real-traffic errors. A health check succeeds in the middle of that window and immediately unejects the host. The host starts receiving traffic again, accumulates more errors, and gets re-ejected. From the outside this looks like flapping: ejections_active rises and falls while membership_healthy stays flat.

Second, there was a confirmed bug where a failed health check could uneject an outlier-ejected host if the FAILED_ACTIVE_HC health flag had not yet been set; it was fixed in Envoy 1.31.0. On versions before 1.31.0 the operator-visible symptom is the same: a host that outlier detection ejected comes back early, before its ejection window expired.

The standard workaround, when you want outlier detection to manage its own timeline independently, is to set successful_active_health_check_uneject_host: false on the cluster’s outlier detection config. With that set, an outlier-ejected host stays ejected for its full base_ejection_time regardless of health check outcomes.

There is a related logging gotcha. When an active health check triggers an uneject, the outlier detection event log records the event with type: CONSECUTIVE_5XX even if the original ejection was for a different reason, such as success rate or gateway failure. The type field on uneject events is left at its zero enum value, which the event schema defines as CONSECUTIVE_5XX (outlier detection issue 35186, still open as of this review). If you are reading outlier detection event logs to diagnose why hosts are coming back, do not trust the type field on uneject entries.

Tradeoffs and when to use each

Use both. The question is not which to enable, but how to tune each and how to handle their interaction.

Health checks alone miss degraded hosts. A host that returns 200 for /healthz but 503 for 30 percent of real requests will stay in rotation indefinitely if you rely only on health checks. The host is not dead, just slow or error-prone under real load, and synthetic probes do not reproduce the condition.

Outlier detection alone misses dead hosts. A host that just crashed and accepts no connections will not be ejected until traffic is sent to it and fails. In a large cluster with many hosts, individual hosts receive traffic infrequently, so detection can be slow. Health checks probe every host on every interval regardless of traffic, so they catch total failures faster.

Outlier detection alone can mass-eject. Because outlier detection reacts to real traffic, a correlated failure such as network jitter or a shared dependency going slow can cause many hosts to cross thresholds simultaneously. As hosts are ejected, load concentrates on the survivors, which then also cross thresholds. The cascade ends when the healthy percentage drops below the panic threshold (default 50 percent), at which point Envoy routes to all hosts including unhealthy ones. Health checks do not cascade because they are independent per host.

The uneject default can mask outlier detection work. With successful_active_health_check_uneject_host: true, any successful health check resets the outlier detection state. If your health check is lenient, such as a simple TCP connect or a 200 from a lightweight endpoint, it will repeatedly rescue hosts that outlier detection correctly identified as degraded. If your intent is for outlier detection to be the authoritative signal for traffic-quality ejection, disable the uneject behavior.

Signals to watch in production

Monitor both systems. Neither is a substitute for the other.

The correlation that matters most: membership_healthy stable while ejections_active is nonzero. That is the signature of a host that passes probes but fails real traffic. If you see the inverse, ejections_active at zero while membership_healthy drops, you are looking at total host failures caught by health checks before traffic had a chance to fail.

Watch for flapping: ejections_enforced_total rate climbing while ejections_active oscillates. That pattern often points to the uneject interaction described above, where health checks and outlier detection fight over the same hosts.

How Netdata helps

• Per-second resolution on both signals. membership_healthy and outlier_detection.ejections_active are collected every second, so flapping between the two systems is visible as it happens rather than averaged away by a longer scrape interval.
• Correlation of ejection type with upstream errors. When ejections_enforced_consecutive_5xx rises alongside upstream_rq_5xx, the ejection is reacting to genuine application errors. When ejections_enforced_local_origin_success_rate rises alongside upstream_cx_connect_fail, the ejection is reacting to network-level failures. The per-type counters let you distinguish the two without reading logs.
• Anomaly flags on ejection rate. A sudden change in ejections_enforced_total rate, even when ejections_active is still small, is an early signal of correlated degradation. Netdata’s anomaly detection flags deviations from the rolling baseline without requiring fixed thresholds.
• Panic threshold visibility. When membership_healthy / membership_total crosses below 50 percent, the dashboard makes the transition visible alongside error rate spikes, so the increased error rate that follows is attributed to panic mode rather than a new failure.
• Cluster-level context. Ejection stats sit next to circuit breaker state, pending queue depth, and upstream latency, so you can tell whether an ejection is protecting the cluster or accelerating a cascade.

Related guides

• How Envoy actually works in production: a mental model for operators
• Envoy membership_healthy dropping: reading the single most important cluster signal
• Envoy monitoring checklist: the signals every production proxy needs
• Envoy monitoring maturity model: from survival to expert
• Envoy no healthy upstream: the 503 when a cluster has no host to route to
• Envoy outlier detection mass ejection: when passive health checks empty a cluster