Envoy panic threshold: why traffic routes to unhealthy hosts at 50%

You are looking at an Envoy cluster where membership_healthy has dropped below half of membership_total. The error rate has climbed. The first instinct is that something new has broken: the health checks are wrong, outlier detection is misfiring, or the upstream has a second fault. In most cases none of that is true. Envoy has entered panic mode, and the elevated error rate is a direct and expected consequence of the design.

Panic threshold is Envoy’s last-resort load balancing policy. When the percentage of healthy hosts in a cluster falls below a configurable threshold (default 50%), Envoy stops respecting health status and distributes traffic across every host in the cluster, including the ones it knows are unhealthy. The alternative is to concentrate all traffic on the few remaining healthy hosts, which typically overloads and kills them within seconds.

What it is and why it matters

Every Envoy cluster tracks host membership across several gauges: membership_healthy, membership_degraded, membership_excluded, and membership_total. Under normal operation the load balancer selects hosts only from the healthy set. When active health checks or outlier detection mark hosts as unhealthy, they are removed from the load balancing rotation but remain in the cluster membership.

The panic threshold is a percentage configured via the healthy_panic_threshold field in the cluster’s CommonLbConfig. The field type is Percent and defaults to 50 if unset. When the ratio of healthy hosts to total hosts drops below this percentage, the cluster enters panic mode.

In panic mode the load balancer ignores health status and routes across the full host set. The membership_healthy gauge continues to report the health-check-derived count, but it no longer reflects which hosts will actually receive traffic. Once panic mode is active, membership_healthy stops being a routing signal and becomes purely informational.

The rationale is overload protection. If a 10-host cluster loses 6 hosts, routing all traffic to the 4 survivors saturates them within seconds. Their latency climbs, their health checks start timing out, and they fail too. The cluster goes from degraded to dead. Panic mode spreads the load across all 10 hosts, accepting a known error rate from the unhealthy ones to keep the survivors alive.

How it works

Envoy evaluates the panic condition on every load balancing decision. The check is straightforward: if membership_healthy / membership_total < healthy_panic_threshold, the cluster is in panic mode for that decision.

flowchart TD
    A["LB decision for cluster"] --> B{"healthy / total >= threshold?"}
    B -- Yes --> C["Route to healthy hosts only"]
    B -- No --> D["PANIC: route to ALL hosts"]
    C --> E["Health check or outlier event changes membership"]
    E --> A
    D --> F["membership_healthy no longer reflects routing"]
    D --> G["lb_healthy_panic increments; expect elevated 5xx"]

The threshold is evaluated per priority level. Most clusters use only the default priority, so the per-priority distinction rarely matters. If you configure priority failover (primary and failover localities), each priority level is checked independently.

When panic mode is active:

• Load balancing ignores membership_healthy and membership_degraded entirely.
• All hosts counted in membership_total become eligible for traffic.
• The lb_healthy_panic counter for the cluster increments with each LB decision made during panic (flat when not in panic).

• Error rates increase because some requests land on hosts that are actively failing health checks or have been ejected by outlier detection.

# Panic counter: increasing means panic mode is active, flat means it is not
curl -s http://localhost:9901/stats | grep 'lb_healthy_panic'

# Check the membership ratio that drives the threshold calculation
curl -s http://localhost:9901/stats | grep 'cluster.my_cluster.membership'

The threshold comparison uses healthy hosts in the numerator. In recent versions, Envoy documentation refers to “available hosts” (healthy or degraded) rather than strictly “healthy hosts.”

In most deployments that use only healthy and unhealthy states, the distinction does not change behavior. If you rely on the degraded state, verify how your version treats it.

Panic mode can be disabled by setting healthy_panic_threshold to 0. Envoy then routes only to healthy hosts and returns 503 if there are none. Istio and Contour both default to 0%.

The ignore_new_hosts_until_first_hc option, available since v1.11.0, prevents newly added hosts from counting in the panic threshold denominator until they complete their first health check. This addresses a common spurious-panic scenario discussed below. Excluded endpoints do not participate in load balancing under normal conditions but may receive traffic if panic mode triggers.

Where it shows up in production

Panic mode is not a rare edge case. It appears in several common operational scenarios.

Outlier detection mass ejection. The most common trigger in production. Outlier detection ejects hosts based on real-traffic error rates. If a correlated failure affects many hosts (network degradation, shared dependency slowness, an AZ-level issue), outlier detection ejects them one by one. As ejections accumulate, the healthy percentage drops. When it crosses the threshold, panic mode activates and traffic flows back to the hosts that were just ejected. The outlier_detection.ejections_active stat will be high, lb_healthy_panic will be increasing, and the error rate you see is Envoy routing to hosts that outlier detection correctly identified as degraded.

Rolling deployments. During a rolling update, endpoints are removed and re-added. If the cluster has few hosts (3 or 4), removing one or two during rollout can temporarily drop the healthy ratio below 50%. Envoy enters panic mode briefly, routes to the draining endpoints, and exits when the new endpoints pass their first health check. This produces a short burst of errors during every deploy.

Small clusters. Clusters with 2 or 3 hosts are structurally vulnerable to panic. A 2-host cluster with 1 unhealthy host sits at exactly 50%, which triggers panic. This is common in DNS-resolved clusters with few upstreams, low-traffic internal services, and development environments. For clusters this small, the overload-protection rationale is weak: concentrating traffic on 1 surviving host is unlikely to kill it, while routing to the known-bad host guarantees errors.

Scaling events with health check delay. When new hosts are added via EDS or DNS resolution, they start unhealthy until their first active health check completes. If enough new hosts are added at once, the ratio of healthy hosts drops temporarily because the new hosts count in membership_total but not yet in membership_healthy. This triggers a spurious panic that resolves as soon as health checks finish. The ignore_new_hosts_until_first_hc option exists specifically to prevent this. Without it, every significant scale-up event can trip a false panic alert.

Tradeoffs and when to tune

The default 50% threshold trades per-request correctness for cluster survival. Tune it intentionally rather than discovering the behavior during an incident.

Keep 50% when cluster survival matters more than per-request success. The right choice for most internal services and traffic-intensive endpoints where partial availability beats total failure.

Disable panic mode (0%) when you would rather fail fast with a 503 than route to known-bad hosts. Appropriate for services where sending traffic to an unhealthy endpoint causes data corruption, non-idempotent side effects, or security problems.

Lower the threshold (for example, 25%) when you want some overload protection but are willing to concentrate traffic on fewer healthy hosts. Useful for clusters large enough that concentrating load on a quarter of the fleet is survivable.

For small clusters (fewer than 5 hosts), consider disabling panic mode entirely. A 3-host cluster in panic mode is routing to hosts it knows are bad, and the survival benefit is marginal.

Whatever you choose, document it. The most common incident pattern is a team that does not know their threshold is set to 50%, sees errors during an upstream partial failure, and spends an hour debugging Envoy when the behavior is correct and the real problem is upstream.

Signals to watch

The key correlation is lb_healthy_panic rate increasing alongside an increase in upstream_rq_503. If those two events are simultaneous, the error rate is expected panic-mode behavior. If errors increase without panic mode activating, investigate a separate issue. If lb_healthy_panic is increasing but you did not expect upstream degradation, check outlier_detection.ejections_active first: passive ejection is the most common path into panic mode.

How Netdata helps

Per-second metric collection makes the panic threshold transition visible as it happens, which matters because the window between threshold crossing and operator response is when bad decisions get made.

• membership_healthy and membership_total are collected every second, so you can see the exact moment the ratio crosses 50% and correlate it with the lb_healthy_panic rate change on a single timeline.
• ML anomaly detection flags the membership ratio deviation before the threshold is crossed, giving lead time to investigate outlier detection settings or upstream health before panic mode engages.
• Correlating lb_healthy_panic rate with upstream_rq_503 and outlier_detection.ejections_active on one chart distinguishes expected panic-mode errors from a genuine new fault.
• Per-cluster dashboards let you compare panic behavior across clusters of different sizes, useful for identifying small clusters that are structurally vulnerable to spurious panics during deploys and scaling events.

Related guides

• How Envoy actually works in production: a mental model for operators
• Envoy membership_healthy dropping: reading the single most important cluster signal
• Envoy monitoring checklist: the signals every production proxy needs
• Envoy monitoring maturity model: from survival to expert
• Envoy no healthy upstream: the 503 when a cluster has no host to route to
• Envoy outlier detection mass ejection: when passive health checks empty a cluster