Envoy monitoring maturity model: from survival to expert

Envoy exposes thousands of stats and dozens of admin endpoints. The trap is not lack of data; it is lack of the right data at the right depth for your operational maturity. A team monitoring /ready and membership_healthy will survive most outages but will be blind to the failure modes that cause multi-hour incidents: silent xDS NACKs, retry amplification, connection pool exhaustion, CFS throttling.

Four levels, each additive. Level 1 catches process death and total upstream loss. Level 2 adds the signals that explain why errors happen. Level 3 adds leading indicators that warn before saturation. Level 4 adds per-host drift, kernel interaction, and certificate runway. You keep every previous level’s coverage and add the next class.

Use this as a self-audit. Most production Envoy deployments plateau at Level 2 because that is where generic dashboards stop. The gap between Level 2 and Level 3 is where the expensive incidents live.

flowchart TD
    L1["Level 1 - Survival
Is it up? Are backends up?"]
    L2["Level 2 - Operational
+ why, how slow, xDS, FDs"]
    L3["Level 3 - Mature
+ workers, pools, overload, NACKs"]
    L4["Level 4 - Expert
+ per-host, retries, kernel, certs"]
    L1 --> L2 --> L3 --> L4

How to use this model

The levels describe monitoring coverage, not team skill. A Level 4 team still needs Level 1 alerts. The model is about which classes of failure mode you can detect and how early.

• Level 1 (Survival). Is Envoy alive, are upstreams healthy, are requests erroring, is memory safe. Catches outages after they are user-visible.
• Level 2 (Operational). Why errors happen (response flags), how slow things are (latency histograms), whether config is live (xDS), whether the process is near hard cliffs (file descriptors). Catches common incident shapes and shortens triage.
• Level 3 (Mature). Leading indicators before saturation: worker blocking, pool depth, overload actions, silent config rejection, cardinality. Gives minutes of warning instead of a page after users fail.
• Level 4 (Expert). Fleet-level drift and kernel interaction: per-host stats, retry effectiveness, conntrack, CFS throttling, certificate runway, config-version consistency. Catches failures that look like Envoy bugs but are environmental.

Stat names below are the actual Envoy stat or admin fields. Admin port is 9901 in standalone Envoy, 15000 in Istio sidecar. Health endpoint is /ready (or /healthz/ready on port 15021 in Istio).

Level 1: Survival

Four signals. If you monitor nothing else, monitor these. They catch the four ways an Envoy deployment dies in a way users notice immediately.

What Level 1 misses. A 503 from a tripped circuit breaker and a 503 forwarded from an upstream app look identical here. membership_healthy == 0 tells you the cluster is dead but not whether health checks failed, outlier detection ejected everything, or the control plane never delivered endpoints. Memory tells you OOM risk but not whether the overload manager is already shedding load. Level 1 is reactive: by the time these fire, users are already failing.

Survival checklist.

• Page on /ready returning non-200 outside planned restarts. A LIVE process that cannot accept connections is the most basic failure.
• Page on membership_healthy == 0 for any cluster with traffic. Combine with upstream_rq_total > 0 and uptime > 600s to avoid cold-start and idle-cluster false positives.
• Ticket on 5xx / total > 0.1% sustained. SLO-dependent, but this ratio is the universal error signal.
• Ticket on memory_allocated approaching 80% of container limit. Below this you have runway; above it you are gambling on the overload manager being configured.

Level 2: Operational

Everything in Level 1, plus the signals that turn “something is wrong” into “here is why.” This is where a production team should be.

The Level 1 to Level 2 gap. This is the response-flag gap. Without %RESPONSE_FLAGS%, every 503 is a mystery. UO means a circuit breaker tripped, so fix the upstream rather than raising Envoy limits. NR after an xDS push means a bad config deployment. UF means upstream connection failure. UT means upstream timeout. These flags are access-log only; they are not exposed as aggregate Prometheus stats, so you need a log pipeline or a Lua/Wasm counter to alert on them.

Operational checklist.

• Alert on connected_state == 0 sustained more than 5 minutes. Stale config is a time bomb; it works until a scaling event or cert rotation reveals it.
• Track retry ratio (upstream_rq_retry / upstream_rq_total) as a dashboard metric. Persistent growth above your baseline signals retry amplification; high sustained ratios indicate a retry storm accelerating the failure it is trying to mask.
• Alert on any sustained NR flag. These are configuration errors in production.
• Monitor FD utilization, not just connection count. Each proxied connection is roughly two FDs, and hot restart briefly doubles total FD usage.
• Monitor ssl.fail_verify_error on every mTLS-enabled cluster. In service mesh, silent SDS failure means certs stop rotating and TLS breaks catastrophically on expiry.

Stats scraping gotcha. Scraping /stats on a high-traffic proxy at 10-15 second intervals can cause memory pressure and latency spikes from serializing high-cardinality output. Use /stats/prometheus format, scrape at 30 seconds or slower, and add ?usedonly to reduce output volume.

Level 3: Mature

Everything in Level 2, plus leading indicators and internal-state signals. The defining trait of Level 3 is that it gives you warning before users fail. Most teams that suffer multi-hour Envoy incidents are missing signals in this band.

The Level 2 to Level 3 gap. Two signals dominate this gap. First, upstream_rq_pending_active: most teams alert on 503s, but by the time 503s appear, the pending queue is already full. Alerting on pending-queue growth gives minutes of warning. Second, update_rejected and listener_create_failure: Envoy can be connected to the control plane (connected_state == 1) but silently NACKing every update. The operator sees “deployment completed” on the control plane side, while Envoy kept the old config. Without monitoring rejection rate, this is invisible until someone wonders why a fix did not work.

Mature checklist.

• Alert on pending_active growth, not just overflow. The queue is the leading indicator; overflow is the cliff.
• Alert on any watchdog_miss increment. Workers should never block; any nonzero value is abnormal and often points to CFS throttling or an expensive filter.
• Track total stat count as a metric. Silent stat drops create monitoring blind spots with no error surfaced.
• Alert on any update_rejected or listener_create_failure. These should be zero; any nonzero value is a control plane or config bug.
• Alert on any overload action going active. If stop_accepting_requests or stop_accepting_connections fires, Envoy is refusing traffic to survive.
• Enable track_remaining: true on circuit breakers. remaining_cx and remaining_pending give headroom visibility before the breaker trips.

Outlier detection versus health checks. These are complementary, not redundant. Health checks detect dead hosts; outlier detection detects degraded hosts. A host can pass health checks but be ejected by outlier detection based on real traffic. Monitor both membership_healthy and ejections_active. When the panic threshold (default 50%) is crossed, Envoy routes to all hosts including unhealthy ones. The increased error rate that follows is expected behavior, not a new failure.

Level 4: Expert

Everything in Level 3, plus fleet-level drift, kernel interaction, and the signals that usually get added after the third or fourth major incident. These catch failures that look like Envoy bugs but are environmental.

The Level 3 to Level 4 gap. This gap is about leaving Envoy’s own stats and correlating with the environment. Conntrack exhaustion on the host causes intermittent upstream_cx_connect_fail with no clear cause in Envoy’s stats. CFS throttling causes watchdog_miss and latency spikes even when CPU usage looks low in cgroup aggregates; the kernel is throttling the quota, not Envoy being computationally overloaded. Per-host stats catch the one host returning 100% errors inside a cluster that looks fine at 1% aggregate error rate.

Expert checklist.

• Monitor container_cpu_cfs_throttled_periods_total on every Envoy container. CFS throttling is the most common misdiagnosed source of tail latency in Kubernetes sidecars.
• Track days_until_first_cert_expiring as a canary for SDS health. In automated rotation environments, a falling runway means the rotation pipeline is broken.
• Compute config-version consistency across the fleet. After every push, verify all instances report the new version_info; instances stuck on old config are a silent inconsistency window.
• Drill into /clusters during incidents. Aggregate cluster stats hide per-host problems; per-host health_flags and rq_success reveal the bad actor.
• Track retry success ratio alongside retry rate. High retry rate with low success ratio means retries are adding load without helping.
• Monitor conntrack table utilization on high-connection-count nodes. Table exhaustion surfaces as silent, intermittent connection failures that look like upstream flakiness.

Common mistakes across all levels. Treating circuit breaker trips as an Envoy problem to fix by raising limits (fix the upstream instead). Alarming on downstream_rq_5xx without separating Envoy-generated from upstream-forwarded errors. Running without an overload manager configured: Envoy goes straight from fine to OOM with no graceful degradation. Assuming health checks and outlier detection are redundant. The maturity model is as much about avoiding these habits as it is about adding signals.

How Netdata helps

Netdata’s per-second collection and anomaly detection shorten the diagnosis path at each level by making correlations immediate rather than retrospective.

• Survival signals with per-second granularity. server.state, membership_healthy, and 5xx rate update every second, so a drain transition or mass ejection is visible as it happens rather than after a 30-second scrape lag.
• Correlating response flags with error rates and circuit breaker state. Netdata collects cluster, listener, and HTTP stats together, so a 503 spike can be cross-referenced against pending_overflow, cx_open, and ejections_active in the same view without joining separate dashboards.
• Leading indicators before saturation. upstream_rq_pending_active, upstream_cx_active, and overload action gauges surface as anomaly candidates before they cross hard thresholds.
• Worker and kernel correlation. watchdog_miss alongside per-core CPU and cgroup CFS throttling metrics (when running the Netdata cgroup collector) distinguishes Envoy-internal blocking from kernel-imposed throttling.
• Config-state continuity. control_plane.connected_state, update_rejected, and listener_create_failure tracked together catch both the disconnected and the silently-NACKing variants of stale config.
• Certificate runway as a first-class signal. SDS-managed cert expiry is monitored alongside TLS handshake errors, so a failing rotation pipeline shows up as a falling runway before handshakes start failing.

Related guides

• Envoy monitoring checklist: the signals every production proxy needs
• How Envoy actually works in production: a mental model for operators
• Envoy no healthy upstream: the 503 when a cluster has no host to route to
• Envoy membership_healthy dropping: reading the single most important cluster signal
• Envoy outlier detection mass ejection: when passive health checks empty a cluster
• Envoy panic threshold: why traffic routes to unhealthy hosts at 50%
• Envoy upstream_cx_connect_fail: failed TCP connections to upstream hosts
• Envoy health checks vs outlier detection: two systems that eject hosts differently
• Envoy upstream_rq_pending_overflow: the pending queue fills and 503s begin
• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy upstream_cx_active near max_connections: the pool filling up
• Envoy connection pool exhaustion: a slow upstream that fills the pool