Envoy monitoring checklist: the signals every production proxy needs

This checklist organizes Envoy monitoring into four maturity levels: survival, operational, mature, and expert. Each level catches failure modes the previous level misses. The levels are cumulative. Level 2 assumes Level 1 is in place. A team alerting on outlier detection ejections without basic server liveness has gaps in the wrong direction.

Use this as an audit tool. Walk each level, confirm each signal is collected and alerted on (or deliberately omitted), and note the gaps. Most production Envoy deployments plateau around Level 2 with a few Level 3 additions.

The admin port question

The Envoy admin interface serves stats, readiness, and config dumps. Confirm the port before debugging anything else.

• Standalone Envoy: admin port is 9901. Readiness is GET /ready.
• Istio sidecar: admin port is 15000. The health probe endpoint is /healthz/ready on port 15021, not /ready on 9901.

If your scraper or probes point at the wrong port, every signal in this checklist will appear missing.

Warning: The admin interface allows destructive operations (including /quitquitquit, which shuts down the server) and exposes private configuration. Restrict network access to it. Do not expose it to the public internet.

flowchart TD
    L1["Level 1 - Survival
liveness, upstream health, traffic, memory"]
    L2["Level 2 - Operational
5xx, latency, response flags, TLS, connect failures"]
    L3["Level 3 - Mature
saturation, circuit breakers, xDS health, outlier ejections"]
    L4["Level 4 - Expert
retry economics, hot workers, per-host state, overload actions"]
    L1 --> L2 --> L3 --> L4

Level 1 - survival

Is Envoy up, are upstreams reachable, is traffic flowing, and is the process about to run out of memory.

A non-LIVE server.state means the process is draining or stuck waiting for initial xDS config. A membership_healthy of zero with nonzero membership_total means traffic has nowhere to go. A flat upstream_rq_total means Envoy is not proxying. A climbing server.memory_allocated without a traffic increase is a leak or a stats cardinality explosion heading toward OOM.

Readiness is not serving capacity: Envoy can return 200 from /ready while having zero healthy upstreams. It will accept connections and return 503s.

Memory during hot restart: server.memory_allocated includes memory from both the old and new processes during hot restart. Compare against your container memory limit, not against server.memory_heap_size, which can be much larger due to allocator fragmentation.

Level 2 - operational

Error visibility, latency, and the signals that distinguish Envoy-generated failures from upstream failures.

The response flags problem: downstream_rq_5xx and upstream_rq_5xx count both Envoy-generated 503s (circuit breaker open, no healthy upstream, no route) and upstream-forwarded 5xx. A 503 from a tripped circuit breaker has a completely different root cause and playbook than a 503 forwarded from a failing backend. Response flags tell them apart, but they are access-log only.

The %RESPONSE_FLAGS% field is available in access logs. It is not exposed as a stats counter. The flags that matter most in production:

For real-time alerting on these, you need a log processing pipeline that counts flag occurrences, or a custom Lua/Wasm filter that exports them as stats. Without that, you are alerting on a blended 5xx rate that conflates “Envoy is protecting itself” with “the backend is broken.”

Latency thresholds: use baseline multiples, not fixed values. P99 above 2x the rolling 24-hour P99 average warrants investigation. P50 above the normal P99 means the entire distribution has shifted. The default upstream_rq_time histogram buckets are wide (0.5ms to 1h); configure custom buckets aligned with your SLOs.

Level 3 - mature

Saturation leading indicators, circuit breaker state, and xDS configuration health. This is where you stop reacting to failures and start seeing them form.

The pending queue is the leading indicator: most teams alert on 503s. By the time 503s appear, the pending queue is full and users are already failing. Alerting on upstream_rq_pending_active growth gives minutes of warning before overflow starts. Any sustained nonzero value means the connection pool is becoming a bottleneck.

The xDS trap: teams monitor control_plane.connected_state but miss cluster_manager.cds.update_rejected (the CDS-level NACK counter). Envoy can be connected to the control plane (state = 1) while silently NACKing every config update. The operator sees “deployment completed” on the control plane side, and Envoy kept the old config. Monitor both connection state and rejection rate. Check Envoy’s process logs for NACK details, since the rejection reason is logged but not exposed via stats.

File descriptors are a hard cliff: at 100% FD utilization, all new connection attempts fail simultaneously with no graceful degradation. The default ulimit on many systems is 1024, which is inadequate for a production proxy. During hot restart, both old and new processes hold FDs, so usage briefly doubles. If baseline usage is above 50% of the limit, hot restart can trigger exhaustion.

Level 4 - expert

Deep signals teams add after repeated incidents: retry economics, worker thread health, per-host granularity, and overload manager visibility.

Retry economics: the ratio upstream_rq_retry / upstream_rq_total should be a standard dashboard metric. Above 0.1 (10%) warrants investigation. Above 0.3 is a retry storm. Track upstream_rq_retry_success alongside it: if retries rarely succeed, they are adding load without helping. In a retry storm, the upstream sees 2x-3x the actual client load, which accelerates the failure retries are trying to mask.

Watchdog misses are not subtle: watchdog_miss should be zero. Any nonzero value means a worker thread was blocked longer than the watchdog timeout (typically 200ms). This directly causes request latency spikes. In Kubernetes, CFS throttling can cause watchdog_miss even when Envoy is not computationally overloaded: the kernel is throttling the CPU quota. Check container_cpu_cfs_throttled_periods_total at the container level.

Overload manager visibility: if the overload manager is not configured (common in default deployments), Envoy has no self-protection against memory exhaustion. It will simply OOM. The overload manager is the mechanism that triggers downstream_cx_overload_reject. Without it configured, Envoy goes straight from “fine” to “dead.” Verify max_heap_size_bytes is set appropriately for your container limits.

Deployment variants that change the checklist

The four levels apply universally, but emphasis shifts by deployment type.

• Sidecar (Istio/service mesh): admin port is 15000, not 9901. Every pod has its own Envoy, so fleet-scale matters more than single-instance depth. xDS control plane load and connected_state across thousands of sidecars is critical. Per-pod memory limits interact with Envoy’s heap behavior. Scale changes the monitoring problem from “is this instance healthy” to “are all instances converging on the same config.”
• Edge/gateway: fewer instances, higher per-instance connection counts. TLS termination dominates CPU. Focus on connection management, TLS handshake rates, and rate limiting behavior. FD limits matter more here than in sidecar mode.
• Front proxy (non-mesh): often static config. xDS signals are less relevant. Focus on upstream health, throughput, and TLS. The connection pool exhaustion cascade is the most common incident pattern.
• HTTP/2 or gRPC upstreams: connection counts are not a proxy for throughput. A single HTTP/2 connection multiplexes many streams. upstream_cx_active can be low while upstream_rq_active (concurrent requests) is high. Monitor request-level concurrency, not connection count.

How Netdata helps

• Per-second collection of server.state, membership_healthy/membership_total, and upstream_rq_time lets you see a saturation cascade form before 503s appear, which is the window where intervention is cheap.
• Correlating upstream_rq_pending_active growth with upstream_rq_time and circuit_breakers.*_open in a single view catches the connection-pool exhaustion pattern without switching between dashboards.
• control_plane.connected_state paired with update_rejected distinguishes “disconnected, running stale” from “connected but silently NACKing,” which are different incidents with different fixes.
• server.memory_allocated and overload_actions.*.active together show whether the overload manager is actively saving the process or whether it is unconfigured and heading straight for OOM.

Related guides

• How Envoy actually works in production: a mental model for operators
• Envoy monitoring maturity model: from survival to expert
• Envoy no healthy upstream: the 503 when a cluster has no host to route to
• Envoy membership_healthy dropping: reading the single most important cluster signal
• Envoy outlier detection mass ejection: when passive health checks empty a cluster
• Envoy panic threshold: why traffic routes to unhealthy hosts at 50%
• Envoy upstream_cx_connect_fail: failed TCP connections to upstream hosts
• Envoy health checks vs outlier detection: two systems that eject hosts differently
• Envoy upstream_rq_pending_overflow: the pending queue fills and 503s begin
• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy upstream_cx_active near max_connections: the pool filling up
• Envoy connection pool exhaustion: a slow upstream that fills the pool