$ guides / envoy ▌

ENVOY · OPERATIONS PLAYBOOK

Envoy fails on purpose: it 503s to protect a slow upstream, tags every response with the reason, and keeps serving stale config when the control plane goes quiet

A multi-threaded L4/L7 proxy where each worker owns its connections and its own copy of cluster state, every upstream call passes a circuit breaker and a connection pool, and the whole configuration arrives over a single xDS stream to a control plane. We trace how that design turns a slow backend into cluster-wide 503s, why the response flag matters more than the status code, and what to watch when the control plane disconnects.

> Start with the monitoring checklist → # Jump to the full guide list

Envoy's defaults get you proxying in minutes, then hand you failure modes that only make sense once you know it fast-fails to protect the upstream, not to serve your request.

The defaults work. Until a backend slows down, connections pile up in a per-worker pool, the pending queue fills, and upstream_rq_pending_overflow starts returning 503s with response flag UO — Envoy protecting the upstream while looking broken. Until a cluster loses its last healthy host and every request comes back no healthy upstream. Until the xDS control plane drops, control_plane.connected_state goes to 0, and Envoy keeps serving last-known-good config while new endpoints, routes, and certificate rotations quietly never arrive. Until a pushed config is NACKed (update_rejected) and the change you deployed silently never takes effect. Until one worker thread pins a core and a slice of requests slow down while aggregate CPU looks fine.

These guides are written for engineers who already run Envoy — as an Istio sidecar, an edge gateway, or a front proxy — not for people learning what a proxy is. The goal is the mental model of how Envoy actually behaves under load, the failure patterns that keep recurring, the signals that catch them before they page anyone, and the runbooks you wish you'd had during your last 503 storm.

How Envoy actually runs in production

Envoy is not just a reverse proxy. It is a multi-threaded event machine where every worker owns its connections and holds its own copy of cluster state, each upstream call passes a route, a load balancer, a connection pool, and a circuit breaker, and the entire configuration is streamed in from a control plane over xDS. Most production failures live between these layers, not inside any one of them.

downstream clients + listeners

Each listener accepts downstream connections on a socket; with <code>SO_REUSEPORT</code> a connection is handed to exactly one worker, which owns it for its whole life. <code>downstream_cx_active</code> is the current load; the listener enforces connection limits and rejects overflow (<code>downstream_cx_overflow</code>) silently at the TCP layer.

LISTEN

network + HTTP filter chains

L4 network filters and, for HTTP, the connection manager plus its L7 filter chain — TLS termination, compression, <code>ext_authz</code>, rate limiting, RBAC, Lua/Wasm, and finally the router. Every filter costs CPU on the worker; an expensive one shows up as <code>downstream_rq_time</code> overhead and can pin a worker.

FILTER

route table (RDS)

The router matches each request against virtual hosts and routes to pick a cluster. No match yields response flag <code>NR</code> (no route) — almost always a bad xDS push. Routes also carry the timeout and retry policy that govern everything downstream of here.

ROUTE

cluster manager + load balancing

Each worker holds a thread-local copy of cluster state. The load balancer picks a host from the healthy set; outlier detection ejects hosts on real-traffic errors, and below the panic threshold (default 50%) Envoy deliberately balances across <em>all</em> hosts, including unhealthy ones.

CLUSTER

connection pools + circuit breakers

Per-worker, per-host, per-protocol pools. Circuit breakers cap connections, pending requests, requests, and retries per cluster. When a pool saturates, requests wait in <code>upstream_rq_pending_active</code>; when that fills, <code>upstream_rq_pending_overflow</code> fast-fails with 503 and flag <code>UO</code>.

POOL

upstream hosts + health

Envoy opens upstream connections (plus TLS if configured) and runs active health checks from the main thread. <code>membership_healthy / membership_total</code> is the cluster's core availability signal; a failed connect increments <code>upstream_cx_connect_fail</code> and surfaces as flag <code>UF</code>.

UPSTREAM

worker threads + overload manager

Workers run independent libevent loops sharing nothing on the hot path. A blocked loop increments <code>watchdog_miss</code>; a single hot worker hides inside aggregate CPU. The overload manager watches heap and connections and, past its thresholds, sheds load (<code>stop_accepting_connections</code>) — or, if unconfigured, Envoy simply OOMs.

WORKER

main thread + xDS control plane

The main thread talks to the control plane over a single xDS stream (CDS/EDS/LDS/RDS/SDS), flushes stats, and serves the admin interface. <code>control_plane.connected_state = 0</code> means stale config; a rejected update (<code>update_rejected</code>) means the change silently never applied.

CONTROL

Why this matters: 'Envoy is returning 503s' can come from a saturated connection pool, a tripped circuit breaker, a cluster with no healthy hosts, a route that no longer matches, a mass host ejection, a retry storm, a saturated worker, or a stale xDS config. The status code is identical; the response flag and the layer differ — and so does the fix.

The failures you'll actually see

Most Envoy incidents fall into a small set of recurring patterns. Recognise the shape — and the response flag — and triage gets dramatically faster.

CRITICAL

The connection pool exhaustion cascade

A backend slows down. Connections are held longer, the per-worker pool fills, new requests pile into upstream_rq_pending_active, and when it hits max_pending_requests Envoy fast-fails with 503 and flag UO. It looks like Envoy is broken; it is protecting the upstream from more load. The give-away is that membership_healthy stays stable throughout.

upstream_rq_pending_overflow incrementing (503s with flag UO)
upstream_rq_pending_active and upstream_cx_active climbing toward max_connections
upstream_rq_time rising while membership_healthy stays flat
circuit_breakers.default.cx_open or rq_pending_open = 1

Investigate →

CRITICAL

No healthy upstream

A cluster's last healthy host drops out — a real outage, a health-check misconfiguration, or an outlier-detection cascade — and every request to it returns a 503 whose body reads no healthy upstream. Below the panic threshold Envoy starts routing to all hosts, healthy or not, so the error rate can look better than the cluster actually is.

503 responses with body 'no healthy upstream'
membership_healthy == 0 with membership_total > 0 and traffic flowing
outlier_detection.ejections_active approaching membership_total
membership_healthy / membership_total below the panic threshold (default 50%)

Investigate →

ACTIVE

The 503 you can't explain

Envoy generates 503 for many unrelated reasons — a tripped circuit breaker, no healthy host, no matching route, a failed upstream connect — and the status code alone cannot tell them apart. The response flag in the access log is the only thing that says why. Teams that don't log or parse flags end up blaming the backend for Envoy's own fast-fails.

Elevated downstream_rq_5xx with no obvious backend fault
Access logs show flags like UO, NR, UF, UT, UC on the failing requests
upstream_rq_5xx counting Envoy-generated and upstream 503s together
No %RESPONSE_FLAGS% in the access log format (flying blind)

Investigate →

ACTIVE

Upstream connect error or reset

Clients get a 503 whose body reads upstream connect error or disconnect/reset before headers, with a trailing reset reason. Envoy could not establish or hold the upstream connection — the host is down, a firewall changed, an idle-timeout mismatch closed the socket, or the connection reset mid-flight. The reset reason and response flag (UF / UC) name the mechanism.

503 body 'upstream connect error or disconnect/reset before headers'
reset reason: connection failure / connection termination / remote reset
upstream_cx_connect_fail rising, response flag UF or UC
upstream_rq_rx_reset climbing during backend deploys or restarts

Investigate →

IMMINENT

The stale-config time bomb

The xDS control plane goes unreachable and control_plane.connected_state drops to 0. Envoy keeps serving its last-known-good config, so nothing breaks immediately — until a deploy adds a service, a scale event moves endpoints, or a certificate needs rotating, and none of it reaches Envoy. The disconnect happened hours before the incident it eventually causes.

control_plane.connected_state = 0 sustained beyond a few minutes
membership_total unchanged despite known scaling events
New services or routes returning 404 (NR) or 503
Config version drifting apart across a fleet of sidecars

Investigate →

IMMINENT

The silent NACK

The control plane pushes a config and Envoy validates it, finds it invalid, and rejects it (update_rejected) — keeping the old config. connected_state stays 1, the control plane reports success, and the change you shipped silently never takes effect. The rejection reason lives only in Envoy's process log, not in stats.

update_rejected or listener_create_failure incrementing after a push
connected_state = 1 but the intended config change is not live
config_dump shows an older version than the control plane pushed
A deployed fix that appears to have done nothing

Investigate →

Choosing a tool

Best Envoy Proxy Monitoring Tools (2026): 10 Ranked

A ranked review of the tools teams actually shortlist here, what each one is genuinely good at, and how the pricing behaves as you scale.

Read the buyer's guide →

Envoy monitoring maturity levels

Envoy observability works in four practical levels. Each is a complete operation, not a stepping stone. Pick the level that matches how much your proxy matters. Most production deployments should land at the second level.

Level 1: Survival

Know that something is wrong

Survival monitoring is the floor: is Envoy up, are its backends up, is traffic flowing, and is it about to run out of memory. It won't tell you why, but it tells you before your users do. Enough for dev proxies and non-critical paths.

Process liveness (/ready, server.state) Is Envoy LIVE and accepting connections, or draining / initializing?
Upstream host health (membership_healthy/total) Does each cluster still have a host to route to?
Downstream 5xx rate Are clients getting errors right now?
Memory allocated vs container limit How close is this instance to an OOM kill?
Traffic flowing (upstream_rq_total) Is the proxy actually passing requests?

Level 2: Operational

Diagnose most incidents on your own

Operational monitoring is what most production Envoy deployments should target. Survival says something is wrong; this says what — which 503, which cluster, saturation, stale config, retries, TLS. With this coverage a team can usually triage on its own.

Response flags from access logs UO / NR / UF / UT say why a 503 happened; the status code can't.
Upstream and downstream latency (rq_time) P99 vs rolling baseline, and the proxy-overhead delta.
Pending queue overflow (upstream_rq_pending_overflow) The leading indicator before a cluster starts 503ing.
xDS connected_state Is Envoy running on live config or last-known-good?
Retry ratio (upstream_rq / downstream_rq) Above ~1.3, retries are inflating upstream load.
Upstream connect failures (upstream_cx_connect_fail) Backends refusing TCP connections.
TLS handshake failures (ssl.fail_verify_error) Certificate or CA problems on downstream or upstream.
File descriptor usage vs limit Cliff-edge: at the ulimit, every new connection fails.

Level 3: Mature

Catch problems before they page anyone

Mature monitoring catches the slow burns: a saturating pool, one hot worker, hosts being ejected, a NACKed config, stats quietly filling their region. None of these page you on day one; they become the incident on day thirty.

Worker saturation (watchdog_miss, poll_delay_us) One blocked event loop the aggregate CPU hides.
Connection pool state (cx_active, pending_active) Per-cluster runway before the circuit breaker trips.
Circuit breaker gauges (*_open) Which breaker is at its limit, per priority.
Outlier detection ejections (ejections_active) Passive health removing hosts on real-traffic errors.
xDS NACKs (update_rejected, listener_create_failure) Config the control plane pushed and Envoy refused.
Overload manager actions (overload_actions.*.active) Envoy shedding load to avoid OOM.
Connection rejections (cx_overflow / overload_reject) Clients turned away at the listener.
Stats cardinality (stat count and growth) A full stats region drops new metrics silently.

Level 4: Expert

Reactive instrumentation after real incidents

Expert signals enter your stack the day after a specific incident proved you needed them. Per-host and per-worker detail, retry effectiveness, connection reuse, and the adjacent-system signals that produce Envoy symptoms with no Envoy cause. Most teams never need every signal here.

Per-host upstream stats (/clusters) A single bad host hidden inside a healthy cluster average.
Retry success ratio (retry_success / retry) Useful retries vs load-adding wasteful ones.
Connection reuse ratio (rq_total / cx_total) Detects pool churn and keepalive misconfiguration.
CFS throttling (cpu_cfs_throttled_periods_total) K8s CPU limits causing watchdog misses at low CPU.
Conntrack table utilization Silent connection drops that look like upstream_cx_connect_fail.
Certificate expiry countdown (/certs) Validates SDS / cert-manager rotation before it breaks.
Config version consistency across the fleet Instances stuck on an old xDS version.
Memory fragmentation (heap_size vs allocated) tcmalloc holding freed memory — alarming but usually benign.

Operating mistakes worth avoiding

The traps Envoy teams keep falling into. Each has a clear, well-known fix. Most teams only learn it after an incident.

Alerting on 503s instead of the pending queue

By the time <code>downstream_rq_5xx</code> spikes, the connection pool is already full and users are already failing. <code>upstream_rq_pending_active</code> growing gives minutes of warning before <code>upstream_rq_pending_overflow</code> starts returning 503s. Watch the leading indicator, not the lagging one.

Reading aggregate worker CPU

Envoy is single-threaded per worker. '20% process CPU' can mean everything is fine or that one worker is pinned at 100% while the rest idle — and the requests on that worker are slow. Per-thread CPU and <code>watchdog_miss</code> are the only way to see a hot worker; the average never will.

Blaming the backend for Envoy's own 503s

A 503 can be the upstream app or Envoy itself (circuit breaker, no healthy host, no route, connect failure). <code>upstream_rq_5xx</code> counts both together. The response flag in the access log is the only thing that separates them — teams without flag logging debug the wrong system for hours.

Watching connected_state but not update_rejected

Everyone eventually alarms on xDS disconnection. Fewer notice that Envoy can be connected and still NACK every config push — <code>connected_state = 1</code>, <code>update_rejected</code> climbing, the old config still live. The deployment looks done and Envoy silently ignored it. Monitor both.

Treating a tripped circuit breaker as something to raise

The reflex when a breaker trips is to increase the limit. That removes the protection without fixing the cause, and the next failure lets an unprotected upstream collapse completely. The breaker is a symptom; the slow or failing upstream is the cause. Fix the upstream.

Configuring retries and forgetting them

Retries feel free until an upstream partially fails and they double or triple its load, accelerating the collapse they were meant to mask. The <code>upstream_rq_total / downstream_rq_total</code> ratio should be a standing dashboard metric; above ~1.3 it is telling you retries are amplifying, not helping.

Running without an overload manager

Many deployments never configure one. Without it Envoy has no self-protection against memory pressure — it goes straight from 'fine' to OOM-killed, with no <code>overload_actions</code> and no graceful shedding. It also needs <code>max_heap_size_bytes</code> set correctly or it never triggers. Configure it in every production instance.

Ignoring stats cardinality until metrics vanish

The stats region is a fixed-size shared-memory block. Dynamic route names or per-request tags blow up cardinality, and when the region fills Envoy silently stops registering new metrics — no error, just growing blind spots. Track <code>curl /stats | wc -l</code> as a metric and prune with a stats matcher.

Envoy runbooks in this section

Each guide is a focused runbook for one symptom or topic. Pick one when you have an incident, or use the categories to learn the area.

▸

Start here

▸

Upstream health and outlier detection

▸

Connection pools and circuit breakers

▸

5xx errors and response flags

▸

Retries and retry amplification

▸

Latency, throughput, and traffic anomalies

▸

xDS control plane and configuration

▸

Memory, CPU, file descriptors, and overload

▸

Downstream connections and listeners

▸

TLS, certificates, and authorization

▸

Process lifecycle, hot restart, and deployment

WHERE TO GO NEXT

Setting up Envoy monitoring, or in the middle of a 503 storm?

If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, decode the response flag first, then jump to the symptom that matches it.

> Start with the checklist > Back to Operations Guides