How Envoy actually works in production: a mental model for operators

Envoy is a multi-threaded, event-driven L4/L7 proxy written in C++. Its architecture directly shapes what you see in metrics, access logs, and user-visible behavior during incidents. If you do not know that worker threads share nothing in the hot path, aggregate CPU utilization will mislead you. If you do not know that circuit breaker 503s are indistinguishable from upstream-generated 503s at the counter level, you will blame the wrong component. If you do not know that Envoy keeps serving traffic on stale configuration after an xDS disconnect, you will miss the slow-burn failure that surfaces hours later.

This article covers the abstractions operators need before debugging: the threading model, the request path through filter chains, per-worker connection pools, and the self-protection mechanisms (circuit breakers, outlier detection, overload manager). Once these are internalized, the failure patterns and the metrics that expose them become predictable.

Threading model

The threading model is the single most important thing to understand about Envoy. It determines how CPU is consumed, how connections are owned, how configuration reaches workers, and why certain metrics must be read at per-worker granularity rather than in aggregate.

Envoy uses three thread categories:

• Main thread: Process lifecycle, xDS communication with the control plane, stats flushing, the admin interface, active health checking, and hot restart coordination. It does not process data plane traffic. A slow stats flush or a busy admin endpoint does not compete with request processing.
• Worker threads: Each worker runs its own libevent event loop and accepts downstream connections via SO_REUSEPORT. Once a worker accepts a connection, it owns that connection for its entire lifetime. Workers share nothing in the hot path. This is how Envoy scales near-linearly, and it is why per-worker reasoning matters when diagnosing latency or saturation.
• File flush thread: Asynchronous access log writes, keeping logging I/O off the data plane entirely.

The number of worker threads is controlled by --concurrency (default: number of hardware threads on the machine).

flowchart TD
    CP["xDS control plane
CDS, EDS, LDS, RDS, SDS"]
    MT["Main thread
xDS client, stats flush,
admin, health checks, hot restart"]
    WT["Worker threads
Each: libevent loop, thread-local
cluster mgr, own conn pools"]
    FF["File flush thread
async access log writes"]
    DC["Downstream clients"]
    US["Upstream hosts"]
    CP -->|"gRPC or REST"| MT
    MT -->|"RCU pointer swaps"| WT
    DC -->|"SO_REUSEPORT
worker owns conn for life"| WT
    WT -->|"upstream requests"| US
    WT -->|"access logs"| FF

Request path and self-protection

The request path

A downstream connection arrives at a listener socket. The kernel balances accepts across workers via SO_REUSEPORT. One worker wins and owns the connection for its lifetime. The connection then passes through:

1. Listener filters (L4): Optional pre-processing such as TLS inspector to detect SNI before routing decisions.
2. Network filter chain (L4): Includes the transport socket for TLS termination. The HTTP connection manager (HCM) is the terminal network filter for HTTP traffic.
3. HTTP connection manager: Decodes the protocol (HTTP/1.1, HTTP/2, HTTP/3) and manages the downstream stream lifecycle.
4. HTTP filter chain (L7): Sequential processing through configured filters: external authorization, rate limiting, Lua or Wasm extensions, compression, and finally the router filter.
5. Router filter: Resolves the route against the route table, selects the target upstream cluster, applies load balancing policy, and initiates the upstream request.
6. Connection pool: The router requests a connection from the per-worker, per-host, per-protocol pool. If no connection is available and the pool has not reached its limit, a new one is established. Otherwise the request queues.
7. Upstream: The request is sent to the upstream host. The response flows back through the filter chain in reverse order.

Thread-local cluster manager and connection pools

Each worker maintains its own copy of cluster state through the thread-local cluster manager: connection pools, load balancing weights, and host health. The main thread receives configuration updates via xDS and distributes them to workers via RCU-style pointer swaps through thread-local storage slots. Configuration changes are lock-free in the hot path but eventually consistent across workers.

Connection pools are per-worker, per-upstream-host, and per-protocol (HTTP/1.1, HTTP/2, HTTP/3, TCP). A cluster with 10 hosts across 8 workers has up to 80 independent connection pools. Per-cluster aggregate stats can hide per-host or per-worker hotspots behind healthy averages.

HTTP/2 and HTTP/3 connection pools multiplex multiple concurrent streams over fewer TCP connections. upstream_cx_active can be low while request concurrency is high. Connection counts are not a throughput proxy for multiplexed protocols. Track upstream_rq_active for concurrent in-flight requests instead.

Circuit breakers

Circuit breakers are configured per-cluster and per-priority (default and high). They cap maximum connections, pending requests, concurrent requests for multiplexed protocols, and retries. When a limit is exceeded, Envoy fast-fails the request locally with a 503 and response flag UO. The request never reaches the upstream.

This is a deliberate protection mechanism, not a bug. Envoy sheds load early to prevent cascading failure. Workers share circuit breaker state via eventually consistent counters, so brief races between threads can allow limits to be marginally exceeded.

Check which circuit breakers are open via the stats endpoint:

curl -s localhost:9901/stats | grep circuit_breakers

Outlier detection vs active health checks

These are independent systems with different signal sources:

• Active health checks run on the main thread. They send synthetic probes to upstream hosts at configured intervals. A host that fails health checks is removed from load balancing rotation.
• Outlier detection is passive. It observes real traffic outcomes and ejects hosts based on consecutive 5xx errors, success rate thresholds, or failure percentage.

A host can pass active health checks but be ejected by outlier detection because real traffic is failing. Conversely, a newly failed host that has not yet received traffic will not be caught by outlier detection until health checks mark it unhealthy. Both systems contribute to membership_healthy, membership_degraded, and membership_excluded gauges.

When the healthy host percentage drops below the panic threshold (default 50%), Envoy enters panic mode and load-balances across all hosts including unhealthy ones. This is intentional. The alternative is concentrating all traffic on the few remaining healthy hosts until they also fail.

Overload manager

The overload manager monitors Envoy’s own resource consumption, primarily heap size. When configured thresholds are crossed, it triggers actions: stop accepting connections, stop accepting requests, disable HTTP keepalive, reduce timeouts, shrink heap.

If the overload manager is not configured, Envoy has no self-protection against memory exhaustion. It will consume memory until the container OOM-kills it. The overload manager depends on max_heap_size_bytes being set correctly. If it is not set or set too high, the monitor never fires.

Deployment variants and what they change

The deployment variant changes which signals matter and what “normal” looks like:

Failure modes built into the design

The same architecture that makes Envoy fast creates specific, predictable failure patterns.

Connection pool exhaustion cascade. Upstream slows down. Connections are held longer. The pool fills. Pending requests queue. The queue overflows. Envoy returns 503 with flag UO. The system looks broken, but it is protecting the upstream from additional load.

Memory pressure spiral. Large request or response bodies combined with buffering filters cause heap growth. If the overload manager is configured, it degrades service to survive. If it is not, the process is OOM-killed.

xDS stale configuration. The control plane disconnects. Envoy keeps serving with last-known-good config. Existing traffic works. New endpoints, route changes, and certificate rotations are invisible. The failure surfaces hours later when stale endpoints receive traffic or new services return 503 with flag NR. Check control_plane.connected_state and compare version_info in the config dump against what the control plane reports as current.

Retry amplification. Aggressive retry policy meets partial upstream failure. Retries add load to the already-degraded upstream. More failures trigger more retries. The upstream collapses under 2-3x normal load, accelerated by the mechanism designed to mask the failure.

Stats cardinality explosion. Dynamic route names or per-request metadata in stat tags fill the shared-memory stats region. New stats silently fail to register. Monitoring develops blind spots with no error or warning.

Hot worker. One worker thread saturates at 100% CPU (expensive filter, TLS handshake storm, lock contention) while others idle. Requests assigned to that worker experience high latency. Aggregate process CPU looks moderate. Only per-thread analysis or watchdog_miss counters expose the imbalance.

Outlier detection mass ejection. Aggressive outlier detection settings eject hosts during a correlated failure (network issue, shared dependency). Remaining hosts overload and get ejected. The panic threshold triggers. Envoy routes to all hosts including unhealthy ones by design.

Hot restart FD exhaustion. The new process starts before the old one finishes draining. Both processes hold file descriptors simultaneously. FD usage briefly doubles. If baseline FD usage is above 50% of ulimit -n, hot restart can trigger FD exhaustion.

Signals to watch

How Netdata helps

Netdata’s Envoy collector scrapes the admin stats endpoint at per-second granularity. The operational value is in correlation across metrics that aggregate dashboards miss:

• Watch upstream_rq_pending_active and circuit_breakers.*_open alongside upstream 503 counters to separate circuit breaker rejections from upstream errors in real time.
• Correlate server.memory_allocated with overload manager action gauges to confirm whether Envoy is shedding load due to its own memory pressure or listener limits.
• Track the upstream_rq_total to downstream_rq_total ratio per cluster to surface retry amplification hidden by aggregate error rates.
• Netdata’s anomaly detection on per-cluster latency histograms and membership ratios catches gradual degradation before fixed thresholds fire.
• server.state, control_plane.connected_state, and cluster_manager.warming_clusters together distinguish planned drain from stuck initialization during hot restarts and xDS reconnects.

Related guides

• Envoy monitoring checklist: the signals every production proxy needs
• Envoy monitoring maturity model: from survival to expert
• Envoy no healthy upstream: the 503 when a cluster has no host to route to
• Envoy membership_healthy dropping: reading the single most important cluster signal
• Envoy outlier detection mass ejection: when passive health checks empty a cluster
• Envoy panic threshold: why traffic routes to unhealthy hosts at 50%
• Envoy upstream_cx_connect_fail: failed TCP connections to upstream hosts
• Envoy health checks vs outlier detection: two systems that eject hosts differently
• Envoy upstream_rq_pending_overflow: the pending queue fills and 503s begin
• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy upstream_cx_active near max_connections: the pool filling up
• Envoy connection pool exhaustion: a slow upstream that fills the pool