Envoy Operations Guides on Netdata

Envoy 502 and upstream resets: rx_reset, tx_reset, and mid-response failures

Mon, 01 Jan 0001 00:00:00 +0000

A client request made it past Envoy’s routing, established a TCP connection to an upstream host, and then the connection died before the response completed. The access log shows a UC or UPE response flag, upstream_rq_rx_reset is climbing, and the client received a 502 or 503.

This is a different failure class from “no healthy upstream” (503, NR flag) or “connection refused” (503, UF flag). Those happen before a connection is established. Resets happen after the handshake succeeds, which means the upstream was reachable and then failed during request processing. The diagnosis and fix are completely different.

Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend

Mon, 01 Jan 0001 00:00:00 +0000

You see 503 responses in access logs with the UO response flag. The flag is what pins the cause: UO means upstream overflow. A circuit breaker fast-failed the request locally before forwarding. The backend did not refuse the connection, did not time out, and may not be aware the request existed. Envoy decided the cluster was at capacity and returned an immediate 503.

This is the most commonly misdiagnosed Envoy 503. Operators chase the backend, restart pods, scale the upstream, and the errors persist because the upstream is not the thing that is broken. The circuit breaker is working correctly. It is protecting an upstream that is either slow, under-provisioned, or receiving more concurrent load than its pool can absorb. The fix is to address the upstream saturation or right-size the breaker limit. Restarting Envoy will not help.

Envoy 504 upstream timeout: upstream_rq_timeout, per-try timeouts, and the UT flag

Mon, 01 Jan 0001 00:00:00 +0000

A 504 from Envoy with response flag UT means the upstream request timeout fired: Envoy sent the request to a backend and did not receive a complete response within the configured budget. The backend may be genuinely slow, the timeout may be too tight for the workload, or a filter may be interfering with the timeout clock. Each case needs a different fix.

One diagnostic trap: Envoy records a timed-out request in the upstream_rq_time histogram at the timeout duration, not at the actual response time. This creates an artificial ceiling in the latency distribution that looks like a real latency wall. Without cross-referencing the upstream_rq_timeout counter against the histogram, you cannot distinguish genuine backend slowness from the timeout itself shaping the data.

Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests

Mon, 01 Jan 0001 00:00:00 +0000

You see cx_open=1 or rq_pending_open=1 on a production cluster. Access logs show 503 responses tagged with the UO response flag. Clients receive fast-failed requests, sometimes with an x-envoy-overloaded header. The circuit breaker gauges are binary: 0 means the breaker has headroom and can admit more work, 1 means it is at capacity and rejecting. Each gauge is scoped per-cluster and per-priority (default or high), so you need the right cluster and priority combination to read the signal correctly.

Envoy clusters stuck warming: warming_clusters non-zero and routes returning 503

Mon, 01 Jan 0001 00:00:00 +0000

cluster_manager.warming_clusters reading non-zero during steady state is a config convergence problem. Envoy has accepted a new cluster (or a cluster update) but cannot activate it because a dependency has not resolved: DNS, an SDS secret, an EDS endpoint set, or an active health-check initialization. Until warming completes, the cluster is invisible to routing for new additions, or held in swap-for-update for modifications. Routes targeting a freshly added but still-warming cluster return 503 with response flag NC (no cluster), or 404 with NR (no route), depending on whether the route table entry has been pushed yet.

Envoy connection churn: a low reuse ratio and keepalive misconfiguration

Mon, 01 Jan 0001 00:00:00 +0000

Envoy’s upstream connection pool amortizes TCP (and TLS) handshakes across many requests. When it stops doing that, you have connection churn: the pool establishes a fresh connection for nearly every request, and the reuse ratio collapses toward 1.0.

The headline signal is the ratio upstream_rq_total / upstream_cx_total. For HTTP/1.1 with keepalive working, expect a number well above 1, often in the tens. For HTTP/2 with stream multiplexing, expect far higher still. When the ratio sits near 1.0, every request pays the full handshake tax.

Envoy connection pool exhaustion: a slow upstream that fills the pool

Mon, 01 Jan 0001 00:00:00 +0000

Clients are seeing 503 responses. Access logs show response flag UO on those requests. But upstream health looks fine: membership_healthy is stable and hosts are passing active health checks. The upstream is not down, yet Envoy is refusing to forward new requests to it.

This is connection pool exhaustion. An upstream that was previously fast has become slow. Each request now holds a connection longer, so the same request rate fills more connection slots. The per-worker connection pool saturates, new requests queue in the pending buffer, the buffer overflows, and Envoy fast-fails those requests with 503 UO rather than piling on more load.

Envoy control_plane.connected_state = 0: running on stale xDS config

Mon, 01 Jan 0001 00:00:00 +0000

control_plane.connected_state is a gauge that flips to 0 the moment Envoy’s gRPC stream to its xDS management server drops. The flip produces no user-visible symptom: existing traffic keeps flowing, listener sockets stay open, and Envoy keeps serving its last-known-good configuration. There is no default expiry on that cached config.

That silence is why this signal is under-monitored. By the time anyone notices, new endpoints have been invisible for hours, removed endpoints have been sending traffic to dead or reassigned IPs, routes never updated, and SDS-managed certificates stopped rotating. The disconnect happened long before the visible incident, which makes root-cause correlation non-obvious.

Envoy DNS resolution failures: STRICT_DNS clusters serving stale endpoints

Mon, 01 Jan 0001 00:00:00 +0000

A STRICT_DNS cluster stops picking up new endpoints, and endpoints you removed from DNS hours ago are still receiving traffic. Envoy’s own dashboard shows the cluster as healthy, error rates look flat, and the upstream service reports traffic to IPs that no longer exist. The first concrete evidence is often a wave of 502s or connection failures when a stale IP gets reassigned to an unrelated workload.

This failure mode is specific to STRICT_DNS, and to a lesser extent LOGICAL_DNS, clusters. In these cluster types, DNS is the endpoint-discovery mechanism: Envoy resolves the configured DNS target on a timer, treats each returned A/AAAA record as an upstream host, and drains connections to hosts that disappear from the result. When resolution fails or returns an unexpected result, no membership update happens, and Envoy keeps load-balancing across the previous endpoint set.

Envoy downstream 4xx spike: 401s, 403s, and 404s from the client side

Mon, 01 Jan 0001 00:00:00 +0000

A spike in http.<stat_prefix>.downstream_rq_4xx is usually a client-side story, not an Envoy story. The proxy is reporting that clients sent bad, unauthorized, or unroutable requests. The single counter lumps 400s, 401s, 403s, and 404s together, and Envoy does not expose per-status-code downstream counters. You will not find downstream_rq_401 or downstream_rq_403 in the stats dump.

Envoy downstream connection flood: slowloris, the cx-to-rq ratio, and oversized requests

Mon, 01 Jan 0001 00:00:00 +0000

The proxy is rejecting new connections, latency is climbing, and upstreams look healthy. The problem is at the front door. A downstream connection flood exhausts Envoy’s resources before a request reaches a filter chain. The classic signal: a high new-connection rate paired with a low request rate. Many TCP handshakes, few HTTP requests. That ratio is the cx-to-rq ratio, and when it inverts you are looking at a slowloris-style attack or a misconfigured client pool.

Envoy downstream request rate dropping to zero: the traffic blackhole

Mon, 01 Jan 0001 00:00:00 +0000

A sudden collapse of http.<stat_prefix>.downstream_rq_total to near zero is one of the loudest availability signals Envoy can emit, and one of the easiest to misread. Operators trained to chase 5xx spikes often treat a falling request rate as “the system is calm.” It is not. A drop below roughly 10% of baseline on a listener that should be receiving traffic is a traffic blackhole: clients have stopped reaching Envoy, Envoy has stopped accepting requests, or traffic is reaching Envoy but being answered by local replies before any upstream work happens.

Envoy downstream_cx_active growing: connection leaks and idle-timeout gaps

Mon, 01 Jan 0001 00:00:00 +0000

listener.<address>.downstream_cx_active is a gauge counting the downstream (client-facing) connections currently held open by a listener. When it climbs without a matching rise in request rate, the usual first call is “connection leak”. That diagnosis is sometimes correct and sometimes wrong, and the fixes are completely different.

The gauge reflects three independent inputs: how fast new connections arrive (downstream_cx_total), how fast old connections close (downstream_cx_destroy), and how long each connection is allowed to live once idle (idle, stream, drain, and connection-duration timers). A leak is only one explanation. Others include clients that legitimately hold connections open (HTTP/2 and gRPC multiplexing), loopback sidecar traffic, an idle timeout set far longer than the workload needs, or a slow upstream that holds streams, and therefore connections, open past the point the idle timer could reclaim them.

Envoy downstream_cx_overflow and overload_reject: connections turned away at the door

Mon, 01 Jan 0001 00:00:00 +0000

When Envoy rejects a new TCP connection at the listener, the client gets no HTTP error or retry hint: the SYN is answered with a RST or dropped, and the client sees connection refused or a connect timeout. The only evidence is three listener-level counters that should always be zero: downstream_cx_overflow, downstream_cx_overload_reject, and downstream_global_cx_overflow.

Each counter corresponds to a distinct rejection mechanism with a different root cause and fix. Confusing them wastes time tuning the wrong limit while the real problem is memory pressure, file descriptor exhaustion, or a missing overload manager.

Envoy downstream_rq_time high: client-observed latency and proxy overhead

Mon, 01 Jan 0001 00:00:00 +0000

High downstream_rq_time means client-observed latency through the proxy is climbing. The histogram lives under http.<stat_prefix>.downstream_rq_time, measured in milliseconds, and it maps most directly to user-perceived slowness. When it breaches SLO, you are in an incident whether the upstreams are healthy or not.

The common reflex is to subtract upstream_rq_time from downstream_rq_time and label the remainder “Envoy overhead.” That delta is useful as a trend, but it is not a clean measurement of proxy processing time. It folds in filter execution, downstream upload and download behavior, retry time across multiple attempts, buffering delays, streaming duration, and how fast the client reads the response. A high delta can mean Envoy is working hard, a client is slow, a request was retried, or a long-lived SSE stream is open. The histogram alone will not tell you which.

Envoy ext_authz failure_mode_allowed: unauthenticated traffic when auth is down

Mon, 01 Jan 0001 00:00:00 +0000

The Envoy ext_authz filter calls an external authorization service on every request that matches the filter chain. That call is on the request critical path: nothing is forwarded upstream until the auth service returns a decision. When the auth service is unreachable, returns an HTTP 5xx, or exceeds its timeout, Envoy either lets the request through without an auth decision or rejects it. That choice is controlled by failure_mode_allow, and every time the fail-open branch fires, Envoy increments http.<stat_prefix>.ext_authz.failure_mode_allowed.

Envoy file descriptor exhaustion: the FD cliff that refuses every new connection

Mon, 01 Jan 0001 00:00:00 +0000

When Envoy runs out of file descriptors, it is not a slow degradation. At the RLIMIT_NOFILE ceiling every new downstream accept() and every new upstream connect() fails at once. Clients see resets, timeouts, or 503s, and the proxy looks dead even though established streams keep flowing. Unlike memory or CPU pressure, there is no graceful ramp and rarely an overload-manager warning before the cliff.

Envoy is FD-hungry because each proxied connection consumes roughly two file descriptors: one downstream socket and one upstream socket. Add listen sockets, the admin socket, xDS gRPC streams, active health-check connections, and access-log files, and a busy proxy can hold tens of thousands of FDs open with nothing wrong. The default 1024 ulimit on many base images is therefore far too low for a production proxy and is the most common root cause.

Envoy health checks vs outlier detection: two systems that eject hosts differently

Mon, 01 Jan 0001 00:00:00 +0000

Envoy has two independent mechanisms for removing unhealthy upstream hosts from load balancing. Active health checks send synthetic probes on a schedule. Outlier detection watches real request traffic and ejects hosts based on observed errors. They are not redundant, and they do not always agree.

A host can pass every health check and still be ejected by outlier detection. The two systems run on different threads, observe different signals, and act on different timelines. Operators who watch only membership_healthy or only outlier_detection.ejections_active miss half the picture.

Envoy hot restart races: draining, epoch churn, and dropped connections

Mon, 01 Jan 0001 00:00:00 +0000

You deployed a new Envoy binary or triggered a reload that invoked a hot restart. Seconds later, clients report connection resets, your 5xx rate ticks up, and server.hot_restart_epoch is climbing on your dashboard. The new process was supposed to inherit listen sockets gracefully while the old one drained.

Hot restart is Envoy’s mechanism for zero-downtime binary upgrades and certain reloads. A new process launches, coordinates with the old one over a Unix domain socket, takes over the listen sockets, and the old process enters a drain sequence. Both processes run simultaneously during the handoff. That coexistence is where the races live.

Envoy listener_create_failure: a listener config Envoy could not apply

Mon, 01 Jan 0001 00:00:00 +0000

The listener_manager.listener_create_failure counter increments when Envoy cannot add a listener object to its workers. In a healthy deployment this counter is zero. Any non-zero value means Envoy received a listener configuration it could not apply: a failed bind (port conflict, permission denied) or invalid configuration.

The signal is easy to miss because Envoy does not crash. It keeps serving the previous listener configuration, traffic on existing routes flows normally, and the listener the operator intended to add or modify never becomes active. Deployment pipelines report success. The control plane believes the config was accepted. Only the process log and this counter reveal the failure.

Envoy membership_healthy dropping: reading the single most important cluster signal

Mon, 01 Jan 0001 00:00:00 +0000

If you watch only one availability metric per Envoy cluster, watch the ratio of cluster.<name>.membership_healthy to cluster.<name>.membership_total. When the ratio collapses, the remaining healthy hosts take proportionally more load, and the cluster is one or two failures away from panic mode. Upstream 5xx rate, latency, circuit breaker state, and retries are all downstream of host availability.

The hard part is reading the gauge correctly, not collecting it. A drop can mean the upstream is genuinely broken, the control plane removed endpoints, or outlier detection ejected hosts that are still passing active health checks. Each case has a different response.

Envoy memory pressure spiral: server.memory_allocated climbing toward OOM

Mon, 01 Jan 0001 00:00:00 +0000

server.memory_allocated is climbing, traffic is flat, and you are hours or minutes from an OOM kill. In a sidecar deployment, the pod dies and the application gets the blame. In an edge or gateway deployment, Envoy vanishes and clients see connection resets across the board.

Envoy has a built-in protection mechanism (the overload manager), but many deployments never configure it. Without it, there is no graceful degradation: Envoy goes straight from “memory looks fine” to “OOM killed” with nothing in between.

Envoy memory_heap_size vs memory_allocated: tcmalloc fragmentation that fools dashboards

Mon, 01 Jan 0001 00:00:00 +0000

Envoy exposes three memory gauges that operators naturally try to map onto process RSS: server.memory_allocated, server.memory_heap_size, and server.memory_physical_size. On a busy proxy the first two diverge by 2x-4x routinely, and dashboards that graph heap_size look like a slow leak even when nothing is wrong.

The gap is not a bug. Envoy ships with tcmalloc as its default allocator, and tcmalloc is built for allocation latency, not for prompt return of freed memory to the OS. It parks freed pages in per-thread caches, central caches, and the page heap free lists so the next allocation is fast. Those pages stay mapped into the process, counted in memory_heap_size, and very often still resident in RSS.

Envoy monitoring checklist: the signals every production proxy needs

Mon, 01 Jan 0001 00:00:00 +0000

This checklist organizes Envoy monitoring into four maturity levels: survival, operational, mature, and expert. Each level catches failure modes the previous level misses. The levels are cumulative. Level 2 assumes Level 1 is in place. A team alerting on outlier detection ejections without basic server liveness has gaps in the wrong direction.

Use this as an audit tool. Walk each level, confirm each signal is collected and alerted on (or deliberately omitted), and note the gaps. Most production Envoy deployments plateau around Level 2 with a few Level 3 additions.

Envoy monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Envoy exposes thousands of stats and dozens of admin endpoints. The trap is not lack of data; it is lack of the right data at the right depth for your operational maturity. A team monitoring /ready and membership_healthy will survive most outages but will be blind to the failure modes that cause multi-hour incidents: silent xDS NACKs, retry amplification, connection pool exhaustion, CFS throttling.

Four levels, each additive. Level 1 catches process death and total upstream loss. Level 2 adds the signals that explain why errors happen. Level 3 adds leading indicators that warn before saturation. Level 4 adds per-host drift, kernel interaction, and certificate runway. You keep every previous level’s coverage and add the next class.

Envoy NC no cluster: the cluster vanished mid-request

Mon, 01 Jan 0001 00:00:00 +0000

A request lands in Envoy, the route matches, the router filter looks up the cluster the route points at, and the cluster manager has no cluster by that name. Envoy fast-fails the request with the NC (No Cluster) response flag. This is not an upstream health problem or a transient network issue. Any non-zero rate of NC in production is a configuration or timing bug.

Unlike UO (circuit breaker) or UF (upstream connection failure), which point at the upstream, NC points at the configuration layer: a route exists in RDS, but the cluster it names is absent from CDS at the instant the request was routed.

Envoy no healthy upstream: the 503 when a cluster has no host to route to

Mon, 01 Jan 0001 00:00:00 +0000

When Envoy returns a 503 with body “no healthy upstream” and access log flag UH, the cluster has zero hosts available for load balancing. The cluster exists, routes point to it, traffic is flowing, but the load balancer cannot select a host.

The response body is literal. The flag UH appears in the %RESPONSE_FLAGS% access log field and means “No healthy upstream hosts in upstream cluster in addition to 503 response code.” It is distinct from UO (circuit breaker overflow), UF (upstream connection failure), and NR (no route).

Envoy NR no route: 404s and 503s after a bad xDS route push

Mon, 01 Jan 0001 00:00:00 +0000

The NR response flag in Envoy access logs means no route matched the request. The HTTP connection manager walked its route table (delivered via RDS or static config) and found no match for the incoming Host header, path, or other match criteria. In production, sustained NR is a configuration error. A spike right after an xDS route push points to a bad or rejected config update.

NR is usually paired with HTTP 404. Operators sometimes see it paired with 503, which is a different root cause: on Envoy versions before 1.18.0, NR was also set when a route matched but the referenced upstream cluster did not exist. Envoy 1.18.0 (April 2021) introduced the NC (No Cluster) flag for that case, so on modern Envoy, NR with 503 suggests you are either on an older version or actually looking at NC.

Envoy outlier detection mass ejection: when passive health checks empty a cluster

Mon, 01 Jan 0001 00:00:00 +0000

membership_healthy is collapsing. outlier_detection.ejections_active is climbing toward membership_total, and ejections_overflow is ticking up because Envoy wanted to eject more hosts than max_ejection_percent allows. This is the outlier detection mass ejection pattern: one of the few Envoy failure modes that can amplify a partial upstream degradation into a cluster-wide outage.

The mechanism is subtle because outlier detection is doing exactly what it was configured to do. It is a passive health check: it ejects hosts based on the actual traffic they are serving, not on synthetic probes. When one host starts returning 5xx or its success rate drops, ejecting it is correct. The problem is what happens next. Load concentrates on the survivors, they get slower, their success rates drop, and they get ejected too. The cascade feeds itself.

Envoy overload manager actions: stop_accepting_connections and the last line before OOM

Mon, 01 Jan 0001 00:00:00 +0000

When server.overload_manager.envoy.overload_actions.stop_accepting_connections.active flips to 1, Envoy has stopped accepting new TCP connections on its listeners. When stop_accepting_requests is active, Envoy returns 503 to new HTTP requests before they reach an upstream. These are not bugs. They are Envoy’s last intentional actions before the kernel OOM-kills the process.

The overload manager connects resource pressure (heap size, connection counts) to a cascade of protective actions. When it fires, you are looking at both a root cause (memory or connection pressure) and a symptom (traffic being refused). Many deployments never configure it, skipping every graceful degradation step and going straight from “fine” to OOM-killed with nothing in between.

Envoy panic threshold: why traffic routes to unhealthy hosts at 50%

Mon, 01 Jan 0001 00:00:00 +0000

You are looking at an Envoy cluster where membership_healthy has dropped below half of membership_total. The error rate has climbed. The first instinct is that something new has broken: the health checks are wrong, outlier detection is misfiring, or the upstream has a second fault. In most cases none of that is true. Envoy has entered panic mode, and the elevated error rate is a direct and expected consequence of the design.

Envoy rate limiting: over_limit, 429s, and fail-open vs fail-closed

Mon, 01 Jan 0001 00:00:00 +0000

Envoy has two HTTP filters for rate limiting with very different failure characteristics. The global rate limit filter delegates every applicable request to an external rate limit service (RLS) over gRPC. The local rate limit filter applies an in-process token bucket with no external dependency. Both can produce 429 responses, but the signals that tell you what happened live in different stat namespaces and mean different things.

The harder operational question is not “are we rate limiting” but “what happens when the rate limit service itself is down.” That answer is controlled by one setting: failure_mode_deny. Its default is false (fail-open), meaning a broken RLS silently disables rate limiting. Fail-closed (failure_mode_deny: true) instead rejects every request through the filter. A misread here is a common source of incidents.

Envoy RBAC: access denied: 403s from the RBAC filter and shadow-mode tuning

Mon, 01 Jan 0001 00:00:00 +0000

A 403 with the body RBAC: access denied is the fingerprint of Envoy’s HTTP RBAC filter blocking a request. It is not the upstream service refusing the request, and it is not the network RBAC filter, which closes the TCP connection instead of returning an HTTP status. When this counter rises you have two questions to answer: which policy matched, and was the match correct.

The filter emits four counters: http.<stat_prefix>.rbac.allowed, http.<stat_prefix>.rbac.denied, http.<stat_prefix>.rbac.shadow_allowed, and http.<stat_prefix>.rbac.shadow_denied. The shadow counters are the safe path for validating new or revised policies against real traffic without enforcing them.

Envoy response flags: decoding UO, NR, UF, UT, UC and the rest

Mon, 01 Jan 0001 00:00:00 +0000

A 503 from Envoy can mean a dozen different things. The HTTP status code tells you what the client saw; it does not tell you why Envoy generated that response. The %RESPONSE_FLAGS% access log field separates a circuit breaker trip from a missing route, a dead upstream, or a client that hung up.

Response flags are the most precise debugging signal Envoy emits, and the most operationally misunderstood. They are not aggregate stats; they appear only in access logs. Most teams discover this the first time they try to alert on UO and find no Prometheus counter for it.

Envoy retry policy tuning: retry_on, budgets, idempotency, and hedging

Mon, 01 Jan 0001 00:00:00 +0000

Retries are one of the few Envoy features that can make an outage measurably worse. A correctly tuned retry policy absorbs transient failures cleanly. A poorly tuned one multiplies upstream load by 2x-3x during a partial failure and accelerates the collapse it was supposed to mask. This article covers the knobs that matter operationally: retry_on conditions, retry budgets, per_try_timeout, and hedging. It also covers the one thing Envoy cannot decide for you: whether a request is safe to repeat.

Envoy retry storm: when the upstream-to-downstream request ratio climbs

Mon, 01 Jan 0001 00:00:00 +0000

A retry storm is one of the most dangerous failure modes in an Envoy-based service mesh or edge proxy. An upstream service starts failing a fraction of requests. Envoy’s retry policy, configured to mask transient errors, fires retries on the failures. The extra load lands on an already-degraded upstream. More requests fail under the additional load. More retries fire. Within minutes, the upstream receives two to three times its normal traffic, mostly retries, and collapses under amplification it cannot escape.

Envoy server.state not LIVE: draining, initializing, and the /ready probe

Mon, 01 Jan 0001 00:00:00 +0000

A Kubernetes readiness probe starts failing. The pod shows NotReady, the service stops sending traffic, and a rolling deploy stalls. You exec in and hit Envoy’s /ready endpoint:

# Check readiness state on the admin port
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:9901/ready
503

The probe is doing its job: Envoy is reporting server.state != LIVE. The real question is whether this is expected (rolling deploy, hot restart, normal warm-up) or genuine capacity loss (stuck init, orphaned drain, crashed child). Non-LIVE is normal during deploys, so alerting on server.state != LIVE alone is noisy. You need to combine it with listener state and uptime before paging.

What this means

Envoy exposes its lifecycle as a single gauge, server.state:

Envoy ssl.fail_verify_error: certificate verification failures on the TLS path

Mon, 01 Jan 0001 00:00:00 +0000

When ssl.fail_verify_error starts climbing on an Envoy proxy, TLS handshakes are failing peer certificate verification. A small burst during a planned rotation is operational noise. A sustained climb past roughly 1% of handshakes means the trust relationship between Envoy and its peers has broken.

This counter is one of several SSL stats Envoy tracks, and the others help narrow the failure mode. ssl.connection_error covers protocol-level issues such as TLS version or cipher mismatch. ssl.no_certificate fires when a client presented no certificate where one was required. ssl.fail_verify_error means a certificate was presented and parsed, but Envoy could not validate it against its configured trust context, SAN matchers, or pin hashes.

Envoy stats cardinality explosion: the stats region fills and new metrics vanish

Mon, 01 Jan 0001 00:00:00 +0000

Envoy builds each stat name from the resource names in your config: clusters, listeners, HTTP connection managers, routes. When those names are stable, cardinality is bounded. When they are dynamic, every distinct value mints a new metric. The budget that backs those stats fills, and new stats stop appearing with no error and no log line.

The failure is quiet by design. Envoy treats stat registration as best-effort relative to its memory budget. A missing metric raises no alert, increments no hot-path error counter, and never appears in access logs. You usually discover it weeks later, when an incident sends you hunting for a per-cluster or per-route metric that was never recorded.

Envoy stuck initializing: waiting on xDS while the pod never turns ready

Mon, 01 Jan 0001 00:00:00 +0000

The pod sits at 0/1 Running indefinitely. kubectl describe shows failing readiness probes, the Envoy sidecar never reports ready, and the application container may be healthy but unreachable because the sidecar gate is closed. Envoy’s /ready admin endpoint returns HTTP 503. /server_info reports PRE_INITIALIZING or INITIALIZING. The process is alive, but it never received its initial xDS configuration, so the listener sockets exist but no listeners are active.

This is not a slow cold start. A healthy Envoy cold start resolves in seconds, not minutes. When the init phase persists, the usual root cause is a slow or unreachable control plane at startup. If a liveness probe also points to /ready with a short timeout, kubelet kills the pod and you get a restart loop that looks like a crash but is a starvation pattern: each restart resets the warming clock and Envoy never gets a fair chance to fetch its first config.

Envoy symptoms from the kernel: conntrack exhaustion and CFS throttling

Mon, 01 Jan 0001 00:00:00 +0000

Envoy looks healthy. The admin endpoint returns 200, cluster membership is stable, upstream_rq_time is in its normal band. But every few minutes a small fraction of requests fail with cluster.<name>.upstream_cx_connect_fail ticking up, and nothing inside Envoy explains it. Upstream hosts are up, the network path looks clean, the response flag is UF with no further detail.

Or: server.watchdog_miss starts incrementing during a traffic burst, tail latency spikes, and yet the cgroup CPU chart shows Envoy using well under its declared limit. No hot worker, no expensive filter, no lock contention. Envoy’s own telemetry says “I am not the problem.”

Envoy TLS certificate expiry: expired certs, broken SDS rotation, and total outage

Mon, 01 Jan 0001 00:00:00 +0000

TLS certificate expiry in Envoy is one of the few failure modes that can take down an entire service mesh at once. A single expired server certificate breaks new handshakes on one listener. An expired CA root breaks every mTLS connection in the data plane, every Envoy-to-control-plane link, and every health check that uses TLS.

Expiry is silent until handshakes start failing. Envoy does not expose certificate expiration as a standard metric. The only authoritative source is the /certs admin endpoint, which must be polled externally. Teams running automated rotation via SDS or cert-manager often assume rotation is working and discover otherwise when TLS breaks across the fleet.

Envoy update_rejected (xDS NACK): the config the control plane pushed and Envoy quietly refused

Mon, 01 Jan 0001 00:00:00 +0000

You deploy a config change through your control plane. The deployment reports success. Traffic flows, error rates are flat, control_plane.connected_state stays at 1. But the change never took effect: the new route is missing, the timeout override is gone, the certificate rotation did not happen.

Envoy received the new config, validated it, found it invalid, and rejected it. It kept the previous config and kept routing. The rejection is recorded in update_rejected and, for listeners, in listener_manager.listener_create_failure. The reason itself is only in the Envoy process log, not in stats. The control plane reports success because the gRPC stream is healthy, not because Envoy accepted the config. Nothing fails visibly. The only symptoms are the things that should have changed but did not.

Envoy upstream connect error or disconnect/reset before headers: reading the reset reason

Mon, 01 Jan 0001 00:00:00 +0000

The 503 body upstream connect error or disconnect/reset before headers. reset reason: <reason> means Envoy selected an upstream host, tried to open a request stream on a pooled connection, and the stream was reset before any response headers came back. The body looks generic, but the trailing reset reason: field is the diagnostic payload. It is one of the values in Envoy’s StreamResetReason enum , and each value points at a different failure mechanism.

Envoy upstream_cx_active near max_connections: the pool filling up

Mon, 01 Jan 0001 00:00:00 +0000

The cluster.<name>.upstream_cx_active gauge reports active connections across all hosts in a cluster. When it trends toward the circuit-breaker max_connections limit, the cluster is approaching saturation. Past the limit, Envoy stops opening new connections, queues new requests in upstream_rq_pending_active, and once max_pending_requests is hit, returns 503s with response flag UO.

Envoy is fast-failing locally to protect an upstream that cannot absorb more concurrent work. Raising max_connections without addressing the upstream removes that protection. The operator’s job is to find what is shrinking effective pool capacity.

Envoy upstream_cx_connect_fail: failed TCP connections to upstream hosts

Mon, 01 Jan 0001 00:00:00 +0000

cluster.<name>.upstream_cx_connect_fail is a per-cluster counter that increments each time Envoy fails to establish a TCP connection to an upstream host. On a healthy cluster the rate is flat. The threshold for concern is connect_fail / connect_total > 0.05 , meaning more than 5% of connection attempts are failing. Sustained nonzero rates point at one of four root causes: the upstream process is down, the upstream is out of file descriptors, a firewall or ACL changed, or the upstream listen backlog is overflowing.

Envoy upstream_cx_connect_ms high: slow TCP connects to upstream hosts

Mon, 01 Jan 0001 00:00:00 +0000

When cluster.<name>.upstream_cx_connect_ms starts climbing, understand what this histogram actually measures: the time Envoy spent establishing a TCP connection to an upstream host. When upstream TLS is configured, the TLS handshake is rolled into the same number. It is not request latency, not application processing time, and not pure network RTT once TLS is in the path.

Same-zone connections typically sit under 2ms at P99. The alert threshold is a sudden 5x rise over rolling baseline. Anything beyond that means the network path has changed, the upstream kernel cannot accept connections fast enough, or something is forcing Envoy to open far more new connections than usual.

Envoy upstream_rq_pending_overflow: the pending queue fills and 503s begin

Mon, 01 Jan 0001 00:00:00 +0000

You see cluster.<name>.upstream_rq_pending_overflow incrementing on one or more clusters. Access logs show 503 with response flag UO. Clients get fast-fail 503s, not timeouts. The curve is cliff-edge: requests were flowing fine, and now a slice of them are rejected with no upstream attempt at all.

This is Envoy’s pending request circuit breaker doing exactly what it was configured to do. Requests arrive, there is no available upstream connection to attach them to, so they wait in upstream_rq_pending_active. When that queue hits max_pending_requests, Envoy stops queueing and starts rejecting. The overflow counter is the rejected count.

Envoy upstream_rq_retry_overflow: the retry budget exhausted and retries dropped

Mon, 01 Jan 0001 00:00:00 +0000

upstream_rq_retry_overflow is climbing on a cluster. Error rates are elevated, and retries should absorb the failures. But upstream_rq_retry is not growing proportionally, and the system looks like it stopped retrying. It did: the retry circuit breaker or retry budget is full, and Envoy is dropping retries to protect the upstream.

The retry system is working as designed, but legitimate retries are being dropped at the moment they are needed most. A budget that is too small loses the resilience retries provide. A budget that is too large risks 2x-3x traffic amplification during partial failures.

Envoy upstream_rq_time high: upstream latency as the proxy sees it

Mon, 01 Jan 0001 00:00:00 +0000

cluster.<name>.upstream_rq_time is the histogram that answers “how long did the upstream interaction take, as Envoy observed it?” When it climbs, the instinct is to page the backend team. That instinct is often wrong, or at least incomplete. The metric is wall-clock time measured at the HTTP router filter. It bundles several distinct latencies: TCP connect, upstream TLS handshake (for new connections), request transmission, upstream processing, and response transfer. It is not pure backend service time.

Envoy URX upstream retry limit exceeded: all retries used, last error returned

Mon, 01 Jan 0001 00:00:00 +0000

When you see URX in Envoy access logs, every configured retry attempt has fired and failed, and the last upstream error is what the client receives. This is distinct from retry_overflow, where retries were never attempted because the retry budget or circuit breaker was full. The distinction matters because the two conditions have opposite root causes and opposite fixes.

URX means the retry mechanism is doing its job mechanically: it tried, it retried, and the upstream kept failing. The question is not “why did Envoy give up” but “why does the upstream keep failing on retried requests.” The answer is usually one of three things: the upstream error is genuinely non-retriable (a 501 Not Implemented will never succeed on retry), the retry window is too small for the upstream to recover, or the retry policy is amplifying a partial failure into a full one.

Envoy watchdog_miss and the hot worker: one saturated event loop hiding in the average

Mon, 01 Jan 0001 00:00:00 +0000

server.watchdog_miss just incremented. Aggregate CPU shows Envoy at 40%, comfortably under capacity. But a slice of requests are spiking P99, P50 is fine, and latency variance is high.

This is the hot worker pattern. Envoy is single-threaded per worker: each worker owns its connections for their entire lifetime and runs its own event loop. When one worker’s event loop blocks past the watchdog timeout (200ms by default), the main thread increments watchdog_miss. A single saturated worker is invisible in process-level CPU averages because the other workers idle along.

How Envoy actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

Envoy is a multi-threaded, event-driven L4/L7 proxy written in C++. Its architecture directly shapes what you see in metrics, access logs, and user-visible behavior during incidents. If you do not know that worker threads share nothing in the hot path, aggregate CPU utilization will mislead you. If you do not know that circuit breaker 503s are indistinguishable from upstream-generated 503s at the counter level, you will blame the wrong component. If you do not know that Envoy keeps serving traffic on stale configuration after an xDS disconnect, you will miss the slow-burn failure that surfaces hours later.