Envoy watchdog_miss and the hot worker: one saturated event loop hiding in the average

server.watchdog_miss just incremented. Aggregate CPU shows Envoy at 40%, comfortably under capacity. But a slice of requests are spiking P99, P50 is fine, and latency variance is high.

This is the hot worker pattern. Envoy is single-threaded per worker: each worker owns its connections for their entire lifetime and runs its own event loop. When one worker’s event loop blocks past the watchdog timeout (200ms by default), the main thread increments watchdog_miss. A single saturated worker is invisible in process-level CPU averages because the other workers idle along.

Aggregate metrics were designed for thread-pool servers where work is shared. Envoy does not work that way. A request that lands on the hot worker waits behind whatever is blocking the loop. A request that lands on any other worker passes through cleanly. The result is bimodal latency and intermittent watchdog misses while everything looks calm on average.

This guide covers finding the hot worker, determining why its loop is blocked, and fixing the root cause.

What this means

Each Envoy worker thread runs an independent libevent event loop. Workers accept connections through SO_REUSEPORT (default true since Envoy v1.20.0), which gives each worker its own kernel listen socket. Once a worker accepts a connection, it owns that connection for its entire lifetime. There is no work stealing and no connection migration between workers.

The main thread runs a watchdog that checks whether each worker has updated its shared timestamp within the miss timeout (200ms by default). If a worker has not touched its timestamp within that window, the watchdog increments server.watchdog_miss. If the blockage persists past the longer mega-miss threshold, server.watchdog_mega_miss increments.

While a worker is blocked, every connection owned by that worker is stalled. In-flight requests on that worker see their processing freeze. New connections arriving on that worker’s socket queue in the kernel. The other workers keep serving normally. The symptom shape is consistent: a slice of traffic experiences severe latency while the rest is fine.

The watchdog supports configurable actions that fire in order of severity: MISS, MEGAMISS, MULTIKILL, and KILL. By default, kill_timeout is 0 (disabled), so the watchdog logs and counts misses but does not terminate the process (multikill_timeout also defaults to 0/disabled; both facts come straight from the bootstrap API reference). If you have enabled kill actions and a transient scheduler delay trips them, a noisy neighbor or a brief CFS throttle can take down the entire Envoy process. Check your watchdog configuration before assuming a crash was caused by application logic.

Common causes

Quick checks

# Check watchdog miss and mega-miss counters
curl -s http://localhost:9901/stats | grep -E 'watchdog_miss|watchdog_mega_miss'

# Check worker concurrency
curl -s http://localhost:9901/stats | grep 'server.concurrency'

# Per-thread CPU: identify which worker thread is hot
top -H -p $(pgrep -x envoy) -b -n 1 | head -20

# Check if dispatcher stats are enabled (loop_duration_us, poll_delay_us)
curl -s http://localhost:9901/stats | grep -E 'loop_duration|poll_delay'

# Check CFS throttling (cgroup v2)
grep 'nr_throttled' /sys/fs/cgroup/cpu.stat

# Check TLS handshake rate (downstream)
curl -s http://localhost:9901/stats | grep 'ssl\.handshake'

# Check per-handler connection distribution across workers
curl -s http://localhost:9901/stats | grep 'downstream_cx_active'

In Istio sidecar mode, replace port 9901 with 15000. The cgroup path differs on cgroup v1 systems, typically under /sys/fs/cgroup/cpu/cpu.stat.

How to diagnose it

The diagnostic flow narrows from “is a worker blocked?” to “why is it blocked?” in five steps.

flowchart TD
    A["watchdog_miss incrementing"] --> B{"CFS throttling?
nr_throttled growing"}
    B -->|"Yes"| C["Increase or remove CPU limit"]
    B -->|"No"| D{"One thread at 100%
in top -H?"}
    D -->|"No"| E["Check megamiss
and hot restart epoch"]
    D -->|"Yes"| F{"Connection
imbalance?"}
    F -->|"Yes: per-handler
cx_active skewed"| G["Try exact_balance
for long-lived connections"]
    F -->|"No"| H{"Correlates with
TLS handshakes?"}
    H -->|"Yes"| I["TLS storm: increase workers
or session resumption"]
    H -->|"No"| J["Expensive filter:
profile Lua/Wasm/ext_authz"]

1. Confirm the miss is real and ongoing. Sample server.watchdog_miss twice, 10 seconds apart. If the counter is not advancing, the miss was transient, possibly during a hot restart where two Envoy processes briefly share CPU. If it is climbing, a worker is actively blocked right now.

2. Identify the hot thread. Run top -H -p $(pgrep -x envoy) -b -n 1 and look for a single thread near 100% CPU while the others are low. If no thread is hot, the block is likely CFS throttling (the kernel is preempting the thread, not the thread burning CPU) or a hot restart in progress. Check server.hot_restart_epoch if you suspect the latter.

3. Determine whether the cause is connection imbalance or computation. Check per-handler downstream connection counts. If one worker holds a disproportionate share of active connections, the problem is SO_REUSEPORT distribution, not per-request cost. If connection distribution is even but one worker is still hot, the problem is per-request work: filter execution, TLS, or buffering.

4. Check for CFS throttling if running in Kubernetes. Read /sys/fs/cgroup/cpu.stat (cgroup v2) and look for a climbing nr_throttled count. Alternatively, check container_cpu_cfs_throttled_periods_total from cAdvisor. If throttling is happening, the kernel is quota-limiting the Envoy process even though the application is not computationally saturated. This is one of the most common root causes of watchdog misses in Istio sidecars and it is entirely invisible in Envoy’s own stats.

5. Enable dispatcher stats if they are not already on. Dispatcher stats provide per-worker loop_duration_us and poll_delay_us histograms that tell you exactly how long each worker’s event loop iterations take. They are disabled by default because they add stats volume. Enable them temporarily in the bootstrap config with enable_dispatcher_stats: true, reproduce the issue, then correlate histogram spikes with the miss timestamps. As an operational baseline, a healthy proxy shows poll_delay_us P50 below 100us and P99 below 5ms (field-experience numbers, not official thresholds — compare against the proxy’s own baseline rather than these absolutes). Do not leave dispatcher stats on permanently in high-throughput production.

Metrics and signals to monitor

Fixes

Expensive Lua or Wasm filter

If the hot thread correlates with traffic to a specific route or virtual host, inspect the filter chain on that route. Lua filters that do body inspection, header manipulation on large headers, or regex matching are common offenders. Wasm filters have additional overhead from the runtime boundary.

Move expensive logic out of the data path where possible. If you cannot eliminate the filter, distribute its cost: split the affected listener across more workers, or move the logic to a sidecar that handles it asynchronously.

SO_REUSEPORT connection imbalance

SO_REUSEPORT uses a kernel hash to distribute incoming connections across worker sockets. For short-lived HTTP/1.1 connections, the distribution is statistically uniform. For long-lived connections (gRPC, HTTP/2 streams, WebSocket), the hash can concentrate connections on one worker. A single gRPC connection that lives for hours lands on one worker and stays there.

For workloads dominated by long-lived connections, set connection_balance_config: { exact_balance: {} } on the listener. exact_balance holds a lock during accept to distribute connections nearly exactly across workers. It trades accept throughput for balance, which is the right tradeoff when connection count is low but each connection is expensive.

CFS throttling in Kubernetes

If /sys/fs/cgroup/cpu.stat shows climbing nr_throttled, the kernel is throttling Envoy’s CPU quota. The CFS period is 100ms by default. A burst of work (TLS handshakes, a spike in request rate) can exhaust the quota within a period, causing the kernel to preempt the thread until the next period. From Envoy’s perspective, this looks exactly like a hot worker: the event loop is blocked because the kernel will not schedule it.

The fix is to increase or remove the CPU limit. Envoy sidecars in Istio are particularly prone to this because the default CPU limits are often too low for TLS-heavy mesh traffic. If you cannot remove limits, raise them to accommodate burst traffic with at least 30% headroom.

TLS handshake storms

TLS handshakes are CPU-intensive. A burst of new connections after a load balancer failover, a client restart, or a certificate rotation that drops existing sessions can saturate a worker with handshake work. The signal is miss rate correlating with the ssl.handshake counter rate, not the request rate.

Mitigations: enable TLS session resumption (session tickets or session IDs) to reduce handshake volume. Increase worker count to spread handshake cost across more cores. If downstream TLS terminates on Envoy at an edge, consider reducing cipher suite cost or offloading TLS.

Blocking filters

If a filter performs synchronous I/O, it blocks the event loop. Common culprits: synchronous DNS resolution in a filter, ext_authz configured with a timeout longer than the watchdog miss window, or any filter that does file I/O on the request path.

Switch to async variants where available. For ext_authz, ensure the timeout is well under 200ms. For DNS, Envoy uses c-ares for async resolution by default; if you have configured synchronous DNS in a custom filter, that is the problem.

Prevention

• Monitor per-thread CPU, not just process CPU. Aggregate CPU is the metric that hides this problem. Track per-core CPU utilization and alert on any thread sustained above 80%.
• Enable dispatcher stats in staging. poll_delay_us and loop_duration_us give direct visibility into event loop health. Enable them in non-production and during incident reproduction. They increase stats volume; do not leave them on permanently in high-throughput production.
• Set CPU limits with headroom for TLS bursts. In Kubernetes, CPU limits adequate for steady-state will throttle during TLS handshake storms. Leave 30% headroom or remove limits if your cluster scheduler supports it.
• Track CFS throttling as a first-class metric. container_cpu_cfs_throttled_periods_total should be on every Envoy sidecar dashboard. Correlate it with watchdog_miss before debugging Envoy internals.
• Profile filter cost before deploying. Lua and Wasm filters should be benchmarked against production-like traffic before deployment. A filter that adds 1ms per request is fine at 100 RPS but blocks the loop at 10,000 RPS on a single worker.

How Netdata helps

• Per-second per-core CPU metrics expose the hot worker immediately. When one core is pinned at 100% and others idle, the per-core breakdown makes the imbalance visible without manual top -H sessions.
• ML anomaly detection on watchdog_miss and watchdog_mega_miss surfaces the first increment, not just the sustained trend. A single miss is the leading indicator before tail latency affects users.
• Correlation between CFS throttling and watchdog misses shortens the Kubernetes diagnosis path. When container_cpu_cfs_throttled_periods_total and server.watchdog_miss move together, the root cause is the scheduler, not Envoy.
• Per-second latency histograms reveal the bimodal distribution characteristic of a hot worker: P50 stable while P99 spikes, with high variance. This pattern is invisible in minute-granularity metrics.
• TLS handshake rate alongside CPU connects handshake storms to worker saturation without a separate query or dashboard switch.

Related guides

• Envoy 502 and upstream resets: rx_reset, tx_reset, and mid-response failures
• Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend
• Envoy 504 upstream timeout: upstream_rq_timeout, per-try timeouts, and the UT flag
• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy clusters stuck warming: warming_clusters non-zero and routes returning 503
• Envoy connection pool exhaustion: a slow upstream that fills the pool
• Envoy control_plane.connected_state = 0: running on stale xDS config
• Envoy downstream_rq_time high: client-observed latency and proxy overhead
• Envoy health checks vs outlier detection: two systems that eject hosts differently
• How Envoy actually works in production: a mental model for operators
• Envoy listener_create_failure: a listener config Envoy could not apply
• Envoy membership_healthy dropping: reading the single most important cluster signal