Envoy symptoms from the kernel: conntrack exhaustion and CFS throttling

Envoy looks healthy. The admin endpoint returns 200, cluster membership is stable, upstream_rq_time is in its normal band. But every few minutes a small fraction of requests fail with cluster.<name>.upstream_cx_connect_fail ticking up, and nothing inside Envoy explains it. Upstream hosts are up, the network path looks clean, the response flag is UF with no further detail.

Or: server.watchdog_miss starts incrementing during a traffic burst, tail latency spikes, and yet the cgroup CPU chart shows Envoy using well under its declared limit. No hot worker, no expensive filter, no lock contention. Envoy’s own telemetry says “I am not the problem.”

Both patterns share a root-cause location: outside Envoy, in the Linux kernel or the container runtime. This article covers the two kernel-originated symptoms that most often surface inside Envoy and waste hours of misdirected debugging: nf_conntrack table exhaustion, and CFS bandwidth throttling from Kubernetes CPU limits. It also covers a third, related trap: load-balancer health check traffic that pollutes downstream_rq_total and downstream_rq_2xx and makes everything else harder to reason about.

What this means

Conntrack exhaustion

Linux nf_conntrack records the state of every network flow that traverses the host’s netfilter layer. In Kubernetes, kube-proxy in iptables mode creates a conntrack entry for every Service IP, NodePort, and pod-to-pod flow. The table has a fixed maximum (nf_conntrack_max); when it fills, the kernel stops accepting new flow entries and silently drops the SYN packet that would have created one.

Envoy sees the result, not the cause. The SYN it sent toward the upstream never receives a SYN-ACK because the kernel on this host (or on the upstream’s host) dropped it. Envoy’s TCP connect attempt eventually fails, upstream_cx_connect_fail increments, and the access log records response flag UF. Nothing in Envoy’s own stats explains why the connect failed, because nothing inside Envoy failed.

The symptom is usually intermittent. Flows expire, the table drains, the next few connects succeed, then pressure rebuilds. This pattern is what makes it easy to chase as a “network blip” or “flaky upstream” for hours.

CFS throttling

Kubernetes CPU limits are enforced through the Completely Fair Scheduler’s bandwidth control. Each scheduling period (default 100ms), the cgroup receives a CPU quota equal to limit * period. If the cgroup’s threads consume their quota before the period ends, the kernel stops scheduling those threads until the next period refills the bucket.

Envoy’s worker threads are event-driven and single-threaded per worker. When the kernel refuses to schedule them, the event loop stalls. The main thread’s watchdog expects each worker to check in within miss_timeout (default 200ms); a stalled worker increments server.watchdog_miss. A longer stall crosses megamiss_timeout (default 1000ms) and increments server.watchdog_mega_miss. The current Envoy watchdog does not kill the process by default; kill_timeout defaults to 0, and the watchdog only increments counters.

The trap is that the container’s CPU utilization looks low. A pod limited to 200m CPU that bursts to 100m of work in the first 20ms of a 100ms period has spent half its quota in 20ms of wall clock. Apparent CPU is low because the kernel refused to give the container more time, not because the container did not want it.

Health check traffic inflation

A separate but related diagnostic hazard. The load balancer in front of Envoy polls a health endpoint at a fixed interval. Every poll lands on the same listener and the same HTTP connection manager as real traffic, so it increments downstream_rq_total and, on success, downstream_rq_2xx. On a low-traffic instance, health checks can be a meaningful fraction of total request volume, and they obscure the signal you actually care about.

Exclude health check paths and user-agents from application-traffic analysis. Treating inflated downstream_rq_total as the baseline makes it harder to spot a real traffic drop, and a sudden change in health check frequency can masquerade as a traffic anomaly.

Common causes

Quick checks

The commands below are read-only. The admin port is 9901 for a standalone Envoy and 15000 for an Istio sidecar. Adjust to match your deployment.

# Watchdog counters - any nonzero value is abnormal
curl -s http://localhost:9901/stats | grep -E 'watchdog_miss|watchdog_mega_miss'

# Per-cluster connection failures - intermittent ticks during conntrack pressure
curl -s http://localhost:9901/stats | grep 'upstream_cx_connect_fail'

# Compare current conntrack occupancy to the max
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max

# Kernel log for explicit conntrack drops
dmesg -T | grep 'nf_conntrack: table full' | tail

# CFS throttling on cgroup v2 - look at nr_throttled and throttled_usec
cat /sys/fs/cgroup/cpu.stat

# CFS throttling on cgroup v1
# Note: read the pod's own cgroup, not the host root. Path depends on cgroup driver
# (cgroupfs vs systemd) and runtime; typically under /sys/fs/cgroup/cpu/<slice-or-pod>/cpu.stat
cat /sys/fs/cgroup/cpu/cpu.stat

# Per-thread CPU for Envoy - confirms whether workers were not scheduled.
# -H shows threads of the matched process; pgrep -x returns the main envoy PID.
top -H -p $(pgrep -x envoy | head -1) -b -n 1

How to diagnose it

flowchart TD
    A[Symptom inside Envoy] --> B{What does the symptom look like?}
    B -->|upstream_cx_connect_fail, UF flag| C[Conntrack path]
    B -->|watchdog_miss, latency spikes| D[CFS throttling path]
    C --> E[nf_conntrack_count vs nf_conntrack_max]
    E --> F{Ratio approaching max?}
    F -->|Yes| G[Conntrack exhaustion confirmed]
    F -->|No| H[Look elsewhere: SYN backlog, firewall, DNS]
    D --> I[Read cpu.stat - nr_throttled]
    I --> J{nr_throttled climbing?}
    J -->|Yes| K[CFS throttling confirmed]
    J -->|No| L[Look for hot worker or filter cost]

Conntrack path

1. Confirm upstream_cx_connect_fail increments are time-correlated with each other rather than tied to a specific upstream host. If only one host is failing, the problem is on that host, not in conntrack.
2. Sample nf_conntrack_count and nf_conntrack_max together. The table is effectively full well before _count reaches _max; once the kernel’s early_drop path is evicting victims, you are already dropping new flows. Treat a sustained ratio above roughly 0.85 as a confirmed problem.
3. Check dmesg -T for the literal string nf_conntrack: table full, dropping packet. This message is the kernel’s only explicit admission that a drop happened.
4. Correlate against kube-proxy mode. iptables mode creates a conntrack entry for every Service and NodePort flow; ipvs mode uses fewer entries per service. A node with hundreds of pods in iptables mode routinely needs nf_conntrack_max at 1M or more.
5. Distinguish from upstream SYN backlog overflow by checking upstream_cx_connect_ms. If TCP connect time is normal but connects still fail, the SYN was dropped, not delayed. A slow connect with eventual success points at the upstream, not at conntrack.

CFS throttling path

1. Confirm watchdog_miss increments line up in time with the latency spikes. If they do not, you have a different problem. Look at the hot worker thread pattern next.
2. Read cpu.stat. On cgroup v2 the path is /sys/fs/cgroup/cpu.stat; on cgroup v1 it is /sys/fs/cgroup/cpu/<pod-cgroup>/cpu.stat. Look for nr_throttled and throttled_usec (v2, microseconds) or throttled_time (v1, nanoseconds) climbing between samples.
3. If you have Prometheus or equivalent, compute the throttled fraction: rate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m]). Anything above a few percent is worth investigating for a latency-sensitive proxy.
4. Read the pod spec. The throttling is a property of the declared CPU limit, not of node-wide CPU pressure. A node with spare CPU will still throttle a pod whose limit is too low for its burst profile.
5. Rule out a hot worker before concluding it is throttling. top -H -p $(pgrep -x envoy) shows per-thread CPU. One thread at 100% with others idle is a hot worker; the fix is different.

Metrics and signals to monitor

Note: downstream_cx_destroy_remote_active_rq is only a weak secondary indicator for conntrack pressure — the link is indirect (conntrack → upstream connect fail → 503 downstream → possible client-side connection teardown) and depends on client behavior.

Fixes

Conntrack exhaustion

The right fix depends on whether you are looking at a long-running capacity gap or a one-off burst.

• Raise nf_conntrack_max via sysctl net.netfilter.nf_conntrack_max=N. Reasonable values on Kubernetes nodes start around 1M and scale with pod and service density. Persist the change in /etc/sysctl.d/ or your node bootstrap; runtime sysctl writes do not survive a reboot. Increasing nf_conntrack_max is generally safe but raises kernel memory use roughly linearly with entry count.

• Set nf_conntrack_buckets at module load time. Use a modprobe config: options nf_conntrack hashsize=N, where hashsize is typically nf_conntrack_max / 4.

• Shorten stale-flow timeouts. Default established timeouts are long. For short-lived proxy traffic, lower values for nf_conntrack_tcp_timeout_established and related timeouts reduce steady-state occupancy. Do not set them so low that legitimate long-lived flows get evicted.

• Switch kube-proxy to ipvs mode. iptables mode creates a conntrack entry per Service flow; ipvs mode is more efficient at scale. This is a cluster-level change, not an Envoy change.

• Use NOTRACK sparingly. You can exclude specific traffic from conntrack entirely via -j NOTRACK, but this breaks stateful firewalling for that traffic. Reserve it for high-volume, stateless paths you fully understand.

CFS throttling

• Remove the CPU limit. This is the workaround the Kubernetes community has converged on for trusted, latency-sensitive workloads. Keep the CPU request for scheduling; drop the limit. Long-running Kubernetes issue #67577 documents that CFS quotas can throttle well-behaved pods even when the node has spare CPU.
• Set --cpu-cfs-quota=false on the kubelet. Cluster-wide alternative to per-pod limit removal. Disables CFS quota enforcement for all pods on that node. This is a disruptive, node-scoped change; roll it deliberately and coordinate with other tenants on the node.
• Raise the limit so bursts do not trip the quota. If you must keep limits for accounting reasons, size them to your burst profile, not to your average CPU. A 100ms period means any single-period burst above the quota triggers throttling, regardless of average utilization.
• Account for thread count. Multi-threaded runtimes (Java, Go, Node.js) sum CPU across all threads. A runtime with 8 threads doing 30ms of work each consumes 240ms of CPU quota in 30ms of wall clock and exhausts a 100ms-period budget in roughly 12ms. Envoy itself is multi-threaded; server.concurrency workers each consume quota.
• Check your kernel version. CFS bandwidth control has had several throttling fixes across kernel releases; older kernel series (4.x and early 5.x) have known throttling regressions that were patched in later point releases. Before upgrading, review the CFS bandwidth control changelog for your kernel series.

Health check traffic inflation

• Tag access logs with user-agent and request path. Most load balancers use a recognizable user-agent or hit a dedicated path like /healthz.
• Compute two downstream_rq_total series: with and without the health check filter. Alert and dashboard on the filtered series for application-traffic analysis; keep the unfiltered series for capacity and listener-availability views.
• Beware changes in health check frequency. A load balancer failover or a probe config push can shift the health check rate enough to look like a traffic anomaly in the unfiltered series.

Prevention

• Alert on nf_conntrack_count / nf_conntrack_max sustained above 0.80. Leave headroom below the point where early_drop starts evicting. Alert on the node, not the pod.
• Alert on container_cpu_cfs_throttled_periods_total rate. Any sustained rate is suspicious for a latency-sensitive proxy. The exact threshold is workload-dependent.
• Alert on any server.watchdog_miss increment. Any nonzero value is abnormal.
• Track pod and service density per node against conntrack capacity. A node that worked fine at 60 pods may pressure conntrack at 110 pods even though CPU and memory look fine.
• Document which workloads run without CPU limits. This is a deliberate choice for latency-sensitive infra components like Envoy; make sure the next operator knows.

How Netdata helps

• The Linux collector exposes nf_conntrack_count and nf_conntrack_max per second, so conntrack pressure is visible at the same resolution as Envoy’s own stats.
• The cgroup collector surfaces container_cpu_cfs_throttled_periods_total and container_cpu_cfs_periods_total per container, with derived throttled-fraction charts.
• The Envoy collector brings server.watchdog_miss, server.watchdog_mega_miss, and cluster.<name>.upstream_cx_connect_fail into the same timeline as the kernel and cgroup signals, which is where the actual diagnosis happens.
• Per-second resolution matters for both failure modes. Conntrack drops and CFS throttling events are short; a 30-second scrape interval can hide the correlation entirely.
• ML-based anomaly detection flags a watchdog_miss transition off its zero baseline without requiring a fixed threshold, and surfaces an nf_conntrack_count rise that has not yet crossed a static line.
• Anomaly hints for downstream_rq_total shifts help distinguish a real traffic change from a health check frequency change.

Related guides

• Envoy 502 and upstream resets: rx_reset, tx_reset, and mid-response failures
• Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend
• Envoy 504 upstream timeout: upstream_rq_timeout, per-try timeouts, and the UT flag
• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy clusters stuck warming: warming_clusters non-zero and routes returning 503
• Envoy connection pool exhaustion: a slow upstream that fills the pool
• Envoy control_plane.connected_state = 0: running on stale xDS config
• Envoy downstream 4xx spike: 401s, 403s, and 404s from the client side
• Envoy downstream connection flood: slowloris, the cx-to-rq ratio, and oversized requests
• Envoy downstream_cx_active growing: connection leaks and idle-timeout gaps
• Envoy downstream_cx_overflow and overload_reject: connections turned away at the door
• Envoy downstream_rq_time high: client-observed latency and proxy overhead