Envoy file descriptor exhaustion: the FD cliff that refuses every new connection

When Envoy runs out of file descriptors, it is not a slow degradation. At the RLIMIT_NOFILE ceiling every new downstream accept() and every new upstream connect() fails at once. Clients see resets, timeouts, or 503s, and the proxy looks dead even though established streams keep flowing. Unlike memory or CPU pressure, there is no graceful ramp and rarely an overload-manager warning before the cliff.

Envoy is FD-hungry because each proxied connection consumes roughly two file descriptors: one downstream socket and one upstream socket. Add listen sockets, the admin socket, xDS gRPC streams, active health-check connections, and access-log files, and a busy proxy can hold tens of thousands of FDs open with nothing wrong. The default 1024 ulimit on many base images is therefore far too low for a production proxy and is the most common root cause.

The other trap is hot restart. During a hot restart the old and new Envoy processes run concurrently and both keep their FDs open until drain completes. If you sit at 60% of the limit at baseline, the restart briefly pushes you to 120% and the new process fails to come up. Target below 50% of the ulimit if you rely on hot restart, and below 80% otherwise.

What this means

File descriptors in Linux are the universal handle for sockets, regular files, pipes, and some IPC objects. Envoy treats a socket as an FD for its entire lifetime, from accept or connect until the kernel releases it after close. When the process hits RLIMIT_NOFILE, calls like accept(), socket(), connect(), and open() return EMFILE and the affected subsystem decides what to do.

The failure mode is hard, not graceful. With no overload-manager resource monitor configured, the listener accept loop fails, new upstream connection attempts fail, access-log writes fail, and the proxy stops taking new work while existing streams drain. From outside it looks like a proxy still serving established connections but refusing all new ones, with 503s, resets, and rising upstream_cx_connect_fail counts as upstream dials also fail.

The mental model is the FD budget:

flowchart TD
    A["FD budget = ulimit"] --> B["Fixed FDs:
listen, admin, xDS, logs, HC"]
    A --> C["Downstream + upstream sockets"]
    C --> D["~2 FDs per proxied connection"]
    D --> E{"At ceiling?"}
    E -- No --> F[Healthy]
    E -- Yes --> G["accept/connect EMFILE
new cx fail at once"]
    A --> H["Hot restart: old + new
briefly doubles total"]
    H --> E

Two things make this cliff worse than a generic process hitting its ulimit:

• FD count tracks connections, not requests. HTTP/1.1 keep-alive and HTTP/2 multiplexing hide request volume from connection count. A service that looks idle in RPS can still hold thousands of long-lived downstream sockets. WebSocket and gRPC streams are the classic offenders.
• Hot restart doubles FD usage for the drain window. FDs are passed between the old and new process over a Unix domain socket, but both keep their copies open until the old one finishes draining. A baseline that is safe at steady state can blow the ceiling during restart.

Common causes

Quick checks

These are read-only and safe to run during an incident.

# Identify the Envoy process
ENVOY_PID=$(pgrep -x envoy | head -1)

# Current open FD count
ls /proc/$ENVOY_PID/fd | wc -l

# The real soft and hard limits inside the container
grep 'Max open files' /proc/$ENVOY_PID/limits

# Downstream and upstream active connection counts
curl -s http://localhost:9901/stats | grep -E 'downstream_cx_active|upstream_cx_active'

# Connection rejection counters (silent from client side)
curl -s http://localhost:9901/stats | grep -E 'downstream_cx_(overflow|overload_reject)|downstream_global_cx_overflow'

# Overload manager state (should be 0)
curl -s http://localhost:9901/stats | grep 'overload'

# Hot restart in progress? Non-zero parent_connections means overlap
curl -s http://localhost:9901/stats | grep -E 'parent_connections|hot_restart_epoch|server\.state'

# Top FD consumer types (socket, regular file, pipe, etc.)
ls -l /proc/$ENVOY_PID/fd | awk '{print $NF}' | sed 's/\[.*\]//' | sort | uniq -c | sort -rn | head

If roughly downstream_cx_active + upstream_cx_active times two plus fixed FDs is close to the Max open files value, you are on the cliff. downstream_cx_overflow, downstream_cx_overload_reject, and downstream_global_cx_overflow will be nonzero if Envoy is actively rejecting connections.

How to diagnose it

1. Confirm the real in-container limit. The host may report millions of FDs while the container is capped at 1024. Only /proc/$ENVOY_PID/limits is authoritative. In Kubernetes, check the container runtime defaults and any namespace LimitRange or node-level kubelet constraints.

2. Compute utilization. Divide the live FD count from /proc/$ENVOY_PID/fd \| wc -l by the soft limit. Above 80% warrants investigation; above 85% is page territory. If you hot restart, your real ceiling is half the limit.

3. Classify the FDs. Use ls -l /proc/$ENVOY_PID/fd and bucket by link target type. Socket FDs dominate for a healthy proxy. A large number of regular files points at access-log leaks or opened-but-unclosed resources. A growing number of pipes points at filter-chain issues.

4. Correlate FD growth with connection metrics. If downstream_cx_active plus upstream_cx_active matches FD growth, the cause is real connection load. If FDs grow while connection counts are flat, suspect a leak. Compare downstream_cx_total (new connections) against downstream_cx_destroy (closed connections); a sustained excess of new over destroy with no traffic increase is a leak signature.

5. Check for hot restart overlap. server.parent_connections nonzero means the old process is still draining. server.state of 1 (DRAINING) on one process and 0 (LIVE) on another is expected during a restart, but if both are above 50% of FD capacity the new process will fail.

6. Inspect access-log FDs. Each access-log destination is a long-lived FD. Misconfigured rotation or duplicate per-destination handles can multiply them. A handful is normal; hundreds is not.

7. Estimate health-check FD pressure. For each cluster, active health checks open a short-lived connection per host per interval. Many clusters x many hosts x short intervals adds up. Compare cluster.<name>.upstream_cx_total rates against cluster size.

8. Verify the overload manager. If server.overload_manager.envoy.overload_actions.stop_accepting_connections.active is 1, Envoy is already refusing connections as a protective measure. If no overload actions are configured, you have no early warning before the cliff.

Metrics and signals to monitor

Fixes

Group fixes by cause. Each has tradeoffs.

Raise the limit (most common fix, often correct)

The default 1024 ulimit is wrong for any production proxy. Raise it at every layer that can constrain it.

• systemd unit: add LimitNOFILE=1048576 (or higher) to the [Service] section and reload.

• Container runtime: set --ulimit nofile=1048576:1048576 for Docker, or the equivalent in your containerd/CRI config.

• Kubernetes: there is no native securityContext field for ulimits. Set the limit in the image entrypoint, via an init container using prlimit, or in the container runtime’s pod defaults. Verify the inherited value from /proc/<pid>/limits.

• Kernel: confirm fs.file-max is high enough for the host total. This is almost never the binding constraint, but verify it.

The new hard limit only takes effect after a restart; an already-running process keeps the limits it started with. There is no live way to raise the hard RLIMIT_NOFILE of a running process.

Hold headroom for hot restart

If you hot restart, the operating limit is half of the configured ulimit. Two options:

• Raise the ulimit so steady-state usage stays under 50%.
• Disable hot restart during incidents and cold restart, accepting a brief outage.

Track server.parent_connections during deploys. If it stays nonzero longer than expected, the old process is not draining and FDs are not being released.

Fix connection leaks

If FDs grow while connection counts and traffic are flat, something is opening sockets or files without closing them.

• Verify downstream idle timeouts are set. Long-lived HTTP/1.1 keep-alive connections without an idle timeout inflate downstream_cx_active indefinitely.
• Verify upstream idle timeouts in the cluster config. Connection pools rely on these to recycle idle sockets.
• Confirm WebSocket and gRPC workloads are not accumulating streams without limits. Each stream holds a downstream and an upstream FD.

Reduce health-check FD churn

Active health checks open their own connections per host per interval. With many clusters, many hosts, and short intervals, the FD churn adds up. Increase the interval, reduce the scope of checked hosts, or rely more on outlier detection for hosts that already receive traffic.

Configure the overload manager

Without overload actions, Envoy goes from fine to refusing connections with no early warning. Configure the heap and connection-count resource monitors and a stop_accepting_connections action so Envoy degrades gracefully before hitting the cliff.

Fix access-log FD leaks

If ls -l /proc/<pid>/fd shows many regular files pointing at log paths, investigate per-destination log handles and rotation issues. A correctly configured file access log should be one or a handful of FDs, not hundreds.

Prevention

• Capacity plan around 2 FDs per proxied connection. Use downstream_cx_active + upstream_cx_active times two plus a 10% fixed overhead (listen sockets, xDS, admin, access logs, health checks) as your working estimate.
• Verify the limit inside the container, not on the host. /proc/<pid>/limits is the only authoritative source. Do this at deploy time and after runtime upgrades.
• Hold 50% headroom if you hot restart, 80% otherwise. This is the single most important preventive rule.
• Alert on FD utilization directly. Track /proc/<pid>/fd \| wc -l divided by Max open files, not just connection counts. Connection counts miss non-socket FDs.
• Track server.parent_connections during deploys. Sustained nonzero values during a hot restart are a leading indicator of an FD cliff.
• Review FD count after every cluster, host, or health-check interval change. Many-cluster service meshes with short health-check intervals are a classic slow-burn source of FD growth.
• Bake the ulimit into your base image and runtime config. Do not rely on inheriting the host limit. Container runtime defaults change.

How Netdata helps

• Per-second FD utilization from /proc/<pid>/fd and /proc/<pid>/limits catches the climb before the cliff, including across thousands of sidecars.
• downstream_cx_active and upstream_cx_active correlate with FD growth so you can tell pressure from real load versus a leak.
• downstream_cx_overflow, downstream_cx_overload_reject, and downstream_global_cx_overflow show the moment Envoy starts refusing connections, separate from generic 5xx counts.
• Overload manager action gauges (stop_accepting_connections.active) flag the transition from tight to actively degrading.
• server.parent_connections correlates FD spikes with deploy windows, so you can prove the cliff is restart-induced rather than load-induced.
• ML anomaly detection on FD and connection counts surfaces slow leaks that look normal in absolute terms but deviate from the per-instance baseline.

Related guides

• Envoy 502 and upstream resets: rx_reset, tx_reset, and mid-response failures
• Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend
• Envoy 504 upstream timeout: upstream_rq_timeout, per-try timeouts, and the UT flag
• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy clusters stuck warming: warming_clusters non-zero and routes returning 503
• Envoy connection pool exhaustion: a slow upstream that fills the pool
• Envoy control_plane.connected_state = 0: running on stale xDS config
• Envoy downstream_rq_time high: client-observed latency and proxy overhead
• Envoy health checks vs outlier detection: two systems that eject hosts differently
• How Envoy actually works in production: a mental model for operators
• Envoy listener_create_failure: a listener config Envoy could not apply
• Envoy membership_healthy dropping: reading the single most important cluster signal