When Envoy runs out of file descriptors, it is not a slow degradation. At the RLIMIT_NOFILE ceiling every new downstream accept() and every new upstream connect() fails at once. Clients see resets, timeouts, or 503s, and the proxy looks dead even though established streams keep flowing. Unlike memory or CPU pressure, there is no graceful ramp and rarely an overload-manager warning before the cliff.

Envoy is FD-hungry because each proxied connection consumes roughly two file descriptors: one downstream socket and one upstream socket. Add listen sockets, the admin socket, xDS gRPC streams, active health-check connections, and access-log files, and a busy proxy can hold tens of thousands of FDs open with nothing wrong. The default 1024 ulimit on many base images is therefore far too low for a production proxy and is the most common root cause.

The other trap is hot restart. During a hot restart the old and new Envoy processes run concurrently and both keep their FDs open until drain completes. If you sit at 60% of the limit at baseline, the restart briefly pushes you to 120% and the new process fails to come up. Target below 50% of the ulimit if you rely on hot restart, and below 80% otherwise.

What this means

File descriptors in Linux are the universal handle for sockets, regular files, pipes, and some IPC objects. Envoy treats a socket as an FD for its entire lifetime, from accept or connect until the kernel releases it after close. When the process hits RLIMIT_NOFILE, calls like accept(), socket(), connect(), and open() return EMFILE and the affected subsystem decides what to do.

The failure mode is hard, not graceful. With no overload-manager resource monitor configured, the listener accept loop fails, new upstream connection attempts fail, access-log writes fail, and the proxy stops taking new work while existing streams drain. From outside it looks like a proxy still serving established connections but refusing all new ones, with 503s, resets, and rising upstream_cx_connect_fail counts as upstream dials also fail.

The mental model is the FD budget:

flowchart TD
    A["FD budget = ulimit"] --> B["Fixed FDs:
listen, admin, xDS, logs, HC"]
    A --> C["Downstream + upstream sockets"]
    C --> D["~2 FDs per proxied connection"]
    D --> E{"At ceiling?"}
    E -- No --> F[Healthy]
    E -- Yes --> G["accept/connect EMFILE
new cx fail at once"]
    A --> H["Hot restart: old + new
briefly doubles total"]
    H --> E

Two things make this cliff worse than a generic process hitting its ulimit:

  • FD count tracks connections, not requests. HTTP/1.1 keep-alive and HTTP/2 multiplexing hide request volume from connection count. A service that looks idle in RPS can still hold thousands of long-lived downstream sockets. WebSocket and gRPC streams are the classic offenders.
  • Hot restart doubles FD usage for the drain window. FDs are passed between the old and new process over a Unix domain socket, but both keep their copies open until the old one finishes draining. A baseline that is safe at steady state can blow the ceiling during restart.

Common causes

CauseWhat it looks likeFirst thing to check
Default ulimit too low (1024)Refused connections or crash at modest connection counts; fresh deployments or after runtime changesgrep 'Max open files' /proc/<pid>/limits
Connection leakFD count climbs monotonically without traffic increase; downstream clients not closing, idle timeouts missing or too longCompare downstream_cx_total rate vs downstream_cx_destroy rate
Hot restart ceilingNew process fails to come up during deploy; old process holds FDs during drainserver.parent_connections nonzero while FD count near limit
Access-log FD leakFDs accumulate even at low connection count; many regular-file FDs pointing at log pathsls -l /proc/<pid>/fd | grep -c <log_path>
Excessive health-check connectionsMany clusters x many hosts x short intervals; connections not pooledcluster.<name>.upstream_cx_total rate vs cluster size and interval
WebSocket or gRPC stream accumulationLong-lived streams hold 2 FDs each; traffic looks low but FD count highdownstream_cx_active steady or climbing while request rate is flat
Container runtime capped the soft limitSudden failure after image or runtime update; /proc/<pid>/limits shows 1024 even though host allows moreVerify inside the container, not on the host

Quick checks

These are read-only and safe to run during an incident.

# Identify the Envoy process
ENVOY_PID=$(pgrep -x envoy | head -1)

# Current open FD count
ls /proc/$ENVOY_PID/fd | wc -l

# The real soft and hard limits inside the container
grep 'Max open files' /proc/$ENVOY_PID/limits

# Downstream and upstream active connection counts
curl -s http://localhost:9901/stats | grep -E 'downstream_cx_active|upstream_cx_active'

# Connection rejection counters (silent from client side)
curl -s http://localhost:9901/stats | grep -E 'downstream_cx_(overflow|overload_reject)|downstream_global_cx_overflow'

# Overload manager state (should be 0)
curl -s http://localhost:9901/stats | grep 'overload'

# Hot restart in progress? Non-zero parent_connections means overlap
curl -s http://localhost:9901/stats | grep -E 'parent_connections|hot_restart_epoch|server\.state'

# Top FD consumer types (socket, regular file, pipe, etc.)
ls -l /proc/$ENVOY_PID/fd | awk '{print $NF}' | sed 's/\[.*\]//' | sort | uniq -c | sort -rn | head

If roughly downstream_cx_active + upstream_cx_active times two plus fixed FDs is close to the Max open files value, you are on the cliff. downstream_cx_overflow, downstream_cx_overload_reject, and downstream_global_cx_overflow will be nonzero if Envoy is actively rejecting connections.

How to diagnose it

  1. Confirm the real in-container limit. The host may report millions of FDs while the container is capped at 1024. Only /proc/$ENVOY_PID/limits is authoritative. In Kubernetes, check the container runtime defaults and any namespace LimitRange or node-level kubelet constraints.
  2. Compute utilization. Divide the live FD count from /proc/$ENVOY_PID/fd \| wc -l by the soft limit. Above 80% warrants investigation; above 85% is page territory. If you hot restart, your real ceiling is half the limit.
  3. Classify the FDs. Use ls -l /proc/$ENVOY_PID/fd and bucket by link target type. Socket FDs dominate for a healthy proxy. A large number of regular files points at access-log leaks or opened-but-unclosed resources. A growing number of pipes points at filter-chain issues.
  4. Correlate FD growth with connection metrics. If downstream_cx_active plus upstream_cx_active matches FD growth, the cause is real connection load. If FDs grow while connection counts are flat, suspect a leak. Compare downstream_cx_total (new connections) against downstream_cx_destroy (closed connections); a sustained excess of new over destroy with no traffic increase is a leak signature.
  5. Check for hot restart overlap. server.parent_connections nonzero means the old process is still draining. server.state of 1 (DRAINING) on one process and 0 (LIVE) on another is expected during a restart, but if both are above 50% of FD capacity the new process will fail.
  6. Inspect access-log FDs. Each access-log destination is a long-lived FD. Misconfigured rotation or duplicate per-destination handles can multiply them. A handful is normal; hundreds is not.
  7. Estimate health-check FD pressure. For each cluster, active health checks open a short-lived connection per host per interval. Many clusters x many hosts x short intervals adds up. Compare cluster.<name>.upstream_cx_total rates against cluster size.
  8. Verify the overload manager. If server.overload_manager.envoy.overload_actions.stop_accepting_connections.active is 1, Envoy is already refusing connections as a protective measure. If no overload actions are configured, you have no early warning before the cliff.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
/proc/<pid>/fd count vs Max open filesDirect utilization of the FD budget>80% (or >50% with hot restart)
downstream_cx_active + upstream_cx_activeProxy for FD usage: roughly 2 FDs per connectionGrowth without traffic increase
listener.<addr>.downstream_cx_activeDownstream socket count per listenerSteady growth without matching traffic
cluster.<name>.upstream_cx_activeUpstream pool size per cluster, per workerTrending toward max_connections
listener.<addr>.downstream_cx_overflowPer-listener connection limit trippedAny sustained nonzero rate
listener.<addr>.downstream_cx_overload_rejectOverload manager refusing connectionsAny nonzero value
cluster.<name>.upstream_cx_connect_failNew upstream dials failing, including EMFILESudden spike across many clusters
server.overload_manager.*.activeSelf-protection mode engagedstop_accepting_connections = 1
server.parent_connectionsHot restart overlap, FDs doubledSustained nonzero with high FD usage
Kernel fs.file-max and per-process nofileAll layers must be sufficientContainer capped below host

Fixes

Group fixes by cause. Each has tradeoffs.

Raise the limit (most common fix, often correct)

The default 1024 ulimit is wrong for any production proxy. Raise it at every layer that can constrain it.

  • systemd unit: add LimitNOFILE=1048576 (or higher) to the [Service] section and reload.
  • Container runtime: set --ulimit nofile=1048576:1048576 for Docker, or the equivalent in your containerd/CRI config.
  • Kubernetes: there is no native securityContext field for ulimits. Set the limit in the image entrypoint, via an init container using prlimit, or in the container runtime’s pod defaults. Verify the inherited value from /proc/<pid>/limits.
  • Kernel: confirm fs.file-max is high enough for the host total. This is almost never the binding constraint, but verify it.

The new hard limit only takes effect after a restart; an already-running process keeps the limits it started with. There is no live way to raise the hard RLIMIT_NOFILE of a running process.

Hold headroom for hot restart

If you hot restart, the operating limit is half of the configured ulimit. Two options:

  • Raise the ulimit so steady-state usage stays under 50%.
  • Disable hot restart during incidents and cold restart, accepting a brief outage.

Track server.parent_connections during deploys. If it stays nonzero longer than expected, the old process is not draining and FDs are not being released.

Fix connection leaks

If FDs grow while connection counts and traffic are flat, something is opening sockets or files without closing them.

  • Verify downstream idle timeouts are set. Long-lived HTTP/1.1 keep-alive connections without an idle timeout inflate downstream_cx_active indefinitely.
  • Verify upstream idle timeouts in the cluster config. Connection pools rely on these to recycle idle sockets.
  • Confirm WebSocket and gRPC workloads are not accumulating streams without limits. Each stream holds a downstream and an upstream FD.

Reduce health-check FD churn

Active health checks open their own connections per host per interval. With many clusters, many hosts, and short intervals, the FD churn adds up. Increase the interval, reduce the scope of checked hosts, or rely more on outlier detection for hosts that already receive traffic.

Configure the overload manager

Without overload actions, Envoy goes from fine to refusing connections with no early warning. Configure the heap and connection-count resource monitors and a stop_accepting_connections action so Envoy degrades gracefully before hitting the cliff.

Fix access-log FD leaks

If ls -l /proc/<pid>/fd shows many regular files pointing at log paths, investigate per-destination log handles and rotation issues. A correctly configured file access log should be one or a handful of FDs, not hundreds.

Prevention

  • Capacity plan around 2 FDs per proxied connection. Use downstream_cx_active + upstream_cx_active times two plus a 10% fixed overhead (listen sockets, xDS, admin, access logs, health checks) as your working estimate.
  • Verify the limit inside the container, not on the host. /proc/<pid>/limits is the only authoritative source. Do this at deploy time and after runtime upgrades.
  • Hold 50% headroom if you hot restart, 80% otherwise. This is the single most important preventive rule.
  • Alert on FD utilization directly. Track /proc/<pid>/fd \| wc -l divided by Max open files, not just connection counts. Connection counts miss non-socket FDs.
  • Track server.parent_connections during deploys. Sustained nonzero values during a hot restart are a leading indicator of an FD cliff.
  • Review FD count after every cluster, host, or health-check interval change. Many-cluster service meshes with short health-check intervals are a classic slow-burn source of FD growth.
  • Bake the ulimit into your base image and runtime config. Do not rely on inheriting the host limit. Container runtime defaults change.

How Netdata helps

  • Per-second FD utilization from /proc/<pid>/fd and /proc/<pid>/limits catches the climb before the cliff, including across thousands of sidecars.
  • downstream_cx_active and upstream_cx_active correlate with FD growth so you can tell pressure from real load versus a leak.
  • downstream_cx_overflow, downstream_cx_overload_reject, and downstream_global_cx_overflow show the moment Envoy starts refusing connections, separate from generic 5xx counts.
  • Overload manager action gauges (stop_accepting_connections.active) flag the transition from tight to actively degrading.
  • server.parent_connections correlates FD spikes with deploy windows, so you can prove the cliff is restart-induced rather than load-induced.
  • ML anomaly detection on FD and connection counts surfaces slow leaks that look normal in absolute terms but deviate from the per-instance baseline.