When a Consul server hits its file descriptor ceiling, every consumer that needs a new FD fails immediately: RPC connections are refused, DNS listeners stop accepting queries, xDS streams to Envoy sidecars cannot open, and outbound health-check connections fail. Gossip probes time out and the failure detection protocol marks otherwise healthy peers as suspect or failed. A healthy-looking cluster a minute ago is now generating cascading pages.

File descriptor exhaustion is not graceful degradation. Each socket, pipe, and file handle counts against a hard kernel limit. Once that limit is reached, every syscall that creates a new FD (accept, socket, open, pipe, epoll_create) returns EMFILE, regardless of available CPU, memory, or network capacity. A consul reload either fails outright (it needs FDs to read config and re-open listeners) or succeeds but leaves the server half-functional. Recovery requires raising the limit and restarting the process.

The default Linux ulimit -n of 1024 is catastrophically low for any Consul server. HashiCorp recommends a minimum of 65536 (LimitNOFILE=65536 in the systemd unit). Servers open FDs for every RPC client connection, gRPC stream, DNS listener, outbound health check, watch handler, and Raft peer connection. A cluster with a few hundred client agents and Connect enabled can legitimately sit at tens of thousands of FDs.

How it fails

The kernel bounds each process’s file descriptor table by a soft limit (what ulimit -n reports) and a hard limit (an upper bound the process can raise into). When the soft limit is hit, Go programs like Consul surface the errors as “too many open files” in logs and socket: too many open files in dial errors.

A server provisioned without explicit LimitNOFILE configuration will run for weeks at a few hundred FDs, then tip over when a deployment adds client agents, a Connect sidecar rollout begins, or a blocking-query leak accumulates past the threshold. Anything below 65536 on a server is a latent incident.

flowchart TD
  A[FD count hits ulimit] --> B[New RPC, DNS, xDS refused]
  A --> C[Outbound health checks fail]
  A --> D[Gossip probes time out]
  B --> E[Client agents cannot sync catalog]
  C --> F[Checks go stale]
  D --> G[Peers marked suspect or failed]
  E --> H[Silent catalog staleness]
  G --> I[False failure detection cascades]
  F --> I
  I --> J[Raft leader churn risk]
  J --> K[Write outage]

Common causes

CauseWhat it looks likeFirst thing to check
Default ulimit still in effectFirst “too many open files” page fires at a few hundred to a thousand FDs/proc/$(pgrep consul)/limits
Connection or stream leakFD count trends upward monotonically, never drops during quiet periodsconsul_runtime_sys_fd_used over 24h
Blocking query or watch accumulationGoroutine count climbs in lockstep with FDsconsul_runtime_num_goroutines vs FD count
consul reload leakStepwise FD growth aligned with reload eventsCount reloads vs FD deltas
consul-template idle HTTP connectionsSteady FD growth on agents running many templateslsof on the consul-template process
xDS stream cap (not FD)Error mentions “xDS streams” not “open files”Consul version and Connect topology
K8s cgroup limit hitContainer restarts or fails inside a node with a higher ulimit/proc/<pid>/limits inside the container and the pod spec

The last two rows are not FD exhaustion but produce symptoms operators frequently misread as FD exhaustion. The xDS stream cap introduced in Consul 1.14 (ResourceExhausted: this server has too many xDS streams open) is a per-server application-level limiter, not a kernel FD limit. The fix shipped in 1.15.0 (PR #15789) for locally-registered proxies. For consul-dataplane against external servers, the common root cause is a TLS SNI mismatch causing gRPC load-balancing failure, fixed by setting tlsServerName in the externalServers stanza.

Quick checks

Read-only and safe on a live server:

# Effective soft and hard limits.
# In Kubernetes, run from inside the container. `ulimit -n` from a host
# shell may report the node value, not the cgroup-imposed limit.
cat /proc/$(pgrep consul)/limits | grep "Max open files"

# Current open FD count
ls /proc/$(pgrep consul)/fd | wc -l

# FD usage and limit from Consul telemetry
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -E "consul_runtime_sys_fd_used|consul_runtime_sys_fd_limit"

# FD breakdown by type (IPv4, IPv6, sock, pipe, anon_inode)
lsof -p $(pgrep consul) -n | awk '{print $5}' | sort | uniq -c | sort -rn

# Established TCP connections to the RPC port (8300)
ss -tnp state established '( sport = :8300 )' | wc -l

# Canonical error in logs
journalctl -u consul --since '1 hour ago' | grep -i "too many open files"

# Goroutine count alongside FDs (correlation hints at a leak)
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep consul_runtime_num_goroutines

How to diagnose it

  1. Confirm the limit and current count. Pull consul_runtime_sys_fd_used and consul_runtime_sys_fd_limit from telemetry, or read /proc/<pid>/fd and /proc/<pid>/limits directly. Compute the ratio: PAGE above 90%, TICKET above 70%.

  2. Categorize the FDs. Run the lsof breakdown above. A high sock or IPv4 count points at network connections (RPC, xDS, gossip). A high pipe or anon_inode count suggests eventfds and epoll handles, often correlated with goroutine count.

  3. Determine whether this is a leak. Plot consul_runtime_sys_fd_used over 24 to 72 hours. Slow, non-reversing growth that does not correlate with cluster activity is a connection or stream leak. Stepwise growth aligned with consul reload events points at the reload leak (issue #3018).

  4. Correlate with goroutines. If consul_runtime_num_goroutines climbs in lockstep with FDs, the leak is in a code path that opens both a goroutine and a connection. Blocking queries and Connect xDS streams are the usual suspects.

  5. Check gossip and Raft side effects. Watch consul_serf_lan_member_status for suspect entries and consul_raft_last_contact for upward drift. FD exhaustion often surfaces here first because gossip probes and Raft heartbeats fail to allocate sockets before the leader fails to commit.

  6. If Connect is enabled, distinguish FD exhaustion from the xDS stream cap. A ResourceExhausted gRPC error mentioning “xDS streams” is the 1.14+ stream limiter, not the kernel FD limit. Confirm by checking Consul version and proxy topology.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
consul_runtime_sys_fd_usedAbsolute FD consumptionMonotonic growth over hours with no load change
consul_runtime_sys_fd_limitThe ceiling the process will hitBelow 65536 on a server is misconfigured
consul_runtime_sys_fd_used / fd_limitHeadroom ratioPAGE above 90%, TICKET above 70%
consul_runtime_num_goroutinesProxy for concurrent connections and streamsClimbing in lockstep with FDs indicates a leak
consul_xds_server_streamsActive xDS streams to Envoy sidecarsUnexpected growth signals reconnection churn
consul_serf_lan_member_statusGossip healthSpike in suspect members hints at FD pressure
consul_raft_last_contactFollower-to-leader reachabilitySustained upward drift risks election timeout
consul_client_rpc_failedAgent-to-server RPC healthSustained non-zero on multiple agents means server is rejecting connections

Fixes

Raise the limit (immediate relief)

Add LimitNOFILE=65536 to the [Service] section of the Consul systemd unit via a drop-in override:

systemctl edit consul

[Service]
LimitNOFILE=65536

systemctl daemon-reload
systemctl restart consul

Warning: systemctl restart consul is disruptive. A restart of the leader causes a brief Raft election and write outage. If this is the leader, drain first or perform the restart during a maintenance window. The restart is the recovery action; raising the limit prevents recurrence.

In Kubernetes, raise the limit through the pod spec or container runtime. Check both the node-level ulimit and the cgroup-imposed limit, since ulimit -n inside a container can report the node value rather than the binding cgroup ceiling.

Stop the leak (longer term)

If FD growth is monotonic, the higher limit only buys time. Capture a goroutine dump during the incident:

curl -s http://127.0.0.1:8500/debug/pprof/goroutine?debug=1 > consul-goroutines.txt

Look for stack traces concentrated in RPC accept loops, blocking query handlers, or xDS stream management. The dominant stack identifies the leaking subsystem.

For consul reload-induced leaks, reduce reload frequency. Watch configurations accumulate handlers across reloads (issue #3018). Consider templating configurations at deploy time rather than relying on live reloads.

For consul-template, watch for idle HTTP connection accumulation (consul-template issue #591). A periodic bounce of consul-template, or upgrading to a version with the fix, addresses the slow leak.

Address the xDS stream cap separately

If the error mentions xDS streams rather than open files, raising the FD limit does not help. The relevant fixes:

  • For Consul 1.14.x with locally-registered proxy services, upgrade to 1.15.0 or later, which disables the stream limiter for locally-registered proxies (PR #15789).
  • For consul-dataplane against external servers, verify tlsServerName in the externalServers stanza. A TLS SNI mismatch causes gRPC load-balancing failure and stream multiplication.

Prevention

  • Set LimitNOFILE=65536 on every server. Anything lower is a misconfiguration that will fail under load.
  • Monitor the FD ratio with low thresholds. PAGE above 90%, TICKET above 70%.
  • Trend FD usage over days. Slow monotonic growth is the leak signature. Catching it at 30% utilization is cheaper than catching it at 90%.
  • Track goroutine count alongside FDs. They move together for connection-driven leaks.
  • Check Kubernetes limits independently. The node ulimit is not the value that binds the pod.
  • Limit consul reload frequency. Each reload is a candidate leak source.
  • Validate Consul version behavior for Connect. The xDS stream limiter behavior changed in 1.14 and was refined in 1.15.

How Netdata helps

  • Per-second consul_runtime_sys_fd_used and consul_runtime_sys_fd_limit from Consul telemetry, surfaced as a ratio so the cliff edge is visible at a glance.
  • ML-based anomaly detection on monotonic FD growth catches leaks days before the kernel limit is hit. Threshold alerts at 70% and 90% provide operational tripwires.
  • Correlation of FD utilization with consul_runtime_num_goroutines distinguishes connection-driven leaks from file-handle or pipe leaks.
  • Per-process file descriptor charts, broken down by type, show whether growth is sockets, pipes, or anon_inode.
  • Cross-signal correlation with consul_serf_lan_member_status, consul_raft_last_contact, and consul_client_rpc_failed shows gossip and Raft side effects in the same view.
  • Side-by-side comparison across all Consul servers makes it obvious when one server accumulates FDs faster than its peers.