NATS file descriptor exhaustion: too many open files and the ulimit cliff

Clients suddenly cannot connect. The server process is alive, CPU is fine, memory is fine, but every new connection fails and the logs are full of too many open files. Existing clients keep working, which makes it look like a network problem until you count sockets.

This is file descriptor exhaustion, and it is a cliff-edge failure. There is no graceful degradation: the moment the process hits its OS ulimit -n, accept() starts failing, JetStream cannot open new storage files, and cluster routes cannot establish. It is one of the most common NATS incidents, because the default ulimit -n of 1024 on Linux is far too low for a production message broker, while the default max_connections in NATS is 65536. The OS limit is the real ceiling, and it is usually the one nobody set.

What this means

Every TCP connection to a NATS server consumes one file descriptor. That includes client connections, cluster routes, gateway connections, and leaf node connections. On top of sockets, the server holds FDs for listener sockets, log files, and, if JetStream is enabled with file storage, message block files (.blk) and index files (.idx) on disk.

Two limits apply, and they are independent:

• max_connections (default 65536) caps client connections at the NATS protocol level. Hitting it produces a clean protocol error: -ERR 'Maximum Connections Exceeded'.
• ulimit -n caps all open files at the OS level. Hitting it produces syscall failures everywhere at once: accept, open, connect.

The nasty part is that routes, gateways, and leaf nodes consume FDs but are not counted against max_connections. Neither are JetStream storage files. So a server can be at 40% of its max_connections and still be pinned against the FD ceiling. With the default ulimit -n of 1024, the server exhausts FDs at roughly 1024 minus listeners, routes, and JetStream file handles, which is far below any max_connections value you would configure for production.

flowchart TD
  A[New connection or file open] --> B{FD available under ulimit -n?}
  B -->|yes| C[Normal operation]
  B -->|no| D[accept fails: too many open files]
  B -->|no| E[JetStream cannot open .blk or .idx files]
  B -->|no| F[Route or gateway connect fails]
  D --> G[New clients rejected, existing clients still work]
  E --> H[Stream writes fail mid-operation]
  F --> I[Cluster partition risk]

Common causes

Quick checks

All of these are read-only and safe to run during an incident.

# 1. Find the server PID and check its actual limits
pidof nats-server
cat /proc/$(pidof nats-server)/limits | grep -i "open files"

The Max open files line shows the soft and hard limits the process is actually running under. This is the number that matters, not what your login shell reports.

# 2. Count currently open FDs
ls /proc/$(pidof nats-server)/fd | wc -l

If this number equals or nearly equals the soft limit, the diagnosis is confirmed.

# 3. See what the FDs are: sockets vs files
lsof -p $(pidof nats-server) | awk 'NR>1 {print $5}' | sort | uniq -c | sort -rn

A high IPv4/IPv6/TCP count is connection-driven. A high REG count on a JetStream server points at storage files.

# 4. Check connection and topology counts from the server itself
curl -s http://localhost:8222/varz | jq '{connections, total_connections, max_connections}'

Compare connections plus route and leaf node counts against the FD count from step 2. The gap is JetStream files, listeners, and other handles. Active route and leaf node detail comes from /routez and /leafz; /varz also exposes aggregate routes and leafnodes counts.

curl -s http://localhost:8222/routez | jq '.num_routes'
curl -s http://localhost:8222/leafz | jq '.leafnodes'

# 5. Confirm the failure in the logs
grep -i "too many open files" /var/log/nats/nats-server.log | tail -20
# or, under systemd:
journalctl -u nats-server --since "1 hour ago" | grep -i "too many open files"

On a JetStream server you may see errors opening storage files, for example failing to open a stream’s msgs/*.idx file, alongside accept errors. That dual failure mode is the signature of FD exhaustion rather than a plain connection limit.

# 6. If systemd manages the service, check the unit
systemctl cat nats-server | grep -i LimitNOFILE

How to diagnose it

1. Confirm the process limit, not the shell limit. Run check 1 above. ulimit -n in your SSH session reflects your PAM session, which is irrelevant to a daemon started by systemd. The only limit that matters is the one in /proc/<pid>/limits.

2. Confirm you are at the ceiling. Compare open FD count (check 2) against the soft limit. FD exhaustion is unambiguous: the count is at the limit and the logs show too many open files. If the count is well below the limit, this is not your problem; look at NATS Maximum Connections Exceeded or NATS server not responding instead.

3. Attribute the FDs. Use check 3 to split sockets from regular files. If sockets dominate and connections in /varz is high, this is connection volume: a leak, a reconnect storm, or genuine load. If regular files dominate on a JetStream server, storage files are the pressure and connection tuning alone will not save you.

4. Identify connection churn. A stable connections value with a fast-growing total_connections counter means clients are connecting and disconnecting at a high rate. Each reconnect briefly holds an FD, and a storm after a network event can push a borderline server over the edge.

5. Check the cluster surface area. In a full-mesh cluster of N servers each server holds N-1 route FDs, plus gateway and leaf node connections if configured. These are small numbers, but they are exactly the FDs you cannot afford to lose: when FDs run out, routes cannot re-establish and the cluster partitions on top of the client-facing failure.

6. Decide whether the limit is wrong or the usage is wrong. If FD usage matches legitimate load, the limit is undersized and the fix is raising it. If usage is driven by a client leak or churn loop, raising the limit buys time but the leak wins eventually. Fix both, but size the limit for legitimate peak first.

Metrics and signals to monitor

During FD exhaustion the health endpoint can become unreachable even though the process is running, because accepting the HTTP connection also needs a descriptor. A health probe failure combined with live existing connections is a strong hint to check FDs before assuming a hung process. See NATS /healthz explained for the probe semantics.

Fixes

Raise the limit for the running deployment

The correct limit for a production NATS server is at least 65536, and a practical rule of thumb is more precise:

ulimit -n >= max_connections + (stream_count * ~10) + 256

The stream_count * ~10 term covers JetStream file storage handles per stream and is an estimate; the real number depends on storage configuration and block layout, so test under load and measure. The 256 covers listeners, log files, and miscellaneous handles. If the computed value is under 65536, use 65536 anyway.

systemd

/etc/security/limits.conf does not apply to systemd services; it only affects PAM login sessions. Set the limit in the unit or a drop-in:

# /etc/systemd/system/nats-server.service.d/fd-limit.conf
[Service]
LimitNOFILE=65536

Applying this requires a service restart, which is disruptive: in a cluster, do it as a rolling restart one node at a time and let routes re-establish between nodes.

systemctl daemon-reload
systemctl restart nats-server   # disruptive; roll through the cluster

Verify afterwards with /proc/$(pidof nats-server)/limits.

Docker

Containers get their nofile limit from the container runtime, not from the host’s ulimit -n or limits.conf. The runtime default depends on the engine, daemon configuration, and version; there is no single value you can assume matches the host. Set it explicitly per container:

docker run --ulimit nofile=65536:65536 ...

or the ulimits.nofile equivalent in Compose. Under Kubernetes, ulimit control depends on the runtime and is a common silent failure point: verify the running process limit from inside the pod rather than trusting the manifest.

Interactive or init-script deployments

ulimit -n 65536

set before launching the server, or via the init script. The soft limit cannot exceed the hard limit, and unprivileged users cannot raise hard limits above what is configured for them.

Reduce FD consumption

If usage is the problem rather than the limit:

• Fix client connection leaks. Clients that open a new connection per operation instead of reusing one will outgrow any limit. Track them via /connz by source IP.
• Dampen reconnect storms. Client libraries with aggressive, unsynchronized retry loops amplify a network blip into an FD-consuming storm. Jittered backoff on the client side is the fix.
• Review JetStream stream count. Many small file-backed streams each hold storage FDs. Consolidating streams reduces the file-handle baseline; the actual per-stream handle count is implementation- and workload-dependent, so measure it on a representative server.

Prevention

• Set the limit at provisioning time, not after the first incident. Every NATS deployment artifact (unit file, container spec, Helm values) should carry an explicit nofile limit of at least 65536. Never rely on OS defaults.
• Alert on FD utilization, not just connection count. Track open FDs as a ratio of the soft limit and ticket above 70%. The degradation curve is cliff-edge, so the leading indicator is all you get.
• Size the limit against the formula, then test under load. Load-test with realistic connection counts plus JetStream traffic and measure actual FD usage. Assumptions about per-stream file handles are where the formula and reality diverge.
• Watch churn, not just levels. A fast-growing total_connections with flat connections is a leak or retry loop burning FDs in transit. Catch it before the level matters.
• Include FD headroom in capacity reviews. When you raise max_connections, raise ulimit -n in the same change. Treating them as separate knobs is how this incident happens.

How Netdata helps

• Netdata’s NATS collector polls /varz and charts connections against max_connections, plus total_connections, so connection pressure and churn are visible on one dashboard next to the process metrics that reveal FD pressure.
• Because Netdata also collects per-process OS metrics, you can correlate NATS connection counts with the nats-server process’s open file descriptors on the same timeline, which is the correlation that confirms this diagnosis.
• Per-second granularity catches reconnect storms: the total_connections rate spike after a network event is visible as it happens, not averaged away at a 60-second scrape.
• Uptime and /healthz tracking lets you distinguish “server down” from “server alive but refusing connections,” which is the FD exhaustion signature.
• Alerting on connection utilization ratios alongside OS-level FD alerts covers both ceilings, since either can be hit first.

Related guides

• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check
• How NATS actually works in production: a mental model for operators
• NATS Maximum Connections Exceeded: new clients rejected at the max_connections wall
• NATS monitoring checklist: the signals every production server needs
• NATS monitoring maturity model: from survival to expert
• NATS pending bytes growing: catching a slow consumer before it is disconnected
• NATS server not responding: healthz failing and the process down or hung
• NATS slow consumer breakdown: clients vs routes vs gateways and blast radius
• NATS slow consumer detected: the write buffer overflowed and messages were dropped
• NATS stalled clients and stale connections: half-dead sockets and write-path distress