New clients cannot connect to your NATS server. Existing connections keep working. Client logs show the protocol error -ERR 'Maximum Connections Exceeded' and then the connection closes. The server itself looks healthy: /healthz returns ok, messages still flow for connected clients, and nothing crashed.

This is the max_connections wall. The server has reached its configured connection limit and is rejecting every new connection during the handshake. There is no queueing and no graceful degradation. One slot short of the limit, everything works. At the limit, every new client is turned away.

The default limit is 65536, but many deployments hit the wall much earlier because an operator set a lower value, an account-level limit subdivides the global cap, or the OS file descriptor ceiling runs out before NATS ever gets there. This guide covers how the limit works, how to tell which ceiling you actually hit, and how to find the clients consuming the slots.

What this means

NATS tracks active client connections per server and compares the count to max_connections on every new connection attempt. When the count reaches the limit, the server accepts the TCP connection, sends -ERR 'Maximum Connections Exceeded' during the handshake, and closes the socket. From the client’s perspective the connection fails immediately; everything already connected is unaffected.

Two limits can reject a connection independently:

  • Server-level max_connections. The global cap for the server. Default 65536.
  • Account-level connection limits. Accounts can carry their own connection limit. A connection is rejected if either the server-level limit or the account-level limit is reached, so one account can hit its wall while the server has thousands of free slots.

There is also a third, silent ceiling: the OS file descriptor limit. Every connection consumes one file descriptor, plus routes, gateways, leaf nodes, JetStream store files, and listener sockets. If ulimit -n for the nats-server process is lower than max_connections, the OS limit bites first and the symptom is similar: new connections fail while existing ones work. This is a cliff-edge failure with no graceful degradation, and default ulimit -n values (often 1024) are catastrophically low for a busy NATS server.

flowchart LR
  C[New client TCP connect] --> D{Under server and account limits?}
  D -- yes --> OK[Handshake completes, client connected]
  D -- no --> ERR["-ERR Maximum Connections Exceeded, socket closed"]
  F[Connection leak or reconnect storm] -. consumes slots .-> D
  G[OS ulimit -n exhausted] -. blocks accept .-> ERR

One subtlety on what counts toward the limit: routes and gateways are internal inter-server connections, and leaf node connections carry multiplexed edge traffic. Whether each type counts against max_connections differs by connection type . Regardless of how they count against max_connections, every one of them consumes file descriptors, so FD exhaustion arithmetic must include routes, gateways, and leaf nodes.

Common causes

CauseWhat it looks likeFirst thing to check
Connection leak in a client applicationconnections climbs monotonically over hours or days; total_connections grows faster than expectedCompare connections vs total_connections on /varz; inspect /connz for old, idle connections
Reconnection stormSharp spike in connections after a network event, deploy, or server restart; CPU spike from TLS handshakestotal_connections delta over the last minutes; correlate with uptime and recent network events
Client pool misconfigurationConnection count is a suspicious round multiple of the number of app instancesCount connections per client IP or name in /connz
Account-level limit reachedOnly one account’s clients are rejected; other accounts connect finePer-account stats via /accstatz against the account’s configured limit
OS file descriptor exhaustionRejections start well below max_connections; accept errors in server logscat /proc/$(pidof nats-server)/limits and count open FDs
Crashed clients holding stale slotsConnections with no traffic that never cleanly closed/varz stale_connections field

Quick checks

All of these are read-only. The monitoring port defaults to 8222.

# Current connections, lifetime total, and configured limit
curl -s http://localhost:8222/varz | jq '{active: .connections, total: .total_connections, max: .max_connections}'

# Utilization as a percentage of the configured limit
curl -s http://localhost:8222/varz | jq '{utilization_pct: ((.connections / .max_connections) * 100)}'

# Stale connections and stalled clients (dead or distressed slots)
curl -s http://localhost:8222/varz | jq '{stale_connections, stalled_clients}'

# Who is holding connections: group by client IP
curl -s "http://localhost:8222/connz?limit=1000" | jq '[.connections[].ip] | group_by(.) | map({ip: .[0], count: length}) | sort_by(-.count) | .[:10]'

# OS file descriptor ceiling for the running server process
cat /proc/$(pidof nats-server)/limits | grep -i "open files"

# Approximate current FD usage of the process
ls /proc/$(pidof nats-server)/fd | wc -l

# Look for accept failures in the server log (adjust path to your logging setup)
grep -i "accept" /var/log/nats/nats-server.log | tail -20
# Or, under systemd:
journalctl -u nats-server --since "30 min ago" | grep -i "accept"

A note on /connz at scale: on servers with many thousands of connections, full /connz scrapes are expensive. Use limit and offset parameters and avoid tight polling loops against it during an incident.

How to diagnose it

  1. Confirm which wall you hit. Pull /varz and compare connections to max_connections. If connections equals max_connections, you hit the NATS limit. If connections is well below max_connections but new clients are still rejected, suspect the OS file descriptor ceiling or an account-level limit.

  2. Rule in or out the FD ceiling. Compare the process Max open files limit against the count of open FDs. The FD budget includes routes, gateways, leaf nodes, JetStream store files, and listener sockets, not just client connections. If the FD count is at or near the limit, this is your wall regardless of what max_connections says. Production NATS servers should run with at least 65536 open files; many teams hit this cliff on their first traffic spike.

  3. Determine whether growth is gradual or sudden. Compute the delta of total_connections over a few minutes. A fast-climbing total_connections with a stable or spiking connections means churn: clients connecting and disconnecting rapidly, which points to a reconnection storm or a crash-looping client that reconnects on every restart. Slow, steady growth in connections with modest churn points to a leak.

  4. Identify who holds the slots. Use /connz grouped by IP, client name, or account to find concentration. A leak usually shows as many long-lived connections from one application or host. A storm shows as many fresh connections with short uptime.

  5. Check for stale slots. Non-zero stale_connections in /varz means clients that failed ping/pong health checks but still occupy slots. Clients that crash without a clean close leave the TCP connection open until the server’s stale detection fires, so a fleet of crashing clients can hold a large number of dead slots during the detection window.

  6. Check account-level limits if only some clients fail. If clients in one account are rejected while others connect normally, compare that account’s usage from /accstatz against its configured connection limit. Per-account limits are cluster-global, not per-server .

  7. Correlate with the trigger. A spike that lines up with a deploy, a network partition healing, or a server restart is a reconnection storm. A slow climb with no external event is a leak. Check uptime on /varz and your deploy timeline before blaming the clients.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
connections / max_connections ratioThe only correct way to alert on this wall; absolute counts break across deployment sizes>80% warn, >95% critical
total_connections rate (churn)Reveals reconnect storms and crash-looping clients that a stable connections count hidesRapid growth while connections is flat or spiking
Process FD usage vs ulimit -nThe OS ceiling often bites before max_connections doesFD count >70% of the open-files limit
stale_connectionsDead clients occupying real slotsAny non-zero value sustained >5 minutes
slow_consumers rateSlow consumer disconnects trigger reconnect cycles that churn connection slotsPositive rate alongside rising total_connections
mem (RSS)Each connection costs memory for buffers and goroutines; growth foreshadows the wallMonotonic growth tracking connection growth
Per-account connections (/accstatz)Account limits reject before the server limit doesAccount usage >80% of its configured limit

Alert on the ratio connections / max_connections, never an absolute count. “Alert when connections > 1000” breaks as soon as the limit or the environment changes; “alert when connections exceed 80% of max_connections” survives both. Apply the same ratio thinking to the FD ceiling.

Fixes

Stop the bleeding: free or reclaim slots

If clients are leaking, restarting the offending application releases its slots immediately. This is disruptive to that application but does not touch the NATS server or its healthy clients. Do not restart the NATS server to fix a client-side leak; the slots refill as soon as the leaking clients reconnect.

If the pressure is a reconnection storm, the server usually recovers on its own once clients settle. Check that the storm is decaying (total_connections rate falling) before intervening. If it is cyclical, the underlying cause (flapping load balancer, DNS failure, network device) is still active and that is where to focus.

Raise the limit, carefully

max_connections is a configuration value. Increasing it is legitimate when the current value is simply too small for real load, but it only moves the wall. Before raising it:

  • Verify the FD ceiling first. Raising max_connections above the process ulimit -n is pointless and gives false confidence. Set the open-files limit high enough to cover max_connections plus routes, gateways, leaf nodes, JetStream files, and headroom.
  • Budget the memory. Every connection carries per-connection buffers and goroutines. A large limit on a small host trades connection rejections for memory pressure.

NATS supports runtime configuration reload, and max_connections is among the reloadable settings, so you can raise it without disconnecting existing clients . Test the reload path in staging before relying on it during an incident.

Fix the leak

The durable fix for a leak is in the client: ensure connections are closed on shutdown and not recreated per operation. Common patterns are creating a new connection per request instead of reusing one, or losing the reference to a connection without closing it. Use the /connz inventory you gathered during diagnosis to confirm the fix: connection count per host should return to the expected pool size.

Fix the pool misconfiguration

If each application instance opens far more connections than intended, correct the pool size in the client configuration. Connection pooling defaults in some frameworks are generous; a few hundred instances times a large default pool can fill a server that looked comfortably sized.

Prevention

  • Alert on the ratio, early. Warning at >80% and critical at >95% of connections / max_connections gives you hours or days of runway for a leak, and at least a fighting chance during a storm. Keep roughly 20% free slots as standing headroom to absorb reconnection bursts.
  • Monitor the FD ceiling as its own signal. Track process FD usage against ulimit -n and alert well before exhaustion. Set ulimit -n to at least 65536 for production servers, more if the connection budget demands it.
  • Watch churn, not just the count. total_connections delta relative to a stable connections count is the earliest indicator of flapping clients and storm conditions.
  • Size account limits deliberately. If you use accounts, set per-account connection limits so one misbehaving tenant cannot consume the entire server budget.
  • Test reconnection behavior. Client libraries reconnect automatically; verify that your clients use backoff and jitter so a healed partition does not produce a synchronized thundering herd. A storm can consume all free slots in seconds even when steady-state usage is tiny.
  • Audit client shutdown paths. Clients that exit without closing connections leave slots occupied until stale detection cleans them up. Graceful close on shutdown is cheap insurance.

How Netdata helps

Netdata’s NATS collector polls the server’s HTTP monitoring endpoints and turns the raw counters into the signals this incident actually needs:

  • Connection utilization as a ratio. Netdata charts connections alongside max_connections from /varz, so you see saturation approaching as a percentage, not an absolute number you have to mentally compare.
  • Churn visibility. total_connections is collected as a cumulative counter, so the rate view exposes reconnect storms and crash-looping clients even when the live connection count looks stable.
  • Leak detection over time. Per-second and historical retention makes the slow monotonic climb of a connection leak obvious, and lets you correlate the start of the climb with a deploy or config change.
  • Correlation with the downstream signals. Slow consumers, memory, and throughput live on the same dashboard, so you can see whether connection churn is coming from slow-consumer disconnect cycles or from an upstream network event.
  • Cluster-wide view. In a multi-server cluster, per-server connection charts show whether the wall is one overloaded node or a systemic growth trend across all nodes.