$ guides / nats / nats-connection-storm ▌

Operations Guides

NATS connection storm: reconnect thundering herd after a network event

A network device fails and recovers. A load balancer health check flaps. A rolling restart drops a node. For a few seconds, every client attached to a NATS server loses its connection. Then the network heals, and every one of those clients tries to reconnect in the same second.

Each reconnect is not cheap: TCP handshake, optional TLS handshake, protocol negotiation, authentication, and a full resubscribe of every subscription the client held. Multiply that by hundreds or thousands of clients arriving simultaneously and you have a connection storm: a sharp spike in connections and total_connections, a CPU spike dominated by TLS handshakes if encryption is enabled, memory climbing as per-connection state is allocated, and in the worst case the server hitting max_connections or the OS file descriptor limit and rejecting clients. Rejected clients retry. Now you have a reject-and-retry loop on top of the storm.

This guide covers how to recognize the pattern, tell it apart from lookalikes, ride out the acute phase, and fix the client behavior that makes storms expensive.

What this means

A connection storm is a thundering herd on the reconnect path. The trigger is always a mass disconnect event followed by mass recovery. The clients did nothing wrong individually; the problem is that they all recovered at the same instant with no spreading of their retry timing.

The server-side cost concentrates in three places:

CPU. Protocol negotiation and auth per connection are modest. TLS handshakes are not. Hundreds of simultaneous TLS handshakes can saturate a core, and while the server is busy negotiating sessions it is not serving the clients that are already connected.
File descriptors. Every connection consumes an FD, on top of routes, gateways, leaf nodes, JetStream file handles, and listener sockets. If the storm pushes the process against ulimit -n, accept() starts failing and no new connections get in. This is a cliff-edge failure with no graceful degradation.
Connection slots. If the server was already at moderate connection counts before the event, the burst can push it past max_connections (default 65536). New attempts are rejected outright; there is no server-side queue. The client sees the rejection and retries, which is where the loop forms.

The pattern has a characteristic shape:

flowchart LR
  A[Network event: blip, LB flap, rolling restart] --> B[Mass client disconnect]
  B --> C[Network recovers]
  C --> D[All clients reconnect simultaneously]
  D --> E[CPU spike: TLS handshakes]
  D --> F[FD and connection-slot pressure]
  E --> G{Server keeps up?}
  F --> G
  G -->|Yes| H[Connections stabilize, CPU drops]
  G -->|No| I[Rejections: max_connections or FD limit]
  I --> J[Clients retry: reject-and-retry loop]
  J --> D

Two things define severity. If the server absorbs the burst and the connection count stabilizes, this is a ticket: expensive but self-healing. If the server starts rejecting connections and clients retry in tight loops, this is a page: the storm is now self-sustaining.

If the drop-and-spike pattern repeats cyclically, the triggering network fault is still active. Treat the network as the incident, not the reconnect behavior.

Common causes

Cause	What it looks like	First thing to check
Network device failure and recovery	One sharp drop in `connections`, one sharp recovery spike, then stable	Network device logs, interface error counters on the NATS hosts
Cloud networking maintenance	Drop and spike correlated with a provider maintenance window	Provider event log for the affected hosts and subnets
DNS resolution failure and recovery	Clients disconnected because they could not resolve, then flooded back	Resolver logs, client-side DNS error timestamps
Load balancer health check flapping	Repeated drop-and-spike cycles as backends are removed and restored	LB health check logs and backend membership history
Rolling restart without connection draining	Storm synchronized with deploy timing; one server drops to zero then refills	Deploy timeline vs `total_connections` delta on each node
Missing client backoff or jitter	Every reconnect attempt lands in the same few milliseconds after recovery	Client library reconnect configuration

Quick checks

All of these are read-only against the monitoring port (default 8222).

# Connection state: active, cumulative, and the configured ceiling
curl -s http://localhost:8222/varz | jq '{active: .connections, total: .total_connections, max: .max_connections}'

# CPU and memory: is the server still working through handshakes?
curl -s http://localhost:8222/varz | jq '{cpu, mem, cores}'

# Uptime: did the server itself restart during the event?
curl -s http://localhost:8222/varz | jq .uptime

# Slow consumers: did the storm push any connections into backpressure?
curl -s http://localhost:8222/varz | jq .slow_consumers

# Route health: in a cluster, did routes flap too?
curl -s http://localhost:8222/varz | jq .routes

# Health probe: is the server responsive at all right now?
curl -s http://localhost:8222/healthz?js-server-only=true

# OS file descriptor ceiling for the server process
cat /proc/$(pgrep nats-server | head -1)/limits | grep "open files"

/varz exposes both an aggregate slow_consumers counter and a slow_consumer_stats breakdown (clients, routes, gateways, leafs), but neither endpoint identifies which specific connection tripped; the server log carries the per-connection detail.

The two numbers to watch together are connections (active gauge) and total_connections (cumulative counter since start). During a storm, total_connections jumps rapidly while connections oscillates. After recovery, connections flattens and total_connections stops climbing. If total_connections keeps climbing at a high rate while connections stays flat or low, clients are connecting and being dropped or disconnected in a loop.

How to diagnose it

Confirm the shape. Pull a time series of connections and total_connections across the event window. A connection storm shows a sharp drop followed seconds later by a near-vertical recovery. A slow consumer cascade or connection leak ramps; a storm steps.
Check whether the server kept up. If connections recovered to roughly the pre-event count and CPU has fallen back, the server absorbed the burst. Verify the slow consumer counter did not tick up during the window, since a burst of reconnecting clients resubscribing and catching up can push weak consumers into backpressure.
Look for rejection. If connections recovered to a plateau below the pre-event count and total_connections is still climbing fast, clients are being rejected and retrying. Compare connections against max_connections (alert-worthy above 85% utilization) and against the process FD limit. On the client side, the Go client surfaces this as a max-connections-exceeded error; in server logs you will see the corresponding rejects.
Attribute the CPU spike. If TLS is enabled and CPU spiked with the reconnect burst while message throughput was still recovering, handshake cost is the driver. Plain TCP reconnects rarely saturate CPU on their own.
Decide if it is cyclical. One drop-and-spike is an event. Repeated drop-and-spike at a regular interval is an active fault: a flapping LB health check, an unstable link, or a DNS resolver failing intermittently. Do not tune clients to tolerate a fault that should be fixed.
Correlate across the cluster. In a multi-server cluster, check uptime and connections on every node. If clients pile onto surviving nodes after one node dropped, the blast radius concentrates there. Route counts on each server should be N-1 for an N-node cluster; a route drop during the same window means the event hit the cluster fabric too, which is a different and larger problem.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`connections` (active)	Current client load; the storm shape lives here	Sharp drop followed by near-vertical recovery
`total_connections` delta	Churn rate; reveals reject-and-retry loops even when active count looks stable	High delta rate while `connections` stays flat
`connections / max_connections`	Proximity to the hard rejection wall	Above 85% during or after a recovery event
FD usage vs `ulimit -n`	The cliff that arrives before `max_connections` does	Headroom below 30% during normal operation
`cpu`	TLS handshake saturation during the burst	Spike synchronized with the reconnect burst, slow to decay
`mem`	Per-connection state allocation during the burst	Step up during the storm that does not release
`slow_consumers`	Reconnected clients catching up can trip backpressure	Any increase during the recovery window
`uptime`	Separates “server restarted” from “network dropped clients”	Reset synchronized with the event means a different incident
`routes`	Cluster fabric health during the same event	Below N-1 for the cluster size

Fixes

Ride out the acute phase

If the server is recovering (connection count stabilizing, CPU falling), do nothing disruptive. Do not restart the server; a restart drops every connection again and re-triggers the exact storm you are trying to exit. Watch total_connections delta and CPU until both settle.

If the server is rejecting connections in a loop:

Confirm which wall you hit. connections pinned at max_connections is the NATS limit. accept() errors in the server log with connections below max_connections is the FD limit. The remediation differs.
If the FD limit is the wall, raising ulimit -n requires a restart to take effect, which itself causes a full storm. Plan it: raise the limit, then restart during a window when a full-client reconnect burst is survivable, or roll through a cluster one node at a time so clients land on peers. Production NATS should run with at least 65536 FDs; the 1024 default on some systems is catastrophically low.
If max_connections is the wall, remember that routes, gateways, and leaf node connections count toward it too, so the pool available to clients is smaller than the number suggests. Raising it buys headroom but does nothing for the FD or CPU limits underneath.

Fix client reconnect behavior (the real fix)

The storm exists because every client retries at the same moment. Spreading retries in time is what turns a storm into a queue the server can drain.

Enable or increase reconnect jitter. The Go client defaults are a 2s reconnect wait with 100ms jitter for plain TCP and 1s jitter for TLS (ReconnectJitter, ReconnectJitterTLS). For large client populations, those defaults spread reconnects over a narrow window. Increasing jitter, especially the TLS value, directly reduces simultaneous handshakes.
Keep bounded retries with backoff. MaxReconnect default is 60 attempts. Clients that give up entirely during a long event become a different incident (a fleet of dead services), so do not fix the storm by disabling reconnects. Fix the timing, not the resilience.
Randomize the server list. Most client libraries randomize the order of the configured server list on connect and reconnect by default, which spreads the herd across cluster nodes. If your clients were configured with an explicit ordered server list and randomization disabled, a recovery event concentrates the entire fleet on the first server in the list.
Watch the reconnect buffer. During the disconnect window, clients buffer publishes (default 8MB in the Go client). Publishers that out-produce the buffer drop messages with a reconnect-buffer-exceeded error. This is a data-loss edge hiding inside the connectivity incident.

Server-side admission control

connection_rate_limit (TLS config block, server v2.7.0+) caps new TLS client connections per second. This is the primary server-side defense against handshake storms: it deliberately slows admission so CPU stays available for established connections. Default is 0 (disabled). It applies to TLS client connections only; there is no equivalent rate limiter for plain TCP connections.
Connection-slot headroom. Keep at least 20% of max_connections free in steady state specifically to absorb reconnection bursts. If your normal operating point is above 80%, the next network event becomes a rejection incident by arithmetic, not bad luck.

Fix the network fault

If the pattern is cyclical, none of the above is the fix. Pull LB health check history, DNS resolver logs, and network device logs for the exact timestamps of the drops. A flapping health check that removes and restores a backend every 30 seconds will produce a connection storm every 30 seconds until someone fixes the check.

Prevent storms from your own restarts

Rolling restarts and config reloads are self-inflicted mass-disconnect events. Lame duck mode drains clients gradually: the server signals clients to reconnect elsewhere over a drain window instead of dropping them all at once. Use it for planned restarts. Even a fast restart without draining drops all connections simultaneously and hands the thundering herd to the surviving servers.

Prevention

Client-side jitter and backoff as a standard. Make reconnect jitter (with a raised TLS jitter for large fleets) part of your standard client configuration, not a per-team choice.
FD limits sized for the storm, not the average. ulimit -n should exceed max_connections plus JetStream file handles plus routes, gateways, and leaf nodes, with margin.
Enable lame duck mode for all planned restarts and verify your deploy tooling uses it.
Alert on churn, not just count. total_connections delta rate is the leading indicator of a reject-and-retry loop; connections alone masks it because a stable active count can hide violent churn underneath.
Distribute clients across the cluster with randomized server lists, and watch per-node connection concentration. If most of an account’s clients sit on one server, losing that server turns a node event into an account-wide storm on the survivors.
Enable connection_rate_limit on TLS servers as cheap insurance against handshake saturation.

How Netdata helps

Storm shape recognition: Netdata collects connections and total_connections from /varz per second, so the drop-and-vertical-recovery signature is visible in the dashboard without waiting for a 60-second scrape to smear it out.
Churn vs count: plotting the total_connections rate against the active connections gauge exposes reject-and-retry loops that a stable-looking active count would hide.
CPU correlation: overlaying process CPU on the reconnect window confirms (or rules out) TLS handshake saturation as the bottleneck during the burst.
Capacity wall proximity: tracking connections against the configured max_connections shows how close a recovery event pushed you toward hard rejections.
Cluster-wide correlation: viewing the same signals across all nodes at once separates a single-server event from a cluster-wide network fault, and shows where the herd re-landed.