NATS connection churn: a stable connection count hiding constant reconnects

Your NATS dashboard shows 4,000 client connections. It showed 4,000 an hour ago, and 4,000 yesterday. Everything looks stable. Meanwhile, clients are connecting and disconnecting hundreds of times per minute. Every reconnect burns CPU on protocol handshakes (and TLS handshakes, if enabled), the server logs fill with connect and disconnect events, and your auth system processes a constant stream of authentication attempts.

This is connection churn, and it is one of the most commonly missed NATS failure modes because the metric everyone charts, the current connections gauge, is designed to hide it. A client that disconnects and reconnects within one scrape interval leaves the gauge unchanged. The churn is real; your chart just cannot see it.

The detection metric is one most teams never chart: the delta of total_connections, the server’s lifetime cumulative connection counter. When connections is flat but total_connections is climbing fast, clients are flapping. This article covers how to detect the pattern, how to find the clients responsible, and how to fix the usual causes.

What this means

NATS exposes two connection figures on the /varz monitoring endpoint:

• connections: the number of active client connections right now. A gauge.
• total_connections: every connection established since the server started. A cumulative counter.

In a healthy deployment, clients are long-lived. connections stays near its steady-state value and total_connections advances slowly, only when clients deploy, scale, or genuinely restart. When something goes wrong on the client side, the two decouple: total_connections climbs while connections stays flat, because each disconnect is immediately replaced by a reconnect.

Each cycle has a real cost: a TCP handshake, optionally a TLS handshake, the NATS protocol handshake, authentication, and subscription re-establishment. A few hundred flapping clients can keep a core busy on handshakes alone while delivering no useful messages. Churn is also a leading indicator of worse things: it frequently accompanies the slow consumer disconnect-reconnect spiral, and reconnect bursts can push the server into max_connections or file descriptor exhaustion.

flowchart LR
  C[Client] -->|connect + auth| S[NATS server]
  S -->|disconnect: slow consumer, timeout, LB drop| C
  C -->|immediate reconnect, no backoff| S
  S --> V[varz: connections gauge flat]
  S --> T[varz: total_connections climbing fast]
  T --> D[churn = rate of total_connections]

The detection formula:

churn rate = delta(total_connections) / interval

If that rate is well above your baseline while connections is steady, you have churn. Baseline is workload-dependent: a batch system with short-lived workers legitimately has a higher connection rate than a fleet of daemons. What matters is deviation from your own norm.

One caveat before you alert on the raw counter: total_connections resets to zero on server restart, and the Prometheus exporter (gnatsd_varz_total_connections) exposes it as a gauge carrying the running total, not a Prometheus counter. rate() still works on it, but a server restart produces a reset that can look like a spike or a dip in increase(). Correlate with uptime before treating a counter discontinuity as a churn event.

Common causes

Quick checks

All checks are read-only against the monitoring port (default 8222).

# Snapshot the three connection figures
curl -s http://localhost:8222/varz | jq '{active: .connections, total: .total_connections, max: .max_connections}'

# Measure churn directly: two samples 30s apart, compute the delta
A=$(curl -s http://localhost:8222/varz | jq .total_connections)
sleep 30
B=$(curl -s http://localhost:8222/varz | jq .total_connections)
echo "new connections in 30s: $((B - A))  (rate: $(( (B - A) / 30 ))/s)"

# Rule out a server restart resetting the counter mid-measurement
curl -s http://localhost:8222/varz | jq .uptime

# Check whether churn is tied to slow consumer events
curl -s http://localhost:8222/varz | jq '{slow_consumers, slow_consumer_stats}'

# Look at recently closed connections: who is leaving, and why
curl -s 'http://localhost:8222/connz?state=closed' | jq '.connections[:20] | .[] | {cid, name, ip, reason, subscriptions}'

# Find connections currently building write backlog (pre-slow-consumer)
curl -s 'http://localhost:8222/connz?sort=pending&limit=10' | jq '.connections[] | {cid, name, ip, pending_bytes, rtt}'

Notes on the closed-connection check: the server holds a bounded window of recently closed connections (configurable with max_closed_clients; the default is the last 10,000). Under heavy churn, that window can cover only a few minutes, so sample it promptly when you see the churn rate spike, not an hour later.

If you scrape via the Prometheus exporter, the equivalent query is:

# New connections per second, per server
rate(gnatsd_varz_total_connections[5m])

Remember the metric is a gauge that resets on restart; ignore samples spanning an uptime reset.

How to diagnose it

1. Confirm churn exists. Take two /varz samples 30 to 60 seconds apart. If total_connections advances by far more than your expected deploy/scale activity while connections is flat, churn is confirmed. Note the rate: single digits per second is a slow leak, hundreds per second is an active storm.

2. Check the clock. Read uptime. A server that restarted 5 minutes ago will show a “climbing” total_connections purely from normal client reconnection after the restart. Churn conclusions are only valid on a server with stable uptime. See NATS crash loop: unexpected uptime resets and repeated restarts if uptime itself is the problem.

3. Correlate with slow consumers. Read slow_consumer_stats from /varz (the clients/routes/gateways breakdown is available since nats-server v2.10.0). If the clients component is rising in step with churn, clients are being disconnected for falling behind and reconnecting into the same backlog. That is the slow consumer death spiral, and the fix belongs on the consumer, not the connection layer. If routes or gateways are non-zero, the churn involves inter-server links and the blast radius is cluster-wide; treat it as more urgent.

4. Identify who is flapping. Pull /connz?state=closed and group the results by ip, name, and reason. One service name or one source subnet dominating the list points at the culprit. Short-lived connections with zero or near-zero subscriptions and LB source IPs point at health-check traffic rather than real clients.

5. Characterize the cycle timing. Compare the start and stop fields in the closed-connection data. Loops of a few seconds suggest no reconnect backoff or immediate failure after connect (auth, permissions, protocol error). Loops of tens of seconds to minutes suggest a client that connects fine, falls behind, and gets disconnected as a slow consumer.

6. Check the client side. For the implicated service: process restart counts, client logs around reconnects, and the reconnect/backoff configuration of its NATS client library. Server-side data tells you who and how often; only client-side data tells you why the loop started.

7. Quantify the cost. Compare cpu on /varz against message throughput (in_msgs, out_msgs). CPU elevated without a corresponding message rate means the server is spending cycles on handshakes and teardown rather than routing. With TLS enabled this gap is especially pronounced.

Metrics and signals to monitor

Fixes

Crash-looping clients

The server is a victim here, not the cause. Fix the client crash first: read the client logs, fix the panic or config error, then let churn settle. If many instances of the same service crash-loop together, stop the rollout at the orchestrator level rather than letting the whole fleet hammer the server. Tradeoff: pausing a rollout leaves you on the old version, but it stops the connection storm immediately.

Missing reconnect backoff

NATS client libraries reconnect automatically, and most include jittered backoff, but custom wrappers and misconfigured options can disable it or set the wait to near zero. Set a minimum reconnect wait and keep jitter enabled so a population of disconnected clients does not reconnect in lockstep. There is no server-side fix for a client that reconnects instantly; this must be changed in the client configuration and redeployed.

Slow consumer disconnect-reconnect spiral

This is the nastiest variant: the server correctly disconnects a client that cannot keep up, the client reconnects, resubscribes, immediately falls behind again, and the loop repeats, incrementing slow_consumers each cycle. Do not “fix” this by raising server buffers; that only delays the disconnect and grows memory. Fix the consumer’s throughput: remove synchronous I/O from the message handler, scale the consumer out, or move the workload to JetStream pull consumers where the consumer controls its own rate. If you genuinely need to adjust how long the server tolerates a slow writer, see NATS write_deadline and buffer sizing. For identifying which connections are building backlog before they are disconnected, see NATS pending bytes growing.

Load balancer health-check flapping

If closed connections are short-lived, carry no subscriptions, and come from LB addresses, your health checks are the churn. Where the load balancer supports it, point health checks at the HTTP monitoring port’s /healthz endpoint instead of opening a fresh TCP connection to the client port on every probe; see NATS /healthz explained for what the variants check. If the LB can only do TCP connects, lengthen the probe interval and exclude LB source ranges from your churn alerting so real client churn stays visible.

Reconnect storms after network recovery

When a partition heals, every disconnected client reconnects at once. This is a burst, not a loop: churn spikes and then decays. The operational risks are CPU saturation from simultaneous handshakes (especially with TLS) and running into max_connections or the OS file descriptor limit during the peak. Keep at least 20% headroom below max_connections precisely to absorb these storms; see NATS Maximum Connections Exceeded. If the “storm” repeats cyclically, the underlying network fault is still active and that is what needs fixing.

Leaf node link cycling

A leaf link under heavy hub-to-leaf load can enter a cycle where the write path to the leaf saturates, protocol liveness responses queue behind data, the leaf declares the connection stale, and reconnects into the same backlog. Treat this like a route-class slow consumer: reduce the traffic volume over the leaf or fix the bottleneck on the receiving side, not the reconnect timing.

Prevention

• Chart the churn rate permanently. Add rate of total_connections (per server) to your standard NATS dashboard next to the connections gauge. This is the single highest-value change; every other item on this list is easier once you can see the signal.
• Alert on deviation, not absolutes. Alert when the churn rate exceeds your rolling baseline by a wide margin for more than a few minutes. Absolute thresholds break across deployments of different sizes.
• Gate alerts on uptime. Suppress churn alerts for servers with uptime under a few minutes; post-restart reconnection looks identical to churn.
• Enforce reconnect backoff in shared client wrappers. If your organization wraps NATS client setup in a library, make minimum backoff and jitter non-overridable defaults.
• Correlate churn with slow consumer signals. A churn alert that also shows rising slow_consumers and per-connection pending_bytes points straight at a backpressure problem and skips an hour of guessing.
• Keep connection headroom. Size max_connections and OS file descriptor limits so a full-fleet reconnect storm fits with room to spare.

How Netdata helps

• Netdata collects both connections and total_connections from /varz on every server, so the churn rate and the steady-state gauge are visible on the same dashboard without building custom scrapes.
• Because collection is per-second, short-lived connect-disconnect cycles that a 30 or 60 second scrape interval would average away still show up in the cumulative counter’s slope.
• Netdata charts slow_consumers alongside connection metrics, making the churn-plus-backpressure spiral a one-screen correlation instead of two separate queries.
• Uptime is collected as a metric, so counter resets from server restarts are easy to distinguish from genuine churn when reviewing an incident timeline.
• Per-server views across a cluster let you see whether churn is isolated to one node (client affinity, LB backend issue) or uniform (client-side bug, network-wide event).
• ML-based anomaly detection on the total_connections rate catches churn that deviates from your normal deploy-driven baseline without hand-tuned thresholds.

Related guides

• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS Maximum Connections Exceeded: new clients rejected at the max_connections wall
• NATS slow consumer detected: the write buffer overflowed and messages were dropped
• NATS slow consumer breakdown: clients vs routes vs gateways and blast radius
• NATS pending bytes growing: catching a slow consumer before it is disconnected
• NATS stalled clients and stale connections: half-dead sockets and write-path distress
• NATS write_deadline and buffer sizing: tuning how long the server waits on a slow writer
• NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check
• NATS monitoring checklist: the signals every production server needs
• How NATS actually works in production: a mental model for operators