NATS route disconnected: a missing cluster route means a partition

One of your NATS servers shows fewer routes than it should. In a full-mesh cluster of N servers, every server should hold N-1 route connections, one to each peer. When that count drops, you do not have a degraded link. You have a partition.

The impact is asymmetric and easy to underestimate. The isolated server still accepts client connections, still reports healthy on /healthz, and still routes messages locally. But subscribers connected to it stop receiving messages published on the other side of the partition, and publishers on it vanish from the rest of the cluster’s view. If you run JetStream with replicated streams, Raft groups whose members span the partition can lose quorum, which turns a messaging gap into write failures.

This guide covers how to confirm a missing route, find the cause, and restore the mesh without making things worse.

What this means

NATS clustering works over routes: full-mesh TCP connections between every pair of servers. Routes carry two things: interest propagation (which subscriptions exist where) and the actual messages that need to cross servers. Each route behaves like an internal client connection with its own read/write buffers, and it can itself become a slow consumer.

When a route is missing:

• Interest from the unreachable peer is gone. The local server no longer knows which subscriptions exist on the far side, so it stops forwarding messages that way.
• Core NATS messages published on either side that target subscribers on the other side are dropped. There is no queue in core NATS. They are simply not delivered.
• JetStream Raft groups with members on both sides may lose quorum. A 3-replica stream with one replica on the isolated server keeps quorum as long as the two other replicas can still reach each other. A 2-node cluster is the worst case: the single route is the entire mesh, and losing it is a complete partition. NATS documentation recommends at least 3 nodes for production clusters for exactly this reason.

flowchart LR
  A[nats-1] ---|route OK| B[nats-2]
  B ---|route OK| C[nats-3]
  A -.->|route MISSING: partition| C
  S1[subscriber on nats-3] --- C
  P1[publisher on nats-1] --- A
  P1 -.->|messages dropped: no route| S1

Routes reconnect automatically, so a brief blip during a rolling restart or a network flap usually self-heals. The signal that matters is a route count below expected, sustained for more than 60 seconds. Anything shorter is reconnection noise.

Common causes

A note on ordering: check whether the peer is alive before assuming a network fault. A crashed or restarting peer is the most common cause and the fastest to confirm.

Quick checks

All of these are read-only. The monitoring HTTP port is 8222 by default and must have been enabled with -m 8222 or http_port: 8222.

# Current route count on this server
curl -s http://localhost:8222/varz | jq .routes

# Detailed route list: how many, and which peers are present
curl -s http://localhost:8222/routez | jq '.num_routes, [.routes[].ip]'

# Per-route health: RTT and backpressure
curl -s http://localhost:8222/routez | jq '.routes[] | {rid, remote_id, ip, rtt}'

# Route-level slow consumer events (high blast radius)
curl -s http://localhost:8222/varz | jq '{slow_consumers, slow_consumer_stats}'

# Is the peer alive at all?
curl -s http://<peer-ip>:8222/healthz?js-server-only=true

# Peer uptime: did it just restart?
curl -s http://<peer-ip>:8222/varz | jq .uptime

Two things to keep in mind while reading the output:

• /varz routes and /routez num_routes should agree. If they disagree materially, check your server version before trusting either number. On v2.10.0 and later with route connection pooling (default pool_size: 3), /varz routes and /routez num_routes count individual route connections, so the expected count becomes (N-1) × pool_size rather than N-1. Verify your configured pool size before comparing against the expected mesh count.
• Routes are not client connections. /routez and /connz are separate worlds. A server can show a healthy client connection count while having zero routes.

How to diagnose it

1. Establish the expected count. In a full-mesh cluster of N servers, each server should show N-1 routes. Write down what each server reports via /varz routes. Do this on every server, not just the alerting one. The pattern tells you the shape of the partition: one server missing routes to everyone is isolated; two servers each missing exactly one route have a broken link between just that pair.

2. Check it is sustained, not a blip. Routes auto-reconnect. Re-poll after 60 seconds. If the count recovered, this was a reconnect event, not a partition. Note it and move on, but if it recurs, treat the flapping itself as the problem.

3. Check whether the missing peer is alive. Hit its /healthz?js-server-only=true and read its uptime. If the peer is down or just restarted, the route is a symptom. Go fix the peer. If the peer is healthy and has been up for hours, the route failure is between the servers, not inside one of them.

4. Check connectivity on the cluster port. From each side, verify the peer’s cluster port is reachable. A firewall or security group change that blocks the route port in one direction produces exactly this signature: both servers healthy, both up, route never re-establishes.

5. Check TLS and DNS. If cluster routes use TLS, confirm the certificate has not expired (openssl x509 -enddate -noout -in <cert> or openssl s_client against the cluster port). If routes are configured by DNS name, resolve the name from the local server. Slow or failing DNS makes route establishment time out and presents as disconnections.

6. Check for route slow consumer events. If slow_consumer_stats.routes is non-zero and climbing, the route was up but got disconnected for falling behind on writes. That points to an overloaded peer or a degraded network path, not a hard partition. Look at /routez per-route pending_size and rtt on the surviving routes for corroboration.

7. Assess JetStream impact. If you run replicated streams, check the meta cluster: curl -s http://localhost:8222/jsz | jq '.meta_cluster | {leader, replicas: [.replicas[]? | {name, current, offline, lag}]}'. An offline or non-current replica that sits on the far side of the partition means reduced redundancy even if quorum holds. Leader changes coinciding with the route drop confirm Raft felt it.

Metrics and signals to monitor

The alert condition worth encoding: expected_routes > 0 AND current_routes < expected_routes sustained for more than 60 seconds. Gating on expected_routes > 0 keeps standalone servers quiet, and gating on expected rather than current means you still fire in the worst case, total route loss, where current is 0.

Fixes

Fixes are ordered by cause. Do not restart servers as a first move: routes auto-reconnect, and a restart that does not address the cause just adds a reconnection storm on top of the partition.

Peer crashed or hung

Restore the peer. The route comes back on its own once the peer rejoins. While it is down, clients configured with multiple server URLs will reconnect to surviving servers; clients pinned to the dead peer are offline until it returns. If the peer keeps restarting, that is a separate incident (see the crash loop guide linked below).

Network partition or firewall change

Restore the path on the cluster port in both directions. This is usually an infrastructure change: a security group edit, a NetworkPolicy, a routing change. Once the path is open, the route re-establishes without server intervention. If the partition was long, watch for a brief interest re-propagation spike on the route afterward; that is normal.

TLS certificate expired or rotated badly

Renew or fix the certificate, then reload or restart one server at a time. Certificate expiry on route connections severs the mesh the same way it severs client connections, and with mutual TLS both sides must present valid certs. Monitor tls_cert_not_after from /varz where your version exposes it, and alert at 30 days, escalating at 7.

DNS failure for route endpoints

Fix name resolution or switch route configuration to stable addresses. Slow DNS does not just delay route establishment; it makes routes flap as connect attempts time out, which is noisier and harder to alert on than a clean partition.

Route slow consumer disconnects

The route was alive but could not drain its write buffer. The cause is an overloaded peer or a degraded network path, so the fix is on that side: reduce load on the peer, fix the network path, or reduce cross-cluster fan-out. Raising the write deadline can reduce flapping but delays detection of genuinely slow routes; treat it as a last resort, not the fix.

After any fix

Verify recovery on every server: route count back to N-1, /routez listing all expected peer IPs, and for JetStream, all meta and stream replicas current=true with offline=false. Then check for the second-order effect: subscribers on the previously isolated side may have missed messages in core NATS. Those messages are gone. If the workload cannot tolerate that, that is an architecture problem (use JetStream), not a route problem.

Prevention

• Alert on the gap, not the absence. Encode expected_routes > 0 AND current < expected sustained >60s. This catches total route loss and ignores standalone servers.
• Run at least 3 servers. A 2-node cluster has a single route. Losing it is a complete partition with no redundancy anywhere. With 3+, a single broken link degrades but does not isolate.
• Watch the precursors. Per-route RTT and pending_size climb before routes die. A route that is slowly backing up is next week’s partition.
• Monitor route slow consumers separately. slow_consumer_stats.routes deserves its own alert, distinct from client slow consumers, because the blast radius is the whole cluster.
• Track certificate expiry. TLS failures between cluster members are self-inflicted partitions. Alert at 30 days, escalate at 7.
• Plan for rolling restarts. Brief route drops during maintenance are expected. Silence or downgrade the alert during planned windows rather than loosening it permanently.

How Netdata helps

• Netdata polls the NATS monitoring endpoints and charts /varz route count per server, so a count dropping below the expected mesh size is visible as a sustained step, not buried in logs.
• Correlating route count with peer uptime and /healthz status in one view answers the first diagnostic question immediately: dead peer or broken path.
• Slow consumer charts, including the per-type breakdown, let you see route-level slow consumer events building before a route is disconnected.
• Comparing route count across all cluster members side by side shows the partition shape (one isolated server vs a single broken link) without logging into each host.
• Uptime and restart correlation catches the case where a flapping peer, not the network, is the reason routes keep dropping.

Related guides

• NATS JetStream AckWait tuning: matching the ack timeout to processing time
• NATS connection churn: a stable connection count hiding constant reconnects
• NATS connection storm: reconnect thundering herd after a network event
• NATS JetStream consumer lag growing: falling behind the stream
• NATS consumer stalled at MaxAckPending: delivery stops until messages are acked
• NATS JetStream redelivery loop: num_redelivered climbing and messages reprocessed
• NATS JetStream consumer stopped receiving messages: the diagnostic tree
• NATS context deadline exceeded: JetStream publish and request timeouts
• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS file descriptor exhaustion: too many open files and the ulimit cliff
• NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check
• How NATS actually works in production: a mental model for operators