NATS JetStream no leader / cluster not currently available: writes blocked

Your publishers are failing with nats: no leader for stream or JetStream cluster not currently available, or every JetStream API call times out with no responders on $JS.API. Stream info shows an empty cluster.leader. Writes stay paused until the affected Raft group elects a leader again.

This is a quorum problem, not a load problem. JetStream uses Raft for consensus: one meta group manages all JetStream metadata and admin operations, and each replicated stream (and consumer) has its own Raft group. A group without a leader blocks everything that depends on it. If the meta group is leaderless, all JetStream admin operations fail cluster-wide. If only a stream’s group is leaderless, that stream stops accepting messages while the rest of JetStream keeps working.

The usual trigger is a network partition, losing multiple nodes at once, or an election storm caused by slow disks or CPU starvation. The fix is to restore quorum: bring the missing members back, or cleanly remove dead peers. The worst response is to thrash the cluster with restarts and forced elections while it is trying to converge.

What this means

Raft requires a quorum of floor(R/2) + 1 voting members to elect a leader. A 3-replica group needs 2 members; a 5-replica group needs 3. Below that, no election can pass, and without a leader the group accepts no writes.

There are two distinct scopes, and telling them apart is the first diagnostic step:

• Meta group leaderless: every JetStream API operation (stream create, consumer create, stream info, placement decisions) fails or hangs. Clients publishing via the JetStream API see JetStream cluster not currently available or no responders on $JS.API subjects. Existing R>1 streams whose own groups still have quorum can keep accepting messages, but you cannot administer anything.
• Stream group leaderless: only that stream is affected. nats stream info shows an empty cluster.leader, publishes to that stream fail with a no-leader error, and other streams work normally.

A related but different condition is leader flapping: the group has a leader, but it keeps changing. Writes pause briefly during each election. If the leader name is changing rather than missing, you are dealing with instability (usually network latency, GC pauses, or disk I/O), not quorum loss, and the recovery path is different.

flowchart TD
  A[Writes blocked: no leader errors] --> B{Does $JS.API respond?}
  B -- "No: no responders / cluster not available" --> C[Meta group has no leader]
  B -- "Yes, but one stream fails" --> D[Stream Raft group has no leader]
  C --> E[Check meta_cluster in /jsz on every node]
  D --> F[Check cluster.replicas in stream info]
  E --> G{Quorum members alive?}
  F --> G
  G -- "Yes: partition or slow" --> H[Fix network, disk, CPU. Let election finish]
  G -- "No: members lost" --> I[Restore nodes or peer-remove dead members]

Common causes

Quick checks

All of these are read-only and safe to run during an incident.

# 1. Meta group state: is there a leader, and which peers are offline or lagging?
curl -s http://localhost:8222/jsz | jq '.meta_cluster | {leader, replicas: [.replicas[]? | {name, current, offline, lag}]}'

# 2. JetStream API pressure: high inflight + rising errors means consensus is stalled or slow
curl -s http://localhost:8222/jsz | jq '{api_total: .api.total, api_errors: .api.errors, inflight: .api.inflight}'

# 3. Cluster routes: each server should have N-1 routes in a full mesh
curl -s http://localhost:8222/varz | jq '{routes, connections, uptime}'

# 4. Per-route detail: pending_size > 0 means inter-server traffic is backing up
curl -s http://localhost:8222/routez | jq '.routes[] | {rid, ip, rtt, pending_size}'

# 5. Full JetStream report from the CLI: meta leader, current/offline peers, lag
nats server report jetstream

# 6. For a single affected stream: is cluster.leader empty, and which replicas are current?
nats stream info STREAM_NAME --json | jq '.cluster | {leader, replicas: [.replicas[]? | {name, current, lag, offline}]}'

# 7. Per-Raft-group state across the node (meta group plus per-stream groups)
curl -s http://localhost:8222/raftz | jq '.'

Two log patterns matter while you do this: repeated “new leader” or “stepping down” entries indicate flapping, and a complete absence of election activity on a leaderless group means the remaining members cannot reach quorum at all.

How to diagnose it

1. Scope the failure. Try any JetStream API read, for example nats stream ls. If it hangs or returns no responders / cluster not currently available, the meta group is leaderless and the blast radius is cluster-wide admin operations. If API reads work but publishes to one stream fail with a no-leader error, it is a single stream’s Raft group.

2. Check meta group health on every node, not just one. /jsz meta_cluster shows the leader as seen by that node and per-replica current, offline, and lag. If different nodes report different leaders or disagree on who is offline, you have a partition, not a clean quorum loss.

3. Count live members against quorum. A 3-node cluster needs 2 members; 5 needs 3. If fewer are reachable and healthy, no election can pass. This is the branch point: restore members, or remove dead peers to shrink the required quorum.

4. Rule out leader flapping. If a leader exists but keeps changing, check route RTT, disk latency on the JetStream storage path, and CPU. Network stalls or disk stalls lasting longer than the election timeout (a handful of seconds) trigger a new election. Slow Raft WAL writes on network-attached storage are the classic cause. Fix the latency source; do not restart nodes.

5. For a single leaderless stream, inspect its replicas: nats stream info shows cluster.replicas with current, lag, and offline. If enough replicas are alive but not electing, look at /raftz for that group and at disk I/O on those nodes. If replicas are gone, you are in peer-removal territory.

6. Check what changed. A mass rename of servers (for example, changing the Helm chart’s serverNamePrefix), a reprovisioned node with a fresh disk, or a rolling deploy that took down multiple members at once are the usual ways operators accidentally destroy quorum. This history determines which recovery path below is safe for your version.

Metrics and signals to monitor

Fixes

Restore the missing members

This is always the preferred path. Bring the downed nodes back with their original server names and their data intact. Once a quorum of the original peer set is reachable, the group elects a leader on its own. Do not force elections, restart surviving nodes, or run admin mutations while the group is converging; every disruption restarts the election timers.

Remove dead peers to shrink quorum

If a member is permanently gone (decommissioned, unrecoverable disk), remove it from the group so the remaining members form a quorum:

# Meta group: remove a stale peer (disruptive; run only when the peer is truly gone)
nats server raft peer-remove <peer-name>

# A specific stream's Raft group:
nats stream cluster peer-remove <stream> <peer-name>

Removing a peer triggers replica rebalancing, with two consequences to know before you run it: durable consumers on the removed peer are reassigned to surviving stream peers, but ephemeral consumers whose only replica lived there are deleted. And for an R1 stream whose only replica was on the removed peer, the stream assignment is dropped and the data is not recoverable from the cluster. Treat peer removal as a deliberate, one-way operation.

Version-specific recovery behavior (2.12 changed the rules)

On nats-server 2.10 and 2.11, adding a fresh empty node to a cluster that had lost quorum could lift the election over the line because the new node forced itself into the peer set. As of 2.12 (PR #7038), that no longer works: an empty node cannot force its way in. On 2.12 and later:

• The safe path is to bring the downed nodes back under their original names.
• If the original nodes come back with empty disks, all originally named peers must be available at the same time for an election to pass.
• If you cannot reclaim the original names at all, there is no in-product recovery path at the time of writing; restore from backup.
• Renaming servers (including via the Helm chart’s serverNamePrefix) after initial install doubles the recorded server count and loses quorum. On 2.12+, reverting the name alone does not unbrick the cluster; the original-named servers must actually come back online.

Break an election storm

If the group has quorum but cannot keep a leader because of latency or resource starvation, reduce load first: pause non-critical publishers, verify disk latency and CPU on all members, and check for noisy neighbors on shared storage. A rolling restart, one node at a time with full convergence between each, can break a self-sustaining storm by temporarily reducing load. Restarting everything at once makes it worse.

Prevention

• Size replica counts for the failure you actually want to survive. R3 tolerates one lost member, R5 tolerates two. R1 streams have no leader election to save them; the data lives on one node.
• Put JetStream storage on local SSDs. Network-attached storage with variable latency is the leading cause of Raft election storms.
• Never rename servers or change name prefixes after initial provisioning. Treat server identity as immutable cluster state.
• Take nodes down one at a time during maintenance, and wait for meta_cluster and stream replicas to report current: true before touching the next node.
• Alert on the leading indicators, not the outage: meta leader changes, any replica offline over 60 seconds, rising api.inflight, and route count below N-1. These fire minutes to hours before writes block.
• Watch disk latency as a first-class signal on every JetStream node, and give the process enough CPU headroom that GC pauses and scheduling delays stay well under election timeouts.

How Netdata helps

Netdata’s NATS collector polls the HTTP monitoring endpoints, which is exactly where this failure is visible:

• Meta group state from /jsz: leader presence, replica current/offline/lag flags, so you see quorum eroding before the leader disappears.
• api.inflight and api.errors rates: the signature of consensus stalling, visible as a rising error rate while publishes back up.
• Route count and per-route pending_size: distinguishes a network partition (quorum still reachable but split) from genuine node loss.
• Uptime tracking across nodes: simultaneous resets point to a multi-node event (deploy, AZ failure) as the cause of quorum loss.
• CPU, memory, and disk latency correlation: lets you tell an election storm caused by slow storage apart from a true partition, which determines whether you fix infrastructure or restore members.
• Consumer lag and stream storage in the same view: after recovery, confirms that paused publishers and consumers are draining backlogs instead of tipping into storage exhaustion.

Related guides

• How NATS actually works in production: a mental model for operators
• NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check
• NATS JetStream API errors: reading the /jsz api.errors counter without false alarms
• NATS context deadline exceeded: JetStream publish and request timeouts
• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS connection storm: reconnect thundering herd after a network event
• NATS JetStream consumer lag growing: falling behind the stream