NATS JetStream Raft election storm: leaders flapping and writes pausing repeatedly

Your JetStream cluster is electing leaders over and over. Logs alternate rapidly between Stepping down and JetStream cluster new leader. Publishes intermittently time out, stream and consumer management calls error out, and some streams briefly report no leader at all. Then it settles for a few minutes and starts again.

This is a Raft election storm. It is not a clean failover and it is not usually a software bug. It is a symptom of resource starvation or latency somewhere in the cluster, and it feeds itself: elections consume CPU and I/O, which delays heartbeats further, which triggers more elections.

JetStream almost always self-heals once you remove the underlying pressure. The job is to find that pressure fast, because while the storm runs, writes to affected Raft groups are paused.

What this means

Clustered JetStream uses Raft for consensus. There is one meta Raft group for cluster-wide JetStream metadata (stream and consumer assignments), and each replicated stream and consumer has its own Raft group. Leaders must heartbeat to followers on time. When heartbeats are delayed, by CPU starvation, disk I/O latency on the Raft WAL, network jitter, or Go GC pauses, followers stop hearing from the leader and start an election.

A single election is normal and cheap. A storm happens when the cause affects all nodes at once: the new leader inherits the same starvation, misses its own heartbeats, and gets voted out. Each election pauses writes for that group, and when many stream and consumer groups elect simultaneously, the election traffic itself adds load. That is the self-reinforcing part.

Two operational reference points from the field:

• More than 1 election per hour in steady state warrants investigation.
• More than 1 election per minute is critical: JetStream writes are effectively blocked.

Election storms pause writes but do not destroy stored data. Messages already committed in streams are safe. The damage is availability: writes and management operations fail during elections, and if the meta group is flapping, every stream and consumer group whose leader sat on the flapping node feels it.

flowchart TD
  A[Underlying pressure: CPU steal, slow WAL disk, network jitter, GC pauses] --> B[Raft heartbeats delayed]
  B --> C[Followers time out and start elections]
  C --> D[Writes pause for electing groups]
  C --> E[Election traffic adds CPU and I/O load]
  E --> A
  D --> F[api.errors spike, api.inflight climbs]
  D --> G[Streams intermittently report no leader]

Common causes

Quick checks

All read-only. Run these during a storm if you can; the signals decay fast.

# 1. Meta group leader: is it stable? Run repeatedly over a few minutes.
curl -s http://localhost:8222/jsz | jq '.meta_cluster | {leader, replicas: [.replicas[]? | {name, current, offline, lag}]}'

# 2. JetStream API pressure: errors and inflight during elections
curl -s http://localhost:8222/jsz | jq '{api_total: .api.total, api_errors: .api.errors, api_inflight: .api.inflight}'

# 3. Election frequency from the logs (the signature of a storm)
grep -c "new leader" /var/log/nats/nats-server.log
grep -c "Stepping down" /var/log/nats/nats-server.log

# 4. Overall JetStream state and Raft group inventory
nats server report jetstream

# 5. Per-stream leader distribution: is leadership concentrated on one node?
nats stream report --json | jq '.[] | {stream: .name, leader: .cluster.leader}'

# 6. Route RTT: is inter-node latency elevated or fluctuating?
curl -s http://localhost:8222/routez | jq '.routes[] | {rid, ip, rtt}'

# 7. Disk latency on the JetStream storage path
iostat -x 2 5

# 8. Kernel-level I/O errors
dmesg | tail -50

# 9. CPU and memory of the server process
curl -s http://localhost:8222/varz | jq '{cpu, cores, mem, uptime}'

On check 1: a healthy clustered deployment holds the same meta leader for days or weeks. If the leader field changes between polls a minute apart, you have your confirmation. For per-stream Raft group detail beyond the meta group, use /raftz (see the diagnostic steps below).

How to diagnose it

1. Confirm the storm and measure its rate. Count elections over a fixed window from the logs: grep "new leader" /var/log/nats/nats-server.log | wc -l over a known log rotation period, or watch meta_cluster.leader across polls. Compare against the reference points: more than 1/hour is worth investigating, more than 1/minute is critical.

2. Determine the blast radius: meta group, stream groups, or both. If only the meta leader is flapping, administrative operations (stream/consumer create, update) fail, but existing stream leaders keep serving reads and writes independently. If stream and consumer groups are also electing, writes to those specific streams pause during each election. Check per-stream state with nats stream info <stream> --json | jq '.cluster' and per-group detail via curl -s http://localhost:8222/raftz.

3. Check the disk first. Network-attached storage with variable WAL write latency is the most common root cause. Raft log appends are latency-sensitive; when they stall, heartbeat processing stalls. During a storm, run iostat -x and watch await and %util on the JetStream device, and check dmesg for I/O errors or filesystem stall messages. If JetStream storage sits on network storage (EBS-class volumes, NFS), treat it as the prime suspect until proven otherwise.

4. Check CPU on every cluster member, not just the current leader. Election storms are usually a whole-cluster condition. Look for saturation, steal time on cloud VMs (oversubscribed hosts), and whether the node and core count are plausible for the number of Raft groups you are running. Thousands of streams and consumers means thousands of Raft groups, each with timers and apply loops; there is a real ceiling per node.

5. Check memory and GC. Growing RSS with deepening sawtooth peaks means GC pressure, and a long GC pause on the wrong goroutine delays heartbeats. Pull /varz mem over time and correlate election bursts with GC recovery drops.

6. Check the network between peers. Pull route RTT repeatedly from /routez on each node. Low average RTT does not rule out jitter; you are looking for variance and spikes that line up with election bursts. Cross-AZ deployments are more exposed here than same-rack ones.

7. Check for a cycling node. If one server keeps restarting (failing health checks during slow JetStream recovery, then being killed by the orchestrator), each restart forces leadership step-downs and re-elections. Compare uptime across nodes; a node with repeatedly resetting uptime during storms is a cause, not a victim.

8. Correlate with application-visible symptoms. During elections, /jsz shows api.inflight climbing and api.errors increasing; clients see publish timeouts and failed management calls. Confirming this correlation tells you the storm, not a separate storage or consumer problem, is what users are feeling. For the client-side symptom view, see NATS context deadline exceeded.

Metrics and signals to monitor

Fixes

Grouped by cause. Do not restart anything until you know which one you are dealing with.

Break the cycle first (any cause)

If the storm is self-sustaining and writes are broadly blocked, you have two levers:

• Reduce load. Pause or throttle non-critical publishers and any JetStream-heavy batch jobs. Fewer in-flight proposals and less WAL pressure gives Raft room to stabilize.
• Rolling restart, one node at a time. Restarting a single node temporarily reduces the active Raft group count on that node and can interrupt the feedback loop. Wait for the node to fully rejoin and for its peers to report current=true before touching the next one. Do not restart more than one node at a time; in a 3-node cluster, losing two members simultaneously loses quorum for every R=3 group, which converts a storm into an outage.

Slow storage: the WAL latency fix

Move JetStream storage to local SSDs. This is the durable fix for the most common cause and the playbook recommendation for production. If you cannot move off network-attached storage immediately, reduce the number of streams and consumers sharing the volume, and check for competing I/O: filesystem backups and snapshots on the same device cause exactly the kind of intermittent latency that produces sporadic (not continuous) flapping.

CPU starvation

Add cores, move nodes off oversubscribed hosts, or reduce the Raft group count by consolidating streams and deleting unused consumers. Note from the field: upgrading instance size alone does not always resolve storms when the group count is very high; the group count itself is the scaling dimension.

Network jitter

Co-locate cluster members (same AZ or at least low-variance paths), and investigate any route RTT variance found in diagnosis. There is no operator-tunable Raft election timeout in NATS; the fix is to remove the jitter, not to tune the protocol.

A cycling node

Fix the restart loop before anything else: generous readiness probe timeouts so JetStream recovery can finish, and check the orchestrator’s reason for killing the pod. Each cycle forces step-downs and elections, and the node rejoining under load makes the storm worse. See NATS crash loop: unexpected uptime resets and repeated restarts.

Prevention

• Local SSDs for JetStream storage. Treat network-attached storage for the Raft WAL as a known-risk configuration, not a default.
• Budget Raft groups. Track stream and consumer counts from /jsz as a capacity metric, the same way you track connections and storage. Growth here is growth in election surface area.
• Alert on election rate, not on “is JetStream up”. Elections give you minutes to hours of warning before availability degrades. Server-level health checks fire after the damage.
• Alert on the leading indicators: api.inflight trend, disk await on the JetStream device, route RTT variance, and per-node CPU steal. These move before the elections start.
• Size headroom for the worst window, not the average: deploys, backups, and recovery storms all stack on top of normal load. Keep CPU and IOPS comfortably below saturation.
• Watch leader distribution. Leadership should be roughly even across members. Concentration on one node concentrates write load and makes that node’s slowdown a cluster-wide election trigger.

How Netdata helps

• Election detection without log plumbing: Netdata tracks JetStream meta cluster state from /jsz, so leader churn and peer current/offline/lag status are visible as time series rather than something you grep for after the fact.
• API pressure correlation: api.errors, api.total, and api.inflight on one dashboard let you confirm in seconds that user-facing failures line up with election windows.
• Disk latency next to JetStream signals: per-device await and utilization charts alongside JetStream API metrics make the WAL-latency root cause visible as a correlation, not a guess.
• CPU, steal time, and memory per node: the three starvation sources (oversubscription, GC pressure, saturation) are charted per second, which matters because election-triggering pauses are short.
• Uptime resets across the cluster: a cycling node shows up immediately when uptime charts for all members are stacked.
• Route and connection health in the same view: since network jitter is a candidate cause, having route connectivity and system network metrics adjacent shortens the elimination step.

Related guides

• NATS JetStream consumer lag growing: falling behind the stream
• NATS consumer stalled at MaxAckPending: delivery stops until messages are acked
• NATS JetStream consumer stopped receiving messages: the diagnostic tree
• NATS context deadline exceeded: JetStream publish and request timeouts
• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check
• How NATS actually works in production: a mental model for operators
• NATS insufficient storage / maximum bytes exceeded: JetStream publishes rejected
• NATS JetStream API errors: reading the /jsz api.errors counter without false alarms
• NATS connection churn: a stable connection count hiding constant reconnects
• NATS connection storm: reconnect thundering herd after a network event
• NATS file descriptor exhaustion: too many open files and the ulimit cliff