$ guides / nats / nats-meta-cluster-leader-flapping ▌

Operations Guides

NATS JetStream meta cluster leader flapping: admin operations that keep failing

Stream and consumer administration is failing, but existing JetStream traffic mostly continues. Creating a stream times out, consumer updates return errors, and retries sometimes succeed only to fail again a minute later.

This pattern points at the JetStream meta Raft group. The meta group manages cluster-wide JetStream metadata: stream and consumer create, update, and delete operations. Existing streams use separate Raft groups for replicated data, so the data plane can remain available while the administrative plane is unstable.

A single leader change during maintenance is not an incident by itself. More than one meta leader change in five minutes is concerning, especially when peers report current=false, offline=true, or persistent replication lag.

What this means

Every clustered JetStream deployment has one meta Raft group for cluster metadata, plus separate Raft groups for replicated streams and consumers. This distinction drives the diagnosis:

An unstable meta leader breaks administrative operations across JetStream.
An unstable stream Raft group affects writes and replication for that stream only.
Both can flap together when the root cause is CPU pressure, disk latency, network instability, or a wider Raft election storm.

During a meta election, an administrative request can reach a server that no longer leads the meta group, wait for consensus that never completes, or fail while leadership moves. Client retries then inflate JetStream API traffic and make the failure look larger than the original event.

flowchart LR
  A[Admin request] --> B[Meta Raft leader]
  B --> C{Quorum commits metadata}
  C -->|Stable leadership| D[Create update delete succeeds]
  B -->|Heartbeat or election failure| E[Leader change]
  E --> F[Admin request times out or errors]
  G[Existing stream Raft groups] --> H[Data plane often continues]
  E -. May also trigger .-> G

The signal that matters is not just the current leader name. It is the rate of leader changes, whether all expected peers are present, and whether each peer is current.

Common causes

Cause	What it looks like	First thing to check
Network latency or partition	Route count drops, route RTT rises, or route pending bytes grow. Leader changes correlate with route events.	Compare route count with the expected `N-1` routes and inspect `/routez`.
Meta leader overload	The current or recent leader has high CPU, memory pressure, or GC pauses. Elections follow resource spikes.	Check `/varz` CPU, memory, uptime, and throughput on every node.
JetStream disk I/O stall	`api.inflight` stays high, API errors rise, and OS-level iowait or disk latency increases.	Check disk latency and look for backups, snapshots, or noisy neighbors.
Lagging or offline meta peer	One or more `meta_cluster.replicas[]` entries report `current=false`, `offline=true`, or sustained lag.	Inspect `/jsz` meta cluster state from more than one server.
Raft election storm	Many stream groups change leaders along with the meta group. CPU and API errors spike across nodes.	Inspect `/raftz` and server logs for repeated leader changes.
Clock or timer instability	Elections occur without a clear route, disk, or CPU cause.	Verify time synchronization on all JetStream nodes.
Expected maintenance	One controlled restart or deployment causes one leader change, then the cluster stabilizes.	Compare the event time with deployment and restart history.

Do not treat every API error as proof of meta instability. Idempotent create or bind operations can produce benign errors, and client retries inflate both api.total and api.errors. Confirm the leader is actually changing before treating this as a Raft incident.

Quick checks

Run these from a host that can reach the NATS monitoring port. They are read-only. The default monitoring port is 8222; use the port configured for your deployment.

# 1. Show the current meta leader and peer state
curl -s http://localhost:8222/jsz | jq '.meta_cluster | {leader, replicas: [.replicas[]? | {name, current, offline, lag}]}'

# 2. Poll for leader changes over two minutes
for i in $(seq 1 12); do
  date -u +%H:%M:%S
  curl -s http://localhost:8222/jsz | jq -r '.meta_cluster.leader // "no-meta-cluster"'
  sleep 10
done

# 3. Check JetStream API pressure and errors
curl -s http://localhost:8222/jsz | jq '{api_total: .api.total, api_errors: .api.errors, api_inflight: .api.inflight}'

# 4. Check route count, route RTT, and route backpressure
curl -s http://localhost:8222/routez | jq '{num_routes, routes: [.routes[]? | {remote_id, ip, rtt, pending_size}]}'

# 5. Check server CPU, memory, uptime, and connection pressure
curl -s http://localhost:8222/varz | jq '{cpu, cores, mem, uptime, connections, slow_consumers}'

# 6. Check whether JetStream itself is enabled
curl -s http://localhost:8222/jsz | jq '{disabled, streams, consumers, memory, storage}'

# 7. Inspect all Raft groups when the problem may extend beyond metadata
curl -s http://localhost:8222/raftz | jq .

# 8. Check disk latency and utilization
iostat -x 1 5

# 9. Look for repeated election events in the server log
grep -Ei 'new (metadata )?leader|stepping down' /var/log/nats/nats-server.log | tail -50

Notes on these checks:

Expected route count is N-1 for a full mesh of N nodes. The previous version of this check read a cluster_size field from /varz; use num_routes from /routez instead and compare against your known node count.
On standalone JetStream, meta_cluster is null because there is no clustered metadata Raft group.
/raftz output can be large on clusters with many streams. Use it after /jsz indicates a wider Raft problem.
iostat requires the sysstat package on most Linux distributions.
The log path varies by deployment; adjust it to where your nats-server writes logs.
Run the /jsz check on more than one node. A single server can have a stale or incomplete view while it is catching up.

How to diagnose it

1. Confirm that administration, not the whole server, is failing

Use a safe read operation through the JetStream API:

# Read stream state through the JetStream API
nats stream report

If this fails while /healthz?js-server-only=true remains healthy and existing publish or consume paths continue, the evidence points at the administrative plane. If basic server health is failing too, widen the investigation beyond the meta group.

2. Measure the leader change rate

Poll meta_cluster.leader for at least five to ten minutes. One change can be a clean failover or a maintenance event. Repeated changes, particularly more than one in five minutes, indicate active instability.

Record:

Old and new leader names.
Time of each change.
Whether the same node repeatedly loses leadership.
Whether changes align with deployments, backups, network events, or CPU spikes.

3. Check whether the meta group has all expected peers

Inspect every meta_cluster.replicas[] entry:

current=false means the peer is behind.
offline=true means the peer is unreachable or not participating.
lag shows replication lag in entries.

A peer briefly behind during a burst can recover. A peer offline for more than 60 seconds, or one whose lag never resolves, reduces the group’s fault tolerance and can prevent stable consensus.

4. Separate meta flapping from per-stream flapping

Use /raftz to inspect individual Raft groups. If only the meta group is changing leaders, existing replicated streams may continue to serve normally. If many stream groups are electing at the same time, you have a broader election storm and should expect publish failures or replication stalls too.

This distinction determines the blast radius. Do not assume the data plane is safe just because messages are flowing on one stream.

5. Correlate API errors with inflight requests

A rising error count with low inflight can be client-side or benign retry noise. Rising errors with sustained high api.inflight is stronger evidence that JetStream cannot complete requests, usually because consensus or disk writes are slow.

Calculate the error ratio over a fixed interval rather than alerting on cumulative counters:

# Compare API totals over a 60 second interval
curl -s http://localhost:8222/jsz | jq '{total: .api.total, errors: .api.errors}'
sleep 60
curl -s http://localhost:8222/jsz | jq '{total: .api.total, errors: .api.errors}'

A sustained error ratio above 5 percent indicates a systemic issue. Aggressive client retries can amplify both counters, so cross-check against inflight before concluding the server is at fault.

6. Inspect the nodes that recently held leadership

For the current and previous leaders, check:

CPU saturation and abrupt CPU spikes.
Memory growth and GC pauses.
Disk latency, iowait, and filesystem activity.
Route RTT and route pending bytes.
Unexpected uptime resets.

The server that loses leadership is often the node under pressure, but the root cause can also be another peer that cannot vote or append Raft entries quickly enough.

7. Look for a wider election storm

Search logs for alternating leader events and check whether stream groups are also unstable. A broad election storm can become self-reinforcing: resource pressure causes elections, elections increase CPU and disk work, and that additional work causes more elections.

If many groups are involved, reduce load and fix the shared resource problem before attempting administrative changes.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Meta leader name	Tracks elections in the metadata control plane.	More than one change in five minutes.
Meta peer `current`	Shows whether each peer has caught up with the leader.	Any peer remains `current=false`.
Meta peer `offline`	Shows whether a peer is participating in consensus.	Any peer remains offline for more than 60 seconds.
Meta peer `lag`	Shows replication lag in Raft entries.	Lag persists or grows instead of returning to zero.
JetStream API inflight	Indicates requests waiting on JetStream processing, consensus, or storage.	Sustained elevation above baseline.
JetStream API error ratio	Confirms that administration or publish requests are failing.	Errors above 5 percent of API requests over a sustained interval.
Route count	Verifies the full cluster mesh.	Fewer than `N-1` routes for more than 60 seconds.
Route RTT and pending size	Detects network delay and inter-server backpressure.	Sustained RTT increase or nonzero route pending size.
Server CPU and memory	Finds leader overload and GC pressure.	CPU above 90 percent for five minutes or monotonic memory growth.
Disk latency and iowait	Finds storage stalls that delay Raft and JetStream writes.	Persistent latency or iowait elevation during elections.
Server uptime	Detects restarts that trigger elections.	Unexpected resets or repeated restarts across nodes.

Fixes

Repair route and network instability

Restore the expected full-mesh route count and investigate latency between JetStream nodes. Check network device events, firewall changes, DNS resolution, and cross-zone or cross-region paths.

Do not restart NATS nodes just because a route briefly disconnected. Routes reconnect automatically, and unnecessary restarts create more elections.

Relieve pressure on the current leader

If elections follow CPU or memory spikes, reduce pressure on the affected node. Pause noncritical publishers or administrative jobs, investigate noisy neighbors, and add CPU or memory capacity where the node is undersized.

Evaluate memory as a trend. Go garbage collection creates a sawtooth pattern, so a temporary rise followed by a drop is normal. Monotonic growth without recovery is not.

Remove the disk bottleneck

If api.inflight, disk latency, and iowait rise together, look for filesystem backups, snapshots, disk degradation, or other tenants sharing the storage device. Pause avoidable I/O work where possible.

For a persistent bottleneck, plan a migration to lower-latency local storage. Network-attached storage with variable latency is a common contributor to Raft instability. Treat storage migration as a planned change, not an in-incident experiment.

Restore unhealthy meta peers

Bring offline or lagging peers back before making cluster membership changes. Verify that routes are healthy, JetStream is enabled, disk latency is normal, and the peer has caught up.

Avoid decommissioning or replacing a node while the meta group is unstable. Membership operations during an active quorum or election problem can make recovery harder.

Break a wider election storm carefully

First reduce incoming load and correct the shared resource problem. If the storm remains self-sustaining, a rolling restart may reduce pressure, but it is disruptive and can trigger more stream and meta elections.

If you use it as a last resort:

Restart one node at a time.
Wait for routes to return.
Verify JetStream is enabled.
Verify meta peers are present, current, and not offline.
Check per-stream Raft health before continuing.

Do not restart the whole cluster at once.

Repair time synchronization

If no resource or network cause explains repeated elections, verify clock synchronization on all JetStream nodes. Correct the time source, then watch whether election frequency returns to normal.

Be cautious with election timeout tuning

Operational guidance differs on whether Raft timing should be tuned, and behavior can be version-dependent. Do not assume there is a supported election timeout configuration for your exact server version. Verify the setting and its semantics against the deployed version before changing it. Fixing network, CPU, or disk delay is usually safer than masking it with longer timeouts.

Prevention

Leader change alerting: Alert when the meta leader changes more than once in five minutes. This threshold separates normal isolated failovers from active flapping.
Peer health alerting: Alert when a meta peer stays current=false or offline=true for more than 60 seconds. A reduced peer set leaves less room for another failure.
API correlation: Track API errors together with api.inflight. Error count alone includes benign retries; errors plus inflight better indicate stalled consensus or storage.
Route health monitoring: Compare current routes with the expected N-1 count and monitor route RTT and pending size. Route instability is a common upstream cause of elections.
Disk headroom: Keep disk utilization below 80 percent and preserve IOPS headroom. JetStream and Raft both become unstable when storage latency rises.
Leader distribution review: Check whether one server leads a disproportionate number of stream groups. Concentrated leadership increases the impact of a single overloaded node.
Deployment correlation: Record restarts, configuration changes, backups, and infrastructure events alongside leader changes. This shortens the search for the trigger.
Health check separation: Use /healthz?js-server-only=true for basic server readiness and treat bare /healthz JetStream failures carefully during recovery. This avoids confusing asset recovery with a process outage.

How Netdata helps

Netdata’s NATS monitoring can surface JetStream API errors, inflight requests, storage usage, stream counts, and consumer counts, helping confirm that administrative pressure is rising.
Server CPU, memory, uptime, throughput, connection, and slow-consumer signals help identify whether the node losing leadership is overloaded or restarting.
Host-level disk utilization, latency, and iowait can be correlated with election times to distinguish a storage stall from a network problem.
Route and connection signals help show whether the meta election follows inter-server connectivity loss rather than a JetStream-specific fault.
Comparing nodes side by side helps identify whether one peer is consistently behind while the rest of the cluster remains healthy.
Netdata collects the meta_cluster structure from /jsz but may not expose leader, peer status, or peer lag as dedicated metrics. Keep a separate /jsz poll or log-based watch for those fields.