NATS JetStream replica lag: a non-current replica that would lose data on failover

A replicated JetStream stream shows current: false with a non-zero lag for one of its replicas in nats stream info. Everything still works: publishes succeed, consumers receive messages, no alerts fire on server health. But the replication factor you configured is not the replication factor you have. If the leader fails right now, a failover to the lagging replica loses every message the leader acknowledged but the replica has not yet written.

This is a dangerous state because it is silent. The stream leader keeps serving traffic, so there is no user-facing symptom. The only place the risk shows up is the per-replica state in stream info and the per-stream Raft group state. This guide covers how to read those fields, decide whether you have a transient catch-up or a real problem, find the cause, and fix it without making things worse.

What this means

Each JetStream stream with replicas > 1 has its own Raft group. The leader appends entries, replicates them to followers, and acknowledges a publish once a quorum of replicas has the entry. The per-replica fields in stream info describe the follower side of that contract:

• current (bool): whether the replica is fully caught up with the leader.
• lag (int): how many messages behind the leader this replica is.
• active (duration): time since the leader last heard from this replica.
• offline (bool): whether the replica is unreachable entirely.

The combinations that matter:

Two severity rules from the field:

• A single non-current replica is a ticket-level problem. Investigate it, but quorum still protects you.
• More than one non-current replica, or an offline replica in a cluster with no remaining quorum redundancy, is page-level. You are one leader failure away from data loss or an unwritable stream.

Thresholds that separate normal from abnormal: lag > 0 sustained for more than 30 seconds is abnormal, active greater than about 5 seconds means the replica has not been heard from recently, and offline: true for more than a minute is a real problem, not a network hiccup. Transient lag during a publish burst is expected; lag that persists is not.

flowchart TD
  A[nats stream info: replica current=false, lag>0] --> B{Offline?}
  B -->|yes, over 1 min| C[Replica down or partitioned: check process, routes, network]
  B -->|no| D{Lag clearing over 30s?}
  D -->|yes| E[Transient catch-up: watch until current=true]
  D -->|no| F[Check replica disk I/O and route backpressure]
  F --> G{iostat await high on replica?}
  G -->|yes| H[Disk bottleneck on replica]
  G -->|no| I{routez pending_size growing?}
  I -->|yes| J[Congested route between servers]
  I -->|no| K[Perpetual 1-2 msg lag: subtle timing or I/O issue, treat as real]

Common causes

The last row is the one people overreact to. A replica that just restarted or rejoined the cluster will be behind by definition. The question is whether it is converging. If lag decreases steadily, leave it alone. If it stays flat or grows, you have one of the first four problems.

Quick checks

All of these are read-only.

# Per-replica state for the stream (run on any server in the cluster)
nats stream info ORDERS --json | jq '.cluster | {leader, replicas: [.replicas[] | {name, current, lag, active, offline}]}'

# Watch the trend: is lag converging or stuck?
watch -n 5 'nats stream info ORDERS --json | jq ".cluster.replicas[] | {name, current, lag, active, offline}"'

# Meta cluster health: is the JetStream metadata Raft group stable?
curl -s http://localhost:8222/jsz | jq '.meta_cluster | {leader, replicas: [.replicas[]? | {name, current, offline, lag}]}'
# /jsz meta state lives under cluster.{name,leader,replicas}; per-Raft-group detail appears
# under the meta/streams accounts sections depending on the accounts argument

# Per-Raft-group detail, including per-stream groups
curl -s http://localhost:8222/raftz | jq '.'

# Route health and per-route backpressure
curl -s http://localhost:8222/routez | jq '.routes[] | {rid, ip, rtt, pending_size}'

# Route count: in an N-node cluster each server should have N-1 routes
curl -s http://localhost:8222/varz | jq '.routes'

# Disk latency on the lagging replica's node (JetStream storage device)
iostat -x 5 3

# Kernel-level storage errors on the replica
dmesg | grep -iE 'i/o error|ext4|xfs' | tail -20

# JetStream API pressure: high inflight correlates with Raft or disk slowness
curl -s http://localhost:8222/jsz | jq '{api_total: .api.total, api_errors: .api.errors, inflight: .api.inflight}'

Two notes on interpretation. First, lag is measured in messages, not bytes. A lag of 500 on a stream of 200-byte telemetry messages is very different from a lag of 500 on a stream of 1 MB payloads. Size the risk before you prioritize. Second, nats stream info is a point-in-time snapshot. A single sample is not a diagnosis; the trend over 30-60 seconds is.

How to diagnose it

1. Confirm the state and the scope. Run nats stream info <stream> --json and record current, lag, active, and offline for every replica. Then check your other replicated streams. If one replica on one stream is behind, the problem is likely stream- or node-specific. If many streams show the same replica lagging, the problem is that server: its disk, its network path, or the server itself.

2. Classify: offline, stuck, or converging. offline: true for over a minute points at a down or partitioned server. current: false with lag trending down is normal catch-up. current: false with lag flat or growing is a live replication problem. The watch command above answers this in under a minute.

3. Check the replica’s server. Is the process up? Check uptime on that node (curl -s http://<replica>:8222/varz | jq .uptime). An unexpected uptime reset near the time lag started means the server restarted and is catching up. If it never catches up, keep going.

4. Check route connectivity and backpressure. From the leader’s node, look at /routez. The route toward the lagging peer should exist, and its pending_size should be at or near zero. Sustained non-zero pending_size on a route means inter-server traffic is backing up, which starves Raft replication just as it starves message routing. Elevated route rtt relative to baseline points the same direction. A route slow consumer event (check /varz slow_consumer_stats.routes) is the loud version of this problem.

5. Check disk I/O on the replica. Run iostat -x 5 3 on the lagging node and look at await and %util for the device holding the JetStream store. Raft log appends and stream writes are latency-sensitive; a replica that cannot fsync fast enough cannot keep up with the leader. High iowait with plenty of free space is the disk stall pattern, distinct from storage exhaustion. Also check for backup or snapshot processes running against the store, a classic cause of transient replication lag.

6. Check for the silent-catastrophe case. If the replica shows a perpetual lag of 1-2 messages that never clears, do not dismiss it because the number is small. This pattern means the replica is structurally just behind the leader at all times, and any failover to it will always lose the most recent writes. It usually traces to a subtle I/O or timing issue: slightly slow fsync, a congested route that adds just enough delay, or compaction timing. Treat it as a real fault, not noise.

7. Decide severity before acting. One non-current replica out of three: ticket, work through causes. Two non-current or any offline replica without remaining quorum redundancy: page, because the next leader failure either loses data or makes the stream unwritable.

Metrics and signals to monitor

One caveat: a replica that reports current: true tells you it is caught up on sequence, not that its on-disk state has been verified message-for-message against the leader. There are community reports of replicas reporting current while holding divergent data after recovery. For streams where this matters, periodic snapshot-and-compare or restore testing is the honest check.

Fixes

Replica server down or unreachable

Bring the node back and let it catch up. Verify the process, then routes: in an N-node cluster each server should have N-1 routes, and a missing route toward the replica means a partition, not a slow disk. Check firewalls, security groups, and DNS if routes will not establish. Once the node is back and routes are up, watch lag trend to zero. Do not assume catch-up succeeded; confirm current: true.

If the node is permanently dead, it still counts toward the stream’s configured replication factor. Replacing it means removing the dead peer from the stream’s Raft group and letting the cluster place a new replica. The nats stream cluster family of commands includes peer removal and leader step-down operations; peer removal is disruptive to the target node’s JetStream assets, so read the current JetStream clustering administration docs for your server version before running it.

Disk I/O bottleneck on the replica

This is the most common root cause and the one with the clearest fix: JetStream storage belongs on local SSDs. Network-attached storage with variable latency is the leading cause of Raft instability generally, and of replica lag specifically, because the Raft log append path is latency-sensitive.

Short term: find what is competing for the disk. Filesystem backups and snapshots against the store cause exactly this symptom, and the fix is scheduling, not hardware. Reduce the number of streams sharing the device or the publish rate if the device is simply undersized.

Long term: move the store to faster local storage. Also check whether the lagging node is carrying an unfair share of stream leaders plus replicas; an overloaded node lags on its follower duties too.

Congested route

If /routez shows growing pending_size or elevated RTT toward the replica, replication traffic is queueing behind route traffic. Check overall byte throughput (in_bytes, out_bytes rates against NIC capacity) and look for what changed: a traffic burst, a new high-fan-out subject, or cross-server replication multiplying network load (replication costs roughly R times the message size in network traffic). Relieve the congestion first; the Raft lag will clear on its own once the route drains. If routes are flapping, treat that as its own incident before chasing the lag.

Perpetual 1-2 message lag

Do not ignore this one. Work through the disk and route checks above on that specific replica, because the cause is real even if the number is small. If nothing shows, look at Raft log compaction and server logs on the replica for warnings around applying entries. A replica that never quite catches up is a failover that will always lose your newest messages.

Prevention

• Alert on the replica state, not just server health. Server up does not mean replication is working. Alert when any replica of an R>1 stream shows current: false or lag > 0 for more than 30 seconds, active over about 5 seconds, or offline: true for more than a minute.
• Alert on the silent-catastrophe pattern separately. A replica with lag pinned at 1-2 messages for hours will not trip a “lag is high” threshold. Track time-since-last-fully-current per replica, not just lag magnitude.
• Local SSDs for JetStream stores. This single decision removes the largest class of replication lag causes.
• Monitor route backpressure as a leading indicator. Route pending_size and route slow consumer events fire before stream replicas fall visibly behind. Watch the precursor.
• Watch leader and replica distribution. A node that is leader for most streams plus follower for the rest will lag first under load.
• Size network for replication. Replication multiplies stream traffic by the replica count on the wire between servers. Keep total throughput under about 70 percent of link capacity so bursts do not starve Raft.
• Test failover, do not assume it. Periodically verify that a step-down on a non-current stream behaves the way you expect, and that your runbook distinguishes “replica exists” from “replica is safe to promote.”

How Netdata helps

• Netdata polls the NATS HTTP monitoring endpoints (/varz, /jsz, /routez) so you can correlate server-level signals with replication state on one timeline: route RTT and pending, API inflight and errors, memory, and uptime resets.
• Uptime tracking makes the “replica restarted and is catching up” case obvious instead of something you reconstruct from logs.
• Per-node disk I/O and iowait charts from the Netdata agent on each NATS host let you see which replica’s storage is too slow without SSHing into three boxes during an incident.
• Route health and slow consumer counters expose inter-server backpressure, the leading indicator that a replica is about to fall behind.
• Alerting on sustained conditions (lag persisting, routes degraded, disk latency elevated) catches the quiet drift cases, including the perpetual small-lag pattern that threshold-on-magnitude alerts miss.

Related guides

• NATS JetStream consumer lag growing: falling behind the stream
• NATS context deadline exceeded: JetStream publish and request timeouts
• NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check
• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS insufficient storage / maximum bytes exceeded: JetStream publishes rejected
• NATS connection storm: reconnect thundering herd after a network event
• How NATS actually works in production: a mental model for operators