$ guides / nats / nats-mirror-source-lag ▌

Operations Guides

NATS JetStream mirror and source lag: stale replicas and DR recovery point

A JetStream stream configured as a mirror of another stream, or sourcing from one or more upstream streams, keeps a local copy that is only as fresh as its sync connection. Stream info reports two fields per mirror or source: lag, the number of messages the local copy is behind, and active, the time since the last sync activity. When those numbers move the wrong way, the copy is going stale.

For a disaster recovery mirror this is not a performance nicety. The lag is your recovery point objective in real time. If the source cluster fails right now and you fail over to the mirror, every message counted in lag is data you do not have. A mirror that reports lag: 0 but has not synced in 20 minutes is arguably worse: it looks caught up while silently falling behind every second the source keeps accepting writes.

What this means

Mirror and source streams replicate asynchronously. The local stream pulls messages from the origin over whatever connectivity links the two: a gateway for cross-cluster mirrors, a route or direct connection for same-cluster setups. Asynchronous means the copy is always slightly behind by design, and the system tolerates the origin being unreachable for extended periods. Brief lag during bursts is normal. What you are hunting for is the pattern that does not recover.

Two failure shapes matter:

Steady or growing lag with recent activity. The sync connection is alive but cannot keep up. The source is publishing faster than the mirror can pull, or the path between them does not have the bandwidth. This degrades your RPO linearly with time.

Lag frozen or growing with stale active. The sync connection itself is broken. Nothing has flowed for longer than the reconnect cycle, which means a gateway drop, an auth or permission change, or an overloaded source that stopped serving the sync consumer. This is the dangerous one for DR: your effective RPO is now “time since active last moved plus current lag,” and it grows until someone notices.

Working thresholds:

lag > 0 sustained for more than 1 minute: warning. Investigate before it compounds.
active greater than 30 seconds: the sync connection may be broken. Treat as a probable fault, not a blip.

Mirror and source lag are distinct from Raft replica lag (cluster.replicas[].lag), which measures replication between peers of the same stream inside one cluster. Mirror/source lag measures a copy of a stream that lives somewhere else. Both can be non-zero at once, and the fixes are different. For how JetStream replication and Raft fit together, see How NATS actually works in production.

Common causes

Cause	What it looks like	First thing to check
Gateway disconnection (cross-cluster mirror)	`active` climbing past 30s, lag frozen then growing	`/gatewayz` on both sides: expected outbound/inbound gateway present?
Overloaded source stream or server	Lag grows steadily, `active` recent but sync rate below publish rate	Source server CPU, disk I/O, JetStream API inflight via `/jsz`
Insufficient bandwidth on the sync path	Lag tracks publish rate minus a ceiling; worse during bursts	Network throughput between clusters vs stream byte rate
Auth or permission change breaking the mirror	`active` jumps to stale right after a config or credential change	Server logs for authorization violations on the sync subjects
Cross-account or leaf node setup missing flow control exports	Mirror stalls after an initial burst of messages, `active` goes stale	Account export/import config for flow control subjects
Source stream content changing too fast (rapid publish-then-delete)	Inconsistent or laggy replication despite healthy connectivity	Whether producers delete or purge messages shortly after publishing

Quick checks

All of these are read-only.

# Mirror lag and last sync activity for a mirror stream
nats stream info MIRROR_STREAM --json | jq '.mirror | {name, lag, active}'

# Per-source lag and activity for a sourcing stream
nats stream info SOURCED_STREAM --json | jq '.sources[] | {name, lag, active}'

# Replication report across streams <!-- TODO: verify nats stream report shows Active/Lag per source and mirror in current CLI versions -->
nats stream report

# Gateway connectivity on the cluster hosting the mirror
curl -s http://localhost:8222/gatewayz | jq '{outbound: (.outbound_gateways | keys), inbound: (.inbound_gateways | keys)}'

# Is the source server itself healthy and not saturated?
curl -s http://localhost:8222/varz | jq '{cpu, mem, connections, slow_consumers}'
curl -s http://localhost:8222/jsz | jq '{api_total: .api.total, api_errors: .api.errors, inflight: .api.inflight}'

# Publish rate on the origin stream: is it outpacing the sync?
nats stream info ORIGIN_STREAM --json | jq '.state | {messages, bytes, first_seq, last_seq}'

Take the lag reading twice, a minute apart. Lag that decreases between samples is a sync that is catching up. Lag that increases is a sync losing ground. Lag that never changes while active grows is a dead connection.

How to diagnose it

flowchart TD
  A[Mirror or source lag detected] --> B{active recent, under 30s?}
  B -- Yes --> C{Lag shrinking between samples?}
  B -- No --> D[Sync connection broken]
  C -- Yes --> E[Transient burst, keep watching]
  C -- No --> F[Throughput problem]
  D --> G[Check gatewayz for missing gateway]
  D --> H[Check logs for auth or permission violations]
  D --> I[Check cross-account or leaf flow control setup]
  F --> J[Compare source publish rate to link bandwidth]
  F --> K[Check source server CPU, disk, api inflight]

Read lag and active for the stream. Use nats stream info with jq as above. Note both numbers and the current time.
Decide which failure shape you have. If active is over 30 seconds, go to step 3: the sync connection is suspect. If active is fresh but lag grows, go to step 5: it is a throughput problem.
Check the connectivity layer. For cross-cluster mirrors, verify the expected gateway exists in /gatewayz on both clusters. A missing outbound gateway means the remote cluster is unreachable and the mirror cannot sync at all. For leaf node or cross-account topologies, verify the leaf connection in /leafz.
Check for auth and permission breakage. If the mirror went stale right after a credential rotation or config change, grep the server logs on the origin side for authorization violations. A mirror is a consumer of the origin stream; anything that revokes its read access kills the sync silently from the mirror’s point of view. Also verify account exports and imports are intact if the mirror crosses accounts.
Measure the throughput gap. Compare the origin stream’s publish rate against the observed sync rate and against the bandwidth of the link between the clusters. If the publish rate exceeds what the path can carry, lag grows no matter how healthy everything is. Bursts that outrun the link are fine as long as lag drains afterward; a permanent deficit is not.
Check the source server’s headroom. High CPU, disk I/O saturation, or elevated JetStream api.inflight on the origin means it is slow to serve the internal sync consumer. The fix belongs on the source, not the mirror.
Confirm recovery. After remediation, lag should fall to zero and active should stay in the low seconds. If active cycles between fresh and stale, the connection is flapping: look at gateway RTT and slow consumer events on the gateway or route connections.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Mirror/source `lag`	This is your DR recovery point in messages	> 0 sustained over 1 minute, or growing between samples
Mirror/source `active`	Time since last sync activity; detects dead sync connections	> 30 seconds
Gateway presence (`/gatewayz`)	Cross-cluster mirrors die when the gateway drops	Expected gateway missing for > 60s
Gateway or route slow consumers (`slow_consumer_stats`)	A slow gateway backs up all cross-cluster traffic including mirror sync	Non-zero gateway slow consumer events
Source server `api.inflight` and CPU	An overloaded source serves sync consumers slowly	Sustained elevation above baseline
Link throughput vs origin publish byte rate	Tells you if lag is a bandwidth deficit	Publish rate persistently above achievable sync rate

If you run the Prometheus exporter, it exposes per-source lag and inactivity gauges (jetstream_stream_source_lag and jetstream_stream_source_active_duration_ns), which are the right basis for alerting.

Two replication semantics worth knowing before you rely on a mirror for DR:

Deletes in the origin stream are not replicated through a mirror or source agreement. Retention-driven expiry and explicit deletes on the origin do not propagate the same way, so message counts can legitimately differ.
Quickly deleting messages after publishing on the origin can produce inconsistent replication, because replication is asynchronous and may not have copied the message before it disappeared. If your producers purge aggressively, a mirror is not a reliable copy.

Fixes

Gateway or connectivity restored. Re-establish the gateway or leaf connection. The mirror recovers on its own once the path is back; expect the sync to take some seconds to re-establish, then lag should drain. If it does not, restart the diagnosis at the auth step. Do not restart servers as a first move: a broken gateway on one side does not need a cluster restart to fix.

Auth or permission change. Restore the read access the mirror’s internal consumer needs on the origin subjects, or roll back the offending credential change. This class of failure is silent on the mirror side, so add it to your change checklist for any permission edit touching JetStream accounts.

Bandwidth deficit. If the origin publish rate exceeds what the inter-cluster link can sustain, no tuning fixes this. Either reduce what the mirror must carry (filter subjects on the source configuration so it only syncs what DR actually needs), provision more bandwidth, or accept a larger RPO explicitly and monitor against it.

Overloaded source. Relieve the origin server: move stream leaders, add capacity, or reduce ingest. Mirror lag here is a symptom of source distress, so fix the source and the mirror follows.

Broken mirror that cannot resync cleanly. As a last resort, recreating the mirror stream forces a full re-sync from the origin’s current state. This is disruptive: the mirror is unavailable or incomplete during the rebuild, and any failover plan depending on it is degraded until lag returns to zero. Treat it like restoring a backup, not like a service restart.

Prevention

Alert on lag and active, not just on gateway state. Gateways can be up while the mirror is stale for auth reasons. The mirror’s own fields are the only ground truth for freshness.
Define your RPO as a lag budget. If your DR plan promises 60 seconds of data loss, alert when lag exceeds 60 seconds worth of messages at the current publish rate, and page when active exceeds 30 seconds.
Test failover against the mirror periodically. A mirror you have never failed over to is a hypothesis, not a DR plan. Verify the copy is usable and that you know the lag at the moment of failover.
Include mirror permissions in credential change review. Any change to accounts, exports, imports, or JetStream permissions on the origin side should be checked against what mirrors and sources consume.
Size the inter-cluster link for peak publish rate, not average. Async replication absorbs bursts only if there is headroom to catch up afterward.

How Netdata helps

Netdata’s NATS collector polls the HTTP monitoring endpoints, so gateway presence, route health, slow consumer breakdowns, and JetStream API errors sit on the same dashboard as the server-level signals that explain mirror lag.
Correlating slow_consumer_stats for gateways with the time a mirror went stale distinguishes a bandwidth or backpressure problem from an auth problem without log diving.
JetStream api.inflight and api.errors trends on the origin server show whether a lagging mirror is a symptom of source overload.
Connection and throughput charts on both clusters let you compare publish rate against link capacity, the deciding evidence for a bandwidth deficit.
Netdata does not currently collect per-stream mirror/source lag fields directly. Pair it with the nats stream info checks above or the Prometheus exporter gauges for the lag itself, and use Netdata for the surrounding cause.