$ guides / nats / nats-consumer-lag-growing ▌

Operations Guides

NATS JetStream consumer lag growing: falling behind the stream

The NATS server is healthy. /healthz returns ok, throughput looks normal, CPU and memory are fine. But one stream keeps growing, and the consumer attached to it is not keeping up. num_pending climbs minute after minute, and every dashboard that only watches server-level metrics shows green.

Consumer lag requires per-consumer polling, not a single server endpoint, so most monitoring setups never see it. The result: a stream with millions of pending messages, effectively down while the server looks healthy.

This article covers how to measure lag correctly, the states that produce growing lag, and the fix for each.

What this means

A JetStream consumer tracks two cursors against the stream’s sequence numbers: what it has delivered and what has been acknowledged. Lag is the distance between where the stream is and where the consumer is.

For an unfiltered consumer:

lag = stream last_seq - consumer delivered.stream_seq

For a filtered consumer (one with a FilterSubject), raw sequence difference overstates lag because the consumer only cares about a subset of messages. Use the consumer’s own num_pending field, which counts only messages matching the filter.

Raw message counts are the wrong unit for alerting. 50,000 pending messages is nothing at 100,000 msg/s and catastrophic at 10 msg/s. Express lag as time behind:

seconds_behind = lag_messages / message_rate_per_second

More than about 5 minutes behind on a latency-sensitive workload is typically concerning. Batch and replay consumers are the exception: they are expected to carry high pending counts, so do not alert on them the same way.

Three per-consumer fields tell you which kind of trouble you are in:

num_pending: messages available for delivery but not yet delivered. Growing means the consumer is falling behind producers.
num_ack_pending: messages delivered but not yet acknowledged. Growing means the consumer received messages but is not acking, either because processing stalled or AckWait is too short.
num_redelivered: messages redelivered after ack timeout. Growing means a NAK/redelivery loop or a consumer crashing mid-processing.

Common causes

Cause	What it looks like	First thing to check
Consumer too slow	`num_pending` grows steadily, `num_ack_pending` stays moderate, processing keeps up partially	Consumer application CPU, downstream latency, publish rate vs ack rate
Disconnected but registered consumer	`num_pending` grows, `num_ack_pending` flat or zero, no deliveries happening	Whether the consumer application is actually running and connected
Stalled at MaxAckPending	`num_ack_pending` pinned at the consumer’s `MaxAckPending` limit, delivery stopped, stream grows silently	Compare `num_ack_pending` to configured `MaxAckPending`
NAK/redelivery loop	`num_redelivered` climbing, same messages cycling, lag never drains	Consumer error logs, whether processing fails partway and NAKs
AckWait too short	Messages redelivered while still being processed, duplicates, `num_redelivered` growing	`AckWait` vs actual p99 processing time

The MaxAckPending stall deserves emphasis. When num_ack_pending reaches the configured limit (default 1000 for consumers with explicit acks), the server stops delivering new messages until some pending messages are acked. No error is raised anywhere. The consumer looks fine, the server looks fine, and the stream grows silently. A pull consumer showing num_ack_pending == MaxAckPending with num_waiting == 0 is full and not requesting more: delivery is stalled.

Quick checks

All of these are read-only.

# Per-consumer state across the whole server (costlier on servers with many consumers)
curl -s 'http://localhost:8222/jsz?consumers=true' | \
  jq '.account_details[].stream_detail[].consumer_detail[] | {name, num_pending, num_ack_pending, num_redelivered}'

# One consumer in detail
nats consumer info STREAM_NAME CONSUMER_NAME --json | \
  jq '{pending: .num_ack_pending, waiting: .num_waiting, redelivered: .num_redelivered, ack_floor: .ack_floor.consumer_seq}'

# Compute delivery lag for an unfiltered consumer
STREAM="mystream"; CONSUMER="myconsumer"
LAST=$(nats stream info $STREAM --json | jq '.state.last_seq')
DELIVERED=$(nats consumer info $STREAM $CONSUMER --json | jq '.delivered.stream_seq')
echo "Lag: $(( LAST - DELIVERED )) messages"

# Stream growth: is first_seq advancing or is the stream just accumulating?
curl -s "http://localhost:8222/jsz?streams=true" | \
  jq '.account_details[].stream_detail[] | {name: .name, messages: .state.messages, bytes: .state.bytes, first_seq: .state.first_seq, last_seq: .state.last_seq}'

# Is the server itself under stress? (rules out server-side causes)
curl -s http://localhost:8222/jsz | jq '{api_total: .api.total, api_errors: .api.errors, inflight: .api.inflight}'

/jsz?consumers=true returns substantially more data than the bare endpoint and is costlier on servers with many consumers. For routine monitoring, poll it at a modest interval; for incident triage, query the one consumer you care about via the CLI.

How to diagnose it

Work through these in order. Each step eliminates one of the lag states.

Confirm lag is real and growing. Sample num_pending (filtered) or last_seq - delivered.stream_seq (unfiltered) two or three times, 30 seconds apart. A single snapshot proves nothing; bursty workloads spike and drain. Growing across samples is the signal.
Convert to time behind. Divide lag by the current publish rate. If the answer is seconds and stable, this is a queue absorbing a burst, not a consumer failing. If the answer is minutes and growing, continue.
Check num_ack_pending against MaxAckPending. If it is pinned at the limit, delivery has stalled at the flow-control ceiling. The question becomes why nothing is being acked: processing hung, AckWait expiring before processing finishes, or the consumer application dead but the durable consumer still registered.
Check num_redelivered. A climbing redelivery counter means messages are coming back. Either processing exceeds AckWait (messages redelivered mid-processing, producing duplicates) or the handler is failing and NAKing in a loop.
Verify the consumer application is alive and connected. A disconnected-but-registered durable consumer accumulates num_pending indefinitely. No server-side error fires for this.
Rule out server-side causes. Check api.inflight and api.errors on /jsz. Persistently high inflight plus rising errors points at Raft or disk I/O trouble slowing the whole JetStream subsystem, which shows up as lag across many consumers at once, not just one. If only one consumer on one stream is behind, the problem is almost always application-side.

flowchart TD
  A[num_pending growing] --> B{num_ack_pending at MaxAckPending?}
  B -->|Yes| C[Delivery stalled: check acking, processing hung or AckWait too short]
  B -->|No| D{num_redelivered climbing?}
  D -->|Yes| E[NAK or redelivery loop: handler failing or AckWait shorter than processing]
  D -->|No| F{Consumer app connected?}
  F -->|No| G[Disconnected durable consumer accumulating backlog]
  F -->|Yes| H[Consumer too slow: publish rate exceeds processing capacity]

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`num_pending` per consumer	The direct measure of how far behind the consumer is	Sustained growth rate above zero
Time behind (lag / publish rate)	The alertable unit; message counts are meaningless without rate	More than ~5 minutes on latency-sensitive streams
`num_ack_pending` vs `MaxAckPending`	Predicts the delivery stall cliff before it fully bites	Ratio above 80%; equality means delivery stopped
`num_redelivered` rate	Detects NAK loops and AckWait misconfiguration	Any sustained positive rate
Stream `messages` and `first_seq` movement	Shows whether retention is draining or the stream is accumulating	`messages` growing while `first_seq` is static
`api.inflight` + `api.errors`	Distinguishes one slow consumer from a sick JetStream subsystem	High inflight plus rising errors across all consumers

On alert severity: consumer lag is a ticket, not a page. It is an application-level concern, ephemeral consumers legitimately lag during scaling events, and batch/replay consumers carry high pending by design. Gate alerts on known-durable, latency-sensitive consumers, and alert on sustained growth rate rather than absolute count.

Fixes

Consumer too slow

Scale the consumer horizontally. With a queue group (a consumer with a DeliverGroup), adding members distributes deliveries across them; note that MaxAckPending is shared across all members of the group. If scaling is not possible, find the processing bottleneck: synchronous database writes in the handler, GC pauses, and serialization hot spots are the usual suspects. Compare publish rate to ack rate to size the gap honestly.

Stalled at MaxAckPending

Two knobs, with tradeoffs:

Raise MaxAckPending. Lets more messages be in flight before delivery pauses. Tradeoff: a crashed consumer now holds more unacked messages, and recovery after failure redelivers a larger batch.
Fix the acking. If the consumer is not acking because processing is hung or AckWait expires first, raising the limit only delays the next stall. Treat the limit as the symptom, not the cause.

NAK/redelivery loop and AckWait too short

If num_redelivered climbs while processing is otherwise healthy, AckWait is shorter than real processing time, so messages are redelivered mid-processing and handled twice. Set AckWait comfortably above your p99 processing latency. If the handler itself is failing (poison message, downstream outage), fix the handler; no server-side tuning will drain a loop.

Disconnected but registered consumer

Restart or redeploy the consumer application. If the consumer is genuinely dead and not coming back, deleting the consumer lets retention proceed. Warning: with interest retention, deleting a consumer immediately makes all its pending messages eligible for deletion. Confirm that data loss is acceptable before running nats consumer rm. On restart, a durable consumer resumes from its last ack, so a consumer that was down for a while will show large initial lag while it drains. That is expected; watch that the drain rate exceeds the publish rate.

Server-side contributors

If many consumers across streams lag simultaneously and api.inflight is persistently high, suspect Raft instability or disk I/O stalls rather than individual consumers. See the related guides below for those paths.

Prevention

Alert on time behind, not message counts. Compute lag per durable consumer, divide by publish rate, alert on sustained minutes-behind. Absolute thresholds break across deployment sizes.
Track the num_ack_pending / MaxAckPending ratio. A consumer regularly running above 80% is one slow processing cycle away from a stall. Catch it before delivery stops.
Track num_redelivered as a rate. It is the earliest signal of AckWait misconfiguration and poison-message loops.
Test failure modes in staging. Kill a consumer, slow a consumer, and poison a message deliberately, and verify your alerts fire. The interaction between MaxAckPending, AckWait, redelivery, and retention policy creates failure modes teams only discover during incidents.
Do not rely on server health as a proxy. /healthz green means the server process is operational. It says nothing about whether any given consumer is keeping up.

How Netdata helps

Netdata collects the aggregate JetStream signals from /jsz (storage, message counts, API totals and errors), which tell you whether lag is a consumer problem or a subsystem problem.
Correlating stream growth (messages rising, first_seq static) against API error and inflight rates separates a stalled consumer from a JetStream subsystem that is slow for everyone.
Per-consumer lag (num_pending, num_ack_pending) requires polling /jsz?consumers=true; pair Netdata’s server-level metrics with a small per-consumer poller or exporter for the critical durable consumers, and alert on the growth rate.
Anomaly detection on publish and delivery rates flags the divergence (publish rate steady, delivery rate falling) that precedes visible backlog, often before absolute thresholds trip.