NATS JetStream consumer stopped receiving messages: the diagnostic tree

A JetStream consumer that stops receiving messages almost never raises an error. The stream keeps accepting publishes, the health endpoint stays green, and your application simply goes quiet. The signals that explain this live per-consumer, not per-server, which is why server-level dashboards show nothing.

There are only a handful of mechanisms that produce this symptom, and each has a distinct fingerprint in consumer state. This article is the diagnostic tree. Start at the top with nats consumer info (or /jsz?consumers=true), read five counters, and the tree tells you which failure you are in and where to go next.

The five counters that matter: num_ack_pending (delivered but not acknowledged), num_pending (in the stream, not yet delivered), num_waiting (outstanding pull requests), num_redelivered (redelivery events), and the delivered/ack floor sequence numbers. Everything below is a reading of those fields plus one Raft check.

What this means

“Stopped receiving messages” decomposes into exactly five cases:

1. Stalled at MaxAckPending. The consumer’s in-flight (unacknowledged) messages reached the configured max_ack_pending limit. The server stops delivering new messages until some are acked, nak’d, or expire past ack_wait. Delivery halts abruptly, with no error. This is the most common cause of this symptom.
2. Registered but not requesting (pull consumers). num_pending > 0 and growing, but num_waiting == 0. The consumer exists on the server but has no outstanding pull requests. The client-side pull loop crashed, or the client reconnected and never re-bound the subscription. Messages are waiting and nobody is asking for them.
3. Disconnected but still registered (push consumers). The durable consumer record exists, delivery cursors exist, but no client subscription is bound. Messages accumulate as num_pending.
4. Ephemeral consumer auto-deleted. The consumer is gone entirely. Ephemeral consumers are removed after an inactivity threshold when no subscription is bound. A brief client disconnect (rolling restart, network blip) can delete the consumer before the client reconnects, and if the client does not recreate it, delivery stops permanently.
5. Stream Raft group without a leader. In clustered JetStream, each stream has its own Raft group. If that group loses quorum, the stream accepts no writes and consumers cannot make progress. From the application side this looks identical to a consumer problem.

flowchart TD
  A[Consumer stopped receiving] --> B{Consumer exists?}
  B -- no --> C[Ephemeral auto-deleted or never created]
  B -- yes --> D{num_ack_pending == max_ack_pending?}
  D -- yes --> E[Stalled at MaxAckPending]
  D -- no --> F{num_pending growing?}
  F -- yes --> G{num_waiting > 0?}
  G -- no --> H[Registered but not requesting]
  G -- yes --> I[Delivery active: check client-side processing]
  F -- no --> J{Stream Raft leader?}
  J -- no --> K[Stream group lost quorum]
  J -- yes --> L[Consumer caught up: check publisher path]

Common causes

Quick checks

All read-only. Run these in order; the first two usually answer the question.

# 1. Get the full consumer state
nats consumer info STREAM_NAME CONSUMER_NAME --json | jq '{
  num_pending, num_ack_pending, num_waiting, num_redelivered,
  delivered: .delivered.stream_seq,
  ack_floor: .ack_floor.stream_seq,
  max_ack_pending: .config.max_ack_pending,
  ack_wait: .config.ack_wait,
  durable: .config.durable_name
}'

# 2. Overview of every consumer on the stream
nats consumer report STREAM_NAME

# 3. Does the consumer still exist at all?
nats consumer ls STREAM_NAME

# 4. Same data via the monitoring endpoint (useful when the CLI is not available)
curl -s 'http://localhost:8222/jsz?consumers=true' | \
  jq '.account_details[].stream_detail[].consumer_detail[] | {name, num_pending, num_ack_pending, num_redelivered}'

# 5. Stream-level view: is the stream itself writable and replicated?
nats stream info STREAM_NAME --json | jq '{messages: .state.messages, last_seq: .state.last_seq, cluster: .cluster}'

# 6. Per-stream Raft group health (clustered JetStream only)
curl -s http://localhost:8222/raftz | jq '.'

# 7. JetStream API errors: are publishes being rejected too?
curl -s http://localhost:8222/jsz | jq '{api_total: .api.total, api_errors: .api.errors, inflight: .api.inflight}'

Note on check 4: /jsz?consumers=true is costlier than bare /jsz on servers with many consumers. Fine for incident response; do not put it on a tight scrape loop. Also, num_pending is eventually consistent and may lag just-published messages slightly.

How to diagnose it

Walk the tree. Each step either identifies the failure or hands you to the next branch.

1. Does the consumer exist? Run nats consumer ls STREAM_NAME. If your consumer is absent and the application expects it, you are in the ephemeral auto-delete case. Ephemeral consumers are deleted after an inactivity threshold when no subscription is bound; a brief client disconnect during a rolling restart is enough. The default threshold is 5 seconds, with up to about 1 second of jitter for pull consumers. Confirm whether the application code creates the consumer on startup or assumes it persists. The fix path is in the Fixes section.

2. Is the consumer stalled at MaxAckPending? Compare num_ack_pending to config.max_ack_pending (default 1000). If they are equal, delivery is stopped by design: the server will not send another message until in-flight messages are acked, nak’d, or expire past ack_wait. This is a cliff, not a slowdown. Now ask why acks stopped: is the client process alive? Is it blocked on a downstream call (database lock, synchronous I/O in the handler)? Is ack_wait shorter than real processing time, so messages cycle between delivered and redelivered without ever completing? Check num_redelivered: if it is climbing, your messages are timing out unacked, the classic crash-loop or too-short-ack_wait signature.

3. Is the consumer registered but not requesting? If num_ack_pending < max_ack_pending but num_pending is positive and growing, check num_waiting. For a pull consumer, num_waiting counts outstanding pull requests. num_pending > 0 with num_waiting == 0 means the server has messages and the consumer has capacity, but nobody is asking. The client-side pull loop died, or the client reconnected after a network event and never re-established the fetch loop. The durable consumer record survives reconnects; the subscription binding does not. There is a reported pattern on some server/client versions where consumers go stale in exactly this shape after reconnection. The nats.java #1272 report was closed without a release fix; its reproducer traced one case to application code that swallowed a consumer-registration failure during reconnect. Check your application reconnect logic and versions before blaming the server.

4. Is the stream’s Raft group healthy? If the consumer looks fine but nothing flows, and other consumers on the same stream are also stalled, look one level down. In clustered JetStream each stream has its own Raft group, independent of the meta group. nats stream info STREAM --json | jq .cluster shows the leader and replicas. A group without a leader cannot accept writes or advance delivery. Server logs in this state carry a no-quorum warning for the affected stream: JetStream cluster stream 'ACCOUNT > STREAM' has NO quorum, stalled. Also check /jsz for api.inflight: sustained high inflight plus rising api.errors alongside a stalled stream points at Raft or disk I/O, not at the consumer.

5. Is anything flowing at all? If num_pending is zero and num_waiting > 0, the consumer is alive and caught up; your problem is upstream. Check the publisher path: is in_msgs still incrementing (curl -s http://localhost:8222/varz | jq .in_msgs)? Are publishes being rejected (rising jsz api.errors, which happens when storage is full)? A consumer “not receiving” is sometimes a publisher “not sending” or a subject mismatch.

6. Check for the stuck-counter edge case. If the application terminated without acking and no new pull requests arrive, num_ack_pending can remain non-zero even after ack_wait expires, because the counter only advances when a new pull request mutates consumer state. Do not mistake a stale non-zero num_ack_pending on an idle consumer for an active stall; issue a pull or restart the consumer before concluding. Server maintainers confirmed this intended behavior, and it remains in current releases.

Metrics and signals to monitor

Two caveats from the field. First, batch and replay consumers legitimately sit at high num_pending; alert on growth rate, not absolute value, and exempt replay workloads. Second, per-consumer data requires /jsz?consumers=true or the NATS API, which is exactly why so many teams have no visibility here: server-level dashboards cannot see any of this.

Fixes

MaxAckPending stall

The immediate fix is to unblock acking: fix the downstream dependency or restart the wedged consumer process so in-flight messages expire past ack_wait and redeliver. Do not just raise max_ack_pending to mask the problem; the limit exists to bound unacked work in memory. The structural fixes depend on what you found:

• If processing time exceeds ack_wait, raise ack_wait to comfortably exceed p99 processing latency. A too-short ack_wait combined with a tight max_ack_pending creates a redelivery treadmill where messages cycle without ever completing.
• If the handler does synchronous I/O, move it off the delivery path or scale consumer instances. max_ack_pending is shared across all members of a delivery group and across all subjects bound to the consumer; if you need per-subject flow control, split into separate consumers per subject.
• If the application crashes mid-processing, fix the crash loop first. Rising num_redelivered tells you this is the mechanism.

Registered but not requesting

Re-bind the client. In practice that means restarting the consumer application or fixing the reconnection logic so the fetch/subscribe loop is re-established after every reconnect, not just at startup. The durable consumer record on the server is fine; the messages are waiting. After the client re-binds, num_waiting goes positive and num_pending drains. If this recurs after network events, review your client library’s reconnect handling and check versions against known stale-consumer issues.

Disconnected durable (push)

Confirm whether anything is supposed to be subscribed. If the consumer is genuinely orphaned (the workload was decommissioned), delete it. Warning: under interest retention, a dead consumer also blocks retention for the whole stream. Deleting it immediately makes its pending messages eligible for deletion, which is exactly what you want if storage is filling, or data loss if it is not. Know which before you delete.

Ephemeral auto-delete

If the workload needs the consumer to survive brief disconnects, use a durable consumer. Durables are not auto-deleted unless you explicitly set an inactivity threshold. If you must stay ephemeral, the application must recreate the consumer on reconnect; verify that code path actually runs, and check whether your reconnect window exceeds the inactivity threshold.

Stream Raft quorum loss

This is no longer a consumer problem. Check which peer is unreachable, look at route health (/varz routes, /routez), disk I/O on the peers, and recent leader elections. Raft instability is usually network latency, CPU starvation, or slow disk on one of the replicas, and the recovery steps belong to the cluster guides linked below, not to consumer tuning.

Prevention

• Alert on the stall condition directly. num_ack_pending / max_ack_pending crossing 0.8 is a ticket; equality is a page for latency-sensitive consumers. This is the single highest-value alert in this article.
• Alert on the fingerprint, not the symptom. For pull consumers, num_pending > 0 AND num_waiting == 0 sustained is a precise, low-false-positive detector for the dead-pull-loop case.
• Use durables for anything that matters. Ephemeral consumers are for genuinely ephemeral workloads. Anything with a delivery guarantee requirement should be durable.
• Size ack_wait from measured processing latency, not from the default, and load-test the failure mode: kill a consumer mid-batch in staging and watch num_redelivered, ack_wait expiry, and recovery. The interaction of max_ack_pending, ack_wait, and retention policy is only discoverable under failure.
• Track consumer count per stream so ephemeral deletion is visible the moment it happens.
• Keep the publisher side honest. Correlate consumer lag with stream growth and jsz api.errors so “consumer not receiving” and “publishes rejected” are never confused.

How Netdata helps

• Netdata polls the NATS HTTP monitoring endpoints (/varz, /jsz, /healthz) and charts aggregate JetStream state: api.errors, api.inflight, storage usage, stream and consumer counts. That covers the server-side half of this symptom (Raft distress, storage exhaustion, API rejections) without manual curl loops.
• Server-level signals like slow_consumers, connection churn (total_connections delta), and uptime resets give you the context around the stall: did the client flap, did the server restart, was the network the trigger.
• Correlating JetStream storage growth against consumer activity on one dashboard separates “consumer stalled, stream filling” from “publisher flooding”, the most common misread during this incident.
• Uptime tracking with restart detection tells you whether a Raft election burst or JetStream recovery window explains a transient delivery pause before you start dissecting consumer state.
• Per-consumer fields (num_ack_pending, num_waiting) are not part of Netdata’s current NATS collector; pair Netdata’s server-level view with the nats consumer info checks in this article, or export per-consumer metrics via the NATS API, to close the gap.

Related guides

• NATS connection churn: a stable connection count hiding constant reconnects
• NATS connection storm: reconnect thundering herd after a network event
• NATS context deadline exceeded: JetStream publish and request timeouts
• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS file descriptor exhaustion: too many open files and the ulimit cliff
• NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check
• How NATS actually works in production: a mental model for operators
• NATS insufficient storage / maximum bytes exceeded: JetStream publishes rejected
• NATS JetStream API errors: reading the /jsz api.errors counter without false alarms
• NATS JetStream disabled unexpectedly: the persistence subsystem failed to come up
• NATS JetStream disk I/O stall: the disk has space but is too slow
• NATS JetStream not enabled for account: persistence calls failing on a core server