NATS JetStream storage exhaustion spiral: stalled consumers that starve retention

JetStream storage is climbing steadily. Not a spike, a slope. Then publishes start failing with API errors, producers back up, and the pipeline degrades. The server looks healthy: connections stable, CPU and memory fine. The disk is filling anyway.

This is the JetStream storage exhaustion spiral: a deadlock where one or more consumers have stalled (usually pinned at MaxAckPending) on a stream using interest or workqueue retention. The server cannot delete a message until every interested consumer has acknowledged it, so a single dead consumer makes its entire backlog undeletable. Messages accumulate, storage approaches the configured limit, and new publishes are rejected. The system is stuck: consumers must process messages to free space, but the consumers are the broken part.

This is not a plain full disk and not a producer flood, and the fix is not “add disk”. Find the stalled consumer and decide whether to revive it, pause it, or delete it so retention can proceed.

What this means

JetStream supports three retention policies. Two of them couple deletion to consumer behavior:

• InterestPolicy: a message is kept as long as any consumer on the stream has not acknowledged it. Once all currently defined consumers have acked it, it is removed.
• WorkQueuePolicy: a message is kept until a consumer has delivered and explicitly acknowledged it.
• LimitsPolicy: retention is purely by age, count, or bytes. Consumer behavior does not block deletion, so limits-based streams do not exhibit this spiral.

On an interest or workqueue stream, a stalled consumer holds a veto over deletion. The most common stall is the MaxAckPending wall: the consumer has num_ack_pending equal to its configured max_ack_pending, the server suspends delivery entirely (this is a cliff, not a slope), and every new message piles up in the stream. The stream keeps accepting publishes until storage runs out.

flowchart TD
  A[Producers publish] --> B[Stream stores messages]
  B --> C{Consumer acking?}
  C -- yes --> D[Retention deletes acked messages]
  C -- no: stalled at MaxAckPending --> E[Messages undeletable, accumulate]
  E --> F[Storage approaches limit]
  F --> G[Publishes rejected, api.errors climb]
  G --> H[Producers back up or fail]

The distinguishing shape is storage climbing steadily over hours or days as the consumer falls behind. A producer flood or a misconfigured max_bytes looks different, and a plain filesystem full (another process ate the disk) shows JetStream storage well below its reserved limit while the disk itself is full. Check both before concluding anything.

Common causes

Quick checks

All read-only. Run these before changing anything.

# 1. Server-level JetStream storage vs configured quota
curl -s http://localhost:8222/jsz | jq '{bytes, messages, memory, storage, reserved_memory, reserved_storage}'

# 2. API errors: publish rejections show up here
curl -s http://localhost:8222/jsz | jq '{api_total: .api.total, api_errors: .api.errors}'

# 3. Which streams are consuming storage, and is first_seq advancing?
nats stream report
# or per-stream detail via HTTP:
curl -s "http://localhost:8222/jsz?streams=true" | jq '.account_details[].stream_detail[] | {name: .name, messages: .state.messages, bytes: .state.bytes, first_seq: .state.first_seq, last_seq: .state.last_seq}'

# 4. Per-consumer state: the smoking gun
curl -s "http://localhost:8222/jsz?streams=true&consumers=true" | jq '.account_details[].stream_detail[].consumer_detail[] | {name: .name, ack_pending: .num_ack_pending, pending: .num_pending, redelivered: .num_redelivered}'

# 5. Confirm a specific stalled consumer
nats consumer info STREAM CONSUMER --json | jq '{ack_pending: .num_ack_pending, max_ack_pending: .config.max_ack_pending, pending: .num_pending, redelivered: .num_redelivered}'

# 6. Is it JetStream or the actual filesystem?
df -h /path/to/jetstream/store

Two cautions. /jsz?consumers=true is costlier on servers with many consumers; use it once for diagnosis, not as a tight poll loop. And per-consumer numbers on follower nodes of a replicated consumer can report zero while the leader holds the real state. Query the leader (or use the NATS CLI, which talks to the system) before concluding a consumer is idle. The zero-on-follower report was observed on 2.9.22; maintainers later aligned follower metrics through multiple fixes, including one released in 2.10.17, but leader-first checks remain the reliable incident procedure.

How to diagnose it

1. Confirm the shape. Look at JetStream storage over time. A steady climb over hours with stable producers is the spiral. A vertical jump is a flood or a config change. Check df to rule out a non-JetStream process filling the disk.

2. Confirm publishes are being rejected. api.errors in /jsz rising alongside storage near the limit means the server is refusing writes. This is the point of user impact.

3. Rank streams by storage. nats stream report shows which streams hold the bytes. Pick the top offender. Check whether first_seq is advancing: if messages grows while first_seq is static, retention is blocked, not just slow.

4. Check the retention policy. nats stream info STREAM shows the policy. If it is limits, this article does not apply; you have a sizing or flood problem. If it is interest or workqueue, continue.

5. Find the stalled consumer. For each consumer on the stream, compare num_ack_pending against max_ack_pending. A consumer sitting at the limit with num_pending growing is stalled: delivery is suspended and its backlog pins retention. Also check num_redelivered: rapid growth means AckWait is expiring before processing finishes, and the consumer is cycling instead of progressing.

6. Check whether the application is alive. A durable consumer whose application crashed hours ago still holds its ack state. The server sees a registered consumer with outstanding acks and refuses to delete the messages. This is the most common root cause.

7. Check for stranded messages at MaxDeliver. On workqueue streams, messages that exhaust MaxDeliver attempts stay in the stream and are not automatically removed. They must be deleted manually via the JetStream API. A backlog of poison messages can hold storage even after the consumer recovers.

Metrics and signals to monitor

The storage graph alone will not save you. Storage tells you the disk is filling; consumer lag tells you why. Correlate the two: storage climbing plus a consumer at MaxAckPending is the spiral. Storage climbing with all consumers acking normally is a producer flood or sizing problem.

Fixes

Order matters. Free storage pressure first, then fix the consumer. Do not restart the NATS server as a first move: it does not unblock retention, and on a large store, recovery replay will make things worse before they get better.

The consumer application is dead or hung

Get the application healthy if you can do it quickly: restart the worker, unblock the stuck downstream call, fix the deadlock. Once it resumes acking, retention follows automatically and storage drains.

If the application cannot be revived soon, the deliberate move is to remove the consumer’s veto over deletion:

• NATS 2.11 and later: pause the consumer with the consumer pause API or PauseUntil. Pausing suspends delivery without destroying ack state, so you can resume later.
• Any version: delete the consumer. This is destructive to that consumer’s position and pending state. Under interest retention, deleting the last interested consumer makes its pending messages eligible for deletion, which is exactly what frees the space. Under workqueue, unacked messages become available to other consumers on the stream if any exist. Understand which semantic you are triggering before running it.

Deleting a consumer to unblock retention is a real tradeoff: you are choosing to drop the unprocessed backlog to keep the stream writable. If the backlog matters, snapshot or drain it first.

MaxAckPending too low for processing latency

If the consumer is alive but perpetually pinned, raise max_ack_pending (default 1000) or parallelize processing so acks keep up. On NATS 2.10 and later, also check stream-level ConsumerLimits: a stream-level cap can override your per-consumer increase, so verify both levels.

AckWait shorter than processing time

Increase AckWait so messages are not redelivered mid-processing, or make processing idempotent and faster. Watch num_redelivered flatten out as confirmation.

Retention policy wrong for the workload

If the stream does not actually need consumer-coupled retention, moving to limits retention with a sane age/bytes bound removes the deadlock class entirely. This changes data durability semantics; confirm consumers tolerate replay gaps.

Emergency capacity

Extending the filesystem or raising the server/account storage limit buys time while you fix the consumer. Treat this as a bridge, not a fix: if the consumer stays stalled, the climb resumes.

Prevention

• Alert on the precursor, not the limit. Warn when any consumer’s num_ack_pending exceeds 80% of max_ack_pending, and when num_pending grows for more than a few minutes. The spiral takes hours to fill the disk; the stall is visible within minutes.
• Track stream growth vs first_seq movement. Retention that has stopped deleting is the earliest storage-side signal.
• Size retention for consumer failure, not just throughput. On interest/workqueue streams, your storage headroom must absorb the longest plausible consumer outage multiplied by publish rate. Keep at least 25% free; more for interest retention, where retention depends on consumer health.
• Avoid unbounded streams. max_bytes: -1 lets one stream consume the entire server quota. Set explicit per-stream limits below the server budget.
• Prefer pull consumers for unreliable workers. Push consumers are subject to the MaxAckPending suspension cliff; pull consumers control their own fetch rate and degrade more gracefully.
• Test the failure mode. Kill a consumer in staging on an interest-retention stream and watch what happens to storage, num_pending, and publish behavior. The interaction of MaxAckPending, AckWait, redelivery, and retention is only obvious once you have seen it break.
• Scrape carefully. Per-consumer data via /jsz?consumers=true is expensive on large deployments; poll it at tens of seconds, not sub-second.

How Netdata helps

Netdata’s NATS collector polls the HTTP monitoring endpoints and surfaces the signals that matter for this failure mode:

• JetStream storage vs reserved quota from /jsz, so the steady climb is visible as a trend with time-to-exhaustion context, not discovered at the publish-rejection wall.
• JetStream API error rate (api.errors), which catches publish rejections the moment storage pressure turns into user impact.
• Stream and message totals to correlate which streams are driving growth.
• Server health, uptime, and throughput alongside, so you can rule out the lookalikes (process restarts, producer floods, connection churn) in the same view.

The diagnostic value is correlation: storage climbing, API errors starting, and publish throughput flat or rising, all on one dashboard, points at retention being blocked rather than at the server or the network. Per-consumer ack-pending detail is the one gap: Netdata currently collects aggregate JetStream figures, so keep the nats consumer info checks from this guide in your runbook for the per-consumer step.

Related guides

• How NATS actually works in production: a mental model for operators
• NATS context deadline exceeded: JetStream publish and request timeouts
• NATS JetStream API errors: reading the /jsz api.errors counter without false alarms
• NATS JetStream disabled unexpectedly: the persistence subsystem failed to come up
• NATS JetStream not enabled for account: persistence calls failing on a core server
• NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check
• NATS connection churn: a stable connection count hiding constant reconnects
• NATS connection storm: reconnect thundering herd after a network event
• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS file descriptor exhaustion: too many open files and the ulimit cliff
• NATS Maximum Connections Exceeded: new clients rejected at the max_connections wall
• NATS Maximum Payload Violation: messages rejected for exceeding max_payload