NATS JetStream retention: limits vs interest vs workqueue and the /dev/null stream

JetStream’s retention policy decides when a stored message is deleted, and the failure modes are asymmetric: get it wrong in one direction and storage grows until publishes are rejected; get it wrong in the other and every message you publish is deleted on arrival while the server reports itself perfectly healthy.

This article covers the three policies (limits, interest, workqueue), the exact conditions under which each one deletes a message, the silent /dev/null failure mode, and the checks that confirm your streams are retaining what you think they are. It assumes a working mental model of streams, consumers, and acks. If not, start with how NATS actually works in production.

What the retention policy actually controls

A stream’s retention policy answers one question: what event makes a message eligible for deletion?

It does not control delivery, acks, redelivery, or flow control. Those live on the consumer. Retention only governs the lifetime of the stored copy. Two other settings interact with it constantly:

• The stream limits (max_msgs, max_bytes, max_age) act as upper bounds regardless of policy. Even an interest or workqueue stream will evict or reject messages when these limits are hit.
• The discard policy (DiscardOld or DiscardNew) decides what happens at the limit: evict the oldest messages, or reject new publishes (visible as JetStream API errors and publish failures; see NATS insufficient storage).

So the real behavior of any stream is: retention policy decides deletion in the normal case, limits decide deletion or rejection in the overflow case.

The three policies

flowchart TD
  P[Message published to stream] --> Q{Retention policy}
  Q -->|limits| L[Kept until max age, bytes, or msgs hit]
  L --> D{Discard policy at limit}
  D -->|DiscardOld| D1[Oldest messages evicted]
  D -->|DiscardNew| D2[New publishes rejected]
  Q -->|interest| I{Any consumer interest on the subject}
  I -->|none| I1[Deleted immediately]
  I -->|one or more| I2[Kept until ALL matching consumers ack]
  Q -->|workqueue| W{Any one consumer acked}
  W -->|yes| W1[Deleted]
  W -->|no consumers yet| W2[Retained in stream]

limits (the default)

Messages are kept until a configured limit is reached: maximum age, maximum byte size, or maximum message count. Consumers have no influence on retention at all. A limits stream with zero consumers still stores everything it receives, up to its limits.

This is the policy that behaves the way people coming from Kafka or RabbitMQ assume everything behaves. It is also the policy most likely to fill your disk if you sized the limits optimistically, because retention is completely decoupled from whether anyone is consuming.

interest

A message is kept only while there is consumer interest in it, and deleted once all consumers whose subject filter matches the message have acknowledged it. Two consequences follow directly:

1. A single stalled consumer blocks retention for the entire stream. If one consumer stops acking, its unacked messages stay, and because deletion requires all interested consumers to ack, nothing it touched can be cleaned up. This is a leading cause of the “consumers stalled, stream grew, publishes rejected” death spiral in NATS insufficient storage.
2. Zero consumers means zero interest, which means immediate deletion. This is the /dev/null failure mode covered below.

One nuance: when the last consumer for a subject set is deleted (manually or via its inactive threshold), the server may defer the actual deletion of the now-uninterested messages for performance reasons rather than purging them synchronously. Treat the outcome as contractual, not the timing.

workqueue

A message is deleted as soon as any one consumer acknowledges it. This is the classic task-queue semantic: each message is processed exactly once, by one worker, and then gone.

Workqueue is frequently confused with interest because both tie deletion to acks. The difference is the quantifier: interest waits for all matching consumers, workqueue waits for any one. If you want fan-out (multiple independent consumers each seeing every message), workqueue is wrong. Workqueue streams enforce this: consumers must have non-overlapping subject filters, and creating overlapping consumers is rejected.

One edge case that bites operators: messages that exhaust their consumer’s MaxDeliver redelivery attempts are never acked, so on a workqueue stream they are never deleted by the ack path. They remain in the stream and must be removed explicitly via the JetStream API. A slow accumulation of poison messages on a workqueue stream is normal, not a bug, and needs an operational answer (alert, dead-letter, periodic purge).

The /dev/null stream

The single most dangerous retention configuration is an interest stream with no consumers:

• Publisher sends a message to the stream’s subject.
• The server evaluates interest. There are no consumers, so there is no interest.
• The message is deleted immediately.
• The publish itself can succeed from the client’s point of view. Nothing errors. No slow consumer fires. Server health is green.

The stream is functionally /dev/null while the operator believes data is being persisted. This is most commonly hit in two situations:

• Ordering mistake during provisioning. The stream is created, publishers are deployed, and the consumer application is deployed later. Everything published in the gap is gone.
• Ephemeral consumer expiry. The only consumers on the stream were ephemeral and were auto-deleted after their inactivity timeout. Interest dropped to zero, and the stream silently became a sinkhole.

The tell in the metrics: publish rate (in_msgs) is healthy, the stream’s stored message count stays at or near zero, and out_msgs is far below in_msgs. This is the JetStream cousin of the zero-subscriber silent loss pattern in core NATS: the server is doing exactly what it was told to do, and only the asymmetry between what went in and what is stored gives it away.

Note the asymmetry between policies. On a limits stream, no consumers means messages accumulate. On a workqueue stream, messages that have not been acked by anyone are retained, so a workqueue stream with temporarily absent workers keeps its backlog (bounded by the stream limits). On an interest stream, no consumers means messages vanish on arrival. Two of these three “no consumer” states are safe for your data and one is not.

Choosing a policy

• Event log, replay, audit, “we might need this later”: limits. Retention is a capacity decision, not a consumer-health decision. Size max_bytes / max_age deliberately, and decide up front whether overflow should evict old data (DiscardOld) or backpressure publishers (DiscardNew).
• Fan-out with delivery guarantee to a known set of consumers: interest. You get “keep it until everyone got it” semantics, at the cost of two operational obligations: never let interested consumer count hit zero, and never let one consumer stall indefinitely. Both need monitoring, not hope.
• Task distribution to a worker pool: workqueue. Each message handled once, storage freed on first ack. Accept that poison messages past MaxDeliver pile up and plan a dead-letter or purge path.

If you find yourself wanting “interest but don’t delete when consumers are briefly gone,” that is not a retention policy; that is limits retention with durable consumers.

Verifying a stream retains what you think it does

All of these checks are read-only.

# Confirm the configured policy and limits
nats stream info ORDERS --json | jq '.config | {retention, discard, max_msgs, max_bytes, max_age}'

# See what is actually stored right now
nats stream info ORDERS --json | jq '.state | {messages, bytes, first_seq, last_seq, consumer_count}'

# All streams at a glance: stored messages vs consumers
nats stream report

# Server-side aggregate: storage used vs reserved, and the API error counter
curl -s http://localhost:8222/jsz | jq '{storage, reserved_storage, accounts, api_errors: .api.errors}'

# Per-stream detail via the monitoring port
curl -s 'http://localhost:8222/jsz?streams=true' | jq '.account_details[].stream_detail[] | {name: .name, messages: .state.messages, bytes: .state.bytes, consumers: .state.consumer_count}'

The specific cross-checks that catch retention misconfiguration:

• Interest stream with consumer_count: 0: verify this is intended. If publishers are active and messages is zero, you are in the /dev/null state.
• messages growing while first_seq never advances: retention is not freeing anything. On interest, find the stalled consumer (check num_ack_pending per consumer via nats consumer info or /jsz?consumers=true). On limits, your limits are simply too generous for the publish rate.
• Interest or workqueue stream, storage growing despite healthy consumers: messages past MaxDeliver never get acked and never get deleted. Look at num_redelivered trends and dead-letter them.
• Gap between last_seq - first_seq + 1 and messages: normal during compaction or after deletions; persistent large gaps on interest streams can indicate deferred deletion after a consumer removal.

One caution on tooling: on servers before v2.11, nats stream view creates an ephemeral consumer to read messages. On an interest-retention stream that ephemeral consumer counts as interest, and its lifecycle can interact with deletion. Prefer nats stream info and direct fetches for inspection on older servers, and upgrade when you can.

Version-specific bugs worth knowing

Retention policy behavior has had real bugs. If you are pinned to an older server, check whether you are exposed before trusting the semantics described above. These are drawn from public nats-server issues:

• Interest + filtered consumers (fixed in v2.7.3): with staggered filtered consumers, acked messages could fail to be removed, causing unbounded growth on interest streams.
• Interest + nats stream view (fixed in v2.11.0): the CLI’s ephemeral viewer consumer could trigger interest-based deletion of messages that had exceeded max delivery.
• Interest + stream sourcing across clusters (fixed in v2.14.0): if the destination cluster went down, the replicated consumer disappeared, the source stream saw zero interest, and deleted messages. On earlier versions, prefer limits retention for sourced streams. The v2.14.0 reliable sourcing/mirroring change makes the source/mirror consumer durable and acknowledgment flow-control based.
• Workqueue with R3 replicas losing max-delivered messages (fixed in v2.14.0): reported on v2.12.3/v2.12.4 in nats-server issue #7817 and fixed by PR #7845; the fix is included in v2.14.0.

The general lesson: interest and workqueue retention depend on consumer state, and consumer state is exactly the kind of distributed state that has edge cases. Limits retention has the fewest moving parts.

Signals to watch in production

How Netdata helps

Retention problems are correlation problems: no single metric says “your policy is wrong,” but two or three together say it clearly. Netdata’s NATS collector polls the server’s HTTP monitoring endpoints and charts the counters that matter here.

• JetStream stored messages and bytes over time makes the /dev/null stream visible as a flat line at zero while publish throughput is healthy, and makes retention failure visible as an unbroken climb.
• JetStream storage vs configured limits shows the trajectory toward exhaustion, so a mis-scoped limits stream pages you as a trend, not as a publish-rejection incident.
• JetStream API error rates catch the DiscardNew and storage-full rejections that are the first externally visible symptom of retention not keeping up.
• Message throughput asymmetry (in vs out) surfaces silent loss patterns, where messages are accepted and immediately deleted or delivered to nobody.
• Consumer and stream counts expose the drop-to-zero-consumers event that turns an interest stream into a sinkhole.

Correlating these on one dashboard is what shortens diagnosis: “interest stream, consumer count hit zero at 14:02, stored messages flat since 14:02, publishes fine” is a one-minute conclusion instead of a multi-hour archaeology session.

Related guides

• NATS connection churn: a stable connection count hiding constant reconnects
• NATS connection storm: reconnect thundering herd after a network event
• NATS context deadline exceeded: JetStream publish and request timeouts
• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS file descriptor exhaustion: too many open files and the ulimit cliff
• NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check
• How NATS actually works in production: a mental model for operators
• NATS insufficient storage / maximum bytes exceeded: JetStream publishes rejected
• NATS JetStream API errors: reading the /jsz api.errors counter without false alarms
• NATS JetStream disabled unexpectedly: the persistence subsystem failed to come up
• NATS JetStream not enabled for account: persistence calls failing on a core server
• NATS Maximum Connections Exceeded: new clients rejected at the max_connections wall