NATS insufficient storage / maximum bytes exceeded: JetStream publishes rejected

Your producers start throwing JetStream publish errors: “insufficient storage”, “maximum bytes exceeded”, or “maximum messages exceeded”. The server itself looks healthy: /healthz returns ok, connections are stable, CPU and memory are normal. Writes to one or more streams are failing anyway.

This is a storage-limit failure, not a server failure. JetStream enforces limits at three levels: per-stream (max_bytes, max_msgs, max_age), per-account storage quotas, and the server-wide JetStream storage reservation. When any of them is hit, the discard policy decides the failure mode. With DiscardNew, new publishes are rejected and the publisher sees an error. With DiscardOld, the server silently evicts the oldest messages to make room, which is data loss for any consumer that has not caught up.

Rejection is invisible unless the publisher waits on the ack. A fire-and-forget publisher keeps “succeeding” while messages go nowhere. This guide covers how to identify which limit fired, which streams are involved, and how to recover without losing more data than you have to.

What this means

Every stream has limits, and the server has a storage reservation on top. When a limit is reached:

• DiscardNew: the publish is rejected. The client gets a JetStream API error on the ack (error code 10047, “insufficient storage resources available”). Synchronous publishes return the error directly; async publishes surface it on the ack future. If your publisher never waits on acks, you will not notice until someone downstream complains.
• DiscardOld (the default): the publish succeeds, but the oldest messages are deleted to make room. There is no error anywhere. Lagging consumers lose those messages permanently.

Three different limits produce the same symptom:

1. Per-stream limits: the stream hit its own max_bytes or max_msgs. Other streams on the same server may be fine.
2. Account-level limits: the account’s JetStream storage quota is exhausted, rejecting publishes to any stream in that account.
3. Server-level reservation: the aggregate JetStream storage reservation (reserved_storage / reserved_memory) is full. This affects every stream on the server.

A fourth cause produces identical symptoms from the outside: the underlying filesystem filled up. JetStream respects its own configured limits, but other processes on the same host can consume the disk out from under it. From the client’s perspective, writes fail either way.

flowchart TD
    A[Publish rejected or messages missing] --> B{Which discard policy?}
    B -->|DiscardNew| C[Publisher sees error on ack]
    B -->|DiscardOld| D[Oldest messages evicted silently]
    C --> E{Which limit?}
    D --> E
    E -->|Stream max_bytes or max_msgs| F[Fix stream limits or consumer lag]
    E -->|Account quota| G[Fix account limits or shrink streams]
    E -->|Server reservation| H[Free storage or raise reservation]
    E -->|Filesystem full| I[Clear disk: other processes]

Common causes

Quick checks

All of these are read-only. The monitoring port defaults to 8222.

# Aggregate JetStream storage vs the configured reservation
curl -s http://localhost:8222/jsz | jq '{bytes, messages, memory, storage, reserved_memory, reserved_storage}'

# API error counters: rising errors here include publish rejections
curl -s http://localhost:8222/jsz | jq '{api_total: .api.total, api_errors: .api.errors, inflight: .api.inflight}'

# Per-stream state: find which streams are at or near their limits
curl -s 'http://localhost:8222/jsz?streams=true' | jq '.account_details[].stream_detail[] | {name, messages: .state.messages, bytes: .state.bytes, first_seq: .state.first_seq, last_seq: .state.last_seq, consumers: .state.consumer_count}'

# Per-consumer state: find stalled consumers on the affected stream
curl -s 'http://localhost:8222/jsz?consumers=true' | jq '.account_details[].stream_detail[].consumer_detail[] | {name, num_pending, num_ack_pending, num_redelivered}'

# Underlying filesystem on the JetStream store directory
df -h /path/to/jetstream/store_dir

If you have the nats CLI, nats stream report and nats stream info <stream> give the same per-stream picture, and nats consumer info <stream> <consumer> shows the detailed ack state.

How to diagnose it

1. Confirm the failure mode. Are publishers seeing errors (DiscardNew) or are consumers reporting gaps (DiscardOld eviction)? Check the stream’s discard policy with nats stream info. If DiscardOld, assume data loss has already occurred for lagging consumers and scope the damage via consumer num_pending and sequence gaps.

2. Identify which limit fired. Compare per-stream state.bytes against the configured max_bytes from /jsz?streams=true. If no individual stream is at its limit but storage is near reserved_storage in /jsz, the server-level reservation is the binding constraint. If only streams in one account are affected while the server has headroom, suspect the account-level quota.

3. Find the growth driver. A stream fills because publishes outpace deletion. Deletion happens via retention (age/limits), consumer acknowledgment (interest/workqueue retention), or DiscardOld eviction. If first_seq is not advancing while last_seq climbs, messages are accumulating. That almost always means a stalled or slow consumer, not a publisher problem.

4. Check the consumers on the affected stream. Growing num_pending means delivery is falling behind. num_ack_pending pinned at the consumer’s MaxAckPending means delivery has stopped entirely: the server will not send more until acks arrive. With interest retention, one stalled consumer blocks retention for the whole stream.

5. Check the filesystem independently. Even when JetStream reports headroom against its own reservation, run df on the store directory. Log files, core dumps, or co-located processes can fill the disk. JetStream cannot write what the filesystem will not give it.

6. In a cluster, compare members. Pull /jsz storage on every node. If one server is at its reservation while peers are not, stream and leader placement is skewed. The per-server limit is what matters; cluster totals hide this.

Metrics and signals to monitor

Fixes

Unblock the stalled consumer

This is the most common root cause and the right first fix. If the consumer application is down, restart it. If num_ack_pending is pinned at MaxAckPending, the consumer is connected but not acking: look for a hung processing loop, a downstream dependency failure, or an AckWait shorter than actual processing time (messages get redelivered without ever completing).

If the consumer is genuinely dead and cannot be revived quickly, deleting it makes its pending messages eligible for deletion under interest retention. That is destructive: those messages may be removed. Do it deliberately, not reflexively.

Purge or trim the stream

If the accumulated messages have no remaining value (or the data is reproducible), purging the stream frees space immediately. This is destructive: purged messages are gone. Prefer trimming by age or sequence over a full purge when only the oldest data is expendable: nats stream purge STREAM --seq 1000 purges up to (but not including) sequence 1000, while --keep 1000 keeps the newest 1,000 messages. Use --force only after you have confirmed the destructive scope.

Raise the limit that fired

If the workload has legitimately outgrown the stream, increase max_bytes on the stream, the account quota, or the server reservation, whichever is binding. Tradeoffs: a bigger stream means longer recovery on restart (JetStream replays and reindexes its store), more disk, and a bigger blast radius the next time a consumer stalls. Raising limits without fixing consumer lag only buys time.

Recover server-level headroom

When the server reservation is the constraint: shrink or delete streams you no longer need, reduce replica counts to free space on removed replicas, or raise the reservation if the disk supports it. Note that stream max_bytes reserves capacity against account and server limits even when the stream is empty; issue #4281 reports cases where deleting and immediately recreating a same-sized stream still failed. Recheck /jsz reserved storage before retrying. In clustered setups there is no live stream migration, so rebalancing means adjusting replicas and placement, not moving streams.

Fix the filesystem

If df shows the disk full while JetStream reports headroom, the fix is outside NATS: clear the co-located logs, core dumps, or whatever consumed the space. Longer term, put the JetStream store_dir on a dedicated filesystem so nothing else can fill it.

Prevention

• Set explicit limits on every stream. A stream created without max_bytes, max_msgs, or max_age grows until it hits the account or server limit. Unlimited streams are the most common cause of server-level exhaustion.
• Choose discard policies deliberately. DiscardOld (default) converts a full stream into silent data loss. DiscardNew converts it into visible publisher errors. Neither is “safe”; pick per stream based on whether producers or consumers should absorb the backpressure, and make sure publishers check acks where DiscardNew is used.
• Monitor the ratio, not the absolute. Alert on storage / reserved_storage approaching 90% and per-stream bytes / max_bytes above 0.8, with enough lead time to act before rejection starts.
• Monitor consumer lag as the leading indicator. num_pending growth and num_ack_pending saturation warn you long before storage fills. Storage is the capacity metric; lag is the operational one.
• Reserve overhead beyond stream limits. Clustered JetStream consumes storage beyond raw message data (Raft logs, consumer state, compaction). Leave headroom between the sum of stream limits and the server reservation rather than sizing them exactly equal.
• Give JetStream its own filesystem. A dedicated store_dir filesystem removes the “someone else filled the disk” failure mode entirely.

How Netdata helps

• Netdata’s NATS collector polls the HTTP monitoring endpoints and tracks aggregate JetStream storage, memory, and reserved quotas from /jsz, so you see the reservation filling as a trend instead of discovering it at 100%.
• The JetStream API error rate (api.errors vs api.total) is charted over time, which is where DiscardNew publish rejections become visible as a rising error rate instead of scattered client-side exceptions.
• Correlating storage growth against message throughput (in_msgs rate) separates a producer burst from a consumer stall: throughput flat while storage climbs points at consumers.
• Per-second collection catches the acceleration at the end of a fill event, which 60-second scrape intervals routinely miss.
• Alerting on the storage/reserved ratio with trend-based time-to-exhaustion gives you lead time to fix a stalled consumer before rejection starts, which is the difference between a ticket and data loss.

Related guides

• NATS context deadline exceeded: JetStream publish and request timeouts
• NATS JetStream disabled unexpectedly: the persistence subsystem failed to come up
• NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check
• NATS monitoring checklist: the signals every production server needs
• NATS messages published but not received: subject mismatches and cross-server gaps
• How NATS actually works in production: a mental model for operators