NATS stream max_bytes: one stream hitting its limit while the server has room

Publishers to one JetStream stream start getting rejected with a “maximum bytes exceeded” error. You check the server: JetStream storage is at 40% of reserved_storage, the disk has hundreds of gigabytes free, and every server-level dashboard is green. Nothing about the server looks full.

What happened: max_bytes is a per-stream limit, independent of server-level and account-level JetStream quotas. One stream reached its own configured ceiling and started enforcing its discard policy while every other stream kept working. If your monitoring only watches aggregate JetStream storage (/jsz without parameters), this failure is invisible until publishers start erroring.

The inverse footgun makes this worse. A stream created with max_bytes: -1 (unbounded, and a common default in copied configs) will grow until it consumes the account-level or server-level storage budget, at which point every stream in that account starts rejecting writes. One misconfigured stream takes the rest down with it.

This guide covers how to confirm which limit was hit, how to fix it without losing data you did not intend to lose, and how to monitor per-stream utilization so the next occurrence pages you at 80% instead of at rejection time.

What this means

JetStream enforces storage limits at three independent levels:

1. Per-stream: max_bytes in the stream configuration caps the total bytes that stream may store. Editable after stream creation.
2. Per-account: JetStream resource limits configured on the account cap total JetStream memory and file storage for all streams in that account.
3. Per-server: max_file_store / max_memory_store cap JetStream storage for the whole server, reported via /jsz as reserved_storage and reserved_memory.

A stream can be hard-blocked by its own max_bytes while levels 2 and 3 have abundant headroom. That is the incident this article covers.

Enforcement at max_bytes depends on the stream’s discard policy:

• DiscardOld (default): the oldest messages are deleted to make room for new ones. Publishes keep succeeding. You lose data silently, and slow consumers lose messages they never saw.
• DiscardNew: new publishes are rejected. Producers see the error immediately. This is the configuration that generates the incident page.

The same per-stream blindness applies on the growth side. If one stream is max_bytes: -1, no stream-level limit stops it. It consumes shared account or server storage until the aggregate limit trips, and then DiscardNew behavior applies account-wide: all streams start rejecting.

flowchart TD
  A[Publish rejected or old data missing] --> B{Stream state.bytes near config.max_bytes?}
  B -->|Yes| C[Per-stream limit hit]
  B -->|No| D{Aggregate jsz.storage near reserved_storage?}
  D -->|Yes| E[Server or account limit hit - likely an unbounded stream]
  D -->|No| F[Not a storage limit - check Raft, disk I/O, API errors]
  C --> G{Discard policy?}
  G -->|DiscardNew| H[Publishes rejected with maximum bytes exceeded]
  G -->|DiscardOld| I[Oldest messages silently deleted]

Common causes

Two of these deserve detail because they are version-dependent and non-obvious:

0-byte files bypassing limits. Unclean shutdowns can leave 0-byte .blk and .fss files in the stream’s message store directory. Affected server versions failed to account for these files when enforcing max_bytes, so the stream grew past its own limit until it hit the server-level store limit. The initial fix in 2.9.11 was incomplete; the fuller fix landed in 2.9.17. If you are running older than 2.9.17 and have had unclean shutdowns, upgrade before trusting per-stream accounting.

Storage reservation by configured max_bytes. JetStream reserves max_bytes when a stream is created or resized, even if actual usage is near zero: memory streams count against max_memory_store, and file streams count against max_file_store. Operators have reported that several streams with large max_bytes values can exhaust the matching server quota and fail creation with “insufficient memory resources available” (error 10028) despite low real usage. Maintainers confirmed that JetStream does not over-commit reservations.

Quick checks

All read-only. Run against the monitoring port (default 8222).

# Per-stream bytes vs configured max_bytes: the core check
curl -s "http://localhost:8222/jsz?streams=true" | \
  jq '.account_details[].stream_detail[] |
      {name: .name,
       bytes: .state.bytes,
       max_bytes: .config.max_bytes,
       msgs: .state.messages,
       first_seq: .state.first_seq,
       discard: .config.discard}'

# Aggregate view: is the server or account actually full?
curl -s http://localhost:8222/jsz | \
  jq '{bytes, messages, memory, storage, reserved_memory, reserved_storage}'

# API errors: publish rejections show up here
curl -s http://localhost:8222/jsz | \
  jq '{api_total: .api.total, api_errors: .api.errors, inflight: .api.inflight}'

# CLI view of all streams with current usage
nats stream report

# Single stream detail: config, state, consumer count
nats stream info STREAM_NAME --json | \
  jq '{max_bytes: .config.max_bytes, discard: .config.discard,
       retention: .config.retention, state: .state}'

# If bytes are climbing, find stalled consumers on the stream
curl -s "http://localhost:8222/jsz?streams=true&consumers=true" | \
  jq '.account_details[].stream_detail[] |
      select(.name == "STREAM_NAME") |
      .consumer_detail[]? |
      {name: .name, num_pending, num_ack_pending, num_redelivered}'

Reading the output:

• max_bytes: -1 means unbounded. Treat every unbounded stream as a liability until justified.
• A first_seq that never advances while messages grows means nothing is being reclaimed: either no discards are happening (DiscardNew) or consumers are not acking (interest or workqueue retention).
• api.errors is cumulative. Compute the rate between two scrapes; a sustained positive rate during the incident window confirms active rejections.

How to diagnose it

1. Confirm the rejection is a per-stream limit, not the aggregate. Compare per-stream state.bytes against config.max_bytes for the erroring stream, and compare /jsz storage against reserved_storage. If the stream ratio is near 1.0 and the aggregate is not, you have the per-stream case.
2. Read the discard policy. DiscardNew means producers are being rejected. DiscardOld means the stream is silently deleting its oldest messages. The remediation and the urgency of data-loss assessment differ completely.
3. Check the retention policy and consumer state. With interest retention, messages are deleted only when all consumers have acknowledged them, so one stalled consumer blocks all space reclamation. With workqueue, any one consumer’s ack frees the message. With limits, retention is purely policy-driven. Find out which you are running before touching anything.
4. Determine whether the stream is over-retaining or undersized. If consumers are healthy and keeping up, the limit is simply too small for current publish volume. If consumers are stalled (num_ack_pending at the configured MaxAckPending, or a dead consumer still registered), the limit may be fine and the real fault is the consumer.
5. If the aggregate limit is the one being hit, find the unbounded stream. Sort per-stream bytes descending. The stream consuming a disproportionate share, especially one with max_bytes: -1, is your culprit. Fixing its limit or retention fixes every other stream.
6. Check the server version against known accounting bugs. If the stream grew past its configured max_bytes at all, suspect the 0-byte file accounting bug on versions before 2.9.17, particularly after unclean shutdowns.
7. For streams failing at creation time, remember the reservation behavior: the sum of configured max_bytes values for memory streams must fit inside max_memory_store, and the sum for file streams must fit inside max_file_store, regardless of real usage.

Metrics and signals to monitor

Use a ratio, not an absolute threshold: alert when state.bytes / config.max_bytes exceeds 0.8 per stream. Absolute thresholds break across streams of different sizes. Per-stream data requires /jsz?streams=true, which returns much more data than the bare endpoint; on servers with very many streams, poll it at a slower interval than your core metrics.

Fixes

The limit is too small for real traffic

Raise max_bytes on the stream. It is editable after creation:

# Raise the limit on an existing stream (example: 50 GiB)
nats stream edit STREAM_NAME --max-bytes 50GB

One destructive caveat: lowering max_bytes on a stream that currently holds more than the new limit deletes messages immediately to fit the new limit, with no grace period. Raising is safe; shrinking deletes data on the spot.

Also confirm the server and account can afford the new per-stream limit. The sum of all stream limits should fit inside reserved_storage with at least 25% aggregate headroom to absorb bursts, consumer slowdowns, and compaction overhead.

The stream is full because consumers are stalled

Raising max_bytes buys time but does not fix the fault. Fix the consumer: restart or scale the consuming application, and check whether num_ack_pending is pinned at the consumer’s MaxAckPending (delivery stalled) or whether the consumer is simply gone while still registered. Under interest retention, a dead consumer blocks all reclamation. Deleting a consumer you have confirmed is dead immediately makes its pending messages eligible for deletion; understand that this discards those messages.

For a DiscardNew stream that has already hit the wall: it does not auto-recover when consumers drain, because acknowledged messages are not removed under limits retention. Your options are to purge messages explicitly, raise max_bytes, or change the discard policy to DiscardOld. Purging is destructive; confirm downstream consumers have processed or no longer need the data first.

An unbounded stream is eating shared storage

Set an explicit max_bytes on the offending stream, sized from its real growth rate, and choose a discard policy deliberately. This immediately bounds its blast radius. If the aggregate limit has already tripped and other streams are rejecting, freeing space from the runaway stream (purge, or a lower max_bytes with the immediate-deletion caveat above) restores service for everything else.

To prevent recurrence at the account level, nats-server supports max_bytes_required (since v2.7.0): when set, every stream in the account must declare an explicit max_bytes, so new unbounded streams cannot be created. Companion account settings memory_max_stream_bytes and disk_max_stream_bytes cap how large any single stream’s limit may be. These guardrails make this incident class structurally impossible.

You are on an affected server version

If per-stream accounting has been violated (stream past its own limit) and you are pre-2.9.17 with a history of unclean shutdowns, upgrade. When sizing streams, size max_memory_store against the sum of configured memory-stream max_bytes, and max_file_store against the sum of configured file-stream max_bytes, not observed usage.

Prevention

• Set max_bytes on every stream, no exceptions. Treat -1 as a misconfiguration. Where you cannot set it organizationally, enforce it technically with account-level max_bytes_required.
• Alert on the per-stream ratio. Warn at state.bytes / config.max_bytes > 0.8, critically above 0.9, per stream. Aggregate storage alerts alone will never catch this incident.
• Choose the discard policy deliberately. DiscardOld is silent data loss for slow consumers. DiscardNew is loud producer failure. Neither is wrong, but you should know which failure mode each stream will exhibit before it happens.
• Match retention to consumer reality. Interest retention plus one flaky consumer equals a stream that never reclaims. If consumers are unreliable, limits retention with a sane max_bytes and DiscardOld may be safer.
• Budget limits against capacity. Sum the per-stream max_bytes values per server and per account, and keep the total inside the aggregate quota with at least 25% headroom. JetStream reserves configured max_bytes against the matching storage quota, so this is not optional.
• Include per-stream utilization in capacity review. A stream trending from 0.5 to 0.8 of its limit over a month gives you weeks of runway to resize. See the capacity section of how NATS works in production for the broader saturation model.

How Netdata helps

• Netdata collects JetStream aggregates from /jsz (bytes, messages, memory, storage, reserved quotas, API totals and errors), so the server-level side of this incident is visible continuously rather than on demand.
• The api.errors rate is the lagging signal for publish rejections; charting it alongside storage utilization tells you whether rejections correlate with a storage ceiling or with something else (Raft, disk I/O).
• Correlating JetStream storage growth with consumer-side symptoms (publish rates, throughput asymmetry) helps distinguish “stream undersized” from “consumer stalled” without logging into the server.
• Uptime tracking catches the unclean-shutdown scenario behind the 0-byte file accounting bug on older server versions, which is when per-stream limits stop being trustworthy.
• Netdata’s NATS collector currently reports aggregate JetStream figures, not per-stream state.bytes / config.max_bytes ratios; per-stream alerting requires polling /jsz?streams=true separately.

Related guides

• NATS monitoring checklist: the signals every production server needs
• How NATS actually works in production: a mental model for operators
• NATS context deadline exceeded: JetStream publish and request timeouts
• NATS JetStream disabled unexpectedly: the persistence subsystem failed to come up
• NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check
• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS connection churn: a stable connection count hiding constant reconnects
• NATS connection storm: reconnect thundering herd after a network event
• NATS file descriptor exhaustion: too many open files and the ulimit cliff
• NATS Maximum Connections Exceeded: new clients rejected at the max_connections wall
• NATS Maximum Payload Violation: messages rejected for exceeding max_payload
• NATS messages published but not received: subject mismatches and cross-server gaps