NATS JetStream disabled unexpectedly: the persistence subsystem failed to come up

The server is up. Clients connect, core NATS routes messages, /healthz?js-server-only=true returns ok. But /jsz reports disabled: true on a node where JetStream should be running, and every stream, consumer, and KV bucket backed by this node is gone or degraded. Publishers that rely on persistence fail; request-reply still works, which is why this gets noticed late.

This state means one of two things: JetStream failed to initialize at startup, or it initialized and was later shut down by the server itself. In both cases the root cause is almost always below the NATS layer: the storage directory, the disk underneath it, or the configuration. The server rarely recovers on its own, and a blind restart can make a corrupt store worse.

This guide covers how to confirm the state, separate initialization failure from runtime shutdown, find the cause, and alert without paging yourself during normal recovery. For the broader picture of how JetStream fits into the server’s internals, see How NATS actually works in production.

What this means

JetStream is a subsystem, not the server. The process can be perfectly healthy, passing basic readiness (/healthz?js-server-only=true), while JetStream is off. The disabled field in /jsz tells you exactly one thing: JetStream is not enabled on this node right now. It does not tell you whether streams are intact on disk, whether a clustered peer has taken over, or whether the subsystem would survive a restart.

Two semantic traps:

disabled=true is not the same as “JetStream is broken.” It means not enabled. A node mid-recovery on a large store can transiently report disabled during initialization. A node where JetStream was never configured also reports disabled, and that is normal.
disabled=false is not the same as “JetStream is working.” Enabled only means the subsystem is loaded. For actual working state, bare /healthz on a JetStream-enabled server performs the full health check, including JetStream readiness, meta recovery, and asset recovery. Use bare /healthz as a TICKET-level signal, because it legitimately fails during post-restart asset recovery on large stores.

The documented failure modes behind this state are: storage directory inaccessible, insufficient disk space, or a configuration error. In practice, filesystem-level events (a volume that remounted read-only, a corrupted Raft log) present the same way, because the server’s response to a critical storage error is to shut JetStream down rather than serve corrupted state.

flowchart TD
  A[/jsz disabled=true/] --> B{Uptime < 600s?}
  B -- yes --> C[Wait: init on large stores\ncan report disabled transiently]
  B -- no --> D{Sustained >= 5 min?}
  D -- no --> C
  D -- yes --> E[PAGE: JS expected but disabled]
  E --> F[Read server logs for disable reason]
  F --> G{Cause}
  G -- storage dir / disk --> H[Fix mount, space, permissions]
  G -- config error --> I[Fix config, reload or restart]
  G -- corrupt store / Raft --> J[Restore peer or rebuild from replicas]

Common causes

Cause	What it looks like	First thing to check
Storage directory inaccessible	JS never comes up after restart; log shows store open/create errors at startup	`ls -ld` and mount status of the configured store directory
Insufficient disk space	JS came up, ran, then went disabled; disk at or near 100% on the store volume	`df -h` on the store directory filesystem
Configuration error	JS disabled from the very first boot of a new or changed config	Validate the jetstream config block; check startup log lines
Filesystem went read-only or errored at runtime	JS ran fine, then disabled mid-flight after a storage event	`dmesg`, mount table, log lines around the disable timestamp
Corrupt store or Raft state	Restart loops or immediate disable after crash/power loss	Log lines referencing store or Raft recovery; peer state in `/jsz` meta_cluster
Config reload dropped JetStream	JS was enabled via CLI flag, then disabled after a config reload	Whether the config file actually contains a jetstream block

On the last row: JetStream enabled only via the -js CLI flag, with no matching jetstream block in the config file, is fragile across config reloads. A reload that replaces runtime flags with file-based config can silently turn the subsystem off. The durable fix is to put JetStream in the config file, not the command line.

Quick checks

All read-only. Default monitoring port is 8222.

# Confirm the disabled state
curl -s http://localhost:8222/jsz | jq .disabled

# Rule out cold start: JS may report disabled during init on large stores
curl -s http://localhost:8222/varz | jq .uptime

# Is the server itself fine? (basic readiness, ignores JS depth)
curl -s http://localhost:8222/healthz?js-server-only=true

# The explicit "is JetStream enabled" check
curl -s http://localhost:8222/healthz?js-enabled-only=true

# Full JS health (readiness, meta recovery, asset recovery)
# Expect this to FAIL while JS is disabled, and also during legit recovery
curl -s http://localhost:8222/healthz

# API pressure before it went dark (errors climbing = it was already sick)
curl -s http://localhost:8222/jsz | jq '{api_total: .api.total, api_errors: .api.errors, inflight: .api.inflight}'

# Disk space on the store filesystem (substitute your configured store dir)
df -h /var/lib/nats/jetstream

# Store directory exists, is a mount if it should be, and is writable by the server user
ls -ld /var/lib/nats/jetstream
mount | grep jetstream
findmnt -no OPTIONS /var/lib/nats/jetstream   # look for "ro"

# Kernel-level storage events around the time JS went down
dmesg -T | grep -iE 'error|remount|read-only' | tail -20

# The server log is authoritative for WHY it disabled JS
grep -iE 'jetstream' /var/log/nats/nats-server.log | tail -50

The log lines are the decisive evidence. When the server disables JetStream at runtime due to a storage fault, it logs a critical error naming the failure before shutting the subsystem down. Older versions logged the resource error without the failing path; newer versions include it.

How to diagnose it

Confirm and gate. Verify /jsz disabled=true, then check uptime. If uptime is under about 10 minutes and the store is large, you may be looking at initialization, not failure. Wait and re-check before touching anything. Treat it as an incident only if the state sustains 5 minutes past the cold-start window.
Read the log at the transition point. Find the timestamp where JetStream went from enabled to disabled (or where startup should have enabled it). The lines immediately before the disable event name the cause: store open failure, no space left, read-only filesystem, or a Raft recovery error. This single step resolves most cases.
Check the filesystem, not just the directory. A store directory that exists is not enough. Verify the underlying filesystem is mounted, has free space, and is mounted read-write. findmnt showing ro after a kernel error remount is a classic cause: the directory looks fine, every write fails.
Distinguish initialization failure from runtime shutdown. If JS never enabled on this boot, the problem is config or store-open time: wrong path, wrong permissions, config syntax, or a store the server cannot recover. If JS ran and then disabled, the problem is a runtime event: disk filled, filesystem errored, or a critical write failure. These have different fixes and different recurrence risks.
If clustered, assess blast radius before acting. Check the meta cluster view: curl -s http://localhost:8222/jsz | jq '.meta_cluster | {leader, replicas: [.replicas[]? | {name, current, offline, lag}]}'. If peers are current, this node’s streams can be rebuilt from replicas, which changes what “fix” is safe. If this node held leaders, expect election activity and elevated api.errors cluster-wide while it is out.
Decide: repair, rebuild, or restore. Config and disk-space problems are repaired in place. Corrupt store or Raft state on a clustered node is usually rebuilt by wiping the local store and letting it re-sync from peers, which is destructive to local data and only safe when replicas are confirmed current elsewhere. A standalone node with a corrupt store is a restore-from-backup situation.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`/jsz` `disabled`	The symptom itself; gate on uptime > 600s and 5 min sustained	`true` when JS is expected
`/varz` `uptime`	Separates cold-start init from a real disable	Uptime high but JS disabled
Bare `/healthz`	Full JS health: readiness, meta recovery, asset recovery	Non-ok outside known recovery windows (TICKET, not PAGE)
`/jsz` `api.errors` rate	JS was often sick before it died: storage-full rejections, internal errors	Sustained positive rate before the disable
`/jsz` `api.inflight`	Raft or disk latency building before failure	Sustained high vs baseline
`/jsz` `storage` vs `reserved_storage`	Disk exhaustion is a leading cause of runtime disable	Ratio approaching 90%
`/jsz` `meta_cluster` replicas	Clustered: tells you if rebuild-from-peers is safe	Peers `offline=true` or `current=false`
Filesystem free space + mount options (OS level)	The actual root cause for most cases	Free space dropping, `ro` remount

Fixes

Config error. Fix the jetstream block in the config file (path, limits, syntax). If JetStream was enabled only via CLI flag, move it into the config file so reloads cannot drop it. A config reload covers some changes; if the store directory itself changed, plan a restart and expect recovery time on large stores.

Storage directory inaccessible. Restore the mount, fix ownership and permissions for the server user, then restart. JetStream re-opens and recovers the store at startup. Do not point it at an empty directory as a shortcut unless you intend to rebuild the data.

Insufficient disk. Free space or grow the volume, then restart. Also fix the reason it filled: consumer stalls and retention policy are the usual upstream causes. If you only add disk, you are buying time, not fixing the leak. See the storage exhaustion pattern in NATS monitoring checklist.

Filesystem read-only or errored. Remount read-write only after the underlying device is confirmed healthy (dmesg clean, no pending I/O errors). If the kernel remounted ro because of corruption, run filesystem checks with the server stopped. A store written to while the filesystem was erroring may need recovery time or a peer rebuild afterwards.

Corrupt store or Raft state, clustered. Confirm replicas on other nodes are current via /jsz meta_cluster and per-stream info. Then, with the node stopped, remove the local store directory and start the node; it rejoins and re-replicates from peers. This is destructive to anything that only existed on this node, which is why the peer-current check comes first. Do not do this if quorum across the remaining nodes is already at risk.

Corrupt store, standalone. Restore from backup/snapshot. There is no peer to rebuild from. If backups do not exist, assess whether the streams can be repopulated from producers, and treat this as the incident that justifies a backup strategy.

In all cases, a restart is part of the fix but never the diagnosis. Restarting without knowing why JS disabled risks a crash-loop pattern if the store is corrupt; see NATS crash loop: unexpected uptime resets and repeated restarts.

Prevention

Alert with the right gate. PAGE on: JS expected on this node, disabled=true, uptime > 600s, sustained >= 5 minutes. The uptime gate eliminates cold-start false pages (init on large stores); the duration gate eliminates transient states.
Use the healthz variants for their intended purpose. js-server-only=true for process-liveness PAGEs, bare /healthz for deep JS health as a TICKET. Details in NATS /healthz explained.
Alert on the leading indicators, not just the corpse. Storage approaching reserved_storage, rising api.errors, and high api.inflight usually fire minutes to hours before the server shuts JetStream down.
Put JetStream in the config file. CLI-flag-only enablement is fragile across reloads.
Monitor the filesystem, not just the store. Free space, mount presence, and read-write state on the store volume are OS-level checks that catch the top root causes directly.
Treat backup and snapshot jobs as risk windows. Backup-induced I/O stalls cause transient JS distress; know when they run so you can distinguish them from real failure.

How Netdata helps

The gated disabled signal: Netdata’s NATS collector polls /jsz and tracks the disabled flag alongside /varz uptime, the exact pair you need for the uptime-gated, duration-gated PAGE without custom scripting.
Leading-indicator correlation: JetStream storage usage vs reserved quota, api.errors rate, and api.inflight are collected together, so the “JS was sick before it died” pattern is visible on one dashboard instead of three curl commands.
Health check semantics built in: server health, JS-enabled state, and deep JS health map to the distinct /healthz variants above, so recovery transients do not page you during legitimate asset recovery.
Context at the transition point: per-second retained metrics let you scroll back to the moment JS disabled and see whether disk, API errors, or inflight moved first.
Cluster view: meta cluster state and per-node disabled flags side by side tell you whether a rebuild-from-peers is safe before you touch the broken node.