NATS JetStream disk I/O stall: the disk has space but is too slow

JetStream publishes are timing out or creeping upward in latency. df -h shows plenty of free space. The /jsz endpoint shows storage nowhere near its reserved limits, yet api.inflight sits high and api.errors keeps climbing. On the host, iowait is elevated and disk latency looks bad.

This is the JetStream disk I/O stall pattern: the disk has space but is too slow. It is a different failure from storage exhaustion, and it is frequently misdiagnosed because the obvious capacity checks all pass.

If the cluster is degraded further, you will also see Raft symptoms: meta cluster leader changes, stream leaders flapping, and write rejections that look like consensus failures but are storage latency underneath.

What this means

JetStream file storage uses append-only message blocks and index files; a replicated stream also appends to its Raft WAL, and the meta group has its own Raft WAL. These append paths share the storage path underneath store_dir, so a slow disk slows everything that depends on completed writes.

The cascade looks like this:

flowchart TD
  A[Slow disk: high fsync latency] --> B[WAL appends slow]
  B --> C[Raft heartbeats delayed]
  C --> D[Election timeouts, leader flapping]
  B --> E[Publish acks delayed]
  D --> F[api.errors climbing]
  E --> G[api.inflight high and sustained]
  F --> H[Client publish timeouts]
  G --> H

Two properties define this failure:

• Capacity metrics stay green. Storage usage, reserved_storage ratios, and filesystem free space are all fine. Only latency metrics are red. This is why teams burn time checking retention policies and stream limits when the real problem is the storage device.
• Raft makes it a cluster problem. A slow disk on one node delays that node’s WAL appends. Raft heartbeats and log replication stall, elections fire, and streams led by that node go unavailable for writes even though other nodes have healthy disks. WAL fsync latency is the strongest predictor of Raft instability in a JetStream cluster.

The most common underlying cause in cloud deployments is network-attached storage (EBS, NFS, EFS and similar) with variable or burstable latency. Local SSDs are recommended for JetStream storage for exactly this reason.

Common causes

One differential check: “no space left on device” errors can occur even when df -h shows free space if the filesystem is out of inodes. JetStream creates many small files (message blocks, WAL segments), so on inode-constrained filesystems this can masquerade as a storage problem. Check df -i before ruling capacity out entirely.

Quick checks

All read-only and safe to run during an incident.

# JetStream API pressure: inflight high + errors climbing is the signature
curl -s http://localhost:8222/jsz | jq '{inflight: .api.inflight, total: .api.total, errors: .api.errors}'

# Confirm storage is NOT the problem (distinguishes from exhaustion)
curl -s http://localhost:8222/jsz | jq '{storage, reserved_storage, memory, reserved_memory}'

# Meta cluster stability: leader changes and lagging peers point at Raft distress
curl -s http://localhost:8222/jsz | jq '.meta_cluster | {leader, replicas: [.replicas[]? | {name, current, offline, lag}]}'

# Disk latency and utilization on the JetStream store device (repeat every 2s)
iostat -xz 2 5

# Inode headroom (rules out the df -h false-negative)
df -i /path/to/jetstream/store

# Kernel-level storage faults
dmesg -T | grep -iE 'i/o error|ext4|xfs|nvme|blocked for more than'

Two notes on interpretation:

• Poll /jsz a few times over a minute rather than once. A single snapshot of api.inflight can catch a burst; the stall pattern is persistently high inflight with a rising error counter.
• In iostat -xz output, await is the average time (ms) a request spends in queue plus service. On healthy local SSD storage for JetStream, expect low single-digit milliseconds under normal load. Sustained double-digit await, or high %util combined with low throughput, is the smoking gun.

How to diagnose it

1. Confirm the symptom pair. Pull /jsz three or four times over 60 seconds. If api.inflight is persistently elevated and api.errors is increasing between polls while storage is well under reserved_storage, you are in the I/O stall pattern, not storage exhaustion.

2. Check client-side write latency. Ask the publishing teams (or check client metrics/traces) whether publish ack latency rose at the same time inflight climbed. Rising publish latency plus rising inflight confirms the write path is blocked downstream of the server logic.

3. Measure the disk directly. Run iostat -xz 2 5 against the device backing the JetStream store_dir. High await, high %util, or throughput far below the device’s known ceiling confirms the storage layer. If await is high but %util and throughput are low, suspect network storage latency rather than local saturation.

4. Check for coinciding processes. Look at backup agents, snapshot jobs, and cron schedules that overlap with the latency window. Filesystem backups and volume snapshots are a classic cause of JetStream I/O distress that resolves on its own when the job finishes. Check dmesg for I/O errors, which indicate hardware degradation instead.

5. Assess Raft impact. Pull meta cluster state from /jsz and per-stream Raft state from /raftz. A recently changed meta leader, replicas with current: false or growing lag, or stream groups cycling leaders mean the stall has propagated into consensus. The blast radius changes from “slow writes on one node” to “intermittent write unavailability for led streams.”

6. Identify which cause fits. Compare findings against the causes table: burstable volume metrics, noisy-neighbor correlation, backup windows, stream count growth, or hardware errors. The fix depends entirely on which one it is.

Metrics and signals to monitor

Fixes

Network-attached storage is the bottleneck

Move the JetStream store_dir to local SSD. This is the durable fix and the configuration NATS operators consistently land on after this incident. Network-attached storage with variable latency is the number one cause of Raft election storms in JetStream clusters, because Raft WAL writes are latency-sensitive and election timeouts care about tail latency, not your monthly average.

If you cannot move off network storage immediately, provisioned-IOPS volumes (rather than burstable general-purpose volumes) reduce tail latency variance. Treat that as mitigation, not resolution.

Burstable IOPS exhausted

If cloud volume metrics show burst credits depleted, either switch to a provisioned-IOPS volume class sized for sustained JetStream write load, or reduce the write rate hitting the volume (fewer streams, lower replication factor, lower message rate). Burstable volumes are a poor fit for a synchronous WAL workload because the failure arrives 30+ minutes into a sustained event, mid-incident.

Backup or snapshot interference

Reschedule backups and volume snapshots away from peak JetStream write windows. If the storage system supports it, use snapshot mechanisms that do not stall writes on the source volume. Alerting nuance: backup-induced JetStream distress can make bare /healthz flap, which is why the playbook recommends /healthz?js-server-only=true for page-level alerts and bare /healthz as a ticket-level signal.

Too many streams sharing one disk

Reduce stream count or spread streams across nodes so no single disk carries the aggregate WAL load. Replicated streams multiply write load: an R=3 stream writes on three servers. Consolidating many small streams into fewer subjects within one stream also reduces the number of independent write paths and Raft groups competing for the same device.

Disk degradation

If dmesg shows I/O errors or latency is climbing monotonically over days, replace the device. In a cluster, migrate stream leaders off the affected node first to limit write unavailability during the swap.

Raft instability already in progress

Fixing the disk is the fix; do not try to tune around a slow disk with Raft timeout changes. Once storage latency recovers, elections should stop on their own. If a specific stream remains leaderless after the disk recovers, check its group via /raftz and investigate that group individually.

Prevention

• Put JetStream storage on local SSDs. The single highest-leverage decision. If organizational constraints force network storage, use provisioned IOPS and load-test fsync latency at p99, not averages.
• Baseline disk latency, not just capacity. Alert on api.inflight sustained above baseline and on OS-level disk await for the store device. Capacity alerts on reserved_storage will never fire for this failure mode.
• Track the error ratio. Alert on api.errors rate and on the errors/total ratio rather than absolute counts; idempotent client operations generate benign errors that inflate the raw counter.
• Separate the health probes. Use /healthz?js-server-only=true for paging (process readiness) and bare /healthz as a ticket signal, so JetStream I/O distress pages as “investigate” rather than “server down.”
• Watch Raft leader stability as early warning. Frequent meta or stream leader changes often precede visible publish failures by minutes. Leader-change rate is a cheap proxy for WAL fsync health.
• Coordinate backup windows. Keep filesystem backups and volume snapshots out of peak write windows, and verify your snapshot mechanism does not quiesce writes.

How Netdata helps

• Netdata polls the NATS HTTP monitoring endpoints, so api.inflight, api.errors, and api.total from /jsz are charted over time, making the “persistently high inflight with rising errors” signature visible at a glance instead of requiring manual polling loops.
• Per-second system metrics (iowait, disk await, disk utilization, throughput per device) sit on the same dashboard as the NATS application metrics, so correlating a JetStream API slowdown with a specific device’s latency spike takes one look instead of two tools.
• Uptime, connections, and throughput charts help rule out the lookalike failures: connection churn, slow consumers, and storage exhaustion present differently in the same views.
• High-resolution history lets you align the exact minute a backup job or burst-credit exhaustion started with the minute api.inflight began climbing, which is usually the fastest path to root cause.
• Alerting on ratios and rates (error rate, inflight vs baseline) rather than absolute counters fits how these JetStream signals actually behave in production.

Related guides

• NATS connection churn: a stable connection count hiding constant reconnects
• NATS connection storm: reconnect thundering herd after a network event
• NATS context deadline exceeded: JetStream publish and request timeouts
• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS file descriptor exhaustion: too many open files and the ulimit cliff
• NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check
• How NATS actually works in production: a mental model for operators
• NATS JetStream API errors: reading the /jsz api.errors counter without false alarms
• NATS JetStream disabled unexpectedly: the persistence subsystem failed to come up
• NATS JetStream not enabled for account: persistence calls failing on a core server
• NATS Maximum Connections Exceeded: new clients rejected at the max_connections wall
• NATS Maximum Payload Violation: messages rejected for exceeding max_payload