NATS context deadline exceeded: JetStream publish and request timeouts

Your application logs fill up with nats: context deadline exceeded. It shows up on js.Publish(), on js.StreamInfo(), on consumer fetches, sometimes on nats CLI commands like nats stream view. The NATS server is running, /healthz returns ok, and core NATS pub/sub traffic is flowing fine. Yet every JetStream operation hangs until the client’s context fires.

This error is a client-side deadline expiring. The client sent a JetStream API request and no response came back before the deadline. The request is not being rejected; it is not being answered at all. Somewhere between the client and the JetStream subsystem, the request is stuck, and the server is usually still healthy enough to look innocent.

The fix is almost never “increase the client timeout.” The timeout is telling you the server is slow to ack publishes or slow to answer API calls, and there are a small number of server-side causes that produce exactly this shape: JetStream disk I/O stalls, Raft leader instability, meta-cluster elections, an overloaded server, or an API backlog the server cannot drain.

What this means

Every JetStream operation, publish-ack, consume, stream info, consumer create, is a request-reply exchange over $JS.API.> subjects. The client library wraps the call in a context with a deadline and waits for the reply. When the reply does not arrive in time, the context fires and you get context deadline exceeded. The client has no visibility into why; it only knows nobody answered.

Server-side, there are four ways a request gets stuck long enough to blow the deadline:

1. The publish-ack is waiting on storage. A JetStream publish is not acked until the message is written to the stream’s WAL. If fsync latency on the storage path spikes, every ack stretches out and publishes start timing out.
2. The write needs Raft consensus and there is no stable leader. In clustered JetStream, writes require the stream’s Raft group, and metadata operations require the meta group. During an election there is no leader, so proposals queue and clients wait.
3. The API backlog is growing faster than the server can drain it. /jsz exposes api.inflight: the number of in-progress JetStream API requests. When inflight climbs and stays high, the server is slow to process API requests, usually because of the two causes above or plain CPU saturation.
4. The server or the network path is shedding the request entirely. Stale connections and broken pipes on the server side present to the client as a deadline, not an error.

This is not the same failure as a core NATS request timeout. A core request that times out or returns no responders available means the responder application was slow or absent; the broker did its job. With JetStream timeouts, the broker itself is the slow responder. If you are seeing plain request-reply timeouts without JetStream involvement, see NATS no responders available for request instead.

flowchart TD
  A[context deadline exceeded
on JetStream call] --> B{api.inflight high
and sustained?}
  B -->|yes| C{meta_cluster leader
changing frequently?}
  B -->|no| D{Server overloaded?
CPU, stale connections}
  C -->|yes| E[Raft instability:
check route RTT, GC pauses,
disk latency on WAL path]
  C -->|no| F{Storage near limits?}
  F -->|no| G[Disk I/O stall:
iowait, fsync latency,
backup or noisy neighbor]
  F -->|yes| H[Storage exhaustion:
consumer stall or retention]
  D -->|yes| I[CPU saturation or
connection path issues]
  D -->|no| J[Per-stream Raft group issue:
check /raftz for that stream]

Common causes

One cause worth singling out because it surprises operators: consumer-count scaling. Operators running on the order of 50-60K consumers on a stream have reported persistent context deadline exceeded, and the maintainer response on nats-server issue #5609 was that R3 consumers each create a full Raft group, which does not scale. The recommended alternatives are fewer consumers with subject filters, or republishing from the stream to core NATS subscribers. If your consumer count is in the tens of thousands, this is probably your root cause, not disk or network.

Quick checks

All read-only. Run them against the monitoring port (default 8222).

# Is JetStream up, and what does the API backlog look like?
curl -s http://localhost:8222/jsz | jq '{disabled, api: .api, streams, consumers}'

# Meta cluster health: leader name and replica state
curl -s http://localhost:8222/jsz | jq '.meta_cluster | {leader, replicas: [.replicas[]? | {name, current, offline, lag}]}'

# Server load and connection-path distress
curl -s http://localhost:8222/varz | jq '{cpu, mem, connections, slow_consumers, stale_connections, stalled_clients}'

# Is this a cold start or recent restart?
curl -s http://localhost:8222/varz | jq .uptime

# Health: full JetStream check vs basic server readiness
curl -s http://localhost:8222/healthz
curl -s http://localhost:8222/healthz?js-server-only=true

Then at the OS level on the JetStream nodes:

# Disk latency and saturation on the JetStream storage path
iostat -x 2 5

# Any backup, snapshot, or compaction process hammering the disk?
ps aux | grep -Ei 'backup|snapshot|rsync|tar'

In the server logs, look for Stale Client Connection - Closing, broken pipe, new leader, and Stepping down entries correlating with the timeout window.

Reading the results:

• api.inflight near zero and errors flat: the server is not backlogged right now. Reproduce the failing call while watching, or suspect a single stream’s Raft group or the connection path.
• api.inflight high and leader stable: point at disk I/O. Confirm with iostat (high await, high %util on the store device).
• api.inflight high plus a leader name that changes between polls: Raft instability. This is the meta cluster if admin calls fail cluster-wide, or per-stream groups if only one stream is affected.
• api.errors climbing with publishes rejected and storage near reserved_storage: exhaustion, a different incident. Deadlines here usually mean the consumer stall behind the exhaustion has also wedged the API.

How to diagnose it

1. Confirm the scope. Does the deadline hit all JetStream calls or only one stream or one account? Compare nats stream info on a known-good stream versus the failing one. If everything times out, think meta cluster, disk, or server-wide overload. If one stream times out, think that stream’s Raft group.

2. Sample api.inflight over 30-60 seconds. A single snapshot lies; brief spikes during stream creation are normal. Poll /jsz a few times. Sustained high inflight means the server is slow to respond, not that clients are misbehaving.

3. Check Raft stability. Poll the meta_cluster leader field repeatedly during the incident window. More than one leader change in 5 minutes is concerning. Check replicas for offline=true or current=false. For a single affected stream, check /raftz and the stream’s cluster state: any replica lagging or offline reduces quorum headroom and can block writes. Standalone (single-node) JetStream has no meta cluster, so meta_cluster will be null; if that is your topology, skip to step 4.

4. Check disk I/O on the storage path. JetStream file stores are fsync-latency sensitive. High iowait, high device await, or an active filesystem backup/snapshot during the incident window is enough to explain stretched publish-acks. Network-attached storage with variable latency is the classic offender and the number one cause of downstream Raft election storms, because slow WAL writes delay Raft heartbeats.

5. Check server saturation. /varz CPU sustained above 90 percent, memory climbing toward the container limit, and non-zero stale_connections or stalled_clients all point to a server too busy to answer API calls promptly. Also count consumers: a very large consumer population, especially replicated, loads the server with Raft groups and is a known timeout cause.

6. Rule out recovery and lifecycle artifacts. If uptime is low, JetStream may still be recovering assets; bare /healthz fails during this window on large stores while ?js-server-only=true stays ok. Also check whether JetStream was toggled by a config reload: if JetStream is disabled or was disabled during a reload, in-flight API calls hang until their contexts expire rather than erroring out.

7. Correlate with the client. Note the client’s configured deadline and which operation timed out. Publish-ack timeouts point at the write path (storage, stream Raft). Admin call timeouts (stream info, consumer create) point at the meta group or API backlog. Consume/fetch timeouts can also be a stalled delivery path, so check num_ack_pending against MaxAckPending on the affected consumer before blaming the server.

Metrics and signals to monitor

Fixes

Disk I/O stall

Move JetStream storage to local SSDs if it is on network-attached storage. Schedule filesystem backups and snapshots away from peak publish windows, or use a mechanism that does not stall the store device. Reduce the number of streams sharing one disk, or reduce the publish rate while you remediate. Faster storage is the real fix; tuning retention and rates only buys headroom.

Raft instability

Fix the underlying trigger, not Raft itself: route RTT between nodes, GC pressure from memory headroom, disk latency on the WAL path. If one node is chronically slow, its stream leaders will flap; investigate that node’s CPU steal, disk, and network. Elections usually settle once the resource problem clears. Do not restart nodes as a first move; a rolling restart is a last resort to break a self-sustaining election storm, and it will itself trigger elections.

Single broken stream

If one stream’s Raft group has lost quorum (two of three replicas offline or partitioned), restore connectivity to the peers first. If the group cannot recover on its own, operator intervention on that stream may be required; treat this as a data-path incident and verify replica state before and after any action.

API backlog and overload

Reduce concurrent administrative operations: clients that poll stream info or recreate consumers in tight loops inflate both api.total and the backlog. Add jitter and backoff to admin tooling. If the backlog is from legitimate load, scale the JetStream nodes or redistribute stream leaders so one node is not answering for everything.

Consumer-count scaling

Consolidate to fewer consumers with subject filters, or republish from the stream to core NATS subscribers instead of giving every subscriber its own durable consumer. Avoid R3 consumers at high counts; each one is a full Raft group. This is an application-design fix, not a server tune.

Client-side mitigations

Use explicit contexts with deadlines sized to your real durability SLO, and treat deadline errors as retryable with backoff, since JetStream publish and admin operations are generally safe to retry when idempotent. Retry without backoff during a server-side stall just deepens the inflight pile. Do not paper over server stalls by raising timeouts to minutes; you will convert visible errors into invisible queueing.

Prevention

• Alert on api.inflight as a leading indicator. It rises before clients start reporting deadlines. Baseline it and alert on sustained deviation.
• Track meta leader stability as a metric, not a log grep. Leader-change rate per 5 minutes is one of the earliest JetStream distress signals.
• Monitor disk latency on the JetStream device independently of capacity. The disk-stall failure mode has free space; it is purely a latency problem.
• Keep storage below 80% of reserved and IOPS below roughly 70% of the device’s proven envelope, per the capacity guidance in the NATS monitoring checklist.
• Set Kubernetes readiness probes with the /healthz semantics in mind. Use ?js-server-only=true for the page-level readiness check and treat bare /healthz failures during recovery as expected; overly aggressive probes restart pods mid-recovery and convert a slow start into a crash loop. See NATS /healthz explained.
• Cap consumer fan-out in application design reviews. If a design calls for thousands of replicated consumers per stream, push back before it reaches production.
• Load-test the write path with fsync-realistic storage before declaring capacity. Benchmarks on tmpfs or burst-credit volumes hide the fsync latency that causes this incident.

How Netdata helps

• Netdata’s NATS collector polls the HTTP monitoring endpoints and charts api.inflight, api.errors, and api.total over time, so you can see whether the backlog built gradually (disk, overload) or step-changed (election, restart) before clients reported deadlines.
• JetStream storage, memory, stream, and consumer counts are collected from /jsz, letting you rule storage exhaustion in or out at a glance during triage.
• Server-side CPU, memory, slow consumers, and connection counts from /varz sit on the same dashboard as the JetStream signals, which is what makes the “overloaded server vs storage vs Raft” split fast.
• Netdata also collects node-level disk I/O latency and iowait, so you can overlay the store device’s await curve on the api.inflight curve and confirm or eliminate the fsync-stall hypothesis in one view.
• Per-second collection granularity catches the short election-and-backlog transients that 60-second scrape intervals miss.

Related guides

• How NATS actually works in production: a mental model for operators
• NATS monitoring checklist: the signals every production server needs
• NATS monitoring maturity model: from survival to expert
• NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check
• NATS no responders available for request: request-reply into the void
• NATS messages published but not received: subject mismatches and cross-server gaps
• NATS connection churn: a stable connection count hiding constant reconnects
• NATS connection storm: reconnect thundering herd after a network event
• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS file descriptor exhaustion: too many open files and the ulimit cliff
• NATS Maximum Connections Exceeded: new clients rejected at the max_connections wall
• NATS Maximum Payload Violation: messages rejected for exceeding max_payload