$ guides / nats ▌

NATS · OPERATIONS PLAYBOOK

NATS has no queue: core messaging is fire-and-forget, and JetStream is a Raft cluster that can stall

A Go subject-router where every message is matched against an in-memory trie and fanned out in real time — so a subscriber that falls behind is dropped, a subject with no listeners is a black hole, and the optional JetStream layer trades that speed for a write-ahead log and per-stream Raft groups that fail in their own ways. We trace how that design behaves under load, where it turns from backpressure into data loss, and what to do when it does.

> Start with the monitoring checklist → # Jump to the full guide list

NATS gets you to production in minutes, then hands you a set of behaviours that only make sense once you understand it never had a queue in the first place.

The defaults work. Until a subscriber falls behind, its write buffer fills, and the server flags it a slow consumer and drops its messages — permanently, with no notification, because core NATS is fire-and-forget. Until a publisher sends to a subject with no subscribers and the message simply vanishes: no error, no log, no metric, only a gap between in_msgs and out_msgs. Until you add JetStream for durability and a consumer quietly freezes at MaxAckPending while every server metric looks green. Until a stream's Raft group loses its leader and writes to it block with context deadline exceeded. Until WAL fsync latency on a network disk starts an election storm that takes the whole persistence layer down.

These guides are written for engineers who already run NATS, not for people learning what a subject is. The goal is the mental model of how the router and JetStream actually behave under load, the failure patterns that keep recurring, the monitoring story that catches them before they page anyone, and the runbooks you wish someone had handed you before your last incident.

How NATS actually runs in production

NATS is a Go process that matches every message against an in-memory subject tree and fans it out to subscribers in real time — there is no queue between publisher and subscriber. JetStream bolts a write-ahead log and per-stream Raft groups on top for durability. Most production failures live between these layers: in the write buffer, on a cluster route, in a consumer's ack state, or in the Raft log — not inside any one of them.

clients / connections

Each TCP connection gets a read and a write goroutine plus a pending write buffer. Every connection costs a file descriptor and memory. <code>max_connections</code> (default 65536) and the OS <code>ulimit -n</code> are two separate walls, and the OS one is usually lower.

CLIENT

subject router (trie)

An in-memory trie maps subjects to subscriptions, with <code>*</code> (one token) and <code>></code> (multi-token) wildcards; queue groups load-balance across members. Matching is real-time fan-out — one publish becomes N deliveries. There is no storage here: an unmatched publish is simply dropped.

ROUTE

slow-consumer backpressure

When a connection cannot drain fast enough its pending buffer grows; past the threshold the server marks it a <code>slow consumer</code> and drops messages (core NATS) or disconnects it. <code>write_deadline</code> sets the server's patience. This applies to clients, routes, gateways, and leaf nodes alike.

BUFFER

cluster fabric

Servers form a full-mesh of TCP <code>routes</code> (N-1 per server); gateways connect clusters into superclusters; leaf nodes bridge edge servers to a hub. A single slow route backs up inter-server delivery cluster-wide — far worse than a slow client.

CLUSTER

JetStream store (WAL)

The optional persistence layer: a write-ahead log on file or memory storage plus an in-memory sequence-to-block index. Retention (<code>limits</code>, <code>interest</code>, <code>workqueue</code>) decides when messages are deleted; hitting a storage limit either rejects new publishes or evicts the oldest.

STORE

JetStream consumers

Each consumer tracks a delivery cursor, ack state, and redelivery timers. <code>MaxAckPending</code> caps in-flight unacked messages; <code>AckWait</code> drives redelivery. <code>num_pending</code>, <code>num_ack_pending</code>, and <code>num_redelivered</code> are the health signals the server itself does not page you on.

CONSUME

JetStream Raft

In a cluster each stream has its own Raft group, plus a meta group for cluster-wide metadata. A leader replicates to followers; lose the leader and writes to that stream block. WAL fsync latency on the storage path is the single strongest predictor of election storms.

RAFT

Go runtime

NATS is Go: RSS follows a GC sawtooth (2x spikes are normal, not a leak), goroutines run ~2 per connection, and <code>GOMAXPROCS</code> defaults to host cores, not the container's cgroup limit. A long GC pause can delay a Raft heartbeat and trigger an election.

RUNTIME

Why this matters: 'NATS dropped my messages' or 'the consumer stopped' can come from one slow client, a slow route degrading the whole cluster, zero subscribers on a fire-and-forget subject, a JetStream consumer frozen at MaxAckPending, a stream whose Raft group has no leader, or WAL fsync latency on a network disk starting an election storm. The symptom rhymes but each layer has a different signal — and a different fix.

The failures you'll actually see

Most NATS incidents fall into a small set of recurring patterns. Recognise the shape, and triage gets dramatically faster.

CRITICAL

The slow consumer cascade

A subscriber cannot keep up, its pending write buffer grows past the limit, and the server flags it a slow consumer — dropping its messages (core NATS, permanent loss) or disconnecting it. The client auto-reconnects, resubscribes, and immediately falls behind again, churning connections. When it happens on a route or gateway instead of a client, inter-server delivery backs up and the blast radius is the whole cluster.

slow_consumers counter incrementing (rate > 0)
slow_consumer_stats showing routes or gateways, not just clients
pending_bytes climbing on specific connections in /connz
out_msgs dropping while in_msgs stays constant

Investigate →

CRITICAL

The file descriptor cliff

Every client, route, gateway, leaf, and JetStream storage file consumes a file descriptor. The default ulimit -n of 1024 is exhausted fast, and this limit is separate from — and usually lower than — max_connections. At the ceiling accept() fails with too many open files, storage files cannot be opened, and routes cannot establish. It is a cliff-edge with no graceful degradation.

too many open files in the server log (accept errors)
New connections refused while existing ones work
connections nowhere near max_connections yet failing
JetStream unable to open new storage files

Investigate →

CRITICAL

JetStream with no leader

A stream's Raft group — or the meta group — cannot elect or keep a leader, so writes to it block. Clients see context deadline exceeded; the JetStream API returns no responders; stream info shows an empty cluster.leader. It follows a partition, a multi-node loss, or an election storm driven by WAL fsync latency. If the meta group has no leader, every JetStream administrative operation fails.

no leader / JetStream cluster not currently available
Rapid Stepping down / new leader log entries
api.errors spiking, api.inflight high in /jsz
Streams intermittently reporting no leader

Investigate →

ACTIVE

The storage exhaustion spiral

Consumers stall, and with interest or workqueue retention the server cannot delete messages until they are acknowledged. Messages accumulate, storage fills, and new publishes are rejected with insufficient storage or maximum bytes exceeded (or, with DiscardOld, the oldest are silently evicted). It is a deadlock: consumers must process to free space, but they are the thing that stalled.

insufficient storage / maximum bytes exceeded on publish
JetStream storage climbing steadily toward the limit
Stream messages growing while first_seq stays static
A consumer pinned at num_ack_pending == MaxAckPending

Investigate →

ACTIVE

The stalled consumer

A JetStream consumer stops receiving even though the server looks completely healthy. num_ack_pending has reached MaxAckPending and the server will not deliver more until messages are acknowledged — no error is raised. The application is not acking (a bug, a crash, a hang) or its processing time exceeds AckWait, so messages redeliver without ever clearing. This is the number-one cause of 'JetStream stopped working'.

num_ack_pending equal to MaxAckPending, num_waiting = 0
num_redelivered climbing with no forward progress
Consumer delivery lag growing while server metrics are green
Consumer connected but ack rate at zero

Investigate →

IMMINENT

Publishing into the void

A request is published to a subject with no subscribers. In request-reply the client fails fast with no responders available for request; in plain pub/sub the message is silently discarded with no signal at all. The responder crashed, was never started, sits behind a down route, or the subject has a typo — and NATS subjects are case-sensitive. Everything at the server looks healthy because, by design, nothing is broken.

nats: no responders available for request (503)
in_msgs active while out_msgs is flat or far lower
subscriptions count below the expected number
Downstream systems reporting missing data

Investigate →

Choosing a tool

Best NATS Monitoring Tools: 9 Ranked for 2026

A ranked review of the tools teams actually shortlist here, what each one is genuinely good at, and how the pricing behaves as you scale.

Read the buyer's guide →

NATS monitoring maturity levels

NATS observability works in four practical levels. Each is a complete operation, not a stepping stone. Pick the level that matches how much your messaging matters. Most production deployments should land at the second level; anyone relying on JetStream needs the third.

Level 1: Survival

Know that something is wrong

Survival monitoring is the floor. With these signals you can answer one question: is the server alive and accepting work? You will not learn what broke, but you will learn that something broke before users do. Survival is enough for dev servers and non-critical pub/sub.

Health probe (/healthz?js-server-only=true) Basic readiness without false-paging during JetStream recovery.
Server uptime An unexpected reset means a crash or forced restart.
Active connections Can clients actually connect, and are any present?
Slow consumers (rate) In core NATS, every event means messages were dropped.
Server memory (RSS) Trend, not spikes — Go GC sawtooth is normal.

Level 2: Operational

Diagnose most incidents on your own

Operational monitoring is what most production deployments should target. Survival tells you something is wrong; operational tells you what. With this coverage your team can usually diagnose an incident on its own: backpressure, saturation, routing loss, JetStream storage, and stalled consumers.

In / out message and byte rates The out/in ratio is your only core-NATS loss signal.
Connections vs max_connections Alert on the ratio, never an absolute count.
Route count vs expected (N-1) A missing route is a cluster partition.
JetStream enabled + api.errors disabled=true or a rising error rate on the persistence layer.
Per-consumer ack_pending and lag For critical consumers — the signal the server won't page on.
Authentication failure rate A spike is credential rotation gone wrong, or probing.

Level 3: Mature

Catch problems before they become incidents

Mature monitoring catches problems before they wake anyone up. Per-connection backpressure building, route RTT creeping, a Raft replica drifting, a consumer redelivering — none of these page you on day one. They become the incident on day thirty, especially anywhere JetStream is load-bearing.

Per-connection pending_bytes Identify a slow consumer before it is disconnected.
Client and route RTT High route RTT is what triggers Raft election timeouts.
Gateway and leaf-node health Cross-cluster and edge connectivity, RTT to the hub.
Raft leader distribution + elections Skew and election frequency lead JetStream outages.
Stream replica state (current/lag) A non-current replica means a lossy failover.
Subscription count trend Growth without new connections is a leak.
TLS certificate expiry An expired cert is a silent, total new-connection outage.
stalled_clients / stale_connections Write-path distress and half-dead sockets.

Level 4: Expert

Reactive instrumentation after real incidents

Expert signals enter your stack the day after a specific incident proved you needed them. WAL fsync latency, Go GC pause, connection churn, per-consumer high-water marks, the $SYS event stream. Most teams never need every signal here. Add the ones your incident history says you do.

JetStream WAL fsync latency Inferred from disk I/O on the JS path — the strongest Raft-instability predictor.
Go GC pause time Pauses over ~10ms can trip Raft election timeouts.
Connection churn rate total_connections delta while the count stays flat.
Fan-out ratio (out_msgs/in_msgs) Shifts reveal subscriber population changes and loss.
Per-stream and meta Raft (/raftz) A stream group can fail while the meta group is fine.
Per-consumer redelivery + max-pending high-water One slow cycle away from a stall.
$SYS event stream Real-time connect/disconnect, slow-consumer, and auth events.
Mirror / source lag For DR mirrors this lag is your recovery point (RPO).

Operating mistakes worth avoiding

The traps NATS teams keep falling into. Each has a clear, well-known fix. Most teams only learn it after an incident.

Monitoring 'is it up?' but not 'is it working?'

A <code>/healthz</code> 200 means the process is ready — not that streams are writable, consumers are draining, or subscribers are connected. Teams check process health and stop there, then discover during an incident that JetStream was disabled, a consumer had stalled, or every subscriber had quietly disconnected. The probe is necessary but nowhere near sufficient.

Not monitoring JetStream consumer lag at all

This is the single most common gap. Server metrics — CPU, memory, throughput — can all look healthy while a consumer is completely stalled and a stream grows toward its limit. Consumer lag is the most important JetStream signal and it is systematically under-monitored because it needs per-consumer polling (<code>num_pending</code>, <code>num_ack_pending</code>), not one server endpoint.

Treating slow_consumers as informational

In core NATS a slow consumer event means messages were <em>dropped</em> — not queued, not retried, permanently gone. Teams from a queuing background assume the messages are somewhere. They are not. Every slow-consumer event in a non-JetStream deployment deserves the same urgency as data loss, and the <code>slow_consumer_stats</code> breakdown tells you whether it is a client (contained) or a route/gateway (cluster-wide).

Leaving ulimit -n at the default

The OS file-descriptor limit is a separate wall from <code>max_connections</code>, and the default of 1024 is catastrophically low. Clients, routes, gateways, leaf nodes, and JetStream storage files all consume descriptors, so the server silently fails to accept connections or open storage files long before <code>max_connections</code> is reached. Production NATS should set <code>ulimit -n</code> to at least 65536 and test under load.

Watching total JetStream storage, not per-stream max_bytes

One runaway stream can hit its own <code>max_bytes</code> and start rejecting publishes while the server has plenty of free storage. Teams that only chart total usage miss it entirely — and <code>max_bytes: -1</code> (unbounded) lets a single stream grow until it trips the account or server limit and takes others down. Monitor each stream against its configured limit.

Ignoring Raft dynamics until JetStream is already down

Teams monitor 'are streams responding?' but not the Raft consensus underneath. Leader-election frequency, replica lag, and leader distribution give minutes to hours of warning, and WAL fsync latency on network-attached storage (EBS/NFS) is the number-one cause of election storms. Most teams only discover Raft problems when JetStream has already become unavailable.

Using absolute thresholds instead of ratios

'Alert when connections > 1000' breaks across deployment sizes; 'alert when connections > 85% of <code>max_connections</code>' does not. The same applies to storage (versus the configured limit), memory (versus the container limit), and ack-pending (versus <code>MaxAckPending</code>). Ratios travel between environments; absolute numbers do not.

Assuming core NATS delivery is reliable

Core NATS is fire-and-forget by design: publish to a subject with no subscriber, or to a slow one, and the message is gone. A subtler trap is a JetStream stream with <code>interest</code> retention and no consumers — it deletes every message immediately and is effectively <code>/dev/null</code> while the operator believes data is being stored. If you need reliability, use JetStream and verify the retention policy and consumers.

NATS runbooks in this section

Each guide is a focused runbook for one symptom or topic. Pick one when you have an incident, or use the categories to learn the area.

▸

Start here

▸

Server health and restarts

▸

Slow consumers and backpressure

▸

Connections and file descriptors

▸

Core NATS delivery and routing

▸

JetStream availability

▸

JetStream storage and disk I/O

▸

JetStream consumers and acks

▸

JetStream clustering and Raft

▸

Routes, gateways, and leaf nodes

▸

Go runtime resources

▸

Authentication, permissions, and TLS

WHERE TO GO NEXT

Setting up NATS monitoring, or putting out a fire?

If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, jump straight to the symptom that matches what you're seeing.

> Start with the checklist > Back to Operations Guides