How NATS actually works in production: a mental model for operators

Most NATS incidents come from a mismatch between how operators think NATS works and how it actually works. Teams coming from Kafka or RabbitMQ carry a queue-centric mental model: messages go somewhere, wait there, and can be retrieved later. Core NATS does not work like that. The mismatch shows up at 3 a.m. as silent message loss, slow consumer cascades, or a Raft election storm nobody saw coming.

This article builds the mental model you need before touching any NATS runbook: what the server is doing internally, where it competes for resources, and which behaviors are by design rather than symptoms.

What NATS is and why the model matters

NATS is a subject-based message router written in Go. That sentence carries more operational weight than it appears to. It is not a broker that stores messages. It is not a queue that holds work. It receives messages on subjects, matches them against registered subscriptions, and writes them to the interested connections, in real time.

The consequence that bites teams hardest: in core NATS, if no subscriber is connected when a message is published, the message is silently discarded. No error, no log, no metric. This is fire-and-forget by design. If your architecture assumes at-least-once delivery, that guarantee has to come from JetStream, not from core NATS.

This one fact re-frames most “NATS is losing messages” reports: the server is usually doing exactly what it was designed to do.

How the routing engine works

The core of the server is an in-memory subject tree, a trie that maps subject strings to sets of subscriptions. Subjects are dot-delimited tokens (for example orders.us.east.created), and the trie supports two wildcards: * matches a single token, > matches one or more trailing tokens. Every inbound message is matched against this trie and fanned out directly to all matching subscribers. There is no intermediate queue between publisher and subscriber.

flowchart LR
    P[Publisher] -->|publish on subject| T[Subject trie]
    T -->|match and fan out| S1[Subscriber A write buffer]
    T -->|match and fan out| S2[Subscriber B write buffer]
    T -->|route interest exists| R[Route to peer server]
    T -->|no match| X[Message dropped silently]
    R --> T2[Peer subject trie]
    T2 --> S3[Subscriber C write buffer]

Two properties of this design matter operationally:

• Fan-out is amplification. One inbound message becomes N outbound writes. A high out_msgs / in_msgs ratio is normal for fan-out workloads, but it also means any latency in the delivery path is multiplied across subscribers.
• The trie is shared state. Subscriptions consume memory in the trie and in per-connection tracking. A subscription leak in one client application inflates the routing table, and in a cluster the leak propagates across routes to every server.

Per-connection machinery: where slow consumers come from

Every TCP connection to the server, whether an application client, a cluster route, a gateway, or a leaf node, gets dedicated read and write goroutines plus a per-connection pending write buffer. The write side is governed by write_deadline, which controls how long the server waits for a connection to drain before giving up.

When a connection cannot consume as fast as the server produces, the pending buffer grows. When it exceeds the threshold, the server marks the connection as a slow consumer and, by default, disconnects it. Two things about this mechanism are routinely misunderstood:

1. A slow consumer event is the server correctly enforcing backpressure. The fault is on the slow connection: a subscriber doing synchronous I/O in its message handler, a GC pause, a congested network path. Investigating the server first wastes time.
2. It applies to every connection type, not just clients. A slow consumer on a route or gateway means inter-server message delivery is backing up, with cluster-wide blast radius.

The leading indicator is per-connection pending bytes, visible via /connz (clients) and /routez (routes, as pending_size). Pending bytes grow before the server declares distress, so watching the precursor lets you act before the disconnect-reconnect spiral begins. That spiral is the classic slow consumer cascade: subscriber falls behind, server disconnects it, client auto-reconnects and resubscribes, the backlog hits immediately, and the cycle repeats as connection churn.

Clustering: routes, gateways, and leaf nodes

NATS scales out through three distinct connection types, and each one is, internally, just another client with its own goroutines and buffers:

Clustering does not change the core semantics. Messages still route in real time, there is still no intermediate queue, and every hop has its own buffer that can back up. A route count below N-1 in a cluster means a partition, and in a 2-node cluster losing the single route is a complete partition.

JetStream: what gets bolted on

JetStream is a persistence layer on top of core NATS, not a replacement for it. Enabling it adds subsystems that compete for the same host resources:

• WAL-based storage. Each stream uses file or memory storage. File storage is a write-ahead log with block-level organization, plus an in-memory index mapping message sequence numbers to file block positions. JetStream disk I/O is sequential writes, historical reads, compaction, and snapshots.
• Consumer state tracking. Delivery cursors, acknowledgment state, and redelivery timers, per consumer. With explicit acks, unacknowledged messages accumulate up to MaxAckPending; at that limit, delivery stalls silently. The stream looks healthy while the consumer is effectively down.
• Raft consensus, at two levels. In clustered JetStream there is a meta Raft group for cluster-wide metadata, and each replicated stream (and its consumers) has its own Raft group. A per-stream group can fail independently while the meta group is healthy.
• Background work. Compaction, retention enforcement, snapshot/restore, and catch-up replication for lagging peers.

Operationally, JetStream converts fire-and-forget into a system where storage, consumer lag, and Raft stability are the primary failure surfaces. Retention interacts with consumer health in ways that surprise people: with interest retention, a message is deleted only after all consumers acknowledge it, so one stalled consumer blocks retention for the whole stream and storage fills. Conversely, a stream with interest retention and zero consumers deletes every message on arrival: effectively /dev/null while looking perfectly healthy.

Where resource pressure shows up

NATS competes for a small set of host resources, each with a characteristic degradation shape:

The interplay worth internalizing: slow consumers buffer in memory, memory pressure lengthens GC pauses, GC pauses delay Raft heartbeats, and delayed heartbeats cause elections that pause writes. A JetStream outage can have its root cause three steps away from where the alert fired.

The failure archetypes to recognize

Recognizing the shape early is most of the battle:

• Slow consumer cascade. Disconnect-reconnect churn on clients, routes, or gateways. Check the slow consumer breakdown by connection type first; the remediation for a slow route is completely different from a slow client.
• File descriptor exhaustion. Sudden inability to accept new connections while existing ones work fine. Check ulimit -n before anything else; it is a cliff-edge with no graceful degradation.
• JetStream Raft instability. Network latency, disk I/O stalls, or GC pauses cause election timeouts. Leader flapping means write failures and stalled consumers, and simultaneous elections across many stream groups can make the cluster effectively unavailable.
• Memory pressure. Slow consumer buffering, subscription trie growth, JetStream cache growth. Alert on the trend, not spikes.
• Silent message loss in core NATS. Publishers send to subjects with zero subscribers and messages vanish. The only signal is out_msgs running far below in_msgs after adjusting for expected fan-out.

Signals to watch in production

These signals map directly onto the machinery above. The monitoring port (default 8222) must be explicitly enabled with -m 8222 or http_port: 8222; if it is not, none of these endpoints exist.

One instrumentation caution: /subsz is expensive on servers with many subscriptions and can degrade the server itself. Track the subscription count from /varz, not the list from /subsz.

How Netdata helps

The mental model above tells you what to correlate; the value of monitoring is seeing those correlations on one timeline instead of assembling them from curl during an incident:

• Slow consumer events against connection churn and pending bytes, so you can tell a one-off client hiccup from an active cascade, and see backpressure building before disconnects start.
• in_msgs against out_msgs over time, which makes zero-subscriber message loss visible as a fan-out asymmetry instead of an application-side mystery.
• Route count and route health alongside throughput, so a cluster partition shows up immediately rather than as unexplained delivery gaps.
• Process RSS and CPU next to connection and subscription counts, which distinguishes expected growth from buffer accumulation or a subscription leak.
• JetStream aggregates (storage, API errors) against OS-level disk latency and iowait, the correlation that separates storage exhaustion from a disk I/O stall, two problems with very different fixes.

Related guides

• NATS monitoring checklist: the signals every production server needs
• NATS monitoring maturity model: from survival to expert
• NATS server not responding: healthz failing and the process down or hung
• NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check
• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS slow consumer detected: the write buffer overflowed and messages were dropped
• NATS slow consumer breakdown: clients vs routes vs gateways and blast radius
• NATS pending bytes growing: catching a slow consumer before it is disconnected
• NATS stalled clients and stale connections: half-dead sockets and write-path distress
• NATS write_deadline and buffer sizing: tuning how long the server waits on a slow writer
• NATS Maximum Connections Exceeded: new clients rejected at the max_connections wall
• NATS file descriptor exhaustion: too many open files and the ulimit cliff