NATS Operations Guides on Netdata

How NATS actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

Most NATS incidents come from a mismatch between how operators think NATS works and how it actually works. Teams coming from Kafka or RabbitMQ carry a queue-centric mental model: messages go somewhere, wait there, and can be retrieved later. Core NATS does not work like that. The mismatch shows up at 3 a.m. as silent message loss, slow consumer cascades, or a Raft election storm nobody saw coming.

This article builds the mental model you need before touching any NATS runbook: what the server is doing internally, where it competes for resources, and which behaviors are by design rather than symptoms.

NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check

Mon, 01 Jan 0001 00:00:00 +0000

A NATS server restarts with a large JetStream store. The process is fine, clients will be served shortly, but your pager fires and Kubernetes kills the pod before recovery finishes. The server starts recovering again, the probe fails again, and you are in a restart loop that looks like an outage but is a health check misconfiguration.

The cause is almost always the same: a liveness probe or a PAGE alert pointed at the bare /healthz endpoint on a JetStream-enabled server. On such a server, bare /healthz does not answer “is the process alive?” It runs the full JetStream health suite, and that suite legitimately fails for minutes while the server replays and recovers its assets after a restart.

NATS Authentication Timeout: clients that connect but never finish the handshake

Mon, 01 Jan 0001 00:00:00 +0000

Your NATS server log shows lines like this, and clients are complaining they cannot connect:

[ERR] 10.2.3.14:51824 - cid:1042 - Authentication Timeout

The confusing part: the TCP connection succeeded. The client reached the server. But the authentication handshake never finished, so the server sent -ERR 'Authentication Timeout' and closed the connection. This is not a bad-credentials problem. It is a timing problem: credentials (or the TLS handshake that must precede them) never arrived within the server’s auth window. For the related case where credentials arrive but are rejected, the log string is Authorization Violation and the diagnosis is different.

NATS Authorization Violation: authentication failures and credential rotation

Mon, 01 Jan 0001 00:00:00 +0000

Your NATS server log is filling with Authorization Violation entries, clients are failing to connect, and the error message tells you almost nothing. That is deliberate: NATS keeps auth error messages vague so they do not leak information to attackers. The side effect is that the same log line covers a typo’d password, an expired user JWT, a bad credential file deployed to a fleet, and someone port-scanning your cluster from the internet.

NATS connection churn: a stable connection count hiding constant reconnects

Mon, 01 Jan 0001 00:00:00 +0000

Your NATS dashboard shows 4,000 client connections. It showed 4,000 an hour ago, and 4,000 yesterday. Everything looks stable. Meanwhile, clients are connecting and disconnecting hundreds of times per minute. Every reconnect burns CPU on protocol handshakes (and TLS handshakes, if enabled), the server logs fill with connect and disconnect events, and your auth system processes a constant stream of authentication attempts.

This is connection churn, and it is one of the most commonly missed NATS failure modes because the metric everyone charts, the current connections gauge, is designed to hide it. A client that disconnects and reconnects within one scrape interval leaves the gauge unchanged. The churn is real; your chart just cannot see it.

NATS connection storm: reconnect thundering herd after a network event

Mon, 01 Jan 0001 00:00:00 +0000

A network device fails and recovers. A load balancer health check flaps. A rolling restart drops a node. For a few seconds, every client attached to a NATS server loses its connection. Then the network heals, and every one of those clients tries to reconnect in the same second.

Each reconnect is not cheap: TCP handshake, optional TLS handshake, protocol negotiation, authentication, and a full resubscribe of every subscription the client held. Multiply that by hundreds or thousands of clients arriving simultaneously and you have a connection storm: a sharp spike in connections and total_connections, a CPU spike dominated by TLS handshakes if encryption is enabled, memory climbing as per-connection state is allocated, and in the worst case the server hitting max_connections or the OS file descriptor limit and rejecting clients. Rejected clients retry. Now you have a reject-and-retry loop on top of the storm.

NATS consumer stalled at MaxAckPending: delivery stops until messages are acked

Mon, 01 Jan 0001 00:00:00 +0000

Your JetStream consumer was processing messages fine, and then deliveries just stopped. No error on the client. No error in the server log. The stream keeps growing, CPU and memory are normal, and the consumer application is still connected. Everything is green except the one thing that matters: no messages are moving.

This is the most common cause of “JetStream consumer stopped receiving messages,” and it is silent by design. When a consumer’s count of delivered-but-unacknowledged messages (num_ack_pending) reaches its configured MaxAckPending limit, the server stops delivering new messages to that consumer. No error is raised, no advisory is emitted. Delivery pauses until pending messages are acknowledged, negatively acknowledged, or expire past AckWait.

NATS context deadline exceeded: JetStream publish and request timeouts

Mon, 01 Jan 0001 00:00:00 +0000

Your application logs fill up with nats: context deadline exceeded. It shows up on js.Publish(), on js.StreamInfo(), on consumer fetches, sometimes on nats CLI commands like nats stream view. The NATS server is running, /healthz returns ok, and core NATS pub/sub traffic is flowing fine. Yet every JetStream operation hangs until the client’s context fires.

This error is a client-side deadline expiring. The client sent a JetStream API request and no response came back before the deadline. The request is not being rejected; it is not being answered at all. Somewhere between the client and the JetStream subsystem, the request is stuck, and the server is usually still healthy enough to look innocent.

NATS crash loop: unexpected uptime resets and repeated restarts

Mon, 01 Jan 0001 00:00:00 +0000

Your NATS server’s /varz uptime keeps resetting. The server was up for 4 minutes, then 2 minutes, then 6. Clients reconnect repeatedly, JetStream streams flap between unavailable and recovering, and every restart replays the WAL from scratch. This is a crash loop, and the fix depends entirely on why the process is dying.

The uptime field on /varz is the fastest confirmation. It reports time since process start as a NATS-specific duration string with y/d/h/m/s suffixes (for example “1d2h3m4s”), not a standard Go duration. When that value drops between scrapes, the process restarted. A single restart may be maintenance. More than 3 restarts in 30 minutes is a crash loop and needs an owner.

NATS file descriptor exhaustion: too many open files and the ulimit cliff

Mon, 01 Jan 0001 00:00:00 +0000

Clients suddenly cannot connect. The server process is alive, CPU is fine, memory is fine, but every new connection fails and the logs are full of too many open files. Existing clients keep working, which makes it look like a network problem until you count sockets.

This is file descriptor exhaustion, and it is a cliff-edge failure. There is no graceful degradation: the moment the process hits its OS ulimit -n, accept() starts failing, JetStream cannot open new storage files, and cluster routes cannot establish. It is one of the most common NATS incidents, because the default ulimit -n of 1024 on Linux is far too low for a production message broker, while the default max_connections in NATS is 65536. The OS limit is the real ceiling, and it is usually the one nobody set.

NATS gateway disconnected: cross-cluster traffic cut in a supercluster

Mon, 01 Jan 0001 00:00:00 +0000

Subscribers in cluster A have stopped receiving messages published in cluster B. Publishers see no errors. Every server’s /healthz returns ok, CPU and memory look normal, and yet a whole class of traffic has silently stopped flowing between two sites. In a NATS supercluster, this is the signature of a gateway disconnection: the outbound gateway connection to the remote cluster is missing, so no cross-cluster forwarding happens at all.

Gateways are the links between independent NATS clusters. Unlike cluster routes (full mesh between servers in one cluster), each server maintains gateway connections keyed by remote cluster name, and all cross-cluster delivery for an account flows through them. When one goes down, the failure is clean and quiet: local delivery keeps working, remote delivery stops, and nothing in the server-wide error counters necessarily moves.

NATS goroutine leak: connection cleanup that never completes

Mon, 01 Jan 0001 00:00:00 +0000

The symptom is a steady, one-directional climb. Goroutine count goes up day after day, RSS follows it, and nothing on the traffic side explains it: client connections are flat, routes are flat, throughput is flat. Then, weeks later, the server gets OOM-killed or starts showing GC and scheduling overhead that has nothing to do with message load.

A NATS goroutine leak is connection (or subsystem) cleanup that never completes. A goroutine is spawned to handle a connection, a timer, or a Raft loop; the work ends, but the goroutine never exits. It sits blocked on a channel receive that will never fire, a lock that will never be released, or a retry loop with nothing to retry. Each one is cheap individually, so the leak is invisible until it is not: at roughly 4-8KB of stack per goroutine, 100k leaked goroutines is about 400-800MB of RSS.

NATS high CPU: subject matching, TLS, GC, and the container GOMAXPROCS trap

Mon, 01 Jan 0001 00:00:00 +0000

A NATS server that is pegged on CPU usually looks worse than it is, or better than it is, depending on what you are measuring. The cpu field in /varz is process CPU where 100 means one full core, so on a 16-core host a value of 800 is only 50% busy. Operators regularly page themselves on a number that is not actually saturation, or dismiss real saturation because the number “only” reads 300 on a 2-core container.

NATS insufficient storage / maximum bytes exceeded: JetStream publishes rejected

Mon, 01 Jan 0001 00:00:00 +0000

Your producers start throwing JetStream publish errors: “insufficient storage”, “maximum bytes exceeded”, or “maximum messages exceeded”. The server itself looks healthy: /healthz returns ok, connections are stable, CPU and memory are normal. Writes to one or more streams are failing anyway.

This is a storage-limit failure, not a server failure. JetStream enforces limits at three levels: per-stream (max_bytes, max_msgs, max_age), per-account storage quotas, and the server-wide JetStream storage reservation. When any of them is hit, the discard policy decides the failure mode. With DiscardNew, new publishes are rejected and the publisher sees an error. With DiscardOld, the server silently evicts the oldest messages to make room, which is data loss for any consumer that has not caught up.

NATS JetStream AckWait tuning: matching the ack timeout to processing time

Mon, 01 Jan 0001 00:00:00 +0000

AckWait is the timer the JetStream server starts when it delivers a message to a consumer. If the client does not ack before the timer expires, the server redelivers the message. That is the entire mechanism, and most “JetStream consumer is stuck” incidents that are not crashes come down to this timer being mismatched to the actual processing time of the work behind it.

Too short, and messages redeliver while they are still being processed: num_redelivered climbs, num_ack_pending stays pinned near the ceiling, and the consumer makes net-zero progress. Too long, and a genuinely dead worker holds a slot in MaxAckPending for minutes while nothing flows. This article covers how to size AckWait against p99 processing time, how it interacts with MaxAckPending and MaxDeliver, and when to stop stretching the timeout and use in-progress acks instead.

NATS JetStream API errors: reading the /jsz api.errors counter without false alarms

Mon, 01 Jan 0001 00:00:00 +0000

You opened /jsz and saw api.errors in the thousands, or an alert fired because the counter moved. The first question is not “what broke” but “is this counter telling me something real.” The JetStream API error counter is cumulative, coarse-grained, and incremented by perfectly healthy client behavior. Alerting on its absolute value or on any movement at all is a guaranteed false-alarm generator.

This article covers reading api.errors correctly: computing a rate, using the error ratio, correlating with api.inflight, and separating idempotent-create noise from genuine failures like storage exhaustion, Raft proposal failures, and permission denials. For the broader JetStream signal model, see How NATS actually works in production.

NATS JetStream consumer lag growing: falling behind the stream

Mon, 01 Jan 0001 00:00:00 +0000

The NATS server is healthy. /healthz returns ok, throughput looks normal, CPU and memory are fine. But one stream keeps growing, and the consumer attached to it is not keeping up. num_pending climbs minute after minute, and every dashboard that only watches server-level metrics shows green.

Consumer lag requires per-consumer polling, not a single server endpoint, so most monitoring setups never see it. The result: a stream with millions of pending messages, effectively down while the server looks healthy.

NATS JetStream consumer stopped receiving messages: the diagnostic tree

Mon, 01 Jan 0001 00:00:00 +0000

A JetStream consumer that stops receiving messages almost never raises an error. The stream keeps accepting publishes, the health endpoint stays green, and your application simply goes quiet. The signals that explain this live per-consumer, not per-server, which is why server-level dashboards show nothing.

There are only a handful of mechanisms that produce this symptom, and each has a distinct fingerprint in consumer state. This article is the diagnostic tree. Start at the top with nats consumer info (or /jsz?consumers=true), read five counters, and the tree tells you which failure you are in and where to go next.

NATS JetStream disabled unexpectedly: the persistence subsystem failed to come up

Mon, 01 Jan 0001 00:00:00 +0000

The server is up. Clients connect, core NATS routes messages, /healthz?js-server-only=true returns ok. But /jsz reports disabled: true on a node where JetStream should be running, and every stream, consumer, and KV bucket backed by this node is gone or degraded. Publishers that rely on persistence fail; request-reply still works, which is why this gets noticed late.

This state means one of two things: JetStream failed to initialize at startup, or it initialized and was later shut down by the server itself. In both cases the root cause is almost always below the NATS layer: the storage directory, the disk underneath it, or the configuration. The server rarely recovers on its own, and a blind restart can make a corrupt store worse.

NATS JetStream disk I/O stall: the disk has space but is too slow

Mon, 01 Jan 0001 00:00:00 +0000

JetStream publishes are timing out or creeping upward in latency. df -h shows plenty of free space. The /jsz endpoint shows storage nowhere near its reserved limits, yet api.inflight sits high and api.errors keeps climbing. On the host, iowait is elevated and disk latency looks bad.

This is the JetStream disk I/O stall pattern: the disk has space but is too slow. It is a different failure from storage exhaustion, and it is frequently misdiagnosed because the obvious capacity checks all pass.

NATS JetStream meta cluster leader flapping: admin operations that keep failing

Mon, 01 Jan 0001 00:00:00 +0000

Stream and consumer administration is failing, but existing JetStream traffic mostly continues. Creating a stream times out, consumer updates return errors, and retries sometimes succeed only to fail again a minute later.

This pattern points at the JetStream meta Raft group. The meta group manages cluster-wide JetStream metadata: stream and consumer create, update, and delete operations. Existing streams use separate Raft groups for replicated data, so the data plane can remain available while the administrative plane is unstable.

NATS JetStream mirror and source lag: stale replicas and DR recovery point

Mon, 01 Jan 0001 00:00:00 +0000

A JetStream stream configured as a mirror of another stream, or sourcing from one or more upstream streams, keeps a local copy that is only as fresh as its sync connection. Stream info reports two fields per mirror or source: lag, the number of messages the local copy is behind, and active, the time since the last sync activity. When those numbers move the wrong way, the copy is going stale.

NATS JetStream no leader / cluster not currently available: writes blocked

Mon, 01 Jan 0001 00:00:00 +0000

Your publishers are failing with nats: no leader for stream or JetStream cluster not currently available, or every JetStream API call times out with no responders on $JS.API. Stream info shows an empty cluster.leader. Writes stay paused until the affected Raft group elects a leader again.

This is a quorum problem, not a load problem. JetStream uses Raft for consensus: one meta group manages all JetStream metadata and admin operations, and each replicated stream (and consumer) has its own Raft group. A group without a leader blocks everything that depends on it. If the meta group is leaderless, all JetStream admin operations fail cluster-wide. If only a stream’s group is leaderless, that stream stops accepting messages while the rest of JetStream keeps working.

NATS JetStream not enabled for account: persistence calls failing on a core server

Mon, 01 Jan 0001 00:00:00 +0000

Your application connects to NATS successfully, then fails on the first persistence call: stream creation, consumer creation, or a publish that waits for an ack. The client error is one of two strings: nats: jetstream not enabled for account (err_code 10039) or nats: jetstream not enabled (err_code 10076). Core pub/sub keeps working the whole time, which is what makes this confusing during an incident.

The server is healthy, clients are connected, and messages route fine. What you have is a configuration mismatch: the client is issuing JetStream operations against a server where JetStream was never enabled, or against an account that was never granted JetStream resources. Nothing will self-heal; the fix is always a configuration change.

NATS JetStream Raft election storm: leaders flapping and writes pausing repeatedly

Mon, 01 Jan 0001 00:00:00 +0000

Your JetStream cluster is electing leaders over and over. Logs alternate rapidly between Stepping down and JetStream cluster new leader. Publishes intermittently time out, stream and consumer management calls error out, and some streams briefly report no leader at all. Then it settles for a few minutes and starts again.

This is a Raft election storm. It is not a clean failover and it is not usually a software bug. It is a symptom of resource starvation or latency somewhere in the cluster, and it feeds itself: elections consume CPU and I/O, which delays heartbeats further, which triggers more elections.

NATS JetStream redelivery loop: num_redelivered climbing and messages reprocessed

Mon, 01 Jan 0001 00:00:00 +0000

A JetStream consumer’s num_redelivered counter is climbing and your application is processing the same messages over and over. Nothing is lost, but nothing completes either: deliveries happen, acks do not, and the server keeps trying again.

This is a redelivery loop. The server delivers a message, the consumer fails to acknowledge it within AckWait, the server redelivers it, and each pass increments num_redelivered. Every redelivery is wasted work: duplicate side effects if your handler is not idempotent, duplicate load on downstream systems, and a num_ack_pending count that never drains.

NATS JetStream replica lag: a non-current replica that would lose data on failover

Mon, 01 Jan 0001 00:00:00 +0000

A replicated JetStream stream shows current: false with a non-zero lag for one of its replicas in nats stream info. Everything still works: publishes succeed, consumers receive messages, no alerts fire on server health. But the replication factor you configured is not the replication factor you have. If the leader fails right now, a failover to the lagging replica loses every message the leader acknowledged but the replica has not yet written.

NATS JetStream retention: limits vs interest vs workqueue and the /dev/null stream

Mon, 01 Jan 0001 00:00:00 +0000

JetStream’s retention policy decides when a stored message is deleted, and the failure modes are asymmetric: get it wrong in one direction and storage grows until publishes are rejected; get it wrong in the other and every message you publish is deleted on arrival while the server reports itself perfectly healthy.

This article covers the three policies (limits, interest, workqueue), the exact conditions under which each one deletes a message, the silent /dev/null failure mode, and the checks that confirm your streams are retaining what you think they are. It assumes a working mental model of streams, consumers, and acks. If not, start with how NATS actually works in production.

NATS JetStream storage exhaustion spiral: stalled consumers that starve retention

Mon, 01 Jan 0001 00:00:00 +0000

JetStream storage is climbing steadily. Not a spike, a slope. Then publishes start failing with API errors, producers back up, and the pipeline degrades. The server looks healthy: connections stable, CPU and memory fine. The disk is filling anyway.

This is the JetStream storage exhaustion spiral: a deadlock where one or more consumers have stalled (usually pinned at MaxAckPending) on a stream using interest or workqueue retention. The server cannot delete a message until every interested consumer has acknowledged it, so a single dead consumer makes its entire backlog undeletable. Messages accumulate, storage approaches the configured limit, and new publishes are rejected. The system is stuck: consumers must process messages to free space, but the consumers are the broken part.

NATS leaf node disconnected: an edge server isolated from the hub

Mon, 01 Jan 0001 00:00:00 +0000

A leaf node connection is the single TCP session that ties an edge NATS server to your hub cluster. When it drops, the edge site keeps running locally, but it is cut off from the rest of the messaging fabric. Subscribers on the edge stop receiving messages published at the hub, and subscribers at the hub stop receiving anything published at the edge.

The blast radius is easy to underestimate. A leaf connection multiplexes traffic for potentially many accounts over one logical connection. One dropped leaf can silently sever dozens of application-level communication paths while every local health check on both servers still returns ok.

NATS Maximum Connections Exceeded: new clients rejected at the max_connections wall

Mon, 01 Jan 0001 00:00:00 +0000

New clients cannot connect to your NATS server. Existing connections keep working. Client logs show the protocol error -ERR 'Maximum Connections Exceeded' and then the connection closes. The server itself looks healthy: /healthz returns ok, messages still flow for connected clients, and nothing crashed.

This is the max_connections wall. The server has reached its configured connection limit and is rejecting every new connection during the handshake. There is no queueing and no graceful degradation. One slot short of the limit, everything works. At the limit, every new client is turned away.

NATS Maximum Payload Violation: messages rejected for exceeding max_payload

Mon, 01 Jan 0001 00:00:00 +0000

A publisher calls publish, the call fails, and the NATS server log shows a Maximum Payload Violation. Seconds later the same client reconnects, publishes again, and gets disconnected again. The server is not broken: it is enforcing the configured max_payload limit, which defaults to 1 MB. The application, meanwhile, is in a publish-disconnect-reconnect loop and its messages are not flowing.

The protocol behavior is strict. When a client sends a message whose payload exceeds max_payload, the server responds with -ERR 'Maximum Payload Violation' and closes the connection. Client libraries that auto-reconnect come straight back and repeat the offense, which is why a single oversized publish looks like connection churn rather than a clean, one-time error.

NATS memory growth and OOM: reading RSS past the Go GC sawtooth

Mon, 01 Jan 0001 00:00:00 +0000

The nats-server process is climbing toward its container memory limit. Grafana shows a jagged line that spikes to nearly double the baseline and drops back, except lately the drops are getting shallower and the floor keeps rising. Then the pod restarts, uptime resets to a few minutes, and the cycle repeats. If JetStream is enabled, the numbers look even worse, and half of what you see is not memory the server is actually holding.

NATS messages published but not received: subject mismatches and cross-server gaps

Mon, 01 Jan 0001 00:00:00 +0000

The publisher’s Publish() call returns no error. The server is healthy. The subscriber receives nothing. This is almost never a server bug. It is how core NATS works: fire-and-forget routing against an in-memory subject tree, with no persistence, no delivery guarantee, and no signal when a message matches zero subscriptions.

In core NATS, a message published to a subject with zero matching subscribers is silently discarded. No error, no log line, no metric. The only server-level evidence is an asymmetry between in_msgs and out_msgs, which most teams never graph.

NATS monitoring checklist: the signals every production server needs

Mon, 01 Jan 0001 00:00:00 +0000

A NATS server can pass every health check and still be losing messages. Core NATS drops messages to slow or absent subscribers by design, JetStream consumers can stall while every server-level metric looks green, and a node can be partitioned from its cluster while its local health endpoint returns 200. “Is the process running?” is the start of a monitoring strategy, not the end of one.

This checklist organizes the signals into four maturity levels: survival, operational, mature, and expert. Each lists the HTTP endpoint and field, what healthy looks like, and what to alert on. Start at Level 1 and add levels as criticality grows. A core NATS standalone server needs less of this list than a clustered JetStream deployment; the notes per signal say when that is the case.

NATS monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Most NATS outages are not caused by a lack of metrics. The server exposes a rich monitoring API on port 8222. The failures happen because teams collect the wrong tier of signals for the failures they actually experience. A /healthz check tells you the process is alive. It tells you nothing about a consumer stalled at MaxAckPending, a route connection backing up with pending bytes, or a Raft meta cluster electing a new leader every ninety seconds.

NATS no responders available for request: request-reply into the void

Mon, 01 Jan 0001 00:00:00 +0000

Your service just started throwing nats: no responders available for request. The client got an answer back from the server almost instantly, and the answer was: nobody is listening on that subject. This is the request-reply counterpart to NATS’s silent message loss: instead of the request vanishing and the client waiting out its timeout, the server short-circuits the call and fails fast with a 503 status in the reply headers.

NATS pending bytes growing: catching a slow consumer before it is disconnected

Mon, 01 Jan 0001 00:00:00 +0000

By the time slow_consumers in /varz increments, the damage is done. The server has declared a connection slow, and in core NATS the default response is to disconnect it. Messages buffered for that client are dropped, not queued, not retried. The client library auto-reconnects, resubscribes, immediately falls behind on the same backlog, and you are in a churn loop.

The precursor signal is sitting in the monitoring endpoints the whole time. Every connection has a write-side pending buffer you can read before the server gives up on it: pending_bytes per client connection via /connz, and pending_size per route via /routez. Watching these lets you name the specific culprit, a client, a route, or a gateway, while there is still time to act.

NATS Permissions Violation: authorized clients publishing or subscribing out of scope

Mon, 01 Jan 0001 00:00:00 +0000

A Permissions Violation entry in the NATS server log means an authenticated client tried to publish or subscribe to a subject outside its permitted scope. The server rejected the operation, logged the event with the subject, client IP, and account, and sent an -ERR back to the client. The connection stays open. This is not an authentication failure: the credentials were accepted, but the permissions attached to them do not cover what the client tried to do.

NATS route disconnected: a missing cluster route means a partition

Mon, 01 Jan 0001 00:00:00 +0000

One of your NATS servers shows fewer routes than it should. In a full-mesh cluster of N servers, every server should hold N-1 route connections, one to each peer. When that count drops, you do not have a degraded link. You have a partition.

The impact is asymmetric and easy to underestimate. The isolated server still accepts client connections, still reports healthy on /healthz, and still routes messages locally. But subscribers connected to it stop receiving messages published on the other side of the partition, and publishers on it vanish from the rest of the cluster’s view. If you run JetStream with replicated streams, Raft groups whose members span the partition can lose quorum, which turns a messaging gap into write failures.

NATS route RTT high: inter-server latency that triggers Raft elections

Mon, 01 Jan 0001 00:00:00 +0000

You are looking at a JetStream cluster that keeps electing new leaders. The /jsz meta cluster leader changes every few minutes, stream writes intermittently fail with API errors, and nats server list or /routez shows round-trip times between servers far above what a same-datacenter cluster should produce. Server CPU and memory look fine. Nothing has crashed. Yet the cluster cannot hold a stable leader.

This is the classic signature of elevated route RTT. Cluster routes are the TCP connections over which NATS servers exchange messages, subscription interest, and JetStream Raft traffic. When latency on those connections rises, two things degrade at once: cross-server message delivery slows, and Raft heartbeats start arriving late. Late heartbeats tip followers into starting elections. Sustained high route RTT is one of the leading causes of election storms, and it is routinely missed because teams monitor route existence (“are routes connected?”) but not route quality.

NATS route slow consumer: inter-server forwarding backing up cluster-wide

Mon, 01 Jan 0001 00:00:00 +0000

Your NATS server’s slow_consumer_stats.routes counter just went non-zero, or /routez shows a pending_size that keeps climbing on one route. This is not the same problem as a slow client. A route is the TCP connection that carries inter-server traffic between two NATS servers in a cluster. When it backs up, every subscriber reachable through that peer falls behind or stops receiving messages entirely, and in a JetStream cluster the degradation can extend to Raft heartbeat timing and leader stability.

NATS server not responding: healthz failing and the process down or hung

Mon, 01 Jan 0001 00:00:00 +0000

Your probe against the NATS monitoring port is failing: /healthz returns non-200, or the port does not answer at all. Clients may still be connected, reconnecting in a storm, or already failing over to other cluster nodes.

Do not start with a restart. A crashed process, a hung event loop, lame-duck shutdown, resource exhaustion, and JetStream recovery can all look like “NATS is down,” and the first safe action is different for each. Restarting too early destroys evidence and can turn a recoverable hang into data loss.

NATS silent message loss: zero-subscriber drops and the in/out asymmetry

Mon, 01 Jan 0001 00:00:00 +0000

Your NATS server is healthy. Health checks pass, connection counts are stable, there are no slow consumers, no errors in the logs. And yet a downstream service insists it has not received a single message in the last hour, while the producer’s metrics show it published thousands.

Both are telling the truth. In core NATS, a message published to a subject with zero subscribers is discarded at the moment of publish. The publisher gets no error. The server writes no log line. No counter records the drop. The message simply does not exist anymore.

NATS slow consumer breakdown: clients vs routes vs gateways and blast radius

Mon, 01 Jan 0001 00:00:00 +0000

Your NATS alert fired: slow_consumers is incrementing. You open /varz, see a non-zero counter, and start hunting for a misbehaving client. That is the right move only some of the time. The aggregate slow_consumers counter mixes four very different kinds of events, and two of them mean your cluster fabric itself is backing up, not a single subscriber.

A client slow consumer costs you one subscriber’s message stream. A route or gateway slow consumer means inter-server forwarding is stalling, which can degrade message delivery for every account and every client on the cluster. Teams that monitor only the aggregate number page on the harmless case and sit on the severe one.

NATS slow consumer detected: the write buffer overflowed and messages were dropped

Mon, 01 Jan 0001 00:00:00 +0000

Your NATS server log shows Slow Consumer Detected, or your client logged nats: slow consumer, messages dropped. In core NATS (no JetStream), that line means messages were dropped. Not queued, not retried, not parked somewhere for later.

The server buffers outbound messages per connection, and when a subscriber cannot drain its write buffer fast enough, the server sheds load by dropping messages for that subscriber or disconnecting it outright. Delivery in core NATS is at-most-once, and the slow subscriber is not told what was lost.

NATS stalled clients and stale connections: half-dead sockets and write-path distress

Mon, 01 Jan 0001 00:00:00 +0000

Two counters in /varz tell you about connections that are unhealthy but not yet dead: stalled_clients and stale_connections. Neither one means a connection has been dropped. That is the point: both describe states that precede visible failure. Stalled clients are on the way to becoming slow consumers; stale connections are sockets that look open but whose peer has stopped responding.

The operational risk is twofold. Stalled clients signal write-path backpressure: the server cannot flush data to a client fast enough, which is the precursor to slow consumer disconnection and, in core NATS, silent message loss. Stale connections are quieter: they consume file descriptors and per-connection memory while doing no useful work, and they usually indicate a network half-partition or a hung client process that will not recover on its own.

NATS stream max_bytes: one stream hitting its limit while the server has room

Mon, 01 Jan 0001 00:00:00 +0000

Publishers to one JetStream stream start getting rejected with a “maximum bytes exceeded” error. You check the server: JetStream storage is at 40% of reserved_storage, the disk has hundreds of gigabytes free, and every server-level dashboard is green. Nothing about the server looks full.

What happened: max_bytes is a per-stream limit, independent of server-level and account-level JetStream quotas. One stream reached its own configured ceiling and started enforcing its discard policy while every other stream kept working. If your monitoring only watches aggregate JetStream storage (/jsz without parameters), this failure is invisible until publishers start erroring.

NATS subscription leak: unbounded subscription growth and subject-trie bloat

Mon, 01 Jan 0001 00:00:00 +0000

The symptom is a number that only goes up. You open /varz on a NATS server and subscriptions is higher than it was yesterday, higher than it was last week, climbing for days while connections stays flat. Memory follows the same slope. Nothing is erroring. No slow consumers, no restarts, no complaints from applications. The server just keeps getting heavier.

That is a subscription leak: an application (or a bug) is registering subscriptions faster than it removes them. Every subscription lives in the server’s subject trie and in per-connection tracking state. Left alone, the leak consumes memory, slows subject matching, and in a cluster propagates interest across routes so that one leaky client inflates the subject trie on every server.

NATS TLS certificate expiry: an expired cert that locks out every new connection

Mon, 01 Jan 0001 00:00:00 +0000

Every new client connection to your NATS server is failing at the TLS handshake with a verification error, typically x509: certificate has expired or is not yet valid. Clients that were already connected still work. That is what makes this failure mode deceptive: the server passes every process-level check while refusing all new work.

If you run mutual TLS on cluster routes and gateways, the blast radius is bigger. Route and gateway connections also fail to establish, so servers that restart or reconnect after a network event cannot rejoin the cluster. A certificate that expires during an unrelated incident can turn a recoverable blip into a full partition.

NATS write_deadline and buffer sizing: tuning how long the server waits on a slow writer

Mon, 01 Jan 0001 00:00:00 +0000

A client gets disconnected with Slow Consumer Detected in the server log. Someone raises write_deadline, the disconnects stop, and the ticket gets closed. Two weeks later the same client is back in the log, and now the server is also showing memory growth because the larger buffer window lets more messages pile up per connection. This is the standard lifecycle of a write_deadline tuning mistake.

write_deadline controls how long the NATS server will block on flushing data to a connection before it declares that connection slow. The per-connection pending buffer controls how much data can accumulate while the server waits. Together they define the backpressure contract between the server and every client, route, gateway, and leafnode connection. Set them too aggressively and healthy-but-bursty clients get disconnected. Set them too loosely and pending buffers pin memory while genuinely broken consumers go undetected longer.