NATS monitoring checklist: the signals every production server needs

A NATS server can pass every health check and still be losing messages. Core NATS drops messages to slow or absent subscribers by design, JetStream consumers can stall while every server-level metric looks green, and a node can be partitioned from its cluster while its local health endpoint returns 200. “Is the process running?” is the start of a monitoring strategy, not the end of one.

This checklist organizes the signals into four maturity levels: survival, operational, mature, and expert. Each lists the HTTP endpoint and field, what healthy looks like, and what to alert on. Start at Level 1 and add levels as criticality grows. A core NATS standalone server needs less of this list than a clustered JetStream deployment; the notes per signal say when that is the case.

How collection works

NATS has no native Prometheus endpoint. The server exposes JSON over an HTTP monitoring port, and something has to poll it: a monitoring agent, a Prometheus exporter such as prometheus-nats-exporter, or your own scraper. Everything here assumes a poller hitting these endpoints on a regular interval.

Three collection rules before you start:

• Enable the port explicitly. The monitoring endpoints are off by default. Enable them with -m 8222 or http_port: 8222 in the server config. The port binds to all interfaces by default and has no built-in authentication, so bind it to localhost or a private interface and protect it at the network layer.
• Poll every 10 to 15 seconds. The endpoints are point-in-time snapshots. Faster scraping rarely helps and the endpoints themselves add load. Nearly all counters are cumulative since server start, so compute rates and handle counter resets on restart.
• Never scrape /subsz on a schedule. Listing all subscriptions can lock a server with a large subject trie. Use the subscription count from /varz. The same caution applies, scaled down, to /connz?subs=1 and /jsz?consumers=true on servers with many consumers: they return much more data and cost more. Reserve them for diagnosis or slow-interval collection of critical assets only.

The four levels at a glance

flowchart TD
  L1["Level 1 - Survival: healthz, uptime, connections, memory"]
  L2["Level 2 - Operational: throughput, slow consumers, routes, JetStream status"]
  L3["Level 3 - Mature: storage, accounts, gateways, leaf nodes, TLS expiry"]
  L4["Level 4 - Expert: Raft health, per-consumer lag, pending bytes"]
  L1 --> L2 --> L3 --> L4

Level 1: survival

The bare minimum to know the server is alive and not about to fall over. Every production NATS deployment needs all four.

Notes on each:

• Server health. The js-server-only=true parameter matters. Bare /healthz on a JetStream-enabled server runs full JetStream health checks, including meta recovery and asset recovery, and can fail for minutes during recovery after a restart on a large store. Paging on bare /healthz produces false pages during exactly the moments you expect. Use js-server-only=true for the page, and keep bare /healthz as a ticket-level signal if you want deeper JetStream health. The uptime gate (>300s) kills cold-start false positives. On non-JetStream servers the two forms behave identically.
• Uptime. The value is a duration string, so parse it as such rather than treating it as a number. In a cluster, correlate uptime across nodes: simultaneous resets mean a cluster-wide event, not a local crash.
• Connections. max_connections defaults to 65536. Alert on the ratio, not an absolute number, so the check survives across deployment sizes. Separately, verify the OS file descriptor limit (ulimit -n) is at least max_connections plus headroom for routes, gateways, leaf nodes, and JetStream file handles. The default of 1024 on many systems will kill the server before max_connections is ever reached, and FD exhaustion is a cliff-edge failure with no graceful degradation.
• Memory. NATS is Go, so a sawtooth pattern from GC is normal. Alert on the trend, not spikes. If JetStream is enabled, remember it adds Raft state, message caches, and consumer tracking on top of per-connection buffers; do not size memory budgets from connection counts alone.

Level 2: operational

What a competent production team runs. This is where you catch the failures that actually take NATS down: slow consumers, silent message loss, and cluster partitions.

Notes on each:

• Throughput asymmetry is your only silent-loss detector. In core NATS, a message published to a subject with zero subscribers is silently dropped. No error, no log, no metric. The only signal is in_msgs climbing while out_msgs stays flat relative to your expected fan-out ratio. Set this check up; most teams never do.
• Slow consumers are the server working correctly. When a connection’s pending write buffer grows past what write_deadline (default 10s) allows, the server flags it and disconnects it. The fault is on the slow side of the connection, not the server. But check slow_consumer_stats, which breaks events down into clients, routes, gateways, and leafs. A route or gateway slow consumer means inter-server delivery is backing up, with cluster-wide blast radius. That is a very different incident from one lagging client.
• Churn reveals the death spiral. The classic pattern: subscriber falls behind, server disconnects it, client library auto-reconnects and resubscribes, backlog hits immediately, repeat. connections looks stable while total_connections climbs fast. High churn plus a rising slow consumer rate is this pattern.
• Route count gating. Alert when expected_routes > 0 AND routes < expected_routes, not when routes drops to some nonzero floor. Gating on the current count being nonzero misses the worst case: total route loss, where routes = 0. Brief drops during rolling restarts are normal; require 60 seconds sustained.
• JetStream disabled. disabled=true means the persistence layer failed to initialize or shut down: storage directory inaccessible, disk full, configuration error. The uptime and duration gates exist because JetStream can report disabled during initialization on large stores.
• API errors. Idempotent create/bind calls generate benign errors, so alert on the rate and the ratio, not the absolute counter. Publish rejections from full streams show up here.

Level 3: mature

Full coverage of capacity and topology. Add these once the deployment carries real traffic or spans multiple nodes.

• JetStream storage usage (/jsz -> storage, memory, reserved_storage, reserved_memory). Alert when storage approaches 90% of reserved. The behavior at the limit depends on retention and discard policy: DiscardOld evicts old messages (data loss for lagging consumers), DiscardNew rejects publishes (visible as API errors). Compute time-to-exhaustion from the growth trend.
• Stream and consumer counts (/jsz -> streams, consumers). Alert on unexpected decreases. Ephemeral consumers auto-delete after inactivity, so gate alerts on known-durable critical consumers rather than totals.
• Per-account metrics (/accstatz), including per-account slow consumers and resource usage against configured account limits. Alert above 80% of any account limit.
• Gateway health (/gatewayz). Outbound and inbound gateways should match the configured topology. Zero traffic is not necessarily a fault: gateways switch to interest-only mode after the initial flood, so no cross-cluster interest means no traffic.
• Leaf node health (/leafz). Track connection state and RTT per leaf. A leaf drop isolates an edge location; rising RTT is an early warning of WAN degradation.
• Subscription count (/varz -> subscriptions). Track the trend. Growth without matching connection growth indicates a subscription leak, and in a cluster the bloat propagates across routes. Spikes during blue-green deploys are expected.
• Stale connections and stalled clients (/varz). Stale connections are half-dead TCP paths failing ping/pong. Stalled clients are in write-path distress and are a precursor to slow consumer events. Any nonzero value sustained over 5 minutes is worth a ticket.
• TLS certificate expiry (monitoring endpoint). Ticket at 30 days, escalate at 7. Expiry kills all TLS clients, routes, gateways, and leaf nodes at once, and it is always knowable in advance. If your server version does not expose the expiry timestamp in /varz, scrape it from the certificate file with openssl x509 -enddate -noout -in <cert> instead.

Level 4: expert

The signals you add after the third major incident. These are leading indicators and per-asset drilldowns; several are expensive to collect, so scope them to critical streams, consumers, and routes.

• Raft meta cluster health (/jsz -> meta_cluster.leader, meta_cluster.replicas[].current/offline/lag). More than one leader change per 5 minutes, or any peer offline over 60 seconds, means instability. Standalone JetStream has no meta cluster; gate accordingly. Per-stream Raft groups are separate and visible via /raftz.
• Per-consumer lag (/jsz?consumers=true -> per-consumer num_pending, num_ack_pending, num_redelivered). This is the most important JetStream operational signal and the most commonly missing one. A stream with millions of pending messages is down even if the server is healthy. Growing num_pending means the consumer is falling behind; num_ack_pending pinned at the consumer’s MaxAckPending means delivery has stalled completely; rising num_redelivered means processing is failing after delivery. Do not alert on replay or batch consumers, where high pending is the design.
• Per-connection pending bytes (/connz?sort=pending). The leading indicator for slow consumer events: pending bytes grow before the server flags the connection. Use top-N sorting rather than scraping every connection on a high-connection server.
• Per-route pending size (/routez). Any sustained nonzero pending on a route means cluster routing is degrading. Treat it with the same urgency as a route slow consumer.
• Fan-out ratio (out_msgs / in_msgs over time). A shift in the ratio means the subscriber population changed, usually silently. Correlate with churn (total_connections delta) and slow_consumer_stats to separate a single bad client from a reconnect storm.
• JetStream API inflight (/jsz -> api.inflight). Sustained high inflight relative to baseline means JetStream is slow to respond, usually Raft consensus delay or disk I/O. High inflight plus rising API errors points at Raft or the disk.
• Monitoring self-impact (/varz -> http_req_stats). Confirm nobody is hammering expensive endpoints. /subsz more than once a minute on a high-subscription server is a self-inflicted incident.

Signals that look healthy but are not

Four silent failures this checklist is designed to catch. Verify yours does:

1. Zero-subscriber message loss. All server metrics green, messages gone. Only in_msgs vs out_msgs asymmetry reveals it.
2. Consumer stalled at MaxAckPending. Delivery has stopped, no error is raised, the stream grows silently. Only per-consumer num_ack_pending reveals it.
3. Local health, cluster partition. A node’s /healthz returns 200 while it has lost every route. Only the route count check reveals it.
4. Stable connection count, heavy churn. connections flat while total_connections climbs: clients are flapping through a slow-consumer reconnect loop. Only the churn rate reveals it.

How Netdata helps

• Netdata polls the NATS HTTP monitoring endpoints directly, which matches how NATS exposes telemetry: there is no Prometheus endpoint to bridge, just a poller that computes rates from the cumulative counters and handles resets on restart.
• Connection saturation, throughput, memory, and slow consumer counters land on one dashboard, so the slow-consumer death spiral shows up as what it is: churn, pending pressure, and slow consumer events rising together.
• JetStream aggregates from /jsz (storage, API totals and errors, stream and consumer counts) are collected alongside server-level signals, so storage growth can be correlated against API error rate to distinguish exhaustion from disk stalls.
• Anomaly detection on throughput and connection counts surfaces the silent cases, like in_msgs climbing while out_msgs goes flat, without requiring a hand-tuned static threshold per deployment.
• Health and uptime tracking turn the Level 1 page conditions into durable alerts with the sustained-duration and cold-start gating described above, instead of raw probe flapping.

Related guides

• How NATS actually works in production: a mental model for operators
• NATS monitoring maturity model: from survival to expert
• NATS server not responding: healthz failing and the process down or hung
• NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check
• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS slow consumer detected: the write buffer overflowed and messages were dropped
• NATS slow consumer breakdown: clients vs routes vs gateways and blast radius
• NATS pending bytes growing: catching a slow consumer before it is disconnected
• NATS stalled clients and stale connections: half-dead sockets and write-path distress
• NATS write_deadline and buffer sizing: tuning how long the server waits on a slow writer
• NATS Maximum Connections Exceeded: new clients rejected at the max_connections wall
• NATS file descriptor exhaustion: too many open files and the ulimit cliff