NATS server not responding: healthz failing and the process down or hung

Your probe against the NATS monitoring port is failing: /healthz returns non-200, or the port does not answer at all. Clients may still be connected, reconnecting in a storm, or already failing over to other cluster nodes.

Do not start with a restart. A crashed process, a hung event loop, lame-duck shutdown, resource exhaustion, and JetStream recovery can all look like “NATS is down,” and the first safe action is different for each. Restarting too early destroys evidence and can turn a recoverable hang into data loss.

Classify the failure first, then correlate health with uptime, CPU, memory, and connection pressure so you page on real failures instead of cold starts and recovery windows.

What this means

The NATS monitoring port, commonly 8222 when enabled with -m 8222 or equivalent config, exposes /healthz. A 200 with {"status":"ok"} means the server considers itself ready under the semantics of the probe you used. Those semantics matter:

• Bare /healthz on a JetStream-enabled server performs full health checks, including JetStream readiness, meta recovery, and asset recovery. It can return non-200 during JetStream recovery after a restart, which can last minutes on large stores.
• /healthz?js-server-only=true performs only basic server readiness, without deeper JetStream asset validation. This is usually the right probe for paging.
• /healthz?js-enabled-only=true checks only whether JetStream is enabled.

A 200 does not prove your streams, consumers, or Raft groups are healthy. A non-200 on the bare endpoint during JetStream recovery does not prove the server is down.

The paging condition that survives false-positive analysis is: /healthz?js-server-only=true returns non-ok or is unreachable, sustained for at least 60 seconds, with uptime > 300 seconds. The 60-second window filters transient stalls; the uptime gate filters cold starts.

Common causes

Version-specific trap: there are reports that some v2.10.x releases can produce readiness false negatives under load with JetStream-related probe settings, causing Kubernetes to detach pods while the server is still serving. Do not carry probe config forward across charts or versions without testing it.

Quick checks

All of these are read-only. Run them from the host or from anywhere that can reach the monitoring port. Some OS checks may need root or equivalent journal access.

# Is the process alive at all?
pgrep -a nats-server

# Basic readiness: the probe you should page on
curl -s -o /dev/null -w "%{http_code}\n" --max-time 5 http://localhost:8222/healthz?js-server-only=true

# Bare healthz: full JetStream health on JS-enabled servers
curl -s --max-time 5 http://localhost:8222/healthz

# Uptime: a recent reset means crash or restart; low uptime means cold start or recovery
curl -s http://localhost:8222/varz | jq '{uptime, now: .now}'

# Resource pressure: CPU, RSS memory, connection load
curl -s http://localhost:8222/varz | jq '{cpu, cores, mem, connections, max_connections, slow_consumers, stale_connections, stalled_clients}'

# JetStream state, if enabled: disabled flag plus API error pressure
curl -s http://localhost:8222/jsz | jq '{disabled, api: .api}'

# OS-level confirmation of OOM or I/O trouble
dmesg | tail -50
journalctl -u nats-server --since "30 min ago"

Two transport-level checks before you conclude anything about the server:

• Is the monitoring port actually enabled? If monitoring was never configured, there is no health endpoint and nothing is wrong with the server.
• Is TLS on the monitoring port? If monitoring is configured in secure mode, plain HTTP can fail in a way that looks like an unresponsive server. Try HTTPS before declaring the process hung.

How to diagnose it

Work top-down. Classify the failure before touching the process.

flowchart TD
  A[healthz failing] --> B{Port answers TCP?}
  B -- no --> C{Process alive?}
  C -- no --> D[Crash or OOM kill: check uptime, logs, dmesg or journal]
  C -- yes --> E[Monitoring disabled or TLS mismatch: check config, try HTTPS]
  B -- yes --> F{js-server-only 200?}
  F -- yes --> G[Bare healthz failing: JetStream recovery or JS distress - check uptime]
  F -- no --> H{CPU or memory saturated?}
  H -- yes --> I[Resource exhaustion: correlate mem trend, cpu, connections]
  H -- no --> J[Hung or lame-duck: capture SIGQUIT stack dump before restart]

1. Classify: unreachable vs non-200. Connection refused or timeout at the TCP level means the process is dead, wedged before the HTTP listener, or the port is misconfigured. A TCP connection that succeeds but returns non-200 means the process is alive and deliberately reporting unready.

2. Check uptime immediately. curl -s http://localhost:8222/varz | jq .uptime. An uptime of seconds or minutes changes the case: you are looking at a restart, crash loop, or JetStream recovery, not a hung steady-state server. In a cluster, compare uptime across nodes; simultaneous resets indicate a cluster-wide event, not a single-node fault.

3. Compare the two probes. If js-server-only=true returns 200 but bare /healthz fails, the core server is ready and the failure is in JetStream health: meta recovery, asset recovery, or Raft not current. After a restart on a large store this can be expected and can last minutes. Do not restart the server to fix it; restarting restarts recovery.

4. Check for resource exhaustion. Pull cpu, mem, and connections from /varz. Monotonic RSS growth without GC sawtooth recovery points to a leak or unbounded buffering; slow consumers can accumulate memory before the server disconnects them. CPU pinned near cores x 100 means saturation. Connections at max_connections, or FDs at ulimit -n, mean new work, possibly including your probe, cannot get in.

5. Rule out lame-duck mode. A server in lame-duck mode stops accepting new connections and evicts clients over lame_duck_duration. Health behavior during lame-duck has varied, so do not treat flapping during a drain as proof of a crash. If orchestration, systemd, a deploy pipeline, or an operator initiated shutdown, the outage is a controlled drain. Check server logs and whatever sent the signal.

6. If the process looks genuinely hung, capture evidence before restarting. Sending SIGQUIT makes the Go runtime dump all goroutine stacks to stderr and then terminate the process. This is destructive: it kills the server. On a server you are about to restart anyway, the stack dump is the difference between “we restarted it” and a root cause. Route stderr somewhere durable before you need it. If you cannot tell whether the event loop is stalled versus the whole process being frozen, the dump is the ground truth either way.

7. Correlate with surrounding signals. stale_connections and stalled_clients climbing alongside health flaps points to network or saturation problems rather than a dead process. Rising slow_consumers points to backpressure building before the server became unresponsive.

Metrics and signals to monitor

Fixes

Crashed or crash-looping process

Get the restart reason before restarting again. Check service manager logs, kernel logs for OOM kills, and the NATS server log for panics. If the loop is OOM-driven, restarting without addressing memory only restarts the clock. If the crash follows an upgrade or a change in exposure, such as new ports or new client types, check NATS security advisories for pre-auth crash CVEs affecting your version; several have been published, and the fix is an upgrade, not a restart. More than 3 restarts in 30 minutes is a crash loop: stop the restart cycle and diagnose with the process stable.

Hung process

Capture a SIGQUIT goroutine dump first if the server is already destined for restart; remember it terminates the process. If hangs recur, use the dump to distinguish GC pressure, lock contention, and blocked I/O. Recurring hangs under load usually trace back to memory pressure or disk stalls on the JetStream storage path.

JetStream recovery masquerading as an outage

Do nothing to the server. Watch bare /healthz and uptime; recovery completes on its own. If your orchestrator readiness probe kills the pod before recovery finishes, that is the actual bug: lengthen probe timeouts and switch the probe to js-server-only=true, or to a lighter check where your server version requires it after verification.

Resource exhaustion

• Memory: identify the driver before raising limits. Per-connection buffers, subscription growth, and JetStream caches show up differently: correlate mem against connections, subscriptions, and /jsz memory. Raising the container limit without finding the driver postpones the next page.
• File descriptors: raise ulimit -n. 1024 is catastrophically low; 65536 is a reasonable floor for production. This requires a restart, so plan it.
• CPU: if TLS-heavy connection churn is the driver, reduce churn with client reconnect backoff and stable network paths rather than only adding cores.

Lame-duck shutdown

If the drain was intentional, let it finish and check why alerting treated a planned shutdown as an outage. If it was not intentional, find what triggered lame-duck: deploy tooling, systemd stop, or an operator. If lame_duck_duration has long expired and the process is still alive, capture a SIGQUIT dump and restart.

Prevention

• Page on the right probe. /healthz?js-server-only=true, non-ok or unreachable, sustained >=60s, gated on uptime > 300s. Alert on bare /healthz as a ticket so JetStream recovery and backup-induced I/O stalls wake nobody up.
• Track uptime resets. Every unexpected reset is a crash postmortem waiting to happen. Correlate resets across cluster nodes to catch cluster-wide events.
• Watch leading indicators, not just the wall. Memory trend rather than spikes, connections as a ratio of max_connections, FD headroom, and stalled_clients give you minutes to hours before the health probe fails.
• Size OS limits for NATS. Use ulimit -n of at least 65536 and memory headroom around 20% above peak RSS. Go GC can raise heap transiently; JetStream mmap’d files can inflate RSS without being leaks.
• Make readiness probes survivable in Kubernetes. Use timeouts generous enough for JetStream recovery on your largest store, and match probe parameters to your server version. An aggressive probe turns every rolling restart into an outage.
• Collect stderr and core evidence by default. The next hang will be diagnosed by whatever you captured this time. Route nats-server stderr to a durable log before the incident, not during it.

How Netdata helps

• Netdata polls the NATS HTTP monitoring endpoints and turns cumulative counters such as in_msgs, out_msgs, slow_consumers, and total_connections into rates, so you can see backpressure building in the minutes before the server stopped responding.
• Uptime, memory, CPU, and connection counts are collected together from /varz, which makes the key correlation in this article, health failure plus uptime reset plus memory trend, a single-view check instead of three curl commands.
• Per-second host metrics for CPU saturation, memory pressure, disk I/O, and OOM kills line up with NATS-level signals on one timeline, which is how you tell a hung process from a starved one.
• Anomaly detection on mem and connections surfaces the monotonic leak pattern and the connection-storm pattern without static thresholds that break across deployment sizes.
• Because HTTP endpoint snapshots can miss sub-second events, a high-resolution agent narrows the gap between “the probe failed” and “what the server was doing in the 60 seconds before.”

Related guides

No related guides are published in this section yet. See the NATS operations guides hub for the full mental model, signal catalog, and failure-pattern reference.