NATS crash loop: unexpected uptime resets and repeated restarts

Your NATS server’s /varz uptime keeps resetting. The server was up for 4 minutes, then 2 minutes, then 6. Clients reconnect repeatedly, JetStream streams flap between unavailable and recovering, and every restart replays the WAL from scratch. This is a crash loop, and the fix depends entirely on why the process is dying.

The uptime field on /varz is the fastest confirmation. It reports time since process start as a NATS-specific duration string with y/d/h/m/s suffixes (for example “1d2h3m4s”), not a standard Go duration. When that value drops between scrapes, the process restarted. A single restart may be maintenance. More than 3 restarts in 30 minutes is a crash loop and needs an owner.

Before touching anything, answer one question: is the process crashing on its own, or is something external killing it? Those two paths share almost no diagnostic steps.

What this means

A NATS server restart is always one of four things:

1. The process died on its own: panic, segfault, or an unrecoverable internal error.
2. The kernel or runtime killed it: the Linux OOM killer, or a cgroup memory limit.
3. Something asked it to stop: SIGTERM from a deploy, pod eviction, or an operator. NATS handles SIGTERM with lame duck mode, draining connections before exiting, so deliberate restarts look graceful in the logs.
4. A health check killed it: an orchestrator readiness or liveness probe failed and the platform restarted the container. On JetStream servers this is a classic trap: bare /healthz fails during asset recovery after startup, and an aggressive probe restarts the server before recovery finishes. Each restart begins recovery again. The loop never converges.

Each restart is expensive. On startup with JetStream, the server replays the WAL and rebuilds in-memory indexes, which can take minutes on large stores. Clients reconnect in a storm, and stream Raft groups that lost their leader hold elections. Even a minor root cause gets amplified by the restarts themselves into secondary failures like Raft leader flapping.

Common causes

Quick checks

# Current uptime: has it reset since your last check?
curl -s http://localhost:8222/varz | jq .uptime

# Memory trend: monotonic growth without GC drops points at OOM
curl -s http://localhost:8222/varz | jq '{mem, cpu, connections, subscriptions}'

# Basic server readiness only (skips JetStream asset checks)
curl -s http://localhost:8222/healthz?js-server-only=true

# Full health including JetStream recovery state (expected to fail during recovery)
curl -s http://localhost:8222/healthz

# JetStream storage footprint: large stores mean long recovery
curl -s http://localhost:8222/jsz | jq '{bytes, messages, streams, consumers, disabled}'

On the host, check for kernel OOM kills. This is read-only:

# Look for OOM killer entries around the restart times
dmesg -T | grep -i -E "out of memory|killed process"

If the server runs under systemd:

# Exit status and restart history for the unit
systemctl status nats-server
journalctl -u nats-server --since "1 hour ago" | grep -E "Shutdown|lame duck|panic|fatal"

On Kubernetes, the restart count and last exit reason tell you most of the story:

# Restart count and last termination reason (OOMKilled vs Completed vs Error)
kubectl get pods -l app=nats
kubectl describe pod <nats-pod> | grep -A5 "Last State"

# Logs from the crashed container instance
kubectl logs <nats-pod> --previous --tail=100

In the --previous logs, look for a Go panic stack trace (process died on its own) or a clean lame duck shutdown sequence (something sent SIGTERM, which points at the orchestrator, not NATS).

How to diagnose it

Work top-down. Each step eliminates a branch.

1. Confirm the loop and its cadence. Poll /varz uptime every 15 to 30 seconds for a few minutes, or graph it in your monitoring. Note the interval between resets. A metronome-regular interval (every 60s, every 120s) strongly suggests a probe or supervisor timeout, not a crash. Irregular intervals suggest OOM or panic under load.

2. Check whether the restarts are deliberate. Pull orchestrator events and deploy history for the restart timestamps. On Kubernetes, kubectl describe pod shows probe failures, evictions, and preemption. A clean lame duck shutdown in the logs means SIGTERM arrived: find who sent it before assuming NATS is broken.

3. Check for OOM. If the container’s last state is OOMKilled, or dmesg shows the kernel killing nats-server, the memory path is confirmed. Then determine whether growth was real (connection buffers, subscription trie, JetStream caches) or a container sizing problem. In containers, set GOMEMLIMIT explicitly: without it the Go runtime sizes against host memory, not the cgroup limit, so the container hits its limit long before Go’s GC feels any pressure. Set it to roughly 80 to 90 percent of the container memory limit so the GC works against the same ceiling the kernel enforces.

4. Check for a probe-induced loop on JetStream servers. Compare the reset interval to the probe period times failure threshold. Bare /healthz on a JetStream server performs full asset recovery checks and fails for the entire recovery window, which can be minutes on large stores. If your readiness probe uses bare /healthz with a tight timeout, the orchestrator kills the pod mid-recovery, recovery restarts from scratch, and the pod never becomes ready. Switch the probe to /healthz?js-server-only=true and give it generous timeouts and failure thresholds.

5. Read the crash itself. If neither OOM nor a probe explains it, the previous-instance logs should contain a panic, a fatal config error, or a startup failure. A server that exits within seconds of starting, especially right after a config change or SIGHUP reload, is a config problem. A panic stack trace with a consistent location across restarts is a bug: check whether you are on an affected version and plan an upgrade.

6. Zoom out to the cluster. Pull uptime from every node. Simultaneous resets across nodes are a cluster-wide event: shared infrastructure, a network event, or a coordinated action like a rolling restart or config push. Staggered resets on one node are a node-local problem. Do not debug a single server when all five reset at the same second.

flowchart TD
    A[Uptime reset detected] --> B{Deliberate SIGTERM in logs?}
    B -- yes --> C[Check deploys, drains, rolling restarts]
    B -- no --> D{OOMKilled or kernel OOM in dmesg?}
    D -- yes --> E[Set GOMEMLIMIT, size memory, find growth source]
    D -- no --> F{Resets at probe interval, JS store large?}
    F -- yes --> G[Use js-server-only probe, raise timeouts]
    F -- no --> H{Panic or config error in previous logs?}
    H -- panic --> I[Capture trace, check version, upgrade]
    H -- config --> J[Validate config, roll back change]
    A --> K{Multiple nodes reset simultaneously?}
    K -- yes --> L[Cluster-wide event: infra, network, coordinated action]

Metrics and signals to monitor

One instrumentation caveat: all /varz counters reset to zero on restart. Any rate-based monitoring must handle counter resets or every restart will look like a massive negative spike in throughput and connection metrics.

Fixes

OOM kills

Set GOMEMLIMIT in every containerized deployment so Go’s GC targets the container limit rather than host memory. Keep peak RSS below roughly 80 percent of the container limit; you want about 20 percent headroom between peak RSS and the limit. If memory growth is genuine, find the source before raising the limit: per-connection buffers scale with connections and slow consumers, the subscription trie scales with subscription count, and JetStream adds caches and Raft state that can dwarf core connection memory. Treat a slow-consumer-driven buffer buildup as a consumer problem first.

Panics

Capture the full stack trace from the crashed instance before the logs rotate. Check the trace against known issues for your server version. JetStream recovery and crash-consistency bugs have been fixed in specific patch releases; if the trace matches a fixed issue, schedule an upgrade rather than working around it. If the panic is in startup recovery of a specific stream, that asset’s store files are the suspect.

Config errors

A server that dies immediately after a config reload or change should be rolled back first, diagnosed second. Validate config changes in staging, and prefer startup validation over discovering a bad directive during a production reload. If you must reload in production, reload one node at a time and watch uptime before proceeding.

Readiness probe killing JetStream recovery

Point readiness probes at /healthz?js-server-only=true so the check reflects process readiness rather than asset recovery completion. Raise probe timeouts and failure thresholds so the server gets minutes, not seconds, to finish WAL replay and index rebuild. Also give the server a generous graceful shutdown window (lame duck drain takes time on busy servers) so scale-downs do not add hard kills on top of the recovery problem. The same logic applies to alerting: page on the basic readiness probe, ticket on bare /healthz.

Cluster-wide events

If multiple nodes reset together, stop debugging the server. Correlate the timestamp against infrastructure changes, network events, and orchestrator actions. Rolling restarts done too fast, a shared storage event, or a simultaneous OOM across nodes all present this way.

Prevention

• Graph uptime per node and alert on drops. A reset is a ticket, not a page: single restarts can be maintenance, and in a cluster one node restarting is not an outage. Page on the readiness probe instead.
• Alert on restart cadence. More than 3 restarts in 30 minutes is a crash loop regardless of cause.
• Set GOMEMLIMIT on every containerized server, and keep peak RSS under 80 percent of the container limit.
• Use js-server-only readiness probes on JetStream servers, with timeouts sized for your largest store’s recovery time. Re-verify after storage grows.
• Handle counter resets in any rate-based monitoring on /varz counters.
• Gate readiness pages on uptime above 300s. This suppresses cold-start false positives during exactly the recovery window this article is about.
• Validate config in staging and reload one node at a time in production.
• Correlate uptime across the cluster in a single view so simultaneous resets are obvious at a glance.

How Netdata helps

• Uptime reset detection: Netdata’s NATS collector polls /varz, so uptime resets show up immediately on the uptime chart and can drive restart detection alerts without building custom polling.
• Memory-before-crash correlation: RSS from /varz graphed next to restart events shows whether memory was climbing into each death, which confirms or rules out the OOM path in seconds.
• Cluster-wide restart correlation: uptime charts for all nodes in one dashboard make simultaneous resets, and therefore cluster-wide events, visible without manual cross-node queries.
• Health probe monitoring: tracking the readiness endpoint alongside restarts exposes probe-induced loops, where healthz failures line up one-for-one with kills.
• Counter reset handling: rate charts for in_msgs, out_msgs, and connections survive server restarts cleanly, so restarts do not masquerade as traffic collapses.
• Recovery window visibility: JetStream aggregate metrics from /jsz show when storage and stream state return to normal after each restart, so you can size probe timeouts from real recovery duration.

Related guides

• How NATS actually works in production: a mental model for operators
• NATS monitoring checklist: the signals every production server needs
• NATS monitoring maturity model: from survival to expert
• NATS server not responding: healthz failing and the process down or hung