NATS memory growth and OOM: reading RSS past the Go GC sawtooth

The nats-server process is climbing toward its container memory limit. Grafana shows a jagged line that spikes to nearly double the baseline and drops back, except lately the drops are getting shallower and the floor keeps rising. Then the pod restarts, uptime resets to a few minutes, and the cycle repeats. If JetStream is enabled, the numbers look even worse, and half of what you see is not memory the server is actually holding.

Three different things produce the same symptom: normal Go garbage collection, reclaimable memory the OS counts against the process, and genuine unbounded growth. Restarting the server hides all three for a few hours and tells you nothing.

Every signal you need is on the monitoring port, and the distinguishing test is simple: stop looking at peaks and start looking at the trend of the troughs.

What this means

NATS is written in Go, so its memory follows a garbage-collected sawtooth. The heap grows between GC cycles, then drops sharply when the collector runs. With the default GOGC=100, the runtime lets the heap double before collecting, so peaks around 2x baseline are expected behavior, not a leak.

The /varz mem field is RSS, the resident set size from the OS perspective. It is not the Go heap. It includes the heap, runtime overhead, goroutine stacks, and, critically, memory-mapped files. JetStream file-based storage uses memory-mapped files for message caching, and the OS counts those pages as RSS even though they are reclaimable under memory pressure. This is the most common source of false “NATS is leaking” reports.

A genuine problem looks different: the floor of the sawtooth rises monotonically over hours and never comes back down, and the rise is not explained by growth in connections, subscriptions, or JetStream assets. That is the leak or unbounded-accumulation signal, and it ends in an OOM kill if you let it run.

Common causes

Quick checks

All read-only, safe to run during an incident.

# Current RSS, connections, subscriptions in one shot
curl -s http://localhost:8222/varz | jq '{mem, connections, subscriptions, slow_consumers, uptime}'

# RSS from the OS side, in KB, to confirm /varz agrees
ps -p $(pgrep nats-server) -o rss=

# Worst offenders by buffered-but-unsent bytes
curl -s "http://localhost:8222/connz?sort=pending&limit=10" | jq '.connections[] | {cid, name, pending_bytes, subscriptions}'

# Route-level backpressure (cluster-wide blast radius)
curl -s http://localhost:8222/routez | jq '.routes[] | {rid, ip, pending_size}'

# JetStream memory and storage footprint
curl -s http://localhost:8222/jsz | jq '{memory, storage, bytes, messages, streams, consumers, reserved_memory}'

# Confirm an OOM kill actually happened (vs a crash or deploy); needs root on hardened systems
dmesg -T | grep -i -E "oom|killed process.*nats"

In Kubernetes, dmesg is often unavailable from inside the pod or the node log has rotated; check kubectl describe pod for OOMKilled in the container’s last state instead.

Two more checks that matter but are not on the monitoring port by default:

• Goroutine count. Expected is roughly 2 x (connections + routes + gateways + leafnodes) plus server internals and JetStream Raft goroutines. If pprof (/debug/pprof/goroutine) or the metrics endpoint is enabled, a count growing well past that while connections stay flat points at a goroutine leak. Each leaked goroutine holds several KB of stack, so tens of thousands of them become hundreds of MB.
• Container limit vs RSS. In a container, compare RSS against the cgroup limit, not host RAM. Also make sure the Go runtime sees the right limits: CPU quota for GOMAXPROCS and memory limit for GC targeting (see GOMEMLIMIT under Fixes).

How to diagnose it

Work through this in order. The whole point is to avoid “fixing” sawtooth peaks that were never a problem.

flowchart TD
  A[RSS climbing or OOM restart] --> B{Did the floor of the
sawtooth rise over 4+ hours?}
  B -- "No: peaks only, floor flat" --> C[Normal GC sawtooth.
No action; fix the alert threshold.]
  B -- Yes --> D{Rise correlates with connections,
subscriptions, or stream counts?}
  D -- "Yes: connections" --> E[Per-connection buffers.
Check pending_bytes and slow consumers.]
  D -- "Yes: subscriptions" --> F[Subject trie growth.
Hunt the subscription leak.]
  D -- "Yes: JetStream assets" --> G[Metadata cache and Raft state.
Check stream/consumer counts and storage config.]
  D -- "No correlation" --> H{JetStream file storage
with high mmap/page cache?}
  H -- Yes --> I[Reclaimable RSS.
Verify OS does not OOM; size limits with headroom.]
  H -- No --> J[Genuine leak or goroutine leak.
Capture goroutine/heap profile; check version bugs.]

1. Confirm the OOM. Check dmesg (or kubectl describe pod) for the OOM killer and /varz uptime for the restart. If uptime reset but there is no OOM record, you may be chasing a crash loop instead; see the crash-loop guide in Related guides.
2. Establish the trend, not the peak. Pull mem over at least 4 hours. Fit a line through the troughs (or eyeball the floor). Flat floor with tall teeth: stop here, your alert is wrong. Rising floor: continue.
3. Correlate against workload drivers. Plot connections, subscriptions, and JetStream stream/consumer counts on the same window. Memory should scale with (connections x per-connection overhead) + (subscriptions x per-sub overhead) + JetStream cache + baseline. If RSS grows in lockstep with one of these, that is your driver and it is a capacity or client-behavior problem, not a leak.
4. Check for backpressure buffering. High pending_bytes on clients (/connz?sort=pending) or routes (/routez) means the server is holding messages in memory for consumers that cannot keep up. RSS climbs until the slow-consumer logic disconnects them, then partially recovers. Chronic slow consumers look like a slow leak.
5. Discount the JetStream mmap effect. If the server has large, actively-read file-store streams, a chunk of RSS is memory-mapped file data the OS will reclaim under pressure. It inflates every reading and survives GC. Do not chase it unless the kernel actually OOM-kills the process.
6. If nothing correlates, suspect a leak. Capture a goroutine profile (if pprof is enabled) and check whether goroutine count is growing independently of connections. Also check whether you are running a version with known memory-growth issues; several NATS releases have had confirmed JetStream memory regressions, fixed in later patches, so compare your version against the changelog before assuming your workload is at fault.

Metrics and signals to monitor

Fixes

If it is the GC sawtooth (most common outcome)

Nothing is broken. Change the alert, not the server. Alert on the trend of RSS (a regression slope over hours) and on RSS as a ratio of the container limit, never on instantaneous peaks. Peaks near 2x baseline are the collector doing its job.

If the peaks themselves are close enough to the limit to risk OOM, you can trade CPU for memory by lowering GOGC (default 100), which makes GC run sooner and caps how far the heap doubles. In containers, setting GOMEMLIMIT gives the collector a target to stay under; be aware this is a soft limit, and RSS can still exceed it because of non-heap memory like mmap’d files and stacks.

If it is slow-consumer buffering

The memory is a symptom; the consumer is the fault. Identify the offenders via /connz?sort=pending, fix or scale the slow application, and check the slow-consumer breakdown: if routes or gateways are the slow ones, you have an inter-server problem with much higher blast radius. See the connection churn guide in Related guides for the reconnect-disconnect spiral that follows.

If it is subscription or connection growth

For subscriptions: find the client that subscribes without unsubscribing (a common reconnect-handler bug that re-subscribes on every reconnect without cleaning up). For connections: the fix is on the client side, such as connection pooling or fixing a connection leak. Do not raise memory limits to accommodate a leak.

If it is JetStream-driven

Check whether stream and consumer counts are growing (metadata cache and Raft state scale with them) and whether storage limits are set sanely. If a specific stream’s in-memory footprint is the problem, review its retention, replica count, and whether it needs to exist at all. Increases to JetStream max_file_store and max_memory_store are applied on config reload; decreases and storage-directory changes require a restart.

If it is a genuine leak

Capture the evidence first: goroutine profile, RSS trend, and the correlation data showing no workload driver. Then check your nats-server version against release notes for known memory regressions. A rolling restart (one node at a time in a cluster) buys time but is not a fix. If you upgrade to escape a known leak, verify the target version specifically addresses it.

Prevention

• Size with headroom for GC and filesystem cache. Keep peak RSS under roughly 70% of the memory available to the process. The remaining 30% absorbs the GC sawtooth peaks and the page cache JetStream file storage depends on for read performance. A container limit set at “what the server uses at baseline” guarantees OOM kills at the first GC peak. At minimum, leave 20% between peak RSS and the limit.
• Alert on slope, not level. A linear regression on mem over the last hour gives you runway estimation: (limit - current_rss) / growth_rate. This pages you days early on a real leak and never fires on GC teeth.
• Alert on the ratio, not absolutes. mem / container_limit at 80% is meaningful across environments; “mem > 2 GB” is not.
• Trend the drivers alongside RSS. Connections, subscriptions, slow consumers, and JetStream asset counts on the same dashboard turn “memory is up” into “memory is up because subscriptions doubled” in one glance.
• Ignore the first 5-30 minutes after restart. Cold-start memory and CPU are not representative: caches are empty, subscription tables are rebuilding, and JetStream is recovering. Do not tune against warmup numbers.

How Netdata helps

• Netdata polls /varz on the monitoring port and charts mem, connections, subscriptions, and slow consumers per second, so the GC sawtooth is visible as a sawtooth instead of being flattened into a misleading average by a 60-second scrape interval.
• Per-second resolution lets you overlay the RSS floor against connection and subscription trends in one view, which is the exact correlation test from the diagnosis steps above.
• Netdata’s ML anomaly detection runs per metric, so a rising RSS trough that deviates from learned behavior flags as anomalous while routine GC peaks do not.
• Uptime monitoring catches the OOM restart itself, and correlating the restart timestamp against the RSS curve confirms whether memory was the cause.
• Because Netdata also collects host-level memory and cgroup metrics, you can compare nats-server RSS against the container limit and host page cache on the same dashboard, which is how you spot the mmap-reclaimable portion without SSHing in.

Related guides

• NATS JetStream AckWait tuning: matching the ack timeout to processing time
• NATS connection churn: a stable connection count hiding constant reconnects
• NATS connection storm: reconnect thundering herd after a network event
• NATS JetStream consumer lag growing: falling behind the stream
• NATS consumer stalled at MaxAckPending: delivery stops until messages are acked
• NATS JetStream redelivery loop: num_redelivered climbing and messages reprocessed
• NATS JetStream consumer stopped receiving messages: the diagnostic tree
• NATS context deadline exceeded: JetStream publish and request timeouts
• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS file descriptor exhaustion: too many open files and the ulimit cliff
• NATS gateway disconnected: cross-cluster traffic cut in a supercluster
• NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check