The nats-server process is climbing toward its container memory limit. Grafana shows a jagged line that spikes to nearly double the baseline and drops back, except lately the drops are getting shallower and the floor keeps rising. Then the pod restarts, uptime resets to a few minutes, and the cycle repeats. If JetStream is enabled, the numbers look even worse, and half of what you see is not memory the server is actually holding.

Three different things produce the same symptom: normal Go garbage collection, reclaimable memory the OS counts against the process, and genuine unbounded growth. Restarting the server hides all three for a few hours and tells you nothing.

Every signal you need is on the monitoring port, and the distinguishing test is simple: stop looking at peaks and start looking at the trend of the troughs.

What this means

NATS is written in Go, so its memory follows a garbage-collected sawtooth. The heap grows between GC cycles, then drops sharply when the collector runs. With the default GOGC=100, the runtime lets the heap double before collecting, so peaks around 2x baseline are expected behavior, not a leak.

The /varz mem field is RSS, the resident set size from the OS perspective. It is not the Go heap. It includes the heap, runtime overhead, goroutine stacks, and, critically, memory-mapped files. JetStream file-based storage uses memory-mapped files for message caching, and the OS counts those pages as RSS even though they are reclaimable under memory pressure. This is the most common source of false “NATS is leaking” reports.

A genuine problem looks different: the floor of the sawtooth rises monotonically over hours and never comes back down, and the rise is not explained by growth in connections, subscriptions, or JetStream assets. That is the leak or unbounded-accumulation signal, and it ends in an OOM kill if you let it run.

Common causes

CauseWhat it looks likeFirst thing to check
Normal Go GC sawtoothPeaks around 2x baseline, sharp drops, flat floor over daysTrend of troughs over 4+ hours, not peaks
JetStream mmap / page cache in RSSHigh stable RSS that does not cause OOM; correlates with active file-store streamsCompare /jsz memory/storage and stream activity with RSS
Slow consumer bufferingRSS climbs with rising pending_bytes on connections or routes/connz?sort=pending and /routez pending sizes
Subscription or subject-trie growthsubscriptions climbs without matching connection growth/varz subscriptions trend vs connections trend
Connection growthRSS tracks connections linearly, ~tens of KB per connection plus buffers/varz connections and total_connections churn
JetStream metadata, caches, Raft stateRSS grows with stream/consumer count, not message rate/jsz streams/consumers counts vs RSS
Goroutine leakRSS and goroutine count grow while connections stay flatpprof goroutine count vs expected ~2x connections
Genuine leak or version bugMonotonic floor rise for 4+ hours, uncorrelated with any workload metricSlope of RSS troughs against all of the above

Quick checks

All read-only, safe to run during an incident.

# Current RSS, connections, subscriptions in one shot
curl -s http://localhost:8222/varz | jq '{mem, connections, subscriptions, slow_consumers, uptime}'

# RSS from the OS side, in KB, to confirm /varz agrees
ps -p $(pgrep nats-server) -o rss=

# Worst offenders by buffered-but-unsent bytes
curl -s "http://localhost:8222/connz?sort=pending&limit=10" | jq '.connections[] | {cid, name, pending_bytes, subscriptions}'

# Route-level backpressure (cluster-wide blast radius)
curl -s http://localhost:8222/routez | jq '.routes[] | {rid, ip, pending_size}'

# JetStream memory and storage footprint
curl -s http://localhost:8222/jsz | jq '{memory, storage, bytes, messages, streams, consumers, reserved_memory}'

# Confirm an OOM kill actually happened (vs a crash or deploy); needs root on hardened systems
dmesg -T | grep -i -E "oom|killed process.*nats"

In Kubernetes, dmesg is often unavailable from inside the pod or the node log has rotated; check kubectl describe pod for OOMKilled in the container’s last state instead.

Two more checks that matter but are not on the monitoring port by default:

  • Goroutine count. Expected is roughly 2 x (connections + routes + gateways + leafnodes) plus server internals and JetStream Raft goroutines. If pprof (/debug/pprof/goroutine) or the metrics endpoint is enabled, a count growing well past that while connections stay flat points at a goroutine leak. Each leaked goroutine holds several KB of stack, so tens of thousands of them become hundreds of MB.
  • Container limit vs RSS. In a container, compare RSS against the cgroup limit, not host RAM. Also make sure the Go runtime sees the right limits: CPU quota for GOMAXPROCS and memory limit for GC targeting (see GOMEMLIMIT under Fixes).

How to diagnose it

Work through this in order. The whole point is to avoid “fixing” sawtooth peaks that were never a problem.

flowchart TD
  A[RSS climbing or OOM restart] --> B{Did the floor of the
sawtooth rise over 4+ hours?}
  B -- "No: peaks only, floor flat" --> C[Normal GC sawtooth.
No action; fix the alert threshold.]
  B -- Yes --> D{Rise correlates with connections,
subscriptions, or stream counts?}
  D -- "Yes: connections" --> E[Per-connection buffers.
Check pending_bytes and slow consumers.]
  D -- "Yes: subscriptions" --> F[Subject trie growth.
Hunt the subscription leak.]
  D -- "Yes: JetStream assets" --> G[Metadata cache and Raft state.
Check stream/consumer counts and storage config.]
  D -- "No correlation" --> H{JetStream file storage
with high mmap/page cache?}
  H -- Yes --> I[Reclaimable RSS.
Verify OS does not OOM; size limits with headroom.]
  H -- No --> J[Genuine leak or goroutine leak.
Capture goroutine/heap profile; check version bugs.]
  1. Confirm the OOM. Check dmesg (or kubectl describe pod) for the OOM killer and /varz uptime for the restart. If uptime reset but there is no OOM record, you may be chasing a crash loop instead; see the crash-loop guide in Related guides.
  2. Establish the trend, not the peak. Pull mem over at least 4 hours. Fit a line through the troughs (or eyeball the floor). Flat floor with tall teeth: stop here, your alert is wrong. Rising floor: continue.
  3. Correlate against workload drivers. Plot connections, subscriptions, and JetStream stream/consumer counts on the same window. Memory should scale with (connections x per-connection overhead) + (subscriptions x per-sub overhead) + JetStream cache + baseline. If RSS grows in lockstep with one of these, that is your driver and it is a capacity or client-behavior problem, not a leak.
  4. Check for backpressure buffering. High pending_bytes on clients (/connz?sort=pending) or routes (/routez) means the server is holding messages in memory for consumers that cannot keep up. RSS climbs until the slow-consumer logic disconnects them, then partially recovers. Chronic slow consumers look like a slow leak.
  5. Discount the JetStream mmap effect. If the server has large, actively-read file-store streams, a chunk of RSS is memory-mapped file data the OS will reclaim under pressure. It inflates every reading and survives GC. Do not chase it unless the kernel actually OOM-kills the process.
  6. If nothing correlates, suspect a leak. Capture a goroutine profile (if pprof is enabled) and check whether goroutine count is growing independently of connections. Also check whether you are running a version with known memory-growth issues; several NATS releases have had confirmed JetStream memory regressions, fixed in later patches, so compare your version against the changelog before assuming your workload is at fault.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
/varz mem (RSS), trendedThe actual OOM inputMonotonic floor rise over 4+ hours; RSS over 80% of container limit
/varz connections + total_connectionsPer-connection buffers are a primary RSS driver; churn adds allocation churnRSS tracking connections; high churn with stable count
/varz subscriptionsSubject trie and per-connection tracking grow with itGrowth without matching connection growth (subscription leak)
/connz pending_bytes, /routez pending_sizeBuffered messages sit in RSS before slow-consumer disconnectsSustained high pending; any sustained route pending
/varz slow_consumers (and slow_consumer_stats where available)The consequence of buffering; routes/gateways have cluster-wide blast radiusAny positive rate; non-zero routes/gateways breakdown
/jsz memory, storage, stream/consumer countsJetStream metadata cache and Raft state live in process memoryGrowth in asset counts or cache uncorrelated with traffic
/varz uptimeDetects the OOM restart itselfUnexpected resets; more than 3 restarts in 30 minutes
Goroutine count (pprof/metrics port)Leaked goroutines hold stack memory in RSSGrowth past ~2x connections with flat connections

Fixes

If it is the GC sawtooth (most common outcome)

Nothing is broken. Change the alert, not the server. Alert on the trend of RSS (a regression slope over hours) and on RSS as a ratio of the container limit, never on instantaneous peaks. Peaks near 2x baseline are the collector doing its job.

If the peaks themselves are close enough to the limit to risk OOM, you can trade CPU for memory by lowering GOGC (default 100), which makes GC run sooner and caps how far the heap doubles. In containers, setting GOMEMLIMIT gives the collector a target to stay under; be aware this is a soft limit, and RSS can still exceed it because of non-heap memory like mmap’d files and stacks.

If it is slow-consumer buffering

The memory is a symptom; the consumer is the fault. Identify the offenders via /connz?sort=pending, fix or scale the slow application, and check the slow-consumer breakdown: if routes or gateways are the slow ones, you have an inter-server problem with much higher blast radius. See the connection churn guide in Related guides for the reconnect-disconnect spiral that follows.

If it is subscription or connection growth

For subscriptions: find the client that subscribes without unsubscribing (a common reconnect-handler bug that re-subscribes on every reconnect without cleaning up). For connections: the fix is on the client side, such as connection pooling or fixing a connection leak. Do not raise memory limits to accommodate a leak.

If it is JetStream-driven

Check whether stream and consumer counts are growing (metadata cache and Raft state scale with them) and whether storage limits are set sanely. If a specific stream’s in-memory footprint is the problem, review its retention, replica count, and whether it needs to exist at all. Changes to JetStream storage limits may require a restart to take effect, so plan accordingly.

If it is a genuine leak

Capture the evidence first: goroutine profile, RSS trend, and the correlation data showing no workload driver. Then check your nats-server version against release notes for known memory regressions. A rolling restart (one node at a time in a cluster) buys time but is not a fix. If you upgrade to escape a known leak, verify the target version specifically addresses it.

Prevention

  • Size with headroom for GC and filesystem cache. Keep peak RSS under roughly 70% of the memory available to the process. The remaining 30% absorbs the GC sawtooth peaks and the page cache JetStream file storage depends on for read performance. A container limit set at “what the server uses at baseline” guarantees OOM kills at the first GC peak. At minimum, leave 20% between peak RSS and the limit.
  • Alert on slope, not level. A linear regression on mem over the last hour gives you runway estimation: (limit - current_rss) / growth_rate. This pages you days early on a real leak and never fires on GC teeth.
  • Alert on the ratio, not absolutes. mem / container_limit at 80% is meaningful across environments; “mem > 2 GB” is not.
  • Trend the drivers alongside RSS. Connections, subscriptions, slow consumers, and JetStream asset counts on the same dashboard turn “memory is up” into “memory is up because subscriptions doubled” in one glance.
  • Ignore the first 5-30 minutes after restart. Cold-start memory and CPU are not representative: caches are empty, subscription tables are rebuilding, and JetStream is recovering. Do not tune against warmup numbers.

How Netdata helps

  • Netdata polls /varz on the monitoring port and charts mem, connections, subscriptions, and slow consumers per second, so the GC sawtooth is visible as a sawtooth instead of being flattened into a misleading average by a 60-second scrape interval.
  • Per-second resolution lets you overlay the RSS floor against connection and subscription trends in one view, which is the exact correlation test from the diagnosis steps above.
  • Netdata’s ML anomaly detection runs per metric, so a rising RSS trough that deviates from learned behavior flags as anomalous while routine GC peaks do not.
  • Uptime monitoring catches the OOM restart itself, and correlating the restart timestamp against the RSS curve confirms whether memory was the cause.
  • Because Netdata also collects host-level memory and cgroup metrics, you can compare nats-server RSS against the container limit and host page cache on the same dashboard, which is how you spot the mmap-reclaimable portion without SSHing in.