NATS goroutine leak: connection cleanup that never completes

The symptom is a steady, one-directional climb. Goroutine count goes up day after day, RSS follows it, and nothing on the traffic side explains it: client connections are flat, routes are flat, throughput is flat. Then, weeks later, the server gets OOM-killed or starts showing GC and scheduling overhead that has nothing to do with message load.

A NATS goroutine leak is connection (or subsystem) cleanup that never completes. A goroutine is spawned to handle a connection, a timer, or a Raft loop; the work ends, but the goroutine never exits. It sits blocked on a channel receive that will never fire, a lock that will never be released, or a retry loop with nothing to retry. Each one is cheap individually, so the leak is invisible until it is not: at roughly 4-8KB of stack per goroutine, 100k leaked goroutines is about 400-800MB of RSS.

The reason most teams find this late is instrumentation. Goroutine count is not on the standard NATS monitoring port: /varz gives you connections, memory, and throughput, but not goroutines. The authoritative source is the Go pprof endpoint (/debug/pprof/goroutine), which is only available if profiling was enabled in the server config. Many teams enable neither pprof nor metrics until they are already in the incident.

What this means

NATS is a Go server, and its concurrency model is predictable. Every client TCP connection gets dedicated read and write goroutines. Routes, gateways, and leaf node connections get the same treatment. On top of that sits a fixed set of server internals, plus JetStream Raft goroutines if persistence is enabled. The expected steady-state count is roughly:

goroutines ~ 2 x (clients + routes + gateways + leafs)
           + server internals (~50-100)
           + JetStream Raft goroutines (if enabled)

That formula is the whole diagnostic key. Goroutine count should track connection count. When connections are flat and goroutines grow, something is being created without being destroyed. Three mechanisms produce that pattern:

1. Cleanup that never completes. A connection closes, but its goroutines stay blocked on a channel or lock, so they never return.
2. Stuck goroutines. A goroutine is parked on a receive from a closed or abandoned channel, or on a mutex held by another stuck goroutine. This is where client library bugs live: a Next() call that blocks forever after the server goes away, or a Drain() that deadlocks against an in-flight Next().
3. Accumulating background work. Raft-related goroutines or timer loops that pile up instead of being reused, common after repeated elections or reconnect churn.

The blast radius is memory and scheduler overhead first, then GC pressure, then OOM. The server usually keeps routing messages correctly for a long time, which is why this class of bug survives to production.

flowchart TD
  A[Goroutine count rising] --> B{Connections also rising?}
  B -->|Yes| C[Not a leak: load growth or churn.
Check total_connections delta]
  B -->|No| D[Leak: capture a goroutine dump]
  D --> E{Where are the stacks parked?}
  E -->|Connection read/write paths| F[Server-side cleanup not completing
or route/gateway stuck]
  E -->|Client library: Next, Drain, reconnect loops| G[Client bug or app leak
new connection per request]
  E -->|Raft loops and timers| H[JetStream Raft accumulation
after election churn]
  F --> I[Upgrade server, restart to reclaim]
  G --> J[Fix app lifecycle, upgrade client]
  H --> K[Upgrade server, review Raft health]

Common causes

Version note: specific stuck-goroutine bugs are version-dependent. Known examples from upstream issues include nats.go Next() blocking indefinitely after server shutdown, and a Drain() deadlock when Next() is already blocked waiting for messages, in older client releases. Server-side, several releases have shipped fixes for goroutine and timer retention in reconnect loops, healthcheck monitors, and interest-tracking goroutines. If your dump matches one of these patterns, the fix is usually an upgrade. Check the release notes for your exact server and client versions before concluding you have a novel bug.

Quick checks

The curl checks are read-only and cheap. The full dump (step 6) and SIGQUIT (step 7) are not; read the warnings before running either.

# 1. Goroutine count via pprof (if enabled with the prof_port config option, typically 6060).
#    The first line of the debug=1 output is the total.
curl -s http://localhost:6060/debug/pprof/goroutine?debug=1 | head -1

# 2. Connection counts to compare against the baseline formula
curl -s http://localhost:8222/varz | jq '{connections, routes, leafnodes, mem, uptime}'

# 3. Route and gateway counts for the full baseline
curl -s http://localhost:8222/routez | jq '.num_routes'
curl -s http://localhost:8222/gatewayz | jq '{out: (.outbound_gateways | length), in: (.inbound_gateways | length)}'

# 4. Connection churn: is the cumulative counter racing while current count is flat?
curl -s http://localhost:8222/varz | jq '{active: .connections, lifetime_total: .total_connections}'

# 5. Full goroutine dump with stack traces (EXPENSIVE: can pause the scheduler
#    and spike latency on a busy server. Run once, off-peak if possible.)
curl -s http://localhost:6060/debug/pprof/goroutine?debug=2 > /tmp/nats-goroutines.txt

# 6. Fallback if pprof is not enabled: SIGQUIT dumps all goroutine stacks to stderr.
#    WARNING: on a default Go runtime, SIGQUIT dumps the stacks and then TERMINATES
#    the process. Treat this as a disruptive action, not a probe. Verify how your
#    nats-server build handles SIGQUIT before using it, and capture stderr from the
#    service manager (journald, container logs) since the dump goes there.
kill -QUIT $(pgrep nats-server)

The server registers handlers only for SIGINT, SIGTERM, SIGHUP, SIGUSR1, and SIGUSR2; SIGQUIT follows default Go runtime behavior (dump stacks to stderr, then terminate the process).

For a client-side leak, the same pprof endpoints work against your application if it exposes them, and SIGQUIT works on any Go client process, with the same terminate-after-dump caveat. For deployments with the system account set up, the NATS CLI can also request a goroutine profile from a server remotely. (the subcommand is nats server request profile goroutine)

A note on Prometheus: if you scrape the prometheus-nats-exporter (default port 7777), its go_goroutines series typically describes the exporter process itself, not the NATS server. Do not use it as the server goroutine count unless you have confirmed your exporter re-exports the server’s runtime stats. The pprof endpoint is authoritative.

How to diagnose it

1. Confirm the trend, not the point value. A single goroutine count is meaningless. You need the slope over hours or days. If you only have pprof and no time series, take three samples an hour apart and compare. Flat connections plus rising goroutines is the leak signature.

2. Compute the baseline. Pull connections, routes, gateway count, and leafnodes from /varz, /routez, /gatewayz. Apply the formula: 2 x (sum of all connection types) + server internals + Raft goroutines. If the actual count is 50% or more above the estimate and still climbing, you have a leak.

3. Check for churn masquerading as growth. If total_connections is racing while connections is flat, clients are flapping. High churn amplifies small per-connection leaks: each reconnect cycle that fails to fully clean up adds a few permanent goroutines. See the connection churn and connection storm guides if this is your pattern.

4. Capture a full dump once. Use debug=2 (or SIGQUIT only if pprof is off and you accept the restart risk). On a server with 100k+ goroutines this file is large and generation stalls the process briefly. Warn your team before running it in production.

5. Group the stacks. Do not read the dump linearly. Count goroutines by their top frames:

# Histogram of the top stack frame each goroutine is parked in
awk '/^goroutine /{getline; print}' /tmp/nats-goroutines.txt | sort | uniq -c | sort -rn | head -30

The leaked population shows up as hundreds or thousands of goroutines with identical stacks, all blocked in the same place: a channel receive, sync.Mutex.Lock, select in a reconnect or iterator loop, or a Raft apply/ticker path.

6. Map the dominant stack to a subsystem. Connection read/write loops point to server-side cleanup. Stacks in client library iterators (Next(), drain, subscription cleanup) point to the client. Raft ticks and apply loops point to JetStream. One dominant stack signature, thousands of copies: that is your leak.

7. Correlate with history. Line the growth curve up against deploys, server restarts, network events, and Raft elections. A leak that steps up after every client deploy is application-side. One that steps up after every partition event is server or route side.

8. Check the client processes too. If the server is clean but clients show the same growth pattern, the leak is in the client library or the application’s connection lifecycle. The most common application bug is trivial: opening a new connection per request or per goroutine and never calling Close() (or Drain() then Close()), leaving old connections and their goroutines alive until timeout.

Metrics and signals to monitor

Fixes

Application connection leak

Reuse connections. A NATS connection is designed to be long-lived and multiplexed; one per process, or a small pool, is the norm. Audit for code paths that connect per request, per message, or per goroutine, and make sure every path calls Close() on shutdown, ideally Drain() then Close() so inflight messages finish first. Tradeoff: pooling adds a little coordination complexity, but it eliminates the entire class of leak.

Client library stuck goroutines

If the dump shows goroutines parked in Next(), drain cleanup, or reconnect loops, you are likely on an affected client version. Upgrade the client library and re-test your shutdown path explicitly: kill the server mid-consume and verify the client’s consumer goroutines exit. If you cannot upgrade immediately, the only reliable recovery for a process full of permanently blocked iterator goroutines is a process restart, so gate it behind supervision that detects stalled consumption.

Server-side cleanup or route/gateway leaks

Check release notes for your server version against goroutine, timer, and cleanup fixes; this class of bug is almost always resolved by upgrading, not configuring. If the leaked goroutines are tied to route or gateway churn, also address the underlying instability (network, RTT, flapping) or the leak will keep re-accumulating after every event.

Raft goroutine accumulation

Treat the election storm as the primary problem: stabilize inter-node latency, disk I/O, and CPU headroom so Raft stops churning, then upgrade the server. Restarting to reclaim the goroutines without fixing the elections just resets the clock.

Reclaiming memory now

The only way to reclaim goroutine stacks in a running Go process is for the goroutines to exit, and leaked ones never do. A restart is the reclamation, and it is legitimate here once you have captured a dump and identified the cause. Do it rolling in a cluster, and capture the dump first: after the restart the evidence is gone.

Prevention

• Collect the goroutine count permanently. Scrape the pprof endpoint on a schedule, or confirm a metrics path that reflects the server’s runtime, so the count is a time series rather than an incident-time surprise. Trend it against the connection baseline and alert on divergence, not on absolute values.
• Alert on the ratio, not the count. Goroutines per connection that climbs steadily is the actionable signal. Absolute thresholds break across deployment sizes.
• Load-test your shutdown path. In staging, kill servers while clients are mid-consume and verify client goroutine counts return to baseline. This is where stuck Next() and drain deadlocks show up.
• Correlate memory with goroutine count in dashboards. RSS climbing in lockstep with goroutines, without connection growth, is the earliest cheap confirmation.
• Track server and client versions against upstream leak fixes. Goroutine cleanup bugs are recurring enough that staying current is real prevention.

How Netdata helps

• Netdata’s NATS collector tracks /varz connection counts, total_connections, mem (RSS), routes, and slow consumers at per-second resolution, giving you the baseline half of the leak equation without manual polling.
• Pairing it with a goroutine-count series (scraped pprof, or the Go runtime metrics if you have confirmed they reflect the server) lets you chart goroutines against connections on the same timeline, which is the exact divergence that defines this leak.
• RSS trend views make the 4-8KB-per-goroutine memory cost visible early, well before the OOM kill, and correlate it with uptime so you can tell leak growth from cold-start noise.
• Connection churn (total_connections delta vs. stable active count) surfaces the flap pattern that turns a small per-connection cleanup bug into a fast leak.
• JetStream signals such as meta cluster leader changes help you connect Raft instability events to steps in the goroutine growth curve.

Related guides

• NATS connection churn: a stable connection count hiding constant reconnects
• NATS connection storm: reconnect thundering herd after a network event
• NATS file descriptor exhaustion: too many open files and the ulimit cliff
• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS route RTT high: inter-server latency that triggers Raft elections
• NATS gateway disconnected: cross-cluster traffic cut in a supercluster
• NATS JetStream consumer lag growing: falling behind the stream
• NATS consumer stalled at MaxAckPending: delivery stops until messages are acked
• NATS JetStream redelivery loop: num_redelivered climbing and messages reprocessed
• NATS JetStream consumer stopped receiving messages: the diagnostic tree
• NATS JetStream AckWait tuning: matching the ack timeout to processing time
• NATS context deadline exceeded: JetStream publish and request timeouts