NATS subscription leak: unbounded subscription growth and subject-trie bloat

The symptom is a number that only goes up. You open /varz on a NATS server and subscriptions is higher than it was yesterday, higher than it was last week, climbing for days while connections stays flat. Memory follows the same slope. Nothing is erroring. No slow consumers, no restarts, no complaints from applications. The server just keeps getting heavier.

That is a subscription leak: an application (or a bug) is registering subscriptions faster than it removes them. Every subscription lives in the server’s subject trie and in per-connection tracking state. Left alone, the leak consumes memory, slows subject matching, and in a cluster propagates interest across routes so that one leaky client inflates the subject trie on every server.

What this means

NATS routes messages through an in-memory subject trie (the Sublist). Each subscription inserts nodes into that trie, and each connection carries its own subscription tracking state. Wildcard subscriptions (* for one token, > for multi-token) add branching to the trie, so pathological wildcard patterns cost more than plain literal subjects. The trie is efficient, but it is not free: every subscription consumes memory, and every published message is matched against the trie, so a bloated trie adds CPU cost to the hot path of every message.

Three properties make this failure mode nasty:

1. It is silent. Subscribing is a normal, successful operation. The server does not log or reject a subscription just because a client already has 200,000 of them (unless you set a limit; see Fixes).
2. It survives reconnects in a disguised form. When a leaky client disconnects, its subscriptions are cleaned up, but a buggy auto-resubscribe loop recreates them immediately, so the count dips and resumes climbing.
3. It spreads. In a cluster, interest propagation advertises subscriptions across routes. A leak on one server inflates routing state on all of them.

flowchart TD
    A[Client subscribes in a loop or without unsubscribe] --> B[subscriptions counter climbs in /varz]
    B --> C[Subject trie nodes accumulate]
    B --> D[Per-connection tracking state grows]
    C --> E[RSS grows without GC recovery]
    C --> F[Subject matching CPU rises]
    B --> G[Interest propagates across routes]
    G --> H[Peer servers' routing state inflates too]

One nuance before you assume the worst: in a cluster, /varz subscriptions includes remote interest. Remote subscriptions from cluster peers show up as binary interest, roughly one entry per subject per remote server, so a clustered server will always report more subscriptions than its local clients hold. Compare like for like: trend over time on the same server, and sanity-check against per-connection counts.

Common causes

A note on the last row: certain older server versions had genuine counting bugs. Notably, v2.2.2 double-counted subscriptions in /varz on every request (fixed the same week in PR #2172), and v2.0.0 had incorrect /subsz counts around remote interest. If the global count climbs but per-connection subscription counts are stable, verify the version before blaming an application.

Quick checks

All of these are read-only against the monitoring port (default 8222).

# 1. Get the subscription count. This is the only number you need for trending.
curl -s http://localhost:8222/varz | jq .subscriptions

# 2. Compare against connection counts. Growth without connection growth = leak.
curl -s http://localhost:8222/varz | jq '{connections, total_connections, subscriptions, mem, uptime}'

# 3. Find the connections holding the most subscriptions.
curl -s "http://localhost:8222/connz?subs=1&limit=20" \
  | jq '[.connections[] | {cid, name, ip, subscriptions}] | sort_by(.subscriptions) | reverse | .[:10]'

# 4. Sample the count twice to compute a leak rate.
curl -s http://localhost:8222/varz | jq .subscriptions; sleep 60; curl -s http://localhost:8222/varz | jq .subscriptions

# 5. In a cluster, compare across servers to find the origin.
for h in nats-1 nats-2 nats-3; do echo -n "$h: "; curl -s http://$h:8222/varz | jq .subscriptions; done

Two hard safety rules:

• Never fetch /subsz?subs=1 on a server with a high subscription count. The full subscription list can lock the server. Use the COUNT from /varz, never the LIST from /subsz. If you must inspect individual subjects, inspect them per-connection via /connz?subs=1 with limit and offset, and even that is expensive at scale.
• Treat /connz with subs=1 as a diagnostic you run a few times during an incident, not something you scrape every 10 seconds.

If you have the NATS CLI tooling installed, nats-top -n 1 -sort subs shows the connections with the largest subscription counts without touching /subsz.

How to diagnose it

1. Confirm the trend. Pull /varz subscriptions at two or three points over a few minutes. A flat line with deployment spikes is normal. A monotonic climb at a steady rate is a leak. Compute the rate: (count2 - count1) / interval.

2. Rule out connection growth. Check connections and total_connections over the same window. If connections is flat while subscriptions climbs, one or more existing clients are accumulating subscriptions. If total_connections is climbing fast while connections is stable, you have churn, and the leak may be a resubscribe bug: each reconnect adds subscriptions without cleaning up. See NATS connection churn for that pattern.

3. Identify the culprit connection. Use /connz?subs=1 with a limit and sort client-side by the subscriptions field (step 3 above). The leaking connection usually stands out by orders of magnitude. Note its cid, name, and source IP.

4. Inspect the subject strings. On the offending connection, look at the actual subjects. The pattern tells you the bug class:

   • Unique token per subscription (foo.<uuid>, events.user.<id>): the app is subscribing per entity or per request and never unsubscribing.
   • _INBOX-style names piling up: request-reply traffic creating subscriptions per request instead of using a single wildcard inbox.
   • The same literal subject thousands of times on one connection: a resubscribe loop that re-registers on every reconnect attempt.

5. Check the blast radius. In a cluster, query /varz subscriptions on every server. If all servers show growth, interest has propagated; the origin server is the one where the culprit connection lives. Also check memory (/varz mem) on each peer to see how much the propagated interest is costing them.

6. Rule out a counting artifact. If per-connection counts do not add up to anywhere near the /varz total, check the server version against the known counting bugs mentioned above. On a clustered server, remember that remote interest legitimately inflates the total.

7. Quantify the resource impact. Watch /varz mem and cpu alongside the subscription count. Memory climbing with the count confirms trie and tracking-state bloat. CPU rising while message rate is steady confirms matching cost.

Metrics and signals to monitor

The single most useful alert is rate-of-change on subscriptions relative to connections. Absolute thresholds break across deployments; tens of thousands of subscriptions is normal for one system and catastrophic for another. A deployment will temporarily double the count (blue-green overlap) and that is fine, which is another reason to alert on sustained slope over hours, not instantaneous spikes.

Fixes

Fix the client bug (the real fix)

Everything else is containment. The common shapes:

• Subscribe once, demux in the handler. If the app subscribes per entity (orders.<id>), replace with a wildcard (orders.*) plus routing logic in the callback, or a shared subscription with a dispatch table.
• Unsubscribe deterministically. Every subscribe needs a matching unsubscribe (or drain) tied to the same lifecycle as the work it serves. If subscriptions are created per request, they must be torn down per request, or replaced by a single wildcard inbox subscription.
• Fix the resubscribe path. Client libraries auto-resubscribe after reconnect by replaying the subscription set. If application code also re-subscribes in a connect callback, you get duplicates on every reconnect. Subscribe in exactly one place.

Set a per-connection limit (containment)

The max_subscriptions server configuration option limits how many subscriptions a single client (or leafnode) connection may hold. The default is 0, meaning unlimited. Setting a real ceiling, sized well above legitimate per-client usage, converts a silent memory leak into a loud client-side error the moment the buggy app crosses the line. That is a much better failure mode than an OOM-killed server.

Tradeoff: a legitimate high-subscription client (an aggregator, a gateway-style service) will hit the wall too, so size the limit from observed baselines, not guesses.

Restart or disconnect (triage only)

Killing the offending connection frees its subscription state immediately. The count drops, memory follows after GC. But if the client bug is still live, the leak resumes on reconnect, sometimes faster. Disconnect the client only to buy time while the fix ships, and expect the count to start climbing again. Do not restart the server as a first response: in a cluster the client will reconnect to a peer and continue leaking there, and you lose the evidence.

Reduce wildcard cardinality

If the bloat comes from overlapping wildcard patterns rather than raw count, consolidate the subject namespace. Many overlapping wildcard subscriptions across high-cardinality subject trees create far more trie branching than the same number of literal subscriptions. Flattening the namespace or replacing overlapping wildcards with fewer, broader ones reduces both memory and matching cost.

Prevention

• Alert on slope, not level. Track subscriptions per second over a rolling window and alert on sustained positive rate-of-change decoupled from connection growth. Gate out deploy windows if blue-green spikes are noisy.
• Baseline per-application subscription counts. Know how many subscriptions each service should hold. A connection with 10x its baseline is a leak in progress, long before server-level totals look scary.
• Set max_subscriptions on every production server as a backstop, sized from observed legitimate maximums.
• Add the count to your monitoring checklist alongside connections, memory, and slow consumers. Subscription count tracking is a Level 3 maturity signal in the NATS monitoring maturity model, and this incident is exactly why.
• Never automate /subsz collection. Keep it out of scrapers and cron jobs. Count from /varz, inspect per-connection with /connz when actively debugging.
• Test reconnect behavior in staging. Bounce a client repeatedly and watch whether its subscription count returns to baseline or ratchets upward. Resubscribe bugs only show up across reconnect boundaries.

How Netdata helps

• Netdata’s NATS collector polls /varz continuously, so subscriptions, connections, and total_connections are charted together at high resolution. The leak signature, a rising subscription line over a flat connection line, is visible at a glance.
• Rate-of-change alerting on the subscription count catches slow leaks that take days to matter. An absolute threshold would either never fire or fire on every deployment.
• Correlating subscriptions with process RSS (mem) and CPU on the same dashboard confirms the bloat is real resource consumption, not a counting artifact, and shows the slope you need for runway estimation.
• Uptime tracking alongside the count exposes restarts that reset the counter and disguise a long-running leak as a fresh baseline.
• In a cluster, per-server subscription charts side by side show propagation: all servers climbing together means interest is spreading from one origin, which narrows the search to the server holding the culprit connection.

Related guides

• NATS monitoring checklist: the signals every production server needs
• NATS monitoring maturity model: from survival to expert
• How NATS actually works in production: a mental model for operators
• NATS connection churn: a stable connection count hiding constant reconnects
• NATS connection storm: reconnect thundering herd after a network event
• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS file descriptor exhaustion: too many open files and the ulimit cliff
• NATS Maximum Connections Exceeded: new clients rejected at the max_connections wall
• NATS pending bytes growing: catching a slow consumer before it is disconnected
• NATS no responders available for request: request-reply into the void
• NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check
• NATS server not responding: healthz failing and the process down or hung