NATS high CPU: subject matching, TLS, GC, and the container GOMAXPROCS trap

A NATS server that is pegged on CPU usually looks worse than it is, or better than it is, depending on what you are measuring. The cpu field in /varz is process CPU where 100 means one full core, so on a 16-core host a value of 800 is only 50% busy. Operators regularly page themselves on a number that is not actually saturation, or dismiss real saturation because the number “only” reads 300 on a 2-core container.

The useful frame is this: NATS CPU should scale roughly linearly with message rate. When CPU climbs without a matching rise in in_msgs/out_msgs, the cost is coming from something other than routing: TLS handshakes, Go garbage collection, pathological wildcard matching, Raft consensus churn, or the Go runtime fighting a container CPU limit it does not know about. That last one, the GOMAXPROCS trap, is the most common cause of “NATS is slow but nothing looks loaded” in Kubernetes.

What this means

NATS spends CPU in a small number of places:

• Subject matching: every inbound message is matched against the subscription trie and fanned out. Cost scales with message rate and subscription complexity.
• TLS: handshake cost per new connection, plus per-byte encryption. On TLS-enabled servers with high connection churn, handshakes dominate.
• Go GC: garbage collection consumes CPU proportional to allocation rate and heap churn. GC shows up as CPU spikes followed by drops.
• JetStream: indexing, compaction, and (in clustered mode) Raft consensus. Raft election storms burn CPU on all nodes simultaneously while blocking writes.
• Runtime mismatch (GOMAXPROCS): not a cost center itself, but a scheduler sized for the host instead of the container multiplies the cost of everything above.

The diagnostic problem is that all of these present as “CPU high”. The split comes from correlating CPU with message rate, connection churn, and cluster state.

flowchart TD
  A[CPU high on nats-server] --> B{Message rate also up?}
  B -- yes --> C[Normal load: matching + fan-out
check TLS share, plan capacity]
  B -- no --> D{Connection churn high?}
  D -- yes --> E[TLS handshake cost
or reconnect storm]
  D -- no --> F{RSS sawtoothing / GOMEMLIMIT set?}
  F -- yes --> G[Go GC pressure]
  F -- no --> H{Clustered JetStream?}
  H -- yes --> I[Raft elections / meta flapping]
  H -- no --> J[Wildcard matching pathology
or wrong GOMAXPROCS]

Common causes

Quick checks

All read-only. Run against the monitoring port (default 8222).

# 1. Read CPU correctly: process CPU and core count together
curl -s http://localhost:8222/varz | jq '{cpu, cores}'

# 2. Throughput: is message rate actually up?
curl -s http://localhost:8222/varz | jq '{in_msgs, out_msgs, in_bytes, out_bytes}'
# Take two snapshots 10s apart and compute rates.

# 3. Connection churn: stable count but fast-climbing total means churn
curl -s http://localhost:8222/varz | jq '{connections, total_connections}'

# 4. Slow consumers and their type breakdown
curl -s http://localhost:8222/varz | jq '{slow_consumers, slow_consumer_stats}'

The slow_consumer_stats field has clients/routes/gateways/leafs breakdowns and is available since nats-server v2.10.0.

# 5. Memory: sawtooth indicates GC activity
curl -s http://localhost:8222/varz | jq .mem

# 6. JetStream meta cluster: is the leader flapping?
curl -s http://localhost:8222/jsz | jq '.meta_cluster | {leader, replicas: [.replicas[]? | {name, current, offline, lag}]}'

# 7. JetStream API health: errors and inflight rise during elections and disk stalls
curl -s http://localhost:8222/jsz | jq '{api_total: .api.total, api_errors: .api.errors, api_inflight: .api.inflight}'

# 8. CPU steal on the host (oversubscribed VM check)
vmstat 1 5

# 9. Effective CPU visibility inside the container
nproc                          # what Go sees by default
cat /sys/fs/cgroup/cpu.max     # cgroup v2 limit, or:
cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us /sys/fs/cgroup/cpu/cpu.cfs_period_us   # cgroup v1

# 10. Capture a CPU profile for the definitive answer (see diagnose step 5)
nats server request profile cpu --timeout 30s

How to diagnose it

1. Normalize the number. Compute cpu / (cores * 100) from /varz. On a container limited to 2 cores on a 64-core host, /varz may report cores as the host count, which makes the utilization math lie in the optimistic direction. Cross-check with nproc inside the container and the cgroup quota files above. Sustained utilization above ~70% is capacity-planning territory; above 90% for more than 5 minutes is saturation.

2. Correlate with message rate. Compute in/out message rates from two /varz snapshots. If CPU and message rate rose together, this is load, and the question is whether the load is expected. If CPU rose and message rate did not, you are in GC, TLS-churn, Raft, matching-pathology, or GOMAXPROCS territory.

3. Check connection churn and TLS. A stable connections count with a fast-rising total_connections means clients are cycling. Each new TLS connection pays a handshake, and a mass reconnect event (network blip, deploy, LB flap) can saturate all cores on handshakes alone for tens of seconds before settling. The signature is a CPU spike tightly correlated with a connection-count drop-and-recovery. See NATS connection storm and NATS connection churn for the reconnect side.

4. Rule GC in or out. Watch /varz mem over several minutes. A sawtooth (climb, sharp drop, repeat) is the Go GC at work and is normal at moderate amplitude. If the sawtooth is tight and fast, or if GOMEMLIMIT is set below the live heap, the GC can run nearly continuously and eat a large fraction of CPU. Check the environment for GOMEMLIMIT and GOGC.

5. Profile, do not guess. Enable profiling via prof_port in the server config and hit the pprof endpoint, or use the NATS CLI: nats server request profile cpu --timeout 30s. The profile tells you directly whether the time is in the sublist/matching code, TLS, GC, or Raft. On a saturated server this is the fastest way to stop arguing about causes. Profiling adds overhead while it runs; 30 seconds is a reasonable window.

6. Check Raft if JetStream is clustered. A meta leader that keeps changing, replicas reporting current: false or non-zero lag, rising api.errors, and high api.inflight together indicate consensus churn. CPU saturation can itself cause this: a leader too busy to process heartbeats triggers elections, and each election adds CPU, disk, and network cost, which can cascade into a storm. Check host-level steal time too: on oversubscribed VMs, steal is a classic election-storm trigger.

7. Check the GOMAXPROCS trap explicitly. Compare what Go sees (nproc) with the cgroup limit. If the container limit is 2 cores and the host has 64, the Go runtime sizes its scheduler for 64 and thrashes: goroutines oversubscribe the 2 available cores, GC workers multiply, and tail latency and CPU both climb while throughput stays flat. Details in the fixes below.

Metrics and signals to monitor

Fixes

Right-size for real load

If CPU tracks message rate and you are simply hot, the fix is capacity: spread connections and subscriptions across more cluster members, reduce fan-out where possible, and keep utilization under ~70% sustained. Do not restart the server as a first move; on JetStream, restarts trigger recovery work and client reconnect storms that temporarily make CPU worse.

Reduce TLS handshake cost

• Fix the churn source. Handshake storms are a symptom of reconnect storms. Stabilize the network path, LB behavior, or client reconnect jitter first.
• Keep connections long-lived. Clients should reuse connections, not open per-operation.
• Confirm TLS is the cost with a profile before changing cipher or cert configuration. If handshakes dominate, staggering client reconnects (jittered backoff in the client) flattens the spike.

Fix pathological subject design

Wildcard subscriptions (*, >) over highly diverse subjects defeat the fast path in the sublist and force repeated matching work. Symptoms: high CPU at modest message rates, with a large subscription count. The fix is subject hygiene: flatten hot subjects, avoid deep wildcard fan-in on high-rate topics, and audit for subscription leaks (subscriptions growing without connection growth). Publishing on many distinct subjects matched by a single broad wildcard is measurably more expensive than a flat subject space.

Tune Go GC

• If GOMEMLIMIT is set, verify it is comfortably above the live heap. Set too low, the GC runs almost continuously and shows up as a permanent CPU tax. In Kubernetes, a common pattern is GOMEMLIMIT at roughly 80% of the container memory limit; the official NATS Helm chart documentation uses this same rule of thumb.
• Give the heap headroom: more headroom means fewer GC cycles. If RSS is pinned near the limit, raise the limit or reduce per-connection buffering before touching GC knobs.
• Confirm GC in the CPU profile before tuning; GC showing high in the profile plus a fast sawtooth in mem is the combination that justifies it.

Fix the container GOMAXPROCS trap

This is the highest-leverage fix for containerized deployments:

• The Go runtime historically defaulted GOMAXPROCS to the host core count, ignoring cgroup CPU limits. A container limited to 2 cores on a 64-core host runs a 64-way scheduler on 2 cores.
• Set GOMAXPROCS explicitly to the container’s CPU limit (or use an automaxprocs-style mechanism where it works; note that quota detection cannot infer a limit when the task has no CPU quota and cpu.cfs_quota_us is -1).
• Go 1.25 made the default cgroup-aware on Linux when a CPU limit is set. If you set only a CPU request and no limit, behavior is unchanged from the old default. The cgroup v1 non-root-mount path bug was fixed in Go 1.26; Go 1.25 releases before that fix may miss CPU limits on affected platforms.
• Align CPU requests and limits for latency-sensitive NATS. Throttling from a tight CFS quota looks like CPU saturation with extra latency and can trigger Raft elections.

Calm a Raft election storm

1. Remove the resource pressure first: CPU saturation, disk I/O latency on the JetStream storage path, and CPU steal are the usual roots. Network-attached storage with variable latency is a repeat offender.
2. Reduce load where possible (pause non-critical publishers) so leaders can process heartbeats.
3. If the storm is self-sustaining, a rolling restart, one node at a time, can break the cycle. This is disruptive: it forces client reconnects and JetStream recovery on each node. Do it only after the underlying pressure is addressed, or the storm resumes.

Prevention

• Pin runtime resources in containers: set GOMAXPROCS (or run a Go version with cgroup-aware defaults and set CPU limits), set GOMEMLIMIT sensibly, and size CPU requests and limits together.
• Baseline CPU per message rate: know your normal slope so “CPU up, rate flat” is immediately visible as an anomaly.
• Monitor leading indicators, not just saturation: connection churn, slow consumer breakdown by type, meta leader stability, api.inflight, and host steal time all move before CPU pegs.
• Design subjects deliberately: keep high-rate subjects flat, avoid broad wildcards over diverse namespaces, and alert on subscription-count growth.
• Put JetStream on local low-latency storage and watch steal on shared VM hosts; Raft heartbeat health depends on both.
• Keep profiling available (prof_port configured, CLI access working) so a CPU incident is a 5-minute diagnosis, not a guessing game.

How Netdata helps

• Netdata collects /varz CPU alongside core count, throughput, connections, slow consumers, memory, and subscriptions at per-second resolution, so the “CPU up but message rate flat” divergence is visible on one dashboard instead of two terminals.
• The slow-consumer breakdown by type (clients, routes, gateways) sits next to connection churn, which separates TLS handshake storms from matching load quickly.
• Memory sawtooth frequency next to CPU makes GC pressure recognizable without pulling a profile first.
• JetStream aggregates (API errors, storage, stream/consumer counts) on the same timeline as CPU expose election-driven CPU cost.
• Host-level metrics (per-core utilization, steal time, cgroup throttling) correlate directly with the NATS process view, which is exactly what the GOMAXPROCS and noisy-neighbor cases need.
• ML anomaly detection on CPU and throughput flags the slope change when matching cost drifts away from message rate, before users notice latency.

Related guides

• NATS JetStream AckWait tuning: matching the ack timeout to processing time
• NATS connection churn: a stable connection count hiding constant reconnects
• NATS connection storm: reconnect thundering herd after a network event
• NATS JetStream consumer lag growing: falling behind the stream
• NATS consumer stalled at MaxAckPending: delivery stops until messages are acked
• NATS JetStream redelivery loop: num_redelivered climbing and messages reprocessed
• NATS JetStream consumer stopped receiving messages: the diagnostic tree
• NATS context deadline exceeded: JetStream publish and request timeouts
• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS file descriptor exhaustion: too many open files and the ulimit cliff
• NATS gateway disconnected: cross-cluster traffic cut in a supercluster
• NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check