NATS slow consumer breakdown: clients vs routes vs gateways and blast radius

Your NATS alert fired: slow_consumers is incrementing. You open /varz, see a non-zero counter, and start hunting for a misbehaving client. That is the right move only some of the time. The aggregate slow_consumers counter mixes four very different kinds of events, and two of them mean your cluster fabric itself is backing up, not a single subscriber.

A client slow consumer costs you one subscriber’s message stream. A route or gateway slow consumer means inter-server forwarding is stalling, which can degrade message delivery for every account and every client on the cluster. Teams that monitor only the aggregate number page on the harmless case and sit on the severe one.

What this means

Every connection a NATS server writes to has a per-connection pending write buffer with dedicated read/write goroutines. When the receiving side cannot drain the buffer fast enough, pending bytes grow until the server flags the write deadline (write_deadline, default 10s). For ordinary client connections the default timeout policy closes the connection and drops its buffered core-NATS messages; routes, gateways, and leaf connections use retry policies in NATS Server 2.14.0 and later, so closure is no longer the universal default.

Routes, gateways, and leaf connections are internally just connections with their own buffers and goroutines. They can become slow consumers exactly like clients can. Since NATS 2.10, /varz exposes the breakdown directly:

"slow_consumers": 14,
"slow_consumer_stats": {
  "clients": 12,
  "routes": 2,
  "gateways": 0,
  "leafs": 0
}

Before 2.10, only the aggregate slow_consumers integer existed. On mixed-version fleets, older nodes will not report the breakdown and you will need /connz and /routez inspection to attribute events.

Blast radius by type:

flowchart TD
  SC[Slow consumer event] --> Q{Which counter incremented?}
  Q -->|clients| C[One subscriber affected]
  Q -->|routes| R[Cluster-wide forwarding stalls]
  Q -->|gateways| G[Cross-cluster delivery stalls]
  Q -->|leafs| L[Edge site affected]
  C --> C1[Fix or scale the consumer]
  R --> R1[Check routez pending_size and RTT]
  G --> G1[Check gatewayz and WAN link]
  L --> L1[Check leafz and edge connectivity]

Common causes

Quick checks

All read-only, safe to run during an incident. Assumes the monitoring port (default 8222) is enabled.

# 1. Get the breakdown - this is the triage starting point
curl -s http://localhost:8222/varz | jq '{slow_consumers, slow_consumer_stats}'

# 2. Identify worst client connections by buffered bytes
curl -s "http://localhost:8222/connz?sort=pending&limit=10" | \
  jq '.connections[] | {cid, name, ip, pending_bytes, subscriptions, rtt}'

# 3. Check route-level backpressure and latency
curl -s http://localhost:8222/routez | \
  jq '.routes[] | {rid, remote_id, ip, rtt, pending_size}'

# 4. Check gateway connections
curl -s http://localhost:8222/gatewayz | \
  jq '{outbound: (.outbound_gateways | keys), inbound: (.inbound_gateways | keys)}'

# 5. Per-account view: connection and subscription pressure
curl -s "http://localhost:8222/accountz?acc=APP" | jq '.account_detail'
# Per-account slow-consumer counters live in the /accstatz endpoint (per shard/account
# entries); /accountz?acc=NAME returns account config and limits, not slow-consumer counts

# 6. Supporting context: churn, memory, throughput
curl -s http://localhost:8222/varz | \
  jq '{connections, total_connections, mem, in_msgs, out_msgs}'

The rate of change matters, not the absolute number. slow_consumers and its sub-counters are cumulative since server start, so a static non-zero value is history. Snapshot twice, 30 to 60 seconds apart, and compare.

How to diagnose it

1. Read the breakdown first. If slow_consumer_stats.clients is the only counter moving, scope is one or more subscribers. If routes or gateways moved even once, treat it as a cluster-fabric event and escalate immediately.

2. Rule out the false positive. During rolling restarts, a rebooting node takes time to become available and digest gossip, so its peers can report it as a route slow consumer. A single event inside a deploy window is not actionable. Check whether the timing matches a restart before paging anyone.

3. For client events, name the offender. Use /connz?sort=pending and look for connections with large or growing pending_bytes. The name, ip, and subscriptions fields tell you which application and which subjects are involved. Confirm with a second poll: a one-off spike can be a burst, sustained growth is the real thing.

4. For route events, find the direction. /routez shows per-route pending_size and rtt. Pending growing toward one peer means that peer is not reading fast enough: check its CPU, memory, and GC pressure. Pending growing on all routes with elevated RTT points at the network between servers.

5. For gateway events, check the link. Confirm the gateway is still connected in /gatewayz, then look at the WAN path. Right after a gateway reconnects it runs in flood mode until interest-only mode converges, which can itself cause transient backpressure.

6. Check for the churn spiral. A disconnected slow consumer reconnects, resubscribes, immediately falls behind again, and gets disconnected again. The symptom is total_connections climbing fast while connections stays flat, with slow_consumer_stats.clients ratcheting up. The counter increments per event, so one looping client can inflate it dramatically.

7. Accept the observability gap. NATS does not track how many messages were dropped for a slow route or gateway connection; there is no such counter. pending_bytes shows backpressure building, but past that there is no server-side statement about delivery. If you need loss accounting, it has to come from application-level sequence tracking.

Metrics and signals to monitor

Alert on rate, not on the cumulative value, and alert on routes and gateways as separate, higher-severity rules than clients. With prometheus-nats-exporter, the breakdown is flattened to separate gauges. The default prefix is gnatsd (for backward compatibility), so the metrics are gnatsd_varz_slow_consumer_stats_clients, ..._routes, ..._gateways, and ..._leafs; deployments using --prefix nats replace that prefix with nats.

Fixes

Client slow consumers

The fault is on the consuming side, and the server is correctly enforcing backpressure. Real fixes, in order of preference:

• Fix the consumer’s processing loop. The common culprits are synchronous database writes in the message handler, GC pauses, serialization bottlenecks, and CPU starvation on the consumer host.
• Scale out with a queue group. Spread the subscription load across more consumer instances so no single connection has to keep up with the full rate.
• Catch it client-side. Client libraries expose an async error callback for slow consumer conditions (nats.ErrSlowConsumer in the Go client), and subscriber pending limits are configurable (default 65536 messages via SetPendingLimits). Catching backpressure in the client before the server disconnects is strictly better than discovering it via the server counter.
• Disconnect the offender as a pressure valve. If one slow subscriber is amplifying cluster route pressure, cutting it reduces fan-out load while you fix it.

Route and gateway slow consumers

• Overloaded peer: relieve the peer. Check its CPU, memory, and disk. Route backpressure toward one peer almost always means that peer is starved, not that the network is broken.
• Network degradation: elevated route RTT alongside pending growth points at the inter-server path. Cross-AZ or cross-region links degrading will show up here before anywhere else.
• Do not just raise write_deadline. Increasing it buffers more data and postpones the flag, which can mask real events. One documented case raised it from 15s to 20s to suppress rolling-restart false positives, at the cost of detecting genuine slow consumers 5 seconds later. Treat it as a deliberate tradeoff, not a fix.

The churn spiral

If the same clients loop through disconnect-reconnect, pausing or scaling the consumer population breaks the cycle faster than anything server-side. The server is a bystander enforcing limits.

Prevention

• Split the alert. Never alert on aggregate slow_consumers alone. Client rate as a ticket; any route/gateway rate outside maintenance windows as high urgency.
• Watch the precursors. pending_bytes and pending_size build before the flag fires. Top-N sampling via /connz?sort=pending catches offenders before disconnection cascades.
• Baseline deploy windows. Rolling restarts legitimately produce route slow consumer events. Suppress or annotate rather than paging, but do not silence the counter entirely.
• Size consumers for peak fan-out, not average. Slow consumer events cluster around traffic bursts.
• Upgrade to 2.10+ everywhere. The breakdown only exists on 2.10 and later; mixed fleets leave you blind on older nodes.

How Netdata helps

• Netdata polls the NATS HTTP monitoring endpoints at per-second granularity, so slow consumer counter deltas show up as rates rather than scrape-interval guesses.
• Charting the slow_consumers rate next to connections churn and total_connections delta makes the disconnect-reconnect spiral obvious at a glance.
• Correlating slow consumer events with mem shows whether backpressure is accumulating in server buffers before disconnects.
• Correlating with in_msgs / out_msgs rates separates “consumers too slow” from “traffic spike exceeded capacity”.
• Per-second varz polling catches transient route events during deploys that a 60-second scrape would average away.

Related guides

• How NATS actually works in production: a mental model for operators
• NATS monitoring checklist: the signals every production server needs
• NATS monitoring maturity model: from survival to expert
• NATS server not responding: healthz failing and the process down or hung