Your NATS server’s slow_consumer_stats.routes counter just went non-zero, or /routez shows a pending_size that keeps climbing on one route. This is not the same problem as a slow client. A route is the TCP connection that carries inter-server traffic between two NATS servers in a cluster. When it backs up, every subscriber reachable through that peer falls behind or stops receiving messages entirely, and in a JetStream cluster the degradation can extend to Raft heartbeat timing and leader stability.

The blast radius difference is the whole story. A slow client affects one application. A slow route affects every account, subject, and JetStream asset whose traffic flows between those two servers. Any sustained non-zero pending_size on a route is concerning, even before the server formally flags it as a slow consumer and closes the connection.

This article covers how to confirm a route slow consumer, identify which route and which side is at fault, and tell a real peer problem apart from a transient event like a rolling restart.

What this means

Each route between two NATS servers is effectively an internal client with its own read/write goroutines and pending write buffer. The server detects a slow consumer two ways: the outbound buffer for the connection exceeds a pending-bytes limit, or a single write to the socket exceeds the write_deadline (default 10s since NATS 2.2; it was 2s before that, which caused frequent false positives during traffic bursts).

When the server gives up on a route, it closes the connection and logs a line of the form:

Slow Consumer Detected: WriteDeadline of 10s exceeded with N chunks of M total bytes

The route then reconnects automatically, but the underlying cause is usually still present, so pending builds again. Disconnect system events carry a reason field identifying the slow consumer, which is useful for automated detection.

The cascade looks like this:

flowchart TD
  A[Peer server slow to drain route] --> B[pending_size grows on outbound route]
  B --> C{Cause?}
  C -->|Network congestion| D[Route RTT elevated]
  C -->|Peer overloaded| E[Go GC spikes / readloop delays on peer]
  C -->|Traffic asymmetry| F[One direction of route saturated]
  B --> G[write_deadline exceeded]
  G --> H[Server closes route - slow consumer event]
  H --> I[All subscribers via that peer degrade]
  H --> J[JetStream: Raft heartbeats delayed - quorum risk]
  H --> K[Route reconnects - cycle repeats if cause persists]

Two version notes matter here. On NATS 2.10 and later, v2 routes create a pool of route connections between each server pair (default pool_size: 3), with the system account on a dedicated route. That changes what “one route” means: /routez may show multiple route connections per peer, and backpressure on one of them may only affect the accounts pinned to it. Pre-2.10, a single route carried everything, so one slow consumer event blocked all inter-server traffic. Also note that pool_size must match on every server in the cluster; a mismatch prevents clustering with a “Mismatch route pool size” error, and changing it via reload breaks connections.

Common causes

CauseWhat it looks likeFirst thing to check
Network congestion or degradation between serversRoute RTT elevated from baseline, pending_size grows on one peer direction/routez rtt field per route
Peer server overloaded (GC pauses, CPU starvation)Readloop processing time: Xs warnings in logs before the event; peer mem/cpu elevatedPeer /varz cpu, mem
Rolling restart in progressRoute slow consumer events correlate with a node restarting; peer briefly cannot drain inbound route dataUptime on peer nodes (/varz uptime)
Traffic burst exceeding route capacitypending_size spikes correlate with in_msgs/out_msgs spikes/varz throughput counters vs route pending
Pre-2.2 server with 2s write_deadlineFrequent slow consumer disconnects during normal burstsServer version, configured write_deadline
Service mesh or proxy between cluster membersPersistent low-grade route backpressure with no clear host saturationWhether routes traverse sidecars/LB

Quick checks

All read-only. Run against the server’s monitoring port (default 8222).

# 1. Slow consumer breakdown - is it routes, clients, gateways, or leafs?
curl -s http://localhost:8222/varz | jq '{slow_consumers, slow_consumer_stats}'

# 2. Per-route pending and RTT - which route is backing up?
curl -s http://localhost:8222/routez | jq '.routes[] | {rid, remote_id, ip, rtt, pending_size}'

# 3. Route count vs expected: N-1 pre-2.10; on 2.10+, N-1 multiplied by pool_size per peer pair
curl -s http://localhost:8222/routez | jq '.num_routes'

# 4. Server load on this node
curl -s http://localhost:8222/varz | jq '{cpu, mem, connections, in_msgs, out_msgs}'

# 5. Throughput asymmetry - are messages arriving but not leaving?
curl -s http://localhost:8222/varz | jq '{in_msgs, out_msgs, in_bytes, out_bytes}'

# 6. Uptime - is a restart in progress on this node?
curl -s http://localhost:8222/varz | jq .uptime

Also grep the server log on both ends of the route:

grep -E "Slow Consumer|Readloop processing time" /var/log/nats/nats-server.log | tail -30

Readloop processing time: Xs warnings preceding the slow consumer event point at the read side being delayed, typically a CPU-starved or GC-pausing peer. Values of 8-30s have been observed ahead of route disconnects.

How to diagnose it

  1. Confirm the type. Check slow_consumer_stats in /varz. If routes is zero and clients is incrementing, you have a client problem instead; see NATS connection churn. Route or gateway increments are the cluster-wide case this article covers.

  2. Identify the specific route. Pull /routez and find the route connection with non-zero or growing pending_size. Note its remote_id and IP. On 2.10+, note which pooled connection it is and, if accounts are pinned, which account it carries.

  3. Determine the direction of fault. Route pending on server A means A cannot flush data to server B. The fault is on B’s read side, the network between them, or both. Log into B and check its cpu, mem, and log for readloop warnings. If B is healthy, suspect the network path.

  4. Check route RTT against baseline. Same-datacenter routes should be well under 5ms; under 1ms is typical. A sustained increase from baseline is the precursor signal. Brief RTT spikes from Go GC on either end are normal; sustained elevation is not.

  5. Rule out maintenance. Correlate event timestamps with rolling restarts or config reloads. A restarting node temporarily cannot drain inbound route data, and its peers legitimately report it as a slow consumer. If events only occur during deploy windows, that is your answer.

  6. Check JetStream impact. If this is a JetStream cluster, check the meta cluster: curl -s http://localhost:8222/jsz | jq '.meta_cluster | {leader, replicas: [.replicas[]? | {name, current, offline, lag}]}'. Route slow consumer events have been observed cascading into “JetStream cluster no metadata leader” via lost Raft heartbeats. If the meta leader is changing or peers are non-current, treat this as urgent.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
slow_consumer_stats.routes (/varz)Count of route slow consumer events; the definitive cluster-impact indicatorAny positive rate of change
pending_size per route (/routez)Leading indicator; backpressure builds before the server disconnectsAny sustained value > 0
Route rtt (/routez)Precursor from network congestion or GC spikes on either endSustained increase from baseline
in_msgs vs out_msgs (/varz)Route backlog shows as delivery falling behind ingressout_msgs dropping relative to in_msgs
Peer cpu/mem (/varz)GC pauses and CPU starvation on the peer slow its read sideCPU > 90% sustained; monotonic mem growth
meta_cluster leader and replica state (/jsz)Route instability cascades into RaftLeader changes, offline or non-current replicas
uptime per node (/varz)Correlates slow consumer events with restartsUnexpected resets near event timestamps

Note that pending_size and pending_bytes are point-in-time snapshots. A spike can appear and clear between scrapes, so scrape at 10-15s or better and alert on sustained values, not single samples.

Fixes

Overloaded peer

If the peer is CPU-starved or GC-thrashing, fix its resource situation: more CPU headroom, memory tuning, or moving load off it. Readloop warnings on the peer are the tell. Do not start by raising write_deadline; that only delays detection of the real problem.

Network congestion

Reduce cross-server traffic or increase path capacity. Route compression (S2) reduces route bandwidth at the cost of CPU; newer versions support an auto-compression mode that applies compression based on RTT thresholds. Verify routes are not traversing sidecar proxies or L7 load balancers; NATS cluster routes need direct persistent TCP connections, and proxies add latency that shows up as false backpressure.

Rolling restart noise

If events only fire during rolling restarts, some operators raise write_deadline modestly (for example from 15s to 20s) to ride out the drain window. Understand the tradeoff: a higher deadline delays detection of genuine slow consumers everywhere. A better long-term fix is slower restarts with drain time between nodes.

Pre-2.2 false positives

If you are running a server older than 2.2, the default write_deadline of 2s causes frequent false-positive route disconnects during normal bursts. Upgrade; the default was raised to 10s for exactly this reason.

Pre-2.10 single-route topology

On older clusters, one route carries all accounts including system traffic, so any backpressure blocks everything. Upgrading to 2.10+ gets you pooled routes with a dedicated system account route, which isolates system and JetStream control traffic from application backpressure. All servers must run the same pool_size, and changing it requires a coordinated rolling reload with the same value on every server, since a reload-induced mismatch breaks route connections.

Message loss caveat

When a route slow consumer event occurs, the server does not report how many messages were dropped in flight. Even if pending_bytes later dissipates, delivery past the peer is not guaranteed. For subjects where loss matters, this is the argument for JetStream-backed delivery, where gaps surface as consumer lag and can be replayed. See NATS JetStream consumer lag growing.

Prevention

  • Alert on the precursor, not the consequence. slow_consumers is lagging; the server has already disconnected the route. Alert on any sustained route pending_size > 0 and on slow_consumer_stats.routes rate > 0, with higher urgency than client slow consumers.
  • Baseline route RTT per peer and alert on sustained deviation, not absolute values. Cross-AZ baselines differ from same-rack.
  • Monitor route quality, not just route existence. Route count matching the expected total (N-1 pre-2.10; scaled by pool_size on 2.10+) tells you the mesh is connected; it says nothing about whether the mesh can move traffic. Pending size and RTT are the quality signals.
  • Correlate slow consumer events with deploys in your tooling so rolling-restart noise is auto-annotated and real events stand out.
  • Size peers for their read side. Route backpressure is often the peer failing to drain, so CPU and memory headroom on every node protects the whole mesh, not just that node’s clients.
  • Keep versions current. The write_deadline default increase, the 2.10 slow_consumer_stats breakdown, and pooled v2 routes all materially improve this failure mode.

How Netdata helps

  • Netdata polls the NATS HTTP monitoring endpoints and tracks slow_consumers as a rate, so route slow consumer events show up as discrete spikes you can correlate with everything else on the node.
  • The slow_consumer_stats breakdown by connection type makes the first triage decision (clients vs routes vs gateways vs leafnodes) visible without querying each server by hand.
  • Correlating route events with the server’s own cpu, mem, and throughput charts distinguishes “this server cannot flush” from “the peer cannot drain” without logging into two machines.
  • Per-second collection catches the pending_size build-up transient that a 60s scrape interval would miss entirely.
  • Uptime and connection-churn charts alongside slow consumer events make rolling-restart correlation immediate.