NATS route slow consumer: inter-server forwarding backing up cluster-wide

Your NATS server’s slow_consumer_stats.routes counter just went non-zero, or /routez shows a pending_size that keeps climbing on one route. This is not the same problem as a slow client. A route is the TCP connection that carries inter-server traffic between two NATS servers in a cluster. When it backs up, every subscriber reachable through that peer falls behind or stops receiving messages entirely, and in a JetStream cluster the degradation can extend to Raft heartbeat timing and leader stability.

The blast radius difference is the whole story. A slow client affects one application. A slow route affects every account, subject, and JetStream asset whose traffic flows between those two servers. Any sustained non-zero pending_size on a route is concerning, even before the server formally flags it as a slow consumer and closes the connection.

This article covers how to confirm a route slow consumer, identify which route and which side is at fault, and tell a real peer problem apart from a transient event like a rolling restart.

What this means

Each route between two NATS servers is effectively an internal client with its own read/write goroutines and pending write buffer. The server detects a slow consumer two ways: the outbound buffer for the connection exceeds a pending-bytes limit, or a single write to the socket exceeds the write_deadline (default 10s since NATS 2.2; it was 2s before that, which caused frequent false positives during traffic bursts). Since NATS Server 2.14.0, a route also has a write-timeout policy: the default is retry, so a slow route is flagged and retried rather than closed; cluster.write_timeout: close restores close-on-timeout behavior. Before 2.14, a route write deadline closed the connection.

A write-deadline event logs a line of the form:

Slow Consumer Detected: WriteDeadline of 10s exceeded with N chunks of M total bytes

Route behavior then depends on cluster.write_timeout: the default retry in NATS Server 2.14.0 and later keeps the route connected and later logs Slow Consumer State/Slow Consumer Recovered; close, or releases before 2.14, closes the route and lets it reconnect. $SYS.ACCOUNT.<account>.DISCONNECT events include reason, but they cover client/leaf billing events, not route closures; use logs and slow_consumer_stats.routes for routes.

The cascade looks like this:

flowchart TD
  A[Peer server slow to drain route] --> B[pending_size grows on outbound route]
  B --> C{Cause?}
  C -->|Network congestion| D[Route RTT elevated]
  C -->|Peer overloaded| E[Go GC spikes / readloop delays on peer]
  C -->|Traffic asymmetry| F[One direction of route saturated]
  B --> G[write_deadline exceeded]
  G --> H[Route flagged as slow consumer]
  H --> I[All subscribers via that peer degrade]
  H --> J[JetStream: Raft heartbeats delayed - quorum risk]
  H --> K[Default retry keeps route; close mode reconnects]

Two version notes matter here. On NATS 2.10 and later, v2 routes create a pool of route connections between each server pair (default pool_size: 3), with the system account on a dedicated route. That changes what “one route” means: /routez may show multiple route connections per peer, and backpressure on one of them may only affect the accounts pinned to it. Pre-2.10, a single route carried everything, so one slow consumer event blocked all inter-server traffic. Also note the version behavior: nats-server 2.11 and later allow different pool_size values during rolling updates by using the larger effective size; older pooled-route releases require matching values. A pool-index violation is rejected as an “Invalid route pool index” protocol error, and a reload-induced topology change can still close/re-form routes.

Common causes

Quick checks

All read-only. Run against the server’s monitoring port (default 8222).

# 1. Slow consumer breakdown - is it routes, clients, gateways, or leafs?
curl -s http://localhost:8222/varz | jq '{slow_consumers, slow_consumer_stats}'

# 2. Per-route pending and RTT - which route is backing up?
curl -s http://localhost:8222/routez | jq '.routes[] | {rid, remote_id, ip, rtt, pending_size}'

# 3. Route count vs expected: N-1 pre-2.10; on 2.10+, N-1 multiplied by pool_size per peer pair
curl -s http://localhost:8222/routez | jq '.num_routes'

# 4. Server load on this node
curl -s http://localhost:8222/varz | jq '{cpu, mem, connections, in_msgs, out_msgs}'

# 5. Throughput asymmetry - are messages arriving but not leaving?
curl -s http://localhost:8222/varz | jq '{in_msgs, out_msgs, in_bytes, out_bytes}'

# 6. Uptime - is a restart in progress on this node?
curl -s http://localhost:8222/varz | jq .uptime

Also grep the server log on both ends of the route:

grep -E "Slow Consumer|Readloop processing time" /var/log/nats/nats-server.log | tail -30

Readloop processing time: Xs warnings preceding the slow consumer event point at the read side being delayed, typically a CPU-starved or GC-pausing peer. Values of 8-30s have been observed ahead of route disconnects.

How to diagnose it

1. Confirm the type. Check slow_consumer_stats in /varz. If routes is zero and clients is incrementing, you have a client problem instead; see NATS connection churn. Route or gateway increments are the cluster-wide case this article covers.

2. Identify the specific route. Pull /routez and find the route connection with non-zero or growing pending_size. Note its remote_id and IP. On 2.10+, note which pooled connection it is and, if accounts are pinned, which account it carries.

3. Determine the direction of fault. Route pending on server A means A cannot flush data to server B. The fault is on B’s read side, the network between them, or both. Log into B and check its cpu, mem, and log for readloop warnings. If B is healthy, suspect the network path.

4. Check route RTT against baseline. Same-datacenter routes should be well under 5ms; under 1ms is typical. A sustained increase from baseline is the precursor signal. Brief RTT spikes from Go GC on either end are normal; sustained elevation is not.

5. Rule out maintenance. Correlate event timestamps with rolling restarts or config reloads. A restarting node temporarily cannot drain inbound route data, and its peers legitimately report it as a slow consumer. If events only occur during deploy windows, that is your answer.

6. Check JetStream impact. If this is a JetStream cluster, check the meta cluster: curl -s http://localhost:8222/jsz | jq '.meta_cluster | {leader, replicas: [.replicas[]? | {name, current, offline, lag}]}'. Route slow consumer events have been observed cascading into “JetStream cluster no metadata leader” via lost Raft heartbeats. If the meta leader is changing or peers are non-current, treat this as urgent.

Metrics and signals to monitor

Note that pending_size and pending_bytes are point-in-time snapshots. A spike can appear and clear between scrapes, so scrape at 10-15s or better and alert on sustained values, not single samples.

Fixes

Overloaded peer

If the peer is CPU-starved or GC-thrashing, fix its resource situation: more CPU headroom, memory tuning, or moving load off it. Readloop warnings on the peer are the tell. Do not start by raising write_deadline; that only delays detection of the real problem.

Network congestion

Reduce cross-server traffic or increase path capacity. Route compression (S2) reduces route bandwidth at the cost of CPU; the RTT-based auto mode is available for cluster routes from NATS Server 2.10.0. Verify routes are not traversing sidecar proxies or L7 load balancers; NATS cluster routes need direct persistent TCP connections, and proxies add latency that shows up as false backpressure.

Rolling restart noise

If events only fire during rolling restarts, some operators raise write_deadline modestly (for example from 15s to 20s) to ride out the drain window. Understand the tradeoff: a higher deadline delays detection of genuine slow consumers everywhere. A better long-term fix is slower restarts with drain time between nodes.

Pre-2.2 false positives

If you are running a server older than 2.2, the default write_deadline of 2s causes frequent false-positive route disconnects during normal bursts. Upgrade; the default was raised to 10s in NATS 2.2.0 for exactly this reason.

Pre-2.10 single-route topology

On older clusters, one route carries all accounts including system traffic, so any backpressure blocks everything. Upgrading to 2.10+ gets you pooled routes with a dedicated system account route, which isolates system and JetStream control traffic from application backpressure. On releases before 2.11, all servers must run the same pool_size; from 2.11, rolling updates can use the larger effective size, but coordinate reloads and verify that pooled routes re-form on both ends.

Message loss caveat

When a route slow consumer event occurs, the server does not report how many messages were dropped in flight. Even if pending_bytes later dissipates, delivery past the peer is not guaranteed. For subjects where loss matters, this is the argument for JetStream-backed delivery, where gaps surface as consumer lag and can be replayed. See NATS JetStream consumer lag growing.

Prevention

• Alert on the precursor, not the consequence. slow_consumers is lagging; the server has already disconnected the route. Alert on any sustained route pending_size > 0 and on slow_consumer_stats.routes rate > 0, with higher urgency than client slow consumers.
• Baseline route RTT per peer and alert on sustained deviation, not absolute values. Cross-AZ baselines differ from same-rack.
• Monitor route quality, not just route existence. Route count matching the expected total (N-1 pre-2.10; scaled by pool_size on 2.10+) tells you the mesh is connected; it says nothing about whether the mesh can move traffic. Pending size and RTT are the quality signals.
• Correlate slow consumer events with deploys in your tooling so rolling-restart noise is auto-annotated and real events stand out.
• Size peers for their read side. Route backpressure is often the peer failing to drain, so CPU and memory headroom on every node protects the whole mesh, not just that node’s clients.
• Keep versions current. The write_deadline default increase, the 2.10 slow_consumer_stats breakdown, and pooled v2 routes all materially improve this failure mode.

How Netdata helps

• Netdata polls the NATS HTTP monitoring endpoints and tracks slow_consumers as a rate, so route slow consumer events show up as discrete spikes you can correlate with everything else on the node.
• The slow_consumer_stats breakdown by connection type makes the first triage decision (clients vs routes vs gateways vs leafnodes) visible without querying each server by hand.
• Correlating route events with the server’s own cpu, mem, and throughput charts distinguishes “this server cannot flush” from “the peer cannot drain” without logging into two machines.
• Per-second collection catches the pending_size build-up transient that a 60s scrape interval would miss entirely.
• Uptime and connection-churn charts alongside slow consumer events make rolling-restart correlation immediate.

Related guides

• How NATS actually works in production: a mental model for operators
• NATS connection churn: a stable connection count hiding constant reconnects
• NATS connection storm: reconnect thundering herd after a network event
• NATS JetStream consumer lag growing: falling behind the stream
• NATS consumer stalled at MaxAckPending: delivery stops until messages are acked
• NATS JetStream redelivery loop: num_redelivered climbing and messages reprocessed
• NATS JetStream consumer stopped receiving messages: the diagnostic tree
• NATS JetStream AckWait tuning: matching the ack timeout to processing time
• NATS context deadline exceeded: JetStream publish and request timeouts
• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS file descriptor exhaustion: too many open files and the ulimit cliff
• NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check