NATS pending bytes growing: catching a slow consumer before it is disconnected

By the time slow_consumers in /varz increments, the damage is done. The server has declared a connection slow, and in core NATS the default response is to disconnect it. Messages buffered for that client are dropped, not queued, not retried. The client library auto-reconnects, resubscribes, immediately falls behind on the same backlog, and you are in a churn loop.

The precursor signal is sitting in the monitoring endpoints the whole time. Every connection has a write-side pending buffer you can read before the server gives up on it: pending_bytes per client connection via /connz, and pending_size per route via /routez. Watching these lets you name the specific culprit, a client, a route, or a gateway, while there is still time to act.

What this means

Each client TCP connection to a NATS server gets dedicated read and write goroutines with per-connection buffers. When a publisher sends a message, the routing engine matches it against the subject trie and writes it into every matching subscriber’s write buffer. If a client cannot drain that buffer as fast as the server fills it, pending bytes grow. The server waits, bounded by write_deadline (10s by default), then flags the connection as a slow consumer.

Two operational consequences:

1. slow_consumers is a cumulative counter of disconnect events. It tells you distress already happened, not that it is coming.
2. The pending buffer is observable in real time. Rising pending bytes on one connection is the leading indicator, and it gives you a window to intervene before the disconnect, the message loss, and the reconnect churn.

The blast radius depends on the connection type. A slow client loses its own messages. A slow route means inter-server traffic is backing up, which degrades delivery for every subscriber reachable through that route. Routes and gateways are internal connections with their own buffers and can become slow consumers exactly like clients, and when they do the impact is cluster-wide. The slow_consumer_stats breakdown in current upstream /varz separates events into clients, routes, gateways, and leafs for exactly this reason.

flowchart TD
  A[Pending bytes rising on a connection] --> B{Which connection type?}
  B -->|Client| C[>1MB sustained: investigate
>10MB sustained: disconnect imminent]
  B -->|Route or gateway| D[Any sustained pending:
cluster-wide impact, treat as urgent]
  C --> E{Spike and clears between scrapes?}
  E -->|Yes| F[Bursty workload, keep watching]
  E -->|No, trend is up| G[Fix consumer or shed load
before slow_consumers fires]
  D --> G

Common causes

The last row deserves emphasis: the server-side write buffer (bounded by write_deadline and internal buffer sizing) and the client library’s own pending limits are two independent buffer boundaries. Operators routinely conflate them. A client can be flagged as slow because its library-side limits tripped even when the server-side buffer looked fine, and vice versa.

Quick checks

All read-only. The monitoring port defaults to 8222 and must be enabled (-m 8222 or http_port: 8222).

# Top 5 client connections by pending bytes, with identity fields
curl -s "http://localhost:8222/connz?sort=pending&limit=5" | \
  jq '.connections[] | {cid, name, ip, pending_bytes, subscriptions, rtt}'

# Per-route pending size: any sustained non-zero value is concerning
curl -s http://localhost:8222/routez | \
  jq '.routes[] | {rid, ip, pending_size, rtt}'

# Slow consumer history and breakdown by connection type
curl -s http://localhost:8222/varz | \
  jq '{slow_consumers, slow_consumer_stats}'

# Connection churn: stable connections with fast-growing total_connections
# means clients are flapping (disconnect/reconnect loops)
curl -s http://localhost:8222/varz | \
  jq '{connections, total_connections}'

# Server memory: pending buffers live in RSS
curl -s http://localhost:8222/varz | jq .mem

For a live view sorted by pending, nats-top -sort pending shows a PENDING column per connection. One timing trap: once the server disconnects the slow client, its pending buffer is gone. If slow_consumers is incrementing but every connection shows pending at zero, you are looking at the aftermath, not the buildup. That is why the trend matters more than any single snapshot.

How to diagnose it

1. Confirm it is a trend, not a snapshot. /connz and /routez are point-in-time. Bursty workloads spike pending bytes and clear them between scrapes. Poll the top-N connections two or three times, 10 to 15 seconds apart. A value that spikes and clears is normal under load; a value that climbs across consecutive samples is a consumer falling behind.

2. Identify the connection type. Check slow_consumer_stats first. If events are under routes or gateways, stop thinking about clients. Route and gateway slow consumers have cluster-wide blast radius and need network and peer-server investigation, not application debugging.

3. Name the culprit client. From /connz?sort=pending, capture the cid, name, ip, and subscription count of the worst connections. The name field is whatever the client set at connect time, which is why requiring a meaningful connection name in every client application pays off during incidents.

4. Check the client’s RTT. High rtt on the same connection suggests a network path problem or a client host under CPU pressure (the RTT is measured via NATS PING/PONG frames, so a CPU-starved client library inflates it). Normal RTT with growing pending points at the application handler itself: GC, blocking I/O, a slow downstream.

5. Correlate with throughput and churn. If out_msgs is dropping relative to in_msgs while pending grows, deliveries are backing up. If total_connections is climbing fast while connections stays flat, the disconnect-reconnect spiral has already started.

6. For routes, check the peer. A route with growing pending_size means the far server cannot keep up or the path between them is degraded. Check route RTT, the peer’s CPU and memory, and whether you are mid-rolling-restart (route slow consumers during restarts are common and usually self-resolve).

Metrics and signals to monitor

Collection cost note: polling /connz for every connection at scrape frequency is expensive on servers with high connection counts. The top-N approach (?sort=pending&limit=N) is much cheaper and catches what matters, since you need the worst offender, not the full distribution.

Fixes

The specific client is falling behind

The problem is on the consumer side, not the server. Treat the server as correctly enforcing backpressure.

• Unblock the application. Look for what the handler is waiting on: a database lock, a slow downstream HTTP call, a GC pause. Synchronous I/O inside the message handler is the classic root cause.
• Scale the consumer group. If the publish rate grew past what one subscriber can process, add members to the queue group so deliveries spread out.
• Raise client-side pending limits if the client library’s own limits are tripping before the workload actually warrants it. This buys headroom for bursts; it does not fix a consumer that is structurally too slow, and it trades message loss for memory growth on the client.
• Shed load deliberately. Disconnecting the slow consumer yourself, or pausing its publishers, beats letting the server do it mid-backlog. In core NATS those buffered messages are gone either way, but a controlled cut avoids the reconnect-into-backlog churn loop.

A route or gateway is backing up

• Check the path and the peer. Route pending almost always means the remote server is overloaded or the network between the servers is degraded. Fix the peer, not the route.
• Expect transient route pending during rolling restarts. A common false alarm. If it clears when the restarted peer finishes rejoining, it was the restart.
• Be cautious raising write_deadline cluster-wide. It can smooth over route slow consumer events during restarts, but it delays detection of genuine slow clients everywhere. That is a real tradeoff, not a free fix.

You only see the aftermath (counter up, pending zero everywhere)

The disconnect already happened. Use the slow_consumer_stats type breakdown plus server logs for the disconnect events, and set up trending on pending bytes so the next occurrence is caught in the buildup phase.

Prevention

• Trend pending bytes, not just the counter. Alert on a client connection with sustained pending_bytes above 1MB, and page-urgent above 10MB. Alert on any sustained non-zero route pending_size. Sustained means across multiple scrapes, never on a single sample.
• Require connection names. An unnamed cid at 3 a.m. means guessing which of forty services is drowning. Make every client set a name that identifies the service and instance.
• Watch churn as a compound signal. total_connections climbing fast with flat connections is the signature of the disconnect-reconnect spiral, and often the first visible symptom when pending-byte alerts were missed.
• Keep scrapes at 10 to 15 seconds. The monitoring endpoints are snapshots served by a single goroutine. Faster polling adds load and still misses sub-second spikes; slower polling misses the buildup window entirely.
• Load-test consumer headroom. Know the publish rate at which your consumers start accumulating pending, and alert well below it.

How Netdata helps

Netdata polls the NATS monitoring endpoints and keeps per-second history, which is what turns pending bytes from a snapshot into a trend:

• Slow consumer counter tracking so the rate of slow_consumers events is visible over time, not just as a number you happened to curl.
• Throughput correlation: overlaying in_msgs vs out_msgs with slow consumer events shows whether deliveries are backing up while publishes continue.
• Connection and churn visibility: connections alongside total_connections growth exposes the disconnect-reconnect spiral that follows slow consumer events.
• Memory correlation: pending write buffers live in the server’s RSS, so a fleet-wide slow consumer event shows up as memory growth; seeing mem climb alongside throughput drops confirms the mechanism.
• Cluster-wide views: correlating route health and slow consumer events across all servers separates a single bad client from inter-server backpressure.

Related guides

• NATS slow consumer detected: the write buffer overflowed and messages were dropped
• NATS slow consumer breakdown: clients vs routes vs gateways and blast radius
• NATS monitoring checklist: the signals every production server needs
• How NATS actually works in production: a mental model for operators
• NATS monitoring maturity model: from survival to expert
• NATS server not responding: healthz failing and the process down or hung
• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check