NATS slow consumer detected: the write buffer overflowed and messages were dropped

Your NATS server log shows Slow Consumer Detected, or your client logged nats: slow consumer, messages dropped. In core NATS (no JetStream), that line means messages were dropped. Not queued, not retried, not parked somewhere for later.

The server buffers outbound messages per connection, and when a subscriber cannot drain its write buffer fast enough, the server sheds load by dropping messages for that subscriber or disconnecting it outright. Delivery in core NATS is at-most-once, and the slow subscriber is not told what was lost.

The server is doing the right thing: enforcing backpressure so one stalled connection cannot drag down the whole routing engine. The fault is almost always on the other end of the connection: a consumer that cannot keep up, a network path that cannot drain the socket, or a cluster route backing up.

What this means

Every client TCP connection to a NATS server gets dedicated read and write goroutines. The write side maintains a pending buffer: messages matched to that connection’s subscriptions are staged there until the kernel accepts them on the socket. Two limits govern this buffer: an internal pending-bytes limit, and write_deadline (default 10s), which caps how long the server will wait for a slow socket write before giving up.

When the buffer exceeds its limit, the server flags the connection as a slow consumer. For client connections the default outcome is that pending messages are dropped and, depending on version and configuration, the connection may be closed. On the client side, libraries expose this through an asynchronous error callback (nats.ErrSlowConsumer in the Go client) which fires before the server acts, giving the application a chance to log or react. Client libraries also enforce their own subscription pending limits (default 65536 messages or 64 MB per subscription in the Go client) and will drop messages client-side when those are exceeded.

Two facts change the blast radius significantly:

• Slow consumers happen on all connection types, not just clients. Routes, gateways, and leaf node connections each have their own buffers and can be flagged. A slow route means inter-server message delivery is backing up, which is a cluster-wide problem, not a single application problem.
• Pull-based JetStream consumers are immune because they pace themselves by explicitly requesting messages. JetStream push consumers are not immune: they ride on a core NATS subscription and can be flagged like any other subscriber.

flowchart TD
  P[Publisher] --> S[NATS server routing]
  S --> B[Per-connection write buffer]
  B -->|client drains fast enough| C[Subscriber receives messages]
  B -->|buffer exceeds limit| SC[Slow consumer flagged]
  SC -->|core NATS| D[Messages dropped - permanent loss]
  SC -->|default action| X[Connection disconnected]
  X --> R[Client auto-reconnects and resubscribes]
  R -->|backlog still there| SC

Common causes

Quick checks

All of these are read-only against the HTTP monitoring port (default 8222).

# 1. Confirm the event and get the breakdown by connection type
curl -s http://localhost:8222/varz | jq '{slow_consumers, slow_consumer_stats}'

# 2. Estimate the current event rate (cumulative counter, so take two samples)
curl -s http://localhost:8222/varz | jq .slow_consumers
sleep 60
curl -s http://localhost:8222/varz | jq .slow_consumers

# 3. Find the worst offending connections right now
curl -s "http://localhost:8222/connz?sort=pending&limit=10" | \
  jq '.connections[] | {cid, name, ip, pending_bytes, subscriptions, rtt}'

# 4. Check route-level backpressure (clustered deployments)
curl -s http://localhost:8222/routez | jq '.routes[] | {rid, ip, pending_size, rtt}'

# 5. Check per-account slow consumer counts (multi-account servers)
curl -s http://localhost:8222/accstatz | jq '.account_statz[] | {account_name, slow_consumers}'

# 6. Confirm it in the server log (path depends on your deployment)
grep "Slow Consumer" /var/log/nats/nats-server.log | tail -20
# or, on systemd units:
journalctl -u nats-server --since "1 hour ago" | grep "Slow Consumer" | tail -20

# 7. Sanity check server health and uptime (rule out a wider problem)
curl -s "http://localhost:8222/healthz?js-server-only=true"
curl -s http://localhost:8222/varz | jq '{uptime, connections, mem}'

A note on /connz: it is a point-in-time snapshot and can be expensive on servers with very high connection counts. Use limit, sample periodically rather than scraping continuously, and never request full subscription listings (/connz?subs=1 or /subsz?subs=1) on a busy server.

How to diagnose it

1. Check the breakdown first. /varz slow_consumer_stats splits events into clients, routes, gateways, and leafs. This single field decides your severity. Client-only events are an application problem. Route or gateway events mean inter-server delivery is failing and the blast radius is the whole cluster. Treat those with much higher urgency.

2. Measure the rate, not the counter. slow_consumers is cumulative since server start. A static non-zero value is history; a positive rate is an active problem. Alert and triage on rate of change.

3. Identify the offending connection. Use /connz?sort=pending and look at pending_bytes. Pending bytes are the leading indicator: before the server flags a connection as slow, its buffer grows. Transient spikes are normal under bursty load; sustained pending above roughly 1 MB on a client connection is a real problem, and sustained growth into the multi-MB range means a slow consumer flag is imminent. On routes, any sustained non-zero pending_size is concerning.

4. Decide which side is slow: the app or the network. Look at the connection’s rtt in /connz. Normal RTT with high pending bytes points at the consumer application (blocked handler, GC pause, CPU starvation). Elevated RTT alongside pending bytes points at the network path or an overloaded client host. Remember that NATS RTT is measured via protocol PING/PONG, so a CPU-bound client inflates it too.

5. Correlate with churn. Compare /varz total_connections growth against the stable connections count. Fast-growing total_connections with a flat connections gauge means clients are flapping: the classic slow consumer death spiral where the server disconnects the client, the library auto-reconnects, the backlog hits immediately, and the cycle repeats.

6. For JetStream, check the consumer type. If the flagged subscription belongs to a JetStream push consumer, the messages are safe in the stream (they will be redelivered per the consumer’s ack policy), but delivery has stalled. Check num_ack_pending against MaxAckPending via /jsz?consumers=true or nats consumer info. A consumer pinned at MaxAckPending has stopped receiving entirely.

7. Check the server version. On older nats-server versions, a single slow subscriber could block the publisher for the full write_deadline, stalling all other subscribers on the same server. If you are on an affected version, the upgrade is the fix for a whole class of secondary symptoms.

Metrics and signals to monitor

Fixes

Consumer application is the bottleneck

The most common root cause: the message handler does something slow (synchronous database writes, downstream HTTP calls, serialization) on the receive path, so the client library cannot drain its internal queue. Fixes, in order of preference:

• Move work off the receive path. Have the handler hand messages to a worker pool or internal queue so the network read loop never blocks.
• Fix the actual stall: database lock contention, GC tuning, CPU limits on the container.
• Scale out horizontally. In a core NATS queue group, a slow worker that gets disconnected has its share immediately rerouted to the remaining members, so queue groups give you both capacity and isolation from one bad instance. Messages already sitting in the disconnected worker’s buffer are still lost.

Raising the client library’s pending limits (SetPendingLimits or equivalent) buys buffer headroom but does not fix throughput. If the consumer is structurally slower than the producer, a bigger buffer only delays the drop.

Message volume exceeds what one subscriber can absorb

If the subscriber simply receives more messages than it can process (high fan-out, broad wildcard subscription), reduce what it must ingest: narrow the subscription subjects, shard the workload across queue group members, or split the subject space. Check the fan-out ratio (out_msgs / in_msgs) to confirm the amplification factor before choosing a shard count.

Tuning write_deadline

You can raise write_deadline above the 10s default to tolerate brief stalls. The official documentation’s warning applies directly: be sure you are not just postponing a slow consumer error. A longer deadline means larger buffers, more memory pressure on the server, and the same drop at the end. Tune it only to absorb known, bounded bursts.

Route or gateway slow consumers

This is a different incident. Check route RTT and pending_size on /routez, look for network degradation between the servers (retransmits, saturated links, a noisy-neighbor VM), and verify the peer server is not CPU-starved. Transient route slow consumer events during a rolling restart are expected as routes reconnect and re-converge; a sustained rate outside maintenance windows is not. If you run a service mesh with sidecar proxies on cluster traffic, added proxy latency can cause false slow consumer events on routes.

The messages matter and loss is unacceptable

Core NATS is fire-and-forget. If losing messages is not acceptable, the architectural fix is JetStream: streams persist messages, and pull-based consumers pace themselves so the slow consumer condition does not apply to them. Push consumers still ride core subscriptions and can be flagged, so for loss-sensitive workloads prefer pull consumers, and monitor num_pending and num_ack_pending per consumer as your lag signals.

Prevention

• Alert on rate, not absolute. slow_consumers is cumulative. Alert on any sustained positive rate, and escalate harder when slow_consumer_stats.routes or .gateways is the source.
• Monitor the precursor, not just the consequence. Track top-N pending_bytes via sampled /connz?sort=pending and per-route pending_size. You get minutes of warning before a disconnection instead of a log line after the loss.
• Instrument the client side. Wire the async error callback in your client libraries so nats: slow consumer events land in your application logs with context, and set explicit pending limits rather than inheriting defaults you have never reviewed.
• Budget memory for buffering. Per-connection buffers live in server RSS. Slow consumer events and rising mem correlate; leave headroom so backpressure does not become an OOM.
• Keep the server current. Older server versions carried write-path behaviors that turned one slow subscriber into a server-wide stall.
• Load-test the slow path. Exercise a deliberately slow consumer in staging so you know your alerts fire, your client callbacks log, and your team recognizes the pattern before 3 a.m.

How Netdata helps

• Netdata polls the NATS HTTP monitoring endpoints and charts slow_consumers as a rate, so you see the onset of events immediately instead of diffing cumulative counters by hand.
• Correlating slow consumer events with connection count and churn on the same dashboard separates a one-off stall from the reconnect-disconnect death spiral.
• Plotting in_msgs against out_msgs next to slow consumer events shows whether deliveries are actually falling behind publishes, and by how much.
• Memory (RSS) and CPU from /varz on the same node view let you confirm whether buffer accumulation is pressuring the server itself.
• Long retention on these signals lets you distinguish chronic low-grade slow consumer problems (a capacity issue) from acute spikes tied to deploys or restarts.

Related guides

• How NATS actually works in production: a mental model for operators
• NATS monitoring checklist: the signals every production server needs
• NATS monitoring maturity model: from survival to expert
• NATS server not responding: healthz failing and the process down or hung