NATS write_deadline and buffer sizing: tuning how long the server waits on a slow writer

A client gets disconnected with Slow Consumer Detected in the server log. Someone raises write_deadline, the disconnects stop, and the ticket gets closed. Two weeks later the same client is back in the log, and now the server is also showing memory growth because the larger buffer window lets more messages pile up per connection. This is the standard lifecycle of a write_deadline tuning mistake.

write_deadline controls how long the NATS server will block on flushing data to a connection before it declares that connection slow. The per-connection pending buffer controls how much data can accumulate while the server waits. Together they define the backpressure contract between the server and every client, route, gateway, and leafnode connection. Set them too aggressively and healthy-but-bursty clients get disconnected. Set them too loosely and pending buffers pin memory while genuinely broken consumers go undetected longer.

This guide covers how detection works, how to size both knobs against your RTT and payload profile, and how to tell a tuning problem from an under-scaled consumer.

What this means

Every connection to a NATS server gets dedicated read and write goroutines with a per-connection write buffer. When the server routes a message to a subscriber, it writes into that connection’s pending buffer, and the write goroutine flushes the buffer to the socket. Two conditions flag the connection as a slow consumer:

1. A flush to the connection takes longer than write_deadline (default 10 seconds since v2.2; it was 2 seconds in v2.0.x through v2.1.x).
2. The pending outbound bytes for the connection exceed the internal buffer limit (the server-side max_pending, 64 MB per connection by default).

When either trips, the server closes the connection and logs a line of the form Slow Consumer Detected: WriteDeadline of Xs exceeded with N chunks of Y total bytes. The client library then auto-reconnects, resubscribes, and if the underlying consumer is still too slow, immediately starts building a backlog again. That is the reconnect-disconnect churn spiral.

flowchart TD
  A[Message matched to subscriber] --> B[Write into connection pending buffer]
  B --> C{Flush completes within write_deadline?}
  C -->|yes| D[Connection healthy]
  C -->|no| E[Flag slow consumer]
  B --> F{Pending bytes exceed buffer limit?}
  F -->|no| C
  F -->|yes| E
  E --> G[Server closes connection]
  G --> H[Client auto-reconnects and resubscribes]
  H --> B

One detail that changes blast radius: this logic applies to every connection type, not just clients. Routes, gateways, and leafnodes each carry their own buffers and their own slow consumer risk. A slow consumer on a route means inter-server delivery is backing up, which is a cluster-wide problem. Check the breakdown before assuming it is a client issue:

# Slow consumer totals and breakdown by connection type
curl -s http://localhost:8222/varz | jq '{slow_consumers, slow_consumer_stats}'

Common causes

Quick checks

All read-only, safe to run against a production server with the monitoring port enabled.

# Overall slow consumer counters and per-type breakdown
curl -s http://localhost:8222/varz | jq '{slow_consumers, slow_consumer_stats}'

# Worst-offending client connections by buffered bytes
curl -s "http://localhost:8222/connz?sort=pending&limit=10" | \
  jq '.connections[] | {cid, name, ip, pending_bytes, subscriptions, rtt}'

# Route-level backpressure (clustered servers only)
curl -s http://localhost:8222/routez | jq '.routes[] | {rid, ip, pending_size, rtt}'

# Memory and churn context: buffers show up in RSS, churn shows in total_connections
curl -s http://localhost:8222/varz | jq '{mem, connections, total_connections}'

# Write-path distress precursors, where exposed
curl -s http://localhost:8222/varz | jq '{stalled_clients, stale_connections}'

# Disconnect events in the server log
grep "Slow Consumer Detected" /var/log/nats/nats-server.log | tail -20

How to diagnose it

1. Confirm which connection type is being flagged. Pull slow_consumer_stats from /varz. If the increment is in clients, continue with client diagnosis. If it is in routes or gateways, skip to step 5; the remediation and the urgency are different.

2. Identify the specific connections. Use /connz?sort=pending and note the name, ip, pending_bytes, and rtt of the top entries. Poll a few times 10 seconds apart. A single snapshot with high pending bytes can be a burst artifact; sustained or growing pending is the real signal. These endpoints are point-in-time, so a buffer can spike and clear between scrapes.

3. Classify the flagged client: slow, or just bursty? Compare the disconnect timestamps in the log against throughput. If disconnects only happen during publish bursts and the client drains quickly otherwise, you are looking at a deadline/buffer sizing problem. If pending bytes grow monotonically until the disconnect regardless of traffic shape, the consumer itself cannot keep up. That is an application capacity problem, and no server-side tuning fixes it.

4. Check RTT on the flagged connections. A connection with elevated RTT drains its buffer more slowly, so the same publish rate that is fine on a LAN client will trip a WAN client. If flagged clients cluster at high RTT, the fix is usually buffer headroom sized to the bandwidth-delay product, not a global deadline increase.

5. For route or gateway slow consumers, treat it as cluster infrastructure. Check /routez pending_size and route RTT. A restarting node catching up on gossip can transiently appear slow on routes; community reports describe rolling restarts tripping route slow consumers, with operators raising the deadline from 15s to 20s as a workaround at the cost of slower detection of real slow consumers. Sustained route pending outside of restarts means network degradation or an overloaded peer.

6. Check the server version. Before v2.6.5, a single slow subscriber could block the publisher’s read loop for the full write_deadline, stalling delivery to other subscribers on the same path (upstream issue #2679, fixed in v2.6.5). If you see Readloop processing time warnings alongside slow consumer events on an old server, upgrade before you tune anything.

Metrics and signals to monitor

Tuning and fixes

Fix the consumer, not the deadline, when the consumer is the bottleneck

If pending bytes grow monotonically regardless of traffic shape, the consumer is under-scaled: blocked on a database write, starved on CPU, stuck in a GC pause, or doing synchronous work in the message handler. Raising write_deadline here only postpones the error while more memory accumulates in per-connection buffers. Scale the consumer out, move blocking work off the handler, or reduce the fan-out to that subscriber.

Raise the deadline deliberately, not defensively

write_deadline takes a Go duration string and is reloadable without a restart:

# nats-server.conf
write_deadline: "10s"

Older v1.x configs used a bare integer in seconds; that form is deprecated in favor of the duration string.

Raising the deadline is legitimate when your traffic is genuinely bursty and consumers demonstrably drain between bursts: the deadline must exceed your worst-case flush stall, including GC pauses on the client and network jitter. The tradeoff is detection latency. A 30-second deadline means a truly dead consumer sits connected, buffering, for 30 seconds before the server acts. On routes, that delay extends to cluster-wide delivery.

Size buffers against RTT and payload

The buffer question is: how many bytes can pile up before the server gives up? A useful floor is the bandwidth-delay product of the connection path: pending headroom >= drain_rate x worst_case_stall_time. Work it from your numbers:

• Estimate peak publish rate to a single connection (messages/s x average payload bytes). Watch byte rate, not just message rate; a 10x jump in average payload size has the same buffer impact as a 10x jump in message rate and only shows up in in_bytes/out_bytes.
• Multiply by the longest stall you are willing to absorb (client GC pause, brief redeploy, network blip). That product is the minimum useful buffer headroom.
• Compare against the server-side per-connection pending limit (max_pending, 64 MB default) and the client-side subscription pending limits (default 65536 messages or 64 MB per subscription, whichever hits first; -1 disables a limit, which trades slow consumer protection for unbounded memory growth).

Larger buffers are not free. Every connection’s pending capacity is potential RSS. Multiply your per-connection headroom by connection count and make sure the result fits in your memory budget with room for Go GC overhead. A reasonable target: keep peak RSS under 70-80% of the container or host limit.

Client-side pending limits as the second line

The client library enforces its own per-subscription pending limits independent of the server. If the server-side knobs are correct but a single greedy subscription starves others on the same connection, tune the client limits per subscription rather than loosening the server globally. Keep at least one of the message-count or byte limits finite; unbounded on both sides is how a stalled consumer turns into a host OOM.

What not to do

• Do not raise write_deadline globally to mask route slow consumers during deploys without understanding you are also delaying detection of real route failures. If restart transients are the only problem, consider whether the deploy sequence can give nodes time to catch up instead.
• Do not treat slow_consumers as a server fault. It is the server correctly enforcing backpressure; the fault is on the slow end of the connection.
• Do not scrape /connz aggressively on high-connection-count servers to watch pending bytes. Poll at 10-15 seconds and use sort=pending with a limit rather than enumerating everything.

Prevention

• Alert on the rate, not the counter. slow_consumers is cumulative; a static non-zero value is history. Alert on any sustained positive rate of change, weighted by type.
• Watch the precursor. pending_bytes and stalled_clients build before the disconnect. A top-N pending check catches the problem minutes earlier than the slow consumer event.
• Load-test burst profiles before choosing the deadline. Measure your real worst-case flush stall (client GC pause plus network jitter) in staging and set the deadline with margin above it, not by doubling whatever value last stopped the pages.
• Budget buffer memory explicitly. Per-connection headroom x connection count must fit in RSS with GC headroom. Revisit the budget whenever you raise limits or connection counts.
• Pin the server version. Anything older than v2.6.5 carries the publisher-blockage bug that makes one slow consumer hurt healthy subscribers.

How Netdata helps

• Netdata’s NATS collector polls the monitoring HTTP endpoints and charts slow_consumers as a rate, so you see disconnection events as they happen rather than discovering them in logs.
• Correlating the slow consumer rate against in_msgs/out_msgs throughput separates “bursty traffic tripping a tight deadline” (spike aligns with traffic peaks) from “consumer cannot keep up” (rate climbs while traffic is flat).
• The connection churn view, total_connections delta against stable active connections, makes the reconnect-disconnect spiral visible as churn rather than as a connection count drop.
• Tracking mem alongside slow consumer events shows whether your buffer sizing is pinning memory, which is the cost side of a loose deadline.
• Route and gateway context on the same dashboard lets you confirm quickly whether the event is a client problem or a cluster-internal one, which changes who gets paged.

Related guides

• NATS slow consumer detected: the write buffer overflowed and messages were dropped
• NATS slow consumer breakdown: clients vs routes vs gateways and blast radius
• NATS server not responding: healthz failing and the process down or hung
• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check
• How NATS actually works in production: a mental model for operators
• NATS monitoring checklist: the signals every production server needs
• NATS monitoring maturity model: from survival to expert