NATS stalled clients and stale connections: half-dead sockets and write-path distress

Two counters in /varz tell you about connections that are unhealthy but not yet dead: stalled_clients and stale_connections. Neither one means a connection has been dropped. That is the point: both describe states that precede visible failure. Stalled clients are on the way to becoming slow consumers; stale connections are sockets that look open but whose peer has stopped responding.

The operational risk is twofold. Stalled clients signal write-path backpressure: the server cannot flush data to a client fast enough, which is the precursor to slow consumer disconnection and, in core NATS, silent message loss. Stale connections are quieter: they consume file descriptors and per-connection memory while doing no useful work, and they usually indicate a network half-partition or a hung client process that will not recover on its own.

This article covers what each counter means, how to find the specific connections behind them, and how to fix the underlying causes. For the broader NATS signal taxonomy, see the NATS monitoring checklist.

What this means

Every client TCP connection gets dedicated read and write loops plus a per-connection pending buffer on the write side. When a client cannot consume messages fast enough, that buffer grows. The server enforces backpressure with write_deadline: if a write to the connection cannot complete within the deadline (default 10 seconds; tunable in server config), the connection is in write-path distress.

The stalled_clients counter in /varz tracks connections that have entered this stalled state. It is a pre-slow-consumer signal: the buffer is under pressure, but the server has not yet flagged the connection as a slow consumer and disconnected it.

Separately, the server runs a ping/pong health check over each connection. If a client fails to respond to protocol pings, the connection is declared stale, counted in stale_connections, and closed. The key property of a stale connection before closure is that the TCP socket still looks alive from the server’s perspective: the peer is not responding at the protocol level, but TCP has not noticed, or has not been allowed to notice. This is the classic half-dead socket. Common causes: a hung client process (GC storm, deadlock, stopped container), a stateful middlebox that silently dropped the flow state, or a network partition where neither side ever sees a RST.

flowchart TD
  A[Client connection] --> B{Write buffer draining?}
  B -- yes --> C[Healthy]
  B -- no, write_deadline hit --> D[stalled_clients]
  D --> E{Buffer keeps growing?}
  E -- yes --> F[slow consumer: disconnect, messages dropped]
  E -- no --> C
  A --> G{Responds to PING?}
  G -- yes --> C
  G -- no --> H[stale_connections: half-dead socket]
  H --> I[FDs and memory held with no useful work]

A useful mental split: stalled_clients is the server pushing data out and failing; stale_connections is the server checking liveness and failing. Different root causes, different blast radii. Never treat them as one combined “bad connections” number.

Both are cumulative counters since server start, so what matters is the rate of change, not the absolute value. A positive rate sustained over more than 5 minutes warrants investigation. A static non-zero value from last Tuesday’s deploy is history, not signal.

Common causes

Quick checks

All of these are read-only against the monitoring HTTP port (default 8222).

# 1. Current counter values
curl -s http://localhost:8222/varz | jq '{stale_connections, stalled_clients}'

# 2. Two snapshots 60s apart give you a rate, not a point value
curl -s http://localhost:8222/varz | jq '{stale_connections, stalled_clients, uptime}'
sleep 60
curl -s http://localhost:8222/varz | jq '{stale_connections, stalled_clients, uptime}'

# 3. Clients with the largest write backlog (pre-slow-consumer)
curl -s "http://localhost:8222/connz?sort=pending&limit=10" | jq '.connections[] | {cid, name, ip, pending_bytes, subscriptions, rtt}'

# 4. Has the slow consumer consequence started firing?
curl -s http://localhost:8222/varz | jq '{slow_consumers, slow_consumer_stats}'

# 5. If clustered: are routes backing up? (higher blast radius than clients)
curl -s http://localhost:8222/routez | jq '.routes[] | {rid, ip, pending_size, rtt}'

# 6. Churn: stable connections with fast-growing total_connections means reconnect loops
curl -s http://localhost:8222/varz | jq '{connections, total_connections}'

# 7. RTT outliers: high-RTT clients are prime slow consumer candidates
curl -s "http://localhost:8222/connz?sort=rtt&limit=10" | jq '.connections[] | {cid, name, ip, rtt}'

Two safety notes. /connz is expensive on servers with tens of thousands of connections; always use limit and sort parameters rather than pulling the full list repeatedly. And /varz counters reset on server restart, so a sudden drop to zero is a restart, not a recovery. Check uptime before celebrating.

How to diagnose it

1. Establish which counter is moving. Poll /varz twice, 60 seconds apart. If stalled_clients is growing, you have a write-path problem; go to step 2. If stale_connections is growing, you have a liveness problem; go to step 4. If both are moving, treat the write-path side first, since it precedes message loss.

2. Identify the stalled connections. Use /connz?sort=pending and look at the top entries by pending_bytes. Note whether the worst offenders are application clients or, if the sort output and /routez suggest it, routes or gateways. A route or gateway with growing pending bytes is a cluster-wide problem, not a client problem. See NATS slow consumer breakdown: clients vs routes vs gateways for the blast-radius analysis.

3. Confirm the trajectory. High pending bytes that clear on the next poll are burst absorption, which is normal. Pending bytes that grow across polls are a consumer falling behind. Check whether slow_consumers has started incrementing: once it does, NATS is disconnecting the connection and, in core NATS, dropping messages for it. At that point this article hands off to NATS slow consumer detected.

4. For stale connections, find the common factor. The question is why the peer stopped answering. Pull the affected connections from /connz and look for patterns in ip, name, or account. A cluster of stale clients from one network segment points at a middlebox or partition. A single stale client with an inflated RTT before going silent points at a hung process on that host.

5. Check the client side. For a hung client, verify process state on the client host: CPU starvation, a long GC pause, or a deadlock in the message handler will all stop protocol responses while leaving TCP open. For suspected network issues, check whether the path involves a NAT, VPN, or L4 load balancer with an idle-flow timeout shorter than the NATS ping interval.

6. Quantify the resource cost. Each stale connection holds a file descriptor plus per-connection buffers and goroutines. If the stale count is large, compare connections against max_connections and check the process FD count against ulimit -n. This is how a quiet stale-connection problem becomes an FD exhaustion cliff later.

Metrics and signals to monitor

Fixes

Fix the slow consumer behind the stalls

The stalled state is the server correctly applying backpressure; the fault is on the connection that cannot keep up. The durable fixes are on the consumer side: move blocking work (database writes, synchronous I/O) out of the message handler, scale the subscriber horizontally (queue groups for core NATS), or reduce fan-out onto the struggling consumer. For JetStream workloads, prefer pull consumers, which control their own consumption rate and are not subject to push-based slow consumer flagging.

Raising write_deadline is a common workaround for transient bursts. Understand the tradeoff: it postpones the stall and the eventual slow consumer disconnect, but it does not fix a consumer that is structurally slower than its message rate. It only trades earlier detection for larger in-memory backlogs.

Reap the stale connections and fix what created them

A connection that has failed ping/pong is closed by the server once detected, which releases its FD and buffers. If stale_connections keeps growing, the interesting question is why clients keep entering that state. For hung client processes, fix the client: the server cannot make a deadlocked process answer pings. For network paths, look for idle-flow timeouts on NAT devices, firewalls, or load balancers between client and server, and make sure the client library’s ping interval is comfortably shorter than any middlebox timeout.

Do not restart the NATS server as a first response to either counter. The server is reporting these states accurately; restarting destroys the evidence and usually re-creates the condition when the same clients reconnect.

Reduce reconnect churn

If stalls are driving disconnect-reconnect loops (fast-growing total_connections with a flat connections count), the churn itself adds CPU and memory pressure. Fixing the slow consumer stops the loop. If you need a temporary pressure valve, shedding the worst offending subscriptions or pausing a non-critical publisher is safer than anything server-side.

Prevention

• Alert on rate, not value. Both counters are cumulative. Alert on a positive delta sustained over 5 minutes, and treat a reset to zero as a restart event worth correlating with unexpected uptime resets.
• Monitor the precursor, not just the consequence. Track top-N pending_bytes via /connz?sort=pending alongside stalled_clients. Pending bytes rise before the stall counter moves, which is before the slow consumer disconnect.
• Keep client ping intervals shorter than middlebox idle timeouts. Any NAT, VPN, or L4 proxy in the path with an idle timeout shorter than the protocol ping interval will manufacture stale connections.
• Separate client from route/gateway distress. Write-path problems on routes and gateways have cluster-wide blast radius. Watch /routez pending_size and slow_consumer_stats so the breakdown is never a surprise.
• Size FD headroom for the stale-connection case. Keep total connections well under both max_connections and ulimit -n, because stale connections consume slots while doing nothing.
• Avoid middleboxes in the data path where possible. NATS clients use persistent TCP connections; L7 proxies and aggressive L4 idle reaping are recurring sources of half-dead sockets.

How Netdata helps

Netdata’s NATS collector polls the server’s HTTP monitoring endpoints and charts the signals around this problem, which shortens the correlation work:

• Slow consumers, including the per-type breakdown (clients, routes, gateways, leafs), so you can see immediately whether write-path distress is a client issue or a cluster issue.
• Connection count and churn (connections and total_connections rates), which makes disconnect-reconnect spirals visible next to the slow consumer events that cause them.
• Throughput (in/out messages and bytes), so you can line up stall onset with fan-out spikes and check for out_msgs dropping relative to in_msgs.
• Server memory and CPU, letting you confirm whether stale connections and stalled buffers are translating into real resource pressure.
• Per-second granularity, which matters here because pending-buffer spikes and brief stall windows are easy to miss at 60-second scrape intervals.

One gap: Netdata’s NATS collector does not currently chart stalled_clients or stale_connections from /varz. Poll those two fields directly with the curl commands above, or wire them into your own scrape job, and use Netdata’s surrounding signals (slow consumers, churn, memory) to corroborate what you find.

Related guides

• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check
• How NATS actually works in production: a mental model for operators
• NATS monitoring checklist: the signals every production server needs
• NATS monitoring maturity model: from survival to expert
• NATS server not responding: healthz failing and the process down or hung
• NATS slow consumer breakdown: clients vs routes vs gateways and blast radius
• NATS slow consumer detected: the write buffer overflowed and messages were dropped