NATS leaf node disconnected: an edge server isolated from the hub

A leaf node connection is the single TCP session that ties an edge NATS server to your hub cluster. When it drops, the edge site keeps running locally, but it is cut off from the rest of the messaging fabric. Subscribers on the edge stop receiving messages published at the hub, and subscribers at the hub stop receiving anything published at the edge.

The blast radius is easy to underestimate. A leaf connection multiplexes traffic for potentially many accounts over one logical connection. One dropped leaf can silently sever dozens of application-level communication paths while every local health check on both servers still returns ok.

This guide covers how to confirm the disconnect, isolate the cause (network, credentials, or backpressure), and understand what happened to your messages during the gap.

What this means

Leaf nodes reconnect automatically, so a brief blip self-heals. The operational problem is a sustained disconnect, or worse, a flapping reconnect loop where the leaf connects, falls behind, gets disconnected, and repeats.

During the disconnect window:

• Core NATS messages are lost. Messages published on either side with subscribers on the other side have nowhere to go. There is no store-and-forward for core NATS over a leaf link.
• JetStream traffic is delayed, not lost. Streams that source or mirror across the leaf link resume syncing after the connection re-establishes, with a recovery interval on the order of 10 to 20 seconds before the sync restarts.

flowchart LR
    subgraph Edge site
        EPUB[Edge publishers] --> ES[Edge NATS server]
        ES --> ESUB[Edge subscribers]
    end
    subgraph Hub
        HS[Hub NATS cluster] --> HSUB[Hub subscribers]
        HPUB[Hub publishers] --> HS
    end
    ES -- "leaf connection :7422 (many accounts multiplexed)" --> HS
    LS{Link down?} -.->|network loss| ES
    LS -.->|credential expiry| HS
    LS -.->|slow consumer on leaf| HS

Because the leaf is one connection carrying many accounts, neither server’s /healthz tells you the leaf is down. Process health is fine on both sides. The only authoritative sources are the hub’s monitoring endpoints and the edge server’s logs.

Common causes

Quick checks

Run these on the hub unless noted. All are read-only.

# 1. How many leaf connections does the hub see right now?
curl -s http://localhost:8222/varz | jq '.leafnodes'

# 2. Per-leaf detail: which account, which remote, what RTT
curl -s http://localhost:8222/leafz | jq '.leafs[] | {name, account, ip, rtt, in_msgs, out_msgs, subscriptions}'

# 3. Is the hub flagging the leaf as a slow consumer?
curl -s http://localhost:8222/varz | jq '{slow_consumers, slow_consumer_stats}'

# 4. Auth failures on the hub (credential expiry, wrong creds)
grep "Authorization Violation" /var/log/nats/nats-server.log | tail -20

# 5. From the edge host: can it even reach the hub leafnode port?
#    (default leafnode port is 7422)
timeout 5 bash -c 'cat < /dev/null > /dev/tcp/HUB_HOST/7422' && echo "reachable" || echo "unreachable"

Notes on reading the output:

• /varz .leafnodes is the current connected count. Compare it against the number of edge sites you expect. Zero when you expect one or more is the confirmed disconnect.
• /leafz shows only currently connected leaves, so a missing edge site is your confirmation. RTT is a Go duration string (for example "1.234ms"); a rising RTT trend before the drop points at WAN degradation.
• slow_consumer_stats.leafs non-zero means the hub has flagged the leaf connection itself as a slow consumer. That is a different problem from a network outage and needs a different fix.

How to diagnose it

1. Confirm the disconnect and its duration. Check /varz .leafnodes on the hub against the expected count. Then check hub and edge logs for the disconnect event and any reconnect attempts. A leaf that disconnected once and reconnected 30 seconds ago is a different incident from one that has been flapping for an hour.

2. Determine which side initiated the drop. The edge server’s log shows whether it lost the connection (network path) or was closed by the hub (slow consumer, auth). The hub’s log shows the inverse. This single observation cuts the cause list roughly in half.

3. Check for slow consumer evidence on the hub. Pull /varz and look at slow_consumer_stats.leafs. If it is incrementing and correlates with the disconnect timestamps, the hub could not write to the leaf fast enough. Causes are either a congested WAN link or traffic volume that exceeds link capacity. On bandwidth-constrained links there is a known nasty failure mode: hub-to-leaf data saturates the hub’s write buffer for the leaf connection, PONG responses queue behind the data, the leaf stops seeing liveness traffic, and it declares the connection stale. The leaf disconnects, reconnects, hits the same backlog, and loops. With default ping behavior (ping interval of 2 minutes, 2 missed pings), each cycle takes minutes to detect.

4. Rule out credentials. If the leaf reconnects TCP but gets closed immediately, check hub logs for Authorization Violation around the reconnect timestamps. Leaf nodes authenticate with credentials (JWT/NKey in operator mode, or user/pass, or TLS). An expired JWT or rotated credential on one side produces exactly this pattern: connect, auth fail, close, retry forever.

5. Check WAN quality. If the leaf is currently connected, watch its RTT in /leafz over several minutes and compare to baseline. If it is disconnected, test the raw path from the edge host: TCP reachability to the hub leafnode port, packet loss, latency. On very high-latency links (satellite-class, ~500 ms RTT), older NATS versions had a hardcoded 1-second timeout waiting for the hub’s INFO during leaf connect, which made the link effectively unusable; newer versions expose this as a configurable first_info_timeout.

6. Assess the message impact. For core NATS subjects crossing the leaf, messages published during the gap are gone. For JetStream, check any streams that source or mirror across this leaf: after reconnect, give them the 10 to 20 second recovery interval, then verify sync is actually resuming (lag shrinking, active recent). There are reports of versions where locally stored messages did not sync back to the hub after a leaf reconnected, so verify rather than assume.

7. Check for a stale zombie connection. In older server versions there was a bug where continuous local client traffic on the edge suppressed outbound PINGs on the leaf connection, so the edge never detected a dead link and sat in a zombie state until TCP keepalive fired (default hours). Fixed upstream in late 2022. If you are on a version older than that and the edge shows the leaf as connected while the hub does not, this is a prime suspect. Restarting the edge server clears it, but upgrading is the real fix.

Metrics and signals to monitor

Fixes

Network path between edge and hub

Restore the underlying connectivity: routing, firewall rules for the leafnode port, VPN or tunnel health. If the link is inherently high-latency, run a server version with configurable first_info_timeout and raise it for that remote. If the link is lossy, expect flapping and consider tuning the leaf’s reconnect interval so the edge retries aggressively without hammering the hub.

Credential expiry

Renew or reissue the leaf’s credentials and reload or restart the edge server so it picks them up. If you run in operator/JWT mode, track credential expiry the same way you track TLS certificate expiry; an expired leaf JWT takes an entire site offline and the only symptom is auth failures in the hub log. Alert on days-to-expiry before it becomes an incident.

Slow consumer on the leaf connection

The hub is enforcing backpressure correctly; the leaf link cannot drain what the hub is sending. Options, in order of preference:

1. Reduce hub-to-leaf traffic. Audit which subscriptions at the edge are pulling traffic across the leaf. Leaf nodes propagate interest, so anything subscribed at the edge pulls matching traffic over the link. Removing a chatty subscription is often the whole fix.
2. Increase link capacity or reduce competing traffic on the WAN path.
3. Move bursty or high-volume flows to JetStream with sourcing or mirroring, so cross-site transfer is pulled and rate-controlled by the stream machinery rather than pushed over the live leaf connection.

Do not respond to leaf slow consumers by simply raising write deadlines. That delays the disconnect while the buffer grows; on a constrained link it makes the eventual failure bigger.

Flapping reconnect loop on a constrained link

If you are in the PONG-starvation cycle described above, the only reliable levers today are reducing hub-to-leaf volume below link capacity or moving bulk transfer to JetStream sourcing. Treat the loop as a capacity problem, not a NATS bug you can tune around.

Prevention

• Alert on leaf count, not just server health. /varz .leafnodes below expected for more than 60 seconds should page or ticket depending on how critical the edge site is. Server health on both ends stays green during a leaf outage, so process checks alone will never catch this.
• Baseline per-leaf RTT and alert on sustained deviation. Latency drift is your earliest warning of WAN degradation.
• Track slow_consumer_stats.leafs as its own signal. A leaf slow consumer has site-wide blast radius, unlike a client slow consumer.
• Monitor leaf credential and TLS expiry with the same 30-day / 7-day escalation you use for certificates.
• Keep both hub and edge servers current. Several leaf-specific liveness bugs (PING suppression, INFO timeout on high-latency links) and recent leafnode-related CVEs are fixed in recent releases. Old versions at the edge, where they are easy to forget, are a recurring source of exactly this incident.
• Decide explicitly which traffic may cross the leaf. Core NATS subjects crossing a leaf are lossy during any disconnect. Anything that must survive a WAN event should be on JetStream with sourcing or mirroring.

How Netdata helps

• Netdata’s NATS collector polls the monitoring endpoints and tracks the leafnodes count from /varz over time, so a dropped leaf shows up as a step change rather than something you discover from user reports.
• Slow consumer totals from /varz are collected continuously, letting you correlate leaf disconnect timestamps with slow consumer events to confirm a backpressure cause.
• Because Netdata also collects system metrics from the same host, you can overlay leaf disconnects with network interface errors, saturation, and packet loss on the hub or edge machine to separate a local NIC problem from a WAN problem.
• Per-second collection catches short flap cycles that a 60-second scrape interval would average into invisibility.
• Connection churn tracking on both servers helps you distinguish a leaf-specific event from a wider restart or reconnection storm at the edge site.

Related guides

• NATS connection churn: a stable connection count hiding constant reconnects
• NATS connection storm: reconnect thundering herd after a network event
• How NATS actually works in production: a mental model for operators
• NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check
• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS file descriptor exhaustion: too many open files and the ulimit cliff
• NATS JetStream consumer lag growing: falling behind the stream
• NATS consumer stalled at MaxAckPending: delivery stops until messages are acked
• NATS JetStream redelivery loop: num_redelivered climbing and messages reprocessed
• NATS JetStream consumer stopped receiving messages: the diagnostic tree
• NATS JetStream AckWait tuning: matching the ack timeout to processing time
• NATS context deadline exceeded: JetStream publish and request timeouts