NATS route RTT high: inter-server latency that triggers Raft elections

You are looking at a JetStream cluster that keeps electing new leaders. The /jsz meta cluster leader changes every few minutes, stream writes intermittently fail with API errors, and nats server list or /routez shows round-trip times between servers far above what a same-datacenter cluster should produce. Server CPU and memory look fine. Nothing has crashed. Yet the cluster cannot hold a stable leader.

This is the classic signature of elevated route RTT. Cluster routes are the TCP connections over which NATS servers exchange messages, subscription interest, and JetStream Raft traffic. When latency on those connections rises, two things degrade at once: cross-server message delivery slows, and Raft heartbeats start arriving late. Late heartbeats tip followers into starting elections. Sustained high route RTT is one of the leading causes of election storms, and it is routinely missed because teams monitor route existence (“are routes connected?”) but not route quality.

The diagnostic path is short. /routez gives you per-route RTT directly, and the cause is almost always one of a small set: a service mesh sidecar in the path, a noisy-neighbor network, cross-AZ placement you did not intend, or a host-level problem (CPU steal, GC pauses, slow disk) that delays the ping/pong responses RTT is measured with.

What this means

In a NATS cluster, every server maintains a full-mesh TCP route to every other server. Each route carries forwarded messages, subscription interest propagation, and, for clustered JetStream, the Raft protocol traffic for the meta group and every replicated stream and consumer group with peers on that server. The server measures RTT on each route using NATS protocol ping/pong frames and exposes it per route in /routez.

That RTT matters because of how Raft liveness works. Raft leaders send heartbeats roughly every second, and followers start a new election if they go without a heartbeat for a randomized election timeout in the range of a few seconds. If the network path adds enough latency and jitter that heartbeats arrive late, followers conclude the leader is dead and call an election. The election pauses writes for that Raft group. If the meta group leader was also leading several stream Raft groups, those groups elect too, and a single latency problem becomes a cluster-wide flurry of elections.

The disconnection threshold is much higher than the election threshold. A route is only torn down when it stops responding to pings for longer than the configured ping_interval (default 2 minutes). So the dangerous operating zone is wide: RTT can be far below the disconnection threshold and still be wrecking Raft stability. A route can look perfectly “connected” in /varz while elections fire continuously underneath.

flowchart TD
  A[Route RTT elevated
mesh sidecar, noisy network,
CPU steal, GC pauses, slow disk] --> B[Raft heartbeats arrive late]
  B --> C[Follower election timeout expires]
  C --> D[New election for meta group
or stream Raft groups]
  D --> E[Writes pause for affected groups]
  D --> F[Meta leader change cascades
into stream leader elections]
  F --> D
  E --> G[api.errors rising, api.inflight high,
publish and admin timeouts]
  A --> H[Cross-server delivery latency up
route pending_size growing]

Common causes

One nuance on measurement: RTT is measured with protocol ping/pong, so it includes processing delay on both ends, not just wire time. A peer that is CPU-starved or under GC pressure will show inflated RTT even on a healthy network. A single GC-induced spike is not the concern; sustained elevation is.

Quick checks

All of these are read-only.

# Per-route RTT, the primary field for this symptom
curl -s http://localhost:8222/routez | jq '.routes[] | {rid, remote_id, ip, rtt, pending_size}'

# Route count: in an N-server cluster each server should have N-1 routes
curl -s http://localhost:8222/varz | jq .routes

# Meta cluster leader and peer health
curl -s http://localhost:8222/jsz | jq '.meta_cluster | {leader, replicas: [.replicas[]? | {name, current, offline, lag}]}'

# JetStream API pressure: elections show up as rising errors and inflight
curl -s http://localhost:8222/jsz | jq '{api_total: .api.total, api_errors: .api.errors, api_inflight: .api.inflight}'

# Slow consumers broken down by type; routes non-zero is high blast radius
curl -s http://localhost:8222/varz | jq '{slow_consumers, slow_consumer_stats}'

# CPU and memory on each peer, to spot a starved or GC-thrashing server
curl -s http://localhost:8222/varz | jq '{cpu, cores, mem, uptime}'

Check the server logs for election activity. Rapid alternation of “stepping down” and “new leader” entries is the election-storm pattern. Count occurrences over a window rather than eyeballing.

If you have the NATS CLI, nats server list shows route RTT between cluster members in one view, the fastest way to see pairwise latency across the whole cluster. Use nats stream info STREAM --json | jq '.cluster' for per-stream Raft leaders and replica lag.

How to diagnose it

1. Confirm the RTT elevation is sustained, not a spike. Poll /routez every 10-15 seconds for a few minutes. A single GC pause inflates one sample; a real problem shows sustained elevation. Note which route (which peer pair) is elevated. One bad route points at that server or its network path. All routes elevated on one server points at that server’s host.

2. Correlate RTT with election events. Take the timestamps of leader changes from the logs and from repeated /jsz meta cluster polls. If elections cluster around RTT spikes, latency is the trigger. If elections happen with flat, low RTT, look at disk I/O or CPU on the leader instead.

3. Check whether a proxy is in the path. If the cluster runs in Kubernetes with a service mesh, determine whether route connections traverse sidecar proxies. Sidecars add per-hop latency and can buffer or delay packets, which both inflates RTT and produces false slow-consumer events on route connections. In a same-DC cluster where baseline should be under 1ms, even a millisecond or two of proxy overhead per hop is a large relative change. Check slow_consumer_stats for non-zero routes as supporting evidence.

4. Rule out host-level causes on the peer. For the server on the far end of the slow route, check /varz cpu against cores, memory trend, and uptime. On the host, check steal time (oversubscribed VMs are a classic cause of Raft instability), network retransmits, and disk latency on the JetStream storage path. Raft log writes are latency-sensitive; network-attached storage with variable latency is a leading cause of election storms and can present alongside elevated RTT.

5. Check clock sync. Raft does not depend on wall-clock agreement, but heartbeat and election timers run on local clocks. Significant skew between servers makes timeout behavior inconsistent. Verify NTP state on all members.

6. Assess blast radius. List per-stream Raft groups with /raftz and check which streams have non-current or lagging replicas. A meta group flap is disruptive; meta plus several stream groups flapping is a write outage for those streams.

Metrics and signals to monitor

Fixes

Remove the proxy from the route path. If sidecars are the cause, exclude NATS cluster traffic from mesh interception. NATS speaks its own TCP protocol over persistent connections; L7 proxy features buy you nothing on routes and cost latency and false slow-consumer detection. How you exclude a port or pod from interception is mesh-specific, so follow your mesh’s documentation. Tradeoff: you lose whatever mTLS and telemetry the mesh was providing on those connections, so supply TLS for routes through NATS configuration instead if you need encryption in transit.

Fix placement. If servers landed in different zones or regions unintentionally, reschedule them so cluster members sit where your latency budget assumes they sit. Same-DC clusters should see sub-millisecond route RTT. If you intentionally run stretch clusters across zones, accept 1-5ms as the floor and treat election frequency as the metric to watch, because your margin before the election timeout is thinner.

Relieve the starved peer. If one server shows inflated RTT to all peers plus high CPU or GC pressure, reduce its load: rebalance stream leaders, move JetStream assets, or give it more CPU. In containers, make sure CPU limits are not throttling the server and that Go’s processor count matches the cgroup limit rather than the host. Raft election timeouts are internal to the server and not exposed as configuration, so you cannot tune your way out of a starved host; you have to fix the resource problem.

Fix the storage path. If disk latency is delaying Raft log appends, move JetStream storage to local SSDs. This requires a planned migration of the affected nodes; do it one at a time so the cluster keeps quorum.

Tune ping_interval only as a last resort, and know what it does. Lowering ping_interval makes dead-route detection faster but does nothing for Raft stability, and raising it to tolerate latency only delays disconnection while elections keep firing. The RTT problem itself is the fix target.

Do not restart servers as a first move. A restart triggers elections for every Raft group that had its leader on that node, adding churn on top of the instability you already have. If the storm is fully self-sustaining, a careful rolling restart, one node at a time with quorum confirmed between steps, can break the cycle, but only after you have identified the underlying cause, otherwise it returns.

Prevention

• Alert on route RTT, not just route count. Scrape /routez rtt per route at a 10-15 second interval and alert on sustained elevation from baseline. Route count only tells you about partitions after the fact.
• Keep NATS cluster traffic out of the service mesh by policy, so a future mesh rollout does not silently reintroduce the latency.
• Track meta leader stability as a metric. Elections per hour should be zero in steady state. Any sustained non-zero rate deserves a ticket before it becomes a storm.
• Watch route pending_size and slow_consumer_stats.routes. These are the leading indicators that inter-server traffic is degrading before RTT alone makes it obvious.
• Pin cluster members to low-latency placement and local storage. Enforce zone affinity and local SSD for JetStream in your platform templates rather than relying on scheduling luck.
• Baseline your RTT by peer pair. Cross-AZ pairs have a higher floor than same-rack pairs; alert on deviation from the pair’s own baseline, not a single global number.

How Netdata helps

• Netdata’s NATS collector polls the monitoring endpoints continuously, so route count, slow consumer counts, and server CPU and memory are charted together at per-second granularity, which lets you line an RTT spike up with a CPU or GC event.
• Slow consumer tracking, including the breakdown by connection type, surfaces route-level backpressure before it escalates into disconnections and elections.
• JetStream API totals, errors, and inflight requests are collected from /jsz, so the downstream symptom of elections (API pressure) is visible on the same dashboard as the server-level resource signals.
• Server uptime and health from /varz and /healthz let you rule out restarts as a confounder when correlating election bursts.
• Because Netdata also collects host-level metrics (CPU steal, disk latency, network retransmits) on the same nodes, you can correlate an inflated route RTT with the underlying host cause without switching tools.

Related guides

• How NATS actually works in production: a mental model for operators
• NATS connection churn: a stable connection count hiding constant reconnects
• NATS connection storm: reconnect thundering herd after a network event
• NATS context deadline exceeded: JetStream publish and request timeouts
• NATS JetStream consumer lag growing: falling behind the stream
• NATS JetStream redelivery loop: num_redelivered climbing and messages reprocessed
• NATS JetStream AckWait tuning: matching the ack timeout to processing time
• NATS consumer stalled at MaxAckPending: delivery stops until messages are acked
• NATS JetStream consumer stopped receiving messages: the diagnostic tree
• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS file descriptor exhaustion: too many open files and the ulimit cliff
• NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check