NATS gateway disconnected: cross-cluster traffic cut in a supercluster

Subscribers in cluster A have stopped receiving messages published in cluster B. Publishers see no errors. Every server’s /healthz returns ok, CPU and memory look normal, and yet a whole class of traffic has silently stopped flowing between two sites. In a NATS supercluster, this is the signature of a gateway disconnection: the outbound gateway connection to the remote cluster is missing, so no cross-cluster forwarding happens at all.

Gateways are the links between independent NATS clusters. Unlike cluster routes (full mesh between servers in one cluster), each server maintains gateway connections keyed by remote cluster name, and all cross-cluster delivery for an account flows through them. When one goes down, the failure is clean and quiet: local delivery keeps working, remote delivery stops, and nothing in the server-wide error counters necessarily moves.

This guide covers how to confirm a gateway is actually disconnected (versus merely idle), what typically cuts the connection, and how to restore and protect cross-cluster traffic.

What this means

Each NATS server in a supercluster tracks its gateway connections in two maps on the /gatewayz monitoring endpoint: outbound_gateways and inbound_gateways, both keyed by remote cluster name. Outbound is what your cluster initiated; inbound is what the remote cluster initiated toward you. Every configured remote cluster should appear. A configured gateway missing from outbound_gateways means cross-cluster delivery to that cluster is failing.

Two things make gateway incidents confusing:

• Zero traffic is not a symptom. Gateways run in interest-only mode: messages for a subject only cross the gateway when the remote cluster has registered interest. If no client in cluster B subscribes to a subject, cluster A correctly sends nothing. Flat gateway traffic counters can be completely normal.
• The server looks healthy. /healthz passes, local pub/sub works, and JetStream (if present) keeps serving local consumers. The only broken thing is the path between clusters.

After a gateway reconnects, there is a brief flood phase before interest-only mode re-converges, during which more messages are forwarded than strictly needed. Expect a temporary bandwidth spike on recovery. A gateway stuck out of interest-only convergence for a long time is itself a warning sign.

Common causes

Quick checks

All read-only, run against the local monitoring port (default 8222):

# Which remote clusters does this server see, outbound and inbound?
curl -s http://localhost:8222/gatewayz | jq '{outbound: (.outbound_gateways | keys), inbound: (.inbound_gateways | keys)}'

# Detail per outbound gateway: is the configured remote cluster present and connected?
curl -s http://localhost:8222/gatewayz | jq '.outbound_gateways'

# Is this server itself healthy and not freshly restarted?
curl -s http://localhost:8222/varz | jq '{uptime, connections, routes}'

# Throughput: did out_msgs drop while in_msgs continues? Cross-cluster fan-out gone.
curl -s http://localhost:8222/varz | jq '{in_msgs, out_msgs}'

# Gateway slow consumer events (a gateway under backpressure can precede a drop)
curl -s http://localhost:8222/varz | jq '{slow_consumers, slow_consumer_stats}'

Interpretation notes:

• Compare .outbound_gateways | keys against the remote clusters in your gateways configuration. The sets should match. A missing configured gateway is the incident.
• A gateway that is present but whose traffic counters are flat is not proof of a problem. Check whether the remote cluster actually has interest (subscriptions) for the subjects you expect to flow.
• slow_consumer_stats breaks slow consumers down by connection type. A non-zero gateway count means the gateway connection itself was falling behind, which has a much bigger blast radius than a slow client and often precedes or accompanies gateway instability. The per-type breakdown is in /varz and is available since nats-server v2.10.0.

How to diagnose it

flowchart TD
  A[Cross-cluster delivery stopped] --> B{Remote cluster in outbound_gateways?}
  B -- Yes, present --> C{Traffic flat?}
  C -- Yes --> D[Check remote interest. Likely normal: no subscribers]
  C -- No --> E[Gateway up. Look at remote consumers or subject mismatch]
  B -- Missing --> F{Remote cluster reachable?}
  F -- No --> G[Network partition or remote cluster down]
  F -- Yes --> H{TLS or config errors in log?}
  H -- Yes --> I[Fix certs / CA bundle / gateway URLs]
  H -- No --> J[Stuck reconnect: check server version]

1. Confirm the gap. Run the /gatewayz key listing above on more than one server in the local cluster. If the remote cluster name is absent from outbound_gateways everywhere, the gateway link is down. If it is absent on one server only, scope the problem to that server.
2. Rule out the zero-traffic false alarm. If the gateway entry exists but counters are flat, verify that a subscriber actually exists on the remote side for the subject in question. Interest-only mode means no interest, no traffic. Many “gateway down” reports end here.
3. Check reachability. From the local server, test TCP connectivity to the remote cluster’s gateway listen address and port. Failure here points to a cross-DC partition, firewall change, or the remote cluster being down. Also check the remote cluster’s own /healthz and uptime: a remote cluster mid-restart or crash-looping explains a missing gateway with no local fault.
4. Read the server log around the drop. Look for gateway connection attempts without a corresponding successful registration, and for TLS handshake failures on the gateway port. TLS failures after a certificate change on either side are a classic cause: if the remote cluster rotated its certificate and your CA bundle only trusts the old CA, the connection is severed at handshake.
5. Verify gateway configuration symmetry. Every server in a cluster must use the same gateway name, and gateway URLs must be reachable from every gateway node in both directions. A name mismatch or an advertised address that peers cannot route to (for example, a pod-internal IP advertised instead of the reachable external address) will keep connections from establishing.
6. Check the server version for the stuck-reconnect bug. On nats-server v2.10.14 and earlier, an outbound gateway connection could get stuck indefinitely during packet loss: the PING timers that detect a dead connection only started after the first INFO response from the remote, so a lost handshake left the connection hanging forever. The fix shipped in v2.11.0. If you are on an affected version and the gateway is stuck rather than cleanly refused, this is likely your cause.

Metrics and signals to monitor

Fixes

Network partition or remote cluster down

Restore connectivity or bring the remote cluster back; the gateway reconnects on its own once the path is up. Do not restart local servers as a first move: it does nothing for a partition and adds a client reconnect storm on top. If the partition is recurrent, treat it as a network engineering problem (flow logs, inter-DC link health) rather than a NATS problem.

Remote cluster mid-restart

Wait for the remote cluster to finish recovering. Note the flood phase after reconnection: expect a short bandwidth spike while interest re-converges, and do not mistake it for a leak.

Gateway URL or name misconfiguration

Fix the gateways block so URLs point at reachable gateway listen addresses, and ensure the gateway name is identical on every server in the cluster. In Kubernetes deployments, make sure the address each server advertises for gateway connections is the address peers can actually reach (a load balancer or external IP), not a pod-internal IP. Operators have hit a failure mode where a configured or discovered gateway URL contains a pod-internal IP, so peers cannot reach it and reconnect attempts keep failing. Setting an explicit reachable advertise address and doing a rolling restart of all clusters resolved it. In current nats-server releases, gateway { advertise } overrides the host and port announced in gateway INFO; the soliciting server tries every URL in a remote gateway’s list in one attempt before waiting one second for the next attempt.

TLS misconfiguration

Align the CA bundle on both sides so each cluster trusts the other’s current certificate. When rotating certificates across a supercluster, use a bundle containing both the old and new CAs and rotate one cluster at a time; rotating the remote side first with a CA bundle that only trusts the old CA severs the gateway immediately. Track expiry in advance (see the monitoring table) so this never becomes the incident cause.

Stuck outbound connection on v2.10.14 or earlier

Upgrade to v2.11.0 or later, where gateway PING timers start before the first INFO response and dead half-open connections are detected and retried. As an interim measure on an affected version, restarting the stuck server forces a fresh connection attempt, but treat that as a workaround, not the fix, and schedule the upgrade. If reconnect storms after recovery are a concern, nats-server v2.12.0 and later support gateway { connect_backoff: true } for exponential connection-attempt backoff; the option applies to both explicit and discovered gateway entries. gateway { connect_retries }, however, limits only implicit gateways.

Prevention

• Alert on topology, not traffic. Page-worthy condition: a configured remote cluster missing from /gatewayz outbound connections, sustained more than 60 seconds. Do not alert on zero gateway traffic; that is interest-only mode working as designed.
• Monitor gateway slow consumers separately. slow_consumer_stats.gateways should be zero. Any rate of change means cross-cluster delivery is backing up before it breaks.
• Coordinate certificate rotation across clusters. One rotation calendar, shared CA bundles during transitions, and expiry alerting well before 30 days.
• Keep servers current. The stuck-outbound-gateway bug class is fixed in maintained versions; running old patch releases across a supercluster is an avoidable risk.
• Test the failure. Periodically verify, in staging, what your alerting does when a gateway drops and that dashboards distinguish “gateway down” from “no cross-cluster interest.”

How Netdata helps

• Netdata polls the NATS monitoring endpoints and charts in_msgs/out_msgs rates per server, so the moment cross-cluster fan-out disappears (out_msgs dropping while in_msgs holds) shows up as a visible divergence across both clusters’ dashboards.
• Slow consumer counters, including the per-type breakdown, are collected over time, letting you see gateway-level backpressure building before a disconnection rather than after.
• Uptime tracking across all supercluster nodes makes correlated restarts obvious, separating a shared event (cert rotation, network maintenance) from a single-node fault.
• Because Netdata collects at per-second granularity, the brief flood-phase bandwidth spike after a gateway reconnect is distinguishable from a sustained traffic anomaly, reducing false alarms during recovery.
• Correlating NATS signals with host network metrics (interface errors, retransmits, link drops on the inter-DC path) shortens the “is it NATS or is it the network” loop that gateway incidents always trigger.

Related guides

• How NATS actually works in production: a mental model for operators
• NATS connection churn: a stable connection count hiding constant reconnects
• NATS connection storm: reconnect thundering herd after a network event
• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check
• NATS file descriptor exhaustion: too many open files and the ulimit cliff
• NATS JetStream consumer lag growing: falling behind the stream
• NATS consumer stalled at MaxAckPending: delivery stops until messages are acked
• NATS JetStream redelivery loop: num_redelivered climbing and messages reprocessed
• NATS JetStream consumer stopped receiving messages: the diagnostic tree
• NATS JetStream AckWait tuning: matching the ack timeout to processing time
• NATS context deadline exceeded: JetStream publish and request timeouts