$ guides / nats / nats-silent-message-loss ▌

Operations Guides

NATS silent message loss: zero-subscriber drops and the in/out asymmetry

Your NATS server is healthy. Health checks pass, connection counts are stable, there are no slow consumers, no errors in the logs. And yet a downstream service insists it has not received a single message in the last hour, while the producer’s metrics show it published thousands.

Both are telling the truth. In core NATS, a message published to a subject with zero subscribers is discarded at the moment of publish. The publisher gets no error. The server writes no log line. No counter records the drop. The message simply does not exist anymore.

This is the most dangerous failure mode in core NATS precisely because it is not a failure. It is the designed behavior of an at-most-once, fire-and-forget message router. Teams migrating from Kafka or RabbitMQ, where a published message is durably stored whether or not anyone is listening, get blindsided the first time it happens. The server is working correctly; your messages are gone.

This article covers how the drop works, how to detect it with the only signal that exists (the asymmetry between in_msgs and out_msgs), how to find the specific subject or subscriber involved, and when the right answer is JetStream instead of more monitoring.

What this means

Core NATS is a real-time router, not a queue. The server maintains an in-memory subject tree mapping subject strings to sets of subscriptions. Every inbound message is matched against that tree and written directly into each matching subscriber’s connection buffer, in real time. There is no intermediate storage between publisher and subscriber.

If the match set is empty, there is nothing to write to. Routing completes with zero destinations and the message is gone. The client’s publish call succeeds, because from the protocol’s perspective the server accepted and routed the message. There is no “messages discarded for lack of interest” metric. The drop is invisible at every layer except one: the aggregate message counters.

The server counts what it received (in_msgs) and what it delivered (out_msgs). In a healthy system, out_msgs >= in_msgs, because each published message fans out to one or more subscribers. When messages are dropped for lack of interest, in_msgs keeps climbing while out_msgs stays flat or falls below the expected fan-out. That asymmetry, adjusted for expected subscribers per subject, is the only server-side evidence that loss is happening.

flowchart LR
  P[Publisher] -->|PUBLISH orders.new| S[NATS server]
  S --> T{Subject tree match}
  T -->|subscriber connected| C1[Consumer A]
  T -->|subscriber connected| C2[Consumer B]
  T -->|zero subscribers| X[Message discarded]
  S --> V[/varz counters/]
  V --> IN[in_msgs increments]
  V --> OUT[out_msgs increments only on delivery]

The publish path is identical in both cases. Only the counter pair diverges.

The same invisible-drop behavior shows up in subtler forms worth knowing before you start diagnosing:

Cluster interest propagation lag. When a client subscribes, the subscription is advertised to other servers in the cluster, but propagation is not instantaneous and there is no signal that it completed. Messages published on another node before the interest arrives are dropped exactly as if no subscriber existed.
Slow consumers. A subscriber that cannot keep up is flagged as a slow consumer and messages for it are dropped (or the client is disconnected). Different mechanism, same result from the application’s perspective: messages vanished. The distinguishing signal is that slow consumer events increment the slow_consumers counter.
JetStream streams with interest retention and no consumers. If a stream uses interest-based retention and has no consumers, every published message is immediately eligible for deletion because there is no interest. The stream is a black hole while looking perfectly healthy.

Common causes

Cause	What it looks like	First thing to check
Subscriber never connected or crashed	`out_msgs` flat while `in_msgs` climbs; subscription count lower than expected	Is the consumer process running and connected? `/connz` for its connection
Subject mismatch between publisher and subscriber	One specific flow is silent, everything else healthy	Compare subject strings byte-for-byte; NATS subjects are case-sensitive
Queue group misconfiguration	One group member receives, the rest see nothing, or a second group is empty	Verify the queue group name matches on every intended member
Cluster interest propagation lag	Messages lost only in the seconds after a subscriber starts or reconnects	Correlate loss windows with subscriber connect/reconnect timestamps
Subscriber on a different server with routes down	Loss only for cross-node subject interest	Route count vs expected N-1 on every node
Publish-before-subscribe at startup	First messages after every deploy or restart are lost	Application startup ordering: does the subscriber subscribe before the publisher publishes?
Wrong retention mental model (JetStream interest retention, no consumers)	Stream accepts publishes but stores nothing	Stream info: retention policy and consumer count

Quick checks

All commands below are read-only against the monitoring HTTP port (default 8222).

# 1. Pull the in/out counters. Take two snapshots to compute rates.
curl -s http://localhost:8222/varz | jq '{in_msgs, out_msgs, in_bytes, out_bytes}'
sleep 10
curl -s http://localhost:8222/varz | jq '{in_msgs, out_msgs, in_bytes, out_bytes}'

# 2. Current subscription count. Lower than you expect?
curl -s http://localhost:8222/varz | jq '{subscriptions, connections}'

# 3. Confirm slow consumers are NOT the explanation.
curl -s http://localhost:8222/varz | jq '{slow_consumers}'

# 4. Verify the expected subscriber is connected and subscribed to the exact subject.
curl -s "http://localhost:8222/connz?subs=1" | jq '.connections[] | {cid, name, ip, subscriptions}'

# 5. If clustered, verify full-mesh routes (N-1 routes per node).
curl -s http://localhost:8222/varz | jq '{routes}'

Two cautions on check 4. First, /connz?subs=1 can be expensive on servers with high connection and subscription counts; use limit and offset if the server is large. Second, all counters are cumulative since server start, so a single snapshot tells you nothing. You need rates: two snapshots as above, or a monitoring system that computes them for you.

How to diagnose it

Confirm the asymmetry is real. Compute rate(in_msgs) and rate(out_msgs) over a few minutes. If publishers are active and out_msgs / in_msgs is below your expected fan-out ratio (or out_msgs is flat), messages are being dropped. If out_msgs is healthy, the loss is elsewhere: look at the subscriber application itself.
Rule out slow consumers. Check slow_consumers in /varz. If it is incrementing, you have a delivery problem (a subscriber that cannot keep up), not a zero-subscriber problem. That is a different diagnosis: see the pending bytes and slow consumer guides linked below.
Count subscriptions. Compare /varz subscriptions against the number your applications should be holding. A count below expected, combined with the asymmetry, points directly at missing subscribers. A sudden drop usually lines up with a deploy, a crash, or a config rollout.
Verify the specific subject. Find the subscriber’s connection in /connz?subs=1 and confirm it is subscribed to the exact subject the publisher uses. Subjects are case-sensitive, and wildcards that do not mean what the author thought (* matches one token, > matches one or more trailing tokens) are a classic cause. A subscriber on orders.new receives nothing published to Orders.new or orders.v2.new.
Check queue group membership. If the subscriber is part of a queue group, verify the group name is identical on every member. A typo splits the group: each distinct name becomes its own independent group, and one of them may have no members receiving the traffic you expect.
In a cluster, check route health. Each node should have N-1 routes. If routes are down, a subscriber connected to node B does not attract messages published on node A, and those publishes are dropped for lack of local and propagated interest. Suspect propagation lag if the loss window is only the first seconds after a subscriber starts or reconnects.
Check startup ordering. If the loss is a fixed number of messages after every deploy, reproduce it: publish to the subject before subscribing, then subscribe. The pre-subscription messages will not arrive. That is publish-before-subscribe loss, and the fix is application ordering, not server configuration.
If this is request-reply, use the no-responders signal. For request-reply patterns, NATS can return an immediate “no responders” reply when no service is subscribed to the request subject, instead of letting the requester hang until its own timeout. That converts invisible loss into an explicit, catchable error at the call site. It only covers request-reply; plain pub/sub has no equivalent. See the no responders guide linked below.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`rate(in_msgs)` vs `rate(out_msgs)` from `/varz`	The only server-side signal for zero-subscriber drops	`out_msgs / in_msgs` falls below expected fan-out, or `out_msgs` flat with active publishers
`subscriptions` from `/varz`	Missing subscribers are the root cause	Count below expected, or sudden drop correlated with a deploy
`slow_consumers` from `/varz`	Rules delivery problems in or out	Should be flat at zero; any increase means drops for connected subscribers
`connections` and `total_connections` delta	Subscriber churn creates repeated zero-interest windows	High churn: stable `connections` with fast-climbing `total_connections`
`routes` from `/varz`	Cross-node interest requires healthy routes	Below N-1 per node in a cluster, sustained more than 60s
Consumer-side delivery counters (application-level)	The server cannot see the loss; your app can	Gap between producer publish count and consumer receive count

The fan-out ratio deserves a concrete alerting recipe, because this is where most teams never finish the job. If you scrape /varz into a metrics system (the Prometheus exporter exposes these counters as gnatsd_varz_in_msgs and gnatsd_varz_out_msgs), an alert along these lines is the starting point:

# Alert when deliveries fall far below receives, meaning messages
# are being dropped for lack of interest. Adjust the ratio threshold
# to your expected minimum fan-out (every critical subject should
# have at least one subscriber).
rate(gnatsd_varz_in_msgs[5m]) > 10
  and on(instance)
rate(gnatsd_varz_out_msgs[5m]) / rate(gnatsd_varz_in_msgs[5m]) < 0.5

Thresholds are deliberately conservative and workload-dependent. Three things make this noisier than it looks:

The ratio is server-wide. It mixes all subjects. A healthy high-fan-out subject can mask a silently dying low-fan-out subject. For critical subjects, track health application-side (publish and receive counters per subject) rather than relying on the aggregate.
Cluster traffic distorts the counters. Cross-server route traffic contributes to the aggregate message and byte counters, so the naive ratio shifts with cluster topology and load. Establish the baseline ratio during known-healthy operation and alert on deviation from it, not on an absolute number.
JetStream internal traffic contributes too. On JetStream-enabled servers, internal messages are included in these counters. Compare like for like when you change what runs on the server.

Alert on deviation from your measured baseline, sustained for several minutes, as a ticket. Add one application-level rule: compare the producer’s published count to the consumer’s received count per critical subject. The server fundamentally cannot tell you which subject lost messages; only the endpoints can.

Fixes

Subscriber crashed or never connected

Get the consumer running and connected, then alert on subscriptions so the next occurrence pages a human instead of a downstream user. If the consumer crashes on startup because of a poison message or bad config, that is the real bug; the NATS drop was the symptom.

Note: the lost messages are not recoverable in core NATS. If that data mattered, you have just learned your reliability requirement. See the JetStream section below.

Subject or queue group mismatch

Fix the string. Make subjects and queue group names configuration values shared between publisher and subscriber rather than string literals on both sides. Add a startup self-check to the subscriber: after connecting, verify its own subscription registered (via the monitoring endpoint or an application-level handshake) before signaling readiness.

Reorder startup: subscribers connect and subscribe before publishers begin publishing. For request-reply, enable the no-responders mode so callers fail fast instead of timing out. For critical handoffs, use a readiness barrier in your deployment: subscriber ready before producer starts.

Cluster interest propagation lag

There is no server-side “interest has propagated” signal, so the practical mitigation is application-side: a short settle delay after subscribing before the service reports ready, or a readiness handshake in which the service receives a probe message before accepting traffic. Keep routes healthy; every route drop restarts interest convergence for the affected subjects.

When reliability is the actual requirement: JetStream

If any of the messages in question must not be lost, the correct fix is architectural: move that traffic to JetStream. JetStream persists messages to a stream independent of whether any consumer is connected, and consumers track delivery and acknowledgment state, so a disconnected consumer picks up where it left off. Core NATS is the right tool for the fast request path and for traffic where loss is tolerable; it is the wrong tool for anything you would describe as “the messages must get through.”

The tradeoffs are real: JetStream adds disk I/O, storage management, consumer lag as a new thing to monitor, and (in clustered mode) Raft consensus. Do not move all traffic by default. Move the subjects whose loss you cannot tolerate, and monitor consumer lag (num_pending, num_ack_pending) on those streams, because JetStream moves the failure mode from silent loss to silent accumulation: a stalled consumer does not lose messages, but the stream grows until retention or storage limits bite.

One JetStream-specific trap: a stream with interest-based retention and no consumers deletes messages immediately, because no consumers means no interest. Verify your retention policy matches your intent, and verify at least one durable consumer exists before you trust the stream.

Prevention

Alert on the in/out asymmetry. A fan-out-ratio alert as described above is the single highest-value check for this failure mode, and most teams never set it up.
Monitor subscription count against an expected floor. A sudden drop in subscriptions is the earliest warning that interest is missing.
Pair producer and consumer counters per critical subject. The server cannot attribute loss to a subject; your applications can. Emit publish and receive counts with the subject as a label and alert on divergence.
Use no-responders for all request-reply traffic. It converts the most common invisible failure into an explicit error at the call site.
Fix startup ordering everywhere. Subscribers subscribe before publishers publish, enforced by readiness checks, not convention.
Document per-subject loss tolerance. For each subject or traffic class, record whether loss is acceptable. That document tells you which subjects need JetStream and which are fine on core NATS.
Test the failure. In staging, kill a subscriber and confirm that (a) the fan-out alert fires and (b) the application-level gap alert fires. If neither fires, you will find out in production.

How Netdata helps

Netdata polls the NATS monitoring endpoints and charts in_msgs, out_msgs, in_bytes, and out_bytes per second, so the in/out asymmetry is a visible divergence on a chart rather than something you compute by hand during an incident.
Subscription count and connection count are collected alongside throughput, so you can correlate a drop in subscriptions with the exact moment out_msgs flattened.
slow_consumers is charted over time, so you can rule the delivery-drop mechanism in or out at a glance instead of guessing which loss mode you are in.
Route count and uptime are collected per node, so cross-node interest failures and restarts line up on the same dashboard as the throughput asymmetry.
Because all of these are time series at high granularity, you can measure your actual baseline fan-out ratio during healthy operation and alert on deviation from it, which is exactly what this failure mode requires.