NATS messages published but not received: subject mismatches and cross-server gaps

The publisher’s Publish() call returns no error. The server is healthy. The subscriber receives nothing. This is almost never a server bug. It is how core NATS works: fire-and-forget routing against an in-memory subject tree, with no persistence, no delivery guarantee, and no signal when a message matches zero subscriptions.

In core NATS, a message published to a subject with zero matching subscribers is silently discarded. No error, no log line, no metric. The only server-level evidence is an asymmetry between in_msgs and out_msgs, which most teams never graph.

This runbook covers the four causes behind nearly every occurrence: a subject mismatch between publisher and subscriber, a wildcard pattern that does not match what the author expected, a queue-group misconfiguration, and the cross-server case where publisher and subscriber sit on different cluster servers and the route between them is down. Each is provable from the monitoring endpoints without guessing.

What this means

Every inbound message is matched against the server’s subject trie and fanned out to the subscriptions that match. If that set is empty, the message vanishes. Two properties make this bite in production:

First, NATS subjects are case-sensitive and subject strings are not validated on publish unless the client enables pedantic mode. A client publishing to orders.created while the subscriber listens on Orders.Created, or on orders.create, produces zero deliveries and zero errors on either side. Even a malformed subject, such as one with a stray space or empty token, can be “successfully” published and then match nothing.

Second, interest-based routing extends the same semantics across a cluster. Servers advertise subscription interest over routes, and a message only crosses a route if the remote server has registered interest for that subject. If the subscriber is on server B, the publisher on server A, and the A-to-B route is down, the message is dropped at A. There is no queuing and no replay in core NATS.

flowchart TD
  P[Publisher sends message] --> M{Subject trie match on this server?}
  M -->|local subscriber| D[Delivered]
  M -->|no local match| R{Interest on a route and route up?}
  R -->|yes| F[Forward over route to peer, deliver there]
  R -->|no or route down| X[Silently dropped]
  M -->|wildcard or case mismatch| X

Common causes

This symptom is distinct from slow-consumer drops (the subscriber receives some messages and then falls behind, with slow_consumers incrementing) and from payload rejections (publishes fail at the protocol level). Here, nothing arrives at all and the server counters look clean.

Quick checks

All of these are read-only HTTP calls against the monitoring port (default 8222).

# 1. Is the subscriber connected, and on exactly which subjects?
curl -s http://localhost:8222/connz?subs=1 | \
  jq '.connections[] | {cid, name, ip, subscriptions: .subscriptions_list}'

# 2. Does a subscription exist for the subject the publisher uses?
curl -s "http://localhost:8222/connz?subs=1" | \
  jq '.connections[] | select(.subscriptions_list | index("orders.created")) | {cid, name, ip}'

# 3. Server-side totals: is anything arriving and leaving?
curl -s http://localhost:8222/varz | \
  jq '{in_msgs, out_msgs, subscriptions, slow_consumers}'

# 4. Clustered? Route count should be N-1 on each server.
curl -s http://localhost:8222/varz | jq .routes
curl -s http://localhost:8222/routez | jq '.routes[] | {rid, ip, rtt, pending_size}'

# 5. Did subscription interest propagate across routes?
curl -s "http://localhost:8222/routez?subs=1" | jq '.routes[] | {rid, subscriptions}'

# 6. Server uptime: did the server restart during the gap window?
curl -s http://localhost:8222/varz | jq .uptime

On check 2: /subsz looks like the obvious tool for listing all subscriptions, but on servers with accounts enabled it is known to misbehave, and on high-subscription servers it is expensive enough to stall the server. Prefer /connz?subs=1 and /routez?subs=1 for verification.

How to diagnose it

Work through these in order. Each step eliminates one cause class.

1. Verify the subscriber is connected to a server at all. Query /connz?subs=1 on the server you believe the subscriber uses and find the connection by name or IP. If the connection is absent, the problem is upstream of subjects: the client is down, stuck reconnecting, or pointed at a different server URL. See NATS connection churn: a stable connection count hiding constant reconnects if the connection flaps.

2. Compare subject strings exactly. Take the subject from the publisher’s code and the subscription from /connz?subs=1 output and compare them character by character. Check case first (orders.Created vs orders.created), then trailing tokens, then whitespace. Subjects cannot contain whitespace, and because publish-side validation is off by default, a generated subject with a stray space or empty token can be published “successfully” and match nothing. If your clients build subjects from variables, log the resolved subject string once per process start and compare it against the subscription list.

3. Check wildcard semantics. foo.* matches foo.bar but not foo.bar.baz; foo.> matches both but must be the last token. The classic bug is expecting * to span multiple tokens. The reverse bug exists too: a subscriber on foo.* silently misses everything published at deeper levels.

4. Check the queue group. If the subscriber uses a queue group, confirm the group name in /connz?subs=1 matches what the rest of the fleet uses. A one-character difference puts the subscriber in its own group, which changes delivery semantics: two groups each get every message, and a subscriber in the wrong group can appear to “miss” messages that went to the intended group. Queue groups need no server-side configuration, so the name on each subscription is the only thing to check.

5. Determine whether publisher and subscriber are on the same server. In a cluster, identify which server each side is connected to via /connz on each node. If they are on the same server and steps 1-4 are clean, delivery should be happening; re-examine the application.

6. If they are on different servers, check the route. Each server in an N-node cluster should show N-1 routes in /varz. If the count is short, or /routez shows a missing or flapping peer, you have the cross-server gap: messages published on server A never reach subscribers on server B while the route is down. Confirm interest propagation with /routez?subs=1: the route should list the subscriber’s subject. A missing route plus absent interest explains the loss completely.

7. Rule out a slow-consumer disguise. Check /varz slow_consumers and the slow_consumer_stats breakdown. If the counter is incrementing, messages are being dropped at the write buffer rather than never routed, which is a different runbook: NATS pending bytes growing: catching a slow consumer before it is disconnected.

8. Check server uptime against the gap window. A server restart drops all subscriptions; clients resubscribe on reconnect. If the subscriber’s reconnect or resubscribe raced or failed, the loss window is explained. Correlate /varz uptime with the timestamp of the first missed message.

Metrics and signals to monitor

The fan-out ratio deserves emphasis. If your subject mix normally delivers each message to three subscribers, out_msgs / in_msgs should hover near 3. A drift to 1 means two thirds of your subscriber population is silently gone, and no error will ever fire.

Fixes

Subject typo or case mismatch

Fix the string on one side and establish a single source of truth for subject names. Shared constants or a generated schema per integration boundary is the durable fix; subjects defined in two codebases will drift. For programmatically generated subjects, log the fully resolved string at startup and, where the client library supports it, enable pedantic mode in non-production so invalid subjects fail loudly during testing instead of silently in production. Subjects are case-sensitive by design, so “normalize everything to lowercase” conventions only work if every team follows them.

Wildcard pattern that does not match

Change the subscription to > when multi-token depth is required, or constrain publishers to a fixed token depth. > widens the match surface: if the subject namespace is shared, a broad foo.> subscription may pick up far more traffic than the consumer was sized for, trading this incident for a slow-consumer incident. Prefer explicit depth (foo.*.*) where the hierarchy is known.

Queue-group misconfiguration

Align the queue group name across all members of the group. Queue groups are purely client-declared, so a typo creates a new group with one member rather than an error. If you need every subscriber to see every message, remove the queue group entirely rather than relying on distinct names; if you need load balancing, verify after every deploy that /connz?subs=1 shows all intended members sharing one group name.

Cross-server gap (routes down)

Restore route connectivity: check the peer process, the network path between servers, firewall or security-group rules on the route port, and DNS resolution if routes use names. Routes reconnect automatically once the path is healthy, but messages published during the partition are gone; core NATS does not buffer or replay cross-server traffic. If the deployment uses configured route pool_size, confirm all servers use the same value, because a mismatch can prevent routes from forming. While you fix the route, clients configured with multiple server URLs will reconnect to surviving servers, which restores delivery for subscribers but not retroactively.

Subscriber absent or not resubscribed after a restart

Fix the client’s reconnect and resubscribe path, then decide whether fire-and-forget is acceptable for this traffic. If the business cannot tolerate loss during subscriber downtime, this is the architectural boundary: move the flow to JetStream, where messages persist in a stream and consumers resume from their last acknowledged position. Core NATS will never give you at-least-once delivery, and no client tuning changes that.

Prevention

• Graph the fan-out ratio. Alert when out_msgs / in_msgs deviates from the established baseline for more than a few minutes. This is the only automated tripwire for zero-subscriber loss.
• Alert on route count. expected_routes > 0 AND current_routes < expected_routes, sustained 60 seconds, catches partitions before user reports do.
• Centralize subject definitions. One shared package or schema per integration, consumed by both publisher and subscriber, with a test that asserts the subscriber’s wildcard pattern matches a sample of real published subjects.
• Verify subscriptions after every deploy. A post-deploy check that hits /connz?subs=1 and asserts every expected (subject, queue group) pair exists catches typos and failed resubscribes in minutes instead of days.
• Use request-reply with no-responder detection where loss is unacceptable but JetStream is too heavy. A request to a subject with no subscribers returns an immediate no-responders error, converting silent loss into a visible failure. See NATS no responders available for request: request-reply into the void.
• Keep monitoring scrapes off /subsz. Use counts from /varz and per-connection detail from /connz; the full subscription list is expensive and unreliable where accounts are enabled.

How Netdata helps

• Netdata polls the NATS monitoring endpoints and computes rates from the cumulative in_msgs and out_msgs counters, so the publish-vs-deliver asymmetry that signals zero-subscriber loss is visible as a chart rather than a manual jq diff.
• Route count and route health are tracked per server, making it immediate to see that one node dropped below N-1 routes at the exact moment subscribers went quiet.
• The slow_consumers counter and its breakdown let you rule the slow-consumer drop mode in or out in seconds, which is the first diagnostic fork in this runbook.
• Uptime and total_connections deltas are correlated on the same dashboard, so a server restart or a reconnect storm lines up visually with the start of the delivery gap.
• Subscription count is trended over time, exposing the slow subscriber-population decay that absolute checks miss.

Related guides

• NATS connection churn: a stable connection count hiding constant reconnects
• NATS connection storm: reconnect thundering herd after a network event
• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS file descriptor exhaustion: too many open files and the ulimit cliff
• NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check
• How NATS actually works in production: a mental model for operators
• NATS Maximum Connections Exceeded: new clients rejected at the max_connections wall
• NATS monitoring checklist: the signals every production server needs
• NATS monitoring maturity model: from survival to expert
• NATS no responders available for request: request-reply into the void
• NATS pending bytes growing: catching a slow consumer before it is disconnected
• NATS server not responding: healthz failing and the process down or hung