NATS no responders available for request: request-reply into the void

Your service just started throwing nats: no responders available for request. The client got an answer back from the server almost instantly, and the answer was: nobody is listening on that subject. This is the request-reply counterpart to NATS’s silent message loss: instead of the request vanishing and the client waiting out its timeout, the server short-circuits the call and fails fast with a 503 status in the reply headers.

The fast failure is a feature. Core NATS is fire-and-forget: a message published to a subject with zero matching subscriptions is dropped with no error, no log line, and no dedicated metric. The no-responders mechanism exists so that request-reply callers at least find out. The server is telling you something specific: at the moment your request was routed, the subject had no subscribers. Your job is to figure out why.

The most important distinction up front: no-responders means the responder is absent. A timeout means the responder is present but did not reply in time. The fixes are completely different, so do not conflate them.

What this means

When a NATS client issues a request, it publishes a message on the target subject with an inbox reply subject, then waits for a response. On NATS Server 2.2.0 and later, with a client library that supports headers, the server checks the subject tree when the request arrives. If no subscription matches, the server immediately publishes an empty reply with a Status: 503 header. The client library surfaces this as a typed error: nats.ErrNoResponders in Go, NATSNoRespondersException in C#, NoRespondersError in Python, and equivalents in the other major clients.

Before 2.2.0, or with a client that does not negotiate header support, the same condition blocks until the client’s request timeout fires, and you get a timeout error indistinguishable from a slow responder. If you are seeing timeouts rather than explicit no-responders errors, check server and client versions before assuming the responder exists.

Because the check runs against the subject tree at routing time, no-responders is a statement about subscription interest, not about whether your responder process is alive. A responder that crashed, was never started, subscribed to a different subject, or sits behind a broken route all produce the same error.

flowchart TD
  A[Request published on subject S] --> B{Any subscription matches S?}
  B -- No --> C[Server replies with Status 503 header]
  C --> D[Client raises no-responders error immediately]
  B -- Yes --> E{Responder replies before client timeout?}
  E -- Yes --> F[Normal response]
  E -- No --> G[Client raises timeout error]
  D --> H[Responder absent: check process, subject spelling, routes]
  G --> I[Responder slow or hung: check its processing path]

Common causes

One gotcha worth internalizing: when a service is reached through an account export/import, the server creates an internal subscription for the import. From the routing engine’s perspective there is always interest on the subject, so the no-responders shortcut never fires. Cross-account service calls can only fail by timeout. If your architecture uses account imports for services, you lose the fast-fail signal entirely and must treat timeouts as your absence detector.

Quick checks

# Is the responder connected and subscribed? List connections with their subscriptions.
# Empty subscriptions_list across all connections means nobody is subscribed.
curl -s "http://localhost:8222/connz?subs=1" | jq '.connections[] | {cid, name, ip, subscriptions_list}'

# Total subscription count. Lower than expected for your deployment?
curl -s http://localhost:8222/varz | jq .subscriptions

# Cluster routes: in an N-node cluster each server should have N-1.
# A missing route means interest from one node is invisible to another.
curl -s http://localhost:8222/varz | jq .routes

# Message flow asymmetry: in_msgs rising while out_msgs stays flat means
# publishes are landing on subjects with no subscribers.
curl -s http://localhost:8222/varz | jq '{in_msgs, out_msgs}'

# JetStream: is it enabled and is the API answering?
curl -s http://localhost:8222/jsz | jq '{disabled, api_total: .api.total, api_errors: .api.errors}'

# JetStream meta cluster: is the leader stable? Frequent leader changes
# correlate with transient no-responders on JetStream publish.
curl -s http://localhost:8222/jsz | jq '.meta_cluster | {leader, replicas: [.replicas[]? | {name, current, offline}]}'

# Server health (basic readiness, avoids JetStream recovery false positives):
curl -s http://localhost:8222/healthz?js-server-only=true

All of these are read-only HTTP GETs against the monitoring port (default 8222, enabled with -m 8222 or http_port). Avoid /subsz on servers with very high subscription counts; it can be expensive. The subscription count from /varz is the safe check.

How to diagnose it

1. Confirm the error type. Look at the actual client error. ErrNoResponders / 503 means absent responder. A timeout error means present but unresponsive responder. If you cannot tell from logs, reproduce with a minimal client call against the same subject.

2. Verify the subject string. Subjects are case-sensitive, token-delimited by dots, and a double dot (foo..bar) creates an empty token that no normal subscription matches. Compare the publisher’s subject against the responder’s subscribe call character by character. This is the most common cause in practice, especially after config refactors or environment variable changes that assemble subjects from parts.

3. Check whether the responder is connected at all. Use the /connz?subs=1 query above. If the responder’s connection is absent, the problem is the responder application: crashed, stuck in a restart loop, failing authentication, or connecting to a different server URL than you think. If connections are churning (compare total_connections growth against the stable connections count in /varz), the responder may be flapping: subscribing, dying, reconnecting.

4. If clustered, check routes. Interest propagates across routes. If the responder is connected to server B and the requester to server A, and the A-B route is down, server A sees zero interest and returns 503. Compare /varz routes on each node against the expected N-1. Brief route drops during rolling restarts are normal; sustained missing routes mean partition.

5. If the failing call is a JetStream publish, check JetStream separately. The synchronous Publish() in the Go client is implemented as an internal request to the JetStream API subject. During meta-group or stream leader elections in clustered JetStream, that API is briefly without a responder, and publishes fail with no-responders. The Go client retries this path (250 ms wait, 2 retries by default, tunable via RetryWait and RetryAttempts). If you see bursts of no-responders that self-resolve in under a second and correlate with leader changes in /jsz, this is your cause.

6. If the call crosses accounts, stop expecting 503. A service import masks absence. Diagnose with a direct subscription check in the exporting account instead, and treat client timeouts as the failure signal.

Metrics and signals to monitor

Fixes

Responder down or not started

Start or fix the responder. Then address the ordering problem: if requesters can start before responders are subscribed, add readiness gating in the requester (retry with backoff on ErrNoResponders, or block startup on a probe request). Client libraries do not retry request-reply for you on this error; the retry policy is application code.

Subject mismatch

Fix the string, then remove the class of bug: define subjects once in a shared constant or config value consumed by both publisher and subscriber. Validate assembled subjects at startup (non-empty, no empty tokens). If multiple environments share a cluster, include the environment prefix in a single place rather than string-concatenating at each call site.

Cluster partition

Restore the route: fix the network path, firewall rule, or DNS entry between the two servers. Verify with /routez that num_routes returns to N-1 and that per-route pending_size drains. Clients with multiple server URLs will fail over, but interest-based routing still requires the full mesh for cross-server delivery.

Transient JetStream no-responders during elections

If bursts correlate with leader changes, the real problem is Raft instability, not the publish path. Check route RTT, CPU, and disk latency on JetStream nodes; elections triggered by slow heartbeats will keep producing these blips until the underlying resource issue is fixed. Tuning the client’s RetryWait/RetryAttempts upward buys tolerance but does not fix the elections.

There is also a known open issue where a freshly created 3-node JetStream cluster (observed on 2.10.17) produces persistent no-responders errors on KeyValue writes and stream publishes; restarting one non-leader node clears it.

Cross-account service imports

You cannot restore the 503 behavior; the import subscription prevents it by design. Set client request timeouts deliberately (not the library default) so absence fails in bounded time, and monitor the exporting side’s subscription count as your absence signal.

Prevention

• Shared subject definitions. One source of truth for every request-reply subject, imported by both sides. Most no-responders incidents in steady state are typos or drift.
• Startup ordering and retries. Treat ErrNoResponders as retryable with capped backoff during deploys, fatal after the deploy window. Treat timeouts as a separate signal (responder slow), not as “service missing”.
• Monitor the asymmetry. Alert on out_msgs falling below the expected fan-out of in_msgs. This catches the silent-loss version of the same failure before any requester notices.
• Alert on subscription count drops for subjects that must always have a responder.
• Watch routes and Raft leader stability so partitions and election storms are caught before they surface as application-level 503s.
• Decide where you need persistence. If a request must not be lost when the responder is down, core request-reply is the wrong primitive regardless of the 503 feature. Use JetStream work queues and monitor consumer lag.

How Netdata helps

• Netdata polls the NATS HTTP monitoring endpoints and charts in_msgs vs out_msgs per server, which makes the publish-without-subscribers asymmetry visible as a divergence instead of something you discover from application logs.
• Subscription count and connection count are tracked continuously, so a responder disconnect shows up as a step down you can correlate with the first no-responders error timestamp from client logs.
• Connection churn (the total_connections delta) exposes flapping responders that produce intermittent, hard-to-reproduce 503s.
• Route count per server is charted against the expected full mesh, so partition-induced no-responders are visible from any node’s dashboard.
• On JetStream servers, API error rate and meta-cluster state from /jsz let you confirm that a burst of publish-side no-responders lines up with a leader election rather than an application bug.

Related guides

• How NATS actually works in production: a mental model for operators
• NATS server not responding: healthz failing and the process down or hung
• NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check
• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS stalled clients and stale connections: half-dead sockets and write-path distress
• NATS slow consumer detected: the write buffer overflowed and messages were dropped
• NATS slow consumer breakdown: clients vs routes vs gateways and blast radius
• NATS pending bytes growing: catching a slow consumer before it is disconnected
• NATS Maximum Connections Exceeded: new clients rejected at the max_connections wall
• NATS write_deadline and buffer sizing: tuning how long the server waits on a slow writer
• NATS monitoring checklist: the signals every production server needs
• NATS monitoring maturity model: from survival to expert