NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check

A NATS server restarts with a large JetStream store. The process is fine, clients will be served shortly, but your pager fires and Kubernetes kills the pod before recovery finishes. The server starts recovering again, the probe fails again, and you are in a restart loop that looks like an outage but is a health check misconfiguration.

The cause is almost always the same: a liveness probe or a PAGE alert pointed at the bare /healthz endpoint on a JetStream-enabled server. On such a server, bare /healthz does not answer “is the process alive?” It runs the full JetStream health suite, and that suite legitimately fails for minutes while the server replays and recovers its assets after a restart.

NATS exposes three variants of /healthz, and they answer three different questions. Picking the wrong one causes false pages, restart loops, or blind spots. This article explains what each variant checks, how the behavior changed across server versions, and which variant belongs in which check.

What /healthz is and why the variant matters

The endpoint lives on the NATS monitoring HTTP server, not the client port. Two prerequisites before any of this works:

• The monitoring port must be enabled. There is no /healthz at all unless you start the server with -m 8222 or set http_port: 8222 in the config. If your probes fail with connection refused on a brand-new deployment, check this first.
• The endpoint exists only on NATS 2.2 and later.

When healthy, the endpoint returns HTTP 200 with {"status":"ok"}. Any other response is a failure. What sits behind that binary answer depends on which variant you call and whether JetStream is enabled.

One operational note: the monitoring server has no built-in authentication. Treat port 8222 as internal-only and restrict it with network policy or equivalent isolation.

The three variants

Consequences that follow directly:

• On a non-JetStream server, bare /healthz and ?js-server-only=true behave identically. The variant question only matters where JetStream is enabled.
• Bare /healthz is a deep correctness check, not a liveness check. During post-restart recovery on a large store it returns non-200 for minutes while the server is doing exactly what it should be doing. Slow recovery is normal.
• js-enabled-only=true is a configuration-drift check, not a serving check. It tells you JetStream failed to initialize or is disabled when it should not be. It says nothing about whether streams and consumers are recovered and serving.
• js-server-only=true answers “can this process accept work?” That is the question a liveness probe and a page-level alert should ask.

flowchart TD
  A[You need a NATS health check] --> B{JetStream enabled on this server?}
  B -->|No| C[bare /healthz is fine - identical to js-server-only]
  B -->|Yes| D{What question are you asking?}
  D -->|Is the process ready to serve?| E[Use /healthz?js-server-only=true - PAGE and liveness]
  D -->|Is JetStream fully healthy including assets?| F[Use bare /healthz - TICKET only]
  D -->|Is JetStream enabled per config?| G[Use /healthz?js-enabled-only=true - drift check]
  E --> H[Gate on uptime over 300s and sustained over 60s]
  F --> I[Expect non-200 for minutes after restart on large stores]

Version differences that change the answer

The semantics of these parameters have shifted across releases, so the right answer depends on what you run:

• v2.9.0 introduced /healthz?js-enabled=true and /healthz?js-server-only=true.
• v2.9.10 deprecated js-enabled=true and replaced it with js-enabled-only=true, with fixed semantics: check the desired state from config against the current JetStream state, then exit. The deprecated parameter is still accepted for backward compatibility but should not appear in new configurations.
• v2.10.x: js-server-only=true still included a meta leader currency check. Under high load during a cluster restart this produced repeated “Falling behind in health check” warnings and readiness probe failures that detached pods from the Kubernetes service even though the servers were recovering normally. The practical workaround on v2.10 was to move the readiness probe to js-enabled-only.
• v2.11: js-server-only=true no longer checks the health of the meta leader, which removes the v2.10 problem. A separate js-meta-only=true option was added for operators who want the old meta-leader check explicitly. See the NATS monitoring documentation and the v2.11 release notes for the authoritative parameter list.

The takeaway: if you tuned your probes around v2.10 behavior, re-check them after upgrading to v2.11. The meaning of js-server-only=true changed underneath you, in this case for the better.

The failure mode everyone hits: probe-driven restart loops

The concrete incident looks like this:

1. A JetStream server restarts: rolling update, node drain, or crash.
2. On startup, the server replays its write-ahead log, rebuilds in-memory indexes, and recovers stream and consumer assets. On a large store this takes minutes. During this window bare /healthz returns non-200.
3. A liveness probe pointed at bare /healthz fails its threshold. Kubernetes kills the pod.
4. The pod restarts, recovery starts over, the probe fails again. The loop never converges because each kill discards the recovery progress.

Two aggravating factors make this worse. First, if probe timeouts are too tight, the orchestrator kills the server before recovery finishes, so readiness probe timeouts on JetStream servers should be generous. Second, bare /healthz also flaps during backup-induced I/O stalls: a filesystem snapshot creates transient JetStream distress that the full check reports as unhealthy even though the server is serving traffic.

The same logic applies to paging. If your page-level alert calls bare /healthz, every rolling restart of a JetStream cluster pages someone for a server that is recovering exactly as designed.

There is one failure where a persistent bare /healthz failure is real: if a stream or consumer cannot be recovered from disk at all (corrupt WAL, filesystem error), the endpoint reports a stream- or consumer-type error until the server is restarted with the corrupt data removed. That is a genuine data-path incident, and it is what the deep check is for. Distinguishing “recovery in progress” from “recovery failed” requires the duration of the condition and the server logs, not the probe alone.

Which variant to use, per check type

PAGE alert and Kubernetes liveness probe: js-server-only=true. You want to know whether the process is hung, crashed, or unable to accept work. Gate it to kill cold-start and transient noise:

• Uptime greater than 300 seconds, so a server still in early startup is not paged.
• Failure sustained for at least 60 seconds, so a single slow probe response is not paged.

This variant is safe against the known false-positive sources (JetStream recovery after restart, backup I/O stalls, batch-processing load) because it checks basic readiness rather than JetStream I/O health.

TICKET alert: bare /healthz. Full JetStream health, including Raft recovery and asset health, is worth knowing about. Route it to a ticket, not a page, because of the recovery transients above. A bare /healthz failure that persists well beyond the expected recovery window, or one that reports a hard asset recovery error, gets promoted to an incident.

Configuration-drift check: js-enabled-only=true. Use this to assert “JetStream is supposed to be on and it is on.” It catches a server that started without initializing JetStream (inaccessible storage directory, config error). For a stronger version of the same assertion, /jsz reporting disabled=true when JetStream is expected is page-worthy, but only with cold-start gating: uptime over 600 seconds and the condition sustained for at least 5 minutes, because JetStream can report disabled during initialization on large stores.

Kubernetes probes in practice. Liveness gets js-server-only=true; readiness can be stricter but must tolerate the recovery window with generous timeouts rather than low failure thresholds. On v2.10 specifically, js-server-only=true included the meta leader currency check, so readiness probes on that version hit false failures under load during cluster restarts. A minimal liveness stanza:

# Liveness probe: basic server readiness only, not full JetStream health
livenessProbe:
  httpGet:
    path: /healthz?js-server-only=true
    port: 8222
  periodSeconds: 10
  failureThreshold: 6

Whatever thresholds you choose, keep the failure budget longer than your worst observed cold start plus a margin.

Signals to correlate when /healthz fails

A failed probe tells you almost nothing by itself. These signals tell you which failure you are looking at:

The decision pattern: bare /healthz failing plus fresh uptime plus heavy disk I/O equals normal recovery. Bare /healthz failing with a hard stream or consumer recovery error in the logs, persisting past any plausible recovery window, equals a data-path incident. js-server-only=true failing with uptime well past startup equals a genuinely sick process. Page on the last one.

Common misuses

• Bare /healthz as a liveness probe on JetStream servers. The restart-loop machine described above. This is the most common one.
• Bare /healthz as a PAGE alert. Same root cause, different blast radius: false pages during every rolling restart and every backup window.
• Using the deprecated js-enabled=true. Still accepted, but replaced by js-enabled-only=true since v2.9.10. Remove it from new configs.
• Assuming js-server-only=true is version-stable. It gained a meta leader check in v2.10 and lost it in v2.11. Probe behavior changed across the upgrade; re-validate after version bumps.
• Treating js-enabled-only=true as proof JetStream is serving. It confirms enablement only. A server can pass it while consumers are stalled and streams are unrecovered.
• Forgetting the monitoring port entirely. No -m 8222 or http_port: 8222, no endpoint. Probes failing with connection refused on a fresh install is usually this, not a sick server.
• Exposing 8222 broadly. The monitoring endpoints have no authentication. Keep the port inside the cluster or host network.

How Netdata helps

• Uptime tracking makes restart loops visible. Netdata collects server uptime from /varz at per-second granularity, so the sawtooth of a probe-driven crash loop (uptime resetting every minute or two) is visible on a dashboard instead of being inferred from probe logs after the fact.
• Correlating probe failures with JetStream pressure. API inflight, API errors, storage usage, and message counters from /jsz and /varz sit on the same timeline as restart events, which is what separates “recovery in progress” from “JetStream is actually broken.”
• Cold-start context for alerting. Because uptime is collected alongside health-adjacent signals, you can build alert logic with the gating this article recommends (uptime over 300s, sustained failure) instead of raw probe flapping.
• Disk I/O correlation. Host-level disk latency and utilization for the JetStream storage path explain slow recoveries and v2.10-era meta leader check failures in the same view.
• Restart-rate detection. Alerting on repeated restarts in a short window catches the loop before anyone reads a probe configuration.

Related guides

• NATS monitoring checklist: the signals every production server needs
• NATS monitoring maturity model: from survival to expert