$ guides / nats / nats-jetstream-api-errors ▌

Operations Guides

NATS JetStream API errors: reading the /jsz api.errors counter without false alarms

You opened /jsz and saw api.errors in the thousands, or an alert fired because the counter moved. The first question is not “what broke” but “is this counter telling me something real.” The JetStream API error counter is cumulative, coarse-grained, and incremented by perfectly healthy client behavior. Alerting on its absolute value or on any movement at all is a guaranteed false-alarm generator.

This article covers reading api.errors correctly: computing a rate, using the error ratio, correlating with api.inflight, and separating idempotent-create noise from genuine failures like storage exhaustion, Raft proposal failures, and permission denials. For the broader JetStream signal model, see How NATS actually works in production.

What this means

Every NATS server that serves JetStream also serves the JetStream API: the $JS.API.> subject space clients use to create streams, bind consumers, publish with acks, and run admin operations. Each failed API call increments a per-server counter, exposed on the monitoring port (default 8222) at /jsz as api.errors, alongside api.total for all API requests.

Two properties define how you should read it:

The counters never reset. Both api.total and api.errors are lifetime counters since server start. A value of 5,000 errors means 5,000 failures ever, which could be one bad hour three weeks ago. Alert on the rate (delta over time), not the value.
The counter is coarse-grained. It increments on any failed API call, including “stream name already in use” from a client doing idempotent creation and “consumer already exists” from a create-or-bind race during a rolling deploy. These are benign, and on busy systems they can dominate the counter.

Practical thresholds for the ratio api.errors / api.total:

Ratio	Reading
< 1%	Normal. Benign idempotent operations and occasional client retries live here.
> 1%	Notable. Worth a look, especially if sustained or rising.
> 10%	Systemic. Something structural is failing: storage, Raft, account limits, or permissions.

Apply a minimum request volume before trusting the ratio. On a quiet cluster with 100 lifetime API calls, three benign “already exists” errors produce a 3% ratio that means nothing. A floor of a few hundred requests in the sample window keeps low-traffic clusters from paging you.

Common causes

Cause	What it looks like	First thing to check
Idempotent create/bind patterns	Steady low error rate, ratio under 1%, errors cluster around deploys	API advisory stream for “already exists” / “already in use” descriptions
Client retry storms	One root error, but api.errors climbs fast; api.total climbs with it	Server logs: same client connection repeating the same failing call?
Stream storage full	Errors on publish/create ops, storage near reserved_storage	`/jsz` storage vs reserved_storage; per-stream sizes via `/jsz?streams=true`
Raft proposal failures / leader instability	Errors plus rising api.inflight, meta leader changing	`/jsz` meta_cluster leader stability and replica state
Account JetStream limits exceeded	Errors confined to one account	`/jsz?accounts=true` per-account api.errors
Permission errors on JS ops	Errors after credential or config changes, from specific clients	Server logs for authorization/permissions violations
Disk I/O stall (not full, just slow)	High sustained api.inflight, errors on write paths, storage has headroom	OS-level iowait and disk latency on the JetStream store path

Quick checks

All read-only. Assumes the monitoring port is 8222.

# Snapshot the API counters
curl -s http://localhost:8222/jsz | jq '{api_total: .api.total, api_errors: .api.errors, inflight: .api.inflight}'

# Compute a rate: take two snapshots 60s apart
curl -s http://localhost:8222/jsz | jq '[.api.total, .api.errors] | @tsv'
sleep 60
curl -s http://localhost:8222/jsz | jq '[.api.total, .api.errors] | @tsv'
# error_rate = (errors2 - errors1) / 60; ratio = (errors2 - errors1) / (total2 - total1)

# Which account is generating the errors?
curl -s 'http://localhost:8222/jsz?accounts=true' | jq '.account_details[] | {account: .name, api_total: .api.total, api_errors: .api.errors}'

# Is storage involved?
curl -s http://localhost:8222/jsz | jq '{memory, storage, reserved_memory, reserved_storage, streams, consumers}'

# Is Raft involved? (clustered JetStream only; null on standalone)
curl -s http://localhost:8222/jsz | jq '.meta_cluster | {leader, replicas: [.replicas[]? | {name, current, offline, lag}]}'

# Watch live API advisories, including per-error detail (requires nats CLI and system account access)
nats event --js-advisory

# Correlate with server logs for the actual error descriptions
grep -i "jetstream" /var/log/nats/nats-server.log | tail -50
grep -iE "permissions violation|authorization violation" /var/log/nats/nats-server.log | tail -20

Two notes. First, api.inflight is the count of API requests currently being processed; treat it as a concurrency gauge, not an error signal. Second, the advisory stream is the fastest way to see which operations are failing, with error code, description, account, and client connection details per event. It is far more diagnostic than the counter alone.

How to diagnose it

Route the error rate into one of three buckets: benign, client-amplified, or genuine server-side failure.

flowchart TD
  A[api.errors rate rising] --> B{Error ratio in window}
  B -->|under 1%| C{Advisory stream shows already-exists or bind races?}
  C -->|yes| D[Benign: idempotent create or bind noise]
  C -->|no| E[Inspect logs for the real error]
  B -->|over 1%| F{api.inflight also elevated?}
  F -->|yes| G[Raft or disk path: check meta_cluster and OS disk latency]
  F -->|no| H{Storage near reserved limit?}
  H -->|yes| I[Storage exhaustion: check per-stream sizes and consumer lag]
  H -->|no| J{Errors confined to one account or one client?}
  J -->|yes| K[Account limits or permission errors: check logs]
  J -->|no| L[Systemic: check Raft health and server logs]

Compute the rate and ratio over a window. Two /jsz snapshots 60 seconds apart give you errors/sec and errors/total for the window. Under 1% with a stable rate is almost always noise. Over 1% and rising deserves the next steps. Over 10% is a real incident until proven otherwise.
Break it down per account. /jsz?accounts=true shows api.total and api.errors per account. If one account owns nearly all the errors, the blast radius is one tenant’s limits, permissions, or clients.
Watch the advisory stream. nats event --js-advisory shows each failed operation as it happens: the API operation type, error code and description, account, and client connection. This is where you separate “consumer already exists” (benign) from “insufficient storage resources available” (not benign). JetStream API errors carry a stable numeric err_code in the 1xxxx range (per ADR-7) plus an HTTP-style code; the human-readable description string is explicitly not covered by SemVer and can change between releases, so alert and script on the codes, not the text.
Check api.inflight alongside. If errors rise and inflight is elevated and sustained, the server is slow to process API calls. That combination points at Raft consensus delays or disk I/O latency, not clients. If inflight is flat and errors are rising, the failures are fast rejections: limits, permissions, or “already exists” responses.
Check Raft health (clustered only). A changing meta_cluster.leader, replicas with current=false or offline=true, or growing replica lag explains transient API error spikes: requests submitted during leader elections fail. Brief spikes during an election that resolve in seconds are expected; repeated elections are the actual problem. See the Raft instability pattern in the NATS monitoring checklist.
Check storage. Compare storage and memory against reserved_storage and reserved_memory. Near the limit, publish and stream-create operations get rejected. Find the largest streams with /jsz?streams=true and check whether stalled consumers are preventing retention from freeing space.
Correlate with logs for permission and auth failures. Permission violations on JetStream operations show up in server logs, not in any counter field. If errors started right after a credential rotation, config reload, or upgrade, logs are the authoritative source.
Check versions. If you upgraded recently, behavior changes can inflate the counter. NATS v2.12 enables a “strict mode” by default that returns errors to clients for configuration mismatches that older versions only logged. Separately, CVE-2025-30215 (fixed in v2.11.1 and v2.10.27) was an authorization gap on certain JetStream admin APIs: on unpatched versions, some cross-account access attempts may not surface as errors at all, so a quiet counter is not proof of a healthy permission model on old versions.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
api.errors rate (delta)	The only honest way to read a cumulative counter	Any sustained positive rate; sharp acceleration
api.errors / api.total ratio	Normalizes for traffic volume	>1% notable, >10% systemic (with a minimum request volume)
api.inflight	Server-side API processing latency proxy	Sustained elevation alongside rising errors: Raft or disk
Per-account api.errors	Blast-radius isolation	One account owning most errors
storage / reserved_storage	Storage-full rejections are a top genuine cause	Approaching 90%
meta_cluster leader changes	Elections cause transient API failures	More than ~1 change per 5 minutes
Server log permission violations	Permission errors only visible in logs	Spike after credential or config changes

On exporters: if you scrape NATS into Prometheus via prometheus-nats-exporter, the JetStream API counters are exported as counters and the same rate-and-ratio logic applies in PromQL.

Fixes

Benign idempotent-create noise

Fix the alert, not the server. If clients use create-or-bind patterns (common during rolling deploys, where two instances race to create the same consumer), the “already exists” errors are the system working as designed. Prefer idempotent client calls (create-or-update semantics) where the client library offers them, and make sure alerting uses rate and ratio with a volume floor, never absolute counter values.

Client retry amplification

One underlying failure can look like a flood because each retry increments both api.total and api.errors. Well-behaved clients retry with backoff; clients that retry immediately amplify the error rate and add load to an already stressed meta leader. Identify the offending client connection in the advisory stream or logs and fix the retry policy. Until then, judge severity by the first error in the burst, not the count.

Storage exhaustion

Free space or raise limits, in that order. Find the largest streams, check whether stalled consumers are blocking retention (a consumer that never acks holds messages forever under interest-based retention), and review retention and discard policies. With DiscardNew, a full stream rejects publishes, which is exactly the API error you are seeing. Do not just raise reserved_storage without understanding why growth outpaced consumption.

If inflight and errors rise together, check disk latency on the JetStream storage path first (iowait, await), then network latency between cluster members, then memory pressure and GC pauses on the current and recent leaders. Network-attached storage with variable latency is a classic cause of Raft instability. Do not restart nodes as a first move: a restart triggers leader elections for every Raft group that had its leader there, adding more transient API errors on top of the ones you are chasing.

Permission errors

These come from misconfigured account permissions or, worse, probing. Fix the credential or the account’s JetStream permissions, and confirm your server version is at or past v2.11.1 / v2.10.27 so the authorization layer is actually enforcing on all JetStream admin APIs.

Prevention

Alert on rate and ratio, never on the counter. api.errors only goes up. Any alert on its absolute value will fire eventually and mean nothing.
Require a volume floor. Only evaluate the ratio when the window has enough API traffic for the percentage to mean something.
Baseline your benign error rate. If your deploy pipeline generates create/bind races, measure that rate and treat it as the zero point. Deviations from baseline matter more than fixed thresholds.
Watch inflight as a companion signal. Errors plus inflight is a server problem; errors alone is usually a client problem. Having both on the same dashboard saves the first 15 minutes of every investigation.
Keep the advisory stream in your incident toolkit. The counter tells you that; $JS.EVENT.ADVISORY.API tells you what, who, and why.
Patch past CVE-2025-30215. On affected versions the counter can under-report because some unauthorized operations were not rejected at all.

How Netdata helps

Automatic rate computation: Netdata’s NATS collector polls /jsz and stores api.total and api.errors as time series, so deltas and ratios are computed for you instead of by hand with jq and sleep.
Counter-reset handling: because the values are cumulative and drop to zero on restart, Netdata treats them as counters, so a server restart does not render as a nonsensical negative spike.
Correlation in one view: the error rate sits next to JetStream storage usage, api.inflight, message throughput, and connection churn, which is exactly the correlation set this diagnosis depends on.
Restart correlation: uptime resets (see NATS crash loop) explain sudden counter resets and post-restart error bursts during JetStream recovery.
Maturity placement: JetStream API error monitoring is a Level 2 signal in the NATS monitoring maturity model; per-account breakdowns and inflight tracking are the Level 3 and 4 refinements.