$ guides / nats / nats-authorization-violation ▌

Operations Guides

NATS Authorization Violation: authentication failures and credential rotation

Your NATS server log is filling with Authorization Violation entries, clients are failing to connect, and the error message tells you almost nothing. That is deliberate: NATS keeps auth error messages vague so they do not leak information to attackers. The side effect is that the same log line covers a typo’d password, an expired user JWT, a bad credential file deployed to a fleet, and someone port-scanning your cluster from the internet.

The diagnostic problem is not reading the error. It is establishing the blast radius and the source pattern. One client failing from one IP is a misconfiguration. Every client failing at the same timestamp is a credential lifecycle event, most often mass JWT expiration or an account resolver outage. A sustained flood from source IPs you do not recognize is scanning or brute force, and the correct response is network policy, not credential debugging.

What this means

When a client connects, the server evaluates the credentials presented in the CONNECT handshake against its configured authorization mode: static user/password or token in the config, NKey, or decentralized JWT under an operator. On failure, the server logs the violation and closes the connection with an authentication failure reason. The client library then decides what to do next, and this is where the noise comes from.

Some client libraries retry aggressively on auth failure. One bad credential file rolled out to a thousand containers can produce thousands of Authorization Violation entries per second. The connection count in /varz stays deceptively stable while total_connections climbs fast, because clients connect, fail, disconnect, and retry in a tight loop. This is the same churn signature described in NATS connection churn: a stable connection count hiding constant reconnects.

The second amplifier is shared expiration. In JWT/NKey mode, if user JWTs were all issued with the same expiration timestamp, every client fails simultaneously the moment that timestamp passes. Connection count drops toward zero, and the log flood is total rather than from one source. The third systemic cause is the account resolver: if the resolver serving account JWTs is unreachable or its storage is broken, every new client connection fails auth while existing connections keep working until they disconnect.

flowchart TD
  A[Authorization Violation in logs] --> B{How many sources?}
  B -->|One client / one IP| C[Single misconfiguration:
wrong creds, bad file, wrong server]
  B -->|All clients at one timestamp| D{Auth mode?}
  B -->|Many unknown IPs, sustained| E[Scanning or brute force:
network policy response]
  D -->|JWT/NKey| F[Mass JWT expiration
or account resolver down]
  D -->|Static user/pass or token| G[Config reload dropped users
or rotated secret not deployed]
  F --> H[Check resolver health and JWT expiry]
  G --> I[Check server config and client credential rollout]

Common causes

Cause	What it looks like	First thing to check
Wrong or rotated credentials on one client	Sporadic violations from a single source IP or client name; other clients fine	Which credential file the failing client presents; when it last changed
Mass JWT/NKey expiration	Every client fails at the same moment; connection count collapses	Expiration timestamps embedded in issued user JWTs
Account resolver unreachable	All new connections fail; existing connections keep working	Resolver storage and connectivity from the server
Credential rotation partially deployed	Violations spike right after a deploy; a subset of clients fail	Whether new credentials reached all client pods/hosts
Scanning or brute force	Sustained failures from IPs that do not map to your fleet	Source IP distribution in the log entries
Client retry amplification	Thousands of log entries per minute for one underlying bad credential	`total_connections` delta vs steady `connections`

Quick checks

All of these are read-only.

# Count violations and see the recent entries
grep -c "Authorization Violation" /var/log/nats/nats-server.log
grep "Authorization Violation" /var/log/nats/nats-server.log | tail -20

# Current vs cumulative connections: a fast-climbing total with a stable
# current count is retry churn from failing clients
curl -s http://localhost:8222/varz | jq '{active: .connections, total: .total_connections}'

# Who is connected right now, and from where
curl -s "http://localhost:8222/connz?auth=true" | jq '.connections[] | {cid, ip, name, account}'

# Distribution of source IPs in the violations (adjust log path)
grep "Authorization Violation" /var/log/nats/nats-server.log | grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort | uniq -c | sort -rn | head

The last check is the single most informative one. It tells you in one step whether you are looking at one broken client, your whole fleet, or the internet.

How to diagnose it

Establish the source pattern. Run the IP distribution check above. One dominant IP means a single misconfigured client. A spread across your fleet’s address space means a systemic credential event. A spread across addresses you do not own means external probing.
Check the blast radius on connections. Compare connections against its recent baseline. A drop greater than 50% in five minutes means legitimate clients are down, which is a page-level application incident regardless of the auth cause. A stable count with a climbing total_connections delta means clients are failing and retrying in a loop.
Identify the failing client. Correlate log timestamps with /connz?auth=true output. Clients that are repeatedly failing may never appear in /connz at all, or appear and disappear between polls. Match log source IPs to your deployment inventory to find which service owns the bad credential.
Determine the auth mode in play. Static user/password or token failures point to a config mismatch between server and client. NKey failures point to a wrong or rotated key. JWT failures point to expiration, revocation, or the account resolver. The remediation path is completely different for each, so do not skip this step.
If JWT mode and all clients failed at once, check the resolver and expiration in that order. An unreachable or broken account resolver fails all new connections while existing ones survive. Mass expiration fails everything including reconnects of previously healthy clients. The distinguishing symptom: with a resolver outage, long-lived connections keep working; with expiration, nothing works.
Check for retry amplification. If log volume is enormous relative to the number of affected clients, the client library retry behavior is multiplying one credential problem into a log flood and connection churn. Quantify it with the total_connections rate. This matters because the churn itself consumes server CPU (especially with TLS handshakes) and file descriptors, so a credential bug can become a capacity incident.
If the sources are external, switch from debugging to containment. Verify the server port is not unintentionally exposed. The fix is firewall or security group policy, not anything on the NATS side.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Auth violation log rate	The primary signal; NATS does not reliably expose auth failures as a `/varz` field, so logs are authoritative	Any spike above 10x baseline; sustained failures from known service accounts
`connections` vs `total_connections`	Reveals retry churn hidden behind a stable connection count	`total_connections` climbing fast while `connections` is flat
`connections` drop	Measures real client impact	Drop > 50% in 5 minutes
`cpu`	TLS handshake cost during retry storms	CPU spike correlated with violation flood
Server uptime	Rules out crash loops as a confounder	Uptime resets during the incident window

Current upstream nats-server may not expose an auth_errors counter in /varz. Treat server log parsing as the authoritative source for this signal, and build alerting on log rate plus the connection-count correlation.

Fixes

Single client with wrong credentials

Identify the service from the source IP, fix its credential file or environment, and restart that client only. Verify by watching the violation rate return to baseline and the client appear stably in /connz. If the credential was recently rotated, confirm the client is presenting the new material and not a cached or stale file.

Mass JWT/NKey expiration

If clients use short-lived JWTs with a refresh mechanism, check why the refresh path stopped working rather than issuing long-lived credentials as a workaround.

Account resolver unreachable

Restore the resolver: check its storage (a full disk on resolver storage is a known cause), its network reachability from the servers, and that account JWTs were pushed within any initial window the resolver requires. Existing connections survive a resolver outage, so prioritize keeping healthy clients connected while you fix it. Do not mass-restart clients during the outage: they will fail to reconnect and convert a partial incident into a total one.

Rotation partially deployed

Roll credentials forward consistently or roll back to the previous valid set; do not leave the fleet split. The correct rotation procedure is overlap, not cutover: issue new credentials while the old ones still validate, deploy to clients, confirm all clients authenticate on the new material, then retire the old credentials. In JWT mode, updated account JWTs can be pushed to the resolver without restarting servers; the operator JWT, by contrast, requires replacing the file on the server and triggering a config reload, which is an easy step to miss.

Scanning or brute force

Do not tune credentials in response to external probing. Restrict reachability of the NATS client port with firewall rules or security groups so only your fleet can attempt connections. If token auth is in use, treat this as the trigger to migrate: bearer tokens are sent in cleartext during the CONNECT handshake unless TLS is enforced, and NKey/JWT auth removes the shared-secret problem entirely.

Prevention

Stagger credential expiration. Never issue user JWTs that share one expiration timestamp across the fleet. Stagger expirations or use short-lived JWTs with an automated refresh path, and alert on time-to-expiry for credentials the way you would for TLS certificates.
Alert on the violation rate, not the event. Baseline the log rate and alert on spikes, with separate handling for “from our fleet” (page the owning team) and “from unknown sources” (security review).
Watch churn, not just count. Track the total_connections delta. A stable connections value hides a retry storm that is burning CPU and log volume.
Test rotation as a drill. Rotate credentials in staging with the overlap procedure and verify zero violations during the overlap window. Rotation procedures that are never rehearsed are the ones that cause the 3 a.m. flood.
Monitor the resolver as a dependency. If you run JWT mode with a resolver, its storage health and reachability are part of your auth availability. Treat resolver failure as an auth outage, because it is one.
Enforce TLS. Without it, token and password credentials cross the wire in cleartext during CONNECT, which turns any network visibility into a credential leak.

How Netdata helps

Netdata collects /varz connection metrics per server, so you can see the connections drop and the total_connections churn rate on the same dashboard as the moment the log flood started.
Correlating connection churn with CPU shows when a retry storm driven by bad credentials has turned into a TLS handshake capacity problem.
Uptime tracking rules out restarts as a confounder when you are establishing whether violations preceded or followed a server event.
Because NATS does not expose auth failures as a standard /varz field, pairing Netdata’s per-second server metrics with log-based alerting on the violation rate gives you both halves of the picture: the symptom in the logs and the blast radius in the connections graph.
In a cluster, per-node correlation shows whether violations hit all servers (systemic credential event) or one server (resolver or config issue local to that node).