NATS Permissions Violation: authorized clients publishing or subscribing out of scope

A Permissions Violation entry in the NATS server log means an authenticated client tried to publish or subscribe to a subject outside its permitted scope. The server rejected the operation, logged the event with the subject, client IP, and account, and sent an -ERR back to the client. The connection stays open. This is not an authentication failure: the credentials were accepted, but the permissions attached to them do not cover what the client tried to do.

Most of the time this is an operational bug, not an attack. The two dominant causes are a credential rotation that assigned the wrong permission set, and a service presenting another service’s credentials. A third, quieter cause is a permission that used to be a broad > wildcard and was recently tightened, exposing every place the client secretly depended on full access. Occasionally, violations from unknown IPs targeting system namespaces are genuine probing, and those are a page, not a ticket.

The client side is unreliable here. The protocol-level errors are -ERR 'Permissions Violation for Publish to <subject>' and -ERR 'Permissions Violation for Subscription to <subject>', and they are non-fatal. Several client libraries do not surface them as synchronous errors: some never throw on publish violations, some only expose them through an async error callback, and some close the connection on a subscription violation. Do not assume the application will tell you it is being denied. The server log is the authoritative source.

What this means

NATS evaluates permissions at the subject level, per operation. A user’s publish and subscribe permission maps each have allow and deny lists, with deny taking priority over allow. When a client publishes or subscribes, the server matches the subject against those lists. On a mismatch, the operation is rejected and the violation is logged. Because the check is per operation, a client can connect and authenticate cleanly and still generate violations on every message it touches.

Two structural facts matter for diagnosis. First, a broad > permission silently grants everything, including system subjects like $SYS.> and $JS.API.>. Over-permissioned credentials are common, and they hide problems until someone tightens them. Second, permissions violations are not exposed through the monitoring HTTP endpoints (/varz, /connz, /jsz). There is no counter to scrape. Log parsing is the only way to see them, which is why these events are chronically under-monitored.

Common causes

Quick checks

All of these are read-only.

# Confirm the violations and see the most recent ones
grep "Permissions Violation" /var/log/nats/nats-server.log | tail -20

# Rate: is this a trickle or a flood?
grep -c "Permissions Violation" /var/log/nats/nats-server.log

# Authentication problems travel with permission problems during bad rotations
grep -c "Authorization Violation" /var/log/nats/nats-server.log
grep -c "Authentication Timeout" /var/log/nats/nats-server.log

# Who is connected to the system account right now?
curl -s "http://localhost:8222/connz?auth=true" | jq '.connections[] | select(.account == "SYSTEM") | {cid, ip, name, subscriptions}'

# Connection churn: a mispermissioned client in a reconnect loop hides behind a stable count
curl -s http://localhost:8222/varz | jq '{active: .connections, total: .total_connections}'

# If violations touch JetStream API subjects, expect API errors to climb too
curl -s http://localhost:8222/jsz | jq '{api_total: .api.total, api_errors: .api.errors}'

The log path above assumes the server logs to /var/log/nats/nats-server.log. Under systemd with journal logging, use journalctl -u nats-server | grep "Permissions Violation" instead.

How to diagnose it

1. Extract the violator identity. Each violation log line carries the subject, the client IP, and the account. Group recent violations by those three fields. A single user hitting a single subject repeatedly is a misconfiguration. Many subjects from one source in a short window looks like probing or a client defaulting to wildcard subscriptions it is not entitled to.

2. Classify the subject. The subject tells you which fix applies. _INBOX.* means request-reply without inbox subscribe permission. $JS.API.* means the client can publish data but was never granted the JetStream management subjects. $SYS.> targets mean someone is reaching for server internals. Ordinary business subjects point at the application’s own allow list.

3. Classify the source. Is the client IP a known service host? If yes, this is a credentials or permissions problem, and the question is which credential the client presents versus which one it should present. If the IP is unknown, stop treating this as a config bug.

4. Check the timing. Violations that begin at a credential rotation, a config reload, or a deployment are almost certainly caused by that change. Correlate the first violation timestamp against your change log before digging into permission semantics.

5. Confirm what the client believes. Because client libraries surface these errors inconsistently, the application may look healthy while every publish is denied. Check for missing acknowledgments, stalled processing, or silent data gaps rather than for logged errors.

flowchart TD
  A[Permissions Violation in server log] --> B{Subject target?}
  B -->|_INBOX subjects| C[Request-reply missing inbox subscribe permission]
  B -->|$JS.API subjects| D[JetStream API subjects not in publish allow list]
  B -->|$SYS or admin subjects| E[Treat as security event: page]
  B -->|business subjects| F{Source client known?}
  F -->|known, after rotation| G[Credential or permission misassignment]
  F -->|known, steady state| H[Service using wrong credentials]
  F -->|unknown IP| E
  C --> I[Fix allow list or use allow_responses]
  D --> I
  G --> I
  H --> I
  E --> J[Isolate source, audit system account connections]

Metrics and signals to monitor

Fixes

Correct the credential or permission mapping

If the violation is a legitimate client doing legitimate work, the permission set is wrong. Fix it by granting the specific subjects the client needs, not by widening to >. Remember that deny wins over allow, so an explicit deny can silently override what looks like a correct allow entry. If you use default_permissions, verify the affected user is not silently inheriting a default narrower than intended.

Fix request-reply permissions

A client using request-reply needs subscribe permission on _INBOX.> to receive responses; without it you get violations on randomly generated inbox subjects. For pure responders, allow_responses is the cleaner tool: it dynamically permits publishing to reply subjects for requests the client actually received, while implicitly denying all other publish subjects. Set as true it allows one response per request with no time limit; a map form with max and expires gives tighter control. For stronger isolation, use per-user private inbox prefixes (for example _INBOX_<username>.>) so users cannot receive each other’s replies.

Grant JetStream API subjects explicitly

Publishing to a stream’s data subject and managing streams or consumers are separate permission domains. A client that creates consumers needs publish access to the relevant $JS.API.STREAM.* and $JS.API.CONSUMER.* subjects in addition to its data subjects. Expect api.errors in /jsz to fall once these are granted.

One caveat worth knowing in multi-tenant JetStream setups: subscribe permissions on core subjects are not enforced for JetStream consumer delivery. Messages arrive on inbox subjects, so a user who can create a consumer on a stream can receive messages from subjects they could not subscribe to directly. Control access at the stream and consumer management level, not at the data-subject level.

Patch wildcard deny bypass (CVE-2026-58252)

If you rely on wildcard deny rules (for example deny: ["foo.>"]) as a security boundary, note that nats-server versions from 2.11.0 up to but not including 2.11.16, 2.12.7, and 2.14.0 have an authorization bypass: an authenticated user can evade wildcard denies with certain subscription forms and queue subscriptions. Upgrade to 2.11.16, 2.12.7, 2.14.0, or later. Until patched, treat wildcard denies as advisory, not enforced.

Respond to probing

Violations from unknown IPs, or violations targeting $SYS.> and $JS.API.>, are a page. Isolate the source at the network layer, audit current connections to the system account with /connz?auth=true, and review whether any account has a bare > grant that reaches system subjects. Watch for unexpected connections to the $SYS system account as a standing signal, not just during an incident.

Prevention

• No bare > grants in production. A > publish or subscribe permission silently includes $SYS.>, $JS.API.>, and every future subject. Replace with explicit allow lists, and add explicit denies for system namespaces on ordinary application users.
• Deny by default where possible. Use default_permissions so users without an explicit permission set get a restrictive baseline rather than open access.
• Scope request-reply deliberately. Use allow_responses for responders and private inbox prefixes per user instead of granting _INBOX.> broadly.
• Lint permission config in CI. Permission regressions ship silently because nothing fails until the client calls the denied subject. Diff and review permission changes like firewall rules.
• Alert on the violation log, not just the server. A modest rate threshold (for example more than 10 per minute from one source, or any violation touching system namespaces) catches both bad rotations and probing before the second incident review.
• Keep the server patched. Authorization logic is security-critical code; the wildcard deny CVE is a reminder that deny rules are only as strong as the version enforcing them.

How Netdata helps

• Netdata’s NATS collector polls the HTTP monitoring endpoints and trends total_connections against connections, so a mispermissioned client stuck in a reconnect-and-deny loop shows up as churn even though the violation itself is log-only.
• When violations target JetStream API subjects, the api.errors versus api.total rate from /jsz corroborates that denied management operations are failing server-side, and its recovery confirms your fix landed.
• Baselines on active connection counts per server make unexpected new clients stand out, which matters when you are checking whether a violation source is a known workload or something that should not be there at all.
• Correlating violation windows with server uptime, CPU, and slow-consumer signals helps separate a permissions incident from the look-alikes: a denied client hammering reconnects can masquerade as the early stage of a connection storm.
• Permissions violations are not exposed on any monitoring endpoint, so pair Netdata’s metric view with log collection on the server log and alert on the Permissions Violation pattern directly.

Related guides

• NATS JetStream AckWait tuning: matching the ack timeout to processing time
• NATS route RTT high: inter-server latency that triggers Raft elections
• NATS connection churn: a stable connection count hiding constant reconnects
• NATS connection storm: reconnect thundering herd after a network event
• NATS JetStream consumer lag growing: falling behind the stream
• NATS consumer stalled at MaxAckPending: delivery stops until messages are acked
• NATS JetStream redelivery loop: num_redelivered climbing and messages reprocessed
• NATS JetStream consumer stopped receiving messages: the diagnostic tree
• NATS context deadline exceeded: JetStream publish and request timeouts
• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS file descriptor exhaustion: too many open files and the ulimit cliff
• NATS gateway disconnected: cross-cluster traffic cut in a supercluster