ZooKeeper KeeperErrorCode = NoAuth: ACL denials on protected znodes

KeeperErrorCode = NoAuth for /path appears in client logs when a ZooKeeper operation is rejected because the calling session lacks the ACL permission required for that operation on that znode. The matching server-side line is Permission denied. This is not a transient connectivity issue. The request reached a server, the server evaluated the znode’s ACL, and the session did not match.

ZooKeeper always enforces ACLs. A fresh ensemble uses OPEN_ACL_UNSAFE (world:anyone with CREATE, READ, WRITE, DELETE, ADMIN). NoAuth only appears after someone has explicitly set a restrictive ACL on a znode using one of the schemes digest, sasl, ip, auth, or x509. The error implies a protected znode exists and a client reached it without the matching credential.

The usual causes are a misconfigured or missing SASL/Digest credential, a client that never called addAuthInfo before issuing operations, a credential rotation that left one client behind, or an unauthorized access attempt. Repeated NoAuth from a single source warrants investigation as either a misconfiguration or a probe.

What this means

ZooKeeper evaluates ACLs per operation. Each znode stores an ACL: a list of (scheme:expression, permissions) entries. When a client performs an operation, the server walks the ACL and allows the operation only if some entry matches the client’s authenticated identity and grants the required permission. The permission bits are CREATE, READ, WRITE, DELETE, and ADMIN. getACL requires READ or ADMIN (since 3.6.0, per ZOOKEEPER-1392). setACL requires ADMIN.

Facts that bite operators:

• ACLs are not recursive. A parent’s ACL does not protect its children. Each znode carries its own ACL, set at creation time. A common cause of NoAuth is assuming an ACL set on a parent propagates to children.
• The auth scheme is a wildcard for “identities this session has authenticated as.” Creating a node with an auth-scheme ACL before the creating session has called addAuthInfo fails with InvalidACL. If a node ends up with an empty or misapplied ACL, no client will match.
• The world scheme with anyone is the permissive default. Once a node is created with a restrictive ACL, only matching identities or the super user can change it.
• exists() now checks READ in versions 3.9.2, 3.8.4, and 3.7.3 and later (ZOOKEEPER-2590). Clients that previously probed for node existence without READ permission now receive a permission error where they previously received a silent ok.

The error surface differs by client. The Java client throws org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /path. The CLI (zkCli.sh) reports KeeperError = NoAuth, or, prior to ZOOKEEPER-3891 (fixed in 3.8.0), a misleading Authentication is not valid message for insufficient permissions. The server log records Permission denied. All four are the same event.

flowchart TD
  A["NoAuth on /path"] --> B{"Client called addAuthInfo?"}
  B -- No --> C["Add addAuthInfo before
protected operations"]
  B -- Yes --> D{"Identity matches ACL?"}
  D -- No --> E["Fix digest password
or SASL principal"]
  D -- Yes --> F{"Upgraded to
3.9.2 / 3.8.4 / 3.7.3?"}
  F -- Yes --> G["exists now needs READ.
Grant it or stop probing."]
  F -- No --> H["Source IP expected?
If not, treat as probe."]

Common causes

Quick checks

These are read-only. In ZooKeeper 3.5.3+, four-letter commands must be whitelisted via 4lw.commands.whitelist in zoo.cfg. If mntr returns nothing, whitelisting is the first thing to fix.

# Recent NoAuth denials in the server log
grep -hE "NoAuth|Permission denied" /var/log/zookeeper/zookeeper.log | tail -30

# Failed authentication handshakes (SASL/digest). Distinct from ACL denials.
echo mntr | nc localhost 2181 | grep zk_auth_failed_count

# Confirm the server is healthy enough to be the source of truth
echo ruok | nc localhost 2181
echo isro  | nc localhost 2181

# Server version. Determines whether the ZOOKEEPER-2590 exists() change applies.
echo srvr | nc localhost 2181 | grep -E "Zookeeper version|Mode"

# Connected clients and source IPs. Expensive on busy servers; use sparingly.
echo cons | nc localhost 2181 | head -40

The log path differs by distribution. /var/log/zookeeper/zookeeper.log is the common default; check your service definition if that file is absent.

How to diagnose it

1. Confirm the error is NoAuth. Search the server log for the exact path and operation. The server records Permission denied along with the client session. Confirm the client log shows KeeperErrorCode = NoAuth for /path. If the client shows ConnectionLoss or SessionExpired, this is a different failure.

2. Read the ACL on the znode. Run getACL /path from a session that has READ or ADMIN on the node. If no live client has access, use the super user (see Fixes) or read the ACL from a snapshot off-line. The ACL tells you which scheme and expression the client must match.

3. Identify the client’s authenticated identity. For digest, the identity is user:<base64(SHA1(user:password))>. For sasl, it is the Kerberos principal. For x509, it is the client certificate subject. Compare this identity against the ACL expression exactly.

4. Check whether addAuthInfo was called before the failing operation. In the Java client this is zoo.addAuthInfo("digest", "user:password".getBytes()) or the SASL equivalent at connection. A common bug is creating the ZooKeeper handle and immediately issuing operations before authentication completes.

5. Check the server version. If you recently upgraded to 3.9.2, 3.8.4, or 3.7.3, exists() now checks READ. Clients that worked before by probing without READ will now fail. This is a behavior change, not a misconfiguration.

6. Check for a Kerberos event correlation. With SASL/Kerberos, look for ticket renewal failures in client logs. ZOOKEEPER-4885 (open as of the 3.9.3 timeframe) describes a case where a non-SASL client is created after a Kerberos failure and never recovers, producing persistent NoAuth even after Kerberos is healthy again. The signal is NoAuth from a single client that began at the same timestamp as a renewal failure.

7. Check the source distribution. If NoAuth comes from one host or one client identity repeatedly, it is almost certainly a misconfiguration or a probe. If it affects many clients at once, suspect a credential rotation or an ACL change.

Metrics and signals to monitor

Fixes

Client never called addAuthInfo

Call addAuthInfo on the ZooKeeper handle before issuing any operation against a protected znode:

ZooKeeper zk = new ZooKeeper(connectString, sessionTimeout, watcher);
zk.addAuthInfo("digest", "user:password".getBytes(StandardCharsets.UTF_8));
// now safe to operate on ACL-protected nodes

addAuthInfo is asynchronous in the Java client. The credential is sent to the server and the server responds. Operations issued before the response may still fail with NoAuth. If you need a synchronous guarantee, issue a no-op such as exists on a world-readable node and wait for it to succeed before issuing protected operations.

Digest credential mismatch

If the ACL was set with digest:user:<sha1> and the client presents user:wrongpassword, correct the password on the client. The hash is base64(SHA1(user:password)). Compute it locally to verify:

# Compute the digest auth hash for verification
echo -n "user:password" | openssl dgst -sha1 -binary | base64

Compare against the ACL expression visible via getACL /path. If you cannot recover the original password, your options narrow:

• Reset the ACL using the super user (see below). Requires zookeeper.DigestAuthenticationProvider.superDigest to have been configured on the server.
• Reconstruct the node: delete and recreate with the correct ACL. This is destructive. Ephemeral children vanish, watches fire, and any data on the node is gone. Do this only outside peak traffic and only after checking what depends on the node.

SASL/Kerberos mismatch

The ACL expression for SASL is the Kerberos principal (for example, [email protected]). Ensure the client’s JAAS config presents the matching principal. Common pitfalls:

• The JAAS config file is not loaded (java.security.auth.login.config system property missing or pointing at the wrong file).
• The principal in the keytab does not match the principal in the ACL.
• The ticket has expired and renewal failed. ZOOKEEPER-4477 (fixed in 3.8.1) addressed a case where a single renewal failure prevented all future renewals on Java 9 and later.
• ZOOKEEPER-4885 (open as of 3.9.3): after a Kerberos failure, the client falls back to a non-SASL client and never recovers, producing persistent NoAuth even after Kerberos recovers. The workaround is to recreate the ZooKeeper handle on the client side.

Rebuilding the ZooKeeper handle drops the session, which clears ephemeral nodes and watches. Do this only after confirming the client is stuck in the ZOOKEEPER-4885 pattern.

ACL set with auth before authentication

The auth scheme expands to “all identities this session has authenticated as.” If the node was created with an auth-scheme ACL before the creating session called addAuthInfo, the server rejects it with InvalidACL. If an empty or unmatchable ACL bypassed validation, the fix is to recreate the node with the correct ACL, or to use the super user to run setACL.

Version-driven exists() change

This is not a misconfiguration. If you upgraded to 3.9.2, 3.8.4, or 3.7.3 and clients now fail exists() where they previously succeeded, you have two options:

• Grant READ on the probed nodes to the probing identity.
• Change the client to not depend on exists() returning ok without READ.

Do not work around this with zookeeper.skipACL=yes. That disables all ACL checking on the server and removes a security control. The exists() change closes a probe that should never have been permitted.

Unauthorized access attempt

If the source IP is not in your client inventory, the NoAuth is doing its job. The fix is to prevent the source from reaching the ZooKeeper port: network policy, firewall rules, or removal of the pod or job. In parallel, audit your ACL posture. Confirm that sensitive subtrees (Kafka assignments, HBase region state, distributed lock nodes) carry restrictive ACLs, because the default world:anyone makes them readable and writable by any client that can reach the port.

Super user recovery

For any cause that requires changing an ACL you cannot reach, configure the super user escape hatch on the server:

-Dzookeeper.DigestAuthenticationProvider.superDigest=super:base64(SHA1(super:password))

Then connect as addauth digest super:password and bypass all ACL checks. This is the only recovery path for a node whose ACL has locked out every legitimate client. Configure it before you need it, and store the password in your secrets manager.

Prevention

• Wire addAuthInfo into the connection path. Every client should call it as part of establishing the handle, before any application-level operation.
• Set restrictive ACLs at creation time. Create znodes with explicit (scheme:expression, permissions) rather than relying on defaults.
• Remember ACLs are not recursive. Children do not inherit the parent’s ACL. Protect each child explicitly, or set the creating client’s default ACL.
• Configure superDigest before you need it. Document the password in your secrets store. You will need it during a lockout.
• Track server version in monitoring. The exists() change in 3.9.2/3.8.4/3.7.3 is breaking. Know when you cross that boundary.
• Keep audit logging on for sensitive subtrees. audit.enable=true in zoo.cfg (3.6+) logs mutations with the session and identity. NoAuth denials themselves are already in the server log.
• Do not use skipACL in production. zookeeper.skipACL=yes disables all ACL checking. It is a debug escape hatch, not a configuration.

How Netdata helps

• Correlate zk_auth_failed_count with server log spikes. Authentication handshake failures often precede or accompany NoAuth storms. Seeing both move together in one view narrows the cause from “client never authenticated” to “client authenticated as the wrong identity.”
• Map connection source IPs to NoAuth bursts. Per-second collection on zk_num_alive_connections and the connection rejected counter lets you spot a single source IP producing a burst, the signature of a misconfigured rollout or a probe.
• Track server version across the ensemble. Knowing which nodes are on 3.9.2 or later tells you immediately whether the exists() ACL change is in play, without grepping release notes mid-incident.
• Surface ensemble-wide auth failures. zk_ensemble_auth_fail increments on server-to-server authentication failures, a different problem (quorum auth) but easy to confuse with client NoAuth. Having both on one dashboard prevents misdiagnosis.
• Catch the Kerberos recovery trap. Sustained NoAuth from a single client that began at the same timestamp as a Kerberos ticket renewal failure matches the ZOOKEEPER-4885 pattern. High-resolution collection makes that timestamp correlation visible.

Related guides

• ZooKeeper data size growing: using ZooKeeper as a database is an anti-pattern
• ZooKeeper autopurge not configured: snapshots and logs filling the disk over months
• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• ZooKeeper “Client session timed out, have not heard from server”: the heartbeat miss
• ZooKeeper connection drops spiking: sessions dying in bursts
• ZooKeeper KeeperErrorCode = ConnectionLoss: the transient disconnect every client hits
• ZooKeeper dataLogDir sharing a disk with snapshots: the #1 fsync-latency footgun
• ZooKeeper “Detected pause in JVM or host machine (eg GC)”: the pause-monitor warning
• ZooKeeper data tree digest mismatch: detecting corruption before it spreads
• ZooKeeper transaction log disk full: the crash with no graceful degradation
• ZooKeeper follower doing a SNAP sync: full snapshot transfer and its blast radius