ZooKeeper with no authentication: the open-by-default coordination store

ZooKeeper ships with no authentication by default. Any TCP client that can reach port 2181 can open a session, read any znode, and write any znode whose ACL has not been explicitly restricted. The default ACL on znodes created without an explicit ACL is OPEN_ACL_UNSAFE: world:anyone with full cdrwa (create, read, write, delete, admin) permissions.

This matters because ZooKeeper is the coordination store for systems that treat its contents as authoritative: Kafka broker registrations and controller elections (pre-KRaft), HBase region assignment and master election, HDFS NameNode HA fencing state. An unauthenticated writer in those subtrees can silently corrupt cluster state, force leader changes, or trigger cascading failovers, and the writes succeed because the ACL permits them. There is no second layer that catches them.

The lockdown order matters: inventory first, then audit logging, then SASL authentication, then per-znode ACLs, then network segmentation. Skipping ahead to “turn on SASL” without an inventory is how teams discover, mid-rollout, that a critical client they forgot about can no longer connect.

What this enables

  • A documented inventory of every IP currently connected to 2181, so you can spot unexpected clients before you flip on auth and break them.
  • An audit trail of every create, setData, delete, setACL, and reconfig operation against critical znodes, with the client session and identity, for post-incident forensics.
  • A path from OPEN_ACL_UNSAFE to authenticated, per-znode ACL-controlled access, where credentials you actually trust are required to mutate coordination data.
  • Defense in depth: if one control fails (a leaked digest, a misconfigured security group, an upstream CVE), the others still constrain the blast radius.

Each layer fails independently and logs differently:

flowchart TD
  A[Client reaches 2181] --> B{Network ACL
allows source IP?}
  B -- no --> X[Blocked at firewall]
  B -- yes --> C{SASL auth
valid credentials?}
  C -- no --> Y[Session rejected]
  C -- yes --> D{Per-znode ACL
permits operation?}
  D -- no --> Z[NoAuth logged]
  D -- yes --> E[Mutation applied]
  E --> F[Audit log records
user, IP, path]

Prerequisites

  • ZooKeeper 3.6.x or newer. Audit logging and sessionRequireClientSASLAuth both require 3.6+.
  • Shell access to each ensemble node, with permission to edit zoo.cfg and rolling-restart the QuorumPeerMain process.
  • A known-client inventory: the list of IPs and CIDRs that should be connecting. Without it, the connection log is noise.
  • A rolling-restart maintenance window once authentication settings change. Existing clients using the open posture will be rejected when you enforce SASL.
  • Familiarity with the difference between client port 2181, AdminServer port 8080 (3.5+, on by default), and the inter-ensemble quorum and election ports.

Procedure

1. Confirm the current exposure

Read-only, safe to run on production.

# Check zoo.cfg path on your distro; common locations:
# /etc/zookeeper/conf/zoo.cfg  (Debian/Ubuntu)
# /etc/zookeeper/zoo.cfg       (RHEL/CentOS)
# /opt/zookeeper/conf/zoo.cfg  (tarball installs)

# Confirm no auth is configured on the client port
grep -E 'authProvider|sessionRequireClientSASLAuth|kerberos|sasl' /etc/zookeeper/conf/zoo.cfg
# Default install: returns nothing

# Confirm audit is disabled
grep -E 'audit.enable|audit.log.dir' /etc/zookeeper/conf/zoo.cfg
# Default install: returns nothing

# Confirm the four-letter-word surface (3.5.3+)
grep '4lw.commands.whitelist' /etc/zookeeper/conf/zoo.cfg
# Default: only 'srvr' is whitelisted

If the SASL/authProvider grep returns nothing and 2181 is reachable from outside the deployment network, you are running open by default.

2. Inventory who is connecting today

Read-only, but the Accepted socket connection log is verbose. Sample it; do not tail indefinitely on a busy node.

# Distinct source IPs recently connected, from the ZK server log
grep 'Accepted socket connection' /var/log/zookeeper/zookeeper.log \
  | awk '{print $NF}' | sort -u
<!-- TODO: verify the exact field where the source IP appears in the 3.9.x
'Accepted socket connection' line. The pattern is real; the awk column
may need adjustment per version. -->

# Live connection snapshot via four-letter-word 'cons'
# (must be in 4lw.commands.whitelist; expensive on busy nodes)
echo cons | nc localhost 2181 | head

Diff the source IPs against your known-client inventory. Anything outside the inventory is the starting point for investigation. Run cons once, off-peak; it is O(n) in connection count and will perturb latency on a loaded node.

3. Turn on audit logging (3.6+)

Edit zoo.cfg on each node, then rolling-restart:

audit.enable=true
audit.log.dir=/var/log/zookeeper/audit

After restart, every mutation (create, setData, delete, setACL, multi, reconfig) is written to zookeeper_audit.log with session, user (once SASL is on), and znode path. Reads are not audited by default.

Verify:

# WARNING: this writes to production ZK. Use a throwaway test path and clean up.
zkCli.sh -server localhost:2181 create /audit-test "hello"
tail -n 5 /var/log/zookeeper/audit/zookeeper_audit.log

# Clean up
zkCli.sh -server localhost:2181 delete /audit-test

Leave audit logging on from this point forward. It is the only forensic record you will have once you start locking things down, and the only way to answer “who mutated /kafka/controller?” after the fact.

4. Stand up SASL authentication

This is the disruptive step. Stage it as two flips, not one.

Digest-MD5 is the lowest-friction option. For each client identity, generate user:password, hash per the digest scheme, and configure ZooKeeper to load a JAAS file. Kerberos, SASL/PLAIN, and mTLS are alternatives; pick based on your existing identity infrastructure.

In zoo.cfg:

authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
sessionRequireClientSASLAuth=true

authProvider.1 enables SASL handling on the server. sessionRequireClientSASLAuth=true (3.6+) forces all client sessions to authenticate; any client that does not present valid SASL credentials is rejected. Roll out authProvider first, confirm no production client is being rejected via zk_auth_failed_count, then flip sessionRequireClientSASLAuth.

Caveats from the upstream security guidance:

  • Digest transmits the password in the clear and stores an unsalted SHA-1 hash. Use SASL/Kerberos or mTLS where credential confidentiality matters.
  • Quorum peer auth (quorum.auth.enableSasl=true) is a separate control and is not enabled by default. CVE-2023-44981 affected this path; if you enable it, run 3.9.1+, 3.8.3+, or 3.7.2.
  • The ip ACL scheme is spoofable. It trusts source IP. Use it only as a secondary check.

5. Apply per-znode ACLs

ACLs in ZooKeeper are not recursive. Each znode must have its ACL set independently. The default OPEN_ACL_UNSAFE (world:anyone:cdrwa) is permissive; lockdown means explicitly setting auth: or digest:user: ACLs on each critical subtree.

For the major ecosystems:

  • Kafka (ZooKeeper mode): set zookeeper.set.acl=true on brokers so the broker applies ACLs to its znodes at creation. Without this, broker-registered znodes remain world-writable even after SASL is on. Kafka is migrating to KRaft; for ZooKeeper-mode clusters, this is still the lever.
  • HBase and YARN: these do not set ACLs for their znodes by default. Locking them down requires a tool like zkpolicy or an explicit per-znode ACL walk after SASL is enabled.
# WARNING: setAcl on production coordination paths is disruptive.
# Test on a non-production node first. Locking /kafka before brokers
# have working SASL credentials will break the cluster.
zkCli.sh -server localhost:2181
# Inside the CLI:
setAcl /kafka auth:cdrwa
setAcl /hbase auth:cdrwa
getAcl /kafka

Children must be walked and set individually. A parent ACL does not flow down, which is the single most common reason a “locked-down” ensemble is still writable by an anonymous client.

6. Tighten the surface around the client and admin ports

  • Four-letter-word whitelist (3.5.3+): explicitly list only what monitoring needs (mntr, ruok, isro, srvr). Do not whitelist cons, wchc, wchp, dump, or envi in production. They expose connection and watch detail and are O(n) expensive.
  • AdminServer (3.5+, port 8080, on by default): bind to localhost or an internal interface, and put it behind network policy. On 3.9.0+, consider admin.snapshot.enabled=false and admin.restore.enabled=false unless you actively use those endpoints.
  • maxClientCnxns (default 60 per source IP) is a DoS guard, not an auth control. Do not raise it to work around an auth change.

7. Network segmentation

Port 2181 should be reachable only from known client CIDRs and the load balancer you control. Port 8080 (AdminServer) and the quorum and election ports should never be reachable outside the ensemble subnet. Apply this at the security-group or firewall layer, not only at ZooKeeper. Adjacent endpoints have had bypasses (CVE-2024-23944 persistent watchers, CVE-2024-51504 AdminServer, CVE-2026-24281 reverse-DNS hostname verification), so do not treat any single control as sufficient.

Verifying it works

After the rolling restart and ACL walk:

# 1. Anonymous client should be rejected at session establishment.
# NOTE: four-letter-word commands like 'ruok' do NOT establish a session
# and are not subject to sessionRequireClientSASLAuth. They will still
# respond. The correct test for anonymous rejection is zkCli without JAAS.
<!-- TODO: verify that 4LW commands bypass sessionRequireClientSASLAuth
on the target ZK version. If they do, replace this check with:
  zkCli.sh -server localhost:2181 ls /
# and confirm it fails with auth error. -->

# 2. Authenticated client should succeed
zkCli.sh -server localhost:2181  # with JAAS configured
ls /kafka

# 3. ACL on a protected path should be visible and restrictive
getAcl /kafka

# 4. Audit log should now record the authenticated user, not anonymous
tail -n 20 /var/log/zookeeper/audit/zookeeper_audit.log

Run a controlled failover of one dependent service (one Kafka broker, one HBase RegionServer) and confirm two things: it can re-register under the new ACL, and the audit log captures the re-registration with the expected identity. If re-registration fails, you have an ACL gap; do not continue the rollout.

Common pitfalls

  • ACLs are not recursive. setAcl /hbase auth:cdrwa does not protect /hbase/rs or any child. You must walk and set each znode, or use a tool that does.
  • The ip ACL scheme is spoofable. It matches source IP and trusts network-level controls. Never use it as the only control on sensitive paths.
  • Digest auth transmits passwords in the clear. Acceptable inside a segmented network, unacceptable across an untrusted one. Use Kerberos or mTLS there.
  • HBase and YARN do not set ACLs by default. Turning SASL on without a parallel ACL walk leaves their znodes world-writable. SASL gates session creation, not per-node writes against OPEN_ACL_UNSAFE.
  • AdminServer is on by default on 8080. If your firewall assumed ZK listens only on 2181, you have a second exposure point. Confirm admin.enableServer and the bind address.
  • Four-letter-word whitelist defaults to srvr only. Monitoring that depends on mntr silently breaks on upgrade to 3.5.3+. Explicitly whitelist what you need.
  • Forcing SASL kicks out existing clients. Apply authProvider first; only flip sessionRequireClientSASLAuth=true once all known clients have working credentials.

Signals to monitor

SignalWhy it mattersWarning sign
zk_auth_failed_countCounts SASL/Digest auth failures. Should be zero in steady state.Sustained non-zero rate after rollout: a client did not get new credentials, or a brute-force attempt.
zk_ensemble_auth_failServer-to-server auth failures between quorum members.Any non-zero increment threatens quorum.
zk_insecure_admin_countAdministrative operations performed without authentication.Non-zero in a hardened deployment means a control is bypassed.
zk_non_mtls_remote_conn_countRemote connections not using mutual TLS.Non-zero in an mTLS-required environment.
zk_unsuccessful_handshake / zk_tls_handshake_exceededTLS handshake failures and timeouts.Spikes after cert rotation or mTLS rollout.
Accepted socket connection log entriesSource IPs reaching 2181.IPs outside the known-client inventory.
NoAuth / Permission denied log entriesACL denials.Repeated denials from a single source warrant investigation.
audit.enable=true + zookeeper_audit.logForensic record of mutations on critical paths.Mutation on a coordination subtree by an unexpected identity.

How Netdata helps

  • Per-second collection of zk_auth_failed_count and connection-state metrics from mntr, so a burst of auth failures (a misconfigured rollout or a probing attempt) surfaces in the same window as the underlying connection changes, not on a 5-minute scrape cadence.
  • The connection and log-derived signals (Accepted socket connection, NoAuth) can be correlated against per-second auth counters in a single dashboard, which is the diagnostic gap that turns a bad SASL rollout into a multi-hour incident.
  • ML anomaly detection on zk_num_alive_connections and zk_packets_received catches the indirect signal of a lockdown change: the client fleet disconnecting and reconnecting as the new auth posture takes effect.
  • Per-node dashboards make a rolling restart visible. You can watch each node cycle and confirm zk_server_state returns to leader or follower without an election storm, which is when a SASL rollout usually goes sideways.