ZooKeeper authentication failures: SASL/Digest auth_failed_count climbing

The zk_auth_failed_count counter exposed by ZooKeeper’s mntr four-letter command increments every time a client fails authentication under the Digest or SASL schemes. In a stable, locked-down production ensemble this counter is effectively flat between restarts. When it moves, a client is connecting with credentials the server rejects.

The metric is per-server and cumulative since process start. It does not break out by auth scheme, source IP, or principal, so the counter alone tells you something is wrong but not who or why. You resolve the “who and why” by reading the ZooKeeper log, the surrounding metrics (zk_ensemble_auth_fail , zk_connection_rejected ), and the deployment timeline.

A short burst right after a credential rotation is expected and self-resolving. A sustained rate above roughly 10 failures per second suggests widespread misconfiguration (a fleet of clients on stale credentials) or an unauthorized access attempt. Either way it deserves a ticket, not silence. When the failures are server-to-server rather than client-to-server, you are looking at a quorum threat, not a client nuisance.

What this means

zk_auth_failed_count covers two authentication paths:

  • Digest authentication: username and password verified locally by the server’s DigestAuthenticationProvider. A failure means the credentials the client sent do not match what the server has on file.
  • SASL authentication: typically GSSAPI/Kerberos in production, sometimes DIGEST-MD5. A failure means the SASL handshake did not complete. Causes include an expired ticket, a missing server-side JAAS entry, a hostname mismatch against the service principal, or a client library that is not configured to do SASL at all.

The counter increments on either path. To know which one you are hitting, read the log line ZooKeeper emits alongside the increment.

Operational notes that shape the diagnosis:

  • Since ZooKeeper 3.5.3, four-letter commands must be whitelisted via 4lw.commands.whitelist. If mntr returns nothing, your monitoring is blind to this counter and everything else it exposes. The AdminServer on port 8080 (3.5+) is an HTTP alternative that can be authenticated separately.
  • zk_auth_failed_count is a client-side signal. The companion metric zk_ensemble_auth_fail tracks server-to-server (quorum peer) authentication failures. Any non-zero value there is more serious because it threatens quorum, not just a single client.
  • zk_connection_rejected is a different failure: the per-source-IP maxClientCnxns limit (default 60) has been hit. Reconnection storms from auth-failing clients can drive both counters together.
  • The ZooKeeper security page is explicit that Digest transmits the password during authentication and stores an unsalted SHA-1 hash. Treat Digest as coarse access control, not as a confidentiality mechanism. Prefer SASL/Kerberos or mTLS where credential secrecy matters.

Common causes

CauseWhat it looks likeFirst thing to check
Credential rotation not propagated to clientsSustained auth_failed spike beginning at deploy time; only some clients failRecently shipped client config or keytabs
Expired or renewal-failed Kerberos ticketWave-like failures across many clients; KDC unreachable or slow during renewalklist on a failing client; KDC reachability from the ZK host
Server-side JAAS misconfigured or missing“No password found for user: null” or “quorum member’s saslToken is null” in the ZK logServer JVM startup flags and the JAAS config file
Hostname alias mismatch against Kerberos principal“GSS initiate failed [Caused by GSSException: … Checksum failed]” in logsServer hostname vs. service principal (ZOOKEEPER-4334)
Unauthorized access attemptFailures from IPs not in the client inventory, climbing rate, no deploy eventSource IPs in the ZK log; client-port network ACLs
maxClientCnxns saturation compounding the stormauth_failed climbing alongside zk_connection_rejectedPer-IP connection counts via cons
SASL Quorum Peer auth bypass (CVE-2023-44981)zk_ensemble_auth_fail increments; unexpected endpoints attempting to join quorumPatch level (fixed in 3.7.2, 3.8.3, 3.9.1)

Quick checks

All of the following are read-only and safe to run on a live server.

# Check the auth_failed counter twice, seconds apart, to confirm it is moving
echo mntr | nc localhost 2181 | grep zk_auth_failed_count

# Server-to-server auth failures (higher-stakes, threatens quorum)
echo mntr | nc localhost 2181 | grep -E 'zk_ensemble_auth'

# Confirm mntr is even whitelisted (empty reply means monitoring is blind)
echo ruok | nc localhost 2181     # imok = process alive

# Confirm the node is functional, not in read-only mode
echo isro | nc localhost 2181     # rw = healthy; ro = quorum lost

# Correlate with connection rejection (per-IP maxClientCnxns saturation)
echo mntr | nc localhost 2181 | grep -E 'zk_(connection_rejected|num_alive_connections)'

# Recent auth-related log lines (log path varies by distribution)
grep -iE 'auth|sasl|denied|noauth' /var/log/zookeeper/zookeeper.log | tail -50

# Per-connection snapshot to identify noisy source IPs (expensive on loaded servers, sample sparingly)
echo cons | nc localhost 2181 | head -30

How to diagnose it

The counter tells you only that something is failing. The diagnostic job is to classify the failure, identify the source, and decide whether the threat is to a single client or to the ensemble.

flowchart TD
    A["zk_auth_failed_count moving"] --> B{"Sustained non-zero rate?"}
    B -- No, isolated burst --> C["Recent credential rotation?"]
    C -- Yes --> D["Expected. Watch for self-heal."]
    C -- No --> E["Single misconfigured client. Identify via logs."]
    B -- Yes --> F{"zk_ensemble_auth_fail also moving?"}
    F -- Yes --> G["Quorum threat. Check CVE-2023-44981 patch level and quorum JAAS."]
    F -- No --> H{"Which auth scheme is failing?"}
    H -- Digest --> I["Verify client credentials and server JAAS digest entries"]
    H -- SASL / GSSAPI --> J["Check Kerberos: klist, KDC reachability, hostnames vs principals"]
    H -- Unknown --> K["Grep ZK log for source IPs and exact error strings"]
  1. Confirm rate, not absolute value. zk_auth_failed_count is cumulative since restart. Run mntr twice, seconds apart, and compute the delta. A flat counter that has been static for weeks is not the problem you are chasing.
  2. Classify the failure window. A short burst that begins within minutes of a known deploy and self-resolves is almost certainly credential propagation. A sustained rate is either fleet-wide misconfiguration or an unauthorized attempt.
  3. Check zk_ensemble_auth_fail first. Server-to-server auth failures are higher-stakes than client auth failures because they threaten quorum. If this counter is moving, prioritize it over zk_auth_failed_count. Verify patch level against CVE-2023-44981, which allowed a SASL Quorum Peer auth bypass on unpatched 3.7.0 through 3.7.1, 3.8.0 through 3.8.2, and 3.9.0.
  4. Read the log for the scheme. The error string disambiguates Digest from SASL. “No password found for user: null” indicates the server’s JAAS config is missing the digest entry for the connecting user. “quorum member’s saslToken is null” indicates a missing or malformed server section in the JAAS config for quorum auth. “GSS initiate failed [Caused by GSSException: … Checksum failed]” points to a Kerberos hostname alias mismatch (ZOOKEEPER-4334) or a stale keytab.
  5. Identify the source. cons is expensive on a loaded server, but a single sampled snapshot will show which IPs are connected. Cross-reference against your client inventory. Unknown IPs in a locked-down environment warrant immediate network-ACL review.
  6. Correlate with zk_connection_rejected. If maxClientCnxns (default 60 per source IP) is being hit in the same window, the failures may be compounding a reconnection storm. Containerized deployments where many pods share a host IP hit this limit quickly.
  7. Cross-reference the deployment timeline. Most sustained spikes in production correlate with a config push, a keytab rotation, a KDC maintenance window, or a network change. The counter rarely lies about when; it just does not say what.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
zk_auth_failed_countThe headline counter; confirms auth is failingAny sustained non-zero rate; >10/sec suggests brute force or fleet-wide misconfig
zk_ensemble_auth_failServer-to-server auth; threatens quorumAny non-zero increment is abnormal
zk_connection_rejectedper-IP maxClientCnxns saturationIncrementing alongside auth failures suggests a reconnection storm
zk_num_alive_connectionsTotal active client connectionsSharp drop precedes or accompanies mass auth-driven disconnects
zk_stale_sessions_expiredSessions dying from heartbeat missSpike indicates cascading impact on dependent services (Kafka, HBase, Solr)
zk_insecure_admin_countAdmin operations performed without authNon-zero in production where admin auth is expected
zk_non_mtls_remote_conn_countRemote connections without mutual TLSNon-zero in mTLS-required environments

Fixes

Credential rotation not propagated

Refresh client configs or keytabs on the affected fleet. For Kerberos, ensure the keytab is readable by the client process and that the principal in the JAAS config matches the keytab contents. For Digest, update the server’s DigestAuthenticationProvider entries and verify the super-digest configuration if you use one. Verify against a single known-good client before rolling fleet-wide.

Expired or renewal-failed Kerberos ticket

ZooKeeper’s SASL login module renews TGTs on a schedule, but renewal fails silently if the KDC is unreachable at the renewal window. Investigate KDC reachability from the ZooKeeper host, then either restart the affected clients with a fresh ticket or fix the KDC. If this is recurring, investigate the renewal interval (jaasLoginRenew ) and the KDC’s ticket lifetime and renewal policy.

Server-side JAAS misconfigured or missing

The JAAS config (-Djava.security.auth.login.config) must be explicitly passed in the ZooKeeper server start script. Missing this is a common source of “ZooKeeper server cannot authenticate itself properly” errors. Verify the config file has a Server section for client-facing SASL and, if you have enabled quorum SASL, the corresponding QuorumServer and QuorumLearner sections.

Hostname alias mismatch (ZOOKEEPER-4334)

If the ZooKeeper server connects using a hostname alias that does not match the service principal in the keytab, GSSAPI checksum failures occur. Align the hostname the client uses with the principal, or regenerate the keytab against the canonical FQDN.

maxClientCnxns saturation

If auth-failing clients are reconnecting in a tight loop and saturating the per-IP connection limit, raise maxClientCnxns with care. The default of 60 is per source IP, and containerized deployments with many pods behind one host IP can exceed it trivially. The real fix is usually to stop the reconnect storm at the source, not to raise the limit indefinitely.

Unauthorized access attempts

If the source IPs are not in your client inventory, this is a security event, not an operational nuisance. Lock down the ZooKeeper client port with network ACLs, enable audit.enable=true (3.6+) for forensic capture of auth and mutation events, and require SASL or mTLS at the server. Do not rely on Digest alone for confidentiality given its cleartext password transmission and unsalted SHA-1 hash storage.

SASL Quorum Peer bypass (CVE-2023-44981)

If zk_ensemble_auth_fail is incrementing, prioritize patching. CVE-2023-44981 allowed a SASL ID without the instance part (for example [email protected] instead of eve/[email protected]) to skip the authorization check entirely when quorum.auth.enableSasl=true. Affected versions include 3.7.0 through 3.7.1, 3.8.0 through 3.8.2, and 3.9.0. Fixed in 3.7.2, 3.8.3, and 3.9.1. Verify patch level on every ensemble member.

Prevention

  • Alert on rate, not absolute value. The counter is cumulative since restart. Alert on increase(zk_auth_failed_count[5m]) > 0 outside maintenance windows, and page on zk_ensemble_auth_fail incrementing at all.
  • Gate credential rotations in maintenance windows. A short burst after a rotation is expected; treat any sustained rate as an incident.
  • Monitor zk_ensemble_auth_fail as a separate, higher-severity signal. Client auth failures are a nuisance. Quorum peer auth failures are an availability and integrity threat.
  • Enable audit logging on production ensembles. audit.enable=true (3.6+) gives you per-operation forensics for sensitive znode mutations and authentication events.
  • Require SASL or mTLS at the server. sessionRequireClientSASLAuth (3.6.0+) lets the server reject clients that do not authenticate via SASL.
  • Lock down the client port with network ACLs. Defense in depth matters. An unknown source hitting auth failures is worth investigating even when auth is enforced at the application layer.
  • Track patch level against the Apache ZooKeeper security advisories. SASL-related CVEs have historically been high-severity and easy to miss on slow-moving ensembles.

How Netdata helps

  • Per-second collection of zk_auth_failed_count and zk_ensemble_auth_fail exposes rate changes long before a slower scrape interval would surface them.
  • Correlation views let you put auth failures next to zk_connection_rejected, zk_num_alive_connections, and zk_stale_sessions_expired to distinguish a misconfigured client from a reconnection storm from a downstream cascade.
  • ML anomaly detection flags unusual increments even when the absolute rate is below a static threshold, useful for catching slow credential drift or low-and-slow unauthorized probing.
  • Per-node dashboards localize which ensemble member is seeing the failures, which matters when only one node is reachable from a problematic network path.
  • Long-term retention makes it cheap to answer “when did this start?” by overlaying the counter on deployment events weeks after the fact.