Apache ZooKeeper Operations Guides on Netdata

How ZooKeeper actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How ZooKeeper actually works in production: a mental model for operators

ZooKeeper is a distributed coordination service built on a replicated state machine. It maintains a hierarchical namespace of data nodes (znodes) entirely in memory and replicates every mutation across an ensemble of servers using the ZAB (ZooKeeper Atomic Broadcast) protocol.

This is the mental model the rest of the ZooKeeper runbooks assume. It is the set of abstractions an on-call engineer needs to reason about why writes stall, sessions expire, a “healthy” node can return stale reads, and why the leader matters more than any other node in the cluster.

ZooKeeper "Cannot open channel to N at election address": the blocked election port

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper “Cannot open channel to N at election address”: the blocked election port

The log line is Cannot open channel to <id> at election address /host:3888. It is emitted by QuorumCnxManager.connectOne() when Socket.connect() to a peer’s leader election port fails with ConnectException (refused) or SocketTimeoutException (timed out). The error is harmless during steady state and fatal during an election.

ZooKeeper ensembles use two inter-server TCP ports. Port 2888 (the quorum port) carries the ZAB proposal/ACK/commit stream between followers and the active leader. Port 3888 (the leader election port) is touched only when FastLeaderElection needs pairwise TCP channels to every voting peer. If 2888 is reachable but 3888 is not, the ensemble runs fine until the leader is lost, at which point no new leader can be elected.

ZooKeeper "Client session timed out, have not heard from server": the heartbeat miss

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper “Client session timed out, have not heard from server”: the heartbeat miss

A Client session timed out, have not heard from server in <ms>ms for session id 0x..., closing socket connection and attempting reconnect log line is the client’s SendThread reporting that no PING response arrived inside its heartbeat window. This is a client-side symptom of a heartbeat miss. It is not the server expiring the session, and it is not yet SessionExpired. The client closes its socket and tries another ensemble member.

ZooKeeper "Detected pause in JVM or host machine (eg GC)": the pause-monitor warning

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper “Detected pause in JVM or host machine (eg GC)”: the pause-monitor warning

The log line looks like this:

Detected pause in JVM or host machine (eg GC): pause of approximately 5234ms

ZooKeeper’s JvmPauseMonitor emits that line when the process froze longer than its configured threshold. The monitor thread sleeps for a fixed interval, wakes, and measures how long the sleep actually took. Anything beyond the expected sleep plus the warn threshold gets logged. If the JVM was not in a visible GC at that moment, the line ends with “No GCs detected”, which is the operator’s cue that something else on the host stole CPU.

ZooKeeper "fsync-ing the write ahead log took too long": the disk warning behind most write stalls

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls

The warning:

fsync-ing the write ahead log in SyncThread:0 took 1234ms which will adversely affect operation latency...

fires when fsync on the transaction log exceeds fsync.warningthresholdms (default 1000ms). The wording is deliberate: every write in ZooKeeper blocks on a quorum of fsyncs. If fsync takes a second, every write takes a second. If fsync takes 10 seconds, you are one missed heartbeat away from a leader election.

ZooKeeper "Packet len is out of range": jute.maxbuffer and oversized znodes

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper “Packet len is out of range”: jute.maxbuffer and oversized znodes

Packet len <N> is out of range! looks like a network framing problem. It is not. The ZooKeeper client is telling you the server’s serialized response exceeded the client’s maximum deserialization buffer, and the client closed the connection rather than read a truncated packet.

On the server side, the same condition produces a terser log line: Len error. That appears when a client attempts a write whose payload exceeds the server’s configured buffer limit. Both sides are governed by one Java system property: jute.maxbuffer, which defaults to 0xfffff (1048575 bytes, just under 1 MB).

ZooKeeper "Too many connections from /IP - max is 60": maxClientCnxns rejecting clients

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper “Too many connections from /IP - max is 60”: maxClientCnxns rejecting clients

WARN ... Error accepting new connection: Too many connections from /1.2.3.4 - max is 60 is ZooKeeper’s maxClientCnxns limiter refusing a new TCP connection from a specific source IP. By the time it appears in the server log, the client has already been denied.

The first trap: maxClientCnxns is enforced per source IP, not as a total. A single ZooKeeper node can hold thousands of healthy sessions while still refusing every new connection from one IP. zk_num_alive_connections, the metric most teams watch, is a total. It can look completely normal while clients behind a shared host IP are being silently turned away.

ZooKeeper "Unable to load database on disk": corrupt snapshot on startup

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper “Unable to load database on disk”: corrupt snapshot on startup

The startup error Unable to load database on disk from FileTxnSnapLog means ZooKeeper cannot reconstruct its in-memory data tree from the on-disk snapshot and transaction log. The node refuses to join the ensemble and exits before serving traffic. You typically see this only on the next restart after the corruption happened, often days later.

The failure is nasty because the running process looks fine until it does not. The corruption was already on disk; the restart made it impossible to ignore. A node that was serving requests an hour ago can refuse to come back after an unclean shutdown, an OOMKill, or a full dataDir disk.

ZooKeeper "X is not executed because it is not in the whitelist": four-letter-word commands blocked

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper “X is not executed because it is not in the whitelist”: four-letter-word commands blocked

The error string is exact. When you run echo mntr | nc localhost 2181 against a ZooKeeper 3.5.3+ server that has not been configured for it, the server replies:

mntr is not executed because it is not in the whitelist.

Same shape for ruok, isro, stat, conf, envi, cons, wchs, and the rest of the four-letter-word (4lw) command set. Only srvr works out of the box, because the bundled zkServer.sh status check depends on it.

ZooKeeper authentication failures: SASL/Digest auth_failed_count climbing

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper authentication failures: SASL/Digest auth_failed_count climbing

The zk_auth_failed_count counter exposed by ZooKeeper’s mntr four-letter command increments every time a client fails authentication under the Digest or SASL schemes. In a stable, locked-down production ensemble this counter is effectively flat between restarts. When it moves, a client is connecting with credentials the server rejects.

The metric is per-server and cumulative since process start. It does not break out by auth scheme, source IP, or principal, so the counter alone tells you something is wrong but not who or why. You resolve the “who and why” by reading the ZooKeeper log, the surrounding metrics (zk_ensemble_auth_fail , zk_connection_rejected ), and the deployment timeline.

ZooKeeper autopurge not configured: snapshots and logs filling the disk over months

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper autopurge not configured: snapshots and logs filling the disk over months

autopurge.purgeInterval defaults to 0, meaning snapshots and transaction logs accumulate forever. On a quiet ensemble the growth is slow enough that nobody notices for months, then the dataLogDir partition hits 100%, ZooKeeper cannot fsync the next write, and the process dies. The leader throws an IOException on the transaction log and the ensemble loses a member, or quorum if more than one node fills simultaneously.

ZooKeeper avg_latency hides write stalls: why the headline number lies

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper avg_latency hides write stalls: why the headline number lies

The dashboard says zk_avg_latency is 1.2 ms. Clients are timing out on writes. Both can be true. On a read-heavy ZooKeeper ensemble, the headline latency number can look healthy while the write path is stalled.

Two properties cause this. First, zk_avg_latency, zk_min_latency, and zk_max_latency aggregate reads and writes into one number. Reads are served from local memory and complete in microseconds. Writes require a quorum round-trip plus a transaction log fsync before acknowledgment. When reads dominate the request mix, a severe write stall is diluted by thousands of cheap reads and disappears into the average.

ZooKeeper connection drops spiking: sessions dying in bursts

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper connection drops spiking: sessions dying in bursts

A burst in zk_connection_drop_count means connections to a ZooKeeper server are closing in a tight window, not one at a time. When the burst pushes zk_stale_sessions_expired up simultaneously, you are looking at a session expiration storm in progress or one about to land on dependent services.

Occasional single drops across a large fleet are background noise. A sustained drop rate above roughly 0.1% of total connections per minute is where the signal stops being normal churn. Bursts that fire on a rhythm (every few minutes, hourly, at the same minute past the hour) almost always point to a JVM garbage collection cycle or a scheduled job that briefly saturates the leader.

ZooKeeper data size growing: using ZooKeeper as a database is an anti-pattern

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper data size growing: using ZooKeeper as a database is an anti-pattern

ZooKeeper is a distributed coordination service, not a datastore. It holds the entire znode tree in JVM heap, replicates every mutation through ZAB, and periodically serializes the full tree to a snapshot on disk. The design point is small, hot, strongly consistent metadata: leader election handles, service discovery registrations, distributed locks, configuration pointers. When teams treat it as a general-purpose key-value store and let zk_approximate_data_size climb, they inherit the operational characteristics of an in-memory database without the tooling, schema, or compaction strategies of one.

ZooKeeper data tree digest mismatch: detecting corruption before it spreads

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper data tree digest mismatch: detecting corruption before it spreads

When zk_digest_mismatches_count increments on a ZooKeeper node, the in-memory data tree on that node has diverged from the checksum ZooKeeper expects. This is a data-integrity alarm, not a performance signal. Clients reading from that node may be receiving wrong answers, and if the divergence came from a ZAB replication bug rather than local corruption, the same divergence may be propagating to other ensemble members.

ZooKeeper dataLogDir sharing a disk with snapshots: the #1 fsync-latency footgun

Mon, 01 Jan 0001 00:00:00 +0000

You are chasing intermittent ZooKeeper write-latency spikes that appear to have no cause. Average latency is fine most of the time. Then, every few minutes, p99 update latency jumps by an order of magnitude, zk_outstanding_requests briefly climbs, and clients on tight timeouts see a flicker of connection churn. By the time you SSH in, the cluster looks healthy again.

ZooKeeper follower doing a SNAP sync: full snapshot transfer and its blast radius

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper follower doing a SNAP sync: full snapshot transfer and its blast radius

A follower that fell too far behind the leader does not catch up transaction by transaction. Once its last-seen zxid is older than the leader’s retained transaction log, the leader ships the entire data tree as a snapshot. This is a SNAP sync, the most expensive recovery path a healthy ensemble runs short of a leader election.

ZooKeeper follower sync time climbing: a follower approaching ejection

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper follower sync time climbing: a follower approaching ejection

When zk_avg_follower_sync_time or zk_max_follower_sync_time starts climbing, a follower is taking longer to process proposals from the leader. The metric measures how close a follower is to being ejected from the quorum.

The hard ceiling is syncLimit x tickTime. With defaults of syncLimit=5 and tickTime=2000ms, that ceiling is 10 seconds. When a follower’s sync time approaches that limit, the leader closes the connection, stops pushing updates, and the follower must re-enter leader discovery. In a 3-node ensemble, that drops you to minimum quorum with zero remaining fault tolerance.

ZooKeeper GC pause cascade: how a Stop-the-World freeze expires sessions and re-elects the leader

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper GC pause cascade: how a Stop-the-World freeze expires sessions and re-elects the leader

A ZooKeeper ensemble loses connections in bursts. Sessions expire en masse. The leader changes without a network cause. Request latency spikes on a rhythm that does not match disk I/O. The cause is usually a JVM Stop-the-World pause, and the fix is on the JVM, not the network.

The mechanism: a JVM STW pause freezes the entire ZooKeeper process. No heartbeats go out. No requests are processed. No quorum ACKs flow. The TCP listener still accepts sockets, so the server looks alive from the outside, but the process does nothing with them. Clients miss their heartbeat window and their sessions expire. Followers miss the leader’s heartbeat and, if the pause runs past syncLimit * tickTime (default 10 seconds with tickTime=2000 and syncLimit=5), they declare the leader dead and start a new election. When the JVM resumes, the queued backlog drains as a latency spike and disconnected clients reconnect in a herd.

ZooKeeper heap usage climbing: catching the GC death spiral before it starts

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper heap usage climbing: catching the GC death spiral before it starts

ZooKeeper keeps its entire data tree on the JVM heap: every znode, its data bytes, ACLs, children lists, watch registrations, and session state. When heap climbs, it is the leading indicator for the OOM that eventually kills the ensemble. The signal that matters is not the sawtooth peaks from young-generation GC, but the rising post-GC trough that means the live set itself is growing.

ZooKeeper KeeperErrorCode = ConnectionLoss: the transient disconnect every client hits

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper KeeperErrorCode = ConnectionLoss: the transient disconnect every client hits

KeeperErrorCode = ConnectionLoss is the error every ZooKeeper client eventually logs. It means the TCP connection between the client and the server it was talking to broke before the operation’s response arrived. It does not mean the operation failed, and it does not mean the session is gone. The outcome of the in-flight operation is unknown, and the correct response is an idempotent retry.

ZooKeeper KeeperErrorCode = NoAuth: ACL denials on protected znodes

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper KeeperErrorCode = NoAuth: ACL denials on protected znodes

KeeperErrorCode = NoAuth for /path appears in client logs when a ZooKeeper operation is rejected because the calling session lacks the ACL permission required for that operation on that znode. The matching server-side line is Permission denied. This is not a transient connectivity issue. The request reached a server, the server evaluated the znode’s ACL, and the session did not match.

ZooKeeper KeeperErrorCode = NodeExists: create failing on an already-created znode

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper KeeperErrorCode = NodeExists: create failing on an already-created znode

The error:

org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /some/path

A create() hit a znode path that already exists. ZooKeeper classifies this as a state exception, not a system fault. The cluster refused to overwrite an existing node, exactly as specified. The question is whether the caller expected that path to be free.

Most production hits follow one of two patterns. The first is a retry after ConnectionLoss: the original create() committed but the response was lost in flight. The client does not know whether the operation succeeded, retries, and receives NodeExists. The second is a genuine race: two candidates trying to create the same ephemeral leader or lock node, where exactly one wins and the other should lose gracefully. Both are expected. The incident starts when a client mishandles the exception, or when an orphaned ephemeral blocks the legitimate owner indefinitely.

ZooKeeper KeeperErrorCode = NoNode: operating on a path that doesn't exist

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper KeeperErrorCode = NoNode: operating on a path that doesn’t exist

A client logs KeeperErrorCode = NoNode for /some/path. The server returned Code.NONODE (integer -101), which the Java client surfaces as KeeperException.NoNodeException. The failed operation was a getData, getChildren, exists, setData, delete, or create against a znode that is not currently in the data tree.

NoNode is not a server fault. It is the API contract enforced correctly: ZooKeeper refuses to operate on a missing path. The operator’s job is to find out why the path is missing. Three cases cover almost every incident: the path was never created (usually a missing parent), the path was deleted by the server (an ephemeral tied to an expired session, or a container/TTL node auto-cleaned), or a deploy changed the znode layout clients expect.

ZooKeeper KeeperErrorCode = Session expired: ephemeral nodes gone, clients evicted

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper KeeperErrorCode = Session expired: ephemeral nodes gone, clients evicted

The exact error clients log is KeeperErrorCode = Session expired. On the ZooKeeper side you see Expiring session 0x... timeout of Nms exceeded. Once that line lands, the client’s ZooKeeper handle is dead and every piece of state it owned through that session is gone: ephemeral znodes deleted, watches invalidated, ACLs no longer enforceable. The client cannot reconnect on the same handle. It must build a new ZooKeeper object, negotiate a new session, and recreate every ephemeral node it relied on.

ZooKeeper leader election storm: an ensemble that keeps re-electing

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper leader election storm: an ensemble that keeps re-electing

zk_looking_count is incrementing on multiple nodes. The ZooKeeper log fills with repeated LEADING and FOLLOWING transitions. Every few seconds or minutes, the ensemble elects a new leader, and that leader quickly loses quorum. Writes are intermittent, and downstream systems that depend on ZooKeeper for coordination (Kafka controllers, HBase region assignment, distributed locks) experience cascading failures.

This is a leader election storm. Unlike a single failover where remaining nodes elect a stable replacement, here every potential leader hits the same wall. The root cause is shared across ensemble members: fsync stalls on overloaded storage, GC pauses exceeding the quorum timeout, or intermittent network failures between server pairs. Because every candidate experiences the same problem, no leader holds the role long enough for the cluster to recover.

ZooKeeper monitoring checklist: the signals every production ensemble needs

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper monitoring checklist: the signals every production ensemble needs

A reference checklist for engineers running production ZooKeeper ensembles. Signals are organized into four maturity levels: survival, operational, mature, and expert. Each level adds visibility for failure modes the previous level cannot see.

The levels are cumulative. Level 2 assumes Level 1 is covered. Skipping to Level 4 without Levels 1 through 3 leaves gaps in the signals that actually page you during incidents: disk stalls, GC cascades, and quorum loss.

ZooKeeper monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper monitoring maturity model: from survival to expert

Most ZooKeeper monitoring stops too early. Teams run ruok as their only health check, collect zk_avg_latency without understanding it is cumulative since the last reset, and never look at fsync latency until a write stall cascades into a Kafka outage. The ensemble looks healthy in dashboards until it suddenly does not, and the postmortem reveals the signals were there all along, uncollected.

ZooKeeper OutOfMemoryError: Java heap space - the OOM that kills the whole ensemble at once

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper OutOfMemoryError: Java heap space - the OOM that kills the whole ensemble at once

You grep the ZooKeeper log and find java.lang.OutOfMemoryError: Java heap space. The process is gone. A minute later another node dies with the same error, then the third. The whole ensemble went down inside a single window, not as a rolling failure. That simultaneity is the signature, not a cascade.

ZooKeeper holds the entire data tree on the JVM heap: every znode, its data, ACL references, children lists, stat structures, plus session state, watch tables, and request queues. Every ensemble member holds the same tree. Whatever fills the heap on one node fills it on all of them at roughly the same rate, so when the tree finally exceeds the heap they OOM near-simultaneously. This is a single-cause total outage.

ZooKeeper outstanding requests growing: the request pipeline is backing up

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper outstanding requests growing: the request pipeline is backing up

zk_outstanding_requests counts requests queued in the server’s request processor pipeline that have not yet completed. In steady state it sits at or near zero. When it climbs, something downstream in the pipeline has stopped draining faster than clients are submitting.

The queue is a leading indicator. Requests pile up before zk_avg_latency reacts, and well before the server starts dropping client traffic. If you wait for latency alerts, you have already lost the lead time needed to keep dependent services like Kafka, HBase, and Solr from feeling the stall.

ZooKeeper pending syncs growing: followers can't keep up with the write rate

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper pending syncs growing: followers can’t keep up with the write rate

zk_pending_syncs is the leader’s count of in-flight sync operations to followers. In steady state it sits at zero. Sustained non-zero values mean at least one follower cannot absorb the proposal stream as fast as the leader generates it. This is ZooKeeper’s replication-lag signal, distinct from generic latency metrics because it points directly at the write pipeline’s fan-out side.

ZooKeeper proposals not committing: proposal_count outpacing commit_count

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper proposals not committing: proposal_count outpacing commit_count

In a healthy ZooKeeper ensemble, every proposal the leader broadcasts is committed a few milliseconds later, after a quorum of followers acknowledges it. The two counters zk_proposal_count and zk_commit_count track the front and back of that pipeline, and on a cluster with active writers their rates should track each other closely.

When zk_proposal_count keeps climbing but zk_commit_count stalls, the leader is generating proposals but cannot reach quorum ACK. Writes are not landing. Clients with operation timeouts start failing, ephemeral nodes are not being created, and dependent systems (Kafka controller, HBase master, Solr overseer) start logging “operation timeout” or “session expired”. A worse signal is both metrics staying flat while writers are active: the write pipeline is fully blocked, or the leader is isolated from the quorum.

ZooKeeper quorum ack latency high: followers slow to acknowledge proposals

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper quorum ack latency high: followers slow to acknowledge proposals

zk_p99_quorum_ack_latency is climbing on the leader and every write in the ensemble is paying for it. This metric measures the time from the leader sending a PROPOSE message to receiving quorum acknowledgments from followers. It is a leader-only signal.

Every ZooKeeper write blocks until a quorum of followers ACK. An ACK means the follower has written the proposal to its transaction log and fsync’d it to persistent storage. So quorum ack latency bundles two things into one number: the inter-node network round trip, and the follower’s fsync plus processing time. When this metric is elevated, the slowest follower in the quorum is bottlenecking writes for every client connected to the ensemble.

ZooKeeper quorum loss: no leader elected and every write is failing

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper quorum loss: no leader elected and every write is failing

Every write to your ZooKeeper ensemble is timing out. Clients report ConnectionLoss and SessionExpired. Downstream systems that depend on ZK for coordination, such as Kafka controller elections or HBase region assignment, are cascading into failure. On the surviving ZK nodes, ruok still returns imok. The process is alive; the ensemble is not.

Quorum loss is ZooKeeper’s worst-case availability scenario. When fewer than floor(N/2)+1 voting members can communicate, no leader can be elected and every write fails. Surviving nodes sit in LOOKING state, unable to make progress through ZAB.

ZooKeeper read latency high: memory reads that should never be slow

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper read latency high: memory reads that should never be slow

Reads in ZooKeeper are local heap lookups. A getData, getChildren, or exists call should return in well under a millisecond because no quorum is involved. The connected server walks its in-memory data tree and replies. When zk_p99_readlatency sits above 50ms for minutes at a time, something on that JVM is competing with request processing.

Do not treat read latency like write latency. Write latency is dominated by transaction-log fsync and quorum ACK. Read latency has none of that machinery, so when it climbs the cause is almost always local: a Stop-the-World GC pause, a pathologically deep or wide znode tree pushing heap pressure, or a watch-delivery backlog stealing CPU from the request thread. Correlating zk_p99_readlatency with zk_jvm_pause_time_ms, zk_znode_count, and zk_watch_count is what separates a 30-second diagnosis from an hour of guessing.

ZooKeeper request throttling: globalOutstandingLimit and TCP backpressure

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper request throttling: globalOutstandingLimit and TCP backpressure

zk_throttled_ops incrementing in production is a saturation alarm, not a tuning knob. By the time this counter moves, the request pipeline is already full: the server has stopped reading from client sockets because the global outstanding request queue has reached globalOutstandingLimit (default 1000), TCP backpressure is propagating to every connected client, and any client that cannot absorb the added latency is on its way to a session expiration.

ZooKeeper server stuck in LOOKING: a node that never rejoins the quorum

Mon, 01 Jan 0001 00:00:00 +0000

A single ZooKeeper node sits in LOOKING long after the ensemble has settled, or flaps between LOOKING and FOLLOWING. The rest of the ensemble holds a stable leader with healthy write throughput. Restart the orphaned node and it drops back into LOOKING. Restart the leader and it briefly rejoins, then falls out again.

This is not quorum loss. Quorum loss is every node entering LOOKING at once because no majority can form. That case is covered in ZooKeeper quorum loss: no leader elected and every write is failing. This article covers the narrower symptom: one permanently-orphaned node while the rest of the ensemble serves traffic.

ZooKeeper session count climbing: leaks and duplicate sessions

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper session count climbing: leaks and duplicate sessions

zk_global_sessions climbing while your client fleet is stable is a slow ZooKeeper failure mode. The ensemble keeps serving reads and writes, latency looks fine, quorum is intact, but the session table keeps growing. Each entry costs heap and periodic heartbeat processing. Eventually you hit a GC death spiral, an OOM, or a maxClientCnxns-shaped outage.

The signal is simple to read but easy to misinterpret. ZooKeeper exposes two distinct populations: global sessions, which the leader echoes across the ensemble, and local sessions (only present when localSessionsEnabled=true, added in ZK 3.5, default false), which live on a single follower and upgrade to global when the client creates an ephemeral node. If you only watch zk_global_sessions you may be looking at a subset of the actual session population, and if you only watch zk_num_alive_connections you cannot tell a leak from a deployment event.

ZooKeeper session expiration storm: the ephemeral-node thundering herd

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper session expiration storm: the ephemeral-node thundering herd

A session expiration storm is the worst-case thundering herd in a coordination service. Many clients lose contact long enough for the ensemble to declare their sessions dead. The cluster then deletes every ephemeral node owned by those sessions and fires every watch attached to those nodes. Every disconnected client reconnects at the same time, recreates its ephemeral nodes, and re-registers watches, hammering an ensemble that is already stressed.

ZooKeeper slow startup: snapshot load and txnlog replay taking minutes

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper slow startup: snapshot load and txnlog replay taking minutes

A ZooKeeper node you just restarted is not answering client requests. The process is up, the port is listening, but mntr hangs or returns nothing useful, and dependent services are logging connection failures and session timeouts. Your dashboard shows zk_uptime climbing past one, two, five minutes with no leader participation, or your health checks have already paged because the node looks dead.

ZooKeeper snapshot errors: recovery safety at risk

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper snapshot errors: recovery safety at risk

zk_snapshot_error_count increments when ZooKeeper fails to serialize the in-memory data tree to disk, or fails to load a snapshot during startup. The node keeps serving reads and writes from its in-memory copy. The cost shows up on the next restart, when recovery cannot find a usable snapshot and either fails outright or replays stale state.

A single increment is often transient. A backup job, a brief disk-full condition, or I/O contention from a colocated batch process can cause one snapshot to fail. The next snapshot cycle (taken every snapCount transactions, default 100,000) succeeds and silently recovers. If you page on every increment, you burn through your team’s attention on events that self-resolve.

ZooKeeper split-brain: two nodes both reporting leader

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper split-brain: two nodes both reporting leader

Your monitoring polls each ZooKeeper node independently and two of them return zk_server_state leader. Two servers in the same ensemble both believe they are the active leader. Treat this as an unconditional page: each side can accept writes that diverge from the other.

ZAB (ZooKeeper Atomic Broadcast) is designed to make sustained split-brain impossible. A leader only becomes durable after a quorum of followers (floor(N/2)+1) has acknowledged the NEW_LEADER proposal. In a clean partition, the minority side cannot reach quorum and must block writes. So when two nodes both report leader for more than a brief convergence window, something has broken the quorum accounting: an asymmetric partition, a disk-full recovery bug, an authentication bypass, or a stale-read protocol corner case.

ZooKeeper stale reads from followers: zxid lag and read-after-write surprises

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper stale reads from followers: zxid lag and read-after-write surprises

A follower can pass every standard health check, serve reads at sub-millisecond latency, and still return data dozens of transactions behind what the leader just committed. No error is logged. No alert fires. The ensemble reports a leader, all followers are synced, and quorum is intact.

This is not a bug. ZooKeeper offers sequential consistency for reads, not linearizability. Reads are served locally from each server’s in-memory data tree, and that tree may lag behind the leader’s committed state by anywhere from a few transactions to far more under load. For configuration data or service discovery where eventual consistency is tolerable, this is fine. For distributed locks, leader election fencing, or read-after-write logic, it can cause duplicate work, lock violations, or lost updates.

ZooKeeper stale requests dropped: requests aging out of the pipeline

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper stale requests dropped: requests aging out of the pipeline

zk_stale_requests_dropped incrementing is a late signal in a ZooKeeper saturation cascade. By the time a request ages out of the pipeline and is dropped, the server has already exhausted queue headroom, engaged throttling, and held the request long enough that the client gave up or the connection died. Treat any non-zero rate as an incident, and treat it as proof that an earlier signal was missed.

ZooKeeper synced_followers below ensemble size: degraded fault tolerance

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper synced_followers below ensemble size: degraded fault tolerance

zk_synced_followers reports how many followers are currently synced with the leader. In a healthy ensemble it equals ensemble_size - 1 (voting members only; observers are excluded). When it drops, a follower is disconnected or lagging, and your fault tolerance margin has shrunk.

This metric is emitted only by the leader. Followers, observers, and standalone nodes do not report it. If your collector scrapes a fixed node or only followers, you have a blind spot: identify the leader dynamically, or scrape every node and keep only the values from the node reporting leader in zk_server_state.

ZooKeeper TLS handshake failures: unsuccessful handshakes and non-mTLS connections

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper TLS handshake failures: unsuccessful handshakes and non-mTLS connections

When ZooKeeper is configured for TLS, the TLS layer becomes a new failure surface between clients and the ensemble. A spike in zk_unsuccessful_handshake or zk_tls_handshake_exceeded means clients are attempting TLS connections that never complete. If your environment requires mutual TLS, any non-zero value in zk_non_mtls_remote_conn_count means a client bypassed mTLS entirely.

These failures are noisy in a particular way. The client sees a connection timeout or a refused handshake, while the server logs an “Unsuccessful handshake” entry that does not always explain why. Certificate expiry, cipher mismatch, hostname verification failure, and low host entropy all produce similar surface symptoms. The fix is rarely the server itself; it is usually a client configuration error, an expired credential, or a monitoring tool sending plaintext to a secure port.

ZooKeeper transaction log disk full: the crash with no graceful degradation

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper transaction log disk full: the crash with no graceful degradation

ZooKeeper has no graceful degradation path for a full dataLogDir partition. When the WAL append fails, the server throws an IOException and dies. There is no read-only fallback, no throttling, and no mntr warning that precedes the crash. The same applies to the snapshot directory when the next snapshot write or pre-allocation fails.

The most common root cause is broken or disabled autopurge. With autopurge.purgeInterval defaulting to 0 (disabled) and autopurge.snapRetainCount defaulting to 3, an ensemble that has never been explicitly configured will accumulate transaction logs and snapshots forever. Disk consumption is silent and cliff-edge. By the time ruok fails, the process is already gone.

ZooKeeper unexpected leader election: finding why the leader dropped

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper unexpected leader election: finding why the leader dropped

An unexpected ZooKeeper leader election is an availability event. While the ensemble is in LOOKING state, no writes are processed. Systems that depend on ZK for coordination queue or fail their mutations, and zk_sum_leader_unavailable_time climbs. Treat every unplanned election like a database failover: the cluster recovered, but you still need the root cause.

The cause is on the old leader or on the path to it. Followers call an election when they have not heard from the leader within syncLimit * tickTime (default 5 * 2000ms = 10 seconds). The question is always: what stopped the old leader from sending heartbeats, or what stopped a quorum of followers from receiving them.

ZooKeeper unrecoverable error: when a node's integrity is compromised

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper unrecoverable error: when a node’s integrity is compromised

You got paged because zk_unrecoverable_error_count incremented. This is one of the few mntr counters you never want to see move. It tracks errors ZooKeeper cannot recover from internally: data corruption, invariant violations, or resource exhaustion that puts the node into a state it cannot safely keep serving from.

ZooKeeper is designed to fail fast. When it hits an unrecoverable condition it does not limp along. The critical thread logs a severe error, notifies the supervision listener, and the process exits. By the time the counter increments, the JVM is usually already gone or about to be, and the question is no longer “is this node healthy” but “do I trust its on-disk state at all”.

ZooKeeper watch storm: thousands of notifications when one hot znode changes

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper watch storm: thousands of notifications when one hot znode changes

A single znode changes. Seconds later, zk_packets_sent spikes to many times its normal rate, CPU on the ZooKeeper process surges, and zk_outstanding_requests begins climbing. Clients report latency spikes, connection timeouts, or session expirations. If many clients watch the same znode, you are looking at a watch storm.

The mechanism: ZooKeeper maintains a watch table mapping znode paths to registered watchers. When a watched znode changes, the server queues a notification for every client that registered a watch on that path. For a path with 10,000 watchers, a single setData call produces 10,000 notification packets. Notification serialization and queuing happen inside the request processing pipeline, competing with all other request handling.

ZooKeeper with no authentication: the open-by-default coordination store

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper with no authentication: the open-by-default coordination store

ZooKeeper ships with no authentication by default. Any TCP client that can reach port 2181 can open a session, read any znode, and write any znode whose ACL has not been explicitly restricted. The default ACL on znodes created without an explicit ACL is OPEN_ACL_UNSAFE: world:anyone with full cdrwa (create, read, write, delete, admin) permissions.

This matters because ZooKeeper is the coordination store for systems that treat its contents as authoritative: Kafka broker registrations and controller elections (pre-KRaft), HBase region assignment and master election, HDFS NameNode HA fencing state. An unauthenticated writer in those subtrees can silently corrupt cluster state, force leader changes, or trigger cascading failovers, and the writes succeed because the ACL permits them. There is no second layer that catches them.

ZooKeeper write latency high: read zk_updatelatency, not just avg_latency

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper write latency high: read zk_updatelatency, not just avg_latency

The dashboard says ZooKeeper is fine. zk_avg_latency is 2ms. Clients are timing out anyway: distributed locks expiring mid-acquisition, Kafka controllers flapping, HBase regions bouncing. The signal you are missing is zk_updatelatency, the write-specific latency family that 3.6+ exposes separately from the misleading aggregate.

The trap is structural. zk_avg_latency, zk_min_latency, and zk_max_latency from mntr combine reads and writes into one cumulative statistic. Reads are local in-memory lookups, typically sub-millisecond. Writes require a leader round-trip, a ZAB proposal, a quorum ACK, a commit, and an fsync. When read volume dominates, a healthy average masks pathological write latency. These are also server-cumulative statistics since the last srst reset, not sliding windows: a single fsync stall from three hours ago still inflates zk_max_latency.

ZooKeeper znode count growing unbounded: the silent heap killer

Mon, 01 Jan 0001 00:00:00 +0000

ZooKeeper znode count growing unbounded: the silent heap killer

The symptom is familiar: a ZooKeeper ensemble that ran cleanly for months suddenly enters a GC death spiral. Heap climbs, full GC pauses stretch from milliseconds to seconds, sessions expire, and the JVM OOMs. The process restarts, the data tree reloads from snapshot, and the cycle repeats. All members OOM at roughly the same time because they carry the same in-memory data tree.