$ guides / zookeeper ▌

APACHE ZOOKEEPER · OPERATIONS PLAYBOOK

ZooKeeper's three pressure points: an fsync on every write, a data tree that lives in the heap, and a quorum a single pause can break

A coordination service that holds its entire znode tree in JVM memory, fsyncs every write to disk before acknowledging it, and depends on a quorum agreeing over ZAB. We trace how that design behaves under load, where a slow disk or a GC pause turns into a cluster-wide outage, and what to do when it does.

> Start with the monitoring checklist → # Jump to the full guide list

ZooKeeper's defaults get you a working ensemble quickly, then hand you a set of cliff-edges that most teams only discover during an incident.

The defaults work. Until the transaction log shares a disk with snapshots, an fsync stalls, and the log warns that it took Nms which will adversely affect operation latency. Until the data tree quietly grows for months, the heap fills, and every node throws java.lang.OutOfMemoryError: Java heap space at the same moment because they all hold the same tree. Until a GC pause trips ZooKeeper's own monitor with Detected pause in JVM or host machine (eg GC), clients miss heartbeats, and sessions expire — deleting the ephemeral nodes that Kafka, HBase, and HDFS depend on. Until a security-group change blocks the election port and the ensemble cannot elect a leader. Until a container fleet behind one IP hits Too many connections from /IP - max is 60 and new clients are refused.

These guides are written for engineers who already run ZooKeeper, not for people learning what a znode is. The goal is the mental model of how the ensemble actually behaves under load, the failure patterns that keep recurring, the monitoring story that catches them before they page anyone, and the runbooks you wish someone had handed you before your last incident.

How ZooKeeper actually runs in production

ZooKeeper is not just a key-value tree. It is a replicated state machine where the leader turns every write into a proposal, a quorum must fsync and acknowledge it before it commits, and the whole tree lives in one JVM heap. Most production failures live between these layers — on the disk, in the GC, or on the wire between members — not inside any single one.

clients + sessions

Each client opens a TCP connection and negotiates a session with a timeout. Sessions are the unit of identity: ephemeral nodes, watches, and ACLs all bind to them. A client that misses its heartbeat window loses the session — and every ephemeral node it created vanishes. <code>maxClientCnxns</code> caps connections per source IP, not in total.

CLIENT

request throttling

Incoming requests enter a submission queue bounded by <code>globalOutstandingLimit</code> (default 1000). When it fills, the server stops reading from client sockets — TCP backpressure — and counts throttled operations. <code>outstanding_requests</code> is the earliest sign the pipeline cannot keep up, filling before latency moves.

THROTTLE

leader / followers / observers

Followers serve reads locally and forward every write to the leader. The leader alone sequences writes. Observers take the commit stream for read scaling but do not vote. Exactly one leader must exist; a node in <code>LOOKING</code> is mid-election and serving nothing.

ROLE

ZAB proposal pipeline

The leader assigns a monotonic <code>zxid</code>, writes the proposal to its own log, and broadcasts it. Once a quorum acknowledges, it commits and applies to the in-memory tree. <code>quorum_ack_latency</code>, <code>proposal_count</code> vs <code>commit_count</code>, and <code>pending_syncs</code> expose the health of this consensus loop.

ZAB

transaction log (WAL) + fsync

Every mutation is appended to the write-ahead log and <code>fsync</code>'d to disk before it is acknowledged — on the leader and on every acknowledging follower. This one disk operation determines write latency and ensemble stability. It is the single most latency-sensitive thing ZooKeeper does, and the cause of more outages than any other.

WAL

in-memory data tree

The entire znode namespace — data, ACLs, children, stat, plus session and watch tables — lives on the JVM heap. Reads are pure memory lookups. Unbounded znode or watch growth is a silent heap leak: it works for months, then trips GC pressure and OOM.

TREE

snapshots + recovery

Periodically the tree is serialized to a fuzzy snapshot; on restart ZooKeeper loads the latest snapshot and replays the log to recover. <code>autopurge</code> must be configured or logs and snapshots fill the disk. A large tree makes both snapshotting and recovery slow, and a corrupt snapshot blocks startup entirely.

SNAPSHOT

JVM + host

One JVM per server. A Stop-the-World GC pause freezes request processing, heartbeats, and quorum ACKs at once — long enough and it expires sessions or triggers an election. Transparent Huge Pages, swap, and NUMA quietly multiply pause times. ZooKeeper's own pause monitor is the direct signal.

JVM

Why this matters: 'ZooKeeper is slow' or 'writes are failing' can come from a saturated transaction-log disk, a GC pause on the leader, a follower falling out of sync, a data tree bloated past the heap, a blocked election port, or an outright quorum loss. The symptom rhymes but each layer has a different signal — and a different fix.

The failures you'll actually see

Most ZooKeeper incidents fall into a small set of recurring patterns. Recognise the shape, and triage gets dramatically faster.

CRITICAL

The write stall

The transaction-log disk cannot fsync fast enough — a shared disk, exhausted cloud IOPS credits, or degrading hardware. Every write blocks on the leader, the proposal pipeline backs up, followers cannot acknowledge in time, and the ensemble can re-elect. If all members share the same storage tier, the new leader inherits the same problem. Reads on followers stay fast, masking the severity.

fsync-ing the write ahead log ... took Nms in the logs
zk_fsynctime p99 elevated (>50ms, often into seconds)
zk_outstanding_requests climbing, zk_throttled_ops incrementing
Write latency up while follower read latency stays normal

Investigate →

CRITICAL

Quorum loss

Enough members drop out or lose contact that fewer than floor(N/2)+1 remain. No leader can be elected, all writes fail, and surviving nodes sit in LOOKING (serving stale reads only if read-only mode is on). A blocked election port is the quiet cause — the ensemble looks fine until it has to elect and cannot.

No node reporting zk_server_state = leader for >60s
Multiple nodes stuck in LOOKING, zk_looking_count climbing
zk_sum_leader_unavailable_time growing; proposals/commits stalled
Cannot open channel to N at election address in the logs

Investigate →

CRITICAL

The GC pause cascade

A Full GC freezes the process — no heartbeats, no request processing, no quorum ACKs. Clients miss heartbeats and sessions expire; if the leader pauses past syncLimit × tickTime the ensemble re-elects. On resume the queued requests drain as a latency spike. With a data tree that has outgrown the heap, each pause is longer than the last: a death spiral.

Detected pause in JVM or host machine (eg GC) in the logs
zk_jvm_pause_time_ms p99 spiking; post-GC heap trough rising
zk_outstanding_requests building then draining rhythmically
Unplanned leader elections timed to the pauses

Investigate →

ACTIVE

The session expiration storm

A network event, load-balancer timeout, or fleet-wide client GC expires many sessions at once. Ephemeral nodes are deleted en masse, watches fire, and every disconnected client reconnects simultaneously — the thundering herd. Dependent systems react hardest: Kafka deregisters brokers, HBase reassigns regions, locks are lost across the fleet.

KeeperErrorCode = Session expired across many clients
V-shape in zk_num_alive_connections: sharp drop then spike
zk_ephemerals_count dropping, zk_stale_sessions_expired jumping
zk_packets_sent surge (watch notifications) with no matching receive

Investigate →

ACTIVE

The silent heap creep to OOM

A framework creates per-task or per-consumer znodes without cleaning them up. The tree grows for weeks, the heap fills, GC pressure rises, and eventually the JVM dies with OutOfMemoryError. Because every member holds the same tree, they OOM at nearly the same moment — a single-cause total outage rather than a rolling one.

zk_znode_count / zk_approximate_data_size growing without plateau
Post-GC heap trough rising over days; Full GCs lengthening
java.lang.OutOfMemoryError: Java heap space in the logs
Snapshot files growing; recovery on restart taking longer

Investigate →

IMMINENT

Clients silently rejected

A group of clients behind one source IP — a container host, a NAT gateway, several JVMs on one box — exceeds maxClientCnxns (default 60 per IP). New connections are refused with no server-side error by default, while the total connection count still looks healthy. Clients see connection refused or timeouts and cannot register.

Too many connections from /IP - max is 60 in the logs
zk_connection_rejected incrementing
zk_num_alive_connections looks normal (the limit is per-IP)
New pods or instances unable to connect after a scale-up

Investigate →

ZooKeeper monitoring maturity levels

ZooKeeper observability works in four practical levels. Each is a complete operation, not a stepping stone. Pick the level that matches how much your ensemble matters. Most production clusters should land at the second level.

Level 1: Survival

Know that something is wrong

Survival monitoring is the floor. With these signals you can answer one question: is the ensemble alive and is there a leader accepting writes? You will not learn what broke, but you will learn that something broke before dependent systems do. Survival is enough for dev ensembles and non-critical coordination.

ruok liveness (imok) Process alive and answering — but not that it can serve requests.
isro read-write state (rw) 'ro' means quorum is lost while the process still looks up.
Server state / a leader exists Exactly one node must report leader across the ensemble.
Disk free on dataDir + dataLogDir A full log partition is an immediate, ungraceful crash.
Process + uptime (QuorumPeerMain) Unexpected uptime resets reveal crashes and OOM kills.

Level 2: Operational

Diagnose most incidents on your own

Operational monitoring is what most production ensembles should target. Survival tells you something is wrong; operational tells you what. With this coverage your team can usually diagnose an incident on its own: latency, backlogs, connection storms, replication degradation, elections.

Request latency (avg + max) The headline health number; watch max for intermittent stalls.
Outstanding requests Fills before latency moves; the best 'keeping up?' proxy.
Alive connections + rejections Mass drops signal disconnects; rejections mean clients refused.
Znode count + data size Data-tree growth is the slow path to heap exhaustion.
Synced followers / pending syncs Leader-only: replication health and fault-tolerance margin.
JVM heap usage + GC pause count The tree lives on-heap; pauses expire sessions and elect leaders.
Leader election events Every unplanned election is an availability event.
File descriptor usage At the limit, new connections and log files both fail.

Level 3: Mature

Catch problems before they become incidents

Mature monitoring catches problems before they wake anyone up. fsync latency creeping, one follower drifting behind, watch counts climbing, session-expiry rate ticking up, snapshots growing. None of these page you on day one. They become page-out incidents on day thirty.

Update vs read latency (p99) Separated write/read tails; the average hides write stalls.
fsync latency on the txnlog disk The root-cause metric for every write stall.
Per-server zxid comparison Real replication lag; zxids should differ by 1-2 at most.
Watch count + distribution A hot path with thousands of watchers is a thundering herd.
Ephemeral count + session-expiry rate Sudden drops are the signature of an expiration storm.
GC pause duration distribution Duration, not just count, decides election and session risk.
Snapshot size + txnlog file count Bloat and broken autopurge lengthen recovery.
OS iowait / disk await / swap Corroborates disk saturation and quiet swapping.

Level 4: Expert

Reactive instrumentation after real incidents

Expert signals enter your stack the day after a specific incident proved you needed them. Quorum ACK latency, data-tree digest verification, SNAP-sync detection, THP and NUMA status, client-side session events. Most teams never need every signal here. Add the ones your incident history says you do.

Quorum ack latency (p99) Leader-only: consensus round-trip across the followers.
Digest mismatch + unrecoverable errors Data-integrity alarms; any increment is a page.
Snapshot / restore error counts Recovery safety — a corrupt snapshot blocks startup.
SNAP-sync events (from logs) A follower needing a full snapshot degrades the leader.
THP + NUMA + swappiness on hosts Silently multiply GC pause times 2-10x.
Client-side session events Reconnects and expirations as the clients experience them.
Ensemble auth + non-mTLS counts Server-to-server auth failures threaten quorum.
Observer sync time Observer lag serves stale reads without threatening quorum.

Operating mistakes worth avoiding

The traps ZooKeeper teams keep falling into. Each has a clear, well-known fix. Most teams only learn it after an incident.

Using ruok as the only health check

This is the most pervasive ZooKeeper monitoring gap. <code>ruok</code> returning <code>imok</code> only means the process is alive — a server in <code>LOOKING</code> state, unable to serve a single request, still answers <code>imok</code>. Nearly every 'monitoring didn't catch it' postmortem involves ruok-only checks. Use <code>isro</code> for read-write state and <code>mntr</code> for real health.

Not monitoring fsync latency on the transaction-log disk

fsync latency is the single most important write-path signal, yet most teams watch average request latency and never track the disk operation that dominates it. They see spikes and blame 'ZooKeeper' when the cause is a shared disk, exhausted cloud IOPS credits, or a degrading SSD. Without <code>zk_fsynctime</code>, root-cause takes hours instead of seconds.

Leaving dataLogDir on the same disk as snapshots

If <code>dataLogDir</code> is unset it defaults to <code>dataDir</code>, so the latency-critical transaction-log fsync competes with bulk snapshot writes. Everything looks fine until write load rises — then fsync latency spikes seemingly without cause, every time a snapshot runs. Putting the log on its own dedicated device is the highest-leverage single-line config change for ZooKeeper.

Trusting avg_latency instead of update latency

<code>zk_avg_latency</code> aggregates fast local reads and slow quorum writes, so a severe write stall vanishes into the average on a read-heavy cluster. It is also cumulative since the last <code>srst</code> reset, so a spike hours ago stays pinned. Monitor <code>zk_updatelatency</code> and <code>zk_readlatency</code> separately (3.6+ percentiles), or compute deltas.

Not monitoring data-tree growth

Heap exhaustion is a silent killer. Teams set the heap once, never watch <code>zk_znode_count</code>, and a framework accumulates per-task nodes for months. Because every member holds the same tree, they all OOM at the same moment. Watch the count and, more importantly, investigate <em>what</em> is growing — deleting millions of nodes later triggers watch storms.

Forgetting maxClientCnxns is per source IP

The default is 60 connections <em>per source IP</em>, not total. In containerised environments where many pods share a host IP, this is exhausted fast, and new connections are refused with no server log by default — only <code>zk_connection_rejected</code> increments. The total connection count looks healthy the whole time, so it cannot detect the problem.

Not alerting on leader elections

A leader election is the ZooKeeper equivalent of a database failover — writes stop for its duration. Many teams do not monitor for elections at all and discover them only when Kafka or HBase logs errors. Every unplanned election should be at least a ticket, with the leader's GC log and fsync latency checked for the trigger.

Leaving autopurge disabled

In many distributions <code>autopurge.purgeInterval</code> defaults to 0 (off). Snapshots and transaction logs then accumulate for months until the disk fills and ZooKeeper crashes with no warning. Set <code>autopurge.purgeInterval</code> and keep <code>autopurge.snapRetainCount</code> at 3 or more — too few forces lagging followers into expensive SNAP syncs.

ZooKeeper runbooks in this section

Each guide is a focused runbook for one symptom or topic. Pick one when you have an incident, or use the categories to learn the area.

▸

Start here

▸

Quorum, elections, and split-brain

▸

Write pipeline: fsync and latency

▸

Throughput, request queue, and throttling

▸

Sessions, connections, and expirations

▸

Replication, sync, and stale reads

▸

JVM heap, GC, and OOM

▸

Data tree, watches, and memory

▸

Transaction logs, snapshots, and disk

▸

Client-facing KeeperException codes

▸

Data integrity and unrecoverable errors

▸

Authentication, ACLs, TLS, and four-letter commands

WHERE TO GO NEXT

Setting up ZooKeeper monitoring, or putting out a fire?

If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, jump straight to the symptom that matches what you're seeing.

> Start with the checklist > Back to Operations Guides