ZooKeeper monitoring checklist: the signals every production ensemble needs

A reference checklist for engineers running production ZooKeeper ensembles. Signals are organized into four maturity levels: survival, operational, mature, and expert. Each level adds visibility for failure modes the previous level cannot see.

The levels are cumulative. Level 2 assumes Level 1 is covered. Skipping to Level 4 without Levels 1 through 3 leaves gaps in the signals that actually page you during incidents: disk stalls, GC cascades, and quorum loss.

Work through the levels in order. If your tooling cannot reliably collect a level’s signals, close that gap before adding more advanced checks.

Collection sources. Signals below come from the mntr four-letter word, the AdminServer HTTP endpoint (default port 8080, ZooKeeper 3.5+), or the Prometheus metrics endpoint (ZooKeeper 3.6+). Since ZooKeeper 3.5.3, four-letter words require whitelisting via 4lw.commands.whitelist in zoo.cfg. At minimum, whitelist mntr, ruok, isro, srvr, and stat. Disk-space and OS-level signals come from the host, not from ZooKeeper.

flowchart TD
    L4["Level 4 - Expert
quorum_ack_latency, digest_mismatches,
unrecoverable_errors, leader_unavailable_time"]
    L3["Level 3 - Mature
fsynctime, updatelatency, readlatency,
jvm_pause_time, watch_count, pending_syncs"]
    L2["Level 2 - Operational
avg_latency, outstanding_requests,
synced_followers, znode_count, connections"]
    L1["Level 1 - Survival
ruok, isro, server_state, uptime,
disk free on dataDir and dataLogDir"]
    L1 -->|add saturation and capacity| L2
    L2 -->|add write-path and GC signals| L3
    L3 -->|add integrity and consensus| L4

Level 1: survival - is ZooKeeper alive and serving?

Minimum to know the process is up, the client port is open, and the node is part of a functioning ensemble. With only Level 1 you catch complete crashes, quorum loss, and full disks. You do not catch latency degradation, GC pressure, replication lag, or capacity creep.

ruok is necessary but not sufficient. “imok” only confirms the process is alive and the client port is open. A node that lost quorum and fell into read-only mode still returns “imok”. Always pair ruok with isro, and confirm zk_server_state across the ensemble: exactly one node should report “leader”, the rest “follower” (plus any configured observers). If isro returns “ro”, writes are impossible and dependent services that need coordination (Kafka leader election, HBase region assignment) are already failing.

Suppress alerts during cold starts. After a restart, ZooKeeper loads the latest snapshot and replays the transaction log. For large data trees this can take minutes, during which ruok may fail and isro may return “ro”. Suppress non-critical alerts when zk_uptime is below 300 seconds.

Level 2: operational - what a competent team monitors

Signals to catch saturation and capacity problems before they become outages. This is the level most teams plateau at.

Leader-only metrics are invisible on followers. zk_followers and zk_synced_followers appear only on the node currently reporting zk_server_state = "leader". Monitoring that queries a single follower or load-balances across nodes will never see them. Query all nodes and filter for the leader, or target the leader explicitly. The same applies to zk_pending_syncs and zk_quorum_ack_latency at higher levels.

maxClientCnxns rejects silently. Default is 60 connections per source IP. In containerized environments where many pods share a host IP, this limit is easily exceeded. New connections are refused with no server-side log entry. The only signal is zk_connection_rejected incrementing. zk_num_alive_connections is a total across all IPs, so it cannot tell you which IP hit the limit.

Level 3: mature - deep operational visibility

Signals that separate write-path stalls from GC stalls, and replication lag from leader overload. This is where most preventable ZooKeeper outages get caught early.

Separate reads from writes. zk_avg_latency aggregates reads (local memory lookups, fast) and writes (quorum round-trip, slow). A write stall can be hidden in the average when read volume dominates. ZooKeeper 3.6+ exposes zk_updatelatency and zk_readlatency with p50, p95, p99, and p999 percentiles. Track both, alert on percentiles, not cumulative averages. The same applies to zk_fsynctime and zk_jvm_pause_time_ms.

fsync latency is the leading write-path signal. Every write blocks until the transaction log is fsync’d to disk. When zk_fsynctime p99 climbs, zk_updatelatency follows. On dedicated SSD, fsync p99 should be under 2ms. If it is not, check whether dataLogDir shares a disk with snapshots, whether cloud burst credits are exhausted, and whether a colocated workload is saturating the device.

GC pauses are the leading election trigger. zk_jvm_pause_time_ms tells you how long the JVM froze. If p99 approaches a meaningful fraction of minSessionTimeout (default 2 x tickTime = 4000ms), sessions start expiring. If it approaches syncLimit x tickTime (default 5 x 2000ms = 10 seconds), followers miss heartbeats and trigger elections. Correlate pause spikes with zk_looking_count and zk_stale_sessions_expired to confirm GC as the cause rather than network or disk.

Level 4: expert - full operational mastery

Signals for data integrity, consensus health, and write availability accounting. These catch silent corruption and failure modes that only surface after an incident.

Data integrity is delta-only. zk_digest_mismatches_count and zk_unrecoverable_error_count are counters. Alert on any increase, never on the absolute value, or you will re-page for historical events. A digest mismatch means the in-memory tree diverged from the expected checksum. Investigate and potentially rebuild the node from a healthy snapshot. If mismatches appear on multiple nodes, escalate immediately.

Track write availability directly. zk_sum_leader_unavailable_time measures cumulative milliseconds during which no leader could serve writes. In steady state this counter should not grow. Any non-zero delta in a recent window means writes were impossible for that duration. Pair it with zk_cnt_leader_unavailable_time to see how many episodes contributed to the total.

Quorum ACK latency isolates consensus cost. zk_quorum_ack_latency (leader-only) measures the time between the leader sending PROPOSE and receiving quorum acknowledgments. If it climbs while zk_fsynctime stays flat, the bottleneck is follower processing or network, not your leader’s disk. It should stay well below syncLimit x tickTime.

Common monitoring gaps that bite production

Mistakes that show up most often in ZooKeeper outage postmortems. If your checklist stops at Level 1 or 2, several of these are probably open right now.

• Using ruok as the only health check. It confirms the process is alive but not that the node can serve requests or is part of quorum. A read-only node returns “imok”.
• Relying on zk_avg_latency alone. Reads and writes are aggregated, so write stalls hide behind high read volume unless you split zk_updatelatency and zk_readlatency.
• Not collecting zk_fsynctime. Disk latency is the root cause of most write stalls. Without it, root cause analysis turns into guesswork.
• Forgetting maxClientCnxns. Default 60 per source IP silently rejects clients under containerized NAT. Watch zk_connection_rejected.
• Not tracking zk_znode_count growth. The data tree lives entirely in heap. Slow growth over months ends in simultaneous OOM across all nodes, because they all hold the same tree.
• Missing leader-only metrics. zk_synced_followers, zk_pending_syncs, and zk_followers only appear on the leader. A follower-only collector never sees them.
• Alerting on cumulative latency extremes. zk_max_latency is a high-water mark since last srst or restart. A spike from days ago stays elevated forever unless you reset stats or use 3.6+ percentiles.
• Skipping data integrity signals. zk_digest_mismatches_count is the only early warning for in-memory data divergence, and almost no one monitors it.
• Not separating dataLogDir from dataDir. A shared disk means snapshot I/O competes with transaction log fsync. This is a one-line config change that prevents intermittent write spikes.
• No autopurge. Without autopurge.purgeInterval and autopurge.snapRetainCount configured, snapshots and transaction logs accumulate until the disk fills and ZooKeeper crashes with no warning.

How Netdata helps

• Per-second mntr collection means every level of this checklist is continuously populated, not spot-checked at scrape intervals that miss transient GC pauses or election spikes.
• Correlating zk_jvm_pause_time_ms with zk_connection_drop_count and zk_stale_sessions_expired separates GC-caused session storms from network-caused ones in a single timeline.
• Correlating zk_fsynctime p99 with zk_updatelatency p99 confirms whether a write stall is disk-bound before it escalates into quorum loss.
• Leader-only metrics (zk_synced_followers, zk_pending_syncs, zk_quorum_ack_latency) are collected per-node and tagged by zk_server_state, so the leader is always identifiable without a separate collector path.
• Host-level disk free and disk latency on the dataDir and dataLogDir mounts sit next to ZooKeeper process metrics in the same dashboard, closing the gap between application signals and the storage they depend on.
• Anomaly detection on slow movers like zk_znode_count, zk_watch_count, and heap usage flags creeping exhaustion weeks before the cliff edge.

Related guides

• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper monitoring maturity model: from survival to expert
• ZooKeeper quorum loss: no leader elected and every write is failing
• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• ZooKeeper leader election storm: an ensemble that keeps re-electing
• ZooKeeper unexpected leader election: finding why the leader dropped
• ZooKeeper split-brain: two nodes both reporting leader
• ZooKeeper server stuck in LOOKING: a node that never rejoins the quorum
• ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls
• ZooKeeper write latency high: read zk_updatelatency, not just avg_latency
• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper quorum ack latency high: followers slow to acknowledge proposals