ZooKeeper monitoring maturity model: from survival to expert

Most ZooKeeper monitoring stops too early. Teams run ruok as their only health check, collect zk_avg_latency without understanding it is cumulative since the last reset, and never look at fsync latency until a write stall cascades into a Kafka outage. The ensemble looks healthy in dashboards until it suddenly does not, and the postmortem reveals the signals were there all along, uncollected.

This article maps ZooKeeper monitoring across four maturity levels: Survival, Operational, Mature, and Expert. Each level answers a harder question than the one before. Survival asks whether the process is alive. Operational asks whether it is serving clients well. Mature asks why it is degrading. Expert asks whether the data is correct and whether the next failure will be silent.

Use this as a gap assessment. If your monitoring stops at Level 1, you will catch crashes but miss the GC death spiral that caused them. Level 3 covers most production incidents. Level 4 is where you catch data corruption, silent read staleness, and the OS-level pathologies that turn one slow disk into an ensemble-wide outage.

flowchart TD
    L1["Level 1: Survival
ruok, isro, server_state, disk"] --> L2["Level 2: Operational
latency, connections, followers, FDs"]
    L2 --> L3["Level 3: Mature
fsync p99, GC pauses, watches, zxid"]
    L3 --> L4["Level 4: Expert
quorum_ack, digest mismatches, THP/NUMA"]

How to use this model

Read each level as a checklist. If you are missing signals from your current level, fill those gaps before adding signals from the next. Jumping straight to Expert metrics without the Operational foundation means you will have rich data on quorum ACK latency but no idea whether the cluster is even accepting connections.

Most metrics listed come from the mntr four-letter command or its equivalents: the AdminServer HTTP endpoints on port 8080 (ZooKeeper 3.5+), or the Prometheus MetricsProvider on port 7000 (3.6+). Since ZooKeeper 3.5.3, four-letter commands must be whitelisted via 4lw.commands.whitelist in zoo.cfg. If mntr is not whitelisted, monitoring systems receive empty responses and interpret them as healthy, which is worse than no monitoring at all.

Level 1: Survival

Question: Is ZooKeeper alive?

At this level, you can detect process crashes, quorum loss, and full disks. You cannot detect latency degradation, GC issues, replication lag, or capacity exhaustion.

SignalWhat it tells you
ruok returns imokThe process is alive and accepting commands on the client port
isro returns rwThe node has quorum and can accept writes
zk_server_state from mntrExactly one node reports leader, the rest report follower
zk_uptimeDetects unexpected restarts; gates cold-start false positives
Disk space on dataDir and dataLogDirA full transaction log partition is immediately fatal

What you catch: process death, OOM kill, port unreachable, disk full, quorum loss via isro returning ro.

What you miss: everything else. A node returning imok can be in LOOKING state with no leader. isro returning rw does not mean latency is acceptable. This is the most common monitoring state in the industry, and it is the source of most “monitoring did not catch it” postmortems.

Common mistake: using ruok as the sole health check. A server in read-only mode, unable to serve any writes, still returns imok. Always pair ruok with isro, and verify zk_server_state rather than trusting either four-letter command alone.

Level 2: Operational

Question: Is ZooKeeper serving clients well?

This is what a competent production team monitors. You add request pipeline visibility, connection health, replication state, and resource pressure.

SignalWhat it tells you
zk_avg_latency, zk_max_latencyRequest processing latency (cumulative since last reset)
zk_outstanding_requestsRequest pipeline backlog; sustained non-zero means saturation
zk_num_alive_connectionsClient connectivity; sharp drops indicate mass disconnection
zk_connection_rejectedClients hitting maxClientCnxns per-IP limit (default 60), silently refused
zk_connection_drop_countConnection instability rate
zk_open_file_descriptor_countFD pressure; approaching the OS limit blocks new connections
zk_znode_countData tree size; unbounded growth leads to heap exhaustion
zk_followers, zk_synced_followers (leader only)Replication health and quorum margin
JVM heap usage and GC pause frequencyMemory pressure and pause impact

What you catch: most operational degradation. Sustained high latency, pipeline saturation, connection storms, FD exhaustion, data tree bloat, and replication degradation.

What you miss: the root cause of latency spikes. zk_avg_latency aggregates reads and writes, so a write stall can hide behind high read volume. Without separated read/write latency and fsync timing, you know something is slow but not why. Cumulative latency metrics (pre-3.6) never reset unless you call srst, so dashboards showing max_latency of 15000ms may reflect a days-old GC pause.

Common mistakes at this level:

  • Cumulative latency confusion. zk_avg_latency and zk_max_latency are cumulative since the last reset or restart, not sliding windows. Without delta computation or 3.6+ histogram metrics, latency monitoring is misleading.
  • Leader-only metric blindness. zk_followers, zk_synced_followers, and zk_pending_syncs only appear on the leader. Monitoring that queries a follower will never see replication health.
  • Silent connection rejection. In containerized environments where many pods share a host IP, the default maxClientCnxns of 60 is easily exceeded. Rejections are silent from the server side. The only evidence is zk_connection_rejected incrementing.

Level 3: Mature

Question: Why is ZooKeeper degrading?

This is full operational visibility for a senior SRE. You separate read and write latency, track the disk write path directly, monitor GC impact on sessions, and watch for watch storms and session expiry.

SignalWhat it tells you
zk_updatelatency p99, zk_readlatency p99 (3.6+)Separated write and read tail latency; isolates write pipeline from local memory lookups
zk_fsynctime p99 (3.6+)Transaction log disk write latency; the root cause metric for write stalls
zk_jvm_pause_time_ms p99 (3.6+)GC pause duration; correlates with session expirations and elections
zk_throttled_opsOperations throttled at globalOutstandingLimit (default 1000); any non-zero rate means saturation
zk_watch_countActive watch registrations; unbounded growth is watch-storm risk
zk_approximate_data_sizeIn-memory data footprint; combined with znode count, estimates heap consumption
zk_looking_countElection frequency; any increment outside maintenance indicates instability
zk_follower_sync_timeHow long followers take to sync; approaching syncLimit * tickTime means ejection risk
zk_pending_syncs (leader only)Replication lag; sustained non-zero means followers cannot keep up
zk_stale_sessions_expiredSession expiration rate; any non-zero increment outside maintenance is abnormal
Per-server zk_zxid comparisonReplication lag detection across ensemble members
OS-level: iowait, disk await, swap usagePhysical root causes behind application-level symptoms

What you catch: the specific subsystem causing degradation. Disk (fsynctime), JVM (pause time), network (quorum ACK), or data structure (znode count, watch count). You can answer this in minutes, not hours.

What you miss: data integrity violations, silent read staleness on followers, and the deepest quorum consensus signals. You also miss the OS-level tuning issues (THP, NUMA) that amplify GC pauses.

Key correlations that define this level:

  • zk_fsynctime p99 spikes alongside zk_updatelatency p99: disk is the write bottleneck.
  • zk_jvm_pause_time_ms p99 spikes alongside zk_stale_sessions_expired and zk_looking_count: GC is driving instability.
  • zk_outstanding_requests growing with zk_throttled_ops incrementing: pipeline is saturated and applying backpressure.
  • Per-server zxid delta growing between leader and a follower: that follower is falling behind and may require a full SNAP sync.

Level 4: Expert

Question: Is the data correct, and will the next failure be silent?

This is full operational mastery. You add quorum consensus internals, data integrity verification, recovery safety, security signals, and the OS-level pathologies that amplify JVM pauses.

SignalWhat it tells you
zk_quorum_ack_latency p99 (leader only)Time for followers to ACK proposals; isolates network from disk causes
zk_proposal_count, zk_commit_countWrite throughput pipeline; divergence means quorum problems
zk_digest_mismatches_countData tree checksum mismatch; any increment means corruption
zk_unrecoverable_error_countCritical internal errors; any increment means integrity is compromised
zk_snapshot_error_countSnapshot creation or loading errors; recovery safety at risk
zk_sync_processor_queue_sizeWrite pipeline depth; growing means write throughput exceeds processing
zk_sum_leader_unavailable_timeCumulative write unavailability; any non-zero delta means writes were impossible
zk_ensemble_auth_failServer-to-server auth failures; threaten quorum
zk_non_mtls_remote_conn_countRemote connections without mutual TLS; encryption compliance
zk_observer_sync_timeObserver lag; stale reads on observers
SNAP-sync detection (leader logs)Followers requiring full snapshot transfer; expensive for the leader
THP status on ZooKeeper hostsTransparent Huge Pages can multiply GC pause duration significantly
NUMA topology for JVM placementCross-node GC memory operations add pause time
Client-side session eventsReconnection loops and expiry events as experienced by clients

What you catch: data corruption before clients notice (zk_digest_mismatches_count). Write unavailability measured directly (zk_sum_leader_unavailable_time). Security violations (zk_ensemble_auth_fail, zk_non_mtls_remote_conn_count). OS-level amplifiers of JVM pauses (THP, NUMA). The gap between server-side health and client-side experience.

What this requires beyond metrics scraping: ZooKeeper does not expose all of these signals natively. SNAP-sync detection requires parsing leader logs for “Sending snapshot.” THP status requires reading /sys/kernel/mm/transparent_hugepage/enabled on each host. Client-side session events require application-level instrumentation in client libraries, since ZooKeeper has no client-side metrics endpoint.

Why this level matters: zk_digest_mismatches_count is the only early warning for data corruption. A single mismatch means a node is serving incorrect data to clients. Without this signal, you discover corruption when applications fail in ways that make no sense, and by then the corrupted state may have propagated.

Where most teams get stuck

The gap between Level 2 and Level 3 is where most teams stall. The reasons are structural:

  • zk_avg_latency looks sufficient until it is not. Reads are fast (local memory lookups) and writes are slow (quorum consensus plus fsync), so a high read volume masks write stalls in the aggregate. Teams see a healthy average and miss writes timing out.
  • fsync latency is not a prominent metric. Pre-3.6, fsync timing comes from ZooKeeper logs or OS-level iostat. Many teams never correlate the two. On 3.6+, zk_fsynctime is available with percentiles but is still rarely collected.
  • Percentile data requires 3.6+. Teams on 3.4.x or 3.5.x collect avg, min, and max and miss tail latency. A p99 fsync of 50ms with an average of 2ms tells a different operational story.
  • Leader-only metrics are missed by follower-only monitoring. zk_followers, zk_synced_followers, and zk_pending_syncs only appear on the leader. Monitoring that queries a random node or only followers will never see replication health. Identify the leader and query it, or query all nodes and filter for leader-reported values.
  • JVM GC is treated as a black box. zk_jvm_pause_time_ms is available in the Prometheus MetricsProvider (3.6+) but rarely collected. GC pauses are a leading cause of session expirations and unnecessary leader elections, yet most teams monitor heap usage without monitoring pause duration.

How Netdata helps

Correlation is where ZooKeeper monitoring earns its keep. A single metric in isolation rarely identifies the problem. Per-second collection across all four maturity levels connects signals that a 60-second scrape interval would miss.

  • Liveness and function on one timeline. zk_server_state, isro status, and zk_outstanding_requests together distinguish a dead process from a live one that has lost quorum.
  • Write path root cause isolation. zk_fsynctime p99 next to zk_updatelatency p99 separates disk-caused stalls from quorum-ACK delays, the two most common write degradation causes.
  • GC and election correlation. zk_jvm_pause_time_ms p99 overlaid with zk_looking_count and zk_stale_sessions_expired reveals whether GC is the root cause of ensemble instability.
  • Replication margin in real time. zk_synced_followers, zk_pending_syncs, and per-server zxid comparison show how close the ensemble is to quorum loss before it happens.
  • OS context on the same host. Disk await, iowait CPU, and swap usage alongside ZooKeeper metrics connect application symptoms to their physical cause without switching tools.