ZooKeeper read latency high: memory reads that should never be slow

Reads in ZooKeeper are local heap lookups. A getData, getChildren, or exists call should return in well under a millisecond because no quorum is involved. The connected server walks its in-memory data tree and replies. When zk_p99_readlatency sits above 50ms for minutes at a time, something on that JVM is competing with request processing.

Do not treat read latency like write latency. Write latency is dominated by transaction-log fsync and quorum ACK. Read latency has none of that machinery, so when it climbs the cause is almost always local: a Stop-the-World GC pause, a pathologically deep or wide znode tree pushing heap pressure, or a watch-delivery backlog stealing CPU from the request thread. Correlating zk_p99_readlatency with zk_jvm_pause_time_ms, zk_znode_count, and zk_watch_count is what separates a 30-second diagnosis from an hour of guessing.

One caveat before you start. Read latency on a follower does not include replication lag. A follower that has fallen slightly behind the leader will still return reads with sub-millisecond latency, but the data may be stale. If the symptom is “reads are wrong” rather than “reads are slow”, this is a different problem and this article will not help.

What this means

A read is a server-local traversal of the in-memory data tree: no leader round-trip, no proposal, no fsync, no quorum ACK. The Apache ZooKeeper ServiceLatencyOverview benchmark measured roughly 0.17ms/op for get on a single-client 3-node ensemble and roughly 0.49ms/op under 20 concurrent clients.

The operator-relevant threshold is direct: sustained zk_p99_readlatency above 50ms is abnormal. Reads should typically complete in under 1ms. Brief spikes during GC events are expected and are not actionable on their own.

Two measurement traps:

• zk_avg_readlatency, zk_min_readlatency, and zk_max_readlatency from mntr are cumulative since server startup (or since the last srst reset). They are not sliding windows. A single 4-second GC pause from two days ago will pin max_readlatency forever. Use the percentile metrics added in ZK 3.6 (zk_p50_readlatency, zk_p95_readlatency, zk_p99_readlatency, zk_p999_readlatency) or compute deltas externally.
• If you aggregate read and write latency into a single alert, write stalls can be hidden by read volume. The headline zk_avg_latency lies for exactly this reason. See ZooKeeper avg_latency hides write stalls.

flowchart TD
    A["zk_p99_readlatency >50ms sustained"] --> B["Overlay zk_jvm_pause_time_ms"]
    B -->|"Spikes coincide"| C["GC pause driven"]
    B -->|"No correlation"| D["Check zk_znode_count, zk_approximate_data_size"]
    D -->|"Growing"| E["DataTree bloat"]
    D -->|"Stable"| F["Check zk_watch_count"]
    F -->|"Growing or spiking on change"| G["Watch backlog or watch storm"]
    F -->|"Stable"| H["Check zk_outstanding_requests, zk_throttled_ops"]
    H -->|"Non-zero sustained"| I["Pipeline saturation"]
    H -->|"Zero"| J["Check isro, snapshot timing"]

Common causes

Quick checks

On ZK 3.5.3+, four-letter commands must be whitelisted via 4lw.commands.whitelist. If mntr returns nothing, that is the config issue, not a ZooKeeper outage.

# Confirm the node is serving reads and not in LOOKING or RO mode
echo isro | nc localhost 2181                       # expect "rw"
echo mntr | nc localhost 2181 | grep zk_server_state

# Read latency percentiles (ZK 3.6+) - the actual signal
echo mntr | nc localhost 2181 | grep zk_.*readlatency

# GC pause time - the most common cause
echo mntr | nc localhost 2181 | grep zk_.*jvm_pause

# Data tree size and watch pressure
echo mntr | nc localhost 2181 | grep -E 'zk_(znode_count|approximate_data_size|watch_count|ephemerals_count)'

# Pipeline saturation - non-zero means requests are queuing
echo mntr | nc localhost 2181 | grep -E 'zk_(outstanding_requests|throttled_ops)'

# Process GC stats in real time
jstat -gcutil $(pgrep -f QuorumPeerMain) 1000 5

Host-level:

# Swap usage - if ZK is swapping, GC pauses multiply by 10x to 100x
grep VmSwap /proc/$(pgrep -f QuorumPeerMain)/status

# Transparent Huge Pages - inflates GC pauses 2x to 10x when enabled
cat /sys/kernel/mm/transparent_hugepage/enabled

How to diagnose it

1. Confirm you are looking at the right metric. zk_avg_readlatency is cumulative and is contaminated by every read since startup. Use zk_p99_readlatency if you have ZK 3.6+, or reset stats with echo srst | nc localhost 2181 after capturing a baseline and watch the new values.

2. Overlay read latency with zk_p99_jvm_pause_time_ms. If the read-latency spikes line up with GC pause spikes, root cause is done. The fix path is heap or GC algorithm, not ZooKeeper tuning.

3. Check heap utilization. jcmd <pid> GC.heap_info or jstat -gcutil. Sustained old-gen usage above roughly 85% is the cliff where full GC frequency skyrockets. Cross-check against zk_znode_count and zk_approximate_data_size to see whether the heap pressure is data-driven.

4. If GC is not the cause, look at the request pipeline. zk_outstanding_requests sustained above zero means the request processing thread is falling behind. On a heavily read-oriented node this is the signature of CPU starvation, usually from watch delivery.

5. Inspect watch load. echo wchs | nc localhost 2181 returns total watches, connections with watches, and paths being watched. High watch count plus a zk_packets_sent spike without a matching zk_packets_received spike is the signature of a watch storm: one heavily-watched znode changed and the server is fanning out notifications to every watcher.

6. Rule out read-only mode. A follower that lost quorum but has readonlymode.enabled=true will still answer reads quickly, but clients depending on writes or session state will be misbehaving. echo isro | nc localhost 2181 returns rw or ro.

Metrics and signals to monitor

Fixes

GC pauses from heap pressure

The most common cause. Two paths, depending on whether the data tree is appropriately sized.

If zk_znode_count and zk_approximate_data_size are reasonable for your workload, the heap is undersized. Increase ZOO_HEAP_SIZE (or -Xmx in zkEnv.sh) and rolling-restart. The official guidance is deliberately conservative: “if you have 4G of RAM, do not set the Java max heap size to 6G or even 4G… it is more likely you would use a 3G heap for a 4G machine.” Do not size heap to total RAM. Leave headroom for the OS page cache, off-heap Netty/NIO buffers, and JVM overhead. Swapping multiplies GC pause times by 10x to 100x.

If the data tree is genuinely too large for the heap, increasing heap only delays the problem. Identify the bloated subtree and delete the leaked znodes (next section). A 2GB heap with a 2GB live data set will full-GC itself to death.

If heap is reasonably sized and full GCs are still long, the GC algorithm is the problem. On JDK 9+, the JVM default is G1GC. On JDK 15+, ZGC dramatically reduces pause times and is a good fit for coordination servers with sub-millisecond latency targets. CMS was deprecated in JDK 9 and removed in JDK 14; if your startup scripts still reference -XX:+UseConcMarkSweepGC, you have a deprecated collector and an upgrade path.

Also check Transparent Huge Pages. THP on Linux can inflate GC pauses 2x to 10x. /sys/kernel/mm/transparent_hugepage/enabled should not report [always] for a ZooKeeper host.

DataTree bloat

Identify the offending subtree. echo dump | nc localhost 2181 shows ephemeral node info; sorting by path will reveal runaway subtrees. Common culprits are old Kafka consumer offset trees, HBase splitWAL or unassigned regions, Solr collection state, and service-discovery trees that never clean up.

Deleting znodes in production is not free. Every deletion fires watches, which fans out notifications. If the subtree has watchers, a cleanup operation can cause a watch storm that worsens the latency problem you are trying to fix. Coordinate with the application owner, clean up in small batches, and watch zk_packets_sent during the operation.

Prevent recurrence at the application layer. Container nodes (3.5+) and TTL nodes (3.5+) auto-clean. If a framework is the source (older Kafka consumers, Curator path caches, Spark or Flink ephemeral registrations), fix the framework or schedule periodic cleanup.

Watch backlog and watch storms

A high sustained zk_watch_count is normal for service-discovery workloads. The problem cases are unbounded growth, or a single path with extreme watcher count.

wchc (per-connection) and wchp (per-path) are disabled by default in 3.6+ because they are O(n) and can themselves cause latency spikes on servers with many watches. If you enable them, use sparingly and never on a hot server.

maxInProcessingDeadWatchers (system property zookeeper.maxInProcessingDeadWatchers, added in 3.6.0) controls the WatcherCleaner backlog. When dead watchers from expired sessions queue past this limit, ZK slows down adding new dead watchers, which applies backpressure to watch cleanup. Raising this limit can help if cleanup is falling behind, but it also raises heap usage.

For the underlying watch-storm pattern (a heavily-watched znode changing frequently), the fix is almost always application-side: fewer watchers, sharded state, or move the hot-changing data out of ZooKeeper entirely.

Request-thread starvation from throttling

If zk_throttled_ops is incrementing and zk_outstanding_requests is sitting near globalOutstandingLimit, the server is applying TCP backpressure. The causes are upstream: write pipeline stall on the leader, GC pauses, or genuinely too much traffic for the cluster.

Do not raise globalOutstandingLimit to fix latency. That deepens the queue, lengthens worst-case latency, and increases the risk of requests going stale and being dropped. Find the upstream cause first. Note also the known race between submitRequest() and incrOutstandingRequests() (ZOOKEEPER-3072), which is reproducible with globalOutstandingLimit=1. Do not tune it aggressively downward either.

On ZK 3.6+, the RequestThrottler is available but requestThrottleLimit (system property zookeeper.request_throttle_max_requests) defaults to 0, meaning disabled. If something has enabled it, that is the place to look.

Snapshot interference

Snapshot serialization is CPU-intensive and briefly increases latency on the snapshotting node. Snapshots trigger every snapCount transactions (default 100,000). Brief periodic spikes that line up with snapshot file creation in dataDir are expected and not actionable.

If the spikes are severe enough to impact p99 read latency, ensure the snapshot disk is not the same physical device as the transaction-log disk. Snapshot I/O contention with fsync is a write-latency problem as well. On 3.9.4+, the lock contention between snapshotting and sync (ZOOKEEPER-4858) is fixed, and the ResponseCache regression that could degrade read performance (ZOOKEEPER-4919) is also fixed.

Read-only mode

If isro returns ro, the node has lost quorum and is serving stale reads. Read latency may look fine. The fix is to restore quorum, not to tune the read path. See ZooKeeper quorum loss: no leader elected and every write is failing.

Prevention

• Monitor zk_p99_readlatency directly, not zk_avg_readlatency. The cumulative averages are noise. The percentile metrics exist since ZK 3.6. Use them.
• Track zk_znode_count and zk_approximate_data_size as capacity signals. Plot the post-GC heap trough alongside. Linear growth without plateau is a leak.
• Track zk_watch_count and the watches-per-connection ratio. Unbounded growth is a watch leak. Spike-on-change is a watch storm waiting to happen.
• Enable GC logging. Without it, you cannot diagnose the most common cause of read latency. On JDK 9+: -Xlog:gc*:file=/var/log/zookeeper/gc.log:time,uptime,level,tags:filecount=5,filesize=100m.
• Size heap conservatively. Leave roughly 30% free. Do not swap. Disable Transparent Huge Pages.
• Run 3.8.x or 3.9.x. The 3.7.x branch is EOL since February 2024. The 3.9.x line includes multiple heap and serialization improvements (ZOOKEEPER-4717, 4718, 4714, 4289) plus fixes for read-relevant bugs (ZOOKEEPER-4919, ZOOKEEPER-4858 in 3.9.4).

How Netdata helps

• Per-second collection of zk_p99_readlatency and the rest of the read-latency percentiles. The 50ms threshold becomes an alert instead of an average you reason about after the fact.
• Side-by-side overlay of zk_p99_jvm_pause_time_ms and zk_p99_readlatency makes the GC-versus-pipeline diagnosis visual. Rhythmic latency spikes matching GC frequency are obvious in a chart.
• zk_znode_count, zk_approximate_data_size, and zk_watch_count are collected as capacity trends, not just point-in-time values. Slow growth that takes months to become an incident becomes visible weeks ahead.
• zk_outstanding_requests and zk_throttled_ops are surfaced as leading indicators with anomaly detection on the queue-depth baseline, so you see pipeline saturation before it becomes a latency cliff.

Related guides

• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper write latency high: read zk_updatelatency, not just avg_latency
• ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls
• ZooKeeper monitoring checklist: the signals every production ensemble needs
• ZooKeeper monitoring maturity model: from survival to expert
• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper quorum loss: no leader elected and every write is failing
• ZooKeeper leader election storm: an ensemble that keeps re-electing