ZooKeeper znode count growing unbounded: the silent heap killer

The symptom is familiar: a ZooKeeper ensemble that ran cleanly for months suddenly enters a GC death spiral. Heap climbs, full GC pauses stretch from milliseconds to seconds, sessions expire, and the JVM OOMs. The process restarts, the data tree reloads from snapshot, and the cycle repeats. All members OOM at roughly the same time because they carry the same in-memory data tree.

The root cause is rarely a sudden event. It is slow, linear znode accumulation from an application or framework that creates nodes without cleaning them up. The ensemble worked fine at 100k znodes. It worked fine at 500k. It failed at 2M because that is where heap pressure became lethal. The failure was months in the making, and the signal was visible the whole time in zk_znode_count.

This guide covers how to identify which subtree is growing, why deleting millions of nodes in production is dangerous, and how to size the heap against the data tree you actually have.

What this means

ZooKeeper holds the entire data tree in memory on every server. Heap consumption scales with both znode count and the data stored in each znode. There is no on-disk fallback for the working set. Every persistent znode a client ever created is still in heap unless something explicitly deleted it.

Each znode carries overhead well beyond its data payload: the path string, the data bytes, ACL references, the stat structure, and children tracking. A reasonable estimate is roughly 300 bytes of metadata per znode before data, and JVM object overhead (object headers, references, HashMap internals) can push the true heap cost to 2-3x that figure. At 1M znodes you are already looking at hundreds of megabytes of heap just for the tree structure, before any data payload.

The dangerous part is the slope, not the absolute number. An ensemble at 800k znodes growing 5k per day is on a collision course with heap exhaustion. An ensemble at 2M znodes that is stable is fine. The metric that matters is the growth rate of zk_znode_count and zk_approximate_data_size over time.

Because every ensemble member carries the full tree, they all OOM at roughly the same time. A rolling restart does not help for long. The new process loads the same snapshot and climbs back to the same heap pressure.

flowchart TD
    A[Framework creates znodes without cleanup] --> B[zk_znode_count grows linearly]
    B --> C[Heap fills with DataTree and overhead]
    C --> D[GC frequency increases]
    D --> E[Full GC pauses stretch to seconds]
    E --> F[Sessions expire, leader elections trigger]
    E --> G[JVM OOM, process killed]
    G --> H[Restart loads same snapshot]
    H --> C

Common causes

CauseWhat it looks likeFirst thing to check
Framework leak (Spark, Flink, old Kafka, Curator bug)Persistent or sequential znodes accumulate under a known framework path; count grows linearly with workloaddump and ls on the framework’s root path
Ephemeral node accumulationzk_ephemerals_count tracks upward without corresponding connection growth; sessions not expiring cleanlyCompare zk_ephemerals_count against zk_num_alive_connections
Application using ZK as a data storezk_approximate_data_size grows faster than zk_znode_count; individual znodes approaching jute.maxbuffer (default 1MB)Ratio of zk_approximate_data_size to zk_znode_count
No cleanup job at allPersistent znodes under application paths never deleted; count only ever goes upAudit the tree structure with recursive ls
Heap sized for the original deployment, not currentOOMs happen even though growth is slow; heap was never increased after onboarding new clientsCompare heap max against estimated tree footprint

Quick checks

All read-only and safe to run in production.

# Total znode count
echo mntr | nc localhost 2181 | grep zk_znode_count

# Approximate in-memory data size (bytes)
echo mntr | nc localhost 2181 | grep zk_approximate_data_size

# Ephemeral count vs alive connections
echo mntr | nc localhost 2181 | grep -E 'zk_ephemerals_count|zk_num_alive_connections'

# JVM pause time percentiles (ZK 3.6+)
echo mntr | nc localhost 2181 | grep zk_.*jvm_pause

# Current heap usage from the JVM
jcmd $(pgrep -f QuorumPeerMain) GC.heap_info

# List ephemeral nodes with owning session
echo dump | nc localhost 2181 | head -50

# On-disk footprint of snapshots and txn logs
du -sh /var/zookeeper/data/version-2/

If four-letter commands return empty, they are not whitelisted. Since ZK 3.5.3 the 4lw.commands.whitelist property controls which commands are allowed (default is stat, ruok, conf, isro, plus mntr and srvr in many distributions). dump and wchp typically need explicit whitelisting. The AdminServer on port 8080 (ZK 3.6+) is an alternative read-only endpoint.

How to diagnose it

The goal is to find which subtree is growing, not just confirm that the tree is large.

  1. Confirm the growth rate. Plot zk_znode_count over days or weeks. A flat line means the tree is not the problem. A steady upward slope confirms the leak. Pair with zk_approximate_data_size to see whether node count or per-node data size is the driver.

  2. Identify the top-level paths. Connect with the ZooKeeper CLI (zkCli.sh, path varies by distribution) and list the root:

    zkCli.sh -server localhost:2181
# then in the CLI:
ls /

    Most ensembles have a small number of top-level paths (/zookeeper for quota and internal state, plus framework-specific paths).

  3. Count children under each top-level path. The CLI has no built-in recursive count. For a quick breakdown of where ephemeral nodes live, parse dump output to enumerate ephemerals by owning session, then group by path prefix:

    # dump output format: "<session-id> <ephemeral-path>" per line under session headers
echo dump | nc localhost 2181 | grep -oE '/[a-zA-Z0-9_./-]+' | awk -F/ '{print $2}' | sort | uniq -c | sort -rn

    This groups ephemeral nodes by their top-level path and shows where the bulk lives.

  4. Distinguish ephemeral from persistent growth. If zk_ephemerals_count is growing without a matching increase in zk_num_alive_connections, sessions are not being cleaned up properly. This points at either a client library leaking ephemeral nodes within long-lived sessions, or a session expiry problem. If persistent node count is growing, the application is creating persistent znodes without a cleanup path.

  5. Check whether watches are amplifying the problem. Run echo mntr | nc localhost 2181 | grep zk_watch_count. A high watch count on the growing subtree means a bulk delete later will trigger a watch storm. Note this before planning any cleanup.

  6. Estimate time to heap exhaustion. Use the growth rate of zk_znode_count plus the ~300 bytes per znode estimate (plus zk_approximate_data_size) to extrapolate when the tree will consume the remaining heap headroom. This tells you whether you have days or months to act.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
zk_znode_countDirect driver of heap usageSustained linear or exponential growth
zk_approximate_data_sizeCaptures per-node data growth that count alone missesGrowing faster than zk_znode_count means nodes getting larger
zk_ephemerals_countShould track with zk_num_alive_connections; divergence indicates session cleanup problemsRatio to connections trending upward
zk_watch_countA high count on the bloated subtree makes cleanup dangerousUnbounded growth, especially concentrated on few paths
zk_avg_jvm_pause_time_ms / zk_p99_jvm_pause_time_msGC pressure is the symptom of heap exhaustionp99 trending up, or approaching a meaningful fraction of minSessionTimeout (default 2 x tickTime = 4000ms)
JVM heap usage (post-GC trough)The actual capacity signalTrough rising over days means live data set growing
zk_snapshot_error_countLarge trees stress snapshot creation and recoveryAny increment warrants investigation

Fixes

Identify and stop the source of growth

Before deleting anything, identify which framework or application is creating the nodes. Check the application’s documentation or configuration for cleanup settings. Common patterns:

  • Spark / Flink: per-task or per-job znodes that are not cleaned up after job completion. Check the framework’s ZK cleanup configuration.
  • Old Kafka consumer state: pre-KRaft Kafka stored consumer offsets and metadata in ZK. Stale consumer group state can accumulate. Modern Kafka (3.3+) uses KRaft mode and no longer requires ZK.
  • Apache Curator: implements garbage collection for ephemeral and persistent nodes, but bugs in specific versions have caused leaks. Check the Curator version against known issues.

Stopping the source is the only permanent fix. Deleting nodes without fixing the leak just resets the clock.

Clean up the existing nodes carefully

Deleting millions of znodes in production is dangerous. Each successful delete() triggers a data watch and a child watch on the deleted znode, plus a child watch on the parent. If thousands of clients are watching the subtree, the delete produces a watch notification storm that can overwhelm the ensemble the same way a session expiry storm does.

Rules for safe cleanup:

  • Never bulk-delete a large subtree during peak traffic. Schedule cleanup during a maintenance window or low-traffic period.
  • Delete in small batches with throttling. A few hundred deletes per second is a safer ceiling than millions at once. Monitor zk_outstanding_requests and zk_packets_sent while deleting; back off if either spikes.
  • Prefer the application’s own cleanup mechanism. If the framework provides a cleanup API or CLI, use it rather than deleting znodes directly. The application can deregister watches and handle its own consistency.
  • Use deleteall (ZK 3.5+) for recursive deletes. The older rmr command was deprecated in 3.5 and removed in 3.6.
  • Check for watches first. echo wchp | nc localhost 2181 shows watch counts per path. This command is expensive on large trees and typically disabled in the default whitelist, so use it sparingly. If the target subtree has heavy watch pressure, expect a storm and plan accordingly.

Increase heap as a stopgap

If you need time to investigate, increasing the JVM heap buys runway. This does not fix the leak; it delays the OOM.

  • Set ZOO_HEAP_SIZE or -Xmx in zkEnv.sh or the startup script.
  • A rolling restart is required. Each node will reload the full snapshot, so the restart itself takes longer with a larger tree.
  • Keep the heap reasonable. The ZooKeeper admin guide warns against swapping; oversized heaps make full GC pauses longer, which can itself trigger leader elections.
  • G1GC is the default in ZK 3.6+ and handles large heaps better than older collectors. ZGC (JDK 15+) provides sub-millisecond pauses but requires more total memory.

Enable quotas (ZK 3.7+)

ZooKeeper 3.7.0 introduced zookeeper.enforceQuota. When enabled, the server rejects writes that exceed a configured hard quota on total bytes or children count under a znode path, returning QuotaExceededException. The default is false (quotas are advisory only, logged as warnings).

Setting a quota on the offending subtree can prevent a runaway framework from filling the tree, but only if the framework handles QuotaExceededException gracefully. If it does not, writes fail silently from the framework’s perspective. Test before relying on this in production.

Prevention

  • Graph zk_znode_count and zk_approximate_data_size on a long time window. Daily, weekly, and monthly views. Slow growth is invisible on a 1-hour dashboard.
  • Alert on growth rate, not absolute count. A threshold of “1M znodes” is meaningless if your normal operating point is 800k. Alert when the slope is sustained and positive over days.
  • Pair the two metrics. Count growth means new nodes. Data size growth without count growth means existing nodes are getting larger, which is a different problem (application misuse).
  • Track zk_ephemerals_count against zk_num_alive_connections. The ratio should be stable. Divergence indicates sessions are not cleaning up.
  • Size heap against the tree you project, not the tree you have. Re-evaluate heap allocation when onboarding new clients or frameworks.
  • Understand every top-level path in your tree. If you cannot explain what owns each path and what its cleanup strategy is, you have a future OOM.
  • Test cleanup procedures before you need them. Bulk deletes during an incident are much riskier than planned cleanup during a maintenance window.

How Netdata helps

Netdata surfaces the signals that reveal znode-driven heap pressure before it becomes an outage:

  • Per-second zk_znode_count and zk_approximate_data_size collection makes slow linear growth visible on long time windows without sparse polling.
  • Correlating znode count with JVM pause metrics (zk_avg_jvm_pause_time_ms, zk_p99_jvm_pause_time_ms) shows when heap pressure from tree growth is starting to cause GC stalls, the leading indicator of the OOM cycle.
  • Anomaly detection on zk_znode_count growth rate flags when the slope deviates from baseline. Absolute thresholds miss slow leaks.
  • Pairing zk_ephemerals_count with zk_num_alive_connections surfaces session cleanup problems early.
  • Correlating tree growth with zk_outstanding_requests and zk_packets_sent helps anticipate watch storm risk during cleanup operations.