ZooKeeper OutOfMemoryError: Java heap space - the OOM that kills the whole ensemble at once

You grep the ZooKeeper log and find java.lang.OutOfMemoryError: Java heap space. The process is gone. A minute later another node dies with the same error, then the third. The whole ensemble went down inside a single window, not as a rolling failure. That simultaneity is the signature, not a cascade.

ZooKeeper holds the entire data tree on the JVM heap: every znode, its data, ACL references, children lists, stat structures, plus session state, watch tables, and request queues. Every ensemble member holds the same tree. Whatever fills the heap on one node fills it on all of them at roughly the same rate, so when the tree finally exceeds the heap they OOM near-simultaneously. This is a single-cause total outage.

The death usually arrives in two stages. First, the heap gets tight enough that the GC runs continuously and reclaims almost nothing, because most live objects are long-lived data tree nodes. Stop-the-world pauses stretch from milliseconds into seconds, sessions start expiring, and JVM pause p99 climbs. Then the JVM either throws OutOfMemoryError directly or is OOMKilled by the kernel because resident set exceeded the cgroup limit. By the time the error string appears in the log, the runaway growth has been happening for days or weeks. The fix is not to restart; it is to find what is filling the heap and size the heap to actually hold it.

What this means

java.lang.OutOfMemoryError: Java heap space means the JVM could not allocate an object because the Java heap was exhausted and a GC cycle could not free enough space to proceed. For ZooKeeper this is almost never a transient allocation spike. It is the terminal event of sustained on-heap growth from the data tree, the watch table, the session table, or all three. Once heap utilization is consistently above roughly 90%, GC frequency skyrockets and produces stop-the-world pauses that cascade into session expirations and leader elections long before the actual OOM lands.

Because every ensemble member replicates the same data tree, the heap pressure curve is nearly identical across nodes. A workload that pushes one node into OOM pushes all of them into OOM. That is why the outage looks coordinated even though there is no communication between the failures. The diagram shows the loop you are trying to break.

flowchart TD
    A[Data tree grows on every node] --> B[Heap approaches max]
    B --> C[GC frequency skyrockets]
    C --> D[Stop-the-world pauses grow]
    D --> E[Sessions expire, heartbeats miss]
    C --> F[GC reclaims almost nothing]
    F --> C
    D --> G[JVM throws OutOfMemoryError]
    G --> H[Process dies or OOMKilled]
    H --> I[Same data tree on every node]
    I --> J[Near-simultaneous OOM on all members]

Two practical implications follow. First, do not treat a single OOM as a single-node incident; assume the rest of the ensemble is hours or minutes behind. Second, the leading indicator is not peak heap usage, it is the post-GC heap trough. If the trough is climbing over days, the live set is growing and the OOM is predetermined unless the tree stops growing or the heap grows.

Common causes

Quick checks

These are read-only. Run them on the node before it dies, or on a surviving peer if the node is gone.

# Data tree footprint
echo mntr | nc localhost 2181 | grep -E "zk_znode_count|zk_approximate_data_size|zk_ephemerals_count|zk_watch_count"

# JVM heap state right now
jstat -gcutil $(pgrep -f QuorumPeerMain) 1000 5
jcmd $(pgrep -f QuorumPeerMain) GC.heap_info

# GC pressure and queued work
echo mntr | nc localhost 2181 | grep -E "zk_(avg|p99)_jvm_pause_time_ms|zk_outstanding_requests" # TODO: verify exact mntr names for JVM pause metrics on target ZK version

# Uptime, to confirm an OOMKill restart
echo mntr | nc localhost 2181 | grep zk_uptime

# Find the error in the log
grep -E "OutOfMemoryError|java.lang.OutOfMemory" /var/log/zookeeper/zookeeper.log | tail -20

# Confirm what heap and GC the JVM is actually running with
ps -o args= -p $(pgrep -f QuorumPeerMain) | tr ' ' '\n' | grep -E "Xmx|Xms|UseGC|MaxRAMPercentage"

# Confirm container memory limit if running in Kubernetes or under systemd
cat /sys/fs/cgroup/memory.max 2>/dev/null || cat /sys/fs/cgroup/memory/memory.limit_in_bytes 2>/dev/null

The last two commands matter because the OOM is often not “ZooKeeper used too much memory” but “the JVM heap was set larger than the container allows” or “the heap was never sized for the data tree in the first place.”

How to diagnose it

1. Confirm the failure mode. The log line is definitive for Java heap space. Distinguish it from Metaspace (class metadata leak, different fix) and from kernel OOMKill (check dmesg for Killed process and the cgroup memory limit). A kernel OOMKill of the JVM is functionally the same incident but shows up as exit code 137, not as a Java stack trace.

2. Pull the data tree size from a surviving peer. Every node holds the same tree, so any peer gives you the answer.

   echo mntr | nc localhost 2181 | grep -E "zk_znode_count|zk_approximate_data_size|zk_watch_count|zk_ephemerals_count"
   

   Multiply zk_znode_count by roughly 300 bytes of per-znode overhead, add zk_approximate_data_size, then double it for JVM object overhead. If that number is close to your -Xmx, the tree alone is the problem.

3. Check the heap trough trend, not the peak. Pull JVM heap usage at per-second granularity over the last week and look at the post-GC minimum. A flat peak with a rising trough means the live set is growing. A sawtooth with a stable trough means the heap is fine and the OOM came from something else, usually a transient allocation spike or a watch storm.

4. Find the subtree that is bloating. Use dump to list sessions with their ephemeral nodes, or walk the tree from the root with the ZooKeeper CLI. Common culprits are framework paths like consumer offsets, leader-election locks, task nodes, or anything that creates sequential or per-task persistent nodes without cleanup.

   # Ephemeral node summary (read-only, safe)
   echo dump | nc localhost 2181 | head -50
   

   Be careful with wchc and wchp; they iterate every watch and can stall a loaded server. They are also disabled by default unless listed in 4lw.commands.whitelist.

5. Check whether the heap was ever actually sized for this tree. Compare the running -Xmx from ps against the tree footprint from step 2. Default heaps are frequently too small for production. Some distributions ship 256MB or 512MB defaults; the upstream zkEnv.sh defaults to a larger value. Either way, if you never set ZK_SERVER_HEAP, ZOO_HEAP_SIZE, or JVMFLAGS explicitly, assume the heap is wrong.

6. If running in a container, compare the JVM heap to the cgroup limit. The JVM respects container limits via UseContainerSupport, but an explicit -Xmx from zkEnv.sh overrides that. If Xmx is larger than memory.max, the JVM will be OOMKilled the moment it tries to grow into that space, often during GC thread creation.

7. Correlate with JVM pause time. If JVM pause metrics were climbing for hours or days before the OOM, you are looking at the GC death spiral, not a sudden event. The OOM is just the end of it.

Metrics and signals to monitor

The single most useful leading indicator is the post-GC heap trough trend. Peaks are noisy; troughs are the live set. Plot the trough over 30 days and extrapolate linearly to roughly 80% of max heap. That is your runway.

Fixes

Right-size the heap

This is the first action when the data tree is legitimately large and there is no leak to clean. Set the heap explicitly so distro or container defaults cannot surprise you.

# In zkEnv.sh or your startup wrapper
export ZK_SERVER_HEAP=4096   # MB; for Bitnami images use ZOO_HEAP_SIZE
# Or via JVMFLAGS:
export JVMFLAGS="-Xms4g -Xmx4g $JVMFLAGS"

A rolling restart is required; there is no way to resize the heap of a running JVM. Plan the restart outside peak write load. Use -Xms equal to -Xmx to avoid heap resizing pauses, and consider -XX:+AlwaysPreTouch so the OS commits the pages at startup rather than during steady state. Leave 25 to 30 percent of the container or host memory free for off-heap buffers, JVM internals, and the OS, otherwise the kernel OOM killer will come for the JVM even with a correctly sized heap.

For container deployments, set the pod memory limit and the heap together. If the limit is 4Gi, set ZK_SERVER_HEAP=3072 or ZOO_HEAP_SIZE=3072, not 4096. The Bitnami chart’s heapSize parameter does not always propagate; set ZOO_HEAP_SIZE directly as a workaround.

Switch the garbage collector

For heaps above 2 to 3 GB, G1GC is the baseline and ZGC (JDK 15+) is preferable for low pause times. CMS and Parallel produce long full GC pauses on large heaps and make the death spiral worse. G1GC has been the ZooKeeper default since 3.6, but only if your startup script does not override it.

export JVMFLAGS="-XX:+UseG1GC -XX:MaxGCPauseMillis=100 $JVMFLAGS"
# Or, on JDK 15+:
export JVMFLAGS="-XX:+UseZGC $JVMFLAGS"

Enable GC logging so the next incident has evidence.

# JDK 9+
export JVMFLAGS="$JVMFLAGS -Xlog:gc*:file=/var/log/zookeeper/gc.log:time,uptime,level,tags:filecount=5,filesize=100m"

Clean up the data tree

If a subtree is leaking, the only durable fix is to stop the leak and clean up. Identify the offending application path, fix the client (proper ephemeral node lifecycle, TTL nodes, container nodes, or explicit cleanup), then delete accumulated nodes.

Deleting nodes in production is risky. Each delete can fire watches and trigger downstream reactions in Kafka, HBase, Solr, or anything else watching that path. Delete in small batches during a maintenance window, and watch zk_watch_count, zk_outstanding_requests, and zk_packets_sent as you go. Prefer TTL nodes (createTtl) or container nodes (createContainer) for new paths so the tree self-cleans without application cooperation.

Fix container memory mismatch

If the JVM crash log shows memory_limit_in_bytes lower than the configured -Xmx, the fix is either to raise the pod memory limit or lower the heap. Raising the limit is usually correct if the data tree justifies it; lowering the heap just defers the OOM. Check both the request and the limit; a low limit with a high request will still OOMKill.

Prevention

• Plot zk_znode_count and zk_approximate_data_size as long-horizon trends. This is the single best early warning. A leak that takes a year to OOM shows up as a straight line within a week.
• Plot the post-GC heap trough, not the peak. The trough is the live set. A rising trough with a flat peak means the tree is growing.
• Set the heap explicitly on every node from day one. Never rely on distro or upstream defaults for production. Document the heap alongside the ensemble topology.
• Set the heap and the container memory limit together. They are one decision, not two. Review both whenever the data tree grows significantly.
• Use G1GC or ZGC. Parallel and CMS make heap pressure incidents worse than they need to be.
• Enable GC logging permanently. You will need it for the next incident and the disk cost is trivial.
• Enable autopurge. autopurge.purgeInterval and autopurge.snapRetainCount keep disk usage bounded, which prevents a related class of crash that masquerades as heap pressure during recovery.
• Audit znode creation patterns in client applications. The leak almost always originates in a framework creating per-task or per-session persistent nodes without cleanup.
• Set jute.maxbuffer deliberately. The default 1MB cap per znode is correct for coordination data; raising it invites applications to store payloads that do not belong in ZooKeeper.
• Test failover and recovery with the production-sized tree. A SNAP sync after a restart of a large tree is when heap pressure becomes most visible, because the leader serializes the entire tree under load.

How Netdata helps

• The Netdata ZooKeeper collector pulls the full mntr family every second, so zk_znode_count, zk_approximate_data_size, zk_watch_count, and zk_ephemerals_count are available as high-resolution trends rather than point-in-time snapshots. The trough of the heap usage curve is what predicts this incident, and per-second resolution is what makes the trough visible.
• JVM pause metrics sit beside heap usage so you can see the GC death spiral forming before the OOM lands.
• ML anomaly detection flags monotonic drift in zk_znode_count and zk_watch_count weeks before the heap is exhausted, even when no static threshold has been crossed.
• Correlating JVM pause spikes with session expirations, connection drops, and leader elections on a single timeline shows whether the pauses are bad enough to be expiring sessions and triggering elections.
• zk_uptime resets are surfaced as anomalous events, which catches silent OOMKill restarts that would otherwise look like normal rolling maintenance.

Related guides

• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• ZooKeeper “Client session timed out, have not heard from server”: the heartbeat miss
• ZooKeeper connection drops spiking: sessions dying in bursts
• ZooKeeper follower doing a SNAP sync: full snapshot transfer and its blast radius
• ZooKeeper follower sync time climbing: a follower approaching ejection
• ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls
• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper leader election storm: an ensemble that keeps re-electing
• ZooKeeper monitoring checklist: the signals every production ensemble needs
• ZooKeeper monitoring maturity model: from survival to expert
• ZooKeeper outstanding requests growing: the request pipeline is backing up