ZooKeeper data size growing: using ZooKeeper as a database is an anti-pattern

ZooKeeper is a distributed coordination service, not a datastore. It holds the entire znode tree in JVM heap, replicates every mutation through ZAB, and periodically serializes the full tree to a snapshot on disk. The design point is small, hot, strongly consistent metadata: leader election handles, service discovery registrations, distributed locks, configuration pointers. When teams treat it as a general-purpose key-value store and let zk_approximate_data_size climb, they inherit the operational characteristics of an in-memory database without the tooling, schema, or compaction strategies of one.

Growth is almost always silent. zk_approximate_data_size is a single counter in mntr output, reported in bytes, and nothing in ZooKeeper alarms when it doubles or triples. The ensemble keeps serving reads and writes at low latency while the tree bloats underneath. The pain surfaces weeks or months later as larger snapshots, slower restarts, and escalating GC pressure. By the time the tree is large enough to cause incidents, cleanup itself is risky: deleting znodes fires watches and can trigger thundering-herd notifications.

What it is and why it matters

zk_approximate_data_size is exposed by the mntr four-letter command and the equivalent AdminServer HTTP endpoint . It is the approximate total size, in bytes, of the data stored in all znodes in the in-memory data tree. It does not include per-znode metadata overhead (path strings, stat structures, ACL references, children lists), session state, watch tables, or request queues. It is a floor on heap consumption, not a ceiling.

Two limits frame the conversation.

Per-znode data limit. jute.maxbuffer, default 0xfffff (1,048,575 bytes, approximately 1 MB). This is a hard cap on the serialized size of a single znode’s data. A single write that exceeds it is rejected and, on 3.6+ , counted in zk_large_requests_rejected. The property is a Java system property with no zookeeper. prefix. It must be set identically on every server and every client; a mismatch surfaces as Len error or Packet len is out of range! on the side with the smaller limit.

Total tree budget. The operational rule of thumb is that the in-memory footprint of the tree, computed as approximate_data_size + znode_count * ~300 bytes of per-znode overhead, should stay below 50% of JVM heap. Above that, GC time spent in collection increases and full GC pauses grow longer.

The 50% rule exists because JVM object overhead is real. The DataTree maintains two parallel in-memory structures (a hashtable from path to DataNode, plus the tree of DataNodes itself), and Java object headers, references, and HashMap internals typically multiply the raw data size by 2-3x. A zk_approximate_data_size of 400 MB on a 2 GB heap is not “20% used”; it is close to the heap ceiling once object overhead, sessions, watches, and request buffers are added.

How it works

Every byte in zk_approximate_data_size lives on the heap for the entire lifetime of the znode. There is no eviction, no compaction, no off-heap storage. Persistent znodes accumulate forever unless an application explicitly deletes them; ephemeral znodes die with their session but can still leak if sessions are not expiring properly.

Snapshots serialize the entire DataTree. Snapshot frequency is controlled by snapCount (default 100,000 transactions), with per-server randomization to avoid synchronized snapshot storms across the ensemble. Snapshot serialization time grows roughly linearly with tree size, and during serialization the snapshot thread holds a read lock on the tree that can contend with concurrent writes under heavy load. A 1 GB tree takes meaningfully longer to snapshot than a 100 MB tree, and every ensemble member pays that cost independently.

Recovery is worse. On restart, ZooKeeper loads the latest snapshot and then replays the transaction log forward from the snapshot’s zxid. Large snapshots mean long deserialization. A node that takes minutes to recover looks dead to health checks and to followers trying to sync.

flowchart TD
    A[Data tree grows
approximate_data_size climbs] --> B[Heap consumption rises
2-3x raw bytes]
    A --> C[Snapshot size grows
linear with tree]
    B --> D[GC pressure builds
post-GC trough climbs]
    C --> E[Snapshot serialization
holds read lock]
    C --> F[Recovery time grows
on cold start]
    D --> G[Stop-the-World pauses
threaten sessions]
    E --> H[Write latency spikes
during snapshot]
    F --> I[Cold start takes
minutes not seconds]
    G --> J[Session expirations
and leader elections]

This cascade is why data-size growth is not a single symptom but a leading indicator for several distinct incidents: GC death spirals, snapshot-induced write stalls, slow failovers, and SNAP sync blast radius when a follower restarts and needs the full tree.

Where it shows up in production

The anti-pattern enters production through several well-worn paths. Recognizing them helps you find the offending subtree when zk_approximate_data_size is climbing.

  • Configuration blobs and JSON documents stored as znode data. SolrCloud stores collection state as JSON in znodes. Custom platforms store feature flag documents, routing tables, or schema definitions. None of these are coordination data; they are application data that happens to need distribution and grows over time.
  • Per-task or per-session znodes with payloads. Frameworks that create a znode per job, per consumer, or per task and store state in the node’s data field. If cleanup is incomplete, the tree grows monotonically. Old Kafka consumer group offset storage (before the __consumer_offsets topic) is the canonical example.
  • Many znodes near the jute.maxbuffer limit. A tree with 10,000 znodes averaging 900 KB each is roughly 9 GB of zk_approximate_data_size. The per-znode limit does not protect you from aggregate growth; it only caps individual writes.
  • Service discovery payloads that grew over time. A service registry that started by storing host and port now stores full metadata, health details, and tags. The change looks small per node but compounds across thousands of registrations.
  • Application state “temporarily” parked in ZooKeeper. An application needs strongly consistent distributed state, reaches for ZooKeeper because it is already in the dependency graph, and starts writing real data volume to it.

Common misuses

The unifying thread is treating ZooKeeper as a strongly consistent, in-memory, replicated key-value store. It is one, mechanically, but that is not the design point. The design point is small coordination data: bytes to low kilobytes per znode, total tree in the low hundreds of megabytes at most. ZooKeeper has no query language, no secondary indexes, no compaction, no eviction, and no TTL by default (TTL nodes exist in 3.5+ but must be explicitly created with a TTL). Every byte is replicated to every ensemble member on every write and serialized into every snapshot.

If you need to store blobs, use object storage and store a pointer in ZooKeeper. If you need a strongly consistent KV store with larger payloads, use a database. If you need append-only event logs, use a log. ZooKeeper’s job in each of those architectures is to coordinate access to the system that actually holds the data, not to hold the data itself.

When you find a bloated subtree, deleting it in production is not free. Deleting a heavily watched znode fires every watch on it in a single notification burst. For large cleanups, delete in small batches during a maintenance window and monitor zk_packets_sent and zk_outstanding_requests as you go. A burst of watch notifications can spike write latency and, in extreme cases, cause session expirations on clients whose watches fire.

Signals to watch in production

SignalWhy it mattersWarning sign
zk_approximate_data_sizeDirect measure of in-memory tree data volumeSustained upward trend not correlated with known workload growth
zk_znode_countTree node count; multiplies per-znode overheadLinear or exponential growth without plateau
zk_large_requests_rejectedCounter of writes exceeding jute.maxbufferAny non-zero increment; an application is misusing individual znodes
JVM heap usage (post-GC trough)True live data set including object overheadTrough rising over weeks, especially if approximate_data_size is also rising
zk_jvm_pause_time_ms (p99)GC pause impact from heap pressurep99 trending up; rhythmic spikes matching GC frequency
Snapshot file sizeDirect measure of serialized tree sizeNewest snapshot significantly larger than previous ones
zk_snapshot_error_countSnapshot failures; recovery safety at riskAny increment
Restart-to-serving timeProxy for recovery costIncreasing over time as tree grows

The most important correlation is zk_approximate_data_size against JVM heap post-GC trough. If both are climbing together, the tree is driving heap pressure. If heap is climbing but approximate_data_size is flat, look at zk_watch_count, zk_znode_count, and session table growth instead, because something other than payload data is bloating the heap.

The second correlation worth running is zk_approximate_data_size against snapshot size and zk_jvm_pause_time_ms. If snapshot size tracks data size (it should) and GC p99 tracks snapshot size, you are watching the full cascade from data growth to GC stall in slow motion.

How Netdata helps

Netdata’s per-second collection of mntr makes growth visible long before it becomes an incident.

  • Per-second zk_approximate_data_size trend with anomaly detection on the derivative surfaces slow leaks that would otherwise take months to notice. A monotonic climb is the signal that a subtree is accumulating without cleanup.
  • Correlation with JVM heap and GC pause metrics on the same timeline. When all three move together, the diagnosis is “data tree is driving GC pressure”; when they diverge, look elsewhere.
  • Correlation with zk_znode_count and zk_ephemerals_count lets you distinguish “many small znodes” from “few large znodes” without running expensive wchc or dump commands against a loaded ensemble.
  • Snapshot size and zk_snapshot_error_count tracking alongside zk_approximate_data_size ties snapshot stalls to tree growth directly, rather than letting the snapshot path look like an independent disk problem.
  • Per-ensemble-leader views ensure you are reading the leader’s mntr for leader-only signals like zk_synced_followers and zk_pending_syncs, which matter most when a large tree is making follower resyncs expensive.