ZooKeeper autopurge not configured: snapshots and logs filling the disk over months

autopurge.purgeInterval defaults to 0, meaning snapshots and transaction logs accumulate forever. On a quiet ensemble the growth is slow enough that nobody notices for months, then the dataLogDir partition hits 100%, ZooKeeper cannot fsync the next write, and the process dies. The leader throws an IOException on the transaction log and the ensemble loses a member, or quorum if more than one node fills simultaneously.

This article covers how to confirm autopurge is the cause, clean up safely without forcing followers into expensive SNAP syncs, and set autopurge.purgeInterval and autopurge.snapRetainCount so it does not recur.

What this means

ZooKeeper persists state in two on-disk artifacts:

  • Transaction logs (log.<zxid>) in dataLogDir (or dataDir if dataLogDir is not set). Every mutation is appended here and fsync’d before acknowledgment.
  • Snapshots (snapshot.<zxid>) in dataDir. The full in-memory data tree serialized to disk roughly every snapCount transactions (default 100,000).

Without autopurge, neither artifact is ever deleted. Snapshots pile up proportional to write throughput, and transaction logs accumulate between snapshots. Transaction log files are pre-allocated in 64MB chunks (zookeeper.preAllocSize, default 65536KB), so even modest write traffic accumulates full 64MB files at every rotation.

Two config keys control cleanup:

  • autopurge.purgeInterval - interval in hours between purge runs. 0 (the default) disables it.
  • autopurge.snapRetainCount - minimum snapshots (and associated logs) to retain. Default 3, which is also the minimum supported value.

The diagnostic signature: snapshot file count far exceeds snapRetainCount, log.* file count far exceeds what one retention window would need, and disk usage trends monotonically upward.

flowchart TD
  A[autopurge.purgeInterval = 0] --> B[Snapshots never deleted]
  A --> C[Txn logs never deleted]
  B --> D[Disk usage climbs over months]
  C --> D
  D --> E[Partition hits 100%]
  E --> F[fsync fails on next write]
  F --> G[IOException, ZK process dies]
  G --> H[Ensemble loses a member, or quorum]

Common causes

CauseWhat it looks likeFirst thing to check
autopurge.purgeInterval left at 0Slow monotonic disk growth across all members; no purge log linesgrep autopurge zoo.cfg returns nothing or only snapRetainCount
Config added but not appliedzoo.cfg has the keys, disk still growsps -ef | grep QuorumPeerMain start time predates the config change
snapRetainCount raised or purgeInterval very highDisk growth slows but does not stopCompare snapshot count to configured retain value
Solr embedded ZooKeeperConfig looks correct but disk still fillsConfirm whether Solr uses embedded ZK (SolrZKServer), which ignores autopurge
Windows deploymentLogs show “Purge task started” and “Purge task completed” but files remainZOOKEEPER-2844 documents autopurge not deleting on Windows

Quick checks

All safe read-only operations.

# Confirm whether autopurge is configured at all
grep -E 'autopurge\.(purgeInterval|snapRetainCount)' /path/to/zoo.cfg

# Disk usage on the two partitions that matter
df -h /var/zookeeper/txnlog /var/zookeeper/data

# Snapshot file count and total size (dataDir/version-2)
ls /var/zookeeper/data/version-2/snapshot.* 2>/dev/null | wc -l
du -sh /var/zookeeper/data/version-2

# Transaction log count and total size (dataLogDir/version-2)
ls /var/zookeeper/txnlog/version-2/log.* 2>/dev/null | wc -l
du -sh /var/zookeeper/txnlog/version-2

# Newest and oldest snapshot, to see the time span being retained
ls -lt /var/zookeeper/data/version-2/snapshot.* | head -3
ls -lt /var/zookeeper/data/version-2/snapshot.* | tail -3

# Verify the running process start time (autopurge is read at startup)
ps -o lstart= -p $(pgrep -f QuorumPeerMain)

# Confirm ZK is still healthy (4lw must be whitelisted in 3.5.3+)
echo isro | nc localhost 2181
echo mntr | nc localhost 2181 | grep -E 'zk_server_state|zk_znode_count|zk_approximate_data_size'

# Check ZK logs for purge activity or disk write failures
grep -E 'Purge task|IOException|No space left on device' /var/log/zookeeper/zookeeper.log | tail -30

The single most telling signal is the relationship between snapshot count and autopurge.snapRetainCount. A healthy node with autopurge running has roughly snapRetainCount snapshots plus a small number of transaction logs. A node with autopurge disabled has dozens or hundreds of snapshots and a transaction log count that grows without bound.

How to diagnose it

  1. Confirm the symptom is accumulation, not data tree growth. If zk_approximate_data_size is also climbing fast, the root cause may be application misuse of ZooKeeper as a database. See the related guide on data size growth. If zk_approximate_data_size is stable but disk usage climbs, this is a retention problem.
  2. Check whether dataLogDir and dataDir share a partition. If they do, snapshot growth compounds transaction log growth, and you also have fsync contention waiting to surface. Separate them when you fix this.
  3. Verify the config is present and non-zero. Empty output from the grep above is the most common finding. A purgeInterval of 0 is equivalent to absent.
  4. Verify the process is running with the config loaded. autopurge.* keys are read at startup. If you added them but never restarted, they are not in effect.
  5. Check for Windows or Solr embedded-ZK edge cases if the config looks correct but accumulation continues.
  6. Rule out external writers. A colocated backup job, log shipper, or monitoring agent writing to the same partition produces the same disk symptom. du -sh on sibling directories confirms.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
df free space on dataDir and dataLogDirThe failure condition is partition full, not a ZK metricFree space trending down over weeks with no workload change
Snapshot file count vs snapRetainCountDirect measure of whether autopurge is workingCount greater than snapRetainCount + 2 sustained
Transaction log file countShould be bounded by recent snapshotsCount growing monotonically
zk_approximate_data_sizeDistinguishes retention problem from data tree bloatGrowing fast indicates a different root cause
iostat await on the txnlog deviceShared disk plus accumulation produces fsync contentionawait rising as the partition fills
ZK log lines containing IOException or No space left on deviceThe crash signature when the partition fillsAny occurrence

Fixes

Enable autopurge

Add or uncomment both keys in zoo.cfg:

autopurge.purgeInterval=1
autopurge.snapRetainCount=3

purgeInterval is in hours, accepts only positive integers. 1 is reasonable for ensembles with steady write traffic. snapRetainCount of 3 is the default and minimum supported value. Do not set it below 3.

Restart ZooKeeper on each node, one at a time, confirming quorum between restarts. Autopurge parameters are read at startup; the running process will not pick them up from a config reload.

Why snapRetainCount must be at least 3

A follower that was offline briefly (maintenance, crash, GC pause) catches up by replaying transaction logs from its last snapshot forward. If the logs it needs have been purged because snapRetainCount is too low, the follower cannot catch up incrementally. The leader must send the entire data tree as a SNAP sync: the leader serializes and transmits the full tree, the follower is unavailable for reads during transfer, and write latency on the leader can degrade while the snapshot is sent.

snapRetainCount=3 keeps enough history that brief follower outages resolve with incremental log replay instead of a full snapshot transfer. See the related guide on SNAP sync blast radius.

Clean up accumulated files safely

If the disk is already critically full, do not wait for autopurge. Use the bundled PurgeTxnLog utility (also wrapped by bin/zkCleanup.sh in some distributions) to do a controlled purge respecting the same retention rules:

# DESTRUCTIVE: deletes old snapshots and transaction logs.
# Stop the ZK process on this node first, or confirm autopurge is disabled,
# to avoid races with the in-process purge task.
java -cp zookeeper.jar:lib/* org.apache.zookeeper.server.PurgeTxnLog \
  /var/zookeeper/data /var/zookeeper/txnlog -n 3

The -n argument is the retain count. Use the same value as autopurge.snapRetainCount. Run it one node at a time, rolling, so the ensemble never loses redundancy.

Avoid manual rm. PurgeTxnLog understands which logs are referenced by which snapshots and will not orphan a snapshot from the transaction log entries it needs for consistent recovery.

Address configuration management

Check all ensembles. Any deployment not explicitly configured is vulnerable. Common gaps:

  • Helm charts that do not expose autopurge in values.yaml
  • Container images that set ZOO_AUTOPURGE_PURGEINTERVAL=0 explicitly
  • Ansible/Chef/Puppet roles copied from a template that never included the keys
  • Embedded ZooKeeper inside Solr or another product, where the parent product’s docs do not surface the ZK-level config

Edge cases that survive correct configuration

  • Windows. ZOOKEEPER-2844 documents autopurge logging success without deleting files on Windows Server. File handle locking is the suspected cause. On Windows, schedule an external PurgeTxnLog run.
  • Solr embedded ZooKeeper. Solr’s SolrZKServer does not honor autopurge.snapRetainCount or autopurge.purgeInterval. Move to an external ensemble, or run PurgeTxnLog externally on a schedule.

Prevention

  • Treat autopurge config as required. Every production ensemble should have both keys explicitly set. A missing key is a deployment defect, not a neutral default.
  • Alert on snapshot count, not just disk percentage. Disk percentage alerts get snoozed because they climb slowly. A snapshot count above snapRetainCount + 5 for more than one purge interval is a low-noise signal that purge is broken.
  • Alert on dataLogDir free space in absolute terms. The failure mode is pre-allocation failure when the partition is nearly full. Alert at 20% free, not 10%, because the 64MB pre-allocation can fail before df shows zero.
  • Separate dataLogDir and dataDir on different volumes. This improves fsync latency and prevents snapshot accumulation from triggering transaction log write failures.
  • Track zk_approximate_data_size independently. If disk growth is from data tree bloat rather than missing purge, fixing autopurge will slow but not stop the problem.
  • Include autopurge verification in deployment checklists. After every install or manifest change, confirm the keys are present, non-zero, and the process has been restarted since they were added.

How Netdata helps

  • Per-second disk usage metrics on the dataDir and dataLogDir partitions surface the slow monotonic climb that percentage-based alerts miss. Long-term retention makes the “growing over months” pattern visible at a glance.
  • Correlation with zk_approximate_data_size, zk_znode_count, and disk I/O latency lets you confirm in one view whether disk growth is a retention problem (autopurge) or a workload problem (data tree bloat).
  • Alerts on partition free space fire before the partition is full, giving you the window needed to enable autopurge and restart safely.