$ guides / apache-pulsar / apache-pulsar-entry-log-gc-lagging ▌

Operations Guides

Apache Pulsar entry log GC falling behind: reclaimed space that never comes back

A bookie’s ledger disk is filling. Retention policies have deleted ledgers, TTL has expired messages, and cursors have advanced past the data. By every logical measure, the space should be free. But disk usage keeps climbing, and the bookie is heading toward read-only.

BookKeeper does not store each ledger in a separate file. Entry logs are large append-only files that interleave entries from many ledgers, written sequentially as they arrive. When a ledger is deleted, its entries remain physically embedded in the entry log alongside data from other ledgers that are still active. The only way to reclaim that space is compaction: the garbage collector reads the live entries from an old entry log, writes them into a new file, then deletes the old one. If compaction stalls, throttles to a crawl, or never triggers, dead data accumulates indefinitely.

The diagnostic signal is the ratio between two gauges. When bookie_ENTRY_LOG_SPACE_BYTES (total bytes across all entry log files) is much larger than bookie_ACTIVE_ENTRY_LOG_SPACE_BYTES (bytes in entry logs that still contain live ledger data), most of the disk holds entries from ledgers that have already been deleted. A growing total with stable active space means GC is losing ground. This accelerates toward the disk usage threshold (default 95%), where the bookie goes read-only and suspends compaction entirely, creating a feedback loop it cannot escape without intervention.

How entry log compaction works

BookKeeper runs two compaction passes that scan entry logs and rewrite those with sufficient dead data:

Minor compaction runs frequently (default interval: 3600 seconds) with a low threshold (default: 0.2, meaning an entry log is eligible only if less than 20% of its data belongs to live ledgers). This picks off entry logs that are almost entirely dead.
Major compaction runs less frequently (default interval: 86400 seconds) with a higher threshold (default: 0.5 in Pulsar documentation; 0.8 in some BookKeeper versions ). This catches moderately dead entry logs.

When compaction cannot keep up, the gap between total and active entry log space widens. As disk usage climbs toward the configured threshold, the bookie transitions to read-only and suspends both minor and major compaction. At that point, the bookie needs compaction to reclaim space, but it cannot run compaction because the disk is full. This is the death spiral:

flowchart TD
    A["Ledgers deleted by retention or TTL"] --> B["Dead entries remain in entry logs"]
    B --> C["Compaction should rewrite and reclaim"]
    C -->|"falling behind"| D["ENTRY_LOG_SPACE_BYTES >> ACTIVE"]
    D --> E["Disk usage grows toward threshold"]
    E --> F["diskUsageThreshold hit: 95%"]
    F --> G["Bookie goes read-only"]
    G --> H["Compaction suspended"]
    H -->|"cannot reclaim space"| I["Stuck read-only"]

Compaction also drives substantial sequential read I/O. The garbage collector scans entry log headers to identify live entries, which triggers OS PageCache prefetch and can pollute the read cache. Operators often see elevated bookie read latency during compaction windows and mistake it for a consumer read storm. Correlating compaction activity with read latency separates the two.

Common causes

Cause	What it looks like	First thing to check
Compaction thresholds too low or intervals disabled	Entry log ratio grows but `bookie_gc_*` counters stay flat; no compaction cycles in logs	`minorCompactionThreshold`, `majorCompactionThreshold`, and interval values in `bookkeeper.conf`
Compaction throttled by entry count, not bytes	GC runs but throughput is too low for the workload; small-entry topics accumulate dead data faster than compaction rewrites	`isThrottleByBytes` setting; default is `false` with `compactionRateByEntries=1000`
Disk-full death spiral	`bookie_SERVER_STATUS` at 0, disk usage above 95%, compaction suspended	`bookie_ledger_dir_*_usage` and `bookie_SERVER_STATUS`
Corrupt entry log file stalling GC thread	GC counters stop incrementing on one bookie; logs show repeated errors on a specific entry log file	BookKeeper version; PR #4544 in 4.17.3 addresses this . Check bookie logs for entry log read errors
Orphan ledgers or inactive topics blocking deletion	Retention appears to be working but ledgers are never marked for deletion in BookKeeper	Managed ledger ledger lists vs. BookKeeper ledger metadata in ZooKeeper

Quick checks

All commands are read-only unless noted. Run them on the affected bookie. If the bookie is already read-only, jump to step 4 of the diagnosis section.

# GC lag ratio: total entry log space vs active
curl -s http://localhost:8000/metrics | grep -E "ENTRY_LOG_SPACE_BYTES|ACTIVE_ENTRY_LOG"

# Bookie server status (1 = writable, 0 = read-only, -1 = unregistered)
curl -s http://localhost:8000/metrics | grep bookie_SERVER_STATUS

# Disk usage per ledger directory
curl -s http://localhost:8000/metrics | grep bookie_ledger_dir

# Writable directories count
curl -s http://localhost:8000/metrics | grep bookie_ledger_writable_dirs

# GC counters for compaction activity
curl -s http://localhost:8000/metrics | grep -i "bookie_gc\|compact\|reclaim"

# Compaction configuration
grep -E "minorCompaction|majorCompaction|isThrottleByBytes|compactionRate|isForceGCAllowWhenNoSpace|forceAllowCompaction" /conf/bookkeeper.conf

# Filesystem-level check of ledger directory
df -h /path/to/ledgers

# Count entry log files on disk
ls -1 /path/to/ledgers/current/*.log 2>/dev/null | wc -l

How to diagnose it

Confirm the GC lag ratio. Pull bookie_ENTRY_LOG_SPACE_BYTES and bookie_ACTIVE_ENTRY_LOG_SPACE_BYTES. If the ratio exceeds 2x with disk usage above 70%, compaction is falling behind. The ENTRY_LOG_SPACE_BYTES metric was added in BookKeeper 4.17.x . On older versions, only ACTIVE_ENTRY_LOG_SPACE_BYTES is available; infer the gap from filesystem usage minus active space.
Check whether compaction is actually running. Look at bookie_gc_* counters over a scrape interval. If reclaimed-space counters are not increasing, compaction is either not triggering or is stuck. Check bookie logs for GC thread activity.
Verify compaction thresholds and intervals. If minorCompactionInterval or majorCompactionInterval is set to 0, that compaction pass is disabled entirely. If thresholds are too low (near 0), few entry logs qualify for compaction and dead data accumulates. Defaults are 0.2 for minor and 0.5 for major.
Check if the bookie is read-only. When disk usage exceeds diskUsageThreshold (default 0.95), the bookie transitions to read-only and suspends compaction. If bookie_SERVER_STATUS is 0, you are in the death spiral. The bookie cannot compact to reclaim space because it is too full to write the new entry logs.
Check for corrupt entry logs. If GC counters stop incrementing on a single bookie while other bookies are fine, a corrupt entry log file may be hanging the garbage collector thread. This was a known bug addressed in BookKeeper 4.17.3 . On older versions, check bookie logs for repeated I/O errors on a specific entry log file.
Investigate orphan ledgers. If a managed ledger deletes a ledger from its list but the BookKeeper metadata deletion fails (for example, broker restart during the two-step delete), the ledger becomes an orphan. Its data remains in entry logs forever because the GC never sees it as deleted. A ledger that exists in ZooKeeper metadata but is not referenced by any managed ledger’s ledger list is an orphan.
Assess compaction-driven read I/O. During compaction windows, bookie read latency (bookkeeper_server_READ_ENTRY_REQUEST) may spike from sequential entry log scanning. If this correlates with GC activity rather than consumer catch-up reads, it is expected behavior, not a consumer problem.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`bookie_ENTRY_LOG_SPACE_BYTES`	Total bytes across all entry log files, including dead data	Growing while `ACTIVE_ENTRY_LOG_SPACE_BYTES` stays flat
`bookie_ACTIVE_ENTRY_LOG_SPACE_BYTES`	Bytes in entry logs that still contain live ledger data	Stable or declining while total grows
`bookie_ledger_dir_*_usage`	Per-directory disk usage percentage	Above 85% requires urgent action; above 95% triggers read-only
`bookie_SERVER_STATUS`	Writable (1) vs read-only (0) vs unregistered (-1)	Transition to 0 means compaction is suspended
`bookie_ledger_writable_dirs`	Count of writable ledger directories	Dropping to 0 means no writes accepted
`bookie_gc_*` counters	Compaction and reclamation activity	Counters not advancing means GC is stuck or disabled
`bookkeeper_server_READ_ENTRY_REQUEST`	Bookie read latency	Spikes during compaction windows; distinguish from consumer reads

Fixes

Compaction thresholds misconfigured or intervals disabled

Set minorCompactionInterval and majorCompactionInterval to nonzero values (defaults are 3600 and 86400 seconds). Set thresholds to reasonable values: 0.2 for minor, 0.5 for major. If intervals are 0, compaction never runs and disk usage grows monotonically. Restart the bookie after changing these settings.

Compaction throttling bottleneck

The default throttle mode (isThrottleByBytes=false) limits compaction to 1000 entries per second. For workloads with small entries, this results in low byte throughput and compaction cannot keep up with deletion. Switch to byte-based throttling by setting isThrottleByBytes=true and adjusting compactionRateByBytes to match your storage throughput. This allows compaction to use more I/O when needed.

Disk-full death spiral

When the bookie is read-only because disk usage exceeded 95%, compaction is suspended and cannot reclaim space. The recovery path depends on your Pulsar version:

On Pulsar 3.2.0 and later, forceAllowCompaction defaults to true . You can trigger a forced GC via the REST API: curl -XPUT http://localhost:8000/api/v1/bookie/gc. This bypasses the suspendMajor/suspendMinor flags to allow compaction even when the disk is full.
On older versions, forceAllowCompaction defaults to false. Set it to true in bookkeeper.conf and restart, or temporarily increase diskUsageThreshold to give the bookie room to write compacted entry logs. Increasing the threshold above 100% risks actual disk exhaustion; only use this as a short window to let compaction run.
Alternatively, add a new ledger volume or expand the existing one to create headroom. The bookie needs free space to write the new (smaller) entry logs before it can delete the old ones.

The isForceGCAllowWhenNoSpace flag (default false) controls a related but different behavior. When set to true, compaction is not suspended when disk is full, but new entry log files may be created before old ones are deleted, which can exhaust disk faster. Use this with caution.

Corrupt entry log file

If the GC thread is hung on a corrupt entry log, identify the problematic file from bookie logs. On BookKeeper 4.17.3 and later, this should not occur . On older versions, you may need to manually remove the corrupt entry log file after confirming no live ledgers reference it.

This is destructive. Verify with bookkeeper shell commands that no live ledger has entries in the target file before deleting anything. Deleting a live entry log causes data loss for any ledger with entries in that file.

Orphan ledgers or retention policy gaps

If ledgers are not being deleted from BookKeeper, check whether the broker is actually marking them for deletion. Inactive topics that are unloaded from brokers (no producers or consumers) may not have their retention policies evaluated, so ledgers are never marked for deletion. Ensure topics with retention or TTL policies are loaded on a broker. For orphan ledgers, a scan comparing managed ledger ledger lists against BookKeeper metadata in ZooKeeper can identify candidates for manual cleanup.

Prevention

Monitor the entry log space ratio. Alert when bookie_ENTRY_LOG_SPACE_BYTES exceeds 2x bookie_ACTIVE_ENTRY_LOG_SPACE_BYTES sustained with disk usage above 70%. This catches GC lag before it becomes a disk-full emergency.
Verify compaction configuration after deployments. A misconfigured bookkeeper.conf with disabled compaction intervals or restrictive thresholds can silently accumulate dead data for weeks before disk fills.
Use byte-based compaction throttling for small-entry workloads. The default entry-count throttle (1000 entries/second) is insufficient when entries are small and many ledgers are being deleted concurrently.
Keep disk usage below 80%. BookKeeper needs headroom to compact and rewrite entry logs. Running at 90%+ leaves no room for the new files compaction must write before deleting old ones.
Upgrade BookKeeper to 4.17.3 or later to pick up the corrupt-entry-log GC fix (PR #4544 ) and the scheduleWithFixedDelay scheduling fix from 4.17.1 (PR #4296 ) that prevents overlapping compaction runs.
Track GC counter progression. If reclaimed-space counters stop advancing on any bookie, investigate immediately. A stalled GC thread is the earliest sign of a corrupt entry log or a configuration regression.

How Netdata helps

Per-second collection of bookie_ENTRY_LOG_SPACE_BYTES and bookie_ACTIVE_ENTRY_LOG_SPACE_BYTES lets you watch the GC lag ratio develop in real time, rather than discovering it when the disk is already at 90%. Correlating this ratio with bookie_ledger_dir_*_usage and bookie_SERVER_STATUS on the same dashboard makes the death spiral visible as a single cascade: disk crosses threshold, status flips to read-only, GC counters flatline.
Per-second bookie_gc_* counter collection distinguishes a configuration problem (compaction disabled or never qualifying) from a corrupt-entry-log problem (compaction was running and then stopped) within a single dashboard view.
ML anomaly detection on read latency (bookkeeper_server_READ_ENTRY_REQUEST) helps separate compaction-driven read I/O spikes from genuine consumer read storms by flagging patterns that deviate from the normal compaction cycle.
Alerting on bookie_SERVER_STATUS transitioning to 0 gives immediate notification when a bookie goes read-only, which is the point where compaction is suspended and the death spiral begins.