$ guides / apache-pulsar / apache-pulsar-bookie-disk-filling ▌

Operations Guides

Apache Pulsar bookie disk filling: runway to read-only and how to reclaim space

Bookie disk usage decides whether your cluster can accept writes. When a bookie’s ledger directories fill past the configured threshold (default 95%), the bookie transitions to read-only mode and stops accepting new entries. If enough bookies go read-only, the write quorum for affected ledgers cannot be satisfied, and producers start seeing errors.

BookKeeper needs disk headroom to compact entry logs and reclaim space from deleted ledgers. With the default isForceGCAllowWhenNoSpace=false, reaching the warn threshold (diskUsageWarnThreshold, 0.90) suspends major compaction; with isForceGCAllowWhenNoSpace=true the bookie forces GC instead. At 95% (diskUsageThreshold), the bookie goes read-only and suspends both major and minor compaction entirely. At that point, shortening retention or deleting topics will not reclaim space because the compaction that rewrites entry logs is itself suspended.

This article covers how to estimate runway to read-only, how to break the read-only GC trap, and how to identify the root cause so disk growth does not recur after you reclaim space.

What this means

BookKeeper stores message data in entry logs that interleave entries from multiple ledgers. When a ledger is deleted (retention expired, topic deleted, or subscription cursor advanced), the entry log file still contains entries from other active ledgers. BookKeeper reclaims that space through garbage collection: minor and major compaction rewrite entry logs, removing entries from deleted ledgers and freeing the old files.

This reclamation requires temporary disk headroom. Compaction reads an entry log, writes a new one containing only live entries, then deletes the old file. If the disk is nearly full, there is no room to write the compacted entry log, so compaction cannot proceed.

When a bookie’s disk usage reaches diskUsageThreshold (default 0.95), the bookie marks itself read-only (bookie_SERVER_STATUS drops to 0). BookKeeper suspends both minor and major compaction. The bookie is stuck: it cannot accept writes, and it cannot reclaim space to become writable again without intervention.

flowchart TD
    A["disk below 70%: normal"] -->|"backlog growth or retention too long"| B["disk 70-90%: plan expansion"]
    B -->|"GC cannot reclaim fast enough"| C["disk hits diskUsageThreshold ~95%"]
    C --> D["bookie_SERVER_STATUS = 0, read-only"]
    D --> E["compaction suspended: GC cannot reclaim"]
    E --> F["stuck read-only without intervention"]
    D -.->|"force GC REST API or add disk capacity"| G["writable: GC resumes, space reclaims"]

There is also a per-volume dimension. Bookies can be configured with multiple ledger directories, potentially on separate disks. The metrics bookie_ledger_dir_{path}_usage report usage per directory. One volume can hit the threshold while others have ample free space. The bookie goes read-only when it runs out of writable directories (bookie_ledger_writable_dirs reaches zero), not when aggregate usage hits a number. Monitoring only the aggregate misses this.

One more pattern to flag: if bookie disk usage is stable but subscription backlog is growing, check retention policy and TTL. Data may be expiring (deleted) before slow consumers can read it. Stable disk with growing backlog is silent data loss from the consumer’s perspective, not a healthy state.

Common causes

Cause	What it looks like	First thing to check
Slow consumers growing backlog	`pulsar_subscription_back_log` increasing alongside disk growth	Consumer health, dispatch rate vs publish rate
Retention too long for capacity	Disk growing steadily even with healthy consumers	Namespace retention and TTL settings
BookKeeper GC suspended or too slow	`bookie_gc_*` counters flat, `bookie_ENTRY_LOG_SPACE_BYTES` growing	Compaction thresholds, `isForceGCAllowWhenNoSpace`, disk headroom
Single volume full, others have space	Per-volume `bookie_ledger_dir_{path}_usage` shows imbalance	Per-directory usage, not aggregate
Orphan ledgers from failed deletion	Disk grows after topics deleted; ledger metadata exists without topic reference	`bookkeeper shell listledgers` vs active topics
Abandoned subscriptions blocking GC	Subscription count growing, old cursors with zero consumers	Topic stats for subscriptions with empty consumer lists

Quick checks

# Check per-volume disk usage on each bookie
curl -s http://<bookie-host>:8000/metrics | grep bookie_ledger_dir

# Check writable directory count (zero means read-only)
curl -s http://<bookie-host>:8000/metrics | grep bookie_ledger_writable_dirs

# Check bookie server status (1=writable, 0=read-only, -1=unregistered)
curl -s http://<bookie-host>:8000/metrics | grep bookie_SERVER_STATUS

# Check entry log space (total vs active indicates GC efficiency)
curl -s http://<bookie-host>:8000/metrics | grep -E "ENTRY_LOG"

# Check filesystem-level usage directly
df -h /path/to/ledger/dirs

# Check per-directory usage if multiple volumes configured
du -sh /path/to/ledgers/current/*

# Check GC-related metrics (compaction state, gc counters)
curl -s http://<bookie-host>:8000/metrics | grep -i "gc\|compact"

# Check backlog growth on critical subscriptions
curl -s http://<broker-host>:8080/metrics | grep pulsar_subscription_back_log

# Check message expiration rate (silent data loss signal)
curl -s http://<broker-host>:8080/metrics | grep pulsar_subscription_msg_rate_expired

How to diagnose it

Identify which bookies are affected. Pull bookie_SERVER_STATUS and bookie_ledger_writable_dirs from every bookie. A bookie with status 0 and zero writable directories is already read-only. A bookie with high usage but still writable has runway you need to calculate.
Check per-volume, not aggregate. Pull bookie_ledger_dir_{path}_usage for every ledger directory on every bookie. If one volume is at 94% while others are at 60%, the issue is distribution or a single-volume capacity limit, not overall cluster storage.
Calculate daily growth rate. Compare current usage to usage 7 days ago:
```
daily_growth = (current_usage_bytes - usage_7d_ago_bytes) / 7
```
Estimate runway to 90%. This is your action threshold, not the 95% read-only threshold. Plan expansion at 70%, act urgently at 85%, and treat anything above 90% as critical:
```
days_to_90 = (0.90 * total_capacity_bytes - current_usage_bytes) / daily_growth
```
Check whether GC is running. Look at bookie_gc_* counters and bookie_ACTIVE_ENTRY_LOG_SPACE_BYTES vs bookie_ENTRY_LOG_SPACE_BYTES. If total entry log space is more than 2x active space and growing, GC is falling behind. If compaction counters are flat while disk grows, compaction may be suspended due to insufficient disk headroom.
Check backlog trend. Pull pulsar_subscription_back_log for critical namespaces. Growing backlog means consumers are not keeping up, which accelerates disk fill because retention cannot expire data that has not been acknowledged (depending on policy).
Check for silent data loss. If disk is stable but backlog is growing, check pulsar_subscription_msg_rate_expired. Non-zero expiration on topics where all messages must be consumed means data is being deleted before slow consumers read it.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`bookie_ledger_dir_{path}_usage`	Per-volume disk fill, the metric that triggers read-only	Any volume trending above 70%
`bookie_ledger_writable_dirs`	Count of writable directories; zero means read-only	Dropping to zero on any bookie
`bookie_SERVER_STATUS`	Bookie mode (writable vs read-only)	Transitions to 0
`bookie_ENTRY_LOG_SPACE_BYTES` vs `bookie_ACTIVE_ENTRY_LOG_SPACE_BYTES`	GC efficiency ratio	Total space greater than 2x active space with disk above 70%
`pulsar_subscription_back_log`	Backlog growth accelerates disk fill	Sustained growth for specific subscriptions
`pulsar_subscription_msg_rate_expired`	Silent data loss from TTL before consumers read	Non-zero on topics requiring full consumption
`bookie_gc_*` counters	Whether compaction is actually reclaiming space	Flat counters while disk grows

Fixes

Breaking the read-only GC trap

Once a bookie is read-only with compaction suspended, deleting topics or shortening retention will not reclaim space. You need to either give the bookie headroom or force GC.

Option 1: Force GC via REST API (least disruptive). BookKeeper’s HTTP API exposes forced garbage collection at PUT /api/v1/bookie/gc. Whether it can act while compaction is suspended depends on configuration: isForceGCAllowWhenNoSpace=true makes the bookie force GC instead of suspending it when the disk fills, and forceAllowCompaction gates whether a forced request overrides suspension. To trigger it manually:

# Trigger forced garbage collection
curl -XPUT "http://<bookie-host>:8000/api/v1/bookie/gc" -d '{"forceMajor":true,"forceMinor":true}'

The forceMajor and forceMinor flags are accepted in the JSON request body. If deleted ledgers exist in entry logs, compaction will reclaim their space. No restart required.

Option 2: Temporarily raise the threshold and restart. On older BookKeeper versions where the force GC API does not bypass suspension:

Warning: This requires a bookie restart. The bookie will be temporarily unavailable for writes during restart. If the cluster is already near write-quorum limits, this can trigger producer errors. Ensure enough other bookies are writable before proceeding.

Increase diskUsageThreshold in bookkeeper.conf (for example, from 0.95 to 0.98).
Restart the bookie.
Trigger GC via the REST API or wait for the next compaction cycle.
Once space is reclaimed, lower the threshold back to a safe value.

Option 3: Add disk capacity. If the bookie is genuinely out of space (not just GC-suspended), adding a new ledger directory or expanding the volume gives compaction the headroom it needs. After adding capacity, trigger GC.

Addressing backlog-driven disk growth

If the root cause is slow consumers:

Identify the subscription with the largest and fastest-growing backlog via pulsar_subscription_back_log.
Check whether consumers are connected and processing. If consumers are down, restart them or add capacity.
If the backlog is unrecoverable (consumers will never catch up), consider skipping the subscription cursor forward. This is destructive: messages between the old and new cursor position are permanently lost. Coordinate with the application team before doing this.
Shorten retention on the namespace if the data has limited value past a certain age. Retention should match your actual storage capacity and consumer catch-up capacity.

Fixing orphan ledgers

Orphan ledgers are ledger segments that still exist in BookKeeper’s metadata store but are no longer referenced by any managed ledger (topic). They consume disk space because the bookie GC sees the metadata as valid and does not delete the entries.

This happens when the two-step ledger deletion process (remove from managed ledger list, then async delete from BookKeeper) fails at step 2, often due to a broker restart or metadata store error.

# List all ledgers known to BookKeeper
bin/bookkeeper shell listledgers | wc -l

# Compare with active topics and their ledger counts
# A significant discrepancy between known ledgers and active topic ledger references indicates orphans

BookKeeper has no dedicated orphan-ledger cleanup command in recent versions: listledgers and listunderreplicated are the detection tools, and confirmed orphans must be removed from the metadata store manually.

Fixing abandoned subscriptions

Abandoned subscriptions create cursors that hold a position in the log, preventing data deletion past that position. Over time, storage grows because the topic cannot garbage collect past the oldest cursor.

# List subscriptions with zero connected consumers
curl -s http://<broker-host>:8080/admin/v2/persistent/<tenant>/<namespace>/<topic>/stats | \
  jq '.subscriptions | to_entries[] | select(.value.consumers | length == 0) | .key'

# Delete abandoned subscription (destructive: loses cursor position and all unread messages)
pulsar-admin topics unsubscribe -s <subscription-name> persistent://<tenant>/<namespace>/<topic>

Consider enabling subscriptionExpirationTimeMinutes on namespaces to prevent recurrence. Abandoned subscriptions from microservices creating unique subscription names per instance are a common root cause.

Prevention

Monitor per-volume disk usage, not aggregate. Alert on any single bookie_ledger_dir_{path}_usage above 80% with a positive growth trend. Aggregate disk usage hides single-volume saturation.
Plan expansion at 70%, not 90%. Bookie disk growth is predictable from retention policy and publish rate. Calculate runway weekly using the formula above.
Track the GC efficiency ratio. Monitor bookie_ENTRY_LOG_SPACE_BYTES vs bookie_ACTIVE_ENTRY_LOG_SPACE_BYTES. If the ratio exceeds 2x, GC is falling behind even if disk is not yet critical.
Watch backlog growth alongside disk growth. Backlog is the leading indicator for disk fill. A subscription growing at 10GB/day of backlog fills the bookie proportionally faster.
Check for silent data loss. If disk is stable but backlog is growing, verify that TTL and retention are not expiring data before consumers read it. Monitor pulsar_subscription_msg_rate_expired on topics where data loss is unacceptable.
Ensure compaction is configured to run. Verify minorCompactionThreshold, majorCompactionThreshold, and their intervals are set to non-zero values. A threshold or interval of 0 means compaction never runs and disk usage grows monotonically.
Set readOnlyModeEnabled explicitly. Decide whether bookies go read-only or shut down on disk full. The default may differ between BookKeeper distributions. Read-only is generally safer for cluster availability because a shut-down bookie immediately reduces the write quorum pool.

How Netdata helps

bookie_ledger_dir_{path}_usage is collected per second, letting you calculate daily growth trends and project runway without waiting for coarse scrape intervals.
Overlaying pulsar_subscription_back_log against bookie_ledger_dir_{path}_usage reveals whether disk fill is consumer-driven or retention-driven.
When disk usage is flat but backlog is growing, divergence between pulsar_subscription_msg_rate_expired and stable disk confirms data is expiring before consumers read it.
bookie_SERVER_STATUS and bookie_ledger_writable_dirs are monitored continuously. When a bookie drops to read-only or loses its last writable directory, the alert includes the disk usage context that explains why.
Entry log space metrics (bookie_ACTIVE_ENTRY_LOG_SPACE_BYTES vs bookie_ENTRY_LOG_SPACE_BYTES) are collected as time series, showing when GC starts falling behind before disk pressure becomes critical.