Bookie disk usage decides whether your cluster can accept writes. When a bookie’s ledger directories fill past the configured threshold (default 95%), the bookie transitions to read-only mode and stops accepting new entries. If enough bookies go read-only, the write quorum for affected ledgers cannot be satisfied, and producers start seeing errors.

BookKeeper needs disk headroom to compact entry logs and reclaim space from deleted ledgers. At 95% (diskUsageThreshold), the bookie goes read-only and suspends GC entirely. At that point, shortening retention or deleting topics will not reclaim space because the compaction that rewrites entry logs is itself suspended.

This article covers how to estimate runway to read-only, how to break the read-only GC trap, and how to identify the root cause so disk growth does not recur after you reclaim space.

What this means

BookKeeper stores message data in entry logs that interleave entries from multiple ledgers. When a ledger is deleted (retention expired, topic deleted, or subscription cursor advanced), the entry log file still contains entries from other active ledgers. BookKeeper reclaims that space through garbage collection: minor and major compaction rewrite entry logs, removing entries from deleted ledgers and freeing the old files.

This reclamation requires temporary disk headroom. Compaction reads an entry log, writes a new one containing only live entries, then deletes the old file. If the disk is nearly full, there is no room to write the compacted entry log, so compaction cannot proceed.

When a bookie’s disk usage reaches diskUsageThreshold (default 0.95), the bookie marks itself read-only (bookie_SERVER_STATUS drops to 0). BookKeeper suspends both minor and major compaction. The bookie is stuck: it cannot accept writes, and it cannot reclaim space to become writable again without intervention.

flowchart TD
    A["disk below 70%: normal"] -->|"backlog growth or retention too long"| B["disk 70-90%: plan expansion"]
    B -->|"GC cannot reclaim fast enough"| C["disk hits diskUsageThreshold ~95%"]
    C --> D["bookie_SERVER_STATUS = 0, read-only"]
    D --> E["compaction suspended: GC cannot reclaim"]
    E --> F["stuck read-only without intervention"]
    D -.->|"force GC REST API or add disk capacity"| G["writable: GC resumes, space reclaims"]

There is also a per-volume dimension. Bookies can be configured with multiple ledger directories, potentially on separate disks. The metrics bookie_ledger_dir_{path}_usage report usage per directory. One volume can hit the threshold while others have ample free space. The bookie goes read-only when it runs out of writable directories (bookie_ledger_writable_dirs reaches zero), not when aggregate usage hits a number. Monitoring only the aggregate misses this.

One more pattern to flag: if bookie disk usage is stable but subscription backlog is growing, check retention policy and TTL. Data may be expiring (deleted) before slow consumers can read it. Stable disk with growing backlog is silent data loss from the consumer’s perspective, not a healthy state.

Common causes

CauseWhat it looks likeFirst thing to check
Slow consumers growing backlogpulsar_subscription_back_log increasing alongside disk growthConsumer health, dispatch rate vs publish rate
Retention too long for capacityDisk growing steadily even with healthy consumersNamespace retention and TTL settings
BookKeeper GC suspended or too slowbookie_gc_* counters flat, bookie_ENTRY_LOG_SPACE_BYTES growingCompaction thresholds, isForceGCAllowWhenNoSpace, disk headroom
Single volume full, others have spacePer-volume bookie_ledger_dir_{path}_usage shows imbalancePer-directory usage, not aggregate
Orphan ledgers from failed deletionDisk grows after topics deleted; ledger metadata exists without topic referencebookkeeper shell listledgers vs active topics
Abandoned subscriptions blocking GCSubscription count growing, old cursors with zero consumersTopic stats for subscriptions with empty consumer lists

Quick checks

# Check per-volume disk usage on each bookie
curl -s http://<bookie-host>:8000/metrics | grep bookie_ledger_dir

# Check writable directory count (zero means read-only)
curl -s http://<bookie-host>:8000/metrics | grep bookie_ledger_writable_dirs

# Check bookie server status (1=writable, 0=read-only, -1=unregistered)
curl -s http://<bookie-host>:8000/metrics | grep bookie_SERVER_STATUS

# Check entry log space (total vs active indicates GC efficiency)
curl -s http://<bookie-host>:8000/metrics | grep -E "ENTRY_LOG"

# Check filesystem-level usage directly
df -h /path/to/ledger/dirs

# Check per-directory usage if multiple volumes configured
du -sh /path/to/ledgers/current/*

# Check GC-related metrics (compaction state, gc counters)
curl -s http://<bookie-host>:8000/metrics | grep -i "gc\|compact"

# Check backlog growth on critical subscriptions
curl -s http://<broker-host>:8080/metrics | grep pulsar_subscription_back_log

# Check message expiration rate (silent data loss signal)
curl -s http://<broker-host>:8080/metrics | grep pulsar_subscription_msg_rate_expired

How to diagnose it

  1. Identify which bookies are affected. Pull bookie_SERVER_STATUS and bookie_ledger_writable_dirs from every bookie. A bookie with status 0 and zero writable directories is already read-only. A bookie with high usage but still writable has runway you need to calculate.

  2. Check per-volume, not aggregate. Pull bookie_ledger_dir_{path}_usage for every ledger directory on every bookie. If one volume is at 94% while others are at 60%, the issue is distribution or a single-volume capacity limit, not overall cluster storage.

  3. Calculate daily growth rate. Compare current usage to usage 7 days ago:

    daily_growth = (current_usage_bytes - usage_7d_ago_bytes) / 7

  4. Estimate runway to 90%. This is your action threshold, not the 95% read-only threshold. Plan expansion at 70%, act urgently at 85%, and treat anything above 90% as critical:

    days_to_90 = (0.90 * total_capacity_bytes - current_usage_bytes) / daily_growth

  5. Check whether GC is running. Look at bookie_gc_* counters and bookie_ACTIVE_ENTRY_LOG_SPACE_BYTES vs bookie_ENTRY_LOG_SPACE_BYTES. If total entry log space is more than 2x active space and growing, GC is falling behind. If compaction counters are flat while disk grows, compaction may be suspended due to insufficient disk headroom.

  6. Check backlog trend. Pull pulsar_subscription_back_log for critical namespaces. Growing backlog means consumers are not keeping up, which accelerates disk fill because retention cannot expire data that has not been acknowledged (depending on policy).

  7. Check for silent data loss. If disk is stable but backlog is growing, check pulsar_subscription_msg_rate_expired. Non-zero expiration on topics where all messages must be consumed means data is being deleted before slow consumers read it.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
bookie_ledger_dir_{path}_usagePer-volume disk fill, the metric that triggers read-onlyAny volume trending above 70%
bookie_ledger_writable_dirsCount of writable directories; zero means read-onlyDropping to zero on any bookie
bookie_SERVER_STATUSBookie mode (writable vs read-only)Transitions to 0
bookie_ENTRY_LOG_SPACE_BYTES vs bookie_ACTIVE_ENTRY_LOG_SPACE_BYTESGC efficiency ratioTotal space greater than 2x active space with disk above 70%
pulsar_subscription_back_logBacklog growth accelerates disk fillSustained growth for specific subscriptions
pulsar_subscription_msg_rate_expiredSilent data loss from TTL before consumers readNon-zero on topics requiring full consumption
bookie_gc_* countersWhether compaction is actually reclaiming spaceFlat counters while disk grows

Fixes

Breaking the read-only GC trap

Once a bookie is read-only with compaction suspended, deleting topics or shortening retention will not reclaim space. You need to either give the bookie headroom or force GC.

Option 1: Force GC via REST API (least disruptive). On BookKeeper 4.15.0 and later, the GC REST API accepts a force parameter that bypasses the compaction suspension that normally kicks in above the warn threshold:

# Trigger forced garbage collection (requires BookKeeper 4.15+)
curl -XPUT "http://<bookie-host>:8000/api/v1/bookie/gc?force=true"

This forces minor and major compaction even when the disk is above the warn threshold. If deleted ledgers exist in entry logs, compaction will reclaim their space. No restart required.

Option 2: Temporarily raise the threshold and restart. On older BookKeeper versions where the force GC API does not bypass suspension:

Warning: This requires a bookie restart. The bookie will be temporarily unavailable for writes during restart. If the cluster is already near write-quorum limits, this can trigger producer errors. Ensure enough other bookies are writable before proceeding.

  1. Increase diskUsageThreshold in bookkeeper.conf (for example, from 0.95 to 0.98).
  2. Restart the bookie.
  3. Trigger GC via the REST API or wait for the next compaction cycle.
  4. Once space is reclaimed, lower the threshold back to a safe value.

Option 3: Add disk capacity. If the bookie is genuinely out of space (not just GC-suspended), adding a new ledger directory or expanding the volume gives compaction the headroom it needs. After adding capacity, trigger GC.

Addressing backlog-driven disk growth

If the root cause is slow consumers:

  1. Identify the subscription with the largest and fastest-growing backlog via pulsar_subscription_back_log.
  2. Check whether consumers are connected and processing. If consumers are down, restart them or add capacity.
  3. If the backlog is unrecoverable (consumers will never catch up), consider skipping the subscription cursor forward. This is destructive: messages between the old and new cursor position are permanently lost. Coordinate with the application team before doing this.
  4. Shorten retention on the namespace if the data has limited value past a certain age. Retention should match your actual storage capacity and consumer catch-up capacity.

Fixing orphan ledgers

Orphan ledgers are ledger segments that still exist in BookKeeper’s metadata store but are no longer referenced by any managed ledger (topic). They consume disk space because the bookie GC sees the metadata as valid and does not delete the entries.

This happens when the two-step ledger deletion process (remove from managed ledger list, then async delete from BookKeeper) fails at step 2, often due to a broker restart or metadata store error.

# List all ledgers known to BookKeeper
bin/bookkeeper shell listledgers | wc -l

# Compare with active topics and their ledger counts
# A significant discrepancy between known ledgers and active topic ledger references indicates orphans

Fixing abandoned subscriptions

Abandoned subscriptions create cursors that hold a position in the log, preventing data deletion past that position. Over time, storage grows because the topic cannot garbage collect past the oldest cursor.

# List subscriptions with zero connected consumers
curl -s http://<broker-host>:8080/admin/v2/persistent/<tenant>/<namespace>/<topic>/stats | \
  jq '.subscriptions | to_entries[] | select(.value.consumers | length == 0) | .key'

# Delete abandoned subscription (destructive: loses cursor position and all unread messages)
pulsar-admin topics unsubscribe -s <subscription-name> persistent://<tenant>/<namespace>/<topic>

Consider enabling subscriptionExpirationTimeMinutes on namespaces to prevent recurrence. Abandoned subscriptions from microservices creating unique subscription names per instance are a common root cause.

Prevention

  • Monitor per-volume disk usage, not aggregate. Alert on any single bookie_ledger_dir_{path}_usage above 80% with a positive growth trend. Aggregate disk usage hides single-volume saturation.
  • Plan expansion at 70%, not 90%. Bookie disk growth is predictable from retention policy and publish rate. Calculate runway weekly using the formula above.
  • Track the GC efficiency ratio. Monitor bookie_ENTRY_LOG_SPACE_BYTES vs bookie_ACTIVE_ENTRY_LOG_SPACE_BYTES. If the ratio exceeds 2x, GC is falling behind even if disk is not yet critical.
  • Watch backlog growth alongside disk growth. Backlog is the leading indicator for disk fill. A subscription growing at 10GB/day of backlog fills the bookie proportionally faster.
  • Check for silent data loss. If disk is stable but backlog is growing, verify that TTL and retention are not expiring data before consumers read it. Monitor pulsar_subscription_msg_rate_expired on topics where data loss is unacceptable.
  • Ensure compaction is configured to run. Verify minorCompactionThreshold, majorCompactionThreshold, and their intervals are set to non-zero values. A threshold or interval of 0 means compaction never runs and disk usage grows monotonically.
  • Set readOnlyModeEnabled explicitly. Decide whether bookies go read-only or shut down on disk full. The default may differ between BookKeeper distributions. Read-only is generally safer for cluster availability because a shut-down bookie immediately reduces the write quorum pool.

How Netdata helps

  • bookie_ledger_dir_{path}_usage is collected per second, letting you calculate daily growth trends and project runway without waiting for coarse scrape intervals.
  • Overlaying pulsar_subscription_back_log against bookie_ledger_dir_{path}_usage reveals whether disk fill is consumer-driven or retention-driven.
  • When disk usage is flat but backlog is growing, divergence between pulsar_subscription_msg_rate_expired and stable disk confirms data is expiring before consumers read it.
  • bookie_SERVER_STATUS and bookie_ledger_writable_dirs are monitored continuously. When a bookie drops to read-only or loses its last writable directory, the alert includes the disk usage context that explains why.
  • Entry log space metrics (bookie_ACTIVE_ENTRY_LOG_SPACE_BYTES vs bookie_ENTRY_LOG_SPACE_BYTES) are collected as time series, showing when GC starts falling behind before disk pressure becomes critical.