Apache Pulsar bookie read-only: disk full and bookie_SERVER_STATUS at zero

bookie_SERVER_STATUS == 0 means the bookie has transitioned to read-only mode and is no longer accepting writes. In most cases this is a self-protective response to ledger disk usage crossing diskUsageThreshold (default 0.95). The bookie keeps serving reads, but any topic whose ensemble includes this bookie may fail to meet its write quorum, causing broker-side write errors and producer timeouts.

The real danger is the cascade. If enough bookies in the ensemble go read-only simultaneously, the write quorum (Qa) cannot be satisfied for new ledger creation, and all writes to affected topics fail. This can look like a cluster-wide outage even though the root cause is disk exhaustion on a few nodes.

What this means

The metric bookie_SERVER_STATUS reports three states:

When any ledger directory crosses diskUsageThreshold (default 0.95), the bookie stops writing to that directory. When all ledger directories exceed the threshold, the bookie transitions to read-only entirely. The companion metric bookie_ledger_writable_dirs drops to 0, confirming that no ledger directory can accept new writes.

The bookie checks disk usage every diskCheckInterval milliseconds (default 10000). The transition is automatic and immediate once the threshold is crossed. There is no grace period.

flowchart TD
    A["Disk usage < 90%"] -->|Normal operation| B["bookie_SERVER_STATUS = 1"]
    A -->|Usage crosses 90%| C["diskUsageWarnThreshold hit\nCompaction suspended"]
    C -->|Usage crosses 95%| D["diskUsageThreshold hit\nDir marked unwritable"]
    D -->|All dirs unwritable| E["bookie_SERVER_STATUS = 0\nbookie_ledger_writable_dirs = 0"]
    E -->|Free space below 95%| F["diskUsageLwmThreshold check"]
    F -->|LWM == threshold: stuck| E
    F -->|LWM < threshold: recover| B

The recovery path reveals a common trap. The bookie does not return to read-write until disk usage falls below diskUsageLwmThreshold (the low-water mark). By default, this is set to the same value as diskUsageThreshold (0.95). A bookie that goes read-only at 95% must drop below 95% before it becomes writable again. If garbage collection cannot reclaim enough space to cross back under the threshold, the bookie stays read-only indefinitely.

Common causes

Quick checks

# Check bookie server status
curl -s http://<bookie-host>:8000/metrics | grep bookie_SERVER_STATUS

# Confirm all ledger dirs are unwritable
curl -s http://<bookie-host>:8000/metrics | grep bookie_ledger_writable_dirs

# Check per-directory disk usage
curl -s http://<bookie-host>:8000/metrics | grep bookie_ledger_dir
# or directly:
df -h /path/to/ledgers

# Check if writes are being blocked
curl -s http://<bookie-host>:8000/metrics | grep bookkeeper_server_ADD_ENTRY_BLOCKED

# Verify bookie process is alive
curl -sf http://<bookie-host>:8000/metrics > /dev/null && echo "UP" || echo "DOWN"

# Check bookie uptime (cold start produces transient read-only)
curl -s http://<bookie-host>:8000/metrics | grep -i uptime

# Check under-replicated ledger count on the auditor
curl -s http://<auditor-host>:8000/metrics | grep auditor_NUM_UNDER_REPLICATED_LEDGERS

How to diagnose it

1. Confirm the bookie is read-only. Check bookie_SERVER_STATUS and bookie_ledger_writable_dirs. Both should be 0 for a genuine disk-full read-only state.

2. Rule out cold journal start. If bookie uptime is under 600 seconds, the bookie may still be replaying its journal and can report read-only during replay. Allow 5-10 minutes for large journals before treating this as a failure. The alert guard described in Prevention requires uptime > 600s to avoid this false positive.

3. Check disk usage on ledger directories. Use df -h on each ledger mount point, or query bookie_ledger_dir_*_usage metrics. If usage is above 95% on all directories, you have confirmed the disk-full cause.

4. Assess quorum impact. Count writable bookies vs the ensemble size for your topics. If writable bookies have dropped below the ensemble requirement, new ledger creation fails with NotEnoughBookiesException. Check broker logs for this error.

5. Check whether compaction is suspended. When isForceGCAllowWhenNoSpace is false (the default), BookKeeper suspends major compaction when disk usage exceeds diskUsageWarnThreshold (default 0.90). The mechanism that reclaims space from deleted ledgers is disabled when you need it most. Check GC metrics for activity.

6. Check the hysteresis configuration. Read diskUsageLwmThreshold from bookkeeper.conf. If it equals diskUsageThreshold (both 0.95 by default), freeing space to 94% is not enough. The bookie must drop below the LWM before it becomes writable again.

Metrics and signals to monitor

Fixes

Free disk space immediately

The most direct recovery is reclaiming space on the ledger directories. Options, in order of safety:

• Trigger manual compaction. If isForceGCAllowWhenNoSpace is false, compaction is suspended above 90% disk usage. You can enable forced GC temporarily to allow compaction to run despite disk pressure.

Modern BookKeeper (4.10+) has no bookkeeper shell gc command; forced GC is triggered through the Bookie HTTP API: curl -X PUT http://<bookie-host>:8080/api/v1/bookie/gc. A GET on the same endpoint reports whether forced GC is currently running.

• Delete abandoned subscriptions. Subscriptions with zero connected consumers that hold old cursor positions prevent data deletion. Use pulsar-admin topics stats to find subscriptions with empty consumer lists, verify they are abandoned, then unsubscribe them.

• Shorten retention policies. If retention is set to keep data longer than disk capacity allows, reduce retention on non-critical namespaces. Shorter retention means older data becomes unavailable.

• Trigger tiered storage offload. If tiered storage is configured, offloading to S3/GCS frees local disk space for hot data.

After freeing space, verify that usage has dropped below diskUsageLwmThreshold and that bookie_SERVER_STATUS returns to 1 within the next disk check interval (10 seconds by default).

Fix the hysteresis trap

If diskUsageLwmThreshold equals diskUsageThreshold (both 0.95 by default), the bookie cannot recover even after moderate space reclamation. Set diskUsageLwmThreshold lower than diskUsageThreshold to create a recovery band:

diskUsageThreshold=0.95
diskUsageLwmThreshold=0.85

This allows the bookie to return to read-write once disk usage drops below 85%, providing enough headroom for GC and compaction to stabilize without flapping back to read-only.

This change requires a bookie restart to take effect. Plan it as a rolling restart to avoid reducing writable bookie count further.

Handle a bookie that cannot decommission

When a bookie disk is completely full, the decommission process itself may hang because it needs disk space to write recovery metadata. If the bookie cannot decommission cleanly, the workaround is to manually clear data from the ledger directory and take the service offline.

Warning: This is destructive. Confirm the data exists on enough other bookies to satisfy the replication factor before touching any ledger files. Verify auditor_NUM_UNDER_REPLICATED_LEDGERS is not growing after the operation.

Add bookies to restore quorum

If multiple bookies are read-only and the write quorum cannot be met, adding new bookies restores the writable pool. Existing ledgers do not automatically rebalance to new bookies; only new ledgers will use them. AutoRecovery must run to replicate under-replicated ledgers to the new bookies.

Consider pausing automatic recovery (lostBookieRecoveryDelay) if recovery I/O is stressing surviving bookies. Resume it at a controlled rate after stabilization.

Prevention

• Monitor disk usage growth rate, not just current percentage. Track the trend and estimate runway to 90%. Plan expansion at 70% utilization.
• Set diskUsageLwmThreshold below diskUsageThreshold. A 10-point recovery band (0.95 threshold, 0.85 LWM) prevents the stuck-read-only scenario.
• Monitor subscription backlog as a leading indicator of disk fill. Growing backlog consumes bookie disk space.
• Configure BookKeeper GC correctly. Verify that minorCompactionThreshold and majorCompactionThreshold are set appropriately for your workload. A stuck GC thread means no disk reclamation.
• Use separate physical disks for journal and ledger storage. Sharing a disk causes I/O contention that slows compaction and accelerates disk fill.
• Alert with a multi-condition guard. Page only when all of the following are true: bookie_SERVER_STATUS == 0 sustained for > 10 minutes, uptime > 600s, writable bookies < ensemble, and publish traffic is active. Without all four conditions, you will get false positives during cold starts, planned maintenance, and idle clusters.

How Netdata helps

Netdata’s value here is correlation speed: seeing disk fill, the read-only transition, and producer impact in one view rather than cross-referencing separate tools.

• Per-second collection on bookie_SERVER_STATUS, bookie_ledger_writable_dirs, and per-directory disk usage shows the read-only transition within seconds of it happening, not on a 60-second scrape interval.
• Disk usage and bookie status on the same dashboard make the causal chain visible immediately when disk crosses 95% and the bookie flips to read-only.
• Anomaly detection on disk growth rate can surface unusual fill patterns before the threshold is hit.
• bookie_ledger_writable_dirs alongside bookie_SERVER_STATUS distinguishes a single-directory problem from a full-bookie problem.
• Broker-side correlation with pulsar_broker_publish_latency and under-replicated ledger count shows whether the read-only bookie is actually impacting producers or whether quorum is still satisfied.
• Multi-condition alert guards (uptime > 600s, sustained read-only > 10 min, writable bookies < ensemble, active traffic) suppress cold-start and maintenance false positives automatically.

Related guides

• Apache Pulsar active connections climbing: connection leaks and file descriptor exhaustion
• Apache Pulsar bookie add-entry queue not draining: writes arriving faster than the disk can commit
• Apache Pulsar broker down: telling a dead broker from a fenced one
• Apache Pulsar broker GC death spiral: heap pressure, stop-the-world pauses, and lost topic ownership
• Apache Pulsar broker lookup failures: new clients cannot find their topic
• How Apache Pulsar actually works in production: a mental model for operators
• Apache Pulsar journal force write queue growing: the earliest write-saturation signal
• Apache Pulsar bookie journal and ledger storage on one disk: the #1 architecture mistake
• Apache Pulsar write stall: bookie journal fsync latency and the blocked write path
• Apache Pulsar monitoring checklist: the signals every production cluster needs
• Apache Pulsar monitoring maturity model: from survival to expert
• Apache Pulsar OutOfDirectMemoryError: the off-heap crash JVM heap dashboards never show