bookie_SERVER_STATUS == 0 means the bookie has transitioned to read-only mode and is no longer accepting writes. In most cases this is a self-protective response to ledger disk usage crossing diskUsageThreshold (default 0.95). The bookie keeps serving reads, but any topic whose ensemble includes this bookie may fail to meet its write quorum, causing broker-side write errors and producer timeouts.

The real danger is the cascade. If enough bookies in the ensemble go read-only simultaneously, the write quorum (Qa) cannot be satisfied for new ledger creation, and all writes to affected topics fail. This can look like a cluster-wide outage even though the root cause is disk exhaustion on a few nodes.

What this means

The metric bookie_SERVER_STATUS reports three states:

ValueMeaningCan accept writes?
1WritableYes
0Read-onlyNo
-1UnregisteredNo

When any ledger directory crosses diskUsageThreshold (default 0.95), the bookie stops writing to that directory. When all ledger directories exceed the threshold, the bookie transitions to read-only entirely. The companion metric bookie_ledger_writable_dirs drops to 0, confirming that no ledger directory can accept new writes.

The bookie checks disk usage every diskCheckInterval milliseconds (default 10000). The transition is automatic and immediate once the threshold is crossed. There is no grace period.

flowchart TD
    A["Disk usage < 90%"] -->|Normal operation| B["bookie_SERVER_STATUS = 1"]
    A -->|Usage crosses 90%| C["diskUsageWarnThreshold hit\nCompaction suspended"]
    C -->|Usage crosses 95%| D["diskUsageThreshold hit\nDir marked unwritable"]
    D -->|All dirs unwritable| E["bookie_SERVER_STATUS = 0\nbookie_ledger_writable_dirs = 0"]
    E -->|Free space below 95%| F["diskUsageLwmThreshold check"]
    F -->|LWM == threshold: stuck| E
    F -->|LWM < threshold: recover| B

The recovery path reveals a common trap. The bookie does not return to read-write until disk usage falls below diskUsageLwmThreshold (the low-water mark). By default, this is set to the same value as diskUsageThreshold (0.95). A bookie that goes read-only at 95% must drop below 95% before it becomes writable again. If garbage collection cannot reclaim enough space to cross back under the threshold, the bookie stays read-only indefinitely.

Common causes

CauseWhat it looks likeFirst thing to check
Disk exhaustion from backlog growthbookie_ledger_dir_*_usage climbing steadily, subscription backlogs growingBacklog size trend vs retention policy
BookKeeper GC not reclaiming spaceDisk full despite topics being deleted; isForceGCAllowWhenNoSpace=false suspends compaction above 90%bookie_ACTIVE_ENTRY_LOG_TOTAL vs bookie_ENTRY_LOG_SPACE_BYTES ratio
Hysteresis trap (LWM equals threshold)Disk freed to 94.5% but bookie stays read-onlydiskUsageLwmThreshold value in bookkeeper.conf
Cold journal replay after restartbookie_SERVER_STATUS == 0 for first 5-10 minutes post-restartBookie uptime
Administrative read-only flagBookie read-only with disk space availableBookie state in metadata store

Quick checks

# Check bookie server status
curl -s http://<bookie-host>:8000/metrics | grep bookie_SERVER_STATUS

# Confirm all ledger dirs are unwritable
curl -s http://<bookie-host>:8000/metrics | grep bookie_ledger_writable_dirs

# Check per-directory disk usage
curl -s http://<bookie-host>:8000/metrics | grep bookie_ledger_dir
# or directly:
df -h /path/to/ledgers

# Check if writes are being blocked
curl -s http://<bookie-host>:8000/metrics | grep bookkeeper_server_ADD_ENTRY_BLOCKED

# Verify bookie process is alive
curl -sf http://<bookie-host>:8000/metrics > /dev/null && echo "UP" || echo "DOWN"

# Check bookie uptime (cold start produces transient read-only)
curl -s http://<bookie-host>:8000/metrics | grep -i uptime

# Check under-replicated ledger count on the auditor
curl -s http://<auditor-host>:8000/metrics | grep auditor_NUM_UNDER_REPLICATED_LEDGERS

How to diagnose it

  1. Confirm the bookie is read-only. Check bookie_SERVER_STATUS and bookie_ledger_writable_dirs. Both should be 0 for a genuine disk-full read-only state.

  2. Rule out cold journal start. If bookie uptime is under 600 seconds, the bookie may still be replaying its journal and can report read-only during replay. Allow 5-10 minutes for large journals before treating this as a failure. The alert guard described in Prevention requires uptime > 600s to avoid this false positive.

  3. Check disk usage on ledger directories. Use df -h on each ledger mount point, or query bookie_ledger_dir_*_usage metrics. If usage is above 95% on all directories, you have confirmed the disk-full cause.

  4. Assess quorum impact. Count writable bookies vs the ensemble size for your topics. If writable bookies have dropped below the ensemble requirement, new ledger creation fails with NotEnoughBookiesException. Check broker logs for this error.

  5. Check whether compaction is suspended. When isForceGCAllowWhenNoSpace is false (the default), BookKeeper suspends major compaction when disk usage exceeds diskUsageWarnThreshold (default 0.90). The mechanism that reclaims space from deleted ledgers is disabled when you need it most. Check GC metrics for activity.

  6. Check the hysteresis configuration. Read diskUsageLwmThreshold from bookkeeper.conf. If it equals diskUsageThreshold (both 0.95 by default), freeing space to 94% is not enough. The bookie must drop below the LWM before it becomes writable again.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
bookie_SERVER_STATUSBinary writable/read-only stateSustained 0 for > 10 minutes with uptime > 600s
bookie_ledger_writable_dirsCount of writable ledger directoriesDrops to 0
bookie_ledger_dir_{path}_usagePer-directory disk usage percentageTrending above 80%
auditor_NUM_UNDER_REPLICATED_LEDGERSData replication health after bookie eventsGrowing instead of trending to zero
bookkeeper_server_ADD_ENTRY_BLOCKEDWrites actively blocked by the bookieNon-zero
bookie_ACTIVE_ENTRY_LOG_TOTAL vs bookie_ENTRY_LOG_SPACE_BYTESGC efficiency ratioTotal space > 2x active space with disk > 70%
pulsar_broker_publish_latencyEnd-to-end write latency as seen by brokersP99 elevation correlates with bookie read-only

Fixes

Free disk space immediately

The most direct recovery is reclaiming space on the ledger directories. Options, in order of safety:

  • Trigger manual compaction. If isForceGCAllowWhenNoSpace is false, compaction is suspended above 90% disk usage. You can enable forced GC temporarily to allow compaction to run despite disk pressure.

  • Delete abandoned subscriptions. Subscriptions with zero connected consumers that hold old cursor positions prevent data deletion. Use pulsar-admin topics stats to find subscriptions with empty consumer lists, verify they are abandoned, then unsubscribe them.

  • Shorten retention policies. If retention is set to keep data longer than disk capacity allows, reduce retention on non-critical namespaces. Shorter retention means older data becomes unavailable.

  • Trigger tiered storage offload. If tiered storage is configured, offloading to S3/GCS frees local disk space for hot data.

After freeing space, verify that usage has dropped below diskUsageLwmThreshold and that bookie_SERVER_STATUS returns to 1 within the next disk check interval (10 seconds by default).

Fix the hysteresis trap

If diskUsageLwmThreshold equals diskUsageThreshold (both 0.95 by default), the bookie cannot recover even after moderate space reclamation. Set diskUsageLwmThreshold lower than diskUsageThreshold to create a recovery band:

diskUsageThreshold=0.95
diskUsageLwmThreshold=0.85

This allows the bookie to return to read-write once disk usage drops below 85%, providing enough headroom for GC and compaction to stabilize without flapping back to read-only.

This change requires a bookie restart to take effect. Plan it as a rolling restart to avoid reducing writable bookie count further.

Handle a bookie that cannot decommission

When a bookie disk is completely full, the decommission process itself may hang because it needs disk space to write recovery metadata. If the bookie cannot decommission cleanly, the workaround is to manually clear data from the ledger directory and take the service offline.

Warning: This is destructive. Confirm the data exists on enough other bookies to satisfy the replication factor before touching any ledger files. Verify auditor_NUM_UNDER_REPLICATED_LEDGERS is not growing after the operation.

Add bookies to restore quorum

If multiple bookies are read-only and the write quorum cannot be met, adding new bookies restores the writable pool. Existing ledgers do not automatically rebalance to new bookies; only new ledgers will use them. AutoRecovery must run to replicate under-replicated ledgers to the new bookies.

Consider pausing automatic recovery (lostBookieRecoveryDelay) if recovery I/O is stressing surviving bookies. Resume it at a controlled rate after stabilization.

Prevention

  • Monitor disk usage growth rate, not just current percentage. Track the trend and estimate runway to 90%. Plan expansion at 70% utilization.
  • Set diskUsageLwmThreshold below diskUsageThreshold. A 10-point recovery band (0.95 threshold, 0.85 LWM) prevents the stuck-read-only scenario.
  • Monitor subscription backlog as a leading indicator of disk fill. Growing backlog consumes bookie disk space.
  • Configure BookKeeper GC correctly. Verify that minorCompactionThreshold and majorCompactionThreshold are set appropriately for your workload. A stuck GC thread means no disk reclamation.
  • Use separate physical disks for journal and ledger storage. Sharing a disk causes I/O contention that slows compaction and accelerates disk fill.
  • Alert with a multi-condition guard. Page only when all of the following are true: bookie_SERVER_STATUS == 0 sustained for > 10 minutes, uptime > 600s, writable bookies < ensemble, and publish traffic is active. Without all four conditions, you will get false positives during cold starts, planned maintenance, and idle clusters.

How Netdata helps

Netdata’s value here is correlation speed: seeing disk fill, the read-only transition, and producer impact in one view rather than cross-referencing separate tools.

  • Per-second collection on bookie_SERVER_STATUS, bookie_ledger_writable_dirs, and per-directory disk usage shows the read-only transition within seconds of it happening, not on a 60-second scrape interval.
  • Disk usage and bookie status on the same dashboard make the causal chain visible immediately when disk crosses 95% and the bookie flips to read-only.
  • Anomaly detection on disk growth rate can surface unusual fill patterns before the threshold is hit.
  • bookie_ledger_writable_dirs alongside bookie_SERVER_STATUS distinguishes a single-directory problem from a full-bookie problem.
  • Broker-side correlation with pulsar_broker_publish_latency and under-replicated ledger count shows whether the read-only bookie is actually impacting producers or whether quorum is still satisfied.
  • Multi-condition alert guards (uptime > 600s, sustained read-only > 10 min, writable bookies < ensemble, active traffic) suppress cold-start and maintenance false positives automatically.