$ guides / apache-pulsar / apache-pulsar-bookie-failure-cascade ▌

Operations Guides

Apache Pulsar bookie failure cascade: recovery I/O that topples surviving bookies

Bookies are failing one at a time. Each time one drops, AutoRecovery starts replicating its ledgers to the survivors. The recovery reads and writes add I/O load to bookies already handling foreground traffic. Journal sync latency climbs, publish latency follows, then the next bookie starts timing out. It fails, and the cycle accelerates.

This is the bookie failure cascade. The root cause is not the initial bookie failure; it is the recovery mechanism competing with production traffic on bookies near their I/O ceiling. With default settings, AutoRecovery triggers immediately (lostBookieRecoveryDelay=0) and reads entries in batches of 100 (rereplicationEntryBatchSize=100) from surviving bookies, then writes them locally.

The key to breaking the cascade is recognizing it early. Bookies fail in sequence, not simultaneously. Publish latency rises on survivors before they fail. The first response is to pause or throttle AutoRecovery to stop the I/O competition, stabilize the cluster, then resume recovery at a controlled rate.

What this means

When a bookie goes down, the auditor (a singleton elected via ZooKeeper) detects the loss and publishes rereplication tasks. Replication workers on surviving bookies pick up these tasks, read entries from other bookies in the ensemble, and write them locally to restore the configured replication factor.

The problem emerges when surviving bookies are already near I/O capacity. Recovery I/O is not throttled separately from foreground traffic. It competes for the same journal disks, the same entry log disks, and the same network bandwidth. If bookies were provisioned for steady-state throughput with little headroom, recovery traffic pushes them past the point where they can service writes within timeout.

flowchart TD
    A["Bookie fails"] --> B["AutoRecovery triggers"]
    B --> C["Recovery I/O hits survivors"]
    C --> D["Journal + publish latency rise"]
    D --> E["Survivor times out, fails"]
    E --> F["More recovery, fewer bookies"]
    F --> C
    G["Operator pauses recovery"] --> H["I/O stops, survivors stabilize"]
    H --> I["Resume at controlled rate"]

The cascade has a specific signature that distinguishes it from other multi-bookie failures:

Sequential failures, not simultaneous. Bookies drop one after another over minutes or hours. Simultaneous failure of multiple bookies points to shared infrastructure: power loss, rack failure, or storage array degradation, not a recovery cascade.
Publish latency rises before bookies fully fail. Recovery I/O inflates journal sync latency on survivors. Brokers wait longer for acks. Producers see higher latency before any bookie is marked failed.
Under-replicated ledger count spikes and does not drain. AutoRecovery creates new under-replicated ledgers (from newly failed bookies) faster than it can resolve existing ones, because the replication workers themselves are starved of I/O.

Common causes

Cause	What it looks like	First thing to check
Aggressive recovery defaults	`lostBookieRecoveryDelay=0` triggers immediate recovery on first failure; `rereplicationEntryBatchSize=100` floods survivors with read/write batches	`bookkeeper.conf` recovery parameters
Bookies provisioned near I/O ceiling	Journal sync latency was already trending up before the first failure; no headroom for recovery traffic	Historical journal sync latency and `iostat` utilization
Shared infrastructure failure domain	Multiple bookies on same rack, same storage array, same power circuit; failures cluster by topology	Map bookie hosts to rack, zone, and storage
Journal and ledger storage on same disk	Recovery reads from entry logs directly compete with journal fsync writes on the same device	`mount` output and disk device mapping

Quick checks

These are safe, read-only checks to confirm the cascade pattern and assess urgency.

# Check bookie server status across all bookies (1=writable, 0=read-only, -1=unregistered)
for host in bookie1 bookie2 bookie3 bookie4 bookie5; do
  echo -n "$host: "
  curl -s http://$host:8000/metrics | grep bookie_SERVER_STATUS
done

# Count under-replicated ledgers
bookkeeper shell listunderreplicated | wc -l

# Check broker publish latency for degradation
curl -s http://<broker-host>:8080/metrics | grep pulsar_broker_publish_latency

# Check journal sync latency on surviving bookies
curl -s http://<surviving-bookie>:8000/metrics | grep bookie_journal_JOURNAL_SYNC

# Check add-entry queue depth on surviving bookies
curl -s http://<surviving-bookie>:8000/metrics | grep bookkeeper_server_ADD_ENTRY_IN_PROGRESS

# Check disk I/O utilization on surviving bookie journal disks
iostat -x 1

# Check current lostBookieRecoveryDelay setting
bookkeeper shell lostbookierecoverydelay --get

How to diagnose it

Confirm sequential failure pattern. Review bookie_SERVER_STATUS transitions or process uptime for each failed bookie. If two or more failed within seconds of each other, investigate shared infrastructure first. If they failed minutes or hours apart with recovery activity in between, this is a cascade.
Verify recovery I/O on survivors. On bookies still writable, check journal sync latency (bookie_journal_JOURNAL_SYNC) and add-entry in-progress count (bookkeeper_server_ADD_ENTRY_IN_PROGRESS). If these are elevated compared to pre-failure baselines, recovery traffic is competing with foreground writes.
Check under-replicated ledger trend. Run bookkeeper shell listunderreplicated | wc -l twice, a few minutes apart. If the count is growing, AutoRecovery is losing ground. If it is stable but non-zero, recovery may be stalled.
Assess remaining writable bookies. Count bookies with bookie_SERVER_STATUS == 1 and compare against your ensemble size, write quorum, and ack quorum. If available bookies are approaching the quorum threshold, the cluster is one failure away from write unavailability.
Check whether recovery is actually running. On Pulsar versions before 3.0.2 (BookKeeper before 4.16.3), a ReplicationWorker deadlock bug (fixed by apache/bookkeeper#4058) could stall recovery entirely. If under-replicated ledgers are not decreasing and recovery I/O is not visible on survivors, the replication worker may be deadlocked rather than causing a cascade.
Map failures to infrastructure topology. If failed bookies share a rack, zone, storage array, or network switch, the root cause is infrastructure, not recovery I/O. Address the infrastructure failure first, then manage recovery carefully.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`bookie_SERVER_STATUS` per bookie	Tracks which bookies are writable vs failed	Multiple transitions to 0 or -1 in sequence
`auditor_NUM_UNDER_REPLICATED_LEDGERS`	Recovery backlog; growing count means recovery is losing	Spike on first failure, then continues growing instead of draining
`pulsar_broker_publish_latency` P99	End-to-end write latency as seen by producers	Sustained 2x or more above baseline after first failure
`bookie_journal_JOURNAL_SYNC` P99 on survivors	Recovery I/O inflates journal fsync latency	P99 rising on bookies that have not failed
`bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE`	Earliest signal of journal disk saturation from recovery writes	Sustained non-zero depth on surviving bookies
`bookkeeper_server_ADD_ENTRY_IN_PROGRESS`	Write queue depth on survivors; grows when disk cannot keep up	Sustained upward trend on surviving bookies
`bookkeeper_server_ADD_ENTRY_BLOCKED`	Bookie is refusing new writes	Non-zero on any surviving bookie
Disk I/O utilization on journal devices	Direct measure of recovery vs foreground I/O competition	%util approaching 100% on journal disk

Fixes

Pause recovery to stop the cascade

Stop the I/O competition first. Pause or delay AutoRecovery to give surviving bookies breathing room.

# Set a delay (in seconds) before recovery triggers after a bookie loss
bookkeeper shell lostbookierecoverydelay --set 3600

# Or disable AutoRecovery entirely (runs without -e; stops all recovery)
bookkeeper shell autorecovery

Warning: Pausing recovery means under-replicated ledgers will not be repaired. The cluster is running at reduced redundancy. This is acceptable for the time it takes to stabilize survivors, but you must resume recovery before another failure occurs or you risk data loss.

Tune recovery batch size

Once the cluster is stable, resume recovery at a controlled rate. The default rereplicationEntryBatchSize=100 entries per batch may be too aggressive for bookies with limited I/O headroom. Reduce it in bookkeeper.conf:

rereplicationEntryBatchSize=10

This limits how many entries the replication worker reads and writes per batch cycle, reducing peak I/O pressure on survivors. There is no separate I/O rate limiter for recovery traffic in current releases; the batch size is the primary throughput control.

Resume recovery gradually

# Re-enable AutoRecovery
bookkeeper shell autorecovery -e

# Or progressively reduce the delay
bookkeeper shell lostbookierecoverydelay --set 300
# Monitor recovery progress and survivor health for several minutes
# Then reduce further if stable

Monitor auditor_NUM_UNDER_REPLICATED_LEDGERS and journal sync latency on survivors during recovery. If latency spikes again, pause and reduce the batch size further before resuming.

Address the root cause

If the initial bookie failure was caused by disk degradation, host failure, or OOM, fix or replace that bookie before resuming full recovery. Adding a fresh bookie gives AutoRecovery a target that is not already under load, spreading recovery writes across more devices.

If the root cause was shared infrastructure (rack power loss, storage array failure), ensure replacement bookies are in a different failure domain.

Add new bookies before resuming recovery

If survivors are too degraded to absorb recovery writes, add new bookies first. Recovery traffic will target the new nodes, distributing I/O across more hardware. Existing ledgers do not automatically rebalance to new bookies. Only recovery writes and new ledger creation use them, but that is sufficient to absorb recovery load.

Prevention

Provision bookie I/O with recovery headroom. If bookies run at 70-80% of journal disk bandwidth during normal operation, there is no room for recovery traffic. Size journal devices for peak foreground write rate plus a recovery burst from the largest single bookie’s data.
Set a non-zero lostBookieRecoveryDelay in production. The default of 0 triggers immediate recovery. A delay of a few minutes gives operators time to assess the situation, verify survivor health, and prepare for the I/O impact.
Keep journal and entry log on separate disks. Recovery reads from entry logs compete with journal writes. On a shared disk, this directly blocks the write path. This is the single most common architecture mistake in Pulsar deployments.
Monitor under-replicated ledger count as a leading indicator. A sudden spike after a bookie event is expected. A count that does not trend toward zero within minutes to hours (depending on data volume) means recovery is failing or cascading.
Test single-bookie failure in staging. Observe what recovery I/O looks like on your hardware. Measure the publish latency impact. This establishes the baseline for recognizing a cascade versus normal recovery behavior.
Verify your BookKeeper version is past the ReplicationWorker deadlock fix. BookKeeper 4.16.3 (PR #4058), bundled with Pulsar 3.0.2, fixed a ReplicationWorker deadlock that could stall recovery entirely (Pulsar issue #21987). On older versions, recovery may silently stop rather than cascade.

How Netdata helps

Per-second bookie status tracking across all bookies makes the failure sequence immediately visible. Sequential transitions confirm a cascade; simultaneous transitions point to infrastructure.
Journal sync latency correlation across all bookies shows recovery I/O impact on survivors in real time. When one bookie fails and latency spikes on the others within seconds, the cascade mechanism is visible before the next failure occurs.
Publish latency and journal queue depth on the same dashboard reveal the cause-and-effect chain: recovery writes inflate journal force-write queues, which inflate fsync latency, which inflates broker publish latency.
Under-replicated ledger count trending shows whether recovery is making progress or losing ground.
Disk I/O utilization per device distinguishes recovery I/O competition on journal disks from entry log pressure, especially when journal and ledger storage are on separate devices.