Apache Pulsar under-replicated ledgers: data at risk after a bookie failure

You lost a bookie. The broker layer kept serving traffic because your write quorum absorbed the failure. But now auditor_NUM_UNDER_REPLICATED_LEDGERS is climbing, and it is not coming back down. Every minute it stays elevated is a minute your data exists on fewer copies than your replication factor demands. If another bookie fails before AutoRecovery finishes rereplicating, those ledgers are gone permanently.

This metric is exposed only on the elected auditor node, not on every bookie. If you are scraping bookie metrics and do not see it, you are either scraping the wrong host or AutoRecovery is not running. In steady state, the count is zero. After a bookie restart or failure, a temporary spike is expected and should trend back to zero within minutes to hours depending on data volume. A count that grows across audit intervals means recovery is failing, or bookies are dying faster than recovery can complete.

The urgency scales with how many bookies you have left and your ensemble headroom. A cluster with ensemble size 3, write quorum 2, and ack quorum 2 can tolerate one bookie failure per ledger. If under-replicated ledgers pile up and a second bookie in the same ensemble fails, entries with only one remaining replica are one failure away from data loss.

What this means

BookKeeper stores each topic’s data across an ensemble of bookies. When a bookie becomes unavailable (crash, disk failure, read-only transition, network partition), any ledger that had entries on that bookie now has fewer valid copies than the configured write quorum. The auditor, an elected leader among the AutoRecovery nodes, periodically scans for these ledgers and marks them under-replicated.

AutoRecovery then rereplicates the affected entries to healthy bookies. This is asynchronous and competes for I/O with foreground traffic on the surviving bookies. Recovery prioritizes open (actively written) ledgers over closed ones, so historical data may stay under-replicated longer.

The danger window is the gap between bookie failure and recovery completion. During that window, ledgers that lost a replica are more fragile than your replication factor implies.

flowchart TD
    A["Bookie fails or goes read-only"] --> B["Auditor marks affected ledgers
under-replicated"]
    B --> C["AutoRecovery begins
rereplication to healthy bookies"]
    C --> D{"Second bookie fails
before recovery completes?"}
    D -- No --> E["Under-replicated count
trends to zero"]
    D -- Yes --> F["Entries with insufficient replicas
may be permanently lost"]
    C --> G["Recovery I/O loads
surviving bookies"]
    G --> H["Surviving bookies slow down
or appear failed"]
    H --> C

Common causes

Quick checks

Run these read-only commands to assess the situation. None modify cluster state.

# Check under-replicated ledger count via BookKeeper shell
bin/bookkeeper shell listunderreplicated | wc -l

# Check whether AutoRecovery is enabled and running
bin/bookkeeper shell autorecovery -status

# Check auditor metrics (only on the elected auditor node)
curl -s http://<auditor-host>:8000/metrics | grep auditor_NUM_UNDER_REPLICATED_LEDGERS

# Check for ledgers with entries that have zero surviving replicas
curl -s http://<auditor-host>:8000/metrics | grep auditor_NUM_LEDGERS_HAVING_NO_REPLICA

# Check bookie status across all nodes
curl -s http://<bookie-host>:8000/metrics | grep bookie_SERVER_STATUS

# Check journal sync latency on surviving bookies (recovery I/O impact)
curl -s http://<bookie-host>:8000/metrics | grep bookie_journal_JOURNAL_SYNC

# Check Pulsar version
<!-- TODO: verify the correct admin API path for broker version in Pulsar 3.x/4.x -->
curl -s http://<broker-host>:8080/admin/v2/brokers/<cluster>/version

The auditor_NUM_LEDGERS_HAVING_NO_REPLICA_OF_AN_ENTRY metric is the most alarming signal in this family. If it is non-zero, some entries have zero surviving copies. Those entries are already lost. The related metrics auditor_NUM_LEDGERS_HAVING_LESS_THAN_WQ_REPLICAS_OF_AN_ENTRY and auditor_NUM_LEDGERS_HAVING_LESS_THAN_AQ_REPLICAS_OF_AN_ENTRY tell you how many ledgers are below write quorum and ack quorum respectively.

How to diagnose it

1. Confirm the count is real, not a stale scrape. Run bin/bookkeeper shell listunderreplicated directly. If the shell output disagrees with your metrics dashboard, your scraper may be hitting the wrong node (the metric lives on the auditor, not every bookie) or the metrics endpoint may be misconfigured.

2. Verify AutoRecovery is running. Run bin/bookkeeper shell autorecovery -status. If it reports disabled, that is your root cause. AutoRecovery is enabled by default (autoRecoveryDaemonEnabled=true), but operators sometimes disable it during maintenance and forget to re-enable it.

3. Check the auditor is alive. The auditor is a single elected node. If it crashes, no new auditing happens, and under-replicated ledgers go undetected until a new auditor is elected. Check the auditor process health on the expected node.

4. Assess surviving bookie capacity. Count writable bookies (bookie_SERVER_STATUS == 1). If the number of writable bookies cannot satisfy your ensemble and write quorum requirements, AutoRecovery cannot place new replicas even if it is running. You need more bookies.

5. Check for recovery-induced I/O cascade. Look at bookie_journal_JOURNAL_SYNC_LATENCY P99 and bookkeeper_server_ADD_ENTRY_IN_PROGRESS on surviving bookies. If recovery I/O is saturating journal disks, surviving bookies may appear slow or time out, which can trigger further fencing and compound the problem. This is the Bookie Failure Cascade pattern.

6. Check for ghost ledgers. If the count plateaus at a non-zero value and never decreases, some ledgers in the under-replicated list may have already been deleted from BookKeeper but their metadata remains in the /underreplicated znode. The ReplicationWorker retries them indefinitely, generating excessive read requests against remaining bookies without making progress.

7. Check your Pulsar version. Pulsar 3.0.0 had a critical bug where the ReplicationWorker would not rereplicate under-replicated ledgers after the Auditor identified them. The ReplicationWorker waited indefinitely for incoming tasks. This was fixed in Pulsar 3.0.2, which upgraded BookKeeper to 4.16.3. If you are on 3.0.0 or 3.0.1, upgrade is the fix.

Metrics and signals to monitor

Fixes

AutoRecovery is disabled

Enable it. If AutoRecovery was disabled for maintenance and never re-enabled, under-replicated ledgers accumulate silently until a second bookie failure causes data loss. This is one of the most common causes of preventable data loss in Pulsar clusters.

# Enable AutoRecovery
bin/bookkeeper shell autorecovery -enable

Verify it is running with bin/bookkeeper shell autorecovery -status, then watch the under-replicated count trend downward.

Insufficient healthy bookies

AutoRecovery cannot rereplicate if there are not enough writable bookies to receive the data. The ensemble placement policy requires enough bookies in distinct failure domains (racks, zones) to satisfy your ensemble size and write quorum.

Add bookies to the cluster. After adding bookies, existing ledgers do not rebalance automatically. Only new ledgers will use the new bookies. The rereplication from AutoRecovery will place recovered copies on the new bookies, but pre-existing healthy ledgers stay where they are.

Recovery I/O cascade

If recovery is running but surviving bookies are degrading under the I/O load, consider increasing lostBookieRecoveryDelay to temporarily pause recovery and let foreground traffic stabilize. This is a judgment call: pausing recovery extends the data-loss vulnerability window but prevents a cascade that takes down more bookies.

# Set a delay (in seconds) before recovery starts after a bookie loss
bin/bookkeeper shell lostbookierecoverydelay -set 300

When surviving bookies are stable, reduce the delay back to its normal value. A production setting of 60 seconds is a common default to avoid unnecessary rereplication during rolling restarts while ensuring recovery proceeds promptly after real failures.

If recovery is consistently too slow, consider deploying dedicated AutoRecovery nodes instead of running the daemon embedded in bookies. This separates recovery I/O coordination from bookie write paths, though the actual rereplication reads and writes still go through the bookies.

Pulsar 3.0.0 ReplicationWorker bug

If you are running Pulsar 3.0.0 or 3.0.1, the ReplicationWorker does not pick up under-replicated ledgers from the Auditor’s work queue. The Auditor detects them, the count grows, and nothing happens. Upgrade to Pulsar 3.0.2 or later, which includes the BookKeeper 4.16.3 fix and multiple deadlock fixes in the auditor and ReplicationWorker.

Ghost ledgers

If ledgers in the under-replicated list have already been deleted from BookKeeper but their metadata persists in the /underreplicated znode, the ReplicationWorker retries them indefinitely. This wastes I/O and prevents the count from reaching zero. This requires manual cleanup of the stale znode entries. Coordinate with your team and the BookKeeper documentation before modifying ZooKeeper metadata directly, as incorrect cleanup can corrupt ledger state.

Metrics not exposed (Pulsar 4.0.10+)

If you have upgraded to Pulsar 4.0.10 or later and auditor metrics have disappeared, the BookKeeper Prometheus metrics provider class changed. The old default org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider was replaced with org.apache.pulsar.metrics.prometheus.bookkeeper.PrometheusMetricsProvider. Using the old class name results in no BookKeeper metrics being exposed, including all auditor metrics. Update your bookkeeper configuration to use the new class.

Prevention

• Set lostBookieRecoveryDelay to a positive value in production. A setting of 60 seconds prevents unnecessary rereplication during rolling restarts while ensuring recovery proceeds promptly after real failures.

• Monitor the under-replicated count as a first-class durability signal. It should be zero in steady state. Any sustained non-zero value after a bookie event that does not trend toward zero is a problem. Alert on count growing across consecutive audit intervals combined with healthy writable bookies dropping below quorum requirements.

• Verify AutoRecovery status after every maintenance window. Operators disable it during planned work and forget to re-enable it. This is the most common path to silent data loss.

• Keep enough bookie headroom. If your ensemble size is 3 with write quorum 2, you need at least 4-5 bookies to tolerate a failure during recovery. Tight bookie counts leave no room for rereplication.

• Separate journal and ledger storage disks on bookies. Recovery I/O hits the journal disk (for writes) and entry log disk (for reads of source data). If they share a disk, recovery amplifies foreground latency. This is the most common architecture mistake in Pulsar deployments.

• Test controlled bookie failures regularly. Know what your metrics look like during a planned failure. When a real failure happens, you need a baseline to distinguish normal recovery from a stalled one.

• Upgrade off Pulsar 3.0.0 and 3.0.1. The ReplicationWorker bug means AutoRecovery silently does nothing. There is no log error, no metric anomaly beyond the growing count. It is invisible until a second failure causes data loss.

How Netdata helps

• Per-second granularity on the under-replicated count lets you see the recovery trend in real time, not just at 15-second scrape intervals. The slope of the curve (flat, decreasing, or increasing) is the most important diagnostic signal, and per-second resolution makes the trend unambiguous.

• Correlation across the bookie fleet. When auditor_NUM_UNDER_REPLICATED_LEDGERS spikes, Netdata lets you overlay bookie_SERVER_STATUS, bookie_journal_JOURNAL_SYNC_LATENCY, and bookkeeper_server_ADD_ENTRY_IN_PROGRESS across all surviving bookies in a single view. This immediately reveals whether recovery is progressing, stalled, or causing a cascade.

• Anomaly detection on recovery signals. Anomaly detection flags unusual patterns in journal sync latency and add-entry queue depth on surviving bookies, catching recovery-induced I/O pressure before it cascades into a multi-bookie failure.

• Auditor-only metric scoping. Netdata’s labeling correctly associates auditor metrics with the elected auditor node, avoiding the common mistake of scraping the wrong bookie and seeing no data.

• Composite alerting. Netdata can correlate the under-replicated count with bookie availability, journal latency, and publish latency, so you get a single alert that says “recovery is failing and here is the I/O evidence” rather than isolated threshold breaches.

Related guides

• How Apache Pulsar actually works in production: a mental model for operators
• Apache Pulsar bookie read-only: disk full and bookie_SERVER_STATUS at zero
• Apache Pulsar bookie disk filling: runway to read-only and how to reclaim space
• Apache Pulsar bookie add-entry queue not draining: writes arriving faster than the disk can commit
• Apache Pulsar journal force write queue growing: the earliest write-saturation signal
• Apache Pulsar write stall: bookie journal fsync latency and the blocked write path
• Apache Pulsar bookie journal and ledger storage on one disk: the #1 architecture mistake
• Apache Pulsar broker down: telling a dead broker from a fenced one
• Apache Pulsar broker GC death spiral: heap pressure, stop-the-world pauses, and lost topic ownership
• Apache Pulsar entry log GC falling behind: reclaimed space that never comes back
• Apache Pulsar broker lookup failures: new clients cannot find their topic
• Apache Pulsar active connections climbing: connection leaks and file descriptor exhaustion