Under-replicated ledgers are the durability canary for your Pulsar cluster. When a bookie fails, ledger fragments on it lose redundancy and the under-replicated count spikes. AutoRecovery is supposed to drive that count back to zero. When the count stays flat or grows instead of trending down, recovery has stalled.
A stall is not slow recovery. Slow recovery means the count is decreasing, just not fast enough. A stall means it is frozen or climbing. The diagnostic paths and fixes are completely different.
The dominant stall patterns: AutoRecovery is disabled (set during maintenance and never reverted), the Replication Worker is deadlocked by a version-specific bug, there are not enough healthy bookies to receive recovered data, or throughput limits are so conservative that the effective recovery rate rounds to zero. Each has a distinct signature.
What this means
AutoRecovery has two cooperating components that must both be functional.
The Auditor. One elected bookie (or a dedicated recovery node) periodically scans for ledgers that have lost replicas and marks them as under-replicated in ZooKeeper. The auditor runs on a single elected leader. If that node dies, a new auditor must be elected before scanning resumes. During the election gap, no new under-replicated ledgers are discovered and no new recovery tasks are created. The default periodic bookie check interval is 24 hours (auditorPeriodicBookieCheckInterval), and the full ledger scan runs every 7 days (auditorPeriodicCheckInterval). Between scheduled scans, the auditor reacts to explicit bookie-failure events via ZooKeeper watches.
The Replication Worker. Reads the under-replicated ledger list from ZooKeeper and copies missing fragments to healthy bookies. On failure, the worker backs off for 5 seconds (rwRereplicateBackoffMs) before retrying. After a failed attempt, the lock on that ledger is held for a grace period of 5 minutes (lockReleaseOfFailedLedgerGracePeriod) before another worker can pick it up. A single problematic ledger can block progress for its full lock duration on each retry cycle.
Both components must be running, connected to ZooKeeper, and able to find healthy target bookies that satisfy your ensemble and rack-awareness constraints. If any link breaks, the under-replicated count freezes.
Recovery prioritizes open (active) ledgers over closed ones. If your stall involves only closed ledgers from a bookie that failed days ago, recovery may be working but deprioritizing them behind active fragments.
flowchart TD
A["UR count flat or growing"] --> B{"Recovery enabled?"}
B -- No --> C["Enable autorecovery"]
B -- Yes --> D{"Auditor alive?"}
D -- No --> E["Fix auditor election or ZK"]
D -- Yes --> F{"Worker active?"}
F -- No --> G["Check BK version for deadlock"]
F -- Yes --> H{"Enough writable bookies?"}
H -- No --> I["Add bookies or free disk"]
H -- Yes --> J["Check throttling or stuck ledgers"]Common causes
| Cause | What it looks like | First thing to check |
|---|---|---|
| AutoRecovery disabled | Count grows monotonically after bookie loss, zero recovery I/O on surviving bookies | autoRecoveryDaemonEnabled in bookkeeper.conf |
| Auditor election gap | Count flat after the auditor node restarted or crashed, no new recovery tasks in logs | Which bookie is the auditor? Is it alive and connected to ZK? |
| Replication Worker deadlock | Auditor lists under-replicated ledgers but worker shows no replication activity | Pulsar/BookKeeper version against known deadlock fixes |
| Insufficient healthy bookies | NotEnoughBookiesException in logs, count flat | Count writable bookies versus ensemble and quorum requirements |
| Rack-awareness constraint | Recovery cannot place fragments despite bookies being available | Check rack distribution and placement policy settings |
| Throughput throttled too low | Count decreasing imperceptibly, recovery I/O near zero on surviving bookies | Replication throughput configuration |
| Deleted ledgers stuck in queue | Count flat, excessive read traffic on healthy bookies with BKNotEnoughBookiesException on non-existent ledgers | Check for ledger IDs that no longer exist on any bookie |
Quick checks
Run these read-only checks to narrow the diagnosis.
# Confirm the stall: sample count twice, 15 minutes apart.
# If both values are identical or the second is higher, recovery is stalled.
bin/bookkeeper shell listunderreplicated | wc -l
# Under-replicated ledger count from the auditor's metrics endpoint
curl -s http://<auditor-host>:8000/metrics | grep auditor_NUM_UNDER_REPLICATED_LEDGERS
# Total bytes of under-replicated data (context for prioritization)
curl -s http://<auditor-host>:8000/metrics | grep auditor_UNDER_REPLICATED_LEDGERS_TOTAL_SIZE
# Bookie server status: 1=writable, 0=read-only, -1=unregistered.
# Run on each bookie to count how many can accept recovered writes.
curl -s http://<bookie-host>:8000/metrics | grep bookie_SERVER_STATUS
# Search for bookie placement failures in broker and recovery logs.
# Log paths vary by deployment; adjust accordingly.
grep -c "NotEnoughBookiesException" /var/log/pulsar/broker.log
# Verify ZooKeeper is responsive from the auditor node.
# Requires the 4lw.whitelist to include "ruok" in newer ZK versions.
echo ruok | nc <zk-host> 2181
How to diagnose it
Confirm it is a stall, not slow recovery. Sample
listunderreplicated | wc -ltwice, 15 minutes apart. If the count decreased, recovery is working but slow. If it is identical or higher, recovery is stalled.Verify AutoRecovery is enabled. Check
autoRecoveryDaemonEnabledinbookkeeper.conf. This istrueby default, but operators sometimes set it tofalseduring maintenance and forget to revert. If disabled, re-enable withbin/bookkeeper shell autorecovery -enable.Identify the auditor node and verify it is alive. The auditor is a single elected bookie or dedicated recovery node. If that process has died or lost its ZooKeeper session, no new recovery tasks are being generated. Check the auditor metrics endpoint at
http://<auditor-host>:8000/metrics. Ifauditor_NUM_UNDER_REPLICATED_LEDGERSis not exposed at all, the auditor may not be running on that node.Check Replication Worker activity. The auditor marks ledgers as under-replicated, but the worker does the actual replication. If the auditor is listing ledgers but the worker is not replicating them, you may be hitting a version-specific deadlock. Look for replication-related log entries on the recovery node.
Verify enough healthy bookies exist. Count the number of writable bookies (
bookie_SERVER_STATUS == 1). If the number of writable bookies is less than your ensemble size or write quorum, recovery cannot place new copies. Also check rack-awareness constraints: ifminNumRacksPerWriteQuorumis enforced, you need enough bookies in distinct racks.Check for
NotEnoughBookiesException. This error in broker or recovery logs means BookKeeper cannot find enough bookies to satisfy the ensemble and placement requirements. The fix is to add bookies or address why existing ones are read-only or unregistered.Check your Pulsar and BookKeeper version. Several AutoRecovery deadlock bugs existed in older releases. If you are running an affected version, the fix is an upgrade.
Check for deleted ledgers stuck in the under-replicated queue. If a ledger was deleted from bookies but still exists in the under-replicated metadata in ZooKeeper, the Replication Worker retries it indefinitely. This generates excessive read traffic on healthy bookies and can block recovery of other ledgers.
Metrics and signals to monitor
| Signal | Why it matters | Warning sign |
|---|---|---|
auditor_NUM_UNDER_REPLICATED_LEDGERS | Primary count of ledgers missing replicas, exposed on the auditor node | Flat or growing after a bookie failure event |
auditor_UNDER_REPLICATED_LEDGERS_TOTAL_SIZE | Total bytes of under-replicated data per audit run | Growing indicates recovery is falling behind even if count looks stable |
bookie_SERVER_STATUS | Whether bookies can accept recovered writes | Multiple bookies at 0 (read-only) or -1 (unregistered) |
| ZooKeeper request latency | Both auditor scanning and worker coordination depend on ZK | Sustained average above 50ms slows the entire recovery pipeline |
| Bookie journal sync latency on surviving bookies | Recovery I/O competes with foreground write traffic | Spikes indicate recovery is active and competing for disk |
bookkeeper_server_ADD_ENTRY_IN_PROGRESS on surviving bookies | Shows write pressure including recovery writes | Growth during recovery may indicate throughput competition |
Fixes
AutoRecovery is disabled
If autoRecoveryDaemonEnabled is false, re-enable it:
# Re-enable AutoRecovery cluster-wide
bin/bookkeeper shell autorecovery -enable
After re-enabling, the auditor picks up the under-replicated ledger list on its next scan cycle or bookie-failure event.
Version-specific Replication Worker deadlock
Several AutoRecovery bugs cause the Replication Worker to silently fail to pick up under-replicated ledgers while the auditor correctly identifies them:
Pulsar before 3.0.2 / BookKeeper before 4.16.3. The Replication Worker fails to find under-replicated ledgers even though the auditor lists them. The auditor shows a growing count, but no replication activity occurs. Fixed in Pulsar 3.0.2, which ships BookKeeper 4.16.3. Restarting the recovery node can trigger a one-time replication pass, but the stall recurs.
Pulsar before 2.11.3. Three separate deadlock bugs affect AutoRecovery: auditor deadlock,
PulsarLedgerUnderreplicationManagernotify failure, and metadata store deadlock fromBookkeeperInternalCallbacks.Processor. All three are fixed in Pulsar 2.11.3 and backported to 3.0.2.Pulsar before 3.0.2. Deleted ledgers that remain in the under-replicated ZooKeeper metadata cause the Replication Worker to retry indefinitely with
BKNotEnoughBookiesException. This generates excessive read traffic on healthy bookies and can block recovery of other ledgers. Fixed in Pulsar 3.0.2.
If you are on an affected version, upgrading is the permanent fix. As a temporary workaround, restarting the recovery node can trigger a one-time replication pass for ledgers the worker can process.
Insufficient healthy bookies
Recovery requires target bookies that are writable (bookie_SERVER_STATUS == 1) and satisfy your ensemble, write quorum, and rack-awareness constraints. If bookies are read-only due to disk pressure, they cannot receive recovered data.
Address the root cause:
- Free disk space on read-only bookies (see the bookie disk filling guide).
- Add new bookies to the cluster.
- Verify rack distribution if rack-aware placement is enforced.
Adding new bookies does not automatically rebalance existing ledgers. Only new ledgers use the new bookies. Recovery of under-replicated ledgers will target the new bookies, but fully healthy ledgers on the old bookies stay where they are.
Recovery throughput too conservative or too aggressive
Recovery I/O competes with foreground write traffic on surviving bookies. BookKeeper throttles replication with settings such as rereplicationEntryBatchSize and auditor concurrency limits (verify the exact knobs for your version) to prevent recovery from overwhelming the cluster, but if set too conservatively, recovery effectively stalls.
If recovery is too aggressive, it can trigger a cascade: recovery I/O stresses surviving bookies, causing timeouts that make them appear failed, triggering more recovery. Use lostBookieRecoveryDelay (default 0, meaning immediate recovery on bookie failure) to add a delay during rolling restarts or Kubernetes pod reschedules. Setting it to 60 seconds or more prevents unnecessary rereplication when a bookie will come back shortly.
Manual recovery
When automated recovery is stuck and you need to force replication of specific ledgers or a specific failed bookie’s data:
# WARNING: resource-intensive. Competes with foreground I/O and can
# stress surviving bookies to the point of triggering a cascade.
# Recover all ledgers from a specific failed bookie
bin/bookkeeper shell recover <bookie-ip>:<port>
# Recover a specific ledger
bin/bookkeeper shell recover <bookie-ip>:<port> --ledger <ledgerID>
Manual recovery bypasses the auditor and Replication Worker pipeline. It directly reads the source fragments and writes new copies to healthy bookies. Use it when the automated pipeline is broken, but monitor surviving bookie I/O closely.
Prevention
Set
lostBookieRecoveryDelayabove zero. The default of 0 triggers recovery immediately on any bookie failure. During rolling restarts or Kubernetes pod evictions, this causes unnecessary rereplication. Set it to 60 seconds or higher to give transient failures time to self-resolve.Upgrade past Pulsar 3.0.2 / BookKeeper 4.16.3. Multiple AutoRecovery deadlock and stall bugs are fixed in these versions. An unexplained recovery stall on an older release may be a known bug rather than a configuration issue.
Alert on the under-replicated ledger count as a trend, not a threshold. A flat or growing count after a bookie event is the signal that recovery is stalled. Alert on sustained non-zero counts lasting more than 15 minutes after a bookie event.
Verify AutoRecovery status after maintenance. If you disable AutoRecovery during maintenance, add re-enabling it to your runbook checklist before closing the change.
Run dedicated AutoRecovery nodes in production. When AutoRecovery runs embedded on bookies, recovery I/O competes with the write path. Dedicated recovery nodes isolate this impact during large recovery operations.
Plan for closed-ledger latency. Recovery prioritizes open ledgers. Closed ledgers from a failed bookie may stay under-replicated for hours or days depending on recovery throughput and data volume. This is expected behavior, not necessarily a stall.
How Netdata helps
Per-second collection of
auditor_NUM_UNDER_REPLICATED_LEDGERS. Per-second granularity makes the difference between a flat line (stall) and a slow decline (working but slow) immediately visible, without waiting for manual samples 15 minutes apart.Correlation between under-replicated count and
bookie_SERVER_STATUSchanges. When a bookie goes read-only or unregistered, the status change and the under-replicated count spike appear on the same timeline.ZooKeeper latency alongside recovery metrics. Both the auditor and Replication Worker depend on ZooKeeper. Correlating ZK latency spikes with recovery stalls pinpoints coordination-layer bottlenecks.
Journal sync latency on surviving bookies during recovery. Per-bookie journal sync latency shows whether recovery is stressing the write path or whether the write path is starving recovery.
Anomaly detection on the recovery trend. Anomaly detection flags when the under-replicated count deviates from the expected post-failure recovery curve, catching stalls earlier than static threshold alerts.
Related guides
- Apache Pulsar active connections climbing: connection leaks and file descriptor exhaustion
- Apache Pulsar bookie add-entry queue not draining: writes arriving faster than the disk can commit
- Apache Pulsar bookie disk filling: runway to read-only and how to reclaim space
- Apache Pulsar bookie read latency high: catch-up reads competing with the write path
- Apache Pulsar bookie read-only: disk full and bookie_SERVER_STATUS at zero
- Apache Pulsar broker down: telling a dead broker from a fenced one
- Apache Pulsar broker GC death spiral: heap pressure, stop-the-world pauses, and lost topic ownership
- Apache Pulsar broker lookup failures: new clients cannot find their topic
- Apache Pulsar entry log GC falling behind: reclaimed space that never comes back
- How Apache Pulsar actually works in production: a mental model for operators
- Apache Pulsar journal force write queue growing: the earliest write-saturation signal
- Apache Pulsar bookie journal and ledger storage on one disk: the #1 architecture mistake