$ guides / apache-pulsar / apache-pulsar-autorecovery-stalled ▌

Operations Guides

Apache Pulsar AutoRecovery stalled: under-replicated ledgers that never heal

Under-replicated ledgers are the durability canary for your Pulsar cluster. When a bookie fails, ledger fragments on it lose redundancy and the under-replicated count spikes. AutoRecovery is supposed to drive that count back to zero. When the count stays flat or grows instead of trending down, recovery has stalled.

A stall is not slow recovery. Slow recovery means the count is decreasing, just not fast enough. A stall means it is frozen or climbing. The diagnostic paths and fixes are completely different.

The dominant stall patterns: AutoRecovery is disabled (set during maintenance and never reverted), the Replication Worker is deadlocked by a version-specific bug, there are not enough healthy bookies to receive recovered data, or throughput limits are so conservative that the effective recovery rate rounds to zero. Each has a distinct signature.

What this means

AutoRecovery has two cooperating components that must both be functional.

The Auditor. One elected bookie (or a dedicated recovery node) periodically scans for ledgers that have lost replicas and marks them as under-replicated in ZooKeeper. The auditor runs on a single elected leader. If that node dies, a new auditor must be elected before scanning resumes. During the election gap, no new under-replicated ledgers are discovered and no new recovery tasks are created. The default periodic bookie check interval is 24 hours (auditorPeriodicBookieCheckInterval), and the full ledger scan runs every 7 days (auditorPeriodicCheckInterval). Between scheduled scans, the auditor reacts to explicit bookie-failure events via ZooKeeper watches.

The Replication Worker. Reads the under-replicated ledger list from ZooKeeper and copies missing fragments to healthy bookies. On failure, the worker backs off for 5 seconds (rwRereplicateBackoffMs) before retrying. After a failed attempt, the lock on that ledger is held for a grace period of 5 minutes (lockReleaseOfFailedLedgerGracePeriod) before another worker can pick it up. A single problematic ledger can block progress for its full lock duration on each retry cycle.

Both components must be running, connected to ZooKeeper, and able to find healthy target bookies that satisfy your ensemble and rack-awareness constraints. If any link breaks, the under-replicated count freezes.

Recovery prioritizes open (active) ledgers over closed ones. If your stall involves only closed ledgers from a bookie that failed days ago, recovery may be working but deprioritizing them behind active fragments.

flowchart TD
    A["UR count flat or growing"] --> B{"Recovery enabled?"}
    B -- No --> C["Enable autorecovery"]
    B -- Yes --> D{"Auditor alive?"}
    D -- No --> E["Fix auditor election or ZK"]
    D -- Yes --> F{"Worker active?"}
    F -- No --> G["Check BK version for deadlock"]
    F -- Yes --> H{"Enough writable bookies?"}
    H -- No --> I["Add bookies or free disk"]
    H -- Yes --> J["Check throttling or stuck ledgers"]

Common causes

Cause	What it looks like	First thing to check
AutoRecovery disabled	Count grows monotonically after bookie loss, zero recovery I/O on surviving bookies	`autoRecoveryDaemonEnabled` in bookkeeper.conf
Auditor election gap	Count flat after the auditor node restarted or crashed, no new recovery tasks in logs	Which bookie is the auditor? Is it alive and connected to ZK?
Replication Worker deadlock	Auditor lists under-replicated ledgers but worker shows no replication activity	Pulsar/BookKeeper version against known deadlock fixes
Insufficient healthy bookies	`NotEnoughBookiesException` in logs, count flat	Count writable bookies versus ensemble and quorum requirements
Rack-awareness constraint	Recovery cannot place fragments despite bookies being available	Check rack distribution and placement policy settings
Throughput throttled too low	Count decreasing imperceptibly, recovery I/O near zero on surviving bookies	Replication throughput configuration
Deleted ledgers stuck in queue	Count flat, excessive read traffic on healthy bookies with `BKNotEnoughBookiesException` on non-existent ledgers	Check for ledger IDs that no longer exist on any bookie

Quick checks

Run these read-only checks to narrow the diagnosis.

# Confirm the stall: sample count twice, 15 minutes apart.
# If both values are identical or the second is higher, recovery is stalled.
bin/bookkeeper shell listunderreplicated | wc -l

# Under-replicated ledger count from the auditor's metrics endpoint
curl -s http://<auditor-host>:8000/metrics | grep auditor_NUM_UNDER_REPLICATED_LEDGERS

# Total bytes of under-replicated data (context for prioritization)
curl -s http://<auditor-host>:8000/metrics | grep auditor_UNDER_REPLICATED_LEDGERS_TOTAL_SIZE

# Bookie server status: 1=writable, 0=read-only, -1=unregistered.
# Run on each bookie to count how many can accept recovered writes.
curl -s http://<bookie-host>:8000/metrics | grep bookie_SERVER_STATUS

# Search for bookie placement failures in broker and recovery logs.
# Log paths vary by deployment; adjust accordingly.
grep -c "NotEnoughBookiesException" /var/log/pulsar/broker.log

# Verify ZooKeeper is responsive from the auditor node.
# Requires the 4lw.whitelist to include "ruok" in newer ZK versions.
echo ruok | nc <zk-host> 2181

How to diagnose it

Confirm it is a stall, not slow recovery. Sample listunderreplicated | wc -l twice, 15 minutes apart. If the count decreased, recovery is working but slow. If it is identical or higher, recovery is stalled.
Verify AutoRecovery is enabled. Check autoRecoveryDaemonEnabled in bookkeeper.conf. This is true by default, but operators sometimes set it to false during maintenance and forget to revert. If disabled, re-enable with bin/bookkeeper shell autorecovery -enable.
Identify the auditor node and verify it is alive. The auditor is a single elected bookie or dedicated recovery node. If that process has died or lost its ZooKeeper session, no new recovery tasks are being generated. Check the auditor metrics endpoint at http://<auditor-host>:8000/metrics. If auditor_NUM_UNDER_REPLICATED_LEDGERS is not exposed at all, the auditor may not be running on that node.
Check Replication Worker activity. The auditor marks ledgers as under-replicated, but the worker does the actual replication. If the auditor is listing ledgers but the worker is not replicating them, you may be hitting a version-specific deadlock. Look for replication-related log entries on the recovery node.
Verify enough healthy bookies exist. Count the number of writable bookies (bookie_SERVER_STATUS == 1). If the number of writable bookies is less than your ensemble size or write quorum, recovery cannot place new copies. Also check rack-awareness constraints: if minNumRacksPerWriteQuorum is enforced, you need enough bookies in distinct racks.
Check for NotEnoughBookiesException. This error in broker or recovery logs means BookKeeper cannot find enough bookies to satisfy the ensemble and placement requirements. The fix is to add bookies or address why existing ones are read-only or unregistered.
Check your Pulsar and BookKeeper version. Several AutoRecovery deadlock bugs existed in older releases. If you are running an affected version, the fix is an upgrade.
Check for deleted ledgers stuck in the under-replicated queue. If a ledger was deleted from bookies but still exists in the under-replicated metadata in ZooKeeper, the Replication Worker retries it indefinitely. This generates excessive read traffic on healthy bookies and can block recovery of other ledgers.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`auditor_NUM_UNDER_REPLICATED_LEDGERS`	Primary count of ledgers missing replicas, exposed on the auditor node	Flat or growing after a bookie failure event
`auditor_UNDER_REPLICATED_LEDGERS_TOTAL_SIZE`	Total bytes of under-replicated data per audit run	Growing indicates recovery is falling behind even if count looks stable
`bookie_SERVER_STATUS`	Whether bookies can accept recovered writes	Multiple bookies at 0 (read-only) or -1 (unregistered)
ZooKeeper request latency	Both auditor scanning and worker coordination depend on ZK	Sustained average above 50ms slows the entire recovery pipeline
Bookie journal sync latency on surviving bookies	Recovery I/O competes with foreground write traffic	Spikes indicate recovery is active and competing for disk
`bookkeeper_server_ADD_ENTRY_IN_PROGRESS` on surviving bookies	Shows write pressure including recovery writes	Growth during recovery may indicate throughput competition

Fixes

AutoRecovery is disabled

If autoRecoveryDaemonEnabled is false, re-enable it:

# Re-enable AutoRecovery cluster-wide
bin/bookkeeper shell autorecovery -enable

After re-enabling, the auditor picks up the under-replicated ledger list on its next scan cycle or bookie-failure event.

Version-specific Replication Worker deadlock

Several AutoRecovery bugs cause the Replication Worker to silently fail to pick up under-replicated ledgers while the auditor correctly identifies them:

Pulsar before 3.0.2 / BookKeeper before 4.16.3. The Replication Worker fails to find under-replicated ledgers even though the auditor lists them. The auditor shows a growing count, but no replication activity occurs. Fixed in BookKeeper 4.16.3, shipped with Pulsar 3.0.2. Restarting the recovery node can trigger a one-time replication pass, but the stall recurs.
Pulsar before 2.11.3. Three separate deadlock bugs affect AutoRecovery: auditor deadlock, PulsarLedgerUnderreplicationManager notify failure, and metadata store deadlock from BookkeeperInternalCallbacks.Processor. All three are fixed in Pulsar 2.11.3 and backported to 3.0.2.
Pulsar before 3.0.2. Deleted ledgers that remain in the under-replicated ZooKeeper metadata cause the Replication Worker to retry indefinitely with BKNotEnoughBookiesException. This generates excessive read traffic on healthy bookies and can block recovery of other ledgers. Fixed in Pulsar 3.0.2.

If you are on an affected version, upgrading is the permanent fix. As a temporary workaround, restarting the recovery node can trigger a one-time replication pass for ledgers the worker can process.

Insufficient healthy bookies

Recovery requires target bookies that are writable (bookie_SERVER_STATUS == 1) and satisfy your ensemble, write quorum, and rack-awareness constraints. If bookies are read-only due to disk pressure, they cannot receive recovered data.

Address the root cause:

Free disk space on read-only bookies (see the bookie disk filling guide).
Add new bookies to the cluster.
Verify rack distribution if rack-aware placement is enforced.

Adding new bookies does not automatically rebalance existing ledgers. Only new ledgers use the new bookies. Recovery of under-replicated ledgers will target the new bookies, but fully healthy ledgers on the old bookies stay where they are.

Recovery throughput too conservative or too aggressive

Recovery I/O competes with foreground write traffic on surviving bookies. BookKeeper has configurable throughput limits to prevent recovery from overwhelming the cluster, but if set too conservatively, recovery effectively stalls.

If recovery is too aggressive, it can trigger a cascade: recovery I/O stresses surviving bookies, causing timeouts that make them appear failed, triggering more recovery. Use lostBookieRecoveryDelay (default 0, meaning immediate recovery on bookie failure) to add a delay during rolling restarts or Kubernetes pod reschedules. Setting it to 60 seconds or more prevents unnecessary rereplication when a bookie will come back shortly.

Manual recovery

When automated recovery is stuck and you need to force replication of specific ledgers or a specific failed bookie’s data:

# WARNING: resource-intensive. Competes with foreground I/O and can
# stress surviving bookies to the point of triggering a cascade.

# Recover all ledgers from a specific failed bookie
bin/bookkeeper shell recover <bookie-ip>:<port>

# Recover a specific ledger
bin/bookkeeper shell recover <bookie-ip>:<port> --ledger <ledgerID>

Manual recovery bypasses the auditor and Replication Worker pipeline. It directly reads the source fragments and writes new copies to healthy bookies. Use it when the automated pipeline is broken, but monitor surviving bookie I/O closely.

Prevention

Set lostBookieRecoveryDelay above zero. The default of 0 triggers recovery immediately on any bookie failure. During rolling restarts or Kubernetes pod evictions, this causes unnecessary rereplication. Set it to 60 seconds or higher to give transient failures time to self-resolve.
Upgrade past Pulsar 3.0.2 / BookKeeper 4.16.3. Multiple AutoRecovery deadlock and stall bugs are fixed in these versions. An unexplained recovery stall on an older release may be a known bug rather than a configuration issue.
Alert on the under-replicated ledger count as a trend, not a threshold. A flat or growing count after a bookie event is the signal that recovery is stalled. Alert on sustained non-zero counts lasting more than 15 minutes after a bookie event.
Verify AutoRecovery status after maintenance. If you disable AutoRecovery during maintenance, add re-enabling it to your runbook checklist before closing the change.
Run dedicated AutoRecovery nodes in production. When AutoRecovery runs embedded on bookies, recovery I/O competes with the write path. Dedicated recovery nodes isolate this impact during large recovery operations.
Plan for closed-ledger latency. Recovery prioritizes open ledgers. Closed ledgers from a failed bookie may stay under-replicated for hours or days depending on recovery throughput and data volume. This is expected behavior, not necessarily a stall.

How Netdata helps

Per-second collection of auditor_NUM_UNDER_REPLICATED_LEDGERS. Per-second granularity makes the difference between a flat line (stall) and a slow decline (working but slow) immediately visible, without waiting for manual samples 15 minutes apart.
Correlation between under-replicated count and bookie_SERVER_STATUS changes. When a bookie goes read-only or unregistered, the status change and the under-replicated count spike appear on the same timeline.
ZooKeeper latency alongside recovery metrics. Both the auditor and Replication Worker depend on ZooKeeper. Correlating ZK latency spikes with recovery stalls pinpoints coordination-layer bottlenecks.
Journal sync latency on surviving bookies during recovery. Per-bookie journal sync latency shows whether recovery is stressing the write path or whether the write path is starving recovery.
Anomaly detection on the recovery trend. Anomaly detection flags when the under-replicated count deviates from the expected post-failure recovery curve, catching stalls earlier than static threshold alerts.