Apache Pulsar NotEnoughBookiesException: new ledgers cannot be created and writes fail

NotEnoughBookiesException (BookKeeper error code -6, surfaced as BKNotEnoughBookiesException) fires when the ensemble placement policy cannot satisfy the ensemble size (E) and write quorum (Qw) requirements for a new ledger. The broker requests a new ledger on size or time rollover, during topic recovery, and during compaction. If the policy cannot form a valid ensemble, ledger creation fails, the managed ledger has nowhere to write, and messages cannot be persisted durably.

This is a data-durability emergency. Any sustained occurrence means producers are blocking, timing out, or receiving errors. Page immediately.

“Enough bookies” does not mean a raw count. The placement policy considers rack-awareness and ensemble diversity. You can have 10 healthy bookies and still fail if your rack-aware policy requires bookies in 3 distinct racks but only 2 racks have writable bookies. Verify both the bookie count and the failure-domain topology before concluding that bookies are “up.”

What this means

The managed ledger cannot advance its write position. Producers see writes fail or hang. The topic is write-blocked until the constraint becomes satisfiable again.

If the old ledger was already fenced (sealed) before the new one could be created, the topic enters a broken state: it can neither write new data nor recover cleanly without intervention. The broker retries ledger creation on subsequent write attempts, flooding the log with repeated exceptions.

flowchart TD
    A["NotEnoughBookiesException"] --> B["Count writable bookies
grep bookie_SERVER_STATUS"]
    B --> C{"Enough bookies for
ensemble and quorum?"}
    C -->|"No"| D["Bookies down,
read-only, or unregistered"]
    C -->|"Yes"| E{"Enough distinct racks
for placement policy?"}
    E -->|"No"| F["Rack diversity failure:
check rack assignments"]
    E -->|"Yes"| G{"Metadata store
reachable?"}
    G -->|"No"| H["Metadata store down:
check ZK or Oxia"]
    G -->|"Yes"| I["Stuck placement:
broker restart may clear"]

Common causes

Quick checks

# Check for NotEnoughBookiesException in broker logs
grep "NotEnoughBookiesException\|BKException.*create" /var/log/pulsar/broker.log | tail -20

# Check bookie server status across all bookies
for bookie in bookie1 bookie2 bookie3 bookie4; do
  echo -n "$bookie: "
  curl -s http://$bookie:8000/metrics | grep bookie_SERVER_STATUS
done

# List writable and read-only bookies separately
bookkeeper shell listbookies -rw
bookkeeper shell listbookies -ro

# Count under-replicated ledgers (skip header line)
bookkeeper shell listunderreplicated 2>/dev/null | tail -n +2 | wc -l

# Check ZK connectivity from broker host.
# Note: 4lw commands must be whitelisted via 4lw.commands.whitelist in zoo.cfg (ZK 3.5+)
echo stat | nc <zk-host> 2181

# Check if publish rate has dropped (writes failing)
curl -s http://<broker-host>:8080/metrics | grep -E "pulsar_(rate|throughput)_in"

# Check bookie disk usage (approaching read-only threshold)
curl -s http://<bookie-host>:8000/metrics | grep -E "bookie_ledger_dir|bookie_ledger_writable"

How to diagnose it

1. Confirm the exception is current, not stale. Check broker logs for the timestamp of the most recent NotEnoughBookiesException. If it stopped, the condition may have self-resolved. If it is repeating, proceed.

2. Count writable bookies. Query bookie_SERVER_STATUS on every bookie. You need at least E writable bookies to create an ensemble. Bookies in read-only or unregistered state cannot participate in new ensembles. If multiple bookies are read-only, check disk usage: BookKeeper transitions to read-only at diskUsageThreshold (default 0.95). (bookie_SERVER_STATUS: 1 = writable, 0 = read-only, -1 = unregistered.)

3. Check rack diversity. If you have enough writable bookies but the exception persists, the placement policy may be failing on rack-awareness. Verify that bookies are distributed across enough distinct racks to satisfy bookkeeperClientMinNumRacksPerWriteQuorum (default 2). If bookkeeperClientEnforceMinNumRacksPerWriteQuorum is true, the policy fails closed rather than relaxing the rack constraint.

4. Verify metadata store connectivity. If ZooKeeper (or Oxia in newer Pulsar versions) is unavailable, the BookKeeper client cannot read bookie registration or rack topology. Run echo stat | nc <zk-host> 2181 from the broker host. A broker may report NotEnoughBookiesException even when all bookie processes are healthy, because it cannot discover them.

5. Check for recent restarts or maintenance. If the error appeared during a rolling upgrade, too many bookies may have been restarted simultaneously. The cluster needs E writable bookies available at all times. Even restarted bookies need time to register and replay their journals before they are usable; cold journal starts can take 5-10 minutes for large journals.

6. Check for version-specific rack mapping bugs. In some Pulsar 3.0.x versions, bookie rack information is lost on restart (reported on 3.0.6, issue #23282): after a bookie restarts its rack can be reported as /defaultRegion/defaultRack. PRs #18672 and #23331 improved rack-info propagation, but related follow-ups remain open. If the exception appears after bookie restarts and all other checks pass, verify the bookies’ registered rack info.

Metrics and signals to monitor

Fixes

Restore bookie availability

If bookies are crashed, restart them. If bookies are read-only due to disk pressure, add capacity or clean up data (check retention policies, compaction status, and backlog quotas). Re-registered bookies need time to become writable: journal replay on startup can take minutes depending on journal size.

If a bookie is permanently lost, decommission it gracefully so AutoRecovery redistributes its data to surviving bookies:

# Destructive: triggers rereplication, adds I/O load on surviving bookies.
# Confirm the bookie will not return before running.
bookkeeper shell decommissionbookie -bookieid <bookie-host>:<bookie-port>

Address rack diversity constraints

If the placement policy cannot find enough bookies in distinct racks, add bookies to underrepresented racks. If that is not immediately possible, you can temporarily set bookkeeperClientEnforceMinNumRacksPerWriteQuorum to false, which allows the policy to fall back to best-effort placement across whatever racks are available.

Tradeoff: Disabling rack enforcement reduces fault tolerance. If two bookies in the same rack participate in an ensemble and that rack fails, you can lose data. Use this only as a temporary measure during an active incident.

Restore metadata store connectivity

If ZK or Oxia is unavailable, fix the metadata store first. Check ZK ensemble quorum, ZK transaction log disk health, and network connectivity between brokers and ZK. Once the metadata store is reachable, the broker re-reads bookie registrations and rack topology on the next ledger creation attempt.

Do not restart brokers while ZK is recovering. The reconnection load adds pressure. Brokers reconnect on their own once ZK is healthy.

Emergency: reduce ensemble/quorum requirements

If you cannot restore enough bookies quickly and writes are blocked, you can temporarily lower managedLedgerDefaultEnsembleSize, managedLedgerDefaultWriteQuorum, and managedLedgerDefaultAckQuorum to values the cluster can satisfy. This requires a broker configuration change and restart.

Tradeoff: Lowering these values reduces data redundancy. E=1, Qw=1, Qa=1 means a single bookie failure causes data loss. Use only as a last resort during an active data-availability emergency, and restore the original values as soon as capacity returns.

Clear stuck placement state

If all bookies and the metadata store are healthy but the exception persists, the broker’s cached placement state or ensemble-change loop may be stuck. Restarting the affected broker clears the cache and forces a fresh topology read. This causes a brief bundle ownership transfer; clients reconnect automatically. Restarting is not guaranteed to clear the state: issue #23807 documents the error persisting after restarts with all bookies healthy, so also look for a deeper cause (stale rack info, metadata store inconsistency).

Prevention

Stagger rolling restarts. Never restart enough bookies to drop below E writable bookies. Account for rack diversity: if all bookies in one rack restart, ensure other racks still have enough bookies for the placement policy. Parallel bookie restarts during upgrades are the most common trigger for this exception.

Set lostBookieRecoveryDelay during maintenance. Increase lostBookieRecoveryDelay to 60 seconds or more during planned bookie restarts. This prevents AutoRecovery from triggering unnecessary rereplication while bookies are briefly down, which adds I/O load to surviving bookies.

Alert on bookie_SERVER_STATUS transitions. Alert on any bookie transitioning to read-only or unregistered before it reduces available bookies below the ensemble requirement.

Track rack distribution. Ensure bookie rack assignments match your physical topology and that each rack has enough bookies to satisfy bookkeeperClientMinNumRacksPerWriteQuorum.

Verify ensemble/quorum against minimum cluster size. Ensure E and Qw never exceed your minimum planned bookie count, including during maintenance windows. The Apache Pulsar Helm chart defaults these values to 2 (matching broker.conf); an explicit E=1/Qw=1/Qa=1 provides no redundancy and a single bookie failure causes data loss. Verify your actual broker.conf values, not just chart defaults.

Watch under-replicated ledger trends. A growing auditor_NUM_UNDER_REPLICATED_LEDGERS count indicates recovery is failing or bookies are failing faster than AutoRecovery can keep up.

Watch for direct memory pressure on sustained exceptions. Repeated NotEnoughBookiesException can trigger ensemble-change retries that hold Netty direct memory buffers on the broker. If the exception persists for extended periods, monitor broker direct memory to avoid a secondary OutOfDirectMemoryError crash. The underlying leak from repeated ensemble-change retries (issue #16998) is still open, so this monitoring advice stands.

How Netdata helps

• Per-second bookie_SERVER_STATUS across all bookies reveals which bookies are writable, read-only, or unregistered the moment the exception fires, without manual curl loops.
• Correlating bookie process health, disk usage trends, and bookie_SERVER_STATUS on a single timeline shows whether the root cause is crashed bookies, disk-full read-only transitions, or registration issues.
• ML anomaly detection on pulsar_rate_in and pulsar_broker_publish_latency surfaces write-path impact before producers time out.
• Under-replicated ledger count tracking on the auditor node provides early warning when data durability is at risk.
• Metadata store latency monitoring catches ZK or Oxia degradation before it manifests as NotEnoughBookiesException.
• Per-second resolution lets you see exactly when a bookie went down relative to when the exception started, critical for distinguishing rolling-restart timing from a real outage.

Related guides

• How Apache Pulsar actually works in production: a mental model for operators
• Apache Pulsar bookie add-entry queue not draining: writes arriving faster than the disk can commit
• Apache Pulsar broker down: telling a dead broker from a fenced one
• Apache Pulsar broker lookup failures: new clients cannot find their topic
• Apache Pulsar monitoring checklist: the signals every production cluster needs
• Apache Pulsar OutOfDirectMemoryError: the off-heap crash JVM heap dashboards never show