Apache Pulsar LedgerFencedException: split ownership and fencing loops

LedgerFencedException appears when a broker tries to write to a BookKeeper ledger that another broker has already fenced. The error string is explicit: “Ledger has been fenced off. Some other client must have opened it to read.”

In most production environments, this is expected noise during planned failover. It becomes a problem when fencing occurs without a corresponding bundle transfer, or when two brokers fence each other’s ledgers in a loop.

Normal fencing correlates 1:1 with bundle unload events and adds a few hundred milliseconds of first-message latency during bundle unload. Abnormal fencing does not. The worst-case is not the fencing itself but what follows: if the new ledger cannot be created because there are not enough healthy bookies (NotEnoughBookiesException), the topic enters a broken state and producers cannot write.

What this means

Fencing is BookKeeper’s mechanism to guarantee single-writer semantics on a ledger. When a new broker takes ownership of a topic (after a bundle unload, broker restart, or failover), it opens the existing ledger in recovery mode. This sends fencing reads to all bookies in the current fragment ensemble. Once (ensemble - ackQuorum + 1) bookies acknowledge the fencing, the ledger is sealed. The previous owner can no longer write to it.

The new owner then creates a fresh ledger and continues accepting writes. In broker logs:

Creating a new ledger after closed <ledger-id>
Created new ledger <new-ledger-id>

This is the normal fencing lifecycle. The cost is metadata operations (ledger creation against ZooKeeper) and a brief latency spike for the first message written to the new ledger, typically under one second.

The diagram below shows the diagnostic decision tree for LedgerFencedException:

flowchart TD
    A["LedgerFencedException in broker logs"] --> B{"Correlates with
bundle unload?"}
    B -->|Yes| C["Normal failover path"]
    C --> D["Seal old ledger,
open new ledger"]
    D --> E["Recovery complete"]
    B -->|No, isolated| F["Check ZK session
and BadVersionException"]
    F --> G{"New ledger
creation fails?"}
    G -->|Yes| H["NotEnoughBookiesException
Topic broken state"]
    G -->|No| I["Monitor for recurrence"]
    B -->|No, looping| J["Topic Ownership
Oscillation"]
    J --> K["Disable load balancer,
manually assign bundle"]

When fencing occurs without a matching bundle transfer, something is causing ownership confusion. Common triggers are ZooKeeper session instability, load balancer thrashing, or a BadVersionException on managed ledger metadata (which, since Pulsar 2.11.1, backported to 2.10.2, 2.9.4, and 2.8.5 via PR #17736, forces the managed ledger to a FENCED state as a split-brain guard).

When fencing loops (broker A fences broker B’s ledgers, then broker B fences broker A’s ledgers, repeatedly), you have Topic Ownership Oscillation. Two brokers are competing for the same bundle, and each transfer triggers a fencing cycle.

Common causes

Quick checks

# Count fencing events in broker logs
grep -c "LedgerFencedException" /var/log/pulsar/broker.log

# Check bundle unload rate: fencing should correlate 1:1 with unloads
curl -s http://<broker>:8080/metrics | grep pulsar_lb_unload_bundle_total

# Check broker-to-ZK session state from each affected broker host (not a Prometheus metric)
echo "stat" | nc <zk-host> 2181

# Check for NotEnoughBookiesException (topic broken state)
grep "NotEnoughBookiesException" /var/log/pulsar/broker.log

# Check for BadVersionException (split-brain guard trigger)
grep -E "BadVersionException|Metadata-BadVersion" /var/log/pulsar/broker.log

# Count writable bookies: need enough for ensemble/quorum
curl -s http://<bookie>:8000/metrics | grep bookie_SERVER_STATUS

# Check ownership transitions in broker logs
grep -E "acquired ownership|released ownership" /var/log/pulsar/broker.log | tail -50

# List available bookies via BookKeeper shell
bin/bookkeeper shell listbookies

How to diagnose it

1. Determine whether the fencing is expected. Pull the bundle unload counter (pulsar_lb_unload_bundle_total) for the same time window as the fencing events. If every LedgerFencedException has a matching bundle unload, and the logs show “Created new ledger” shortly after, this is normal failover. No action needed beyond monitoring first-message latency.

2. If fencing is unexpected, check for ownership oscillation. Look at broker logs for rapid cycling of “acquired ownership” and “released ownership” messages for the same bundle. If two brokers are trading ownership back and forth within minutes, you have Topic Ownership Oscillation. The load balancer is the root cause.

3. Check ZooKeeper health. Check broker-to-ZK session state from each affected broker (echo "stat" | nc <zk-host> 2181). If sessions are dropping, fencing is a downstream symptom of ZK instability. Address ZK latency or session timeout first. Do not restart brokers during a ZK-driven cascade; the reconnections add more load.

4. Check for BadVersionException. If the managed ledger sees a version conflict on metadata updates, Pulsar 2.11.1+ (backported to 2.10.2, 2.9.4, 2.8.5) forces it to FENCED state (PR #17736). This is a protective measure to prevent split-brain, but the root cause is metadata contention, not the fencing itself.

5. Verify bookie availability. If fencing succeeds but the new ledger cannot be created, check for NotEnoughBookiesException in broker logs. Count writable bookies (bookie_SERVER_STATUS == 1). If the count is below the ensemble size or quorum requirement, the cluster cannot satisfy ledger creation.

6. Check for operator-induced fencing. If someone ran pulsar-admin topics stats-internal --metadata on an active topic, it can trigger ledger recovery and fencing on the active writer. This was a known issue in Pulsar 2.10.x and earlier.

Metrics and signals to monitor

Fixes

Normal failover fencing (no fix needed)

If every LedgerFencedException correlates 1:1 with a bundle unload, and new ledgers are created successfully, no action is needed. The fencing is doing its job. The only cost is first-message latency, typically under one second.

If this latency is unacceptable, consider PIP-192’s “transfer” operation (available since Pulsar 3.0 and enabled by default via loadBalancerTransferEnabled=true on the Extensible load balancer), which pre-assigns the destination broker before closing client connections, reducing the fencing window.

Topic Ownership Oscillation

When two brokers are trading a bundle back and forth, each transfer fences ledgers, opens new ones, and drops client connections. This is self-perpetuating.

1. Disable the load balancer temporarily to stop the oscillation:

   # WARNING: this stops all automatic bundle rebalancing cluster-wide.
   # Use only to break an active oscillation loop, then re-enable.
   pulsar-admin brokers update-dynamic-config \
     --config loadManagerClassName \
     --value org.apache.pulsar.broker.loadbalance.NoopLoadManager
   

2. Manually assign the contested bundle to a specific broker:

   # WARNING: forces bundle unload, which drops all client connections
   # on the current owner. Producers and consumers will reconnect.
   pulsar-admin namespaces unload <tenant>/<namespace> --bundle <bundle-range>
   

   Then verify the bundle lands on the intended broker via pulsar-admin broker-stats topics.

3. Debug the load balancer configuration. Check overload and underload thresholds. If two brokers are at similar load levels, the load balancer may be overshooting on each transfer. Widen the thresholds or address the underlying load imbalance (hot bundle, insufficient bundle splitting).

ZooKeeper session instability

If fencing is a downstream symptom of ZK session expiry, address the ZK layer first. See the playbook’s ZooKeeper session cascade pattern.

Key actions:

• Check ZK transaction log disk I/O (iostat -x 1 on the ZK data directory).
• Check ZK watch count (echo wchs | nc <zk-host> 2181). Watch explosions during consumer reconnect storms degrade ZK latency.
• Consider temporarily increasing metadataStoreSessionTimeoutMillis (zooKeeperSessionTimeoutMillis before Pulsar 4.0) if broker GC pauses are causing the session loss. This buys time but does not fix the underlying memory pressure.

NotEnoughBookiesException after fencing

This is the worst case. Fencing succeeded, but the cluster cannot create the new ledger. The topic is in a broken state.

1. Identify why bookies are unavailable. Check bookie_SERVER_STATUS on all bookies. Read-only bookies (status 0) cannot accept new writes.
2. Restore bookie availability: bring failed bookies back, free disk space on read-only bookies, or add new bookies.
3. Once enough writable bookies are available, the broker will retry ledger creation automatically.
4. If the topic remains stuck, unload and reload the namespace bundle to force a fresh ownership acquisition.

BadVersionException and the FENCED state

Since Pulsar 2.11.1 (backported to 2.10.2, 2.9.4, 2.8.5; PR #17736), a BadVersionException during managed ledger metadata updates forces the managed ledger to FENCED state. This prevents two brokers from processing the same ledger’s metadata simultaneously, which the PR described as a split-brain-like condition. The guard is correct and protective, but frequent BadVersionExceptions indicate metadata contention, typically from rapid ownership changes or ZK instability.

If you are running Pulsar 2.10.1 or earlier (pre-PR #17736), upgrade: the guard first shipped in 2.10.2, 2.9.4, 2.8.5, and 2.11.1. Without this guard, two brokers can silently process the same ledger’s metadata, which is more dangerous than the fencing itself.

Non-durable subscription failures on fenced topics

Before Pulsar 3.0.8 / 3.3.3 / 4.0.1 (PR #23579), non-durable subscriptions on fenced topics would fail permanently with “Attempted to use a fenced managed ledger.” Durable subscriptions and producers triggered topic close-and-recreate on fencing, but non-durable subs did not. If you see this error and are on an older version, upgrade to a patched release (3.0.8+, 3.3.3+, 4.0.1+).

Prevention

• Correlate fencing with bundle unloads as a dashboard signal. The most useful preventive metric is the ratio of fencing events to bundle transfers. A 1:1 ratio is healthy. Any fencing without a matching unload warrants investigation.

• Monitor ZK session stability proactively. ZK session loss is the upstream cause of most pathological fencing. Track broker-to-ZK session state and ZK request latency as first-class signals. Treat sustained ZK latency above 10ms as an early warning.

• Ensure enough bookie headroom. Fencing followed by NotEnoughBookiesException is catastrophic. Maintain writable bookie count above the ensemble size with margin. Monitor bookie_SERVER_STATUS and bookie disk usage to prevent read-only transitions.

• Avoid pulsar-admin topics stats-internal --metadata on active topics in Pulsar 2.10.x and earlier. This command can trigger ledger recovery and fencing on the active writer, disrupting production traffic.

• Configure topicFencingTimeoutSeconds if running Pulsar 4.0.x. This setting controls how long a topic can remain fenced before being forcefully closed. The default is 0 (disabled), meaning a fenced topic can stay fenced indefinitely if recovery fails. Set a non-zero value to force bounded recovery.

• Upgrade to Pulsar 2.11.1+ (or 2.10.2+, 2.9.4+, 2.8.5+) if still on an older version. The BadVersionException-to-FENCED guard (PR #17736) prevents silent split-brain metadata processing. The non-durable subscription fix (PR #23579) prevents permanent subscription failure on fenced topics.

How Netdata helps

• Per-second metric resolution captures fencing-related latency spikes that 15-second scrape intervals miss. The brief stall during ledger rollover and the spiky publish latency during ownership oscillation are both visible at 1-second granularity.

• Correlate bundle unload rate with publish latency on the same timeline. Overlay pulsar_lb_unload_bundle_total against pulsar_broker_publish_latency to verify the 1:1 correlation that distinguishes normal fencing from pathological patterns.

• ZooKeeper session state as an anomaly signal. Netdata’s ML-based anomaly detection flags transitions in pulsar_zookeeper_connected before they cascade into ownership changes and fencing loops.

• Bookie health at a glance. bookie_SERVER_STATUS across all bookies in a single view, with anomaly detection on read-only transitions, gives early warning before fencing meets NotEnoughBookiesException.

• Cross-component correlation. Netdata collects broker, bookie, and ZooKeeper metrics from the same cluster, so you can trace a fencing event from ZK session loss through bundle ownership change to publish latency spike without switching between dashboards.

Related guides

• Apache Pulsar broker down: telling a dead broker from a fenced one
• Apache Pulsar broker GC death spiral: heap pressure, stop-the-world pauses, and lost topic ownership
• Apache Pulsar bookie read-only: disk full and bookie_SERVER_STATUS at zero
• Apache Pulsar broker lookup failures: new clients cannot find their topic
• How Apache Pulsar actually works in production: a mental model for operators
• Apache Pulsar bookie disk filling: runway to read-only and how to reclaim space
• Apache Pulsar journal write stall: bookie journal fsync latency and the blocked write path