Apache Pulsar topic ownership oscillation: brokers fighting over the same bundle

Broker A takes ownership of a namespace bundle, opens managed ledgers, and begins serving clients. The load balancer sees A as overloaded and unloads the bundle. Broker B picks it up, fences the ledgers, creates new ones, and starts serving. The balancer now sees B as overloaded and moves it back. The cycle repeats indefinitely.

Each iteration forces ledger fencing, new ledger creation, ZK metadata writes, and client disconnection and reconnection storms. Producers see intermittent publish latency spikes. New clients fail lookups during the brief ownership gap between fencing and re-acquisition. The cluster looks healthy in aggregate because no single broker is down, but the affected topics are in a constant state of transition.

Normal rebalancing converges in minutes. Oscillation persists. The first priority is to freeze the load balancer, then fix the configuration or constraints that caused it.

What this means

A namespace bundle is the unit of load balancing in Pulsar. Each bundle maps to exactly one owning broker at a time. When the load balancer redistributes load, it unloads a bundle from one broker and lets another acquire it. During the unload, the old broker closes its topics and the managed ledger is fenced: the current BookKeeper ledger is sealed permanently. The new owner creates a fresh ledger, and clients reconnect through topic lookup.

In a healthy rebalance, this happens once per bundle and the system converges. In oscillation, the load balancer’s shedding decisions fight themselves. Broker A looks overloaded because it holds the hot bundle. The balancer moves it to B. Now B looks overloaded for the same reason. Back it goes.

flowchart LR
    A[Broker A acquires bundle] --> B[Opens managed ledgers]
    B --> C[Balancer: A above threshold]
    C --> D[Unload bundle from A]
    D --> E[Broker B acquires bundle]
    E --> F[Fences ledgers, opens new ones]
    F --> G[Balancer: B above threshold]
    G --> H[Unload bundle from B]
    H --> A

Default confirmed at 30 minutes; per the broker.conf comment the guard is per-topic (“Prevent the same topics to be shed and moved to other broker more than once within this timeframe”).

The loadBalancerSheddingGracePeriodMinutes setting (default 30 minutes) should prevent the same bundle from being unloaded repeatedly within that window. If you observe oscillation with a period shorter than 30 minutes, the issue is likely one of: different bundles in the same namespace cycling in sequence, a race condition in the ownership cache, or a version-specific bug. If the period is longer than 30 minutes, the grace period is functioning but the underlying load distribution problem has not been solved.

The distinguishing feature is that oscillation does not converge. Normal rebalancing settles after a few minutes as the load distribution stabilizes. Oscillation produces a repeating pattern of bundle load and unload events that continues indefinitely until you intervene.

Common causes

Quick checks

These commands are read-only and safe to run during an active incident.

# Check bundle unload rate across all brokers
curl -s http://<broker-host>:8080/metrics | grep pulsar_lb_unload_bundle_total

# Check lookup failures (clients failing to find their topic)
curl -s http://<broker-host>:8080/metrics | grep pulsar_broker_lookup

# Check active connections (fluctuating = reconnection storm)
curl -s http://<broker-host>:8080/metrics | grep pulsar_active_connections

# Check publish latency (spiky = each fencing cycle causes latency)
curl -s http://<broker-host>:8080/metrics | grep pulsar_broker_publish_latency

# Check broker-to-ZK session state (not a Prometheus metric; use the four-letter command from the broker host)
echo "stat" | nc <zk-host> 2181

# Check for ledger fencing events in broker logs
grep -c "fenced\|LedgerFencedException" /var/log/pulsar/broker.log

# Check for rapid ownership changes in broker logs
grep -E "acquired ownership|released ownership" /var/log/pulsar/broker.log | tail -50

# Check bundle distribution across brokers
curl -s http://<broker-host>:8080/admin/v2/broker-stats/topics

# Check current load balancer configuration
curl -s http://<broker-host>:8080/admin/v2/brokers/configuration | jq

# Check per-broker topic count (identify hot brokers)
curl -s http://<broker-host>:8080/metrics | grep pulsar_topics_count

How to diagnose it

1. Confirm the oscillation pattern. Grep broker logs for ownership change messages on the affected namespace. If you see the same bundle being acquired and released every few minutes on two different brokers, you have confirmed oscillation. Normal rebalancing shows a single ownership transfer per bundle and then stability.

2. Check whether ZK is the root cause. If broker-to-ZK session state is flapping on the affected brokers (checked with echo "stat" | nc <zk-host> 2181), the ownership changes may be caused by ZK session expiry, not the load balancer. Check ZK latency and watch count before adjusting load balancer settings.

3. Identify the oscillating bundle. Use pulsar-admin broker-stats topics or check the broker logs to find which namespace and bundle range is cycling. Note the two or more brokers involved.

4. Check load balancer configuration. Review the current dynamic config: shedding thresholds, shedding interval, and grace period. If thresholds are too tight relative to the actual load variance between brokers, the balancer will never converge.

5. Check anti-affinity and isolation policies. If namespace isolation policies or anti-affinity groups are configured, verify that the constraints are satisfiable. If the number of anti-affinity namespaces exceeds the number of available placement slots (failure domains times brokers), the balancer will cycle indefinitely trying to satisfy an impossible constraint.

6. Check NIC speed reporting on cloud instances. On AWS EC2 and similar, the OS may report a higher NIC speed than the actual bandwidth. The load manager uses this to calculate network resource usage. If the reported speed is too high, the broker appears underloaded, receives excess bundles, then triggers shedding.

7. Check for known version bugs. If running Pulsar older than 3.0.8, 3.3.3, 4.0.1, or 4.1.0, the ownership cache race condition fixed in PR #23515 may be contributing. The symptom is a broker that holds ZK ownership but cannot serve the topic because the local cache is empty, creating a lookup loop.

Metrics and signals to monitor

Fixes

Emergency: stop the oscillation

The fastest way to break the cycle is to switch to NoopLoadManager. This freezes all bundle ownership in place and stops the fencing loop.

# WARNING: this disables all automatic load balancing. Re-enable after fixing the root cause.
pulsar-admin brokers update-dynamic-config --config loadManagerClassName \
  --value org.apache.pulsar.broker.loadbalance.NoopLoadManager

Both are valid: loadBalancerSheddingEnabled=false stops shedding without changing the load manager class (bundles keep their ownership), while NoopLoadManager disables rebalancing entirely.

After switching, bundles stay where they are. No new shedding occurs. Clients stop seeing fencing-related latency spikes. This is a stopgap, not a fix. Re-enable the load balancer after correcting the underlying configuration.

Pin the affected bundle

With NoopLoadManager active, unload the oscillating bundle from its current broker so it lands on a single broker and stays there.

pulsar-admin namespaces unload <tenant/namespace> --bundle <range>

The bundle will be re-acquired by whichever broker picks it up first. With NoopLoadManager active, it stays on that broker because no shedding runs. Verify ownership with pulsar-admin broker-stats topics.

Fix aggressive load balancer thresholds

If the root cause is thresholds that are too tight, widen them. Key settings:

• loadBalancerBrokerOverloadedThresholdPercentage (default 85): if set too low, normal traffic spikes trigger shedding.
• loadBalancerBrokerThresholdShedderPercentage (default 10): if set too high, brokers never appear underloaded and bundles never settle.
• loadBalancerSheddingIntervalMinutes: if too short, the balancer evaluates and sheds too frequently.
• loadBalancerSheddingGracePeriodMinutes: the default of 30 minutes should prevent the same bundle from being unloaded repeatedly. If this has been reduced, restore it.

loadBalancerBrokerLoadTargetStd is confirmed as the TransferShedder key on the Extensible load manager.

On ExtensibleLoadManagerImpl (available since Pulsar 3.0; ModularLoadManagerImpl remains the default in broker.conf), the loadBalancerBrokerLoadTargetStd setting (default 0.25) controls the target standard deviation of load across brokers. If your brokers have naturally similar load profiles, this target may be unreachable, causing the shedder to try repeatedly. The TransferShedder strategy includes anti-oscillation logic: it checks whether a target bundle unload was effective and stops further unloading if it was not.

Fix anti-affinity or isolation constraints

Verify that anti-affinity constraints are physically satisfiable. The number of anti-affinity namespaces must not exceed the number of distinct placement targets (failure domains times brokers per domain). If it does, the load balancer will never find a valid assignment and will cycle indefinitely.

Review namespace isolation policies for conflicting or overly restrictive rules that prevent bundles from landing on available brokers.

Fix NIC speed misreporting

On cloud instances where the reported NIC speed exceeds actual bandwidth, set loadBalancerOverrideBrokerNicSpeedGbps to match the real NIC speed. This corrects the load manager’s network utilization calculation.

Upgrade if hitting a known bug

If running a version affected by the ownership cache race (PR #23515), upgrade to 3.0.8+, 3.3.3+, 4.0.1+, or 4.1.0+. The fix re-acquires ownership into the local cache if ZK says the current broker owns the bundle, preventing the lookup loop.

Prevention

• Set loadBalancerSheddingGracePeriodMinutes to at least the default 30 minutes. Do not reduce it without understanding the consequences.
• Ensure load balancer thresholds reflect your actual broker capacity and workload variance.
• Verify anti-affinity constraints are satisfiable before deploying them. Count the placement targets and compare against the number of anti-affinity namespaces.
• On cloud instances, set loadBalancerOverrideBrokerNicSpeedGbps to match actual bandwidth if the OS-reported speed is incorrect.
• Keep Pulsar current. ExtensibleLoadManagerImpl with TransferShedder (available since Pulsar 3.0) includes anti-oscillation logic that older load balancers lack. If still running the deprecated SimpleLoadManagerImpl, migrate.
• Monitor bundle unload rate as a time series. A sustained rate above 1 per minute outside of maintenance is the earliest sign of oscillation.

How Netdata helps

• Netdata’s per-second resolution on pulsar_lb_unload_bundle_total makes the oscillation pattern visible as a repeating sawtooth, rather than smoothing it away with longer scrape intervals.
• Overlaying pulsar_active_connections against bundle unload events shows whether each unload is triggering a reconnection storm.
• Correlating pulsar_broker_publish_latency spikes with unload events confirms that each ownership transfer causes measurable producer degradation.
• pulsar_broker_lookup_failures rising during ownership gaps identifies which clients are affected, even though existing connections continue to work.
• pulsar_zookeeper_connected lets you distinguish load-balancer-induced oscillation from ZK-session-loss-induced ownership changes, which require a different fix.
• Anomaly detection on bundle unload rate can alert on the repeating pattern before it escalates to user-visible latency.

Related guides

• Apache Pulsar active connections climbing: connection leaks and file descriptor exhaustion
• Apache Pulsar bookie add-entry queue not draining: writes arriving faster than the disk can commit
• Apache Pulsar AutoRecovery stalled: under-replicated ledgers that never heal
• Apache Pulsar backlog age vs size: the latency depth alone cannot show
• Apache Pulsar backlog quota exceeded: producers held or rejected when consumers stall
• Apache Pulsar bookie disk filling: runway to read-only and how to reclaim space
• Apache Pulsar bookie failure cascade: recovery I/O that topples surviving bookies
• Apache Pulsar bookie read latency high: catch-up reads competing with the write path
• Apache Pulsar bookie read-only: disk full and bookie_SERVER_STATUS at zero
• Apache Pulsar broker down: telling a dead broker from a fenced one
• Apache Pulsar broker GC death spiral: heap pressure, stop-the-world pauses, and lost topic ownership
• Apache Pulsar broker lookup failures: new clients cannot find their topic