Apache Pulsar bundle unload thrashing: the load balancer that never converges

The primary signal is pulsar_lb_unload_bundle_total, a counter exposed on the broker Prometheus metrics endpoint. In steady state it barely moves: bundle ownership changes are rare, ideally less than once per hour. When this counter climbs past one unload per minute and there is no rolling upgrade, no broker restart, and no planned maintenance, the load balancer is thrashing.

Each unload transfers a namespace bundle (a hash-range slice of topics) from one broker to another. Each transfer drops the TCP connections for every producer and consumer on those topics. Clients reconnect automatically, but reconnection generates metadata operations against ZooKeeper, new ledger creation in BookKeeper, and a latency blip measured in tens of milliseconds per affected topic. When unloads happen every few seconds, those blips stack into sustained degradation: elevated publish latency, client reconnection storms, and metadata store pressure that feeds back into more instability.

The Extensible load balancer (Pulsar 3.x) mitigates some of this by pre-assigning destination brokers during bundle transfer, which speeds reconnection compared to the older Modular load balancer. Neither load balancer is immune to thrashing. Root causes range from misconfigured shedding thresholds to broker resource exhaustion to known bugs in specific Pulsar versions.

Normal vs. thrashing unloads

Each namespace is divided into hash-range bundles (default 4 per namespace, configurable). Each bundle is owned by exactly one broker at a time. Topic lookup resolves a topic name to a bundle via hashing, then resolves the bundle to its owning broker via ZooKeeper metadata. When the load balancer decides to move a bundle, it instructs the current owner to unload it: close all producer and consumer connections for topics in that bundle, release ZooKeeper ownership, and let the destination broker take over. On the Modular load balancer, the destination is determined after the unload via client lookups. On the Extensible load balancer, the destination is pre-assigned before the unload completes.

Normal unload scenarios:

• Rolling broker restarts or upgrades: every restarting broker sheds its bundles, and surviving brokers absorb them. Unload spikes proportional to bundle count are expected.
• Manual bundle unload via admin API for rebalancing after adding brokers.
• Broker failure: its bundles are reassigned to survivors.

Thrashing is different. The load balancer repeatedly moves the same or similar bundles because no stable assignment satisfies its shedding criteria. The system never reaches equilibrium. Each cycle of unload, reassignment, reconnect, and re-evaluation generates overhead that can itself prevent convergence, especially when ZooKeeper latency rises from the metadata storm.

Common causes

Quick checks

These are read-only and safe to run on any broker.

# Check the unload counter trend across all brokers
curl -s http://<broker-host>:8080/metrics | grep pulsar_lb_unload_bundle_total

# Verify broker process health
curl -s -o /dev/null -w "%{http_code}" http://<broker-host>:8080/admin/v2/brokers/health

# Check active broker count
curl -s http://<broker-host>:8080/admin/v2/brokers/<cluster> | jq '. | length'

# Check topic count distribution across brokers for hotspot detection
curl -s http://<broker-host>:8080/metrics | grep pulsar_topics_count

# Check ZooKeeper connectivity from the broker (broker-to-ZK session state is not exposed as a
# Prometheus metric; use the four-letter command from each broker host)
echo "stat" | nc <zk-host> 2181

# Check for lookup failures (indicates metadata or ownership churn)
curl -s http://<broker-host>:8080/metrics | grep pulsar_broker_lookup

# Check active connections for reconnection storm signature
curl -s http://<broker-host>:8080/metrics | grep pulsar_active_connections

# Verify the configured load balancer class
curl -s http://<broker-host>:8080/admin/v2/brokers/configuration | jq '.loadManagerClassName'

# Check the Pulsar version (relevant for the infinite unloading bug)
curl -s http://<broker-host>:8080/admin/v2/brokers/<cluster>/<broker-host>:8080 | jq '.version'

If the unload counter is rising on multiple brokers simultaneously and there is no maintenance activity, proceed to diagnosis.

How to diagnose it

flowchart TD
    A["pulsar_lb_unload_bundle_total
climbing > 1/min"] --> B{"Maintenance or
rolling restart active?"}
    B -- Yes --> C["Expected: monitor
until complete"]
    B -- No --> D{"Multiple brokers losing
ZK sessions?"}
    D -- Yes --> E["ZK latency feedback loop
Check ZK health first"]
    D -- No --> F{"One broker overloaded
or GC spiraling?"}
    F -- Yes --> G["Resource exhaustion
Fix broker capacity"]
    F -- No --> H{"Pulsar version
has infinite unload bug?"}
    H -- "Pre-3.1.0 etc." --> I["Apply PR #20822
or upgrade"]
    H -- "Patched" --> J{"Config thresholds
too aggressive?"}
    J -- Yes --> K["Tune shedding config
or disable LB temporarily"]
    J -- No --> L["Check NIC speed
and bundle count"]

1. Rule out maintenance. Confirm no rolling restart, upgrade, or manual pulsar-admin unload command is active. Check the active broker count: if it dropped recently, the unload spike is redistribution from a failed or restarting broker and should subside.

2. Check ZooKeeper health. High ZK latency causes brokers to lose sessions, which triggers bundle reassignment, which increases ZK load. Look for Session expired or Connection loss messages in broker logs. Check ZK request latency via echo stat | nc <zk-host> 2181 and watch count via echo wchs | nc <zk-host> 2181. If ZK latency is above 50ms sustained, address ZK first: the unload thrashing is a symptom, not the cause.

3. Check broker resource utilization. If one broker is shedding bundles because it cannot hold them, look at JVM heap usage, GC pause times, and direct memory. A broker in a GC death spiral will repeatedly lose ZK sessions, shed bundles, recover, and shed again. The oscillation pattern in pulsar_active_connections (dropping and spiking) is the signature.

4. Check the Pulsar version. Before Pulsar 3.1.0 (and the equivalent backports: 2.10.6, 2.11.3, 3.0.1), a bug could cause the load balancer to continuously unload bundles in an infinite loop. This was fixed in PR #20822, described in release notes as “Avoid infinite bundle unloading.” If your cluster is on an older version and the configuration looks reasonable, this is a likely cause.

5. Check shedding configuration. On the Extensible load balancer with TransferShedder, loadBalancerBrokerLoadTargetStd (default 0.25) controls how tightly the load balancer tries to equalize broker load. A value too tight for heterogeneous hardware or bursty traffic causes perpetual rebalancing. loadBalancerSheddingConditionHitCountThreshold controls how many shedding intervals a condition must persist before triggering an unload. Too low means transient spikes trigger unnecessary shedding.

6. Check NIC speed reporting. On cloud instances, the OS may report a NIC speed higher than the actual instance limit. For example, an AWS EC2 instance with a 1 Gbps NIC may report 10 Gbps to the OS. The load balancer then underestimates network utilization and over-sheds. Set loadBalancerOverrideBrokerNicSpeedGbps to the actual NIC speed.

7. Check bundle count and splitting. If each namespace has the default 4 bundles but one namespace handles most of the throughput, the load balancer cannot redistribute finely enough. Automatic bundle splitting should trigger when a bundle exceeds the load threshold, but if splitting is disabled or the thresholds are too high, hot bundles oscillate between brokers.

Metrics and signals to monitor

Fixes

Disable the load balancer temporarily to stop the bleeding

If thrashing is causing visible client impact, disable the load balancer immediately. Bundles stay where they are, and you diagnose without further churn.

# WARNING: this stops all automatic rebalancing. Bundles will not move
# even if a broker fails. Re-enable as soon as the root cause is fixed.
pulsar-admin brokers update-dynamic-config \
  --config loadManagerClassName \
  --value org.apache.pulsar.broker.loadbalance.NoopLoadManager

To re-enable, set the config back to the original value (org.apache.pulsar.broker.loadbalance.ModularLoadManagerImpl — the default in all current versions — or org.apache.pulsar.broker.loadbalance.ExtensibleLoadManagerImpl if you run the Extensible load balancer).

Fix shedding thresholds

If the load balancer is over-sensitive, loosen the convergence criteria:

• Increase loadBalancerSheddingConditionHitCountThreshold so transient traffic bursts do not trigger immediate shedding. The default is 3, meaning a condition must persist across three shedding intervals before shedding triggers. Raising it further requires more sustained imbalance before shedding.
• Increase loadBalancerBrokerLoadTargetStd (Extensible LB with TransferShedder) from the default 0.25 to 0.5 or higher for clusters with heterogeneous hardware or bursty workloads. This allows more variance in broker load before triggering rebalancing.
• Increase loadBalancerSheddingGracePeriodMinutes (default 30) to prevent the same bundle from being shed again within the grace window. The broker.conf comment for this setting says it prevents “the same topics” from being shed and moved more than once within the timeframe, i.e. the guard is per-topic.

These are broker.conf settings. Several load-balancer keys (including loadBalancerSheddingEnabled, loadBalancerSheddingGracePeriodMinutes, and loadManagerClassName) are dynamic in recent Pulsar versions and change without a restart; check which keys are updatable in your version with pulsar-admin brokers list-dynamic-config before relying on a runtime change.

Fix ZooKeeper latency

If ZK latency is the root cause, the unload thrashing will not stop until ZK recovers. See Apache Pulsar broker down: telling a dead broker from a fenced one for related fencing and ownership patterns.

Immediate actions:

• Check ZK transaction log disk I/O. A slow transaction log disk is the most common ZK performance killer.
• Check ZK watch count via echo wchs | nc <zk-host> 2181. Watch explosions from consumer reconnect storms can overwhelm ZK.
• Do not restart brokers to fix ZK issues. Restarts generate more reconnection load and worsen the cycle.

Upgrade to fix the infinite unloading bug

If your cluster is running a version older than 3.1.0 (or the backported fixes in 2.10.6, 2.11.3, or 3.0.1) and the thrashing matches the infinite bundle unloading pattern, upgrade or apply the patch from PR #20822. The release notes describe this fix as “Avoid infinite bundle unloading.” The fix targets the Modular load balancer’s threshold-shedding path; the Extensible load balancer’s TransferShedder has separate anti-oscillation logic.

Fix NIC speed misreporting

On cloud instances where the reported NIC speed exceeds the actual instance network limit:

# Set the actual NIC speed in broker.conf or via dynamic config
# Example for a 1 Gbps instance:
loadBalancerOverrideBrokerNicSpeedGbps=1

This prevents the load balancer from underestimating network utilization and over-shedding to compensate.

Increase bundle count for fine-grained rebalancing

If the default 4 bundles per namespace cannot distribute load evenly:

• Lower the bundle splitting thresholds so hot bundles split sooner.
• Manually split hot bundles: pulsar-admin namespaces split-bundle <tenant>/<namespace> --bundle <range>.
• Increase the initial bundle count for new namespaces: pulsar-admin namespaces set-bundles <tenant>/<namespace> --num-bundles <n>.

Bundle count only grows via splitting. There is no automatic merging. Over time you may accumulate many small bundles.

Prevention

• Monitor pulsar_lb_unload_bundle_total as a rate. Alert on sustained rates above 1/min outside maintenance windows. The grace period and maintenance detection must be built into the alert to avoid false positives during upgrades.

• Track ZK latency as a leading indicator. Sustained ZK latency above 10ms is an early warning. By the time it exceeds 50ms, bundle thrashing may already be starting.

• Run supported versions. Stay on 3.1.0+ or the backported branches (2.10.6+, 2.11.3+, 3.0.1+) to avoid the infinite unloading bug.

• Size shedding thresholds to your workload. The defaults are conservative. Clusters with heterogeneous broker hardware, bursty traffic, or high topic counts typically need looser thresholds.

• Verify NIC speed on cloud deployments. Check what the OS reports versus the actual instance network limit. Set loadBalancerOverrideBrokerNicSpeedGbps if they differ.

• Consider the Extensible load balancer on 3.x. Its TransferShedder with bundle pre-assignment reduces client disruption during unloads and provides better convergence properties than the Modular load balancer’s ThresholdShedder.

• Monitor per-broker topic and throughput distribution. Hot brokers and hot bundles are the precondition for thrashing. Detect imbalance early before the load balancer starts oscillating.

How Netdata helps

Netdata’s per-second metric collection is useful for bundle thrashing because the failure mode involves rapid oscillation that coarser scrape intervals miss.

• Rate detection on pulsar_lb_unload_bundle_total: Per-second collection captures unload bursts that 15-second or 60-second scrapes average away. ML anomaly detection flags unusual unload patterns even when the rate is below a static threshold.
• Correlation with ZK latency: Thrashing and ZK latency form a feedback loop. Overlay unload rate, ZK request latency, and broker session events in the same time window to determine which is cause and which is effect.
• Connection churn visibility: pulsar_active_connections oscillation is the client-facing signature of thrashing. Per-second resolution makes the drop-spike pattern immediately visible.
• Per-broker resource pressure: Heap usage, GC pause times, and direct memory across all brokers in one view lets you quickly identify whether one overloaded broker is driving the shedding cycle.
• Lookup failure tracking: pulsar_broker_lookup_failures rising alongside unload rate confirms that ownership churn is affecting new client connections, not just existing ones.

Related guides

• Apache Pulsar active connections climbing: connection leaks and file descriptor exhaustion
• Apache Pulsar bookie add-entry queue not draining: writes arriving faster than the disk can commit
• Apache Pulsar AutoRecovery stalled: under-replicated ledgers that never heal
• Apache Pulsar backlog age vs size: the latency depth alone cannot show
• Apache Pulsar backlog quota exceeded: producers held or rejected when consumers stall
• Apache Pulsar bookie disk filling: runway to read-only and how to reclaim space
• Apache Pulsar bookie failure cascade: recovery I/O that topples surviving bookies
• Apache Pulsar bookie read latency high: catch-up reads competing with the write path
• Apache Pulsar bookie read-only: disk full and bookie_SERVER_STATUS at zero
• Apache Pulsar broker down: telling a dead broker from a fenced one
• Apache Pulsar broker GC death spiral: heap pressure, stop-the-world pauses, and lost topic ownership
• Apache Pulsar broker lookup failures: new clients cannot find their topic