Apache Pulsar broker hotspot: one broker owning far more topics than the rest

You have a multi-broker Pulsar cluster where one broker carries a disproportionate share of topic ownership. Cluster-wide averages look acceptable, but that single broker shows elevated GC pressure, higher publish latency, more active connections, or a larger heap footprint than its peers. The imbalance may have built for hours or days without triggering an alert because aggregate metrics hid it.

In Pulsar, topics are not assigned to brokers individually. They are assigned at the namespace bundle level. Each namespace is sharded into bundles (hash-range slices of the topic namespace), and each bundle is owned by exactly one broker at a time. A topic lands in a bundle by hashing its name. When a bundle accumulates too many topics, too much throughput, or too many sessions, the load balancer should split it and redistribute ownership. When this does not happen, one broker ends up owning a hot bundle and becomes a bottleneck.

Topic count is a proxy for load, not a direct measure. One topic doing 1M messages/sec outweighs a thousand idle topics. Always correlate topic count with per-broker throughput before deciding the imbalance is the root cause.

What this means

As a rule of thumb, the standard deviation of topic count across brokers should be under 50% of the mean. If one broker owns more than 2x the cluster average sustained for more than 30 minutes, you have a hotspot.

The hotspot broker becomes the bottleneck for several reasons:

• Heap pressure: Each topic and its subscriptions consume heap memory for managed ledger state, cursors, and dispatchers. A broker with 10x the topic count of its peers carries 10x the heap overhead.
• Connection concentration: Producers and consumers connect to the broker that owns their topic. More topics means more connections, more file descriptors, and more direct memory for Netty buffers.
• GC amplification: More heap-resident objects means more GC work. If the hot broker is near heap limits, frequent GC pauses can cascade into ZooKeeper session expiry.
• Dispatcher contention: More active topics means more dispatcher threads competing for CPU and memory bandwidth.

The load balancer should detect this and trigger bundle unloads to redistribute ownership. If it is not doing so, the problem is in the load balancer configuration, the bundle split thresholds, or a silent split failure.

Common causes

Quick checks

# Check topic count per broker from Prometheus metrics
curl -s http://<broker-host>:8080/metrics | grep pulsar_topics_count

# Check bundle distribution across brokers
curl -s http://<broker-host>:8080/metrics | grep pulsar_broker_bundles_count

# Check topic ownership via admin API (verbose on large clusters; pipe through jq)
pulsar-admin broker-stats topics

# Check namespace bundle configuration
pulsar-admin namespaces bundles <tenant>/<namespace>

# Check if the load balancer is shedding bundles
curl -s http://<broker-host>:8080/metrics | grep pulsar_lb_unload_bundle_total

# Check bundle split metrics
curl -s http://<broker-host>:8080/metrics | grep pulsar_lb_bundles_split_count

# Check per-broker throughput (correlate with topic count)
curl -s http://<broker-host>:8080/metrics | grep -E "pulsar_broker_rate_(in|out)"

# Check load balancer dynamic configuration
pulsar-admin brokers get-all-dynamic-config

# Check which load manager class is active
pulsar-admin brokers get-all-dynamic-config | grep loadManagerClassName

How to diagnose it

1. Confirm the imbalance. Pull pulsar_topics_count aggregated per broker. Compare the maximum against the cluster mean. If the standard deviation across brokers exceeds 50% of the mean, the distribution is skewed.

2. Determine if the load balancer is trying to fix it. Check pulsar_lb_unload_bundle_total on the hot broker and across the cluster. If the counter is increasing, the load balancer is attempting to shed bundles but something is preventing convergence (ownership oscillation, grace period, or no suitable destination). If it is flat, the load balancer does not see a problem.

3. Check per-broker throughput, not just topic count. A broker with 5000 idle topics may be healthier than one with 500 active topics. Pull pulsar_broker_rate_in and pulsar_broker_rate_out per broker. If throughput is also skewed, the imbalance is load-relevant. If topic count is skewed but throughput is even, the imbalance may be benign but still a risk if traffic patterns shift.

4. Investigate why bundle splitting is not occurring. Check the namespace’s current bundle count against loadBalancerNamespaceMaximumBundles (default 128). If the namespace already has 128 bundles, the split logic is silently skipped. If it has far fewer, check whether the bundle split thresholds are appropriate for the workload.

flowchart TD
    A["Hot broker: > 2x avg topic count"] --> B{"Load balancer shedding?
pulsar_lb_unload_bundle_total rising"}
    B -->|"No"| C{"Bundle splitting?
pulsar_lb_bundles_split_count rising"}
    B -->|"Yes"| D["Check for ownership oscillation
or grace period blocking"]
    C -->|"No"| E{"Namespace at max bundles?"}
    C -->|"Yes"| F{"Split creates new bundle
but stays on same broker?"}
    E -->|"Yes"| G["Increase loadBalancerNamespaceMaximumBundles
or pre-split namespace"]
    E -->|"No"| H["Check split thresholds
vs actual per-bundle load"]
    F -->|"Yes"| I["ThresholdShedder blind spot
or NIC speed misreport"]
    D --> J["Verify shedding interval
and grace period config"]

5. Check the active load balancer type and shedding configuration. Run pulsar-admin brokers get-all-dynamic-config and look for loadManagerClassName. The modular load balancer (ModularLoadManagerImpl) is the default in Pulsar 2.x through 3.x and uses ThresholdShedder by default. The extensible load manager (ExtensibleLoadManagerImpl), available since 3.0, uses TransferShedder by default and pre-assigns destination brokers during unload, reducing topic unavailability.

6. Check NIC speed reporting on cloud instances. AWS EC2 instances with a 1Gbps NIC may report 10Gbps to Linux. The load manager then calculates resource usage as a percentage of an inflated capacity, never triggering shedding. If running on cloud instances, verify loadBalancerOverrideBrokerNicSpeedGbps is set to the actual NIC speed.

7. Check for the ThresholdShedder idle-broker blind spot. With 10 brokers at 80% load and 1 idle broker at 0%, the cluster average is approximately 72.7%. The ThresholdShedder unload threshold is average plus 10%, or 82.7%. Since 80% is below 82.7%, no unloading triggers, and the idle broker sits unused. If this matches your topology, lowerBoundarySheddingEnabled=true resolves it.

Metrics and signals to monitor

Fixes

Pre-split namespaces with sufficient bundles

The default initial bundle count for a new namespace is 4. For namespaces expected to hold many topics across a multi-broker cluster, this is insufficient. The load balancer must auto-split and rebalance, which takes time and causes transient client disconnections during each split.

Pre-create namespaces with more bundles than brokers. For example, for 1000 topics across 16 brokers, start with 64 bundles:

# Create a namespace with a specific bundle count
pulsar-admin namespaces create <tenant>/<namespace> --bundles 64

Manually unload bundles from the hot broker

If the load balancer is not acting, manually trigger bundle unloads. This forces client reconnections and causes brief latency blips (typically tens of milliseconds per unload). Use this when the imbalance is causing acute degradation.

# Unload a specific namespace bundle range
pulsar-admin namespaces unload <tenant>/<namespace> --bundle <bundle-range>

# Unload all bundles in a namespace (DISRUPTIVE: all clients reconnect)
pulsar-admin namespaces unload <tenant>/<namespace>

Enable lower boundary shedding on ThresholdShedder

If the ThresholdShedder is not triggering because idle brokers depress the cluster average below the shedding threshold:

pulsar-admin brokers update-dynamic-config \
  --config lowerBoundarySheddingEnabled \
  --config-value true

This makes the shedder consider the lower boundary of resource usage, not just the average plus threshold. It enables unloading from brokers that are above average even when the absolute values do not cross the traditional threshold. This is a dynamic config change and takes effect without a broker restart.

Override NIC speed on cloud instances

If cloud instances report inflated NIC speeds, set the actual speed so the load balancer calculates resource usage correctly. This requires a broker restart:

# In broker.conf
loadBalancerOverrideBrokerNicSpeedGbps=1

Verify the actual NIC speed before setting this. A value that is too low causes over-shedding.

Increase maximum bundle count

If the namespace has hit the default maximum of 128 bundles and splitting is silently skipped:

# In broker.conf
loadBalancerNamespaceMaximumBundles=256

This is a static config and requires a broker restart. Existing namespaces do not automatically split to the new maximum; they still need to hit split thresholds or be manually split.

Adjust bundle split thresholds

If bundles are not splitting because the workload does not trigger the default thresholds, tune them for your traffic patterns:

Lower thresholds if bundles are growing too large before splitting. More bundles means more metadata operations in ZooKeeper and more overhead per split cycle.

Temporarily disable the load balancer for ownership oscillation

If two brokers are rapidly cycling ownership of the same bundle (visible as rapid pulsar_lb_unload_bundle_total increments and ledger fencing events in broker logs), disable the load balancer temporarily to stop the thrashing:

# WARNING: this disables all automatic load balancing cluster-wide.
# Only use during active troubleshooting. Re-enable as soon as stable.
pulsar-admin brokers update-dynamic-config \
  --config loadManagerClassName \
  --config-value org.apache.pulsar.broker.loadbalance.NoopLoadManager

Then manually assign bundles to brokers, investigate the threshold configuration, and re-enable the load balancer once stable.

Prevention

• Pre-create namespaces with enough bundles. Start with more bundles than brokers for namespaces expected to hold significant topic counts. Do not rely on auto-splitting from the default of 4.
• Alert on distribution skew. Trigger when any broker exceeds 2x the cluster average for more than 30 minutes. Track pulsar_lb_bundles_split_count as a trend; a flat line while topics are being created means the load balancer is not adapting.
• Set NIC speed overrides on cloud instances during provisioning, not as a reactive fix.
• Enable lowerBoundarySheddingEnabled if some brokers are typically idle or the fleet is heterogeneous.
• Verify load balancer configuration after upgrades. Defaults and behavior can change between versions, particularly around the modular-to-extensible transition in 3.0+.

How Netdata helps

Netdata collects Pulsar broker metrics at per-second resolution and correlates them across the fleet. For this specific problem:

• Distribution skew at a glance: Per-broker pulsar_topics_count alongside pulsar_broker_rate_in and pulsar_broker_rate_out on a single timeline distinguishes topic-count imbalance from throughput imbalance.
• Load balancer activity: pulsar_lb_unload_bundle_total and pulsar_lb_bundles_split_count as time series show whether the balancer is working or stalled.
• Downstream impact: JVM GC pauses, heap usage, active connections, and publish latency on the hot broker appear alongside the distribution metrics that caused them.
• Anomaly detection: Flags gradual drift in topic distribution before a fixed threshold would fire.

Related guides

• Apache Pulsar active connections climbing: connection leaks and file descriptor exhaustion
• Apache Pulsar bookie add-entry queue not draining: writes arriving faster than the disk can commit
• Apache Pulsar AutoRecovery stalled: under-replicated ledgers that never heal
• Apache Pulsar backlog age vs size: the latency depth alone cannot show
• Apache Pulsar backlog quota exceeded: producers held or rejected when consumers stall
• Apache Pulsar bookie disk filling: runway to read-only and how to reclaim space
• Apache Pulsar bookie failure cascade: recovery I/O that topples surviving bookies
• Apache Pulsar bookie read latency high: catch-up reads competing with the write path
• Apache Pulsar bookie read-only: disk full and bookie_SERVER_STATUS at zero
• Apache Pulsar broker down: telling a dead broker from a fenced one
• Apache Pulsar broker GC death spiral: heap pressure, stop-the-world pauses, and lost topic ownership
• Apache Pulsar broker lookup failures: new clients cannot find their topic