$ guides / apache-pulsar / apache-pulsar-throttled-connections ▌

Operations Guides

Apache Pulsar throttled connections: the broker shedding load under pressure

Your dashboard shows pulsar_broker_throttled_connections climbing from zero, producers are complaining about elevated send latency, and some clients are timing out on new operations. The broker process is up, the health endpoint returns 200, and heap looks fine. The broker is not broken. It is defending itself.

Connection throttling is a protective mechanism. When the broker’s internal send queues on a connection back up beyond a configured ceiling, it stops reading new requests from that TCP connection (it disables auto-read on the Netty channel) until the backlog drains. The throttled connection count tells you how many connections are currently in that state. In a healthy cluster this gauge is zero. Any sustained non-zero value means the broker is at or near its capacity for the work being pushed through it.

This guide covers how to confirm what is actually saturated behind the throttle, and how to make the right call between adding broker capacity and reducing the load you are asking the broker to carry. For the broader failure catalogue and where this signal sits in the overall model, see How Apache Pulsar actually works in production.

What this means

Each client connection to a broker carries producers, consumers, and the acknowledgements and dispatch traffic for all of them. When the broker cannot drain its per-connection pending send queue as fast as requests arrive, it applies backpressure by pausing reads on that socket. The client sees this as rising publish or receive latency and, if it persists, timeouts and retries. Retries from many clients at once turn a throttled broker into a reconnection storm, which makes the underlying resource problem worse.

The throttle is a symptom, not a root cause. It fires when something downstream of the connection cannot keep up. The usual suspects are:

Direct memory pressure: every connection and in-flight message holds Netty off-heap buffers. Approaching MaxDirectMemorySize, allocation slows and queues back up.
Too much work per broker: too many owned topics, too much throughput, or too many connections for the CPU and memory allocated.
A slow write path: bookie journal stalls mean publish acknowledgements are late, so pending send state accumulates on the broker.
Connection storms: mass client reconnects after bundle unloads, broker restarts, or a client retry-loop bug.

flowchart TD
  A[Resource saturation] --> B[Pending send queue backs up]
  B --> C[Broker pauses reads on connection]
  C --> D[throttled_connections goes non-zero]
  D --> E[Client latency and timeouts]
  E --> F[Client retries and reconnects]
  F --> B
  A1[Direct memory pressure] --> A
  A2[Too many topics or throughput] --> A
  A3[Slow bookie write path] --> A
  A4[Connection storm] --> A

The dangerous property of this pattern is the feedback loop: throttling causes timeouts, timeouts cause retries and reconnects, and the churn adds load to an already saturated broker. Breaking the loop requires fixing the resource at the top of the diagram, not restarting the broker at the bottom.

Common causes

Cause	What it looks like	First thing to check
Direct memory exhaustion approaching	Process RSS far above heap, throttling grows with connection count, eventually `OutOfDirectMemoryError` in logs	Direct buffer pool usage via JMX; RSS vs heap gap
Too many topics on the broker	`pulsar_topics_count` much higher than peer brokers, GC pressure climbing	Per-broker topic count comparison
Throughput exceeds broker capacity	CPU saturated, publish latency elevated before throttle events, rates high on all topics	Broker CPU and `pulsar_broker_rate_in` trend
Bookie write path stall	`bookie_journal_JOURNAL_SYNC` P99 spiking, publish latency rising before throttling	Journal sync latency and force write queue on bookies
Connection storm	`pulsar_active_connections` spiking sharply, churn metrics diverging, recent bundle unload or restart	Connection created vs closed counters
Client retry loop bug	Connection count grows monotonically over days, throttling appears at peaks	Long-term connection count trend

Quick checks

All of these are read-only and safe to run during an incident.

# Current throttled connection count (should be zero in steady state)
curl -s http://<broker-host>:8080/metrics | grep throttled_connections

# Active connections: is the broker also carrying a connection spike?
curl -s http://<broker-host>:8080/metrics | grep pulsar_active_connections

# Publish latency: is the write path slow, or is the broker the bottleneck?
curl -s http://<broker-host>:8080/metrics | grep pulsar_broker_publish_latency

# Topic count on this broker: compare against peers for a hotspot
curl -s http://<broker-host>:8080/metrics | grep pulsar_topics_count

# Bundle unload rate: recent unloads explain reconnect storms
curl -s http://<broker-host>:8080/metrics | grep pulsar_lb_unload_bundle_total

# On the bookies: is the journal the real bottleneck behind the throttle?
curl -s http://<bookie-host>:8000/metrics | grep bookie_journal_JOURNAL_SYNC
curl -s http://<bookie-host>:8000/metrics | grep JOURNAL_FORCE_WRITE_QUEUE_SIZE
curl -s http://<bookie-host>:8000/metrics | grep bookkeeper_server_ADD_ENTRY_IN_PROGRESS

# Check direct memory pressure (not exposed as a Pulsar Prometheus metric)
jcmd <broker-pid> VM.native_memory summary
# And the RSS vs heap gap
grep VmRSS /proc/<broker-pid>/status
jcmd <broker-pid> GC.heap_info

# Established socket count at the OS level
ss -tn state established '( dport = :6650 or sport = :6650 )' | wc -l

# Look for throttle and memory signatures in broker logs
grep -i "OutOfDirectMemoryError\|Direct buffer memory" /var/log/pulsar/broker.log

How to diagnose it

Work top-down from the throttle signal to the saturated resource. The order matters because the fix is completely different for each branch.

Confirm the throttle and its shape. Scrape pulsar_broker_throttled_connections over a few minutes. A brief spike during a bundle unload or deploy is normal churn. A sustained non-zero value, or a value that ratchets upward, is the incident. Note whether one broker is affected or several. One broker points at a hotspot or a local resource problem; several brokers points at cluster-wide capacity or a shared dependency like bookies.
Check publish latency and message rates. If pulsar_broker_publish_latency P99 rose before or alongside the throttling, and pulsar_rate_in is dropping, the write path is backing up. Go to step 4 (bookies). If publish latency is roughly flat but throttling is rising with connection count, suspect the broker’s own resources: continue to step 3.
Check broker-local resources: direct memory, connections, topic count. Compare process RSS to heap usage. A large gap means direct memory is heavily used; if it is approaching MaxDirectMemorySize (broker default 4GB), buffer allocation is slowing and queues back up into the throttle. Check pulsar_active_connections against its multi-week baseline: a monotonic climb indicates a connection leak, a sharp spike indicates a storm. Check pulsar_topics_count against peer brokers: a broker owning far more topics than its peers is a hotspot the load balancer has not corrected.
Check the bookie write path. On the bookies, look at bookie_journal_JOURNAL_SYNC P99, bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE, and bookkeeper_server_ADD_ENTRY_IN_PROGRESS. Journal sync latency spiking, a force-write queue that stays above zero, and an add-entry queue that does not drain within seconds of a burst all mean the storage layer cannot keep up, and the broker throttle is a downstream consequence. Verify with iostat -x 1 on the journal disk: high %util and await confirm disk saturation. Remember that one slow bookie in a write quorum stalls every topic writing through it.
Look for the churn amplifiers. Check pulsar_lb_unload_bundle_total for recent bundle unloads and pulsar_connection_created_total_count against pulsar_connection_closed_total_count for churn. A throttle that started right after a broker restart, a rolling upgrade, or a load-balancing event is likely a reconnect storm; the storm itself can keep the broker throttled even after the original event is over.
Decide: capacity or load. With the saturated resource identified, the decision tree is simple. Storage path saturated: fix or add bookies; adding brokers makes it worse. Broker direct memory or CPU saturated at legitimate load: add brokers or rebalance bundles so the load spreads. Topic count or connection count growing from leaks and abandoned clients: shed load, because adding hardware only delays the cliff.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`pulsar_broker_throttled_connections`	The protective throttle itself; should be zero	Any sustained non-zero value
`pulsar_active_connections`	FD and direct memory load; reveals leaks and storms	Unexplained growth over weeks, or spikes > 2x baseline
`pulsar_broker_publish_latency` P99	Tells you whether the write path or the broker is the bottleneck	> 2x rolling baseline
`bookie_journal_JOURNAL_SYNC` P99	The storage-side root cause behind many throttles	> 2x baseline, or above device-appropriate ceiling
`bookkeeper_server_ADD_ENTRY_IN_PROGRESS`	Bookie write queue depth; leads the latency spike	Queue not draining within ~30s after a burst
`pulsar_topics_count` per broker	Hotspot detection; topic-heavy brokers throttle first	One broker > 2x cluster mean
Direct buffer pool usage (JMX)	The invisible resource behind most connection throttling	> 75% of `MaxDirectMemorySize`
`pulsar_lb_unload_bundle_total`	Explains reconnect storms that amplify throttling	Sustained > 1/min outside maintenance

Fixes

If the bookie write path is the root cause

The throttle on the broker is collateral. Throttling-aware producers will not help; the journal disk is the wall.

Identify whether one bookie or all bookies show journal latency. One bookie means degraded hardware or a noisy neighbor on that disk; decommission or replace it. All bookies means aggregate write load exceeds the storage fleet: add bookies or reduce ingest.
Verify no other process shares the journal disk. Journal and ledger storage must be on separate devices; a shared journal disk produces exactly this failure shape.
As a temporary relief valve only, shape producer traffic (client-side rate limits or batching tuning) to let queues drain. This trades producer latency for stability.

If direct memory is the root cause

Reduce the memory consumers: close leaked connections (fix clients that do not close cleanly), reduce per-connection buffer pressure from oversized messages, and cap connection counts per client where the client library allows pooling.
Increase -XX:MaxDirectMemorySize if the host has physical headroom. Remember that direct memory is off-heap: heap plus direct memory plus JVM overhead must fit in the container or host memory limit, or you trade a throttle for an OOM kill. See the companion guide on the off-heap crash JVM heap dashboards never show.
Do not restart the broker as a first response. A restart drops all connections and triggers a mass reconnect, which is precisely the churn that feeds throttling. Restart only if the broker is already unresponsive.

If the broker is simply overloaded with topics or throughput

Rebalance: if one broker owns a disproportionate share of topics, trigger or wait for bundle unload to spread load, and check why the load balancer has not already done so (thresholds, sticky bundles).
Split hot namespaces into more bundles so the load balancer has finer-grained units to move.
Add brokers and let the load balancer migrate bundles onto them. This is the right answer when all brokers are uniformly hot at legitimate load.
Shed topics: delete test, temporary, and abandoned topics and subscriptions. High topic counts consume heap, FDs, and dispatcher state even at low traffic.

If a connection storm is the amplifier

Find the trigger: recent bundle unloads, a broker restart, a network blip, or a client deploy with an aggressive retry configuration.
Fix clients in retry loops. Exponential backoff and connection reuse matter more here than any broker-side change. A client that opens a new connection per operation instead of reusing one will keep the broker throttled indefinitely.
After the storm subsides, verify pulsar_active_connections returns to baseline. If it does not, you have a leak, not a storm.

Prevention

Alert on the signal, not the aftermath. Ticket on any sustained non-zero pulsar_broker_throttled_connections. It fires earlier than publish latency collapse and much earlier than client-visible outages.
Watch the leading indicators. Direct memory headroom (under 50 percent comfortable, over 75 percent urgent), active connection trend over weeks, and per-broker topic count variance all move before the throttle does. The Pulsar monitoring checklist covers the full baseline set.
Size bundles for balance. Enough bundles per namespace that no single bundle holds an outsized share of traffic, and load balancing thresholds that actually trigger before a broker is saturated.
Harden clients. Enforce connection reuse, bounded retry with backoff, and clean shutdown in your client libraries. Most throttled-connection incidents have a client behavior amplifier.
Capacity-plan the write path. Journal sync latency trending up over weeks is your runway signal for adding bookies before brokers ever throttle.

How Netdata helps

Netdata collects the broker Prometheus endpoint directly, so pulsar_broker_throttled_connections, pulsar_active_connections, pulsar_topics_count, and publish latency land on the same per-second timeline, which makes the sequence (what saturated first) visible instead of inferred.
Per-broker comparison is built in: overlaying topic count, connection count, and throttle events across brokers exposes the single hot broker that cluster averages hide.
Bookie metrics (journal sync latency, force write queue, add-entry in progress) are collected alongside broker metrics, so you can see the bookie stall precede the broker throttle in one view rather than correlating two dashboards by hand.
Host-level collection adds process RSS, CPU, and disk await/%util for the journal device, filling the direct-memory and disk-saturation gaps that the Pulsar metrics endpoint does not expose.
ML-based anomaly detection on connection and latency charts flags the slow, multi-week connection leak that threshold alerts tuned for spikes will miss.