Apache Pulsar broker GC death spiral: heap pressure, stop-the-world pauses, and lost topic ownership

A broker is losing topic ownership, recovering, and losing it again. Clients reconnect in bursts. Lookup failures climb. Publish latency is erratic even though traffic is flat and CPU looks busy for no obvious reason. The broker process never actually dies, which is why restarts and instance health checks keep “fixing” it for ten minutes at a time.

This is the GC death spiral: the broker JVM is under memory pressure, full GC pauses stop the world long enough to miss ZooKeeper heartbeats, the session expires, the broker is fenced and loses its bundles, and the resulting client reconnect and metadata churn allocates even more memory, which triggers more GC. The loop feeds itself. The signature is oscillation, not a clean failure.

This article covers how to recognize the pattern, how to confirm GC is the trigger rather than ZooKeeper itself, and how to break the loop by right-sizing heap against the managed ledger cache.

What this means

Every Pulsar broker holds an ephemeral ZooKeeper session that backs its bundle ownership registrations. If the broker stops heartbeating for longer than the session timeout (zooKeeperSessionTimeoutMillis, commonly 30-60s in production), ZooKeeper declares the session expired. The broker’s ownership claims disappear. Surviving brokers pick up the orphaned bundles, every client connected to topics in those bundles gets disconnected, and those clients immediately look up and reconnect to the new owners.

A stop-the-world pause does not need to last the full timeout to start this. A broker paused for several seconds comes back to a backlog of heartbeats, pending metadata writes, and queued client work, all of which it must process while still memory-constrained. If the heap is too small for the working set, the next full GC arrives before the broker has stabilized. Sessions flap, bundles churn, reconnect storms add allocation pressure, and the cycle repeats.

flowchart TD
  A[Heap pressure on broker] --> B[Frequent full GC, stop-the-world pauses]
  B --> C[Broker misses ZK heartbeats]
  C --> D[ZK session expires, broker fenced]
  D --> E[Bundle ownership lost, clients disconnected]
  E --> F[Clients reconnect to new owners, metadata storm]
  F --> G[Allocation spike on old and new owners]
  G --> A
  E --> H[Lookup failures and publish latency spikes]

Two version-dependent facts change how often you will see this:

• GC algorithm. ZGC has been the default broker GC since Pulsar 2.11+, with pause times far shorter than G1. A broker still running G1 (or an older release) is substantially more prone to session-killing pauses. Any full GC pause over 1 second warrants investigation regardless of collector.
• Session expiry behavior. On older releases, an expired ZK session could shut the broker down outright; newer releases attempt to re-establish the session. Reconnection is less brutal than a halt, but the broker still loses bundle ownership either way. Do not rely on reconnect behavior to save you from a heap problem. The zookeeperSessionExpiredPolicy default is reconnect from Pulsar 2.10 (older releases defaulted to shutdown).

Common causes

Note what is not on this list: ZooKeeper itself. A slow or overloaded ZK ensemble produces a similar-looking cascade (session expiries, bundle churn, reconnect storms), but the trigger is metadata store latency, not GC. The diagnosis section separates the two.

Quick checks

Run these on the affected broker. All are read-only.

# 1. Recent ZK session state changes (a "Session expired" line means ownership was lost)
grep -E "Session expired|Session established" /var/log/pulsar/broker.log | tail -20

# 2. GC pressure: frequency and pause times, sampled every second
jstat -gc <broker-pid> 1000

# 3. Current heap occupancy snapshot
jcmd <broker-pid> GC.heap_info

# 4. Session expiry and fencing events in the broker log
grep -E "Session expired|ConnectionLoss|SessionExpired" /var/log/pulsar/broker.log | tail -50

# 5. Bundle churn: is the load balancer actually moving things?
curl -s http://<broker-host>:8080/metrics | grep pulsar_lb_unload_bundle_total

# 6. Managed ledger cache pressure
curl -s http://<broker-host>:8080/metrics | grep pulsar_ml_cache

# 7. Connection oscillation (drops and reconnects)
curl -s http://<broker-host>:8080/metrics | grep pulsar_active_connections

# 8. Lookup health: are new clients failing to find topics?
curl -s http://<broker-host>:8080/metrics | grep pulsar_broker_lookup

If jstat and jcmd are not available in your runtime image, GC logging (-Xlog:gc* on modern JVMs) or a JMX exporter gives you the same information. Depending on version, the broker’s Prometheus endpoint may not expose the heap and GC detail you need; if those series are missing from /metrics, get them from JMX or GC logs. Pulsar’s /metrics endpoint includes standard jvm_* series (heap, GC, threads) on brokers.

How to diagnose it

1. Confirm the oscillation. Look at the broker log’s ZK session events and pulsar_lb_unload_bundle_total over the last few hours. The death spiral shows repeated session drops and a sustained unload rate well above baseline (steady state should be rare, under roughly 1 unload per hour outside maintenance). A single session drop with a clean recovery is not the spiral.

2. Correlate pauses with session loss. Align GC log timestamps (or jstat samples) against the “Session expired” lines in the broker log. If every session expiry is immediately preceded by a long GC pause, GC is the trigger. If expiries happen with a quiet GC log, suspect the metadata store instead and check ZK latency (echo stat | nc <zk-host> 2181, and the wchs four-letter command for watch count).

3. Determine which memory space is under pressure. Compare jcmd <pid> GC.heap_info (heap) against process RSS (grep VmRSS /proc/<pid>/status). A large gap between RSS and heap means direct memory is doing the damage, which points at Netty buffers and connection count, not the ledger cache.

4. Check the post-GC baseline. In jstat -gc output, watch old-generation occupancy after full collections. If the floor keeps rising across collections, you have a genuine leak or a structurally undersized heap, not a transient burst. A stable high floor with frequent full GCs points at cache-vs-heap misconfiguration.

5. Check cache configuration against memory limits. Read managedLedgerCacheSizeMB from broker.conf and the JVM flags (defaults are -Xms2g -Xmx2g -XX:MaxDirectMemorySize=4g; the cache default is derived from available direct memory). If cache evictions (pulsar_ml_cache_evictions) are high while heap is also pinned above 85%, the cache and everything else are fighting over the same memory budget.

6. Rule out a function or connector leak. If the affected broker hosts Pulsar Functions or IO connectors, check whether heap growth correlates with function activity. A leaking function is a common cause of the monotonic-growth variant of this pattern.

Metrics and signals to monitor

Escalation guidance: a single broker’s GC event often self-resolves and is a ticket. Page only when session expiries repeat across multiple brokers, lookup failures persist beyond 5 minutes, bundle churn stays elevated, and traffic is non-zero.

Fixes

Right-size the managed ledger cache against heap

This is the canonical fix and the most common root cause. The cache default is a fraction of direct memory, but on brokers where operators raised -Xmx without revisiting managedLedgerCacheSizeMB, or where many subscriptions inflate cursor state on heap, the combined footprint leaves no headroom for GC to work.

• Reduce managedLedgerCacheSizeMB so the cache, heap working set, and Netty buffers fit comfortably in physical memory with room to spare.
• Or raise heap, but only if RSS plus the new heap plus direct memory still fits on the host. Growing heap on a memory-pinned host converts a GC problem into an OOM kill.
• Tradeoff: a smaller cache means more reads hit bookies. Watch pulsar_ml_cache_misses_rate and bookie read latency after the change. For tailing workloads the hit rate should stay above 80%.

Move to ZGC if you are on G1

If the broker runs G1 (pre-2.11 defaults, or carried-over JVM flags), switching to ZGC removes the class of multi-second stop-the-world pauses that kill sessions. Verify the actual flags on the running process, not just the release notes; older pulsar_env.sh files have a way of surviving upgrades. ZGC does not fix an undersized heap; it only shortens the pauses. If the heap is genuinely too small, ZGC buys you shorter stalls and higher GC CPU, not stability.

Fix direct memory pressure

If diagnosis showed RSS far above heap with “Direct buffer memory” errors: check pulsar_active_connections for leaks (created vs closed counters diverging), and increase MaxDirectMemorySize if the host has physical headroom. Direct memory is not visible in heap metrics, so this variant hides from most dashboards.

Reduce per-broker load

If the broker simply owns too much: check pulsar_topics_count distribution across brokers. A broker holding more than 2x the cluster average is a hotspot candidate. Let the load balancer redistribute, and investigate why bundles are sticky (misconfigured shedding thresholds, or a hot bundle that needs splitting).

Break the cycle on the worst broker

Restarting the most affected broker is a last-resort measure, not a fix. It breaks the loop by forcing a clean ownership handoff and clearing allocation pressure, but if the memory budget is wrong, the spiral resumes within hours. Restart one broker at a time, watch bundle redistribution, and treat it as buying time to apply the cache/heap fix. Do not restart the whole fleet: mass restarts create a reconnection thundering herd and make everything worse.

Prevention

• Budget memory explicitly. Heap + direct memory (cache + Netty) + JVM overhead must fit in physical memory with headroom. In Kubernetes, set pod memory limits to cover heap + direct + roughly 20% overhead, or eviction becomes your failure mode.
• Alert on the leading indicators, not the cascade. Full GC > 1s, heap > 85% sustained, and rising post-GC old-gen occupancy all fire before the first session expires.
• Instrument JVM metrics. If your Pulsar version’s Prometheus endpoint does not expose heap, GC, and direct memory, run a JMX exporter or equivalent so pauses and occupancy are time-series, not log archaeology.
• Watch bundle unload rate in steady state. A low, stable baseline makes the spiral’s churn unmistakable.
• Audit JVM flags after upgrades. Confirm the intended GC is actually active and that no legacy G1 or CMS flags survived in environment scripts.
• Contain functions and connectors. A leaking function worker should not be able to take a broker into the spiral. Isolate or resource-limit function workloads where possible.
• Keep ZK healthy as a separate concern. Even a perfectly tuned broker spirals if session timeouts come from ZK latency instead of GC. Monitor ZK request latency and watch counts so you can tell the two cascades apart quickly.

How Netdata helps

The spiral is a correlation problem: five independent-looking symptoms that are actually one feedback loop. Netdata shortens the diagnosis by putting the loop’s signals on one timeline:

• Per-second JVM visibility. Heap usage, GC count, and GC pause duration at high resolution, so the pause that preceded each session drop is visible instead of averaged away.
• Process-level memory. RSS versus JVM heap side by side, which is how you distinguish the direct-memory variant from classic heap pressure.
• Pulsar metrics in context. pulsar_lb_unload_bundle_total, pulsar_active_connections, and pulsar_ml_cache_* scraped from the broker endpoint and correlated with the JVM data, so the oscillation signature (session drop, unload spike, reconnect burst, repeat) shows up as one pattern.
• System-level cross-checks. Host memory pressure, CPU spent in GC, and network reconnect bursts on the same dashboards, which helps rule ZK-side and host-side causes in or out.
• Anomaly detection on latency. Erratic pulsar_broker_publish_latency with flat traffic is flagged even when static thresholds would not fire, giving you an early nudge before fencing starts.

Related guides

• Apache Pulsar broker down: telling a dead broker from a fenced one
• Apache Pulsar monitoring checklist: the signals every production cluster needs