A broker’s ZooKeeper session expires when it fails to send heartbeats within the session timeout window. The expiry triggers immediate fencing of the broker, loss of all namespace bundle ownership, and reassignment of those bundles to surviving brokers. Clients connected to the fenced broker see errors like “Topic is temporarily unavailable” or “Attempting to add producer to a fenced topic” until the topics are picked up elsewhere.

The most common root cause is a long JVM garbage collection pause on the broker that prevents ZK heartbeats from being sent in time. The second most common is elevated ZK server latency, where heartbeats are sent but not processed before the timeout. Both paths converge on the same outcome: the broker’s ephemeral znodes are deleted, ownership transfers, and the cluster enters a state of high metadata churn.

What this means

When metadataStoreSessionTimeoutMillis (default 30 seconds) elapses without a successful ZK heartbeat from the broker, ZK considers the session expired. The sequence that follows is deterministic:

flowchart TD
    A["GC pause or ZK latency
exceeds session timeout"] --> B["ZK expires broker session"]
    B --> C["EPHEMERAL znodes deleted
(bundle ownership lost)"]
    C --> D["Load manager detects
broker disappeared"]
    D --> E["Bundles reassigned
to surviving brokers"]
    E --> F["Original broker topics
fenced by BookKeeper"]
    F --> G["Clients see topic
unavailable errors"]
    G --> H["Clients reconnect to
new owning brokers"]
    H --> I["Metadata operation spike
on ZK from reconnections"]

In Pulsar 2.10 and later, the default zookeeperSessionExpiredPolicy is reconnect. The broker does not shut down on session expiry. It stays alive, continues serving existing topics where possible, and attempts to establish a new ZK session. Before 2.10, the default was shutdown, which caused the broker to halt via Runtime.halt(-1).

The reconnect policy avoids the restart cost and is generally better for cluster stability, but it has a known issue: when the broker re-establishes its session, it does not automatically unload topics that were reassigned to other brokers during the outage. Stale fenced topic objects can remain in memory, and clients routed to the original broker continue seeing “Topic is temporarily unavailable” until those topics are manually unloaded or the broker is restarted. The workaround is to set topicFencingTimeoutSeconds to a small positive value (for example, 5), which causes the broker to forcefully close topics that remain fenced beyond that duration.

Common causes

CauseWhat it looks likeFirst thing to check
Long GC pause on brokerpulsar_zookeeper_connected drops to 0 on one broker; GC logs show pauses over 1s; session expires silentlyBroker GC logs or jstat -gc output
ZK server latency spikeMultiple brokers lose sessions simultaneously; ZK latency over 50ms; outstanding requests growing`echo “stat”
ZK ensemble quorum lossAll brokers lose sessions; ZK cluster cannot maintain quorumZK process health on each ensemble member
Network partition to ZKBrokers on one network segment lose sessions while others do notNetwork connectivity between broker and ZK hosts
Session timeout too lowFrequent session expirations during normal GC pauses; no correlation with ZK issuesCurrent metadataStoreSessionTimeoutMillis value

Quick checks

Run these read-only checks to scope the incident. None are disruptive.

# Check broker ZK connection state (1 = connected, 0 = disconnected)
curl -s http://<broker-host>:8080/metrics | grep pulsar_zookeeper_connected

# Check broker health (does it respond at all?)
curl -sf http://<broker-host>:8080/admin/v2/brokers/health

# Check ZK ensemble health from each ZK node
echo "stat" | nc <zk-host> 2181

# Check ZK watch count (watch explosions degrade ZK)
echo "wchs" | nc <zk-host> 2181

# Check ZK-related latency metrics on the broker
curl -s http://<broker-host>:8080/metrics | grep -i zookeeper

# Check bundle unload rate (ownership churn from reassignment)
curl -s http://<broker-host>:8080/metrics | grep pulsar_lb_unload_bundle_total

# Check lookup failures (new clients failing to find topic owners)
curl -s http://<broker-host>:8080/metrics | grep pulsar_broker_lookup

# Check active connections (reconnect storms cause spikes)
curl -s http://<broker-host>:8080/metrics | grep pulsar_active_connections

# Check broker GC behavior (1-second intervals)
jstat -gc <broker-pid> 1000

# Check current session timeout configuration
grep -E "metadataStoreSessionTimeoutMillis|zooKeeperSessionTimeoutMillis" <broker-conf-path>/broker.conf

How to diagnose it

  1. Identify which brokers lost sessions. Grep pulsar_zookeeper_connected across all brokers. Any broker showing 0 has lost or is losing its session. One broker affected points to GC; multiple brokers affected simultaneously points to ZK.

  2. Determine the root cause: GC or ZK latency. If one broker is affected, check its GC logs. A full GC pause exceeding 30 seconds will expire a session with the default timeout. If multiple brokers are affected simultaneously, the problem is on the ZK side. Broker logs will show “Session expired” or “Connection loss” events.

  3. Check ZK server health. Run echo "stat" | nc <zk-host> 2181 on each ZK ensemble member. Look at outstanding requests, latency, and whether the node is in the ensemble. Check ZK transaction log disk I/O with iostat -x 1 on the ZK hosts. ZK is single-threaded for writes; a slow transaction log disk bottlenecks everything.

  4. Check for watch explosions. Run echo "wchs" | nc <zk-host> 2181. A very high watch count indicates that consumer reconnection storms are generating excessive watch registrations, which degrades ZK performance. This is a common contributor to cascading session expirations.

  5. Assess client impact. Check pulsar_broker_lookup_failures across brokers. Rising lookup failures mean new client connections cannot resolve topic ownership. Check publish and dispatch rates: if pulsar_rate_in drops to zero on affected topics, producers are blocked.

  6. Check for fenced topics that are stuck. If the broker has reconnected (session state back to 1) but clients still report “Topic is temporarily unavailable” on that broker, the topic may be stuck in a fenced state. This is the known issue where fenced topics are not automatically unloaded after reconnect.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
pulsar_zookeeper_connectedBinary indicator of broker-to-ZK session health. Any transition from 1 to 0 is an incident.Value drops to 0 on any broker
ZK request latencyLeading indicator for session expiry. When request queueing grows, heartbeats may not be processed within the timeout window.Sustained average over 50ms; 100ms+ means the ZK ensemble cannot keep up and sessions are at risk
ZK watch count (wchs)Watch explosions degrade ZK throughput, contributing to latency spikes that expire sessions.Count growing without bound during reconnect storms
Broker GC pause durationThe most common root cause. Pauses longer than the session timeout guarantee expiry.Any full GC pause over 1 second
pulsar_lb_unload_bundle_totalBundle ownership churn indicates the cluster is actively reassigning work after fencing.Rate over 1 per minute outside maintenance windows
pulsar_broker_lookup_failuresNew clients cannot find topic owners during the reassignment window.Failure rate over 1% of total lookups sustained
pulsar_active_connectionsConnection count fluctuations indicate reconnect storms after fencing.Rapid drops followed by spikes
pulsar_broker_publish_latencyLatency rises during ownership transitions and remains elevated if the cascade feeds itself.P99 over 2x rolling baseline

Fixes

Do NOT restart brokers

This is the most important guidance. Brokers running with zookeeperSessionExpiredPolicy=reconnect (the default in Pulsar 2.10+) will re-establish their ZK sessions on their own once the underlying problem (GC pressure or ZK latency) resolves. Restarting a broker during a ZK outage adds reconnection and re-ownership load, which can worsen the cascade.

The one exception: if a broker has reconnected but topics remain permanently fenced (the stale fenced topic issue), you may need to unload those topics or restart the broker as a last resort.

If GC is the root cause

  1. Check GC logs for frequency and duration of full GC events. Look for pauses exceeding metadataStoreSessionTimeoutMillis.
  2. If heap pressure is the issue, reduce managedLedgerCacheSizeMB or increase JVM heap.
  3. If direct memory is exhausted, check pulsar_active_connections for leaks and consider increasing -XX:MaxDirectMemorySize.
  4. Consider switching to a lower-pause GC algorithm. G1GC is the standard; ZGC or Shenandoah can reduce pause times but require testing under your workload.

If ZK latency is the root cause

  1. Check ZK transaction log disk health with iostat -x 1 on the ZK host. The transaction log must be on a dedicated disk.
  2. If a ZK ensemble member has failed, check quorum status and replace or restart it.
  3. If watch count is the problem, reduce topic-level watch pressure by consolidating subscriptions or reducing topic count per namespace.
  4. As a temporary measure during an active cascade, increasing metadataStoreSessionTimeoutMillis can buy time by allowing brokers to survive longer ZK latency spikes. This is a bandage, not a fix.

If topics are stuck fenced after reconnect

  1. Set topicFencingTimeoutSeconds to a small positive value (for example, 5). This causes the broker to forcefully close topics that remain fenced for that duration, allowing clients to redo lookups and connect to the correct owner broker.
  2. Manually unload specific stuck topics using pulsar-admin topics unload.
  3. On versions before the fix in PR #21035 , BookKeeper ensemble information may be stale after ZK client reconnect. Upgrading resolves this.

If proxy lookup cache is stale

After topics are reassigned, the Pulsar proxy may cache the old broker’s service URL. Clients can see handshake failures for up to approximately 51 seconds . This is documented in issue #9297 . There is no manual fix beyond waiting for the cache to expire.

Prevention

Tune the session timeout deliberately. The default metadataStoreSessionTimeoutMillis is 30 seconds. Too low and normal GC pauses cause spurious session expirations. Too high and failure detection is slow, meaning a genuinely dead broker holds its bundles longer before they are reassigned. Most production deployments use 30 to 60 seconds. Match the timeout to your GC profile: if your worst-case full GC pause is 20 seconds, a 30-second timeout is marginal. Consider 60 seconds or reduce GC pause times.

Set topicFencingTimeoutSeconds. With the default value of 0 (disabled), fenced topics after reconnect are never automatically cleaned up. Setting this to a small value like 5 prevents the stale fenced topic issue from affecting clients after recovery.

Monitor ZK as a first-class component. ZK latency is the leading indicator for cluster-wide cascades. Alert on sustained average latency above 50ms and treat 100ms as critical. Track watch count with echo "wchs" | nc <zk-host> 2181.

Monitor broker GC pauses. Correlate GC pause duration with ZK session state. If sessions expire during GC pauses, the heap or GC algorithm needs attention before the next cascade.

Avoid rolling operations during ZK stress. Broker restarts trigger bundle redistribution, which generates ZK metadata operations. If ZK is already under stress, rolling restarts can push it over the edge into a session expiry cascade.

How Netdata helps

  • Per-second pulsar_zookeeper_connected collection catches session state transitions as they happen. Correlating the exact timestamp of a session drop with GC pause data from the same host identifies the root cause.
  • ZK latency metrics alongside broker metrics on the same dashboard let you distinguish a slow ZK server from a broker that cannot reach ZK.
  • GC pause duration tracking via JVM integration shows whether sessions are expiring because of GC, not ZK.
  • Bundle unload rate and lookup failure metrics show the downstream blast radius: how many bundles are churning and how many client connections are failing.
  • Anomaly detection on ZK latency, active connections, and bundle unload rate flags unusual patterns that can precede a cascade.