Apache Pulsar ZooKeeper session expired: brokers fenced and topics reassigned

A broker’s ZooKeeper session expires when it fails to send heartbeats within the session timeout window. The expiry triggers immediate fencing of the broker, loss of all namespace bundle ownership, and reassignment of those bundles to surviving brokers. Clients connected to the fenced broker see errors like “Topic is temporarily unavailable” or “Attempting to add producer to a fenced topic” until the topics are picked up elsewhere.

The most common root cause is a long JVM garbage collection pause on the broker that prevents ZK heartbeats from being sent in time. The second most common is elevated ZK server latency, where heartbeats are sent but not processed before the timeout. Both paths converge on the same outcome: the broker’s ephemeral znodes are deleted, ownership transfers, and the cluster enters a state of high metadata churn.

What this means

When metadataStoreSessionTimeoutMillis (default 30 seconds) elapses without a successful ZK heartbeat from the broker, ZK considers the session expired. The sequence that follows is deterministic:

flowchart TD
    A["GC pause or ZK latency
exceeds session timeout"] --> B["ZK expires broker session"]
    B --> C["EPHEMERAL znodes deleted
(bundle ownership lost)"]
    C --> D["Load manager detects
broker disappeared"]
    D --> E["Bundles reassigned
to surviving brokers"]
    E --> F["Original broker topics
fenced by BookKeeper"]
    F --> G["Clients see topic
unavailable errors"]
    G --> H["Clients reconnect to
new owning brokers"]
    H --> I["Metadata operation spike
on ZK from reconnections"]

In Pulsar 2.10 and later, the default zookeeperSessionExpiredPolicy is reconnect (PIP-118, issue #13304). The broker does not shut down on session expiry. It stays alive, continues serving existing topics where possible, and attempts to establish a new ZK session. Before 2.10, the default was shutdown, which caused the broker to halt via Runtime.halt(-1).

The reconnect policy avoids the restart cost and is generally better for cluster stability, but it has a known issue: when the broker re-establishes its session, it does not automatically unload topics that were reassigned to other brokers during the outage. Stale fenced topic objects can remain in memory, and clients routed to the original broker continue seeing “Topic is temporarily unavailable” until those topics are manually unloaded or the broker is restarted. The workaround is to set topicFencingTimeoutSeconds (a broker.conf key, default 0) to a small positive value (for example, 5), which causes the broker to forcefully close topics that remain fenced beyond that duration.

Common causes

Quick checks

Run these read-only checks to scope the incident. None are disruptive.

# Broker-to-ZK session state is not exposed as a Prometheus metric; check with the ZK four-letter command from each broker host
echo "stat" | nc <zk-host> 2181

# Check broker health (does it respond at all?)
curl -sf http://<broker-host>:8080/admin/v2/brokers/health

# Check ZK ensemble health from each ZK node
echo "stat" | nc <zk-host> 2181

# Check ZK watch count (watch explosions degrade ZK)
echo "wchs" | nc <zk-host> 2181

# Check ZK client latency metrics on the broker
curl -s http://<broker-host>:8080/metrics | grep -E "zk_(read|write)_latency|pulsar_metadata_store"

# Check bundle unload rate (ownership churn from reassignment)
curl -s http://<broker-host>:8080/metrics | grep pulsar_lb_unload_bundle_total

# Check lookup failures (new clients failing to find topic owners)
curl -s http://<broker-host>:8080/metrics | grep pulsar_broker_lookup

# Check active connections (reconnect storms cause spikes)
curl -s http://<broker-host>:8080/metrics | grep pulsar_active_connections

# Check broker GC behavior (1-second intervals)
jstat -gc <broker-pid> 1000

# Check current session timeout configuration
grep -E "metadataStoreSessionTimeoutMillis|zooKeeperSessionTimeoutMillis" <broker-conf-path>/broker.conf

How to diagnose it

1. Identify which brokers lost sessions. Run echo "stat" | nc <zk-host> 2181 from each broker host (or search broker logs for “Session expired”). A broker whose session expired is gone from ZK’s client list. One broker affected points to GC; multiple brokers affected simultaneously points to ZK.

2. Determine the root cause: GC or ZK latency. If one broker is affected, check its GC logs. A full GC pause exceeding 30 seconds will expire a session with the default timeout. If multiple brokers are affected simultaneously, the problem is on the ZK side. Broker logs will show “Session expired” or “Connection loss” events.

3. Check ZK server health. Run echo "stat" | nc <zk-host> 2181 on each ZK ensemble member. Look at outstanding requests, latency, and whether the node is in the ensemble. Check ZK transaction log disk I/O with iostat -x 1 on the ZK hosts. ZK is single-threaded for writes; a slow transaction log disk bottlenecks everything.

4. Check for watch explosions. Run echo "wchs" | nc <zk-host> 2181. A very high watch count indicates that consumer reconnection storms are generating excessive watch registrations, which degrades ZK performance. This is a common contributor to cascading session expirations.

5. Assess client impact. Check pulsar_broker_lookup_failures across brokers. Rising lookup failures mean new client connections cannot resolve topic ownership. Check publish and dispatch rates: if pulsar_rate_in drops to zero on affected topics, producers are blocked.

6. Check for fenced topics that are stuck. If the broker has reconnected (session state back to 1) but clients still report “Topic is temporarily unavailable” on that broker, the topic may be stuck in a fenced state. This is the known issue where fenced topics are not automatically unloaded after reconnect.

Metrics and signals to monitor

Fixes

Do NOT restart brokers

This is the most important guidance. Brokers running with zookeeperSessionExpiredPolicy=reconnect (the default in Pulsar 2.10+) will re-establish their ZK sessions on their own once the underlying problem (GC pressure or ZK latency) resolves. Restarting a broker during a ZK outage adds reconnection and re-ownership load, which can worsen the cascade.

The one exception: if a broker has reconnected but topics remain permanently fenced (the stale fenced topic issue), you may need to unload those topics or restart the broker as a last resort.

If GC is the root cause

1. Check GC logs for frequency and duration of full GC events. Look for pauses exceeding metadataStoreSessionTimeoutMillis.
2. If heap pressure is the issue, reduce managedLedgerCacheSizeMB or increase JVM heap.
3. If direct memory is exhausted, check pulsar_active_connections for leaks and consider increasing -XX:MaxDirectMemorySize.
4. ZGC has been the default GC algorithm since Pulsar 2.11.0 (2.10 and earlier default to G1GC). ZGC’s low-pause design reduces the risk of a GC-pause session expiry; Shenandoah is an alternative where your JDK supports it. Test under your workload before switching.

If ZK latency is the root cause

1. Check ZK transaction log disk health with iostat -x 1 on the ZK host. The transaction log must be on a dedicated disk.
2. If a ZK ensemble member has failed, check quorum status and replace or restart it.
3. If watch count is the problem, reduce topic-level watch pressure by consolidating subscriptions or reducing topic count per namespace.
4. As a temporary measure during an active cascade, increasing metadataStoreSessionTimeoutMillis can buy time by allowing brokers to survive longer ZK latency spikes. This is a bandage, not a fix.

If topics are stuck fenced after reconnect

1. Set topicFencingTimeoutSeconds to a small positive value (for example, 5). This causes the broker to forcefully close topics that remain fenced for that duration, allowing clients to redo lookups and connect to the correct owner broker.
2. Manually unload specific stuck topics using pulsar-admin topics unload.
3. On versions before 2.10.6 / 2.11.3 / 3.0.2 (PR #21035), BookKeeper ensemble information may be stale after ZK client reconnect. Upgrading resolves this.

If proxy lookup cache is stale

After topics are reassigned, the Pulsar proxy may cache the old broker’s service URL. Clients can see handshake failures for up to approximately 51 seconds (the most common recovery time documented in issue #9297, with worst cases around 1m40s). There is no manual fix beyond waiting for the cache to expire.

Prevention

Tune the session timeout deliberately. The default metadataStoreSessionTimeoutMillis is 30 seconds. Too low and normal GC pauses cause spurious session expirations. Too high and failure detection is slow, meaning a genuinely dead broker holds its bundles longer before they are reassigned. Most production deployments use 30 to 60 seconds. Match the timeout to your GC profile: if your worst-case full GC pause is 20 seconds, a 30-second timeout is marginal. Consider 60 seconds or reduce GC pause times.

Set topicFencingTimeoutSeconds. With the default value of 0 (disabled), fenced topics after reconnect are never automatically cleaned up. Setting this to a small value like 5 prevents the stale fenced topic issue from affecting clients after recovery.

Monitor ZK as a first-class component. ZK latency is the leading indicator for cluster-wide cascades. Alert on sustained average latency above 50ms and treat 100ms as critical. Track watch count with echo "wchs" | nc <zk-host> 2181.

Monitor broker GC pauses. Correlate GC pause duration with ZK session state. If sessions expire during GC pauses, the heap or GC algorithm needs attention before the next cascade.

Avoid rolling operations during ZK stress. Broker restarts trigger bundle redistribution, which generates ZK metadata operations. If ZK is already under stress, rolling restarts can push it over the edge into a session expiry cascade.

How Netdata helps

• Per-second pulsar_zookeeper_connected collection catches session state transitions as they happen. Correlating the exact timestamp of a session drop with GC pause data from the same host identifies the root cause.
• ZK latency metrics alongside broker metrics on the same dashboard let you distinguish a slow ZK server from a broker that cannot reach ZK.
• GC pause duration tracking via JVM integration shows whether sessions are expiring because of GC, not ZK.
• Bundle unload rate and lookup failure metrics show the downstream blast radius: how many bundles are churning and how many client connections are failing.
• Anomaly detection on ZK latency, active connections, and bundle unload rate flags unusual patterns that can precede a cascade.

Related guides

• Apache Pulsar active connections climbing: connection leaks and file descriptor exhaustion
• Apache Pulsar bookie add-entry queue not draining: writes arriving faster than the disk can commit
• Apache Pulsar AutoRecovery stalled: under-repuplicated ledgers that never heal
• Apache Pulsar backlog age vs size: the latency depth alone cannot show
• Apache Pulsar backlog quota exceeded: producers held or rejected when consumers stall
• Apache Pulsar bookie disk filling: runway to read-only and how to reclaim space
• Apache Pulsar bookie failure cascade: recovery I/O that topples surviving bookies
• Apache Pulsar bookie read latency high: catch-up reads competing with the write path
• Apache Pulsar bookie read-only: disk full and bookie_SERVER_STATUS at zero
• Apache Pulsar broker down: telling a dead broker from a fenced one
• Apache Pulsar broker GC death spiral: heap pressure, stop-the-world pauses, and lost topic ownership
• Apache Pulsar broker lookup failures: new clients cannot find their topic