$ guides / apache-pulsar / apache-pulsar-zookeeper-session-cascade ▌

Operations Guides

Apache Pulsar ZooKeeper session cascade: reconnect storms and the thundering herd

Multiple brokers lose their ZooKeeper sessions within seconds of each other. Bundle ownership churns across the cluster as surviving brokers acquire orphaned bundles. Every connected producer and consumer simultaneously discovers its topic has moved or its connection has dropped, triggering a mass reconnection event. Each reconnection generates topic lookups, ephemeral node registrations, and watch re-establishments that hit the already overloaded ZK ensemble. Latency rises further, more sessions expire, and the cascade tightens.

The critical diagnostic signal: broker processes are running, their HTTP admin endpoints respond, and their JVMs show no distress. The problem is in the coordination layer, not the serving layer. Operators who mistake this for a broker problem and start restarting brokers make it worse. Each restart generates metadata operations, client reconnections, and bundle reassignments. The correct first response is to stabilize ZK. Brokers will recover sessions and re-acquire bundles once ZK is healthy.

What this means

When ZK request latency exceeds the broker session timeout window, brokers cannot maintain their heartbeat sessions. Each expired session causes the broker to lose ownership of all namespace bundles assigned to it. Since Pulsar 2.10 (PIP-118), the default metadataSessionExpiredPolicy is reconnect: the broker does not shut down on session expiry but attempts to re-establish its session and re-acquire bundles. The data plane may continue serving already-loaded topics, but new topic lookups, topic creation, and metadata-dependent operations stall until the session is restored.

When a session is re-established, the broker’s local ZK cache is invalidated, forcing re-reads of all metadata. On a cluster with thousands of topics, this means a burst of metadata reads on top of already elevated ZK load.

The thundering herd is the second phase. Every client connected to a broker that lost and regained ownership simultaneously disconnects and reconnects. Each client reconnection triggers a topic lookup (ZK read), a connection registration (ZK write), and watch re-registration. With tens of thousands of clients, this synchronized metadata storm extends ZK overload well beyond the original trigger.

flowchart TD
    A["ZK overload"] --> B["ZK latency spike"]
    B --> C["Broker sessions expire"]
    C --> D["Bundle ownership churn"]
    D --> E["Thundering herd
client reconnects"]
    E --> F["Metadata storm on ZK"]
    F --> A

The feedback loop is what makes this a page-level incident. It does not self-correct: each phase of recovery generates load that re-triggers the overload. Breaking the loop requires stabilizing ZK directly, not cycling brokers.

Key distinction from broker-level failures: broker processes are alive and responding to local health checks. If you see broker process health failing alongside ZK latency spikes, you may have a compound failure (a GC death spiral triggering the cascade), but the first response is still to stabilize ZK. See Apache Pulsar broker GC death spiral for that pattern.

Common causes

Cause	What it looks like	First thing to check
ZK transaction-log disk I/O saturation	ZK latency rises; `iostat` shows high `%util` and `await` on the ZK data disk	`iostat -x 1` on each ZK node, focus on the transaction log mount
ZK JVM GC pauses	ZK latency spikes correlate with GC events; ZK process briefly unresponsive	ZK JVM GC logs; check for long full GC pauses
ZK snapshot or compaction pause	Periodic latency spikes at regular intervals matching snapshot frequency	ZK server logs for snapshot timing; snapshot file sizes
Watch explosion	ZK watch count grows rapidly; often triggered by mass client reconnection	`echo wchs \| nc <zk-host> 2181` on each ZK node
ZK ensemble member failure	Quorum degraded; one ZK node unreachable; remaining nodes handle full load	`echo stat \| nc <zk-host> 2181` per node; check leader/follower status and mode
Metadata operation surge	Large batch topic creation, namespace policy changes, or ledger metadata churn	Broker logs for bulk admin operations; topic count growth rate

Quick checks

Run these before changing anything. All are read-only.

# Check ZK ensemble status on each node (latency, mode, outstanding requests)
echo stat | nc <zk-host> 2181

# Check ZK watch count (watch explosions are a cascade trigger and amplifier)
echo wchs | nc <zk-host> 2181

# Check ZK health via admin server (alternative if four-letter commands are disabled)
curl -s http://<zk-host>:8080/commands/stats

# Check broker ZK connectivity across the fleet (1 = connected, 0 = disconnected)
curl -s http://<broker-host>:8080/metrics | grep pulsar_zookeeper_connected

# Check bundle unload rate (elevated during ownership churn)
curl -s http://<broker-host>:8080/metrics | grep pulsar_lb_unload_bundle

# Check lookup failures (new clients cannot find their topics)
curl -s http://<broker-host>:8080/metrics | grep pulsar_broker_lookup

# Check active connections (drop then spike = thundering herd signature)
curl -s http://<broker-host>:8080/metrics | grep pulsar_active_connections

# Check ZK transaction-log disk I/O on each ZK node
iostat -x 1

How to diagnose it

Verify broker processes are alive. Check GET http://<broker-host>:8080/admin/v2/brokers/health on each broker. If brokers respond with HTTP 200, the problem is not a broker crash. If some brokers are down, you may have a compound failure, but proceed with ZK diagnosis first.
Check ZK ensemble health on every node. Run echo stat | nc <zk-host> 2181 on each ZK server. Look for: mode (leader vs follower), latency (min/avg/max), outstanding requests, and node count. Any node showing high latency or high outstanding requests is under pressure. If one node is unreachable, the ensemble has lost a member and quorum may be degraded.
Check ZK transaction-log disk I/O. Run iostat -x 1 on each ZK node. Focus on the disk holding the ZK data directory (where the transaction log lives). High %util or elevated await means ZK cannot commit transactions fast enough. This is often the root cause.
Check for watch explosions. Run echo wchs | nc <zk-host> 2181 on each ZK node. Compare the total watch count against your baseline. A sudden spike in watches during the incident means client reconnections are amplifying ZK load through watch re-registration.
Confirm simultaneous session expiry. Check pulsar_zookeeper_connected across all brokers. Multiple brokers transitioning from 1 to 0 within the same window confirms the cascade signature, as opposed to a single-broker GC issue.
Confirm the thundering herd. Check pulsar_active_connections across brokers. The pattern is a sharp drop (clients disconnected during bundle ownership loss) followed by a spike (all clients reconnecting at once). Correlate with pulsar_broker_lookup_failures rising during the reconnection window.
Check for the original trigger. After confirming the cascade, look for what started it. Check ZK JVM GC logs, ZK snapshot timing, recent bulk admin operations, and any ZK member that recently restarted or lost network connectivity.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
ZK request latency (avg and max)	Leading indicator for all cascade symptoms; degrades before brokers show distress	Sustained avg > 50ms; max > 100ms means failure is imminent within minutes
ZK watch count (`wchs` output)	Watch explosions amplify metadata load and stress the ZK event thread	Rapid growth from baseline; correlates with client reconnection storms
`pulsar_zookeeper_connected` per broker	Binary session state; transitions to 0 mean the broker lost its session	Multiple brokers transitioning to 0 simultaneously
`pulsar_lb_unload_bundle_total`	Bundle ownership churn generates metadata writes that add ZK load	Sustained unload rate above baseline outside maintenance windows
`pulsar_broker_lookup_failures`	New connections cannot resolve topic ownership when ZK is slow or sessions are expired	Failure rate above 1% of total lookups, sustained
`pulsar_active_connections`	Connection count reveals the thundering herd: mass disconnect followed by mass reconnect	Sharp drop then spike across multiple brokers simultaneously
ZK outstanding requests	Queue depth shows ZK cannot process requests as fast as they arrive	Sustained non-zero; growing queue means ZK is falling further behind

Fixes

Stabilize ZooKeeper first

The cascade feeds on itself. Your priority is to reduce ZK load or improve ZK capacity until the feedback loop breaks. Do not touch brokers until ZK latency returns to baseline (under 10ms average).

ZK transaction-log disk I/O

If iostat shows the ZK data disk is saturated, ZK cannot commit transactions fast enough. Short-term: identify and stop any non-ZK processes writing to that disk. Long-term: move the ZK transaction log to a dedicated, faster device (NVMe preferred). The ZK transaction log is the write-ahead log for all metadata changes; it is fsync-heavy and latency-sensitive, similar to a BookKeeper journal disk.

Temporarily raise session timeout

Increasing metadataStoreSessionTimeoutMillis gives brokers more time before sessions expire during transient ZK latency spikes. The default is 30000 (30 seconds). Raising it to 60 seconds during an active incident can prevent further session expirations while you address the root cause.

ZooKeeper enforces server-side limits on the negotiated session timeout. The server caps the timeout at maxSessionTimeout, which defaults to 20 * tickTime. With the default tickTime of 2000ms, maxSessionTimeout is 40000ms. If you set metadataStoreSessionTimeoutMillis=60000 on the broker, ZK silently caps the negotiated timeout to 40 seconds. To get a 60-second session, you must also increase tickTime or set maxSessionTimeout explicitly in the ZK server configuration (zoo.cfg) and restart the ZK ensemble.

The lower bound is minSessionTimeout, which defaults to 2 * tickTime (4000ms with default tickTime).

Reduce metadata load during the cascade

If ZK is overloaded by metadata operations from bundle churn and client reconnections, reduce the metadata write rate:

Pause the load balancer to stop bundle unload and reassignment operations. This is a dynamic config change, no restart needed:

# DISRUPTIVE: disables automatic load balancing cluster-wide
# Re-enable after ZK stabilizes
pulsar-admin brokers update-dynamic-config \
  --config loadManagerClassName \
  --config-value org.apache.pulsar.broker.loadbalance.NoopLoadManager

Stop any bulk admin operations (mass topic creation, namespace policy changes, large-scale subscription cleanup).
If bookie registration is flapping (bookies losing and regaining ZK sessions), consider pausing auto-recovery to reduce ledger metadata churn.

Do not restart brokers

Brokers with the reconnect session expired policy will re-establish their ZK sessions and re-acquire bundles automatically once ZK recovers. Restarting a broker forces it to drop all client connections, close all managed ledgers, and re-acquire all bundles from scratch. During a ZK cascade, this adds a burst of metadata operations that makes the overload worse.

The one exception: if you are running a version before Pulsar 2.10 where the default session expired policy is shutdown, brokers will have halted themselves on session expiry. In that case, you have no choice but to restart them, but do so in a controlled, staggered manner after ZK is stable.

Check for session timeout mismatch with BookKeeper client

Before Pulsar 4.2.0, the metadataStoreSessionTimeoutMillis value was not properly propagated to the BookKeeper client configuration. The BookKeeper client could use its own default ZK session timeout (typically shorter than the broker’s), causing the BK client to lose its ZK session before the broker’s metadata store session expired. This can trigger ledger fencing failures and additional metadata churn that worsens the cascade. If you are on a version before 4.2.0, verify the BK client timeout is configured to match the broker timeout. Pulsar 4.2.0 fixes this propagation (issue #24624).

Prevention

Monitor ZK request latency as a first-class signal. Alert on sustained average latency above 10ms and treat anything above 50ms as urgent.
Monitor ZK watch count. Track the wchs output over time and alert on rapid growth.
Ensure dedicated, fast storage for the ZK transaction log. The transaction log disk is the ZK write bottleneck, analogous to a BookKeeper journal disk.
Verify the negotiated session timeout. Check that metadataStoreSessionTimeoutMillis is not silently capped by ZK server maxSessionTimeout. Adjust tickTime in zoo.cfg if needed.
Upgrade to Pulsar 4.2.0 or later if the BookKeeper client session timeout propagation fix applies to your deployment.
Watch for slow ZK degradation over days and weeks. ZK latency can creep from 1ms to 50ms gradually as topic count and metadata volume grow. Alert on the trend, not just the absolute value.
Confirm the reconnect session expired policy is active. On Pulsar 2.10+, this is the default. On older versions, brokers halt on session expiry, which turns a recoverable cascade into a full fleet restart.

How Netdata helps

Per-second ZK latency metrics expose the initial spike before session expirations cascade. Standard 15-second scrape intervals can miss it.
ZK latency and pulsar_zookeeper_connected across all brokers in one view makes simultaneous session expiration visible, distinguishing a ZK cascade from individual broker failures.
Bundle unload rate and lookup failure metrics on the same timeline as ZK latency show the relationship between metadata overload and client impact.
ML anomaly detection on ZK latency catches gradual degradation (1ms to 50ms over hours or days) before it reaches session-timeout territory.
Active connection count correlation with ZK latency shows the thundering herd pattern: sharp drop then spike, synchronized across brokers.