Multiple brokers lose their ZooKeeper sessions within seconds of each other. Bundle ownership churns across the cluster as surviving brokers acquire orphaned bundles. Every connected producer and consumer simultaneously discovers its topic has moved or its connection has dropped, triggering a mass reconnection event. Each reconnection generates topic lookups, ephemeral node registrations, and watch re-establishments that hit the already overloaded ZK ensemble. Latency rises further, more sessions expire, and the cascade tightens.
The critical diagnostic signal: broker processes are running, their HTTP admin endpoints respond, and their JVMs show no distress. The problem is in the coordination layer, not the serving layer. Operators who mistake this for a broker problem and start restarting brokers make it worse. Each restart generates metadata operations, client reconnections, and bundle reassignments. The correct first response is to stabilize ZK. Brokers will recover sessions and re-acquire bundles once ZK is healthy.
What this means
When ZK request latency exceeds the broker session timeout window, brokers cannot maintain their heartbeat sessions. Each expired session causes the broker to lose ownership of all namespace bundles assigned to it. Since Pulsar 2.10 (PIP-118), the default metadataSessionExpiredPolicy is reconnect: the broker does not shut down on session expiry but attempts to re-establish its session and re-acquire bundles. The data plane may continue serving already-loaded topics, but new topic lookups, topic creation, and metadata-dependent operations stall until the session is restored.
When a session is re-established, the broker’s local ZK cache is invalidated, forcing re-reads of all metadata. On a cluster with thousands of topics, this means a burst of metadata reads on top of already elevated ZK load.
The thundering herd is the second phase. Every client connected to a broker that lost and regained ownership simultaneously disconnects and reconnects. Each client reconnection triggers a topic lookup (ZK read), a connection registration (ZK write), and watch re-registration. With tens of thousands of clients, this synchronized metadata storm extends ZK overload well beyond the original trigger.
flowchart TD
A["ZK overload"] --> B["ZK latency spike"]
B --> C["Broker sessions expire"]
C --> D["Bundle ownership churn"]
D --> E["Thundering herd
client reconnects"]
E --> F["Metadata storm on ZK"]
F --> AThe feedback loop is what makes this a page-level incident. It does not self-correct: each phase of recovery generates load that re-triggers the overload. Breaking the loop requires stabilizing ZK directly, not cycling brokers.
Key distinction from broker-level failures: broker processes are alive and responding to local health checks. If you see broker process health failing alongside ZK latency spikes, you may have a compound failure (a GC death spiral triggering the cascade), but the first response is still to stabilize ZK. See Apache Pulsar broker GC death spiral for that pattern.
Common causes
| Cause | What it looks like | First thing to check |
|---|---|---|
| ZK transaction-log disk I/O saturation | ZK latency rises; iostat shows high %util and await on the ZK data disk | iostat -x 1 on each ZK node, focus on the transaction log mount |
| ZK JVM GC pauses | ZK latency spikes correlate with GC events; ZK process briefly unresponsive | ZK JVM GC logs; check for long full GC pauses |
| ZK snapshot or compaction pause | Periodic latency spikes at regular intervals matching snapshot frequency | ZK server logs for snapshot timing; snapshot file sizes |
| Watch explosion | ZK watch count grows rapidly; often triggered by mass client reconnection | echo wchs | nc <zk-host> 2181 on each ZK node |
| ZK ensemble member failure | Quorum degraded; one ZK node unreachable; remaining nodes handle full load | echo stat | nc <zk-host> 2181 per node; check leader/follower status and mode |
| Metadata operation surge | Large batch topic creation, namespace policy changes, or ledger metadata churn | Broker logs for bulk admin operations; topic count growth rate |
Quick checks
Run these before changing anything. All are read-only.
# Check ZK ensemble status on each node (latency, mode, outstanding requests)
echo stat | nc <zk-host> 2181
# Check ZK watch count (watch explosions are a cascade trigger and amplifier)
echo wchs | nc <zk-host> 2181
# Check ZK health via admin server (alternative if four-letter commands are disabled)
curl -s http://<zk-host>:8080/commands/stats
# Check broker ZK connectivity across the fleet (1 = connected, 0 = disconnected)
curl -s http://<broker-host>:8080/metrics | grep pulsar_zookeeper_connected
# Check bundle unload rate (elevated during ownership churn)
curl -s http://<broker-host>:8080/metrics | grep pulsar_lb_unload_bundle
# Check lookup failures (new clients cannot find their topics)
curl -s http://<broker-host>:8080/metrics | grep pulsar_broker_lookup
# Check active connections (drop then spike = thundering herd signature)
curl -s http://<broker-host>:8080/metrics | grep pulsar_active_connections
# Check ZK transaction-log disk I/O on each ZK node
iostat -x 1
How to diagnose it
Verify broker processes are alive. Check
GET http://<broker-host>:8080/admin/v2/brokers/healthon each broker. If brokers respond with HTTP 200, the problem is not a broker crash. If some brokers are down, you may have a compound failure, but proceed with ZK diagnosis first.Check ZK ensemble health on every node. Run
echo stat | nc <zk-host> 2181on each ZK server. Look for: mode (leader vs follower), latency (min/avg/max), outstanding requests, and node count. Any node showing high latency or high outstanding requests is under pressure. If one node is unreachable, the ensemble has lost a member and quorum may be degraded.Check ZK transaction-log disk I/O. Run
iostat -x 1on each ZK node. Focus on the disk holding the ZK data directory (where the transaction log lives). High%utilor elevatedawaitmeans ZK cannot commit transactions fast enough. This is often the root cause.Check for watch explosions. Run
echo wchs | nc <zk-host> 2181on each ZK node. Compare the total watch count against your baseline. A sudden spike in watches during the incident means client reconnections are amplifying ZK load through watch re-registration.Confirm simultaneous session expiry. Check
pulsar_zookeeper_connectedacross all brokers. Multiple brokers transitioning from 1 to 0 within the same window confirms the cascade signature, as opposed to a single-broker GC issue.Confirm the thundering herd. Check
pulsar_active_connectionsacross brokers. The pattern is a sharp drop (clients disconnected during bundle ownership loss) followed by a spike (all clients reconnecting at once). Correlate withpulsar_broker_lookup_failuresrising during the reconnection window.Check for the original trigger. After confirming the cascade, look for what started it. Check ZK JVM GC logs, ZK snapshot timing, recent bulk admin operations, and any ZK member that recently restarted or lost network connectivity.
Metrics and signals to monitor
| Signal | Why it matters | Warning sign |
|---|---|---|
| ZK request latency (avg and max) | Leading indicator for all cascade symptoms; degrades before brokers show distress | Sustained avg > 50ms; max > 100ms means failure is imminent within minutes |
ZK watch count (wchs output) | Watch explosions amplify metadata load and stress the ZK event thread | Rapid growth from baseline; correlates with client reconnection storms |
pulsar_zookeeper_connected per broker | Binary session state; transitions to 0 mean the broker lost its session | Multiple brokers transitioning to 0 simultaneously |
pulsar_lb_unload_bundle_total | Bundle ownership churn generates metadata writes that add ZK load | Sustained unload rate above baseline outside maintenance windows |
pulsar_broker_lookup_failures | New connections cannot resolve topic ownership when ZK is slow or sessions are expired | Failure rate above 1% of total lookups, sustained |
pulsar_active_connections | Connection count reveals the thundering herd: mass disconnect followed by mass reconnect | Sharp drop then spike across multiple brokers simultaneously |
| ZK outstanding requests | Queue depth shows ZK cannot process requests as fast as they arrive | Sustained non-zero; growing queue means ZK is falling further behind |
Fixes
Stabilize ZooKeeper first
The cascade feeds on itself. Your priority is to reduce ZK load or improve ZK capacity until the feedback loop breaks. Do not touch brokers until ZK latency returns to baseline (under 10ms average).
ZK transaction-log disk I/O
If iostat shows the ZK data disk is saturated, ZK cannot commit transactions fast enough. Short-term: identify and stop any non-ZK processes writing to that disk. Long-term: move the ZK transaction log to a dedicated, faster device (NVMe preferred). The ZK transaction log is the write-ahead log for all metadata changes; it is fsync-heavy and latency-sensitive, similar to a BookKeeper journal disk.
Temporarily raise session timeout
Increasing metadataStoreSessionTimeoutMillis gives brokers more time before sessions expire during transient ZK latency spikes. The default is 30000 (30 seconds). Raising it to 60 seconds during an active incident can prevent further session expirations while you address the root cause.
ZooKeeper enforces server-side limits on the negotiated session timeout. The server caps the timeout at maxSessionTimeout, which defaults to 20 * tickTime. With the default tickTime of 2000ms, maxSessionTimeout is 40000ms. If you set metadataStoreSessionTimeoutMillis=60000 on the broker, ZK silently caps the negotiated timeout to 40 seconds. To get a 60-second session, you must also increase tickTime or set maxSessionTimeout explicitly in the ZK server configuration (zoo.cfg) and restart the ZK ensemble.
The lower bound is minSessionTimeout, which defaults to 2 * tickTime (4000ms with default tickTime).
Reduce metadata load during the cascade
If ZK is overloaded by metadata operations from bundle churn and client reconnections, reduce the metadata write rate:
- Pause the load balancer to stop bundle unload and reassignment operations. This is a dynamic config change, no restart needed:
# DISRUPTIVE: disables automatic load balancing cluster-wide # Re-enable after ZK stabilizes pulsar-admin brokers update-dynamic-config \ --config loadManagerClassName \ --config-value org.apache.pulsar.broker.loadbalance.NoopLoadManager - Stop any bulk admin operations (mass topic creation, namespace policy changes, large-scale subscription cleanup).
- If bookie registration is flapping (bookies losing and regaining ZK sessions), consider pausing auto-recovery to reduce ledger metadata churn.
Do not restart brokers
Brokers with the reconnect session expired policy will re-establish their ZK sessions and re-acquire bundles automatically once ZK recovers. Restarting a broker forces it to drop all client connections, close all managed ledgers, and re-acquire all bundles from scratch. During a ZK cascade, this adds a burst of metadata operations that makes the overload worse.
The one exception: if you are running a version before Pulsar 2.10 where the default session expired policy is shutdown, brokers will have halted themselves on session expiry. In that case, you have no choice but to restart them, but do so in a controlled, staggered manner after ZK is stable.
Check for session timeout mismatch with BookKeeper client
The metadataStoreSessionTimeoutMillis to BookKeeper client timeout propagation (issue #24624) landed in Pulsar 4.2.0.
Before Pulsar 4.2.0, the metadataStoreSessionTimeoutMillis value was not properly propagated to the BookKeeper client configuration. The BookKeeper client could use its own default ZK session timeout (typically shorter than the broker’s), causing the BK client to lose its ZK session before the broker’s metadata store session expired. This can trigger ledger fencing failures and additional metadata churn that worsens the cascade. If you are on a version before 4.2.0, verify the BK client timeout is configured to match the broker timeout. Pulsar 4.2.0 fixes this propagation (issue #24624).
Prevention
- Monitor ZK request latency as a first-class signal. Alert on sustained average latency above 10ms and treat anything above 50ms as urgent.
- Monitor ZK watch count. Track the
wchsoutput over time and alert on rapid growth. - Ensure dedicated, fast storage for the ZK transaction log. The transaction log disk is the ZK write bottleneck, analogous to a BookKeeper journal disk.
- Verify the negotiated session timeout. Check that
metadataStoreSessionTimeoutMillisis not silently capped by ZK servermaxSessionTimeout. AdjusttickTimeinzoo.cfgif needed. - Upgrade to Pulsar 4.2.0 or later if the BookKeeper client session timeout propagation fix applies to your deployment.
- Watch for slow ZK degradation over days and weeks. ZK latency can creep from 1ms to 50ms gradually as topic count and metadata volume grow. Alert on the trend, not just the absolute value.
- Confirm the
reconnectsession expired policy is active. On Pulsar 2.10+, this is the default. On older versions, brokers halt on session expiry, which turns a recoverable cascade into a full fleet restart.
How Netdata helps
- Per-second ZK latency metrics expose the initial spike before session expirations cascade. Standard 15-second scrape intervals can miss it.
- ZK latency and
pulsar_zookeeper_connectedacross all brokers in one view makes simultaneous session expiration visible, distinguishing a ZK cascade from individual broker failures. - Bundle unload rate and lookup failure metrics on the same timeline as ZK latency show the relationship between metadata overload and client impact.
- ML anomaly detection on ZK latency catches gradual degradation (1ms to 50ms over hours or days) before it reaches session-timeout territory.
- Active connection count correlation with ZK latency shows the thundering herd pattern: sharp drop then spike, synchronized across brokers.
Related guides
- Apache Pulsar broker GC death spiral: heap pressure, stop-the-world pauses, and lost topic ownership
- Apache Pulsar broker down: telling a dead broker from a fenced one
- Apache Pulsar broker lookup failures: new clients cannot find their topic
- Apache Pulsar active connections climbing: connection leaks and file descriptor exhaustion
- Apache Pulsar bookie failure cascade: recovery I/O that topples surviving bookies