$ guides / apache-pulsar / apache-pulsar-zookeeper-watch-explosion ▌

Operations Guides

Apache Pulsar ZooKeeper watch explosion: thousands of watches killing metadata latency

Topic lookups slow down. Bundle ownership transfers stall. Broker sessions flicker between connected and disconnected. ZooKeeper reports thousands, sometimes tens of thousands, of registered watches. This is a watch explosion, and the root cause is rarely in ZooKeeper itself. It is in client connection churn.

Watches accumulate when consumers disconnect and reconnect in waves. Each reconnection registers metadata watches for topic ownership, subscription state, and policy changes. During a mass reconnection event (broker restart, network blip, load balancer cycle), thousands of ephemeral znodes are created and deleted in rapid succession. Each creation and deletion triggers watch registration and notification. ZK processes requests sequentially, so as notification load grows, request latency climbs. Once latency exceeds session timeout, brokers lose sessions, triggering bundle unloads and another round of client reconnections. The feedback loop tightens until the cluster thrashes.

What this means

A ZK watch is a one-time trigger registered against a znode. When the znode changes, ZK notifies the watching client, which must re-register for continued notifications. In Pulsar, brokers, bookies, and clients maintain watches on metadata paths: bundle ownership, topic policies, schema registry, cluster configuration.

A watch explosion occurs when the rate of znode creation, deletion, and modification overwhelms ZK’s request processing pipeline. The ensemble spends more time dispatching watch notifications than servicing metadata operations. Request latency rises because of ZK’s sequential processing model: as the outstanding request queue grows, each new request waits longer, causing client timeouts, which trigger reconnections, which generate more watch registrations.

The degradation curve is not linear. ZK latency stays flat until a tipping point, then spikes hard. By the time dashboards show elevated latency, the cascade has often already begun.

flowchart TD
    A[Consumer disconnect storm] --> B[Mass ephemeral znode churn]
    B --> C[Watch registrations spike]
    C --> D[ZK request latency rises]
    D --> E{Exceeds session timeout?}
    E -->|No| F[Metadata ops slow
lookups fail]
    E -->|Yes| G[Broker sessions expire]
    G --> H[Bundles unloaded
to other brokers]
    H --> I[Clients reconnect
to new owners]
    I --> C

Common causes

Cause	What it looks like	First thing to check
Consumer disconnect/reconnect storm	Watch count spikes alongside `pulsar_active_connections` fluctuations and bundle unload events	`echo wchs \| nc <zk-host> 2181` for current count
Excessive topic count	High baseline watch count proportional to topic count, slow degradation over time as topics accumulate	`pulsar_topics_count` per broker vs cluster average
Load balancer thrashing	Bundle unload rate (`pulsar_lb_unload_bundle_total`) sustained above baseline; each unload triggers client reconnections	Broker logs for rapid ownership cycling
ZK transaction log disk I/O bottleneck	ZK latency spikes correlate with disk I/O on the ZK transaction log volume	`iostat -x 1` on the ZK transaction log disk
ZK ensemble member failure	Quorum degraded, latency rises as remaining members absorb load	`echo stat \| nc <zk-host> 2181` on each node

Quick checks

All read-only and safe to run during an active incident. Note: ZK 4-letter words require 4lw.commands.whitelist to include the relevant commands in zoo.cfg.

# Total watch count on each ZK node
echo wchs | nc <zk-host> 2181

# Watches by session (shows which sessions hold the most watches)
# WARNING: can be expensive on large ensembles; use judiciously during incidents
echo wchc | nc <zk-host> 2181

# Detailed ZK monitoring output: watch count, latency, node count, followers
echo mntr | nc <zk-host> 2181

# ZK server stats: latency, connections, outstanding requests
echo stat | nc <zk-host> 2181

# Broker ZK session connectivity
curl -s http://<broker-host>:8080/metrics | grep pulsar_zookeeper_connected

# Bundle unload rate (reconnection source)
curl -s http://<broker-host>:8080/metrics | grep pulsar_lb_unload_bundle_total

# Active connections for reconnection storm signature
curl -s http://<broker-host>:8080/metrics | grep pulsar_active_connections

# Lookup failures (early failure indicator)
curl -s http://<broker-host>:8080/metrics | grep pulsar_broker_lookup

# Broker logs for session expiry events
grep -i "Session expired\|Connection loss" /var/log/pulsar/broker.log | tail -20

How to diagnose it

Confirm the watch count is abnormal. Run echo wchs | nc <zk-host> 2181 on each ZK ensemble member. Compare against your established baseline. Thousands of watches on a cluster that should have hundreds is the signature. Sustained ZK request latency above 10ms is an early warning; above 100ms means failure is likely imminent.

Identify the source of watches. Run echo wchc | nc <zk-host> 2181 to see which sessions hold the most watches. If a single broker session holds a disproportionate share, that broker is the source. If watches are distributed across many sessions, the issue is broad client churn.
Correlate with connection churn. Check pulsar_active_connections across brokers. Fluctuating counts (drops followed by spikes) indicate a reconnection storm. If available, check pulsar_connection_created_total_count and pulsar_connection_closed_total_count; the rate of change between them indicates churn velocity.
Correlate with bundle unload rate. Check pulsar_lb_unload_bundle_total. Each bundle unload drops client connections for affected topics, causing reconnections that generate new watch registrations. A sustained unload rate above 1 per minute outside maintenance indicates load balancer thrashing.
Check ZK transaction log disk. ZK writes every transaction to its log disk synchronously before acknowledging. If that disk is slow or shared with another workload, request latency rises independently of watch count. Run iostat -x 1 on the ZK node and examine %util and await on the transaction log device.
Verify ensemble quorum health. Run echo stat | nc <zk-host> 2181 on each member. Confirm all members are present, latency is consistent across nodes, and the outstanding request count is near zero. A degraded member (slow disk, GC pause, network issue) drags the entire ensemble because writes require quorum acknowledgment.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
ZK watch count (`wchs` or `mntr`)	Direct measure of watch pressure on the ensemble	Count growing without corresponding traffic growth
ZK request latency	Leading indicator for session timeout cascade	Sustained average > 50ms; > 100ms is critical
`pulsar_lb_unload_bundle_total`	Each unload triggers client reconnections and new watches	Sustained > 1/min outside maintenance
`pulsar_active_connections`	Reconnection storms generate watch churn	Fluctuating counts or unexplained growth
`pulsar_broker_lookup_failures`	Lookup failures indicate ZK cannot serve metadata	Failure rate > 1% of total lookups sustained
`pulsar_zookeeper_connected`	Session loss is the cascade trigger	Transition from 1 to 0 on any broker
ZK transaction log disk I/O	ZK writes are synchronous to this disk	`await` elevated or `%util` consistently high
ZK client connection count	More sessions mean more watches	Growing trend correlating with latency

Fixes

Stop the reconnection storm

The immediate goal is to break the feedback loop between watch churn and ZK latency.

If a consumer fleet is in a reconnection loop (client bug, expired credential, broker repeatedly dropping connections), identify the applications with the highest reconnection rate from broker logs. Tight retry loops with no backoff are the most common trigger. Ensure client libraries use exponential backoff with jitter.

If the storm was triggered by a broker restart or bundle unload, wait for the system to settle. Do not restart brokers to “fix” the issue. Restarts generate additional reconnection load and worsen the cascade.

Reduce ZK pressure from topic count

If the watch baseline is high because of topic count rather than churn, the cluster has outgrown its ZK capacity.

Audit and reduce topic count. Consolidate topics where possible. Partitioned topics with many partitions multiply metadata load because each partition is a separate managed ledger with its own znodes.
Enable metadata batching (Pulsar 2.10+). Set metadataStoreBatchingEnabled=true in broker.conf. This batches metadata operations to reduce ZK request count. Relevant parameters: metadataStoreBatchingMaxDelayMillis=5, metadataStoreBatchingMaxOperations=1000, metadataStoreBatchingMaxSizeKb=128. Batching introduces up to 5ms of artificial latency per operation. For clusters with few topics, disabling batching may yield better latency.

Tune session timeout

If ZK latency is elevated but not catastrophic, increasing zooKeeperSessionTimeoutMillis buys time by preventing premature session expiry. Pulsar’s default is 30 seconds. Raising it to 60 seconds gives brokers more headroom during latency spikes.

The tradeoff: a higher session timeout means slower detection of genuinely dead brokers. The load balancer will not reassign bundles from a dead broker until its session expires.

Configure session expiry policy (Pulsar 2.10+)

Set zookeeperSessionExpiredPolicy=reconnect to prevent broker shutdown on ZK session expiry. Instead of shutting down, the broker reconnects and re-owns bundles from its in-memory cache. This prevents the simultaneous broker shutdown that turns a ZK latency spike into a cluster-wide outage.

Add ZK resources (buys time, does not fix the root cause)

Scaling the ZK ensemble (more members, more CPU, faster transaction log disks) increases throughput and reduces latency. This is a valid stopgap but does not address the underlying watch pressure. If topic count or client churn continues to grow, you will hit the wall again.

The transaction log disk is the most impactful upgrade. ZK writes are synchronous and sequential. A dedicated NVMe device for the transaction log eliminates disk I/O as a bottleneck. Do not share this disk with snapshots, ZK data, or any other workload.

Consider Oxia migration

Pulsar 5.0 introduces Oxia as the recommended metadata store, with a migration framework for transitioning from ZooKeeper. For clusters hitting persistent ZK capacity limits, Oxia eliminates the watch-based coordination model that causes watch explosions. This is a strategic infrastructure decision, not an incident response action.

Prevention

Monitor ZK watch count proactively. Track wchs or mntr output over time, establish a baseline, and alert on sustained growth.
Correlate watch growth with bundle unload rate and client reconnection rate. The three signals together tell the story: reconnections drive watch churn, which drives ZK latency, which drives session expiry, which drives more reconnections.
Treat sustained ZK latency above 10ms as an early warning. Do not dismiss intermittent latency spikes as transient.
Audit topic count growth. Topics accumulate over time from testing, abandoned features, and dynamic topic creation. High topic count is the multiplier that turns normal reconnection churn into a watch explosion.
Configure client reconnection with backoff and jitter. Tight retry loops are the most common trigger for reconnection storms. Ensure all client libraries use exponential backoff.
Set zookeeperSessionExpiredPolicy=reconnect on all brokers. This is the single most effective configuration change to prevent a ZK latency spike from cascading into a full cluster outage.

How Netdata helps

Per-second metric collection captures ZK latency spikes and watch count changes that 15-second scrape intervals miss. The feedback loop between watch churn and latency can develop in under a minute.
Correlating ZK watch count with pulsar_lb_unload_bundle_total, pulsar_active_connections, and pulsar_broker_lookup_failures on a single timeline makes it immediately visible whether a latency spike is caused by watch pressure, bundle thrashing, or disk I/O.
Anomaly detection on ZK request latency and connection count flags early deviation from baseline before latency crosses critical thresholds.
JVM metrics (heap usage, GC pause times) for ZK servers reveal whether latency spikes are caused by ZK-side GC pauses rather than watch pressure.
Disk I/O metrics on the ZK transaction log volume distinguish disk-bottlenecked latency from CPU-bottlenecked watch processing.