Apache Pulsar ZooKeeper watch explosion: thousands of watches killing metadata latency

Topic lookups slow down. Bundle ownership transfers stall. Broker sessions flicker between connected and disconnected. ZooKeeper reports thousands, sometimes tens of thousands, of registered watches. This is a watch explosion, and the root cause is rarely in ZooKeeper itself. It is in client connection churn.

Watches accumulate when consumers disconnect and reconnect in waves. Each reconnection registers metadata watches for topic ownership, subscription state, and policy changes. During a mass reconnection event (broker restart, network blip, load balancer cycle), thousands of ephemeral znodes are created and deleted in rapid succession. Each creation and deletion triggers watch registration and notification. ZK processes requests sequentially, so as notification load grows, request latency climbs. Once latency exceeds session timeout, brokers lose sessions, triggering bundle unloads and another round of client reconnections. The feedback loop tightens until the cluster thrashes.

What this means

A ZK watch is a one-time trigger registered against a znode. When the znode changes, ZK notifies the watching client, which must re-register for continued notifications. In Pulsar, brokers, bookies, and clients maintain watches on metadata paths: bundle ownership, topic policies, schema registry, cluster configuration.

A watch explosion occurs when the rate of znode creation, deletion, and modification overwhelms ZK’s request processing pipeline. The ensemble spends more time dispatching watch notifications than servicing metadata operations. Request latency rises because of ZK’s sequential processing model: as the outstanding request queue grows, each new request waits longer, causing client timeouts, which trigger reconnections, which generate more watch registrations.

The degradation curve is not linear. ZK latency stays flat until a tipping point, then spikes hard. By the time dashboards show elevated latency, the cascade has often already begun.

flowchart TD
    A[Consumer disconnect storm] --> B[Mass ephemeral znode churn]
    B --> C[Watch registrations spike]
    C --> D[ZK request latency rises]
    D --> E{Exceeds session timeout?}
    E -->|No| F[Metadata ops slow
lookups fail]
    E -->|Yes| G[Broker sessions expire]
    G --> H[Bundles unloaded
to other brokers]
    H --> I[Clients reconnect
to new owners]
    I --> C

Common causes

Quick checks

All read-only and safe to run during an active incident. Note: ZK 4-letter words require 4lw.commands.whitelist to include the relevant commands in zoo.cfg.

# Total watch count on each ZK node
echo wchs | nc <zk-host> 2181

# Watches by session (shows which sessions hold the most watches)
# WARNING: can be expensive on large ensembles; use judiciously during incidents
echo wchc | nc <zk-host> 2181

# Detailed ZK monitoring output: watch count, latency, node count, followers
echo mntr | nc <zk-host> 2181

# ZK server stats: latency, connections, outstanding requests
echo stat | nc <zk-host> 2181

# Broker metadata-store session events (Pulsar exposes no zookeeper_connected Prometheus metric; see the log grep below)

# Bundle unload rate (reconnection source)
curl -s http://<broker-host>:8080/metrics | grep pulsar_lb_unload_bundle_total

# Active connections for reconnection storm signature
curl -s http://<broker-host>:8080/metrics | grep pulsar_active_connections

# Lookup failures (early failure indicator)
curl -s http://<broker-host>:8080/metrics | grep pulsar_broker_lookup

# Broker logs for session expiry events
grep -i "Session expired\|Connection loss" /var/log/pulsar/broker.log | tail -20

How to diagnose it

1. Confirm the watch count is abnormal. Run echo wchs | nc <zk-host> 2181 on each ZK ensemble member. Compare against your established baseline. Thousands of watches on a cluster that should have hundreds is the signature. Sustained ZK request latency above 10ms is an early warning; above 100ms means failure is likely imminent. (These thresholds come from operational incident reports; Pulsar’s own documentation does not define them.)

2. Identify the source of watches. Run echo wchc | nc <zk-host> 2181 to see which sessions hold the most watches. If a single broker session holds a disproportionate share, that broker is the source. If watches are distributed across many sessions, the issue is broad client churn.

3. Correlate with connection churn. Check pulsar_active_connections across brokers. Fluctuating counts (drops followed by spikes) indicate a reconnection storm. If available, check pulsar_connection_created_total_count and pulsar_connection_closed_total_count; the rate of change between them indicates churn velocity.

4. Correlate with bundle unload rate. Check pulsar_lb_unload_bundle_total. Each bundle unload drops client connections for affected topics, causing reconnections that generate new watch registrations. A sustained unload rate above 1 per minute outside maintenance indicates load balancer thrashing.

5. Check ZK transaction log disk. ZK writes every transaction to its log disk synchronously before acknowledging. If that disk is slow or shared with another workload, request latency rises independently of watch count. Run iostat -x 1 on the ZK node and examine %util and await on the transaction log device.

6. Verify ensemble quorum health. Run echo stat | nc <zk-host> 2181 on each member. Confirm all members are present, latency is consistent across nodes, and the outstanding request count is near zero. A degraded member (slow disk, GC pause, network issue) drags the entire ensemble because writes require quorum acknowledgment.

Metrics and signals to monitor

Fixes

Stop the reconnection storm

The immediate goal is to break the feedback loop between watch churn and ZK latency.

If a consumer fleet is in a reconnection loop (client bug, expired credential, broker repeatedly dropping connections), identify the applications with the highest reconnection rate from broker logs. Tight retry loops with no backoff are the most common trigger. Ensure client libraries use exponential backoff with jitter.

If the storm was triggered by a broker restart or bundle unload, wait for the system to settle. Do not restart brokers to “fix” the issue. Restarts generate additional reconnection load and worsen the cascade.

Reduce ZK pressure from topic count

If the watch baseline is high because of topic count rather than churn, the cluster has outgrown its ZK capacity.

• Audit and reduce topic count. Consolidate topics where possible. Partitioned topics with many partitions multiply metadata load because each partition is a separate managed ledger with its own znodes.
• Enable metadata batching (introduced in Pulsar 2.10; the default is true — verify it is not disabled in broker.conf). This batches metadata operations to reduce ZK request count. Relevant parameters: metadataStoreBatchingMaxDelayMillis=5, metadataStoreBatchingMaxOperations=1000, metadataStoreBatchingMaxSizeKb=128. Batching introduces up to 5ms of artificial latency per operation. For clusters with few topics, disabling batching may yield better latency.

Batching applies to metadata read/write operations (get/put/delete/children); watch registrations are not batched (AbstractBatchedMetadataStore).

Tune session timeout

If ZK latency is elevated but not catastrophic, increasing zooKeeperSessionTimeoutMillis buys time by preventing premature session expiry. Pulsar’s default is 30 seconds (zooKeeperSessionTimeoutMillis=-1 falls back to metadataStoreSessionTimeoutMillis=30_000; earlier configs set the literal 30000). Raising it to 60 seconds gives brokers more headroom during latency spikes.

The tradeoff: a higher session timeout means slower detection of genuinely dead brokers. The load balancer will not reassign bundles from a dead broker until its session expires.

Configure session expiry policy (Pulsar 2.10+)

Set zookeeperSessionExpiredPolicy=reconnect to prevent broker shutdown on ZK session expiry. Instead of shutting down, the broker reconnects and re-owns bundles from its in-memory cache. This prevents the simultaneous broker shutdown that turns a ZK latency spike into a cluster-wide outage. The default is reconnect since Pulsar 2.10 (2.8/2.9 shipped shutdown).

Add ZK resources (buys time, does not fix the root cause)

Scaling the ZK ensemble (more members, more CPU, faster transaction log disks) increases throughput and reduces latency. This is a valid stopgap but does not address the underlying watch pressure. If topic count or client churn continues to grow, you will hit the wall again.

The transaction log disk is the most impactful upgrade. ZK writes are synchronous and sequential. A dedicated NVMe device for the transaction log eliminates disk I/O as a bottleneck. Do not share this disk with snapshots, ZK data, or any other workload.

Consider Oxia migration

Current Pulsar docs recommend Oxia for new clusters and document a live migration framework from ZooKeeper (Oxia support landed experimentally in Pulsar 3.3.0, PIP-335). For clusters hitting persistent ZK capacity limits, Oxia eliminates the watch-based coordination model that causes watch explosions. This is a strategic infrastructure decision, not an incident response action.

Prevention

• Monitor ZK watch count proactively. Track wchs or mntr output over time, establish a baseline, and alert on sustained growth.
• Correlate watch growth with bundle unload rate and client reconnection rate. The three signals together tell the story: reconnections drive watch churn, which drives ZK latency, which drives session expiry, which drives more reconnections.
• Treat sustained ZK latency above 10ms as an early warning. Do not dismiss intermittent latency spikes as transient.
• Audit topic count growth. Topics accumulate over time from testing, abandoned features, and dynamic topic creation. High topic count is the multiplier that turns normal reconnection churn into a watch explosion.
• Configure client reconnection with backoff and jitter. Tight retry loops are the most common trigger for reconnection storms. Ensure all client libraries use exponential backoff.
• Set zookeeperSessionExpiredPolicy=reconnect on all brokers. This is the single most effective configuration change to prevent a ZK latency spike from cascading into a full cluster outage.

How Netdata helps

• Per-second metric collection captures ZK latency spikes and watch count changes that 15-second scrape intervals miss. The feedback loop between watch churn and latency can develop in under a minute.
• Correlating ZK watch count with pulsar_lb_unload_bundle_total, pulsar_active_connections, and pulsar_broker_lookup_failures on a single timeline makes it immediately visible whether a latency spike is caused by watch pressure, bundle thrashing, or disk I/O.
• Anomaly detection on ZK request latency and connection count flags early deviation from baseline before latency crosses critical thresholds.
• JVM metrics (heap usage, GC pause times) for ZK servers reveal whether latency spikes are caused by ZK-side GC pauses rather than watch pressure.
• Disk I/O metrics on the ZK transaction log volume distinguish disk-bottlenecked latency from CPU-bottlenecked watch processing.

Related guides

• Apache Pulsar active connections climbing: connection leaks and file descriptor exhaustion
• Apache Pulsar bookie add-entry queue not draining: writes arriving faster than the disk can commit
• Apache Pulsar AutoRecovery stalled: under-replicated ledgers that never heal
• Apache Pulsar backlog age vs size: the latency depth alone cannot show
• Apache Pulsar backlog quota exceeded: producers held or rejected when consumers stall
• Apache Pulsar bookie disk filling: runway to read-only and how to reclaim space
• Apache Pulsar bookie failure cascade: recovery I/O that topples surviving bookies
• Apache Pulsar bookie read latency high: catch-up reads competing with the write path
• Apache Pulsar bookie read-only: disk full and bookie_SERVER_STATUS at zero
• Apache Pulsar broker down: telling a dead broker from a fenced one
• Apache Pulsar broker GC death spiral: heap pressure, stop-the-world pauses, and lost topic ownership
• Apache Pulsar broker lookup failures: new clients cannot find their topic