ZooKeeper watch storm: thousands of notifications when one hot znode changes

A single znode changes. Seconds later, zk_packets_sent spikes to many times its normal rate, CPU on the ZooKeeper process surges, and zk_outstanding_requests begins climbing. Clients report latency spikes, connection timeouts, or session expirations. If many clients watch the same znode, you are looking at a watch storm.

The mechanism: ZooKeeper maintains a watch table mapping znode paths to registered watchers. When a watched znode changes, the server queues a notification for every client that registered a watch on that path. For a path with 10,000 watchers, a single setData call produces 10,000 notification packets. Notification serialization and queuing happen inside the request processing pipeline, competing with all other request handling.

What this means

One mutation triggers N notifications. The blast radius scales with the number of watchers on the changed path. The server does not batch or coalesce notifications: each watcher gets its own packet.

flowchart TD
    A["Hot znode changes"] --> B["ZK iterates watch table for path"]
    B --> C["Queue N notifications, one per watcher"]
    C --> D["zk_packets_sent spikes"]
    C --> E["CPU spikes: serialization + network I/O"]
    D --> F["Request pipeline stalls"]
    E --> F
    F --> G["zk_outstanding_requests climbs"]
    G --> H["Latency spikes for all clients"]
    H --> I["Clients miss heartbeats"]
    I --> J["Sessions expire, ephemeral nodes vanish"]
    J --> K["More watches fire from deletions"]
    K --> C

The cascade:

  1. A heavily-watched znode changes (setData, create, or delete).
  2. ZooKeeper iterates the watch table for that path and queues a notification for every registered watcher.
  3. zk_packets_sent spikes sharply, disproportionate to zk_packets_received because notifications are server-initiated, not responses to concurrent client reads.
  4. CPU rises from notification serialization and network I/O.
  5. The request pipeline stalls. zk_outstanding_requests climbs.
  6. If the stall is severe enough, clients miss heartbeats, sessions expire, ephemeral nodes vanish, and more watch notifications fire. The storm cascades.

Two properties make watch storms dangerous. First, a single client writing to a single znode can trigger them; one misbehaving application can degrade the entire ensemble. Second, the watch table distribution is invisible in normal mntr output. zk_watch_count gives the total, not the distribution. A cluster with 50,000 watches evenly distributed across thousands of paths is healthy. A cluster with 50,000 watches where 40,000 sit on one path is a bomb waiting for the next write.

Common causes

CauseWhat it looks likeFirst thing to check
Service discovery on a single registry znodeAll consumers watch one provider list node; any provider change triggers full fanoutwchs summary showing high watches-per-path concentration
Client watch leakzk_watch_count grows monotonically without corresponding connection growthwatch-per-connection ratio trending upward over days
Persistent/recursive watches (3.6+)Sustained high watch count and notification volume per change; watches do not expire after firingClient library version and whether addWatch is used
Leader election herd effectMass notifications when election parent or leadership znode changesClient election recipe: are all clients watching the same node?

Quick checks

Run these read-only checks to confirm or rule out a watch storm. All are safe for production.

# Total watch count - the headline number
echo mntr | nc localhost 2181 | grep zk_watch_count

# Watch summary: total watches, connections with watches, paths being watched
echo wchs | nc localhost 2181

# Packets sent vs received - sent should track received unless notifications are firing
echo mntr | nc localhost 2181 | grep zk_packets_

# Outstanding requests - should be 0 in steady state
echo mntr | nc localhost 2181 | grep zk_outstanding_requests

# Latency - watch delivery competes with request processing
echo mntr | nc localhost 2181 | grep -E 'zk_(avg|min|max)_latency'

# <!-- TODO: verify per-operation read/write latency availability. Standard mntr only
# exposes aggregate avg/min/max latency. Read-specific latency may be available
# via JMX beans or the Prometheus endpoint in 3.6+, not via mntr. -->

# Connection count - watch storms can cascade into session drops
echo mntr | nc localhost 2181 | grep zk_num_alive_connections

In ZooKeeper 3.5.3+, four-letter commands require whitelisting via 4lw.commands.whitelist in zoo.cfg. If mntr, wchs, or any other command returns empty, check that whitelist. The AdminServer (HTTP interface on port 8080) is an alternative that does not require the four-letter-word whitelist; its endpoints live under /commands/.

How to diagnose it

Step 1: Confirm the fan-out pattern.

Correlate zk_packets_sent with zk_packets_received. During a watch storm, zk_packets_sent spikes far above the rate of zk_packets_received. The server is sending notifications that were not requested by those clients at that moment. If both spike proportionally, you have a request-volume problem, not a watch storm.

Step 2: Check the watch count and distribution.

zk_watch_count gives the total. wchs gives a summary: total watches, number of connections holding watches, and number of paths being watched. The key ratio is watches per watched-path. If 50,000 watches sit on 100 paths, that is roughly 500 per path, likely fine for service discovery. If 50,000 watches sit on 5 paths, one of those paths is a thundering-herd candidate.

Step 3: Identify the hot path (with caution).

wchp lists watches grouped by path, telling you exactly which znode has the most watchers. But wchp is O(n): it iterates the entire watch table. On a server with hundreds of thousands of watches, running wchp can itself cause a latency spike. If you must run it, do so during a low-traffic window or on a follower rather than the leader.

wchc lists watches grouped by session, useful for diagnosing watch leaks by identifying which sessions hold disproportionate numbers. Same caveat: O(n), avoid under load.

Step 4: Determine the watch type.

Standard watches in ZooKeeper 3.x are one-shot. After a watch fires, the client must re-register it on the next read operation (exists, getData, getChildren). A rapidly-changing hot znode causes repeated watch storms at the rate of change, plus a wave of re-registration reads after each fire. Clients using one-shot watches on a fast-changing node also miss intermediate changes between the watch firing and re-registration completing.

In ZooKeeper 3.6+, persistent and recursive watches do not need re-registration. They sustain higher counts and produce notification volume proportional to the change rate of every node in the watched subtree. If clients use persistent recursive watches on a busy subtree, notification volume can be dramatically higher than with one-shot watches on a single node.

Check client library version and whether it uses the persistent watch API.

Step 5: Check for a watch leak.

Compute zk_watch_count / zk_num_alive_connections over time. If this ratio grows without bound, clients are registering watches without removing them. Common causes: client code that registers a new watch object on every read without reusing or removing the previous one, session reconnection logic that creates duplicate registrations, or a framework bug where watches are not cleaned up on session close.

Step 6: Assess downstream impact.

Check zk_outstanding_requests, zk_avg_latency, zk_max_latency, and zk_num_alive_connections. If outstanding requests are climbing and latency is elevated, the storm is affecting all clients, not just the watchers. If connections drop sharply, sessions are expiring, which means the cascade has begun.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
zk_watch_countTotal registered watches on this serverGrowing without bound or sudden jump
zk_packets_sentIncludes watch notifications, not just responsesSpike far exceeding zk_packets_received rate
zk_outstanding_requestsPipeline backlog from notification delivery competing with request processingSustained non-zero during or after a storm
zk_avg_latency / zk_max_latencyWatch delivery competes with request processingElevated max during storm events
zk_num_alive_connectionsWatch storms can cascade to session expiry and mass disconnectSharp drop following the spike
CPU utilizationNotification serialization and network I/O are CPU-intensiveSpike correlated with packets_sent spike

Fixes

Redistribute watches off the hot path

If a single znode has thousands of watchers, shard the fan-out. Instead of all clients watching /services/provider, use /services/provider/instance-1, /services/provider/instance-2, and so on, with each client watching a subset. A change to one shard notifies only that shard’s watchers.

This requires client-side changes and is not always possible if the watched data is a single coherent unit like a configuration document. In that case, evaluate whether the data truly needs to live in a single znode or whether it can be split into smaller pieces with independent watch registrations.

Fix client watch leaks

If the watch-per-connection ratio is climbing, the client is leaking watches. Ensure watch objects are reused rather than re-created on every read. Use removeWatches (available since 3.5.0) to explicitly remove watches when no longer needed. Audit reconnection logic to prevent duplicate watch registration on session reconnect.

Evaluate persistent watch usage

Persistent and recursive watches (3.6+) sustain higher notification volumes. If clients use addWatch with recursive mode on a busy subtree, every descendant change fires a notification. Evaluate whether recursive scope is necessary, or whether targeted one-shot watches on specific children would produce less fan-out.

Fix election herd effect

If the storm comes from a leader election recipe where all clients watch the same election znode, switch to the standard sequential-node recipe. Each participant creates a sequential ephemeral node and watches only the immediately preceding node. When the leader changes, only one client is notified, not the entire fleet.

Prevention

Monitor the watch-per-connection ratio. Track zk_watch_count / zk_num_alive_connections over time. A stable ratio means clients are managing watches correctly. A growing ratio points to a leak before it becomes a storm.

Alert on watch count growth. zk_watch_count growing monotonically without a corresponding increase in connections is a leading indicator. Set a threshold based on your baseline, not an absolute number. Service discovery systems naturally have high watch counts.

Audit hot paths before they become hot. Any single path with more than 10,000 registered watchers is a thundering-herd risk. Use wchs periodically during low-traffic windows to track path concentration. The summary is cheaper than wchp and sufficient for trend monitoring.

Never run wchc or wchp under load. These commands are O(n) and can trigger the latency spike you are trying to diagnose. Schedule them for maintenance windows or run them on followers.

Review persistent watch adoption during upgrades. When upgrading clients to 3.6+ libraries, audit whether persistent or recursive watches are being used and on which subtrees. Notification volume from persistent recursive watches on a busy subtree can be dramatically higher than one-shot watches on the same data.

How Netdata helps

The signature of a watch storm is zk_packets_sent spiking far above zk_packets_received. Per-second resolution on both metrics catches short storms that minute-level scraping misses. Correlating packets_sent against CPU, zk_outstanding_requests, and zk_avg_latency on the same timeline confirms the diagnosis without manual cross-referencing.

A slow watch leak grows over days before triggering a storm. Anomaly detection on the zk_watch_count / zk_num_alive_connections ratio flags the upward trend before it becomes operational impact. When a storm cascades to session expiry, zk_num_alive_connections drops in lockstep with the packets_sent spike, distinguishing a watch-driven cascade from a GC-pause or network-driven one.