$ guides / apache-pulsar / apache-pulsar-metadata-store-latency-high ▌

Operations Guides

Apache Pulsar metadata store latency: the leading indicator before every cluster-wide failure

Metadata store latency is the earliest signal that a Pulsar cluster is heading toward a widespread outage. In most 3.x deployments the metadata store is ZooKeeper. Pulsar 3.3.0 introduced experimental Oxia support as an eventual replacement. Every broker, bookie, and load balancer operation that touches cluster topology, bundle ownership, schema lookups, ledger metadata, or cursor persistence routes through this store. When round-trip latency for those operations rises, the symptoms appear downstream in brokers and bookies, but the root cause is upstream.

The failure pattern is predictable. Metadata store latency degrades over days or weeks. A broker occasionally loses bundle ownership, then recovers. Operators dismiss it as transient. Then latency crosses the session timeout threshold, multiple brokers lose their sessions simultaneously, bundle ownership thrashes across the fleet, clients reconnect en masse, and the resulting metadata operation storm pushes the store further into saturation. The curve is exponential once timeouts start causing reconnections, because reconnections generate more load on the already-saturated store.

If you are not monitoring metadata store latency as a first-class signal, you are reacting to second-order effects while the root cause compounds.

How metadata store latency degrades

Metadata store latency is the round-trip time for metadata operations from a broker to the store. It covers reads, writes, and watch notifications. In a healthy cluster this is under 10ms. When it exceeds 50ms sustained, broker operations begin to stall: namespace bundle ownership changes slow down, schema lookups time out, consumer connection registration backs up, and ledger metadata operations queue. When it exceeds 100ms, failure is imminent within minutes because the store can no longer service heartbeats and ephemeral node updates within session timeout windows.

The degradation is a positive feedback loop. Rising latency causes broker operations to take longer, which keeps connections open longer, which holds more resources, which generates retry traffic when those operations eventually time out. Retries add load to an already-saturated store. If the loop crosses the session timeout boundary, the cluster enters a cascade that requires deliberate intervention to break.

flowchart TD
    A[Metadata store latency rising] --> B[Broker metadata ops slow]
    B --> C[Bundle ownership updates delayed]
    C --> D[Lookup failures increase]
    B --> E[Session heartbeat delays]
    E --> F{Latency exceeds session timeout?}
    F -->|No| G[Transient ownership loss
Brokers recover, dismissed as flake]
    F -->|Yes| H[Session expiration on multiple brokers]
    H --> I[Bundle unloads en masse]
    I --> J[Client reconnection storm]
    J --> K[More metadata ops on store]
    K --> A
    G --> A

Common causes

Cause	What it looks like	First thing to check
ZK transaction log disk I/O saturation	Latency spikes correlate with disk write latency on the ZK host. Transaction log disk is shared or undersized.	`iostat -x 1` on the ZK transaction log disk
Watch explosion	ZK watch count in the tens of thousands. Latency rises after a consumer disconnect or reconnect storm.	`echo wchs \| nc <zk-host> 2181`
ZK ensemble member failure	One ZK node unreachable or far behind. Quorum still intact but every write waits for the slowest follower.	`echo stat \| nc <zk-host> 2181` on each ensemble member
Broker GC death spiral	ZK session expirations on one or more brokers correlate with JVM GC pause times exceeding 1 second.	JVM GC logs or `jstat -gc <pid> 1000`
Network congestion or partition	Latency between broker and ZK hosts is elevated. No disk or JVM issue present. May affect only cross-rack or cross-AZ traffic.	TCP retransmit rate, inter-host ping latency
Metadata bloat	ZK snapshot and transaction log sizes growing steadily. Operations increasingly slow even at low load. Topic deletion or bundle unload takes minutes.	ZK data directory size, znode count

Quick checks

Run these in order. They are all read-only and safe for production.

ZooKeeper 4lw commands: The stat, wchs, and mntr commands used below are disabled by default in ZooKeeper 3.5+. You must whitelist them via 4lw.commands.whitelist in zoo.cfg. If nc returns “Command is not executed because it is not in the whitelist,” that is why.
Warning on wchs under load: wchs enumerates all watches on the server. On an ensemble with hundreds of thousands of watches, this can cause a multi-second pause. Avoid running it during an active incident unless necessary.

# Check ZK server responsiveness and latency stats
echo stat | nc <zk-host> 2181

# Check ZK watch count (watch explosion detection)
# Note: expensive on large ensembles, see warning above
echo wchs | nc <zk-host> 2181

<!-- TODO: verify exact admin server endpoint path (stat vs stats) -->
# Check ZK outstanding requests and latency from admin server
curl -s http://<zk-host>:8080/commands/stat

# Check broker-side ZK connectivity state
curl -s http://<broker-host>:8080/metrics | grep -i "zookeeper"

# Check broker lookup failure rate (correlates with metadata store issues)
curl -s http://<broker-host>:8080/metrics | grep pulsar_broker_lookup

# Check bundle unload rate (ownership thrashing indicator)
<!-- TODO: verify exact metric name for bundle unload counter -->
curl -s http://<broker-host>:8080/metrics | grep pulsar_lb_unload_bundle

# Check ZK transaction log disk I/O on the ZK host
iostat -x 1 5

# Check ZK JVM process health
jstat -gc $(pgrep -f QuorumPeerMain) 1000

If your Pulsar version exposes the metadata store abstraction metrics directly, look for pulsar_metadata_store_ops_latency in the broker’s Prometheus output. This histogram, labeled by cluster and operation name (get, put, delete), provides per-operation latency directly from the broker’s perspective. In Pulsar 3.3.0+, if you are running Oxia, the broker-side metric names will differ. In either case, the ZK-direct checks above remain applicable for ZooKeeper-backed deployments.

How to diagnose it

Confirm the latency elevation is metadata-store-specific. Check whether broker publish latency and bookie journal sync latency are also elevated. If they are elevated but metadata store latency rose first, the store is the root cause. If metadata store latency is a symptom of a broker GC death spiral, the GC pauses will be visible in JVM metrics before the store shows problems.
Determine scope: single ZK node or ensemble-wide. Run echo stat | nc <zk-host> 2181 against each ensemble member. Compare the latency and outstanding request counts. If one member is an outlier, it is pulling down the ensemble because ZooKeeper writes require quorum acknowledgment from a majority of followers.
Check for watch explosion. Run echo wchs | nc <zk-host> 2181. Watch counts in the tens of thousands indicate that client reconnect storms or excessive topic-level watches are generating sustained read load. This commonly happens when many consumers disconnect and reconnect simultaneously, each registering watches on namespace and topic metadata.
Inspect ZK transaction log disk I/O. The ZK transaction log is a write-ahead log with synchronous fsync per transaction. If the disk serving this log is shared, saturated, or cloud storage with variable latency (such as EBS gp3), every ZK write stalls. Use iostat -x 1 and look at w_await and %util on the transaction log disk. This disk must be dedicated to ZK transactions.
Correlate with broker-side symptoms. Check pulsar_broker_lookup_failures and pulsar_lb_unload_bundle across the broker fleet. Rising lookup failures combined with increasing bundle unload rates confirm that the metadata store degradation is already affecting cluster operations. The broker logs will show session expiration events (Session expired or Connection loss to ZooKeeper).
Assess proximity to session timeout. The default ZK session timeout is typically 30 seconds. If metadata store latency is approaching the point where heartbeats cannot be serviced within this window, the cluster is minutes away from cascade. At sustained latency above 100ms with high operation volume, the effective heartbeat round-trip may exceed the timeout under load.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Metadata store ops latency (`pulsar_metadata_store_ops_latency`)	Direct measurement of broker-to-store round-trip time. The primary leading indicator.	Sustained average > 50ms. Critical > 100ms.
Batch metadata store executor queue size	Queue depth rises before latency spikes when the batching layer cannot drain fast enough.	Sustained non-zero growth after traffic bursts.
Batch metadata store queue wait time	Time operations spend waiting in the batch queue before execution. Rising wait time predicts rising latency.	Upward trend over minutes.
ZK watch count (`wchs`)	Excessive watches create sustained read load. Watch explosions happen during consumer reconnect storms.	Tens of thousands of watches, or rapid growth.
ZK outstanding requests	Requests queued but not yet processed. Near-zero in healthy operation.	Sustained non-zero values indicate CPU or I/O bottleneck on ZK.
Broker lookup failures	Lookup failures mean new clients cannot find topic owners. First user-visible symptom of metadata store problems.	Failure rate > 1% of total lookups sustained.
Bundle unload rate	High unload rates outside maintenance windows indicate ownership thrashing caused by metadata store instability.	> 1 unload per minute sustained outside maintenance.
Broker ZK session state	Session loss triggers mass bundle reassignment and client reconnection. Binary signal: connected or not.	Any transition to disconnected on multiple brokers.

Fixes

ZK transaction log disk saturation

The ZK transaction log disk is the most common bottleneck. Every metadata write requires a synchronous fsync. If this disk is shared with other workloads, undersized, or cloud storage with variable latency, the entire cluster’s coordination layer stalls.

Immediate: Identify and remove any non-ZK processes writing to the transaction log disk. If using cloud storage, check whether the volume is being throttled (IOPS or throughput limits).

Short-term: Move the ZK transaction log to a dedicated, higher-performance volume. For ZooKeeper, the dataLogDir must point to a device separate from the dataDir (snapshot directory).

Tradeoff: Faster storage costs more. But ZK transaction log disk is not the place to save money. A single fsync stall propagates to every broker in the cluster.

Watch explosion

When consumer reconnect storms register thousands of watches simultaneously, ZK cannot service them without latency spikes. This is common in deployments with many partitioned topics where each partition registers watches.

Immediate: Identify the reconnect source. Check for client retry loops, mass consumer redeployments, or broker-side bundle unloads triggering cascading reconnections.

Short-term: Reduce the number of topic-level watches by consolidating subscriptions or reducing partition counts where possible.

Tradeoff: Reducing partition count reduces parallelism. Balance watch load against consumer throughput requirements.

Broker GC death spiral causing session expiration

If broker JVM GC pauses exceed the ZK session timeout, the broker cannot send heartbeats, and ZK expires its session. The broker loses all bundle ownership. Other brokers pick up the bundles, triggering client reconnections, which create more metadata operations and memory allocation, causing more GC.

Immediate: Check broker GC logs for full GC pause durations. If pauses exceed 1 second, the JVM is in trouble.

Short-term: Reduce managedLedgerCacheSizeMB to free heap, or increase JVM heap. If using G1GC, evaluate ZGC for shorter pause times. ZGC requires JDK 15 or later for production use.

Tradeoff: Smaller managed ledger cache means more cache misses, which sends more reads to bookies. But a broker that stays connected to ZK is strictly better than one that thrashes ownership.

Temporary session timeout increase

If metadata store latency is elevated but the root cause is being addressed, temporarily increasing the ZK session timeout can buy time before sessions start expiring. This is a stopgap, not a fix.

Caveat: A longer session timeout means slower failure detection. A genuinely dead broker takes longer to be recognized. Use this only as a bridge while fixing the underlying latency issue.

Do not restart brokers during a ZK latency storm. Brokers will reconnect on their own once ZK recovers. Restarting creates additional reconnection load and metadata operations, worsening the storm.

Prevention

Treat metadata store latency as a first-class alerting target. Alert on sustained average latency above 50ms. Page on sustained latency above 100ms. These thresholds are not workload-dependent. The metadata store either responds quickly or the cluster is at risk.
Dedicate the ZK transaction log disk. No other process should write to it. This is the equivalent of the bookie journal disk rule. Shared transaction log disk is the most common preventable cause of ZK latency storms.
Monitor ZK watch count as a trend. Watch count growing over weeks indicates metadata accumulation or subscription sprawl. Catch watch explosions before they happen by tracking the trend, not just the current value.
Track ZK snapshot and transaction log file sizes. Growing snapshot sizes mean metadata bloat. Topics with long retention accumulate thousands of ledger segments, each represented as a znode. Large znode counts slow every ZK operation.
Correlate metadata store latency with broker GC pause times. If session expirations correlate with GC pauses, the root cause is broker memory, not the store. Fix the GC issue before it manifests as a metadata store cascade.
Monitor metadata store batching metrics. The batch metadata store executor queue size and queue wait time are leading indicators. When the queue backs up, latency will follow. The default batching configuration is metadataStoreBatchingEnabled=true, metadataStoreBatchingMaxDelayMillis=5, metadataStoreBatchingMaxOperations=1000, and metadataStoreBatchingMaxSizeKb=128.

How Netdata helps

Per-second metric collection captures metadata store latency spikes that 15-second scrape intervals miss. Sub-second latency bursts are the earliest sign of store degradation.
Correlation across layers lets you overlay metadata store latency with broker GC pause times, bundle unload rates, lookup failure rates, and active connection counts in a single view. The feedback loop between store latency and broker behavior is visible immediately rather than reconstructed after the fact.
ML-based anomaly detection on metadata store latency identifies the slow drift from 5ms to 15ms to 30ms over days or weeks. This is the exact pattern that gets dismissed as transient until it crosses the session timeout boundary.

ZK-side metrics provide the server-side view: outstanding requests, watch count, and filesystem latency. Combining broker-side metadata store latency with ZK-side metrics pinpoints whether the bottleneck is in the store or in the network path.
Pre-built alert thresholds for the 50ms warning and 100ms critical levels are available out of the box, so you are not assembling alerting rules during an incident.