ZooKeeper pending syncs growing: followers can’t keep up with the write rate

zk_pending_syncs is the leader’s count of in-flight sync operations to followers. In steady state it sits at zero. Sustained non-zero values mean at least one follower cannot absorb the proposal stream as fast as the leader generates it. This is ZooKeeper’s replication-lag signal, distinct from generic latency metrics because it points directly at the write pipeline’s fan-out side.

Two traps to know up front. First, this metric is leader-only. A monitoring setup that scrapes followers or picks a random ensemble member never sees it; if dashboards are blank for zk_pending_syncs, you are looking at the wrong node. Second, a slow follower does not immediately break quorum, so this signal often fires before anything the business notices. A follower stuck behind long enough eventually crosses syncLimit x tickTime, gets ejected, and triggers re-election.

This article covers how to read the metric, find the lagging member, and isolate follower disk, network, GC, or an in-progress SNAP sync.

What this means

For every write, ZAB does the following: the leader writes the proposal to its own transaction log, broadcasts PROPOSE to followers, each follower writes the proposal to its own transaction log and ACKs, and once quorum ACKs arrive the leader commits. A “pending sync” is the leader’s bookkeeping for an outstanding follower operation that has not completed. When the count is non-zero, a follower has work queued that it has not acknowledged.

Pending syncs growing is not the same as zk_outstanding_requests growing on a single node. Outstanding requests is the local request-processing backlog. Pending syncs is specifically replication fan-out state. Both can be elevated at once, but pending syncs isolates the “leader is waiting on followers” axis.

The threshold from the playbook is sustained non-zero for more than one minute. Brief spikes during leader election recovery or after a rolling restart are expected. What you are hunting for is a follower structurally slower than the write rate, which manifests as zk_pending_syncs refusing to return to zero between bursts.

flowchart TD
  A[Leader writes proposal to txn log] --> B[Leader broadcasts PROPOSE to followers]
  B --> C{Follower ACKs in time?}
  C -- yes --> D[Commit applied, pending_syncs drain]
  C -- no: disk or GC or network slow --> E[zk_pending_syncs grows on leader]
  E --> F{Follower catches up before syncLimit x tickTime?}
  F -- yes --> D
  F -- no --> G[Follower ejected from ensemble]
  G --> H[Quorum shrinks, possible re-election]

Common causes

Quick checks

All read-only.

# Identify the current leader (pending_syncs is leader-only)
for host in zk1 zk2 zk3; do
  echo "$host: $(echo srvr | nc -w 2 $host 2181 | grep '^Mode')"
done

# Confirm pending_syncs is sustained above 0 on the leader.
# Substitute the leader hostname you identified above.
echo mntr | nc leader 2181 | grep -E 'zk_(pending_syncs|synced_followers|followers)'

# Compare zxids across all nodes; the laggard is the one with a smaller value
for host in zk1 zk2 zk3; do
  echo "$host: $(echo mntr | nc -w 2 $host 2181 | grep zk_zxid)"
done

# Fsync latency on each follower (root cause of "slow follower")
for host in zk1 zk2 zk3; do
  echo "== $host =="
  echo mntr | nc -w 2 $host 2181 | grep -E 'zk_.*fsynctime'
done

# JVM pause time on each follower (GC as root cause)
for host in zk1 zk2 zk3; do
  echo "== $host =="
  echo mntr | nc -w 2 $host 2181 | grep -E 'zk_.*jvm_pause'
done

# Outstanding requests on the leader (writes backing up while waiting for ACKs)
echo mntr | nc leader 2181 | grep -E 'zk_(outstanding_requests|throttled_ops)'

<!-- TODO: verify the exact mntr keys for fsynctime, jvm_pause, follower_sync_time, and quorum_ack_latency across ZK 3.6/3.7/3.8 - some are only emitted when the MetricsProvider is enabled -->

If 4lw.commands.whitelist is not configured to include mntr and srvr (required since ZooKeeper 3.5.3), these commands return empty. Use the AdminServer HTTP endpoint at http://<host>:8080/commands/monitor instead.

How to diagnose it

1. Confirm you are looking at the leader. zk_pending_syncs is only emitted on the node reporting Mode: leader. If your dashboard is blank, you are scraping a follower. Either target the leader explicitly or scrape all nodes and use only the leader’s values.

2. Identify the lagging follower by zxid. Walk the ensemble and compare zk_zxid values. In steady state they should be identical. The follower with the smaller zxid is the one holding up the pipeline. Decode the zxid carefully: the upper 32 bits are the epoch (incremented per election) and the lower 32 bits are the counter. A lower counter under the same epoch is real lag.

3. Estimate the size of the gap. A few hundred to a few thousand transactions behind is normal catch-up territory (DIFF or TRUNC sync). A follower so far behind that the leader no longer holds the relevant transaction log entries requires a full snapshot transfer (SNAP sync), which is expensive for the leader and takes the follower out of read service while it runs. Check the leader log for “Sending snapshot” and the follower log for snapshot loading messages.

4. Once you have a suspect, check the three root causes in order:

   • Follower disk: zk_fsynctime p99 on the lagging follower is the smoking gun. Cross-check with iostat -x 1 on its txnlog device. Sustained %util near 100 or await in tens of milliseconds means the disk cannot keep up with the proposal stream.
   • Network: zk_quorum_ack_latency p99 captures PROPOSE-to-quorum-ACK time. If it is elevated but per-follower fsync looks fine, the inter-node path is the bottleneck. Check retransmits, NIC error counters, and whether the ensemble spans datacenters.
   • Follower GC: zk_jvm_pause_time_ms p99 on the lagging follower. A Stop-the-World pause freezes the follower’s request processing including ACK generation, which surfaces to the leader as a sync stall.

5. Check whether syncLimit x tickTime is at risk. The default is 5 x 2000ms = 10 seconds. If follower sync time is climbing toward that boundary, the leader is about to eject the follower. That causes a transient quorum reduction and possibly a re-election. This is the escalation path from “noisy metric” to “incident”.

Metrics and signals to monitor

Fixes

Follower disk too slow

This is the most common cause. The follower accepts a proposal, then blocks on fsync before it can ACK. The leader’s pending sync count grows while it waits.

• Verify dataLogDir is on a dedicated device. If dataLogDir is unset, the transaction log shares a disk with snapshots and contention produces fsync stalls. Pointing dataLogDir at a dedicated low-latency device is the single most impactful configuration change. It requires a rolling restart to take effect.
• Check cloud storage throttling. On EBS gp2/gp3 or equivalent, burst credit exhaustion produces a sudden cliff in fsync latency. Move to provisioned IOPS or a higher tier.
• Check for co-located I/O. Backup agents, logging sidecars, or monitoring scrapers writing to the same device will starve fsync.
• Watch for fsync-ing the write ahead log took warnings in the ZooKeeper log. These are emitted when fsync exceeds fsync.warningthresholdms (default 1000ms). Any appearance of this line on a follower is significant.

Saturated inter-node network

Quorum ACK is bounded by the slowest path between leader and any quorum member. Inter-ensemble network saturating under replication load produces elevated zk_quorum_ack_latency with normal per-follower fsync.

• Measure inter-node bandwidth. Replication traffic and client traffic share the same NIC. If the ensemble spans datacenters, expect a higher baseline and a tighter ceiling.
• Look for asymmetric connectivity. A network policy or security group that degrades one leader-follower pair shows up as one lagging follower regardless of which node holds the role.
• SNAP sync makes it worse. When a follower falls behind enough to require a snapshot transfer, the leader serializes and ships the entire data tree. That compounds the bandwidth problem and can extend the outage.

Follower GC pauses

A long GC pause on a follower freezes ACK generation. The leader sees the follower as stalled, pending syncs grow, and if the pause exceeds syncLimit x tickTime the follower is ejected.

• Read the GC log on the lagging follower. If GC logging is not enabled, enable it: -Xlog:gc*:file=/var/log/zookeeper/gc.log:time,uptime,level,tags:filecount=5,filesize=100m for JDK 9+. Tune the path to match your install.
• Check heap sizing. A data tree that has grown into the heap produces longer and more frequent full GCs. Cross-reference zk_znode_count and zk_approximate_data_size.
• Switch GC algorithm if running CMS or Parallel. G1GC is the default on JDK 11+. ZGC (production in JDK 15) dramatically reduces pause times.

SNAP sync in progress

If a follower recently rejoined after maintenance or a crash, it may be receiving a full snapshot from the leader. zk_follower_sync_time will be high and zk_pending_syncs may sit above zero until the transfer completes.

• Confirm it is progressing. The lagging follower’s zxid should be advancing toward the leader’s.
• Watch leader load. Snapshot serialization is CPU and network intensive on the leader. If write latency to clients degrades during the sync, consider shifting read traffic away from the leader.
• Prevent recurrence. SNAP sync is triggered when the leader no longer holds the transaction log entries the follower needs. The default autopurge retention (autopurge.snapRetainCount = 3) is usually fine; the more common cause is extended follower downtime. Keep planned maintenance windows shorter than the autopurge retention window.

Prevention

• Monitor per-node zxid deltas continuously. A few-hundred-transaction delta is the earliest sign of a slow follower, well before zk_pending_syncs becomes interesting.
• Scrape the leader explicitly for leader-only metrics. zk_pending_syncs, zk_synced_followers, and zk_followers exist only on the leader. Scrape all nodes and filter, or track leader identity and target it.
• Keep dataLogDir on a dedicated device on every ensemble member. This eliminates the most common cause of slow fsync at the source.
• Capacity-plan the write rate against the slowest follower, not the median. The leader’s write throughput is bounded by the slowest quorum member.
• Enable GC logging and ship it. When a follower starts lagging, you want the GC log already in hand.
• Gate alerts on zk_uptime. Cold starts produce transient pending syncs while followers catch up. Suppress non-critical alerts for the first 5 to 10 minutes after a restart.

How Netdata helps

• Per-second zk_pending_syncs collection on the leader. Sustained non-zero values are visible immediately rather than on a 60-second scrape cadence.
• Per-node zxid collection across the ensemble. Divergence between any follower and the leader is observable as a correlated step, faster than waiting for the leader’s aggregate metric to react.
• Per-follower fsync and JVM pause metrics. When zk_pending_syncs grows, pivot directly to the suspect follower’s zk_fsynctime and zk_jvm_pause_time_ms without leaving the correlated view.
• Anomaly detection on proposal and commit counters. Diverging zk_proposal_count and zk_commit_count rates surface the same fan-out stall from a different angle, helping separate “slow follower” from “quorum problem”.
• Leader identification in dashboards. Netdata tags which node is currently the leader, so leader-only metrics are not lost in an aggregate.

Related guides

• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper quorum ack latency high: followers slow to acknowledge proposals
• ZooKeeper outstanding requests growing: the request pipeline is backing up
• ZooKeeper proposals not committing: proposal_count outpacing commit_count
• ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls
• ZooKeeper leader election storm: an ensemble that keeps re-electing
• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper monitoring checklist: the signals every production ensemble needs