Kafka ZooKeeper on Netdata

Kafka NOT_LEADER_FOR_PARTITION: stale metadata, controller lag, and client retries

Mon, 01 Jan 0001 00:00:00 +0000

Producers and consumers log NOT_LEADER_FOR_PARTITION. Broker response metrics show spikes in failed produce or fetch requests. The cluster usually self-heals within seconds as clients refresh metadata. When the error persists for minutes, or flaps across many partitions, the root cause is typically a controller that cannot keep up with leadership changes. Distinguishing a routine leader election from a controller queue backup that blocks metadata propagation is the first step.

What this means

Kafka clients cache partition leadership metadata. When a leader moves (rolling restart, broker failure, preferred replica election), a client with a stale view sends requests to the previous leader. That broker returns NOT_LEADER_FOR_PARTITION. The Java client treats this as a retriable error and refreshes metadata eagerly. A short spike during a restart is normal and usually clears immediately.

Kafka ZooKeeper Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Kafka ZooKeeper Monitoring

What Is Kafka ZooKeeper?

Kafka ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. It is a critical component in the architecture of distributed systems, especially for managing and coordinating services like Apache Kafka.

Monitoring Kafka ZooKeeper With Netdata

Monitoring Kafka ZooKeeper with Netdata offers a comprehensive view of the system’s performance and health. To monitor Kafka ZooKeeper, Netdata uses an openmetrics (Prometheus) exporter called the Kafka ZooKeeper Exporter. This allows Netdata to ingest data from any Prometheus exporter efficiently. With Netdata, you get automated dashboards, alerts, and more without the need for a Prometheus server or Grafana. This streamlined approach with the Netdata $name monitoring tool ensures ease of use and rapid insights into your Kafka ZooKeeper instances.

ZooKeeper "fsync-ing the write ahead log took too long": the disk warning behind most write stalls

Mon, 01 Jan 0001 00:00:00 +0000

The warning:

fsync-ing the write ahead log in SyncThread:0 took 1234ms which will adversely effect operation latency...

fires when fsync on the transaction log exceeds fsync.warningthresholdms (default 1000ms). The wording is deliberate: every write in ZooKeeper blocks on a quorum of fsyncs. If fsync takes a second, every write takes a second. If fsync takes 10 seconds, you are one missed heartbeat away from a leader election.

This is the canary for the failure pattern the playbook calls “Disk Sync Deadlock”: the single most common cause of ZooKeeper outages, more than GC, more than network. When you see this warning, the bottleneck is almost never ZooKeeper itself. It is the disk under dataLogDir. The fix is usually storage, not config.

ZooKeeper "X is not executed because it is not in the whitelist": four-letter-word commands blocked

Mon, 01 Jan 0001 00:00:00 +0000

The error string is exact. When you run echo mntr | nc localhost 2181 against a ZooKeeper 3.5.3+ server that has not been configured for it, the server replies:

mntr is not executed because it is not in the whitelist.

Same shape for ruok, isro, stat, conf, envi, cons, wchs, and the rest of the four-letter-word (4lw) command set. Only srvr works out of the box, because the bundled zkServer.sh status check depends on it.

ZooKeeper dataLogDir sharing a disk with snapshots: the #1 fsync-latency footgun

Mon, 01 Jan 0001 00:00:00 +0000

You are chasing intermittent ZooKeeper write-latency spikes that appear to have no cause. Average latency is fine most of the time. Then, every few minutes, p99 update latency jumps by an order of magnitude, zk_outstanding_requests briefly climbs, and clients on tight timeouts see a flicker of connection churn. By the time you SSH in, the cluster looks healthy again.

The disk is not full, iostat averages look reasonable, and the spikes do not line up with any obvious workload change. The transaction log and snapshot directory are both on the same volume, and that is exactly the problem.