Kafka Operations Guides on Netdata

How Kafka actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How Kafka actually works in production: a mental model for operators

What it is and why it matters

Each broker is a concurrent system of subsystems sharing disk, network, page cache, and threads. Tail latency spikes and offline partitions usually stem from interactions between these subsystems, not single component failures.

Underneath the distributed commit log abstraction, each broker runs a reactor-pattern server with three saturation layers: network threads, a bounded request queue, and I/O handler threads. Between I/O threads and the log sits a delayed-operation timer wheel called the purgatory. A separate controller maintains cluster metadata in a single-threaded event queue. Consumer groups are managed by a Group Coordinator that hashes group IDs to the __consumer_offsets topic. Log data lives in segment files that rely on OS page cache and zero-copy sendfile. These subsystems compete for the same disk I/O, network bandwidth, file descriptors, and cache. Treating them as independent pipelines leads to unnecessary restarts and missed bottlenecks.

Kafka __consumer_offsets growing huge: compaction failure on the offsets topic

Mon, 01 Jan 0001 00:00:00 +0000

Kafka __consumer_offsets growing huge: compaction failure on the offsets topic

One broker’s disk is climbing faster than its peers, or __consumer_offsets has grown to multiple gigabytes while producer traffic is flat. This internal topic is compacted by default; its size should stay roughly proportional to active consumer groups and partitions. Unbounded growth means the log cleaner has stalled or crashed. The failure is silent: producers and consumers keep working, but every offset commit appends a record that compaction will never remove. Growth is not evenly distributed: __consumer_offsets is partitioned by group.id hash, so a stalled cleaner on one broker affects only the partitions in that broker’s log.dirs. Eventually the log directory fills, and the broker may mark it offline.

Kafka 'Broker may not be available': clients that can't connect or stay connected

Mon, 01 Jan 0001 00:00:00 +0000

Kafka ‘Broker may not be available’: clients that can’t connect or stay connected

Client logs show a warning like this:

WARN [Producer clientId=...] Connection to node -1 could not be established. Broker may not be available.

Python clients may raise kafka.errors.NoBrokersAvailable, while librdkafka-based clients report Connection refused against the broker address. The bootstrap server is often reachable; ping and telnet succeed, yet the client still fails. This happens because Kafka clients use the bootstrap connection only to fetch metadata. After that, they disconnect and try to open fresh TCP connections to the host:port pairs advertised in the metadata response. If those endpoints are unreachable, misconfigured, or secured differently than the client expects, the connection fails even though the bootstrap succeeded.

Kafka ActiveControllerCount not equal to 1: no controller or split brain

Mon, 01 Jan 0001 00:00:00 +0000

Kafka ActiveControllerCount not equal to 1: no controller or split brain

When the cluster-wide sum of kafka.controller:type=KafkaController,name=ActiveControllerCount is not 1, the cluster has either no active controller or multiple active controllers. A sum of 0 means no broker is steering metadata. A sum greater than 1 means split brain. Either way, this is a control-plane failure. Existing partition leaders usually keep serving produce and fetch requests, so the data plane may look healthy at first. Any broker failure, partition reassignment, or topic operation will stall because there is no controller to process it, or multiple controllers are processing conflicting operations.

Kafka authentication failures: SASL/mTLS errors, credential rotation, and brute force

Mon, 01 Jan 0001 00:00:00 +0000

Kafka authentication failures: SASL/mTLS errors, credential rotation, and brute force

AuthenticationException lines in broker logs are spiking and failed-authentication-total is climbing on one or more listeners. Producers or consumers are failing to connect, and every failed attempt consumes a network thread and a file descriptor before the broker closes the connection. The same metric pattern can mean routine credential rotation, a client misconfiguration, or an active brute-force attempt. Telling the difference determines whether you need a config fix, a secret rotation, or a security incident response.

Kafka authorization failures: ACL denials, wrong-topic clients, and audit trails

Mon, 01 Jan 0001 00:00:00 +0000

Kafka authorization failures: ACL denials, wrong-topic clients, and audit trails

Your consumers or producers throw TOPIC_AUTHORIZATION_FAILED or CLUSTER_AUTHORIZATION_FAILED. Your security team spots repeated log lines in the broker logs. You need to know within minutes whether this is a deployment misconfiguration, a client pointing at the wrong topic, or a security incident.

In clusters where authorizer.class.name is configured, Kafka writes authorization decisions to kafka-authorizer.log. Denials log at INFO level; allowed operations are silent unless you enable DEBUG logging for the authorizer.

Kafka broker out of disk: log.dirs full, the cliff-edge shutdown, and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Kafka broker out of disk: log.dirs full, the cliff-edge shutdown, and recovery

When a Kafka broker exhausts disk space on a log directory, it does not throttle. It marks that directory offline or crashes entirely. Partitions on the failed directory become unavailable immediately. If all log.dirs fail, the broker exits. You may see under-replicated partitions or producer timeouts seconds before failure, but often the first sign is a hard process exit with No space left on device.

Kafka CommitFailedException: rebalanced-out consumers and poll loop timeouts

Mon, 01 Jan 0001 00:00:00 +0000

Kafka CommitFailedException: rebalanced-out consumers and poll loop timeouts

CommitFailedException with the message that the group has already rebalanced and assigned the partitions to another member means the time between poll() calls exceeded max.poll.interval.ms. The coordinator evicted the consumer and rejected the in-flight offset commit.

When one consumer is evicted, the group rebalances. If other consumers are also slow, or if the rebalance itself takes long enough that healthy consumers miss the same deadline, the group enters a rebalance storm: it oscillates between JoinGroup and SyncGroup without stabilizing, and lag grows without bound.

Kafka connection storms: connection-count spikes, FD pressure, and network threads

Mon, 01 Jan 0001 00:00:00 +0000

Kafka connection storms: connection-count spikes, FD pressure, and network threads

Connection-count jumps on one or more brokers. An alert fires for open file descriptors, or consumers report timeouts while broker logs show nothing dramatic. In Kafka, a connection storm is insidious: every TCP connection costs one file descriptor and network-thread time, and the reactor architecture means network-thread saturation blocks everything, including metadata requests.

Incremental fetch sessions keep consumer connections open even when idle. This is normal, but it means brokers run with a high steady-state connection count. The danger is the rate of change, not the absolute number. When connections spike above twice the established baseline, network threads drop below 30% idle, and FD usage approaches the process limit, the broker is in a connection storm.

Kafka consumer group lag growing: detection, lag-as-time, and root causes

Mon, 01 Jan 0001 00:00:00 +0000

Kafka consumer group lag growing: detection, lag-as-time, and root causes

When a consumer group falls behind, lag climbs monotonically. If the committed offset crosses the retention boundary, consumers hit OffsetOutOfRangeException and must reset to earliest or latest, reprocessing or skipping data. Restarting the consumer is a common first reaction, but broker-side fetch latency, page cache eviction, and network saturation are equally common culprits. Detect lag accurately, convert it to time, and trace the root cause without guessing.

Kafka consumer group rebalancing too often: heartbeats, session timeout, and assignors

Mon, 01 Jan 0001 00:00:00 +0000

Kafka consumer group rebalancing too often: heartbeats, session timeout, and assignors

Consumer group lag is growing, application logs are full of JoinGroup and SyncGroup messages, and kafka-consumer-groups.sh shows the group flipping between Stable and PreparingRebalance. Healthy groups rebalance only during membership changes and planned deployments. More than two or three rebalances per hour for a stable group signals instability. The usual cause is a mismatch between processing latency and one of three timeouts: session.timeout.ms, heartbeat.interval.ms, or max.poll.interval.ms. The assignor strategy and static membership configuration determine how painful each rebalance is.

Kafka consumer group stuck Empty or Dead: no members consuming

Mon, 01 Jan 0001 00:00:00 +0000

Kafka consumer group stuck Empty or Dead: no members consuming

A consumer group that should be active shows Empty or Dead state with no members. For a real-time pipeline, every new message piles up unread. For a batch job, Empty between runs is expected. The difference between an intentional idle state and a silent outage is simple: is lag growing, and was the group supposed to be active?

An Empty group still exists in the coordinator. Committed offsets are retained until they expire, so consumers can resume on restart. A Dead group has been removed, usually because offsets expired while no members were active. If the group is Dead and data is still being produced, new consumers join a fresh group state and default to auto.offset.reset, which can skip or reprocess data.

Kafka consumer rebalance storm: stuck in PreparingRebalance and max.poll.interval.ms

Mon, 01 Jan 0001 00:00:00 +0000

Kafka consumer rebalance storm: stuck in PreparingRebalance and max.poll.interval.ms

Consumer group lag climbs while brokers report zero under-replicated partitions, normal produce and fetch latency, and normal request handler idle percent. The consumer group oscillates between Stable, PreparingRebalance, and CompletingRebalance without settling long enough to make progress. Every rebalance cycle pauses consumption; lag grows because time spent rebalancing dwarfs time spent fetching. This is a consumer rebalance storm. It is almost always a client-side timeout or processing issue. Look for CommitFailedException or max.poll.interval.ms exceeded in consumer logs.

Kafka controller event queue backing up: overwhelmed controller and stalled metadata

Mon, 01 Jan 0001 00:00:00 +0000

Kafka controller event queue backing up: overwhelmed controller and stalled metadata

You see NOT_LEADER_FOR_PARTITION errors spike in client logs. Leader elections stop completing. The active controller’s event queue grows and does not drain. Partitions that need new leaders stay offline. The single thread processing partition state changes cannot keep up, and metadata operations stall.

In ZooKeeper mode, each event writes to ZooKeeper. In KRaft mode, each event appends to the Raft metadata log. The active controller processes them sequentially. When the queue backs up, the control plane slows. The data plane may continue serving existing leaders, but any failure requiring a state change gets stuck.

Kafka disk I/O latency high: await, LocalTimeMs, and the slow-disk broker

Mon, 01 Jan 0001 00:00:00 +0000

Kafka disk I/O latency high: await, LocalTimeMs, and the slow-disk broker

iostat shows await climbing, maybe a disk alert fired. But await alone is not a pageable event. On SSDs and RAID arrays, %util hits 100% under modest load because it measures device busy time, not saturation. What matters is await, the average time for I/O requests to be served. At the Kafka layer, the mirror image is LocalTimeMs in the request latency breakdown. Both spike during normal operations – broker restart with cold page cache, log compaction, and partition reassignment all drive up disk latency without indicating hardware fault. This guide shows how to distinguish a transient spike from a slow disk that will shrink your ISR and block produce requests.

Kafka disk space planning: retention, replication, and runway estimation

Mon, 01 Jan 0001 00:00:00 +0000

Kafka disk space planning: retention, replication, and runway estimation

When a Kafka broker exhausts a log.dirs volume, the directory goes offline. If it is the only volume, the broker halts. Partitions on that volume become unavailable and the broker cannot rejoin until space is freed. Disk planning is a continuous operational calculation, not a one-time purchase. You need a working estimate of steady-state usage, a runway model against an operational full threshold, and an explicit map of the retention rules that actually delete data.

Kafka enable.auto.commit data loss: committed offsets that outrun processing

Mon, 01 Jan 0001 00:00:00 +0000

Kafka enable.auto.commit data loss: committed offsets that outrun processing

Downstream systems are missing events, but your Kafka consumer group reports lag near zero and the group state is Stable. There are no broker errors, no rebalances, and no visible backpressure. The application crashed or deployed minutes ago, then caught up instantly on recovery. Messages are lost silently. This is the signature of enable.auto.commit=true: offsets are committed to __consumer_offsets on a fixed schedule, not after processing completes.

Kafka FailedProduceRequestsPerSec rising: the single best 'producers are hurting' signal

Mon, 01 Jan 0001 00:00:00 +0000

Kafka FailedProduceRequestsPerSec rising: the single best ‘producers are hurting’ signal

FailedProduceRequestsPerSec rising is broker-side confirmation that producers are actively being rejected. Unlike client-side timeouts from network blips or producer memory pressure, it only increments when the broker receives a produce request and fails to complete it. The MBean kafka.server:type=BrokerTopicMetrics,name=FailedProduceRequestsPerSec rolls up server-side produce failures into a single rate: NotEnoughReplicasException, NotLeaderOrFollowerException, CorruptRecordException, and others. A sustained nonzero rate means data is not being accepted, and the window for data loss or unavailability is open.

Kafka fetch request latency high: FetchConsumer vs FetchFollower and page cache misses

Mon, 01 Jan 0001 00:00:00 +0000

Kafka fetch request latency high: FetchConsumer vs FetchFollower and page cache misses

Your tail consumers are lagging, or your under-replicated partition count is climbing. On the broker, kafka.network:type=RequestMetrics,name=TotalTimeMs,request=FetchConsumer or FetchFollower is elevated. Raw fetch latency is a poor signal: consumer long-polling means TotalTimeMs routinely includes the full fetch.max.wait.ms wait even on an idle topic, and follower fetches are paced by the leader’s ability to serve segments. The actionable metric is LocalTimeMs, the time the leader spends reading the log. When FetchConsumer LocalTimeMs spikes, the data was not in the OS page cache. When FetchFollower LocalTimeMs spikes, the leader is slow to serve replication reads and ISR shrinks will follow. The sections below show how to tell the two apart, confirm page cache misses, and fix the root cause.

Kafka ISR shrinking: IsrShrinksPerSec, flapping, and the cascade to offline

Mon, 01 Jan 0001 00:00:00 +0000

Kafka ISR shrinking: IsrShrinksPerSec, flapping, and the cascade to offline

IsrShrinksPerSec is climbing on your leaders and UnderReplicatedPartitions is no longer zero. If it is flapping – shrinks followed by expands every few minutes – the path ends with OfflinePartitionsCount rising and acks=all producers throwing NotEnoughReplicasException. This guide covers that path: how a lagging follower becomes a cluster-wide problem, how to separate flapping from one-way degradation, and how to stop the cascade before partitions go offline.

Kafka JVM heap and Full GC pauses: ISR drops, session timeouts, and right-sizing the heap

Mon, 01 Jan 0001 00:00:00 +0000

Kafka JVM heap and Full GC pauses: ISR drops, session timeouts, and right-sizing the heap

Sporadic UnderReplicatedPartitions and ISR shrinks that do not correlate with disk I/O or network faults, combined with consumer rebalances and NotEnoughReplicasException from producers using acks=all, point to broker JVM heap pressure. Check broker logs for GC pauses in the Old Generation lasting several seconds.

Brokers use the JVM heap for metadata, request buffers, and message format conversion. They do not store messages on the heap; the OS page cache handles that. When the heap is misconfigured or under pressure, garbage collection pauses can freeze a broker long enough to trigger ZooKeeper session expirations, follower lag, and cascading availability issues. Full GC pauses exceeding five seconds are the common threshold where these symptoms begin.

Kafka KRaft metadata log lag: standby controllers and brokers falling behind

Mon, 01 Jan 0001 00:00:00 +0000

Kafka KRaft metadata log lag: standby controllers and brokers falling behind

When client errors like NOT_LEADER_FOR_PARTITION or LeaderNotAvailableException hit a stable-looking cluster, check whether standby controllers and broker metadata lag are climbing. In KRaft mode, the active controller is the Raft quorum leader. It appends metadata changes to __cluster_metadata. Standby controllers and brokers apply that log asynchronously. When they fall behind, they act on stale partition leadership, outdated ISR memberships, and old broker registrations. This guide shows how to distinguish a slow follower, a sick leader, and a network partition.

Kafka KRaft quorum has no leader: current-leader = -1 and frozen metadata

Mon, 01 Jan 0001 00:00:00 +0000

Kafka KRaft quorum has no leader: current-leader = -1 and frozen metadata

Topic creation hangs. Partition reassignments stall. Broker logs show metadata operations timing out. On controller nodes, JMX reports kafka.server:type=raft-metrics,attribute=current-leader with value -1, and the quorum state is frozen. Existing producers and consumers continue to read and write, but the control plane is stuck.

What this means

The quorum leader acts as the active controller. When current-leader is -1, the local controller has not discovered a leader, which means no node in the quorum can commit new entries to the metadata log. Topic creation, deletion, configuration updates, reassignments, and ISR changes are blocked. Leader elections for partitions that lose their broker also cannot proceed.

Kafka LEADER_NOT_AVAILABLE: causes during elections, restarts, and topic creation

Mon, 01 Jan 0001 00:00:00 +0000

Kafka LEADER_NOT_AVAILABLE: causes during elections, restarts, and topic creation

LEADER_NOT_AVAILABLE means a client asked a broker to produce or fetch from a partition that has no assigned leader. In healthy clusters this is brief during rolling restarts, controller elections, or topic creation. Persistent errors correlate with OfflinePartitionsCount > 0 and indicate the data plane is broken for those partitions. Distinguish this from NOT_LEADER_FOR_PARTITION, which means a leader exists but the client contacted the wrong broker and needs a metadata refresh.

Kafka LeaderElectionRateAndTimeMs spiking: election storms and slow elections

Mon, 01 Jan 0001 00:00:00 +0000

Kafka LeaderElectionRateAndTimeMs spiking: election storms and slow elections

LeaderElectionRateAndTimeMs climbing outside maintenance windows means the controller is struggling to keep the partition map consistent. Producers and consumers may see NOT_LEADER_FOR_PARTITION or request timeouts. Partitions can hang under-replicated or offline while the controller works through a backlog.

This metric has two dimensions. The rate is how often leadership changes. The time is how long the metadata store takes to commit each change. High rate with low time means broker flapping or repeated administrative actions. Normal rate with high time means the controller event queue is backed up, ZooKeeper writes are slow, or the KRaft quorum is lagging. Determine which mode you are in before the cluster degrades further.

Kafka leadership imbalance: LeaderCount skew and preferred replica election

Mon, 01 Jan 0001 00:00:00 +0000

Kafka leadership imbalance: LeaderCount skew and preferred replica election

One broker in your cluster is handling 40 percent of all produce requests. Its RequestHandlerAvgIdlePercent is dropping toward 0.3 while the rest of the cluster is nearly idle. Cluster-level throughput dashboards look healthy because aggregate ingress is within capacity, but tail latency on the hot broker is climbing and consumers connected to it are falling behind. The root cause is leadership skew: a disproportionate number of partition leaders have landed on one broker, and every leader carries the full request load for its partitions. This happens silently after rolling restarts, controlled shutdowns, or broker decommissions. Even a perfectly even PartitionCount across brokers can hide severe leadership imbalance. You will not see it without monitoring LeaderCount per broker.

Kafka log compaction falling behind: the dead cleaner thread and unbounded disk growth

Mon, 01 Jan 0001 00:00:00 +0000

Kafka log compaction falling behind: the dead cleaner thread and unbounded disk growth

Disk utilization climbs steadily on one or more brokers while producer traffic stays flat. __consumer_offsets balloons from a few gigabytes to hundreds. Under-replicated partitions are at zero, producers show no errors, and standard Kafka alerts are silent. The usual culprit is a dead log cleaner thread: it hit a corrupt record or an OOM during compaction, crashed, and never restarted. Every compacted topic now accumulates records without bound.

Kafka Log directory failed / OfflineLogDirectoryCount > 0: disk errors and JBOD recovery

Mon, 01 Jan 0001 00:00:00 +0000

Kafka Log directory failed / OfflineLogDirectoryCount > 0: disk errors and JBOD recovery

What this means

When Kafka catches an IOException on a log.dirs path, it marks that log directory offline. The broker increments kafka.log:type=LogManager,name=OfflineLogDirectoryCount and logs the failure. Partitions with replicas on the failed directory lose those replicas. If the partition leader was on that directory and unclean.leader.election.enable=false, the partition becomes unavailable until the controller elects a new leader from the remaining ISR. Producers with acks=all see NotEnoughReplicasException when the surviving ISR drops below min.insync.replicas.

Kafka LogFlushRateAndTimeMs high: fsync latency and a failing disk

Mon, 01 Jan 0001 00:00:00 +0000

Kafka LogFlushRateAndTimeMs high: fsync latency and a failing disk

When kafka.log:type=LogFlushStats,name=LogFlushRateAndTimeMs p99 stays above 200 ms, the broker’s fsync path is slow. On SSD-backed clusters, p99 should stay below 50 ms; sustained values above 2 s point to disk degradation. Because most deployments leave log.flush.interval.messages and log.flush.interval.ms unset and rely on replication plus OS lazy flush, this metric reflects kernel-driven flushes or explicit fsyncs. A slow flush raises produce LocalTimeMs, blocks request handler threads, and can push followers out of ISR. This guide shows how to tell whether the cause is a failing disk, a bad flush policy, or transient I/O contention.

Kafka min.insync.replicas and acks: configuring durability you actually have

Mon, 01 Jan 0001 00:00:00 +0000

Kafka min.insync.replicas and acks: configuring durability you actually have

Most operators set producers to acks=all and assume the cluster acks only when every replica has the message. It does not. With acks=all, the broker waits only for the current in-sync replica set (ISR). Because the ISR shrinks dynamically when followers lag, a partition with replication factor three can have an ISR of one – the leader itself. Without raising min.insync.replicas from its default, the leader acks with zero followers caught up. Your durability guarantee collapses to leader-only persistence, and you only find out when the leader dies and data is missing.

Kafka monitoring checklist: the signals every production cluster needs

Mon, 01 Jan 0001 00:00:00 +0000

Kafka monitoring checklist: the signals every production cluster needs

Kafka failures follow predictable paths: an ISR shrinks, a controller queue backs up, a disk fills while the cleaner thread hangs, or a consumer rebalance storm hides behind healthy broker metrics. You need to know which signals matter and when they justify a 3 AM page.

This checklist organizes broker-side signals into four levels. Each builds on the last: Level 1 prevents data loss. Level 2 prevents surprises. Level 3 exposes leading indicators. Level 4 catches silent killers. Use it to audit dashboards, tune alert severity, and justify instrumentation.

Kafka monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Kafka monitoring maturity model: from survival to expert

Broker logs and a single health check are not enough for production Kafka. Monitoring everything at once creates noise and hides the signals that prevent outages. This model gives you a prioritized path: start with telemetry that prevents total data loss, then add layers that catch degradation before it becomes an incident.

Use this as an onboarding checklist for new clusters, an incident reference, and a roadmap when you have instrumentation budget. Signals come from Kafka JMX, OS metrics, and admin APIs. In KRaft mode (mandatory in Kafka 4.0+), substitute ZooKeeper-specific signals with the corresponding Raft quorum metrics.

Kafka network egress saturation: BytesOutPerSec, replication amplification, and fan-out

Mon, 01 Jan 0001 00:00:00 +0000

Kafka network egress saturation: BytesOutPerSec, replication amplification, and fan-out

Kafka is usually sized for producer ingress because that is what the business reports. In practice, the first resource to saturate is almost always network egress. Every byte written to a partition leader is read at least once by each follower replica and once by every active consumer group keeping up with the log. A broker can hit its NIC ceiling while producer throughput looks comfortable and disk I/O is barely warmed up.

Kafka NetworkProcessorAvgIdlePercent low: network thread saturation and TLS overhead

Mon, 01 Jan 0001 00:00:00 +0000

Kafka NetworkProcessorAvgIdlePercent low: network thread saturation and TLS overhead

NetworkProcessorAvgIdlePercent drops on one or more brokers. A ticket may fire when the value falls below 0.3, or producers and consumers may timeout while broker CPU looks fine and I/O threads are not obviously saturated. Network thread saturation blocks all socket activity, including metadata requests, so every client suffers. With the default num.network.threads set to 3, brokers running TLS and dense client pools are especially vulnerable. Sustained values below 0.1 make the broker effectively unreachable even though the process is still running.

Kafka NOT_LEADER_FOR_PARTITION: stale metadata, controller lag, and client retries

Mon, 01 Jan 0001 00:00:00 +0000

Kafka NOT_LEADER_FOR_PARTITION: stale metadata, controller lag, and client retries

Producers and consumers log NOT_LEADER_FOR_PARTITION. Broker response metrics show spikes in failed produce or fetch requests. The cluster usually self-heals within seconds as clients refresh metadata. When the error persists for minutes, or flaps across many partitions, the root cause is typically a controller that cannot keep up with leadership changes. Distinguishing a routine leader election from a controller queue backup that blocks metadata propagation is the first step.

Kafka NotEnoughReplicasException: acks=all writes rejected below min.insync.replicas

Mon, 01 Jan 0001 00:00:00 +0000

Kafka NotEnoughReplicasException: acks=all writes rejected below min.insync.replicas

Producers are throwing org.apache.kafka.common.errors.NotEnoughReplicasException or NotEnoughReplicasAfterAppendException, and acks=all writes are failing while acks=1 or acks=0 writes may still succeed. The affected partitions no longer have enough in-sync replicas to satisfy min.insync.replicas. The immediate operational question is whether the ISR shrink is a transient recovery blip or a sustained degradation that will block writes until you fix the follower.

What this means

The leader tracks followers caught up within replica.lag.time.max.ms in the In-Sync Replica set (ISR). For acks=all, the leader waits for all current ISR members before acknowledging the producer. If the ISR size drops below min.insync.replicas, the leader rejects the produce request. The broker-level default for min.insync.replicas is 1, so a lone leader can acknowledge alone. In practice, with replication.factor=3 and acks=all, set min.insync.replicas=2 so a single follower loss blocks writes instead of silently weakening durability.

Kafka OfflinePartitionsCount > 0: partitions with no leader and how to recover

Mon, 01 Jan 0001 00:00:00 +0000

Kafka OfflinePartitionsCount > 0: partitions with no leader and how to recover

When kafka.controller:type=KafkaController,name=OfflinePartitionsCount is nonzero, at least one partition has no active leader. Those partitions are completely unavailable: producers receive errors, consumers stall, and no data is written or read until a leader is elected. This is a data-plane outage.

This metric is only meaningful on the active controller. Non-controller brokers always report zero. If the controller itself is down, the metric may be stale or unreachable at the exact moment you need it. Brief spikes can occur during controller re-election or ungraceful broker shutdown, but any sustained nonzero value past 60 seconds is an active incident that requires immediate intervention.

Kafka OffsetOutOfRangeException: when retention deletes data before the consumer reads it

Mon, 01 Jan 0001 00:00:00 +0000

Kafka OffsetOutOfRangeException: when retention deletes data before the consumer reads it

OffsetOutOfRangeException means the consumer requested an offset the broker no longer holds. The log segment containing the consumer’s committed position was deleted by retention before the consumer caught up. This is not a transient fetch error; it is data loss, and the outcome depends entirely on auto.offset.reset. Many clients default to latest, which turns this exception into silent skipping.

Kafka page cache thrashing: the backfill consumer that 100x's tail latency

Mon, 01 Jan 0001 00:00:00 +0000

Kafka page cache thrashing: the backfill consumer that 100x’s tail latency

Tail latency jumps from milliseconds to seconds while producer throughput and replication stay flat. CPU is normal. Every fetch request from tail consumers starts hitting disk. The culprit is usually a single backfill consumer reading historical data, evicting the hot working set from the OS page cache and turning a memory-speed system into a disk-bound one.

The write path stays green. Under-replicated partitions do not increase. No broker has crashed. The only visible signs are elevated disk read latency and slow consumers. If the cluster slowed suddenly with no broker fault, suspect page cache thrashing from a backfill consumer.

Kafka produce request latency high: reading the TotalTimeMs breakdown

Mon, 01 Jan 0001 00:00:00 +0000

Kafka produce request latency high: reading the TotalTimeMs breakdown

Producers are timing out or retrying. Client-side request-latency-avg is elevated and broker TotalTimeMs p99 is spiking. The total is unactionable by itself. Kafka breaks it into five sub-components, each implicating a different subsystem. You need all five via JMX or an equivalent metrics collector to know whether to fix disk, scale a thread pool, or replace a follower.

What this means

TotalTimeMs is the wall-clock time from when the broker’s network thread receives a produce request until the response is fully sent. It is the arithmetic sum of five phases:

Kafka producer timeout cascade: when retries pile load onto a slow broker

Mon, 01 Jan 0001 00:00:00 +0000

Kafka producer timeout cascade: when retries pile load onto a slow broker

Your producers are timing out. P99 produce latency is climbing. You see more requests hitting the brokers, yet actual throughput of new messages is flat or falling. This is not a traffic spike. It is a producer timeout cascade: one slow broker causes clients to retry, and those retries add load to the same overloaded broker, closing the loop until the cluster is pinned.

Kafka purgatory size growing: delayed produce and fetch operations

Mon, 01 Jan 0001 00:00:00 +0000

Kafka purgatory size growing: delayed produce and fetch operations

JMX shows kafka.server:type=DelayedOperationPurgatory,name=PurgatorySize climbing on one or more brokers. Produce purgatory holds acks=all requests waiting for ISR completion. Fetch purgatory holds consumer and follower requests waiting for fetch.min.bytes. A growing queue means requests spend more time inside the broker than clients expected. Producers time out and retry. Consumers sit idle. The cluster is not dead, but it is backing up at a precise choke point. This guide distinguishes normal long-polling from replication crisis.

Kafka quota throttling: throttle-time, runaway clients, and protecting the cluster

Mon, 01 Jan 0001 00:00:00 +0000

Kafka quota throttling: throttle-time, runaway clients, and protecting the cluster

Your Kafka producers are timing out. Consumer lag is growing. Client dashboards show elevated request latency, but the brokers are healthy: UnderReplicatedPartitions is zero and disk I/O looks fine. The culprit is often quota throttling: the cluster is enforcing per-client or per-user byte-rate limits, and one or more clients have hit the wall. The broker exposes this as throttle-time in JMX. That backpressure slows the client, but if the client is a runaway service or a backfill consumer, the throttling can cascade into widespread latency.

Kafka RecordTooLargeException / MESSAGE_TOO_LARGE: message size limits across the path

Mon, 01 Jan 0001 00:00:00 +0000

Kafka RecordTooLargeException / MESSAGE_TOO_LARGE: message size limits across the path

Two errors surface: the producer client throws RecordTooLargeException before the request hits the wire, or the broker returns MESSAGE_TOO_LARGE in the produce response. Both mean a record batch exceeds a limit somewhere in the path, but the fix depends on exactly which limit and where the size is measured. These settings are not a single knob: max.request.size, message.max.bytes, max.message.bytes, replica.fetch.max.bytes, fetch.max.bytes, max.partition.fetch.bytes, and socket.request.max.bytes must all align. Raise one limit without raising the others and you will wedge replication, strand consumers, or silently lose data on the next large message.

Kafka replica MaxLag growing: slow followers and replica fetcher health

Mon, 01 Jan 0001 00:00:00 +0000

Kafka replica MaxLag growing: slow followers and replica fetcher health

When kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica climbs on a broker, the worst follower is failing to replicate fast enough. This metric is the maximum offset distance between a leader and its most lagging follower. In a healthy cluster it stays near zero. If the gap persists longer than replica.lag.time.max.ms, the leader removes the follower from the ISR. Once enough replicas drop, partitions can fall below min.insync.replicas, and producers using acks=all hit NotEnoughReplicasException.

Kafka request queue filling up: RequestQueueSize, queued.max.requests, and backpressure

Mon, 01 Jan 0001 00:00:00 +0000

Kafka request queue filling up: RequestQueueSize, queued.max.requests, and backpressure

Your Kafka producers are timing out. Broker logs show no errors, but client-side metrics reveal growing latency and retries. On the broker, RequestQueueSize climbs toward queued.max.requests (default 500). Once the queue fills, network threads block on enqueue and stop reading from their sockets. Clients see TCP backpressure, time out, and retry. Retries add load, deepening the queue. This feedback loop correlates with falling RequestHandlerAvgIdlePercent.

Kafka REQUEST_TIMED_OUT: produce requests that expire before replication completes

Mon, 01 Jan 0001 00:00:00 +0000

Kafka REQUEST_TIMED_OUT: produce requests that expire before replication completes

Producers using acks=all throw TimeoutException (error code REQUEST_TIMED_OUT) when the broker accepts a produce request but cannot complete replication before request.timeout.ms expires. The leader appends the record to its local log and waits in purgatory for acknowledgments from all in-sync replicas. If the ISR ack does not arrive before the client deadline, the producer disconnects and surfaces the error. The broker may still finish the write, but the producer has already moved on, creating a hidden replication backlog and a potential retry storm.

Kafka RequestHandlerAvgIdlePercent low: I/O thread saturation and overload

Mon, 01 Jan 0001 00:00:00 +0000

Kafka RequestHandlerAvgIdlePercent low: I/O thread saturation and overload

You get paged because RequestHandlerAvgIdlePercent is below 0.3 and falling. This is an exponentially weighted moving average, not a spike metric. A low value means the broker’s I/O handler threads have been saturated long enough that the request queue is backing up and clients are timing out.

Treat above 0.5 as healthy, below 0.3 as critical, and below 0.1 as active overload. Even a stable 0.45 is dangerous: one broker failure can shift enough load to collapse the survivors.

Kafka retention not deleting old segments: retention.ms, retention.bytes, and the active segment

Mon, 01 Jan 0001 00:00:00 +0000

Kafka retention not deleting old segments: retention.ms, retention.bytes, and the active segment

You set retention.ms to 24 hours, but broker disk keeps climbing. A partition shows segment files older than the threshold, or the active segment has grown so large it consumes most of the volume. A topic with both retention.ms and retention.bytes may still appear to ignore them.

Kafka retention is not a continuous sweep. retention.ms and retention.bytes apply independently to closed segments, and the check interval adds latency. The active segment is never deleted by retention alone. On compacted topics, retention.bytes is ignored. Per-topic overrides shadow broker defaults.

Kafka Too many open files: file descriptor exhaustion from segments and connections

Mon, 01 Jan 0001 00:00:00 +0000

Kafka Too many open files: file descriptor exhaustion from segments and connections

Producers time out, consumers disconnect, and the broker log shows java.io.IOException: Too many open files. The broker may still run, but it cannot open new log segments or accept additional TCP connections. File descriptor exhaustion is a cliff-edge failure: the broker operates normally until it hits the hard limit, then the data path stops.

Kafka brokers hold a file descriptor for every log segment and every network connection. Each partition maintains an active segment and older retained segments. A broker with thousands of partitions and tens of segments each, plus hundreds of client connections, can hold tens of thousands of open file descriptors. The default Linux per-process limit of 1024 is far below production requirements.

Kafka too many partitions per broker: controller load, recovery time, and the 4000 guideline

Mon, 01 Jan 0001 00:00:00 +0000

Kafka too many partitions per broker: controller load, recovery time, and the 4000 guideline

Every replica on a Kafka broker carries fixed overhead: file descriptors, memory-mapped index files, replication fetcher load, and controller metadata. The guideline of roughly 4000 partitions per broker is not a hard architectural limit. It is an operational warning based on a serial bottleneck: the controller processes partition state changes one at a time. When a broker with 5000 partitions fails, the active controller queues thousands of events. Until the queue drains, leader elections stall, metadata propagation slows, and clients see NOT_LEADER_FOR_PARTITION. Restarting the broker forces every replica to recover its logs and catch up from leaders before rejoining the ISR. Recovery time is a step function; a routine restart can become a multi-hour incident. Partition count is easy to ignore in steady state and catastrophic to discover during an outage.

Kafka UncleanLeaderElectionsPerSec > 0: confirmed silent data loss

Mon, 01 Jan 0001 00:00:00 +0000

Kafka UncleanLeaderElectionsPerSec > 0: confirmed silent data loss

Your UncleanLeaderElectionsPerSec alert fired. The JMX metric kafka.controller:type=ControllerStats,name=UncleanLeaderElectionsPerSec shows OneMinuteRate > 0, or Count has incremented since your last check. This is confirmed data loss: a partition leader was elected from outside the ISR, and acknowledged records that the new leader does not possess are silently truncated.

Producers that received acks for those records were given a durability guarantee the cluster just broke. Stop additional loss, understand scope, and fix the conditions that allowed it.

Kafka UnderMinIsrPartitionCount: confirming the write path is blocked

Mon, 01 Jan 0001 00:00:00 +0000

Kafka UnderMinIsrPartitionCount: confirming the write path is blocked

UnderMinIsrPartitionCount is non-zero and acks=all producers are failing. Unlike UnderReplicatedPartitions, which only opens a durability window, this metric confirms the broker is actively rejecting writes. It counts leader partitions where the in-sync replica set has shrunk below min.insync.replicas. In steady state it must be zero.

This article explains the metric, when to PAGE, and the exact checks to run when it fires.

Kafka UnderReplicatedPartitions > 0: the most important metric and how to clear it

Mon, 01 Jan 0001 00:00:00 +0000

Kafka UnderReplicatedPartitions > 0: the most important metric and how to clear it

UnderReplicatedPartitions climbing above zero means a follower has fallen behind, the ISR has shrunk, and the cluster’s durability guarantee is degraded. One more failure on the wrong broker could make partitions unavailable or cause acks=all writes to be rejected. Determine whether this is a transient blip from maintenance or the start of a cascading replication failure.

What this means

The JMX MBean kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions reports the number of partitions where this broker is the leader and the ISR count is below the configured replication factor. It is a per-broker gauge. A broker without leadership always reports zero, even when the cluster is degraded, so aggregate across all brokers to see the full picture.

Kafka ZooKeeper request latency high: metadata slowdowns in ZK mode

Mon, 01 Jan 0001 00:00:00 +0000

Kafka ZooKeeper request latency high: metadata slowdowns in ZK mode

If you run Kafka in ZooKeeper mode and kafka.server:type=ZooKeeperClientMetrics,name=ZooKeeperRequestLatencyMs is climbing, a sustained p99 above 100 ms means the controller event queue is already backing up. Above 1 s, you approach the zookeeper.session.timeout.ms boundary and brokers risk session expiry. In ZK mode, every leader election, ISR change, and topic update flows through the ensemble. When ZK stalls, the controller stalls, metadata propagation freezes, and the cluster degrades in a way that looks like a broker problem but originates in the metadata plane.

Kafka ZooKeeper session expired: GC pauses, ISR drops, and controller loss

Mon, 01 Jan 0001 00:00:00 +0000

Kafka ZooKeeper session expired: GC pauses, ISR drops, and controller loss

In ZooKeeper mode, a session expiry means ZK declared a broker dead. The Java process may still be running, but the cluster treats it as gone. If the broker was the controller, the metadata plane re-elects. If it was a leader, every partition it led starts a new leader election. If it was a follower, leaders remove it from the ISR. The default zookeeper.session.timeout.ms is 18000 ms, and the most common trigger is a Full GC pause longer than that window. This guide applies to Kafka clusters running in ZooKeeper mode; KRaft mode does not use ZK sessions.