$ guides / kafka ▌

KAFKA · OPERATIONS PLAYBOOK

Running Kafka in production, without the 3 a.m. surprises

A distributed commit log with an ISR durability contract, a single-threaded controller, fork-free fork-style replication, and a performance model that lives in the OS page cache. The mental model of how Kafka actually behaves under load, where it tends to break, what to monitor as your operation matures, and the runbooks for the incidents you'll see.

> Start with the monitoring checklist → # Jump to the full guide list

Kafka is famously durable and famously unforgiving once the partition count, the write rate, or the consumer fan-out grows past what the defaults assumed.

The defaults work. Until a follower falls behind replica.lag.time.max.ms, the ISR shrinks below min.insync.replicas, and every acks=all producer starts getting NotEnoughReplicasException. Until a leader dies with no in-sync replica and the partition goes offline — or, if someone enabled unclean elections, comes back as a behind follower with acknowledged data silently gone. Until a consumer's offset slips behind retention and it throws OffsetOutOfRangeException. Until the log cleaner thread dies on a corrupt record, stops compacting __consumer_offsets, and the disk fills weeks later. Until a broker hits the FD limit and answers with Too many open files.

These guides are written for engineers who already run Kafka, not for people learning what a topic is. The goal is to give you the mental model of how the cluster actually behaves under load, the failure patterns that keep recurring, the monitoring story that catches problems before they page anyone, and the runbooks you wish someone had handed you before your last incident.

How Kafka actually runs in production

Kafka is not just a queue. It is a reactor pattern feeding a replicated commit log, served almost entirely from the OS page cache, coordinated by a single controller and a per-group coordinator. Most production failures live between these layers, not inside any one of them.

producers / clients

Producers, consumers, replica fetchers, Connect workers, and admin clients. Each connection costs a file descriptor and network-thread time. The <code>acks</code> setting (0, 1, all) decides how much durability a write actually buys, and producer retries can turn one slow broker into a load amplifier.

CLIENT

network threads + request queue

<code>num.network.threads</code> accept connections and read requests, then hand them to a bounded request queue (<code>queued.max.requests</code>, default 500). TLS handshakes burn CPU here. When the queue fills, network threads block and clients see connection-level backpressure.

NETWORK

I/O handler threads + purgatory

<code>num.io.threads</code> (default 8) do the real work — append produces, serve fetches, answer metadata. <code>acks=all</code> produces and long-poll fetches park in the purgatory timer wheel until satisfied. <code>RequestHandlerAvgIdlePercent</code> is the best single saturation signal.

EXEC

the log (segments + page cache)

Each partition is a directory of segment files. Writes append to the active segment; reads are served from the OS page cache via zero-copy <code>sendfile</code> when the working set fits. A backfill consumer that reads cold data evicts the hot set and turns every tail read into a disk read.

LOG

replication + ISR

Each partition has one leader and N-1 followers. Followers caught up within <code>replica.lag.time.max.ms</code> form the In-Sync Replica set. The ISR is the durability contract: drop below <code>min.insync.replicas</code> and <code>acks=all</code> writes are rejected; lose the last in-sync replica and the partition goes offline.

REPLICA

controller + metadata

Exactly one broker is the active controller, processing leadership changes and ISR updates sequentially from a single-threaded event queue. In ZooKeeper mode the metadata lives in ZK; in KRaft mode (mandatory in 4.0+) it lives in a Raft quorum. A backed-up controller stalls leadership and brokers disagree about who owns a partition.

CONTROL

consumer groups + coordinator

A group coordinator (the leader of the relevant <code>__consumer_offsets</code> partition) tracks joins, heartbeats, offset commits, and rebalances. A consumer that misses <code>max.poll.interval.ms</code> is kicked, triggering a rebalance — and one slow consumer can start a storm the group never recovers from.

GROUP

disk + log directories

Local NVMe, EBS, or JBOD volumes under <code>log.dirs</code>. Kafka does not degrade gracefully when a disk fills or errors — it takes the log directory offline. Disk <code>await</code> and free space are cliff-edge signals, and multiple <code>log.dirs</code> fill unevenly.

DISK

JVM heap + GC

A modest 4–8 GB heap holds metadata, connection state, and request buffers — not messages. A Full GC longer than <code>zookeeper.session.timeout.ms</code> ejects the broker from the cluster and shrinks ISRs. An oversized heap means longer pauses and less RAM for the page cache that Kafka actually depends on.

JVM

Why this matters: 'Kafka is slow' or 'producers are failing' can come from a lagging follower, a backed-up controller, a saturated I/O-thread pool, a page-cache eviction storm, a slow disk, a Full GC pause, or a rebalancing consumer group. The symptom rhymes but each layer has a different signal — and a different fix.

The failures you'll actually see

Most Kafka incidents fall into a small set of recurring patterns. Recognise the shape, and triage gets dramatically faster.

CRITICAL

The ISR shrink to blocked writes

A follower can't keep up — slow disk, network blip, GC pause — and the leader drops it from the ISR. If enough followers fall out, the ISR drops below min.insync.replicas and every acks=all producer is rejected with NotEnoughReplicasException. This is Kafka's most common degradation mode, and UnderMinIsrPartitionCount is the signal that confirms writes are actually blocked.

IsrShrinksPerSec elevated without matching IsrExpandsPerSec
UnderReplicatedPartitions nonzero across one or more leaders
UnderMinIsrPartitionCount > 0 (acks=all writes rejected)
FailedProduceRequestsPerSec rising in step

Investigate →

CRITICAL

The partition goes leaderless

A leader dies and no ISR member is available to take over. With unclean.leader.election.enable=false (the safe default), the partition goes completely offline — no reads, no writes — and clients see LEADER_NOT_AVAILABLE. OfflinePartitionsCount on the controller is the ground truth; every second offline is messages dropped or queued on the producer side.

OfflinePartitionsCount > 0 on the active controller
Clients logging LEADER_NOT_AVAILABLE for the affected partitions
All replicas for a partition on brokers that are down
RF=1 topics losing their single broker

Investigate →

ACTIVE

The consumer rebalance storm

A consumer is slow or crashes, a rebalance starts, consumption pauses, other consumers miss their max.poll.interval.ms deadline, get kicked, and trigger another rebalance. The group oscillates through PreparingRebalance and CompletingRebalance, never stabilizing, while lag grows each cycle. Broker metrics look healthy — this is a client-side problem surfacing through CommitFailedException.

Consumer group state flipping, never reaching Stable
CommitFailedException and 'max.poll.interval.ms exceeded' in consumer logs
Consumer lag growing with every rebalance cycle
JoinGroup / SyncGroup request rates elevated on the coordinator

Investigate →

IMMINENT

The controller queue backup

Several brokers fail at once, or a broker with thousands of partitions dies. The single-threaded controller event queue grows faster than it drains — each leadership change is a ZK write or a Raft append. Partitions that need a new leader sit in the queue, offline but not yet processed, and brokers disagree about leadership, so clients see NOT_LEADER_FOR_PARTITION. Restarting more brokers only generates more events.

ControllerEventQueueSize growing without draining
Clients logging NOT_LEADER_FOR_PARTITION on stale metadata
LeaderElectionRateAndTimeMs election time climbing
OfflinePartitionsCount misleadingly low (reflects what the controller knows)

Investigate →

CRITICAL

The disk cliff

A volume under log.dirs fills, or the filesystem returns an I/O error. Kafka does not degrade gracefully — it takes the log directory offline and the partitions on it become unavailable. The log records Shutdown log directory and OfflineLogDirectoryCount goes nonzero. Often the underlying cause is a silently dead log cleaner that stopped compacting weeks ago.

OfflineLogDirectoryCount > 0
'Shutdown log directory' / 'Log directory failed' in server.log
Disk free on a log.dirs volume near zero, or disk I/O errors in dmesg
Disk growing on stable traffic (dead cleaner, max-dirty-percent high)

Investigate →

IMMINENT

The producer timeout cascade

A broker slows down — disk, GC, or an overloaded I/O-thread pool — and acks=all produce requests start expiring with REQUEST_TIMED_OUT. Producers retry (default retries is effectively unbounded), piling more requests onto the already-slow broker. Request rate rises while successful throughput falls, the request queue fills, and the loop feeds itself.

Producers seeing REQUEST_TIMED_OUT / TimeoutException
RequestHandlerAvgIdlePercent dropping below 0.3, RequestQueueTimeMs growing
Produce TotalTimeMs spiking (RemoteTimeMs for acks=all)
BytesInPerSec rising while MessagesInPerSec stays flat (retries)

Investigate →

Kafka monitoring maturity levels

Kafka observability works in four practical levels. Each is a complete operation, not a stepping stone. Pick the level that matches how much your Kafka matters. Most production clusters should land at the second level.

Level 1: Survival

Know that something is wrong

Survival monitoring is the floor. With these signals you can answer one question: is the cluster still alive and is data still flowing? You will not learn what broke, but you will learn that something broke before users do. Survival is enough for dev clusters and non-critical pipelines.

Broker process liveness Is each broker process up and is port 9092 reachable?
UnderReplicatedPartitions Any nonzero value means the durability window is open.
OfflinePartitionsCount On the controller — any nonzero value is an active outage.
ActiveControllerCount Cluster-wide sum must be exactly 1.
Consumer lag for critical groups Are your important consumers keeping up with production?
Disk space on every log.dirs volume Kafka does not degrade gracefully when a disk fills.
JVM heap and Full GC occurrence A Full GC longer than the session timeout ejects the broker.

Level 2: Operational

Diagnose most incidents on your own

Operational monitoring is what most production clusters should target. Survival tells you something is wrong; operational tells you what. With this coverage your team can usually diagnose an incident on its own: ISR shrinks, thread saturation, request latency, consumer rebalances, write rejection.

IsrShrinksPerSec / IsrExpandsPerSec Are replicas falling out of sync, and are they coming back?
UnderMinIsrPartitionCount Confirms acks=all writes are actually being rejected.
UncleanLeaderElectionsPerSec Any occurrence is confirmed, silent data loss.
RequestHandlerAvgIdlePercent Best single broker-saturation signal — keep it above 0.5.
NetworkProcessorAvgIdlePercent Network-thread saturation affects every client.
Produce / FetchConsumer p99 latency TotalTimeMs and its breakdown, not just the total.
BytesInPerSec / BytesOutPerSec Throughput and egress vs NIC capacity per broker.
FailedProduceRequestsPerSec The single best 'producers are hurting' signal.
Consumer group state Stuck in PreparingRebalance is a rebalance storm.
Open file descriptors vs limit Cliff-edge: 'Too many open files' breaks the broker.

Level 3: Mature

Catch problems before they become incidents

Mature monitoring catches problems before they wake anyone up. The controller event queue creeping, the log cleaner falling behind, leadership drifting onto one broker, page cache starting to thrash, purgatory filling. None of these will page you on day one. They become page-out incidents on day thirty.

RequestQueueSize / ResponseQueueSize Pressure between network threads and I/O threads.
Produce / Fetch purgatory size Produce purgatory growing means followers are slow to ack.
ControllerEventQueueSize Growing without draining means the controller is overwhelmed.
LeaderElectionRateAndTimeMs Election storms and slow elections signal instability.
Log cleaner max-dirty-percent / DeadThreadCount The #1 silent killer — a dead cleaner fills the disk.
Page cache pressure (pgmajfault rate) Rising major faults mean reads are hitting disk, not cache.
Consumer lag as time (lag / produce rate) Alert on lag-as-time, not absolute offsets.
Consumer rebalance rate per group More than 2–3 per hour for a stable group is abnormal.
Partition / LeaderCount per broker Is leadership skewed onto one broker?

Level 4: Expert

Reactive instrumentation after real incidents

Expert signals enter your stack the day after a specific incident proved you needed them. Offline log directory counts, TCP retransmits, follower fetch breakdowns, KRaft metadata lag, message down-conversion. Most teams never need every signal here. Add the ones your incident history says you do.

OfflineLogDirectoryCount A log directory going offline is a binary disk/FS failure.
FetchFollower latency breakdown Which follower fetch is being served slowly?
TCP retransmit rate per broker Network degradation behind replication and fetch latency.
Per-partition lag distribution Catches hot partitions and skew the aggregate hides.
MessageConversionsPerSec Old clients forcing on-heap down-conversion, losing zero-copy.
KRaft metadata log lag Standby controllers and brokers behind on metadata.
ZooKeeper session expiry rate (ZK mode) Session expiry ejects a broker and shrinks ISRs.
Partition reassignment status Gate other alerts — reassignment causes expected URP.

Operating mistakes worth avoiding

The traps Kafka teams keep falling into. Each has a clear, well-known fix. Most teams only learn it after an incident.

Not monitoring log compaction health

The #1 silent killer. The log cleaner thread crashes silently on a corrupt record or OOM and does not restart. No alert fires. Compacted topics — <code>__consumer_offsets</code> first — grow without bound, and the disk fills days or weeks later. Monitor <code>max-dirty-percent</code> and the cleaner <code>DeadThreadCount</code>, not just disk space.

Running without min.insync.replicas

With <code>min.insync.replicas=1</code> (the default), <code>acks=all</code> only guarantees the leader received the message — the leader can acknowledge with zero followers in the ISR. For topics where durability matters, set <code>min.insync.replicas=2</code> with <code>replication.factor=3</code>, and understand that this is deliberately trading availability for durability.

Alerting on absolute lag instead of rate of change

'Alert if lag > 100,000' means one second of lag on a high-throughput topic and two days on a low-throughput one. Convert lag to time (<code>lag / produce_rate</code>) and alert on whether it's growing. Lag that outruns <code>retention.ms</code> becomes <code>OffsetOutOfRangeException</code> and permanent consumer-side data loss.

Watching total request latency, not the breakdown

A 500ms <code>TotalTimeMs</code> is not actionable. Is it <code>RequestQueueTimeMs</code> (I/O threads behind), <code>LocalTimeMs</code> (slow disk), <code>RemoteTimeMs</code> (slow followers), or response-side queueing? Collect and alert on the breakdown components — they point straight at the bottleneck.

Giving the broker a huge JVM heap

Kafka stores messages in the OS page cache, not the heap. A 32 GB heap doesn't help — it lengthens GC pauses and steals RAM from the page cache Kafka actually depends on. Keep the heap at 4–8 GB and leave the rest to the OS. A Full GC longer than <code>zookeeper.session.timeout.ms</code> ejects the broker and shrinks ISRs.

Setting ulimit -n at the default

Each log segment costs two file descriptors and each connection costs one. With hundreds of partitions and many clients, the default limit of 1024 is exhausted fast, and the broker fails with <code>Too many open files</code> — unable to open segments or accept connections. Set <code>ulimit -n</code> to at least 100,000.

Ignoring partition and leadership skew

Even partition counts can hide severely uneven leadership — one broker leading 40% of partitions handles disproportionate produce and fetch traffic and has the least headroom. Monitor <code>LeaderCount</code> per broker, alert on deviation from the mean, and remember that preferred-replica election may not auto-rebalance after restarts.

Treating Kafka as stateless during maintenance

Killing a broker means a cold page cache, replica state that must re-fetch from leaders, controller events for every partition it led, and every client reconnecting. A single restart can take 30+ minutes to fully recover. Plan rolling maintenance with per-broker recovery time in mind, and keep monitoring active — don't blanket-suppress alerts.

Kafka runbooks in this section

Each guide is a focused runbook for one symptom or topic. Pick one when you have an incident, or use the categories to learn the area.

▸

Start here

▸

Replication, ISR, and durability

▸

Leadership, offline partitions, and unclean elections

▸

Controller, ZooKeeper, and KRaft metadata

▸

Consumer lag and rebalances

▸

Request handlers, network threads, and saturation

▸

Request latency and the breakdown

▸

Disk, log directories, and storage

▸

Log compaction, retention, and segments

▸

Producers, clients, and connections

▸

Security, ACLs, and quotas

WHERE TO GO NEXT

Setting up Kafka monitoring, or putting out a fire?

If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, jump straight to the symptom that matches what you're seeing.

> Start with the checklist > Back to Operations Guides