Kafka Consumer Lag on Netdata

Apache Kafka Consumer Lag: Troubleshooting & Fixes

Wed, 23 Jul 2025 00:00:00 +0000

You’ve built a powerful, real-time data pipeline with Apache Kafka, but suddenly, things grind to a halt. The dreaded consumer lag is exploding, alerts are firing, and your downstream applications are starved for data. This scenario is all too common for teams running Kafka at scale. Often, the culprit is a subtle interplay between consumer group rebalancing, partition assignment, and suboptimal consumer configurations. Understanding these mechanics is not just about fixing a problem; it’s about building resilient, high-throughput streaming systems from the ground up.

Kafka CommitFailedException: rebalanced-out consumers and poll loop timeouts

Mon, 01 Jan 0001 00:00:00 +0000

CommitFailedException with the message that the group has already rebalanced and assigned the partitions to another member means the time between poll() calls exceeded max.poll.interval.ms. The coordinator evicted the consumer and rejected the in-flight offset commit.

When one consumer is evicted, the group rebalances. If other consumers are also slow, or if the rebalance itself takes long enough that healthy consumers miss the same deadline, the group enters a rebalance storm: it oscillates between JoinGroup and SyncGroup without stabilizing, and lag grows without bound.

Kafka consumer group lag growing: detection, lag-as-time, and root causes

Mon, 01 Jan 0001 00:00:00 +0000

When a consumer group falls behind, lag climbs monotonically. If the committed offset crosses the retention boundary, consumers hit OffsetOutOfRangeException and must reset to earliest or latest, reprocessing or skipping data. Restarting the consumer is a common first reaction, but broker-side fetch latency, page cache eviction, and network saturation are equally common culprits. Detect lag accurately, convert it to time, and trace the root cause without guessing.

What this means

Consumer lag is log-end-offset minus the last committed offset for each partition in a group. Monotonic growth means the consumer is not keeping up with the production rate. The immediate risk is committed offset falling behind log retention, forcing a reset to earliest or latest.

Kafka consumer group rebalancing too often: heartbeats, session timeout, and assignors

Mon, 01 Jan 0001 00:00:00 +0000

Consumer group lag is growing, application logs are full of JoinGroup and SyncGroup messages, and kafka-consumer-groups.sh shows the group flipping between Stable and PreparingRebalance. Healthy groups rebalance only during membership changes and planned deployments. More than two or three rebalances per hour for a stable group signals instability. The usual cause is a mismatch between processing latency and one of three timeouts: session.timeout.ms, heartbeat.interval.ms, or max.poll.interval.ms. The assignor strategy and static membership configuration determine how painful each rebalance is.

Kafka Consumer Lag Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Kafka Consumer Lag Monitoring

What Is Kafka Consumer Lag?

Kafka Consumer Lag represents the difference between the latest offset of a partition and the offset of the consumer group. It is a crucial metric within Kafka’s architecture as it indicates the latency for messages consumed by consumers in a Kafka topic. Properly monitoring Kafka Consumer Lag ensures efficient message queue management and prevents bottlenecks in your streaming applications.

Monitoring Kafka Consumer Lag With Netdata

To monitor Kafka Consumer Lag, Netdata uses an openmetrics (Prometheus) exporter. Netdata’s $name monitoring tool allows you to ingest data from any Prometheus exporter seamlessly, offering automatic dashboards, alerts, and comprehensive insights without requiring a Prometheus server or Grafana. By leveraging Netdata’s dashboards, DevOps and IT teams can effectively track Kafka performance, ensuring real-time data processing stays smooth and efficient.

Kafka consumer rebalance storm: stuck in PreparingRebalance and max.poll.interval.ms

Mon, 01 Jan 0001 00:00:00 +0000

Consumer group lag climbs while brokers report zero under-replicated partitions, normal produce and fetch latency, and normal request handler idle percent. The consumer group oscillates between Stable, PreparingRebalance, and CompletingRebalance without settling long enough to make progress. Every rebalance cycle pauses consumption; lag grows because time spent rebalancing dwarfs time spent fetching. This is a consumer rebalance storm. It is almost always a client-side timeout or processing issue. Look for CommitFailedException or max.poll.interval.ms exceeded in consumer logs.

Kafka monitoring checklist: the signals every production cluster needs

Mon, 01 Jan 0001 00:00:00 +0000

Kafka failures follow predictable paths: an ISR shrinks, a controller queue backs up, a disk fills while the cleaner thread hangs, or a consumer rebalance storm hides behind healthy broker metrics. You need to know which signals matter and when they justify a 3 AM page.

This checklist organizes broker-side signals into four levels. Each builds on the last: Level 1 prevents data loss. Level 2 prevents surprises. Level 3 exposes leading indicators. Level 4 catches silent killers. Use it to audit dashboards, tune alert severity, and justify instrumentation.

Kafka OffsetOutOfRangeException: when retention deletes data before the consumer reads it

Mon, 01 Jan 0001 00:00:00 +0000

OffsetOutOfRangeException means the consumer requested an offset the broker no longer holds. The log segment containing the consumer’s committed position was deleted by retention before the consumer caught up. This is not a transient fetch error; it is data loss, and the outcome depends entirely on auto.offset.reset. Many clients default to latest, which turns this exception into silent skipping.

This is a lag problem wearing a fetch error. The consumer was too slow, paused too long, or was offline longer than the topic’s retention.ms. Once the log start offset moves past the committed offset, every subsequent fetch fails.