Logstash Kafka input: consumer group lag and rebalances

The consumer group lag for your logstash group is climbing and not draining, or the Kafka input periodically drops to zero for seconds or minutes at a time while the brokers look fine. In both cases the evidence lives outside Logstash: consumer-group lag is a Kafka-side signal, and you will not find it in the Logstash monitoring API on port 9600.

Rising lag means one thing: Logstash is consuming slower than producers are writing. Frequent rebalances mean the group cannot hold a stable assignment, and every rebalance is a window where consumption stops entirely. The two often appear together, because the most common rebalance trigger (slow event processing) is also the most common lag trigger.

This guide covers reading the lag signal, telling a rebalance storm apart from a throughput problem, and separating “Logstash is the bottleneck” from “Kafka is the bottleneck” using the Logstash-side signals: queue growth, worker utilization, and backpressure.

What this means

When the Kafka input plugin runs, each Logstash instance joins a consumer group (default group id: logstash) and gets assigned a subset of the topic’s partitions. Lag is the per-partition difference between the log end offset and the consumer’s current committed offset. If production rate exceeds Logstash’s end-to-end delivery rate, lag grows monotonically.

Two distinct failure modes produce the symptoms:

• Sustained lag growth without rebalances. The consumer is healthy but too slow. Work is flowing, just not fast enough. The bottleneck is somewhere in the Logstash pipeline (filters, outputs, workers) or the consumer is under-provisioned (threads, instances).
• Rebalance storms. Consumption repeatedly halts while the group reassigns partitions. The classic trigger is a batch taking longer than max_poll_interval_ms (plugin default 300000 ms, 5 minutes) to process, after which the broker evicts the member and reassigns its partitions. Missed heartbeats relative to session_timeout_ms (plugin default 10000 ms) do the same thing faster. Deploys, restarts, and crash loops also force rebalances.

During a rebalance, input throughput for that group drops to zero. If rebalances happen every few minutes, effective throughput collapses even though Logstash looks alive and the queue looks reasonable.

A note on architecture: when Logstash consumes from Kafka, the Kafka topic itself is the durable backlog. Logstash’s internal queue sits downstream of the consumer. This is why lag, not queue depth, is the truest “is Logstash keeping up” signal for Kafka-sourced pipelines. See How Logstash actually works in production for the full pipeline model.

Common causes

Quick checks

Run these before changing anything. All are read-only.

# 1. Read the lag directly from Kafka (this is the primary signal)
kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group logstash
# Watch the LAG column per partition. Also note which members hold assignments.

# 2. Check for rebalance and commit-failure evidence in the Logstash log
grep -Ei '(rebalance|rebalanced|revoking|commit cannot be completed|max.poll.interval)' \
  /var/log/logstash/logstash-plain.log | tail -n 50

# 3. Check Logstash-side flow: is the pipeline actually saturated?
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty
# Look at flow.input_throughput, flow.output_throughput,
# flow.queue_backpressure, flow.worker_utilization, and queue.events_count

# 4. Rule out GC as the rebalance trigger
curl -sS http://127.0.0.1:9600/_node/stats/jvm?pretty
# Old-gen collection_time_in_millis rising fast relative to wall time means GC stalls

# 5. If workers look busy, find out what they are busy with
curl -sS 'http://127.0.0.1:9600/_node/hot_threads?pretty'

Two things to note on step 1. First, lag per partition matters more than the total: one partition with huge lag and the rest near zero points at a skewed key or a single struggling member, not a fleet-wide capacity problem. Second, run --describe twice a minute apart. Lag that oscillates but returns to baseline between bursts is normal buffering; lag that ratchets upward is a real deficit.

How to diagnose it

Work through these in order. The goal is to land on one of three conclusions: Logstash compute bottleneck, downstream/output bottleneck, or Kafka-side problem.

1. Characterize the lag shape. Steady monotonic rise means a persistent throughput deficit. Sawtooth that never returns to zero means borderline capacity. Flat for minutes then a jump means rebalances are halting consumption. This shape decides which branch you take next.

2. Count the rebalances. If the log shows repeated partition revocation and reassignment, find the trigger. “Commit cannot be completed since the group has already rebalanced” points at batches exceeding max_poll_interval_ms. Rebalances aligned with instance starts and stops point at membership churn, not processing speed.

3. Correlate with the Logstash queue. If lag is rising and the Logstash queue (queue.events_count, queue.queue_size_in_bytes) is also growing, the pipeline itself is the constraint. If lag is rising but the queue is flat and worker utilization is low, the pipeline has spare capacity and the constraint is the consumer (too few threads, rebalance downtime) or Kafka itself.

4. Split compute-bound from output-bound. High worker utilization with high CPU points at expensive filters. High worker utilization with low CPU points at workers blocked on output I/O. This distinction decides the fix: more CPU helps the first and makes the second worse. Logstash queue full: inputs blocked and the backpressure wedge covers the output-bound case in depth.

5. Rule out the JVM. Old-gen GC pauses freeze the poll loop and the heartbeat thread. If GC time is a significant fraction of wall time, fix heap pressure before touching Kafka settings; tuning max_poll_interval_ms to survive GC pauses treats the symptom.

6. Only then look at Kafka. If Logstash has spare capacity everywhere, verify broker health, and check whether producer rate on the topic simply exceeded what the current consumer group can ever drain.

flowchart TD
  A[Kafka lag rising] --> B{Rebalances in Logstash log?}
  B -->|frequent| C[Poll interval, session timeout, or instance churn]
  B -->|rare| D{Logstash queue growing?}
  D -->|yes| E{CPU high or low?}
  E -->|CPU high| F[Compute-bound filters]
  E -->|CPU low| G[Downstream output blocking]
  D -->|flat, workers idle| H[Consumer under-provisioned or Kafka-side issue]

Metrics and signals to monitor

Fixes

Match the fix to the diagnosis. Do not start by restarting Logstash; a restart forces a rebalance and makes the lag picture worse before it gets better.

Batches exceed max_poll_interval_ms

The consumer must finish processing each polled batch and return to poll() within max_poll_interval_ms (default 300000). Two knobs, in order of preference:

• Reduce max_poll_records (default 500) so each batch is small enough to process within the interval. This is usually the safer change because it also smooths heap pressure.
• Increase max_poll_interval_ms if batches legitimately take longer (heavy enrichment, slow downstream). This delays detection of genuinely dead consumers, so raise it deliberately, not as a reflex.

If the real reason batches are slow is a blocked output, neither knob fixes anything; fix the downstream first.

Downstream output blocking

This is the backpressure cascade: output slow, workers block, queue fills, poll loop slows, lag grows, then rebalances start. Resolve the destination problem (Elasticsearch cluster health, bulk rejections, network) before touching the consumer. If the persistent queue is absorbing the backlog, estimate runway before it fills; see Logstash persistent queue runway.

Compute-bound filters

Profile with hot threads and per-plugin flow.worker_utilization to find the expensive filter, then simplify it or add workers if CPU headroom exists. If all cores are saturated, the fix is more CPU or more instances, not more consumer threads.

Rebalance churn from restarts and deploys

• Use static group membership (group_instance_id on the input) so a restarting instance rejoins as the same member and avoids triggering a full rebalance. This is especially useful in containerized environments with frequent restarts.
• Stagger deploys so only one group member leaves at a time.
• Fix crash loops before anything else; each crash-and-rejoin is two rebalances plus lost processing time.
• On Kafka 4.0 and newer, the plugin documents a group_protocol option that can be set to consumer for incremental, cooperative rebalancing instead of the stop-the-world classic protocol. Verify your bundled plugin version supports it before relying on it (group_protocol was added in logstash-integration-kafka 12.0.0, which upgrades the Kafka client to 4.1.0; the 8.x line through 8.19 ships 11.x without it — check with bin/logstash-plugin list --verbose logstash-integration-kafka).

Consumer under-provisioned

Each partition can be consumed by exactly one consumer thread in the group. If the topic has 12 partitions and you run 3 Logstash instances with consumer_threads => 1, at most 3 partitions are ever being consumed at once and the rest sit idle. Scale consumer threads or instances toward a 1:1 ratio with partition count. Adding consumers beyond the partition count does nothing for lag.

Prevention

• Alert on lag trend, not lag magnitude. A rolling-rate alert on per-partition lag catches deficits hours before they become user-visible. Absolute thresholds break when traffic grows.
• Alert on rebalance rate. Rebalances per hour is a leading indicator of poll-interval pressure and instance churn. Zero is normal during steady state; recurring bursts are not.
• Size consumers to partitions at topic creation time. Changing partition counts later is painful; under-partitioned topics cap your consumer parallelism forever.
• Keep GC healthy. The poll loop and heartbeats cannot survive multi-second old-gen pauses. Heap headroom is a Kafka-consumer availability feature, not just a JVM concern.
• Include Kafka lag in your pipeline checklist. The full signal set is in Logstash monitoring checklist; Kafka lag is the input-side complement to the Logstash-internal signals there.

How Netdata helps

• Netdata charts Kafka consumer-group lag alongside the Logstash node’s flow.input_throughput, so you can see lag rising at the same moment input throughput sags, without switching tools.
• flow.worker_utilization and flow.queue_backpressure per pipeline let you apply the diagnostic split from this article directly: high utilization plus rising lag is a pipeline bottleneck; flat utilization plus rising lag is a consumer or Kafka-side problem.
• JVM metrics (heap, old-gen GC time) on the same dashboard make it obvious when rebalance bursts line up with GC pauses rather than deploy events.
• Queue metrics (events_count, PQ occupancy and growth) show whether the internal backlog is growing in step with Kafka lag, which confirms the constraint is inside Logstash.
• Per-second collection catches the short consumption outages that individual rebalances cause, which per-minute scraping typically averages away.

Related guides

• Logstash flow.queue_backpressure: the input-throttling metric explained
• How Logstash actually works in production: a mental model for operators
• Logstash monitoring checklist: the signals every production pipeline needs
• Logstash queue events count growing: reading the in-flight backlog
• Logstash queue full: inputs blocked and the backpressure wedge
• Logstash pipeline stalled: output rate at zero while the process looks alive
• Logstash persistent queue runway: how long until the PQ fills