$ guides / logstash / logstash-queue-full-inputs-blocked ▌

Operations Guides

Logstash queue full: inputs blocked and the backpressure wedge

The symptom usually arrives from upstream first: Filebeat stops shipping, Kafka consumer lag climbs, or an HTTP input starts refusing connections. Logstash itself looks alive. The API answers on port 9600, the process is running, CPU is often unremarkable. But events are not moving, because the internal queue between inputs and workers is full, and every input thread is blocked trying to push into it.

This is the backpressure wedge: outputs slow or fail, workers cannot drain the queue, the queue fills, inputs block, and upstream systems back up or drop events. The queue is where the pain shows up, but it is almost never where the problem lives.

The most important thing to know before touching anything: the fix is downstream, not the queue. Restarting Logstash, enlarging the queue, or adding workers treats the symptom and often makes the underlying failure worse.

What this means

Logstash is a queue-backed batch processor. Each input thread decodes events and pushes them into a bounded queue. Worker threads pull batches (default pipeline.batch.size = 125, pipeline.batch.delay = 50ms), run them through the filter chain sequentially, and push them to outputs. A worker does not fetch its next batch until the current one is acknowledged by the output.

That acknowledgment dependency is the wedge mechanism. One slow output blocks a worker. Enough blocked workers stop queue consumption. A stopped queue fills. A full queue blocks inputs.

flowchart LR
  U[Upstream: Beats / Kafka / TCP] --> I[Input threads]
  I -->|push events| Q[Bounded queue]
  Q -->|pull batches| W[Worker threads]
  W --> F[Filter chain]
  F --> O[Output plugins]
  O --> D[Destination: ES / Kafka / HTTP]
  D -.->|slow or failing: workers block| W
  W -.->|no drain: queue fills| Q
  Q -.->|full: inputs block| I
  I -.->|upstream backs up or drops| U

Queue type determines how fast you hit the wall, not whether you hit it:

Memory queue (default): small bounded buffer, roughly pipeline.workers times pipeline.batch.size events. It fills in seconds under a stalled output and blocks inputs almost immediately. Zero durability: a crash loses whatever is queued.
Persistent queue (PQ): page-based on-disk queue (default 64MB pages, default 1GB max via queue.max_bytes). It absorbs a downstream outage for hours. Once it reaches max_bytes, inputs block identically. PQ buys time; it does not change the failure.

CPU is the key differential signal. In a pure backpressure wedge, workers wait on output I/O, so host CPU is usually low or moderate. If CPU is pegged, you are looking at a compute bottleneck (grok hell) that produces the same queue growth through a different mechanism. The diagnostic steps below separate these.

Common causes

Cause	What it looks like	First thing to check
Slow or failing output (Elasticsearch, Kafka, HTTP endpoint)	Queue grows, output throughput drops, retries/errors in logs, CPU low to moderate	Destination health directly: `curl localhost:9200/_cluster/health`; output errors in `logstash-plain.log`
Elasticsearch bulk rejections (429)	Retries rising, workers blocked, queue filling, ES-side thread pool saturation	ES logs and indexing stats; Logstash output `bulk_requests.failures`
Output auth/TLS failure	Output rate zero, handshake or 401/403 errors repeating in logs, queue growing	`grep -Ei ‘(SSL
CPU-bound filters (grok backtracking, ruby, DNS)	Queue grows with CPU pegged, worker utilization >90%, output healthy	`GET /_node/hot_threads`; per-plugin `flow.worker_utilization`
GC death spiral	Queue grows, throughput near zero, API sluggish, old-gen GC rising	`/_node/stats/jvm`: GC time as fraction of wall clock
PQ reached `max_bytes` after long downstream outage	PQ occupancy at 100%, inputs fully blocked, hours of apparent “health” before it	`queue.queue_size_in_bytes / queue.max_queue_size_in_bytes`
Too few workers for the workload	Queue grows during peaks, CPU and outputs healthy, drains between peaks	`flow.worker_utilization` near 100% with healthy output duration

Quick checks

All read-only, safe to run during an incident.

# 1. Liveness and API responsiveness
curl -sS --connect-timeout 5 http://127.0.0.1:9600/

# 2. Pipeline flow metrics: the wedge in one view
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty
# Read per pipeline: flow.queue_backpressure, flow.worker_utilization,
# flow.input_throughput, flow.output_throughput, queue.events_count

# 3. Queue occupancy (PQ)
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | grep -E 'queue_size_in_bytes|max_queue_size_in_bytes|events_count'

# 4. What are workers actually doing right now
curl -sS 'http://127.0.0.1:9600/_node/hot_threads?threads=10&human=true'

# 5. Output errors and retries
grep -Ei '(retry|error|exception|failed|reject|unavailable|timeout|429|503)' /var/log/logstash/logstash-plain.log | tail -n 200

# 6. JVM pressure (is this GC, not backpressure?)
curl -sS http://127.0.0.1:9600/_node/stats/jvm?pretty

# 7. Process CPU (compute bottleneck vs I/O wait)
curl -sS http://127.0.0.1:9600/_node/stats/process?pretty

# 8. Disk headroom on the PQ volume
df -h /var/lib/logstash

For multi-pipeline deployments, query each pipeline individually (/_node/stats/pipelines/<pipeline_id>). Aggregate stats routinely mask one wedged pipeline behind four healthy ones.

How to diagnose it

Confirm the wedge. In the pipeline stats, check flow.queue_backpressure. This is the fraction of time input threads spend blocked pushing into the queue. A sustained value significantly above the pipeline’s baseline (working bands: >0.2 notable, >0.5 significant, >0.8 severe) confirms inputs are being throttled. Cross-check with events.queue_push_duration_in_millis: it should be near zero in a healthy pipeline; any sustained non-trivial value is direct evidence of queue-side blocking.
Establish direction of flow. Compare flow.input_throughput to flow.output_throughput. Input above output with a growing queue means the pipeline is falling behind. Output at zero with input positive is a complete stall. If both are dropping together, look upstream or at the inputs themselves rather than the queue.
Split backpressure from compute. Look at process.cpu.percent and flow.worker_utilization together:
- Queue full + low/moderate CPU + high worker utilization = workers blocked on output I/O. This is the classic wedge. Go to step 4.
- Queue full + CPU pegged + high worker utilization = compute bottleneck. Skip to step 5.
- Queue growing + erratic throughput + slow API = suspect GC. Check jvm.gc.collectors.old.collection_time_in_millis deltas against wall time; >20% GC overhead is severe, >50% is a death spiral.
For output blockage: interrogate the downstream. Hot threads showing workers parked in output plugin code confirms the wedge. Check per-output events.duration_in_millis rising before throughput dropped; that ordering is the distinguishing feature of a downstream cause. Then verify the destination independently of Logstash: cluster health, bulk rejection rate, auth validity, network path. Output retries in the Logstash log are the tell. A partial bulk failure (HTTP 200 with per-document rejects) still counts as events out while silently losing data.
For compute bottleneck: find the expensive stage. Hot threads pointing at grok, ruby, or DNS filter code, plus per-plugin plugins.filters[].flow.worker_utilization showing one filter dominating, identifies the culprit. A new log format triggering worst-case regex backtracking is the usual trigger.
Estimate your runway before acting. For PQ: (max_queue_size_in_bytes - queue_size_in_bytes) / fill_rate. flow.queue_persisted_growth_bytes gives you the fill rate directly. Runway under 30 minutes with the downstream still impaired is a paging condition: inputs will block imminently. For the memory queue, runway is effectively zero; you are already in the wedge.
Check the adjacent systems. If Logstash consumes from Kafka, consumer group lag is your backlog signal. If Beats ships to Logstash, Filebeat registry stalls are a symptom of Logstash-side backpressure, not a Beats problem. The cascade crosses layers; only correlating both sides makes it diagnosable.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`flow.queue_backpressure`	Fraction of input-thread time lost to queue pressure; earliest direct wedge signal	Sustained rise above pipeline baseline
`queue.events_count` trend	Backlog growth; for memory queue, any monotonic rise over 15 minutes is concerning	Steady growth with output below input
`queue.queue_size_in_bytes / max_queue_size_in_bytes` (PQ)	Occupancy and runway before inputs block	>80% sustained; page at >90% with positive growth and runway <30 min
`flow.queue_persisted_growth_bytes` (PQ)	Direct fill rate; positive means filling, negative means draining	Positive over a smoothed 5-15 min window
`flow.output_throughput` vs `flow.input_throughput`	Whether the pipeline keeps up; zero output with active input is a living-dead pipeline	Output >50% below rolling baseline with input non-zero
`flow.worker_utilization` (pipeline and per-plugin)	Separates compute saturation from I/O blocking	>90% sustained; one plugin dominating >80% of processing time
`events.queue_push_duration_in_millis`	Input-side wait to enter the queue; fires before the queue visibly fills	Sustained non-trivial average per event
Output errors/retries in logs	The downstream tell behind most wedges	Any sustained non-zero retry/error pattern
`jvm.gc.collectors.old.*`	Rules the GC death spiral in or out	GC overhead >10% of wall time; rising old-gen count
`dead_letter_queue.queue_size_in_bytes`	Events permanently diverted; the DLQ does not relieve backpressure	Any unexpected growth

Two false positives to gate out: cold start (queue builds while the JVM warms up; gate alerts on jvm.uptime_in_millis > 300s) and bursty input (short queue spikes that drain between bursts are healthy buffering).

Fixes

Downstream output is slow or failing

Fix the destination. Restore Elasticsearch cluster health, clear the bulk rejection backlog, repair credentials or certificates, resolve the network fault. Logstash will drain the queue and self-heal once the output recovers. PQ drain can take much longer than the fill did, and throughput during drain is reduced; that is expected, not a second incident.

If recovery will take longer than your queue runway, shed load early: pause non-critical inputs, route lower-value pipelines to a standby sink, or throttle at the source. Doing this at 40% PQ occupancy is a calm operational decision. Doing it at 95% is a scramble.

Tradeoff to understand: enlarging queue.max_bytes buys more buffer for future outages but extends recovery time and disk exposure. It does nothing for the current wedge. PQ disk usage can also exceed queue.max_bytes noticeably because pages are allocated in fixed-size chunks (default 64MB) and freed only when fully drained and checkpointed. Size max_bytes relative to the filesystem, and keep the PQ partition under 70% full at peak queue utilization.

Elasticsearch bulk rejections

Rejections mean the destination is saturated, and Logstash retries amplify the load. Increase ES-side capacity or reduce indexing pressure (shards, refresh interval, ingest concurrency). Reducing pipeline.batch.size shrinks each bulk request at the cost of more requests; this can smooth a rejecting cluster but is a tuning bandage, not a fix for an undersized cluster.

CPU-bound filters

Simplify the expensive filter: replace catastrophic-backtracking grok patterns, use dissect for structured logs, cache or remove DNS lookups, move heavy enrichment out of the hot path. Increasing pipeline.workers helps only if CPU headroom exists; on a fully pinned host it adds context switching, not capacity. Long term, the fix is CPU or filter efficiency, since the output side was never the problem.

GC death spiral

This is the one case where an immediate restart is the correct action: the JVM will not recover on its own. If PQ is enabled, queued events survive. With the memory queue, in-flight events are lost; weigh that against the fact that the pipeline is delivering nothing anyway. After restart, raise the heap (-Xms/-Xmx equal; the 1GB historical default is too small for production) and investigate why live objects grew: large events, field explosions, an oversized batch footprint (batch size times worker count), or a filter leak.

Chronically undersized workers

If the queue grows during peaks and drains between them with healthy outputs and available CPU, raise pipeline.workers. Watch for the queue returning to baseline between peaks; if it never does, runway is already negative and you need throughput, not buffer.

Version-specific gotcha

Logstash 9.2.0 fails to start if queue.max_bytes is set to 2147483648 bytes (2 GB) or greater. No 9.2.x patch fixed this; the fix shipped in Logstash 9.3.0 (elastic/logstash#18366). If you raise PQ capacity while planning around a wedge, do not land on that combination.

Prevention

Alert on runway, not occupancy. Compute (max_queue_size - queue_size) / growth_rate continuously and page when runway drops under 30 minutes with the downstream impaired. Occupancy-only alerts fire too late on a fast fill.
Watch flow.queue_backpressure as a leading indicator. It rises before the queue visibly fills and before upstream notices anything.
Correlate both sides of the boundary. Monitor Elasticsearch bulk rejections or Kafka broker health alongside Logstash output duration. The cascade starts at the destination; that is where the earliest signal lives.
Monitor per pipeline, not aggregate. Query each pipeline’s stats individually in multi-pipeline deployments.
Size PQ honestly. Enough capacity to absorb the longest realistic downstream outage you intend to survive, on a partition with the disk to back it. Then monitor flow.queue_persisted_growth_bytes so the buffer is never silently consumed.
Keep the DLQ in perspective. It captures permanently failing events; it is not a backpressure relief valve and does not protect a full queue. It is also disabled by default, so permanent output failures without it are logged and lost.
Baseline-relative thresholds. Absolute events-per-second thresholds break the first time the workload changes. Alert on deviation from rolling baselines for input and output throughput.

How Netdata helps

Queue occupancy and growth trends: Netdata tracks queue.events_count and PQ queue_size_in_bytes per pipeline over time, so a steadily filling queue is visible hours before inputs block, not after upstream pages you.
Backpressure and worker flow metrics: flow.queue_backpressure and flow.worker_utilization charted alongside throughput make the wedge’s signature (input throttled, workers occupied, output falling) readable in one view.
CPU correlation for cause separation: process CPU next to queue depth and worker utilization is exactly the split that distinguishes output blockage (low CPU) from grok hell (pegged CPU) without an SSH session.
JVM heap and GC overlays: old-gen collection time against output throughput rules the GC death spiral in or out in seconds instead of minutes of log archaeology.
Downstream co-visibility: with Elasticsearch or Kafka monitored on the same dashboard, the retry-then-fill causal chain is a single glance rather than a cross-system investigation.