The symptom usually arrives from upstream first: Filebeat stops shipping, Kafka consumer lag climbs, or an HTTP input starts refusing connections. Logstash itself looks alive. The API answers on port 9600, the process is running, CPU is often unremarkable. But events are not moving, because the internal queue between inputs and workers is full, and every input thread is blocked trying to push into it.

This is the backpressure wedge: outputs slow or fail, workers cannot drain the queue, the queue fills, inputs block, and upstream systems back up or drop events. The queue is where the pain shows up, but it is almost never where the problem lives.

The most important thing to know before touching anything: the fix is downstream, not the queue. Restarting Logstash, enlarging the queue, or adding workers treats the symptom and often makes the underlying failure worse.

What this means

Logstash is a queue-backed batch processor. Each input thread decodes events and pushes them into a bounded queue. Worker threads pull batches (default pipeline.batch.size = 125, pipeline.batch.delay = 50ms), run them through the filter chain sequentially, and push them to outputs. A worker does not fetch its next batch until the current one is acknowledged by the output.

That acknowledgment dependency is the wedge mechanism. One slow output blocks a worker. Enough blocked workers stop queue consumption. A stopped queue fills. A full queue blocks inputs.

flowchart LR
  U[Upstream: Beats / Kafka / TCP] --> I[Input threads]
  I -->|push events| Q[Bounded queue]
  Q -->|pull batches| W[Worker threads]
  W --> F[Filter chain]
  F --> O[Output plugins]
  O --> D[Destination: ES / Kafka / HTTP]
  D -.->|slow or failing: workers block| W
  W -.->|no drain: queue fills| Q
  Q -.->|full: inputs block| I
  I -.->|upstream backs up or drops| U

Queue type determines how fast you hit the wall, not whether you hit it:

  • Memory queue (default): small bounded buffer, roughly pipeline.workers times pipeline.batch.size events. It fills in seconds under a stalled output and blocks inputs almost immediately. Zero durability: a crash loses whatever is queued.
  • Persistent queue (PQ): page-based on-disk queue (default 64MB pages, default 1GB max via queue.max_bytes). It absorbs a downstream outage for hours. Once it reaches max_bytes, inputs block identically. PQ buys time; it does not change the failure.

CPU is the key differential signal. In a pure backpressure wedge, workers wait on output I/O, so host CPU is usually low or moderate. If CPU is pegged, you are looking at a compute bottleneck (grok hell) that produces the same queue growth through a different mechanism. The diagnostic steps below separate these.

Common causes

CauseWhat it looks likeFirst thing to check
Slow or failing output (Elasticsearch, Kafka, HTTP endpoint)Queue grows, output throughput drops, retries/errors in logs, CPU low to moderateDestination health directly: curl localhost:9200/_cluster/health; output errors in logstash-plain.log
Elasticsearch bulk rejections (429)Retries rising, workers blocked, queue filling, ES-side thread pool saturationES logs and indexing stats; Logstash output bulk_requests.failures
Output auth/TLS failureOutput rate zero, handshake or 401/403 errors repeating in logs, queue growing`grep -Ei ‘(SSL
CPU-bound filters (grok backtracking, ruby, DNS)Queue grows with CPU pegged, worker utilization >90%, output healthyGET /_node/hot_threads; per-plugin flow.worker_utilization
GC death spiralQueue grows, throughput near zero, API sluggish, old-gen GC rising/_node/stats/jvm: GC time as fraction of wall clock
PQ reached max_bytes after long downstream outagePQ occupancy at 100%, inputs fully blocked, hours of apparent “health” before itqueue.queue_size_in_bytes / queue.max_queue_size_in_bytes
Too few workers for the workloadQueue grows during peaks, CPU and outputs healthy, drains between peaksflow.worker_utilization near 100% with healthy output duration

Quick checks

All read-only, safe to run during an incident.

# 1. Liveness and API responsiveness
curl -sS --connect-timeout 5 http://127.0.0.1:9600/

# 2. Pipeline flow metrics: the wedge in one view
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty
# Read per pipeline: flow.queue_backpressure, flow.worker_utilization,
# flow.input_throughput, flow.output_throughput, queue.events_count

# 3. Queue occupancy (PQ)
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | grep -E 'queue_size_in_bytes|max_queue_size_in_bytes|events_count'

# 4. What are workers actually doing right now
curl -sS 'http://127.0.0.1:9600/_node/hot_threads?threads=10&human=true'

# 5. Output errors and retries
grep -Ei '(retry|error|exception|failed|reject|unavailable|timeout|429|503)' /var/log/logstash/logstash-plain.log | tail -n 200

# 6. JVM pressure (is this GC, not backpressure?)
curl -sS http://127.0.0.1:9600/_node/stats/jvm?pretty

# 7. Process CPU (compute bottleneck vs I/O wait)
curl -sS http://127.0.0.1:9600/_node/stats/process?pretty

# 8. Disk headroom on the PQ volume
df -h /var/lib/logstash

For multi-pipeline deployments, query each pipeline individually (/_node/stats/pipelines/<pipeline_id>). Aggregate stats routinely mask one wedged pipeline behind four healthy ones.

How to diagnose it

  1. Confirm the wedge. In the pipeline stats, check flow.queue_backpressure. This is the fraction of time input threads spend blocked pushing into the queue. A sustained value significantly above the pipeline’s baseline (working bands: >0.2 notable, >0.5 significant, >0.8 severe) confirms inputs are being throttled. Cross-check with events.queue_push_duration_in_millis: it should be near zero in a healthy pipeline; any sustained non-trivial value is direct evidence of queue-side blocking.

  2. Establish direction of flow. Compare flow.input_throughput to flow.output_throughput. Input above output with a growing queue means the pipeline is falling behind. Output at zero with input positive is a complete stall. If both are dropping together, look upstream or at the inputs themselves rather than the queue.

  3. Split backpressure from compute. Look at process.cpu.percent and flow.worker_utilization together:

    • Queue full + low/moderate CPU + high worker utilization = workers blocked on output I/O. This is the classic wedge. Go to step 4.
    • Queue full + CPU pegged + high worker utilization = compute bottleneck. Skip to step 5.
    • Queue growing + erratic throughput + slow API = suspect GC. Check jvm.gc.collectors.old.collection_time_in_millis deltas against wall time; >20% GC overhead is severe, >50% is a death spiral.

  4. For output blockage: interrogate the downstream. Hot threads showing workers parked in output plugin code confirms the wedge. Check per-output events.duration_in_millis rising before throughput dropped; that ordering is the distinguishing feature of a downstream cause. Then verify the destination independently of Logstash: cluster health, bulk rejection rate, auth validity, network path. Output retries in the Logstash log are the tell. A partial bulk failure (HTTP 200 with per-document rejects) still counts as events out while silently losing data.

  5. For compute bottleneck: find the expensive stage. Hot threads pointing at grok, ruby, or DNS filter code, plus per-plugin plugins.filters[].flow.worker_utilization showing one filter dominating, identifies the culprit. A new log format triggering worst-case regex backtracking is the usual trigger.

  6. Estimate your runway before acting. For PQ: (max_queue_size_in_bytes - queue_size_in_bytes) / fill_rate. flow.queue_persisted_growth_bytes gives you the fill rate directly. Runway under 30 minutes with the downstream still impaired is a paging condition: inputs will block imminently. For the memory queue, runway is effectively zero; you are already in the wedge.

  7. Check the adjacent systems. If Logstash consumes from Kafka, consumer group lag is your backlog signal. If Beats ships to Logstash, Filebeat registry stalls are a symptom of Logstash-side backpressure, not a Beats problem. The cascade crosses layers; only correlating both sides makes it diagnosable.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
flow.queue_backpressureFraction of input-thread time lost to queue pressure; earliest direct wedge signalSustained rise above pipeline baseline
queue.events_count trendBacklog growth; for memory queue, any monotonic rise over 15 minutes is concerningSteady growth with output below input
queue.queue_size_in_bytes / max_queue_size_in_bytes (PQ)Occupancy and runway before inputs block>80% sustained; page at >90% with positive growth and runway <30 min
flow.queue_persisted_growth_bytes (PQ)Direct fill rate; positive means filling, negative means drainingPositive over a smoothed 5-15 min window
flow.output_throughput vs flow.input_throughputWhether the pipeline keeps up; zero output with active input is a living-dead pipelineOutput >50% below rolling baseline with input non-zero
flow.worker_utilization (pipeline and per-plugin)Separates compute saturation from I/O blocking>90% sustained; one plugin dominating >80% of processing time
events.queue_push_duration_in_millisInput-side wait to enter the queue; fires before the queue visibly fillsSustained non-trivial average per event
Output errors/retries in logsThe downstream tell behind most wedgesAny sustained non-zero retry/error pattern
jvm.gc.collectors.old.*Rules the GC death spiral in or outGC overhead >10% of wall time; rising old-gen count
dead_letter_queue.queue_size_in_bytesEvents permanently diverted; the DLQ does not relieve backpressureAny unexpected growth

Two false positives to gate out: cold start (queue builds while the JVM warms up; gate alerts on jvm.uptime_in_millis > 300s) and bursty input (short queue spikes that drain between bursts are healthy buffering).

Fixes

Downstream output is slow or failing

Fix the destination. Restore Elasticsearch cluster health, clear the bulk rejection backlog, repair credentials or certificates, resolve the network fault. Logstash will drain the queue and self-heal once the output recovers. PQ drain can take much longer than the fill did, and throughput during drain is reduced; that is expected, not a second incident.

If recovery will take longer than your queue runway, shed load early: pause non-critical inputs, route lower-value pipelines to a standby sink, or throttle at the source. Doing this at 40% PQ occupancy is a calm operational decision. Doing it at 95% is a scramble.

Tradeoff to understand: enlarging queue.max_bytes buys more buffer for future outages but extends recovery time and disk exposure. It does nothing for the current wedge. PQ disk usage can also exceed queue.max_bytes noticeably because pages are allocated in fixed-size chunks (default 64MB) and freed only when fully drained and checkpointed. Size max_bytes relative to the filesystem, and keep the PQ partition under 70% full at peak queue utilization.

Elasticsearch bulk rejections

Rejections mean the destination is saturated, and Logstash retries amplify the load. Increase ES-side capacity or reduce indexing pressure (shards, refresh interval, ingest concurrency). Reducing pipeline.batch.size shrinks each bulk request at the cost of more requests; this can smooth a rejecting cluster but is a tuning bandage, not a fix for an undersized cluster.

CPU-bound filters

Simplify the expensive filter: replace catastrophic-backtracking grok patterns, use dissect for structured logs, cache or remove DNS lookups, move heavy enrichment out of the hot path. Increasing pipeline.workers helps only if CPU headroom exists; on a fully pinned host it adds context switching, not capacity. Long term, the fix is CPU or filter efficiency, since the output side was never the problem.

GC death spiral

This is the one case where an immediate restart is the correct action: the JVM will not recover on its own. If PQ is enabled, queued events survive. With the memory queue, in-flight events are lost; weigh that against the fact that the pipeline is delivering nothing anyway. After restart, raise the heap (-Xms/-Xmx equal; the 1GB historical default is too small for production) and investigate why live objects grew: large events, field explosions, an oversized batch footprint (batch size times worker count), or a filter leak.

Chronically undersized workers

If the queue grows during peaks and drains between them with healthy outputs and available CPU, raise pipeline.workers. Watch for the queue returning to baseline between peaks; if it never does, runway is already negative and you need throughput, not buffer.

Version-specific gotcha

Logstash 9.2.0 fails to start if queue.max_bytes is set to 2147483648 bytes (2 GB) or greater. If you raise PQ capacity while planning around a wedge, do not land on that combination.

Prevention

  • Alert on runway, not occupancy. Compute (max_queue_size - queue_size) / growth_rate continuously and page when runway drops under 30 minutes with the downstream impaired. Occupancy-only alerts fire too late on a fast fill.
  • Watch flow.queue_backpressure as a leading indicator. It rises before the queue visibly fills and before upstream notices anything.
  • Correlate both sides of the boundary. Monitor Elasticsearch bulk rejections or Kafka broker health alongside Logstash output duration. The cascade starts at the destination; that is where the earliest signal lives.
  • Monitor per pipeline, not aggregate. Query each pipeline’s stats individually in multi-pipeline deployments.
  • Size PQ honestly. Enough capacity to absorb the longest realistic downstream outage you intend to survive, on a partition with the disk to back it. Then monitor flow.queue_persisted_growth_bytes so the buffer is never silently consumed.
  • Keep the DLQ in perspective. It captures permanently failing events; it is not a backpressure relief valve and does not protect a full queue. It is also disabled by default, so permanent output failures without it are logged and lost.
  • Baseline-relative thresholds. Absolute events-per-second thresholds break the first time the workload changes. Alert on deviation from rolling baselines for input and output throughput.

How Netdata helps

  • Queue occupancy and growth trends: Netdata tracks queue.events_count and PQ queue_size_in_bytes per pipeline over time, so a steadily filling queue is visible hours before inputs block, not after upstream pages you.
  • Backpressure and worker flow metrics: flow.queue_backpressure and flow.worker_utilization charted alongside throughput make the wedge’s signature (input throttled, workers occupied, output falling) readable in one view.
  • CPU correlation for cause separation: process CPU next to queue depth and worker utilization is exactly the split that distinguishes output blockage (low CPU) from grok hell (pegged CPU) without an SSH session.
  • JVM heap and GC overlays: old-gen collection time against output throughput rules the GC death spiral in or out in seconds instead of minutes of log archaeology.
  • Downstream co-visibility: with Elasticsearch or Kafka monitored on the same dashboard, the retry-then-fill causal chain is a single glance rather than a cross-system investigation.