How Logstash actually works in production: a mental model for operators

Most Logstash incidents are diagnosed badly because the operator is reasoning about the wrong system. They see a “log shipper” and check whether the process is up. The process is up. The API returns 200. Nothing is being delivered. Or they see high CPU and assume the host is undersized, when the real problem is one grok pattern with catastrophic backtracking. Or they watch queue depth stay flat for hours and conclude all is well, while the persistent queue quietly absorbs a downstream outage that will page someone at 4 a.m. when it fills.

Logstash is not a log shipper. It is a JVM-based, queue-backed, multi-threaded batch processor with a plugin architecture. Every characteristic failure falls out of that description once you understand the data path and where backpressure flows. This article builds that mental model: the internal machinery, the resources it competes for, the five failure archetypes that cover nearly everything you will see in production, and the signals that map to each part of the machine.

If you already run Logstash and want the monitoring model first, read this before touching thresholds or runbooks.

What it is and why it matters

A Logstash process hosts one or more pipelines. Each pipeline is an independent event-processing assembly line: input plugins pull events from the outside world, a codec turns raw bytes into Event objects, a central queue buffers those events, and a fixed pool of worker threads pulls batches off the queue and runs them through the filter chain and then the outputs.

Two properties of this design drive almost all operational behavior:

The worker pool is fixed and shared. The number of workers is set by pipeline.workers (default: CPU core count). Each worker processes one batch at a time, through every filter in order, and then blocks until the output acknowledges the batch. A worker stuck waiting on a slow output is not processing anything else.
Backpressure propagates backwards through the whole path. Slow output blocks workers. Blocked workers stop draining the queue. A full queue blocks input threads. Blocked inputs stall upstream senders (Beats, Kafka consumers, TCP clients), which then buffer, lag, or drop depending on their own behavior. One slow Elasticsearch cluster can, through this chain, take down an entire telemetry path while every component reports itself “running.”

How it works: the data path

flowchart LR
  subgraph inputs["Input stage"]
    I1["Input plugin 1
(own thread)"]
    I2["Input plugin 2
(own thread)"]
  end
  C1["Codec
(line, json, multiline)"]
  Q["Queue
(memory or persistent)"]
  W["Worker pool
(pipeline.workers)"]
  F["Filter chain
(sequential, in-worker)"]
  C2["Codec"]
  O["Output plugins"]
  UP["Upstream senders
(Beats, Kafka, TCP)"]
  UP --> I1
  UP --> I2
  I1 --> C1
  I2 --> C1
  C1 -->|events| Q
  Q -->|batches of 125| W
  W --> F
  F --> C2
  C2 --> O
  O -.->|slow output blocks worker| W
  W -.->|workers stop draining| Q
  Q -.->|full queue blocks inputs| C1
  C1 -.->|senders buffer, lag, or drop| UP

Inputs. Each input plugin (Beats, Kafka, TCP, HTTP, file, S3, JDBC, syslog, and so on) runs in its own thread. Inputs push raw bytes through a codec (line, json, multiline) which deserializes them into Logstash Event objects. Note what this implies: codec work happens on the input thread, before the queue. A misbehaving multiline codec does not show up as filter cost. It shows up as reduced input throughput, and bytes dropped before they become events are invisible to events.in entirely, because the counter only sees events that made it through the codec into the queue.

Queue. The queue is the central buffer and the most architecturally important element. Two modes:

Memory queue (default): a bounded in-memory buffer. No durability. If the process dies, queued events are gone. It is small by design, so it fills fast and backpressures inputs almost immediately when workers fall behind. Failure here is abrupt: you get little warning and lose in-flight data on crash.
Persistent queue (PQ): page-based on-disk queue (default 64MB pages, default 1GB max via queue.max_bytes). Events are written to page files and checkpointed, so they survive restarts. The tradeoff is a whole new failure surface: disk I/O saturation, page corruption after unclean shutdown, checkpoint lag, and disk space exhaustion. The PQ also delays failure visibility. It can absorb a downstream outage for hours, looking perfectly healthy, right up until it hits queue.max_bytes and inputs block instantly.

Workers, filters, outputs. Workers pull batches from the queue (pipeline.batch.size, default 125 events; pipeline.batch.delay, default 50ms, caps how long a worker waits for a full batch). Each worker runs its batch through the entire filter chain sequentially: every filter plugin, in order, in the same thread. Then the same worker pushes the batch to the output plugins and waits for acknowledgment.

This is the single most important mechanical fact for diagnosing Logstash: filters and outputs execute in the same thread, and the output call blocks that thread. There is no separate output pool absorbing downstream slowness. A 2-second Elasticsearch bulk response does not just add 2 seconds of latency; it removes one worker from circulation for 2 seconds. Enough slow responses and every worker is parked on I/O, the queue stops draining, and the pipeline wedges with low CPU, because waiting threads do not burn cycles.

Dead letter queue (DLQ). Optional and disabled by default. Captures events that permanently fail specific output failure classes (coverage depends on the output plugin and failure mode; not all failures qualify). It has its own size limit (default 1GB) and a storage policy (drop_newer or drop_older) for when it fills. Two operational consequences: without a DLQ, events that fail permanently after retries are logged and lost; and with a DLQ, replay is a manual operation most teams never perform, so the DLQ becomes silent data loss with extra disk usage unless you monitor it.

Multiple pipelines. Since 6.0, one Logstash process can run many pipelines via pipelines.yml. Each has its own queue, workers, and plugin chain, but they share the JVM heap. This matters for monitoring: aggregate node stats average away localized failures. One dead pipeline out of five drops aggregate throughput by 20%, below most alert thresholds. Per-pipeline queries against /_node/stats/pipelines/<pipeline_id> are not optional in multi-pipeline deployments.

Resources the pipeline competes for

Resource	How Logstash uses it	Failure signature when exhausted
JVM heap	In-flight events, filter state, plugin buffers, memory queue	GC pauses, death spiral, OOM kill
CPU	Grok, dissect, JSON parsing, Ruby filters, date parsing	Queue grows with high CPU and high worker utilization
Disk I/O	PQ page writes, DLQ writes, sincedb, logs	PQ drain slows, `queue_push_duration` rises
File descriptors	One per tailed file, per connection, PQ pages	New connections and file opens fail abruptly
Network	Output connections, input listeners	Output duration climbs, retries rise
Worker threads	Fixed pool, one batch at a time	Queue grows even with idle CPU (threads parked on I/O)

The two traps worth memorizing are heap and worker threads. The JVM heap holds in-flight events, and GC pauses freeze the entire pipeline, including the monitoring API. The worker pool is the hard concurrency ceiling: when all workers are blocked on output I/O, adding CPU does nothing, because the bottleneck is waits, not compute.

The five failure archetypes

Nearly every Logstash incident is one of these, and each has a distinct signal fingerprint:

1. Backpressure wedge. Outputs slow or fail, queue fills, inputs block, upstream backs up. CPU is usually low because workers wait on I/O. The tells: output duration rising, output errors or retries in the log (429s, timeouts, connection failures), queue growing, flow.queue_backpressure rising. Hot threads show workers parked in output calls.

2. Compute bottleneck (“grok hell”). Expensive filters consume all CPU: complex regex with catastrophic backtracking, Ruby filters, heavy JSON manipulation, DNS enrichment. CPU is high, worker utilization is pinned near 100%, queue grows, and per-plugin stats show one filter dominating worker_utilization or duration_in_millis. Distinguished from the backpressure wedge by CPU and by the absence of output errors.

3. GC death spiral. Heap fills, GC runs longer and more often, processing time collapses, events accumulate, heap fills faster. The process stays alive and the API may still answer, but useful throughput is near zero. The correct signal is not peak heap usage but the post-GC floor: if the heap level after collection keeps ratcheting upward and old-gen GC time climbs, you are in or approaching the spiral. Alerting on heap_used_percent > 80% fires on every normal GC peak, trains everyone to ignore it, and is already silenced when the real crisis arrives.

4. PQ exhaustion. A downstream outage lasts longer than the persistent queue’s capacity. Occupancy climbs steadily, everything looks stable, and then queue.max_bytes is reached and inputs block with no grace period. The only advance warnings are occupancy trend and fill rate (flow.queue_persisted_growth_bytes). The question to always be able to answer: (queue.max_bytes - queue_size_in_bytes) / current_fill_rate = how many minutes of runway remain.

5. Silent correctness failure. The pipeline is up, throughput is normal, and the data is wrong: _grokparsefailure tags rising, DLQ growing, field drift, event duplication, in/out ratio drifting from the intended transformation ratio. Every availability and throughput metric stays green. This is the most dangerous archetype precisely because availability monitoring cannot see it; it only surfaces through correctness signals like the grok filter failures counter, DLQ growth, and destination-side tag queries.

Where the deployment shape changes the model

Memory vs persistent queue. Memory queue makes failure abrupt and queue depth less visible; PQ adds disk monitoring, corruption risk, and the “healthy until suddenly not” masking pattern.
Containerized Logstash. Cgroup CPU limits (CFS quota throttling) make moderate-looking CPU usage misleading: the process may be throttled while reporting well under 100%. Pod termination may not wait for queue drain, and OOM kills against a PQ create the corruption scenario.
Kafka input. When Logstash consumes from Kafka, consumer group lag is the primary backlog signal, not the internal queue, and Kafka retention gives far longer runway than any Logstash queue.
Logstash-to-Logstash. In two-tier (shipper to aggregator) deployments, the central instance’s input queue is the bottleneck and monitoring shifts to connection health between tiers.
Load-balanced instances. Health checks must tolerate Logstash’s slow startup (JVM warmup, filter and regex compilation: expect low throughput and high CPU for the first 30-60 seconds, and gate alerts on jvm.uptime_in_millis).

Signals to watch in production

These map directly onto the machine described above. All come from the node stats API (default port 9600) unless noted.

Signal	What part of the machine it measures	Warning sign
`pipelines.<name>.flow.output_throughput`	End-to-end useful work	Zero while input throughput is non-zero (the “living dead”)
`pipelines.<name>.queue.events_count` / `queue_size_in_bytes`	Queue occupancy	Monotonic growth over 15 minutes; PQ occupancy over 80%
`pipelines.<name>.flow.queue_backpressure`	Input threads blocked on the queue	Sustained rise above baseline (inputs being throttled)
`pipelines.<name>.flow.worker_utilization`	Worker pool saturation	Sustained above 90%; per-plugin breakdown shows which stage dominates
`pipelines.<name>.flow.queue_persisted_growth_bytes`	PQ fill rate	Positive during a downstream incident: compute runway immediately
`jvm.mem.pools.old.used_in_bytes` (post-GC floor)	Heap health	Floor ratcheting upward; old-gen above 85% of its max
`jvm.gc.collectors.old.collection_time_in_millis` (rate)	GC overhead	Old-gen GC consuming over 10-20% of wall time
`process.open_file_descriptors` vs `max_file_descriptors`	FD pressure	Ratio over 80%, or slow growth over weeks (leak)
`plugins.filters[].failures` (grok)	Data correctness	Any sustained rise above baseline
`pipelines.<name>.dead_letter_queue.queue_size_in_bytes`	Permanent delivery failures	Any unexpected growth; replay is manual
`pipelines.<name>.reloads.failures`	Config drift	Any failure: running config now differs from deployed config

Two measurement cautions from the field. First, pipeline stats reset on config reload, creating artificial zero-dips in graphs; do not mistake them for outages. Second, do not poll the API faster than about every 10 seconds: the metrics endpoint itself can slow an extremely loaded instance.

How Netdata helps

The failure archetypes above are distinguishable only by correlation, and correlation is where per-second collection pays off:

Backpressure vs compute, at a glance. Queue growth plus low CPU plus rising output errors is a downstream problem; queue growth plus high CPU plus high worker utilization is filter cost. Watching both planes on one dashboard separates the two wedges without a hot-threads capture.
PQ runway before the page. Queue occupancy, queue_persisted_growth_bytes, and filesystem free space trended together turn “PQ masking an outage” from an invisible time bomb into a countdown you can act on.
The post-GC floor, not the peak. Per-second JVM heap and old-gen collection metrics make the sawtooth floor visible, so heap alerts key on accumulation instead of firing on every normal GC peak.
The living dead. Process liveness says nothing about delivery. Alerting on output throughput gated on input throughput and JVM uptime catches the alive-but-useless states (GC spiral, wedged workers, failed reload) that systemd checks miss.
Per-pipeline and per-plugin breakdowns. In multi-pipeline deployments, per-pipeline throughput and per-plugin duration surface the one failed pipeline or the one expensive filter that node-level aggregates average away.
Cold-start noise suppression. Uptime-aware alerting avoids paging on the normal 30-60 seconds of JVM warmup, JIT compilation, and PQ replay after restarts.