$ guides / logstash ▌

LOGSTASH · OPERATIONS PLAYBOOK

Logstash's quiet failure mode: a pipeline that looks alive while it moves nothing

A JVM event pipeline where a fixed pool of worker threads sits between the inputs and the outputs, buffered by a queue that is either in memory or on disk. When an output slows, backpressure propagates all the way to the source while every process stays up and the API keeps returning 200. We trace how that pipeline behaves under load, the handful of failure archetypes it keeps producing, and what to do when the events stop flowing.

> Start with the monitoring checklist → # Jump to the full guide list

Logstash's defaults get events flowing in minutes, then hand you a set of failure modes that all look identical from the outside: the process is up, the API answers, and nothing is moving.

The defaults work. Until Elasticsearch returns 429 on a bulk request, the output retries, workers block, the queue fills, and inputs are backpressured while Beats agents quietly buffer upstream. Until the heap fills and the JVM spends more time in GC than doing work — throughput collapses but the process never dies. Until a source changes its log format and every event sails through tagged _grokparsefailure, green on every dashboard and useless in every index. Until a persistent queue that absorbed a downstream outage for six hours hits max_bytes and blocks — or an unclean kill corrupts its pages and Logstash will not start at all.

These guides are written for engineers who already run Logstash, not for people learning what a pipeline is. The goal is the mental model of how the pipeline actually behaves under load — why output throughput, not process liveness, is the real health signal — the failure patterns that keep recurring, the monitoring story that catches them before data is lost, and the runbooks you wish someone had handed you before your last incident.

How Logstash actually runs in production

Logstash is not just a filter engine. It is a JVM process where each input runs its own thread, events are buffered in a queue, and a fixed pool of worker threads pulls batches and runs the entire filter chain and outputs inline. Two things decide whether it keeps up: whether workers can drain the queue faster than inputs fill it, and whether the heap stays ahead of GC. Most production failures live between these layers, not inside any one of them.

inputs / sources

Each input plugin (Beats, Kafka, TCP, HTTP, file, JDBC, syslog) runs in its own thread and pushes events downstream. When the queue is full, inputs block here — and that block propagates to the source: Beats stops advancing its registry, Kafka consumer lag grows, files pile up.

INPUT

codecs

Codecs (line, json, multiline) deserialize raw bytes into Logstash Event objects on the way in. A codec mismatch — plain text arriving on a <code>json</code> codec, or a multiline pattern that splits wrong — corrupts events before any filter runs.

CODEC

queue (memory or persistent)

The central buffer and the most important architectural choice. The memory queue is small, fast, and lost on crash; the persistent queue writes 64MB pages to disk, survives restarts, and can mask a downstream outage for hours. Either way, when it is full, inputs block. This is where backpressure originates.

QUEUE

pipeline workers

A fixed pool (<code>pipeline.workers</code>, default = CPU cores) pulls batches (<code>batch_size</code> default 125) from the queue. Each worker runs the whole filter chain and the outputs sequentially, inline. The pool is the processing bottleneck: if workers are all busy or all blocked, the queue grows.

WORKER

filter chain

grok, json, date, dissect, ruby, geoip run in order inside the worker thread. This is where CPU goes — a grok pattern with catastrophic backtracking or a heavy ruby filter pins a worker and per-event duration climbs. It is also where correctness is decided: a failed match tags the event and passes it through.

FILTER

outputs

Elasticsearch, Kafka, HTTP, and others deliver events, retrying on failure. This is where downstream trouble enters the pipeline: a slow or rejecting output raises output duration before throughput drops, and workers block waiting on it — the start of every backpressure cascade.

OUTPUT

JVM heap + GC

In-flight events, plugin buffers, filter state, and the in-memory queue all live on the heap. Sustained pressure triggers frequent GC, and GC pauses freeze all processing. The dangerous signal is the post-GC floor rising, not the sawtooth peak — that is the GC death spiral forming.

JVM

DLQ + monitoring API

The dead-letter queue (disabled by default) captures events an output permanently rejects; without it they are logged and silently lost. The node stats API on port <code>9600</code> is where every signal above is read — but a 200 from it only proves the JVM is alive, not that the pipeline is working.

OBSERVE

Why this matters: 'Logstash is slow' or 'events stopped arriving' can come from a full queue blocking inputs, a slow output backpressuring workers, a CPU-bound grok filter, a GC death spiral, a persistent queue that filled or corrupted, a single failed input hidden in a multi-input pipeline, or a parse failure quietly wrecking data quality. The symptom rhymes but each layer has a different signal — and a different fix.

The failures you'll actually see

Most Logstash incidents fall into a small set of recurring patterns. Recognise the shape, and triage gets dramatically faster.

CRITICAL

The backpressure wedge

The output slows, rejects, or times out — Elasticsearch returns 429, a broker fails, the network stalls. Output duration rises before throughput drops, workers block waiting on acknowledgement, the queue fills, and inputs are backpressured. Upstream Beats and Kafka consumers back up while every component still reports 'running'. CPU is low — workers are waiting on I/O, not computing.

Output rate falling while input rate holds, then both drop
Output errors / retries rising (429, timeouts, connection failures)
queue.events_count or PQ occupancy growing steadily
CPU low despite the growing queue — the tell vs grok hell

Investigate →

CRITICAL

The GC death spiral

The heap fills, GC runs more often and for longer, less CPU is left for events, so events accumulate and the heap fills faster. Full-GC pauses freeze everything. Throughput collapses while the process stays alive — health checks based on process existence pass. Left alone it ends in OutOfMemoryError or an OOM kill.

Post-GC heap floor rising (not just the sawtooth peak)
Old-gen collection count and time climbing
GC overhead > 20% of wall-clock time
Output rate wobbling or near zero, API responsiveness degrading

Investigate →

CRITICAL

The pipeline that won't start

Logstash refuses to come up. The most common cause is a stale lock: another instance using the configured data.dir after an unclean exit or a duplicate service. Others: a persistent queue corrupted by a hard kill, a config error, or an input port already in use. Until it is resolved nothing runs, and in multi-pipeline setups one bad pipeline can fail while the JVM starts the rest.

'could not be started because there is already another instance' on boot
java.io.IOException / checkpoint errors from a corrupted PQ
'Address already in use' on a Beats / TCP / HTTP input port
Expected pipeline id absent from /_node/stats/pipelines

Investigate →

ACTIVE

The silent correctness failure

Logstash is up, busy, and delivering events — but the data is wrong. A source changed its log format and the grok pattern no longer matches, so events flow through tagged _grokparsefailure with raw, unstructured fields. The most dangerous pattern, because every uptime and throughput metric stays green. Evidence lives only in failure tags, DLQ growth, and downstream field inspection.

Grok filter failures counter rising in per-plugin stats
_grokparsefailure / _jsonparsefailure tags climbing at the destination
events.in vs events.out ratio drifting from the intended shape
Throughput normal; downstream users report missing or malformed fields

Investigate →

IMMINENT

The persistent-queue time bomb

A persistent queue absorbs a downstream outage and makes a broken system look healthy — for hours. Occupancy climbs quietly while output stays below input. The missing question is always 'how long until full?'. At max_bytes the graceful fill becomes a cliff: inputs block instantly. And if the process is killed hard with pages in flight, the queue can corrupt and refuse to reload.

PQ occupancy rising past 80% with positive smoothed growth
flow.queue_persisted_growth_bytes sustained positive
Output rate below input rate for the whole window
queue.data.free_space_in_bytes shrinking on the PQ volume

Investigate →

ACTIVE

File-descriptor exhaustion

The process is alive but starts failing at the edges: new connections are refused, PQ page files will not open, new tailed files are ignored. A slow leak — a file input matching thousands of files, output reconnection churn, or connections that never close — climbs for days then hits a cliff. It is frequently misdiagnosed as a disk problem because file opens fail.

'Too many open files' in logstash-plain.log
open_file_descriptors / max_file_descriptors > 0.8
FD count climbing steadily and never returning to baseline
Connection or file-open errors while CPU and heap look fine

Investigate →

Choosing a tool

Best Logstash Monitoring Tools Ranked for 2026

A ranked review of the tools teams actually shortlist here, what each one is genuinely good at, and how the pricing behaves as you scale.

Read the buyer's guide →

Logstash monitoring maturity levels

Logstash observability works in four practical levels. Each is a complete operation, not a stepping stone. Pick the level that matches how much your pipeline matters. Most production deployments should land at the second level.

Level 1: Survival

Know that something is wrong

Survival monitoring is the floor. With these signals you can answer one question: is the pipeline alive and actually moving events? You will not learn what broke, but you will learn that something broke — including the 'living dead' case where the process is up and throughput is zero. Survival is enough for dev instances and non-critical pipelines.

Process / API reachability (9600) Is the JVM up and the node stats API answering at all?
Pipeline output rate (events.out) The real health signal — zero output with live input is the living dead.
Queue growth / occupancy Rising queue is backpressure; the earliest sign inputs will block.
Output error / retry activity The most common cause of queue growth and eventual outage.
JVM heap usage Sustained high heap is the precursor to the GC death spiral.

Level 2: Operational

Diagnose most incidents on your own

Operational monitoring is what most production deployments should target. Survival tells you something is wrong; operational tells you what. With this coverage your team can usually diagnose an incident on its own: backpressure, compute bottlenecks, GC pressure, parse failures, and disk runway.

Per-pipeline stats (not just global) Aggregate metrics average away one failed pipeline of many.
Worker utilization flow.worker_utilization sustained > 90% means no headroom for spikes.
GC overhead (old-gen) Old-gen collection time and frequency, not just heap percent.
Disk space on PQ / DLQ / log volumes A full shared partition crashes the process regardless of PQ max_bytes.
Grok failures counter Parse quality — almost never watched, directly in per-plugin stats.
Dead-letter queue growth Events being permanently rejected and diverted (or silently lost).
File descriptor ratio Cliff-edge: at the limit, connections and file opens fail.
Input vs output rate The gap predicts a backlog before queue depth confirms it.

Level 3: Mature

Catch problems before they become incidents

Mature monitoring catches problems before they wake anyone up. A persistent queue quietly filling, one plugin's per-event cost creeping, a reload that failed and left stale config running, event counts drifting from their intended ratio. None of these page you on day one. They become page-out incidents on day thirty.

Per-plugin performance breakdown plugins.filters[].worker_utilization localises the one bad stage.
Queue fill rate and runway flow.queue_persisted_growth_bytes answers 'how long until full?'.
Queue backpressure metric flow.queue_backpressure — how much ingestion is being throttled.
Event processing duration trend Per-event cost rising reveals filter regressions and ReDoS.
Config reload state reloads.failures — the deployed config quietly differs from running.
Source traffic baseline Expected vs actual input rate per source, not an absolute floor.
Event cardinality drift in/out ratio vs the pipeline's intended transformation ratio.
Incident-time hot threads /_node/hot_threads separates filter burn from output waits.

Level 4: Expert

Reactive instrumentation after real incidents

Expert signals enter your stack the day after a specific incident proved you needed them. End-to-end freshness, the post-GC old-gen floor, sincedb health, Elasticsearch per-document bulk failures, config-drift detection. Most teams never need every signal here. Add the ones your incident history says you do.

Per-pipeline latency / freshness SLOs Throughput can hold while data arrives late — stale is as bad as missing.
Post-GC old-gen floor The heap level after collection, extrapolated to the cliff.
Sincedb health (file inputs) Corruption re-reads files from the start — duplicates downstream.
ES per-document bulk-failure rate HTTP 200 with rejected docs is counted as success — silent loss.
End-to-end event latency Source timestamp vs destination index time; not a built-in metric.
Recovery drain rate after outage How fast the PQ drains once downstream returns, not just detection.
TCP retransmit rate to outputs Network-layer trouble under a healthy-looking output.
Config drift detection Running config vs source-controlled config, reconciled continuously.

Operating mistakes worth avoiding

The traps Logstash teams keep falling into. Each has a clear, well-known fix. Most teams only learn it after an incident.

Monitoring the process, not the pipeline

Teams alert when <code>logstash</code> dies, but miss the 'living dead': the process is up, the API returns 200, the queue is full, and zero events have been processed for hours. Output throughput (<code>events.out</code>) is the real health signal, not process existence. A liveness check that passes while data stops is worse than no check, because it manufactures false confidence.

Ignoring the dead-letter queue (or not having one)

The DLQ is disabled by default, so many deployments run without it — events that permanently fail an output are logged and silently lost. Teams that enable it often do not monitor it, so the DLQ fills, <code>drop_newer</code> starts discarding, and the safety net fails too. This is silent data loss that surfaces weeks later as 'where are last Wednesday's logs?'.

Monitoring throughput but not data quality

Teams know events-per-second and queue depth but have no idea that 15% of events carry <code>_grokparsefailure</code> and are producing useless data in their indices. The grok filter's <code>failures</code> counter is right there in per-plugin stats and almost never monitored. Throughput stays green while a source's format drift quietly wrecks correctness.

Treating JVM heap as a simple number

Alerting on <code>heap_used_percent > 80%</code> fires on every normal GC sawtooth peak (alert fatigue) and then stays silenced during the real crisis. The correct signal is the post-GC floor — heap after collection — and the old-gen trend. A steady 75% with efficient GC is healthy; a rising floor is a real leak or accumulation.

Missing the Elasticsearch partial bulk failure

A bulk request can return HTTP 200 while individual documents fail on mapping or type errors. Logstash counts the batch as 'out' — the event counter looks healthy while data is lost. Catching this needs Elasticsearch-side per-document rejection metrics, not just the Logstash output count. 'events.out looks fine' does not prove delivery.

Not monitoring persistent-queue runway

Teams enable the PQ for durability and treat it as set-and-forget. But the PQ makes a broken system look healthy for hours, and the missing question is always 'how long until full at this fill rate?'. <code>flow.queue_persisted_growth_bytes</code> answers it directly. Without runway monitoring, the first sign of trouble is inputs blocking at the cliff-edge.

Only watching aggregate metrics in multi-pipeline setups

With <code>pipelines.yml</code>, aggregate stats average away localised failures — one failed pipeline of five drops total throughput ~20%, below most thresholds. Per-pipeline monitoring (query <code>/_node/stats/pipelines/<id></code> and alert per pipeline) is essential, because multi-pipeline isolates queues and a stall in one is invisible in the totals.

Setting arbitrary absolute thresholds

'Alert if events-per-second drops below 1000' works until the workload changes: low-traffic hours fire continuously and high-traffic periods never fire. Baseline-relative thresholds — percent deviation from a rolling average for the same time window — are more work to implement but dramatically more useful, and they are the only sane way to alert on a variable pipeline.

Logstash runbooks in this section

Each guide is a focused runbook for one symptom or topic. Pick one when you have an incident, or use the categories to learn the area.

▸

Start here

▸

Queue backpressure and the pipeline wedge

▸

Persistent queue: disk, runway, and corruption

▸

JVM heap, garbage collection, and OOM

▸

Throughput, workers, and CPU-bound filters

▸

Outputs, retries, and downstream backpressure

▸

Parse failures, DLQ, and silent data loss

▸

Inputs, sources, and per-input failures

▸

Pipelines, config reload, and multi-pipeline

▸

Liveness, file descriptors, and disk

▸

Authentication, TLS, and config integrity

WHERE TO GO NEXT

Setting up Logstash monitoring, or putting out a fire?

If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, jump straight to the symptom that matches what you're seeing.

> Start with the checklist > Back to Operations Guides