Logstash monitoring maturity model: from survival to expert

Most Logstash deployments are monitored at the wrong level. Teams alert on process liveness and heap percentage, then get paged by users asking where Wednesday’s logs went. The process was up, the API returned 200, and throughput looked fine the whole time. The gap is not tooling. It is which signals the team decided to watch.

This article defines a four-level maturity model for Logstash monitoring: Survival, Operational, Mature, and Expert. Each level names the specific signals to collect, why they matter, and what class of failure becomes visible that was invisible at the level below. Use it as an audit checklist against your current setup, and as a roadmap for what to add next. The levels are cumulative: every level assumes everything below it is already in place.

Two caveats before the lists. First, pipeline flow metrics (flow.input_throughput, flow.output_throughput, flow.queue_backpressure, flow.worker_concurrency, flow.worker_utilization) exist only in Logstash 8.6 and later (pipeline-level worker_utilization was added in 8.13); on older versions you must derive rates from cumulative counters. Second, thresholds here are baseline-relative on purpose. Absolute thresholds like “alert below 1000 events/sec” fire all night on quiet pipelines and never fire after traffic doubles.

flowchart TD
  L1["Level 1: Survival
Is it alive and delivering?"]
  L2["Level 2: Operational
Per-pipeline and resource health"]
  L3["Level 3: Mature
Per-plugin breakdown and runway"]
  L4["Level 4: Expert
SLOs and silent-loss detection"]
  L1 --> L2 --> L3 --> L4

Level 1: Survival

The minimum to avoid complete blind spots. With only these signals you can detect total failure, but nothing about degradation, data quality, or impending problems.

Pipeline output rate. pipelines.<name>.events.out or flow.output_throughput from GET /_node/stats/pipelines. Output rate is the real health signal. Zero output while inputs are active is the “living dead” scenario that process checks miss entirely.
Queue growth. queue.events_count and, for persistent queues, queue_size_in_bytes / max_queue_size_in_bytes. A monotonically growing queue for more than 15 minutes means the pipeline is falling behind.
Output errors and retries. Log patterns for retry, error, reject, timeout, 429, and 503 in logstash-plain.log, plus per-output event stats. Sustained nonzero retries are abnormal and usually precede queue growth.
JVM heap usage. jvm.mem.heap_used_percent from /_node/stats/jvm. At this level it is a coarse memory-pressure check, nothing more.
API liveness. curl -sS --connect-timeout 5 http://127.0.0.1:9600/ plus systemctl is-active logstash. A 200 only means the JVM and HTTP server are alive; pair it with output rate or you will miss the living-dead case.

What Level 1 cannot see: partial pipeline failures in multi-pipeline setups, GC death spirals in their early stage, parse failures, and anything about data correctness.

Level 2: Operational

What a professional team running Logstash in production needs. The theme of this level is moving from global aggregates to per-pipeline and per-resource views, because aggregates average away localized failures. One dead pipeline out of five drops aggregate throughput by 20%, below most alert thresholds.

Per-pipeline stats. Query /_node/stats/pipelines/<pipeline_id> individually instead of relying on node-level aggregates. Also verify expected pipeline IDs are present in the response; a missing pipeline after a failed reload is a partial outage.
Worker utilization. flow.worker_utilization. Sustained values above 90% during normal peaks mean the pipeline cannot absorb bursts without queue growth. High utilization with low host CPU points at blocking I/O or lock contention, not compute.
GC overhead. delta(jvm.gc.collectors.*.collection_time_in_millis) / delta(wall time). Above 10% is concerning, above 20% is severe. Track old-gen separately: rising old.collection_count is far more dangerous than rising young-gen counts.
Disk on PQ, DLQ, and log volumes. queue.data.free_space_in_bytes from the API plus df -h /var/lib/logstash /var/log/logstash. PQ max_bytes limits queue size but not disk usage if the queue shares a partition with logs or the DLQ. A full filesystem crashes the process even when the queue is within limits.
Grok failure counter. plugins.filters[].failures for grok filters, direct from the API. This is the cheapest data-quality signal available and almost nobody monitors it.
DLQ growth. dead_letter_queue.queue_size_in_bytes. Any unexpected growth on production data is a correctness failure: events are being permanently diverted instead of delivered. Note that DLQ is disabled by default; without it, permanently failing events are logged and silently lost.
File descriptor ratio. process.open_file_descriptors / process.max_file_descriptors from /_node/stats/process. Alert above 80%. File inputs hold one FD per tailed file, and FD leaks grow for weeks before they bite.
Input vs output comparison. flow.input_throughput vs flow.output_throughput. A sustained input/output ratio above 1.1 for more than 15 minutes with no drain periods means a backlog is building. Account for pipelines that legitimately transform event counts with clone, split, or drop.

What Level 2 cannot see: which specific plugin is the bottleneck, how much runway remains before the queue fills, and composite failures that require correlating several signals at once.

Level 3: Mature

This is where monitoring starts answering “how long do we have” and “exactly which component is at fault” instead of “is something wrong.”

Per-plugin breakdown. plugins.filters[].flow.worker_utilization, worker_millis_per_event, and events.duration_in_millis. Logstash problems are usually plugin-local. One filter consuming more than 80% of pipeline processing time is your culprit. Assign explicit id values to plugins in config so the stats map back to readable names.
Queue runway. flow.queue_persisted_growth_bytes gives the fill rate directly; positive means growing, negative means draining. Runway is (max_queue_size_in_bytes - queue_size_in_bytes) / growth_rate. Page when occupancy exceeds 90%, smoothed growth is positive over a 5-15 minute window, output rate is below input rate, and runway is under 30 minutes.
Queue backpressure. flow.queue_backpressure, the fraction of time input threads spend blocked pushing into the queue. Treat it as baseline-relative: the magnitude depends heavily on pipeline shape and cannot be compared across pipelines.
Processing-duration trend. delta(events.duration_in_millis) / delta(events.filtered) for per-event average. A sustained 2x rise over baseline without a change in event complexity means filters or enrichment got more expensive.
Composite pattern detection. Correlate signals into the known archetypes: backpressure wedge (low CPU, growing queue, output errors), grok hell (high CPU, high worker utilization, growing queue), GC death spiral (rising post-GC floor, GC overhead above 20%, throughput collapse). Single signals page too often; the combinations are the reliable triggers.
Reload state. pipelines.<name>.reloads.successes, .failures, and .last_error. Any new failure means the running config has diverged from the deployed config, invisibly.
Cardinality drift. events.in vs events.filtered vs events.out against the intended transformation ratio. Unexplained drift catches accidental drops, clones, and sincedb corruption re-reading files.
Hot threads on demand. GET /_node/hot_threads during incidents. Repeated snapshots separate filter burn from output waits. Not an alerting signal; forensic evidence.

Level 4: Expert

Deep signals for catching subtle or chronic issues, and for turning “is it up” into “is it meeting its commitment.”

Per-pipeline latency and freshness SLOs. End-to-end freshness is not a built-in metric. Derive it by comparing source-set @timestamp against destination indexing time. Throughput can look perfect while events arrive minutes late, and for alerting and security use cases stale data is as bad as missing data.
Post-GC floor. The heap level after garbage collection, tracked via jvm.mem.pools.old trend. A rising floor is the early-warning leak signal; the sawtooth peak is noise. This replaces the useless “heap > 80%” alert that fires on every normal GC peak.
Sincedb health. For file inputs, sincedb corruption causes re-reads from the beginning: a spike in events.in with duplicates downstream. The Node Stats API has no sincedb health fields; this requires custom file/state checks on the sincedb files.
Elasticsearch per-document bulk failures. Bulk requests can return HTTP 200 while individual documents fail mapping or type checks. Logstash counts the batch as out. Detecting this requires ES-side bulk rejection metrics or document-level output stats, not Logstash output counters; the Node Stats API exposes no per-document bulk failure counter.
Config-drift detection. Running config versus source-controlled config. Reload counters tell you a reload failed; drift detection tells you the deployed and running configs differ even when nothing recently failed. This is a custom external check (e.g. config hash comparison) unless you use Central Pipeline Management, which sources pipeline config from Elasticsearch.

Choosing your target level

Level	Catches	Still misses
Survival	Total failure, living-dead process	Degradation, partial pipeline failure, correctness
Operational	Per-pipeline failure, resource exhaustion, DLQ and parse failures	Plugin-level root cause, time-to-full, composite patterns
Mature	Bottleneck attribution, queue runway, reload drift, GC vs CPU vs backpressure split	Latency SLO breaches, silent ES bulk loss, config drift
Expert	Silent data loss, leak early warning, freshness SLOs, drift	Business-logic correctness (out of scope for metrics)

A reasonable target for most production teams is Level 2 everywhere, Level 3 for pipelines that feed alerting or compliance reporting, and Level 4 only where a silent-loss incident has already cost you something.

How Netdata helps

Netdata polls the Logstash monitoring API directly, so Survival and Operational signals (output throughput, queue occupancy, heap, GC, FD ratio, reload counters) are collected per second without hand-rolled curl scripts.
Per-pipeline and per-plugin charts make Level 2 and Level 3 breakdowns visible by default, which is where aggregate-only monitoring hides failures.
Correlating worker_utilization against host CPU on one dashboard is the fastest way to split the two most common patterns: compute bottleneck (high both) versus output blocking (high utilization, low CPU).
Persistent queue occupancy alongside disk free space on the same volume exposes the PQ-masked-outage pattern before the queue hits max_bytes.
Baseline-relative anomaly detection on throughput handles the workload-variation problem that breaks absolute thresholds, without you hand-tuning rolling averages per pipeline.