Most Logstash monitoring setups answer the wrong question. They answer “is the process running?” when the question that matters is “are events leaving the pipeline?” A Logstash JVM can be alive, healthy by systemd’s standards, and returning 200 from its monitoring API while the queue is full, workers are blocked on a dead Elasticsearch, and zero events have been delivered for an hour. Process liveness is necessary. It is nowhere near sufficient.

This checklist organizes the signals a production Logstash deployment needs into four maturity levels, from the floor that avoids total blindness to the deep signals that catch silent correctness failures. Use it to audit an existing setup or to build one that will not betray you at 3 a.m.

Almost everything here comes from the Logstash monitoring API, which binds to 127.0.0.1:9600 by default (configurable via api.http.port). The two workhorse endpoints are /_node/stats/pipelines and /_node/stats/jvm. In containerized deployments the API still binds to loopback by default, so you must publish the port and may need to bind to a non-loopback address to scrape it. Do not poll faster than every 10 seconds; the stats API competes with the pipeline for JVM resources on loaded instances.

How to use this checklist

Work down the levels in order. Level 1 is not a recommendation; it is the floor. If you are missing a Level 1 signal, you have a blind spot that has already caused or will cause an undetected outage. Levels 2 and 3 are where most teams should aim. Level 4 signals pay off after you have been burned by the failure modes they detect.

One rule applies across every level: thresholds should be baseline-relative, not absolute. “Alert below 1000 events/sec” fires all night and never during a real incident after traffic doubles. Percent deviation from a rolling average for the same time window is more work and dramatically more useful. Also gate time-sensitive alerts on JVM uptime (jvm.uptime_in_millis > 300000); cold starts produce false positives in almost every signal for the first 30 to 120 seconds.

flowchart TD
  L1["Level 1: Survival - output rate, queue, output errors, heap, API liveness"]
  L2["Level 2: Operational - per-pipeline stats, workers, GC, DLQ, FDs, parse failures"]
  L3["Level 3: Mature - per-plugin breakdown, runway, backpressure, reloads, drift"]
  L4["Level 4: Expert - post-GC floor, freshness, recovery path, bulk failure detail"]
  L1 --> L2 --> L3 --> L4

Level 1: survival

The minimum to avoid complete blind spots.

  • Pipeline output rate. The rate of events successfully emitted to outputs: pipelines.<name>.events.out as a counter, or flow.output_throughput as a pre-computed rate. This is the primary functional health signal. Zero output with non-zero input (gate on flow.input_throughput > 0 to avoid idle-server false positives) means the pipeline is alive but delivering nothing. Alert when output drops more than 50% below the rolling baseline, or to zero sustained, while input is active.
  • Queue growth and occupancy. pipelines.<name>.queue.events_count and, for persistent queues, queue.queue_size_in_bytes against queue.max_queue_size_in_bytes. A steadily growing queue means events arrive faster than they leave. For memory queues, any monotonic increase over 15 minutes is concerning. For PQ, alert above 80% occupancy sustained.
  • Output error and retry activity. Grep the log for retry, error, reject, timeout, 429, and 503 patterns, and check per-output plugin stats in /_node/stats/pipelines. Sustained non-zero retry activity is abnormal; brief retries during downstream failover self-heal. Severity escalates when retries coincide with queue growth.
  • JVM heap usage. jvm.mem.heap_used_percent from /_node/stats/jvm. Heap pressure triggers GC, which pauses all processing. A steady 75% with an efficient sawtooth is normal; alert fatigue from naive “>80%” rules is why teams miss real crises. See Level 4 for the correct version of this signal.
  • Process and API reachability. curl -sS --connect-timeout 5 http://127.0.0.1:9600/ plus systemctl status logstash. The first check in any incident. A 200 only proves the JVM and HTTP server are alive; always pair it with output throughput. During severe GC pauses the API can time out while the process recovers, so treat standalone unreachability as a ticket and let the output-rate signal carry page weight.

Level 2: operational

Everything in Level 1, plus what a professional team needs to run Logstash without surprises.

  • Per-pipeline stats, not aggregates. In multi-pipeline deployments (pipelines.yml), one dead pipeline out of five drops aggregate throughput by 20%, below most thresholds. Query /_node/stats/pipelines/<pipeline_id> individually and verify each expected pipeline ID is present and running. On recent 8.x, the /_health_report endpoint gives structured pipeline health states; prefer it over inferring state from raw stats.
  • Input rate and input-vs-output ratio. events.in / flow.input_throughput versus the output equivalents. Input exceeding output by more than about 10% for over 15 minutes with no drain periods means a backlog is building. Know the intended transformation ratio per pipeline first: clone and split filters legitimately produce more output than input, drop filters fewer.
  • Worker utilization. flow.worker_utilization (flow metrics are available in 7.14+ and standard in 8.x). Sustained above 90% during normal peaks means no headroom for bursts. Interpret with CPU: high utilization plus high CPU is a compute bottleneck; high utilization plus low CPU is workers blocked on output I/O.
  • GC overhead. jvm.gc.collectors.old.collection_time_in_millis and collection_count, computed as a rate: delta(collection_time) / delta(wall time). Over 10% of wall clock is concerning, over 20% severe. Track old-gen separately; rising old-gen frequency is far more dangerous than young-gen activity.
  • Disk space on PQ, DLQ, and log volumes. queue.data.free_space_in_bytes reflects filesystem free space, not just PQ allocation. PQ max_bytes does not protect you if the queue shares a partition with logs or the DLQ. A full disk crashes the process even when the queue is within its configured limits.
  • Parse failure indicators. The grok filter’s plugins.filters[].failures counter is directly available in per-plugin stats and almost never monitored. Track its rate, not the absolute value. A rising failure rate with normal throughput is a silent correctness disaster: events reach the destination with wrong or missing fields. Cross-check downstream for _grokparsefailure and _jsonparsefailure tags, since aggregate rates can hide 100% failure on one small critical stream.
  • DLQ growth. pipelines.<name>.dead_letter_queue.queue_size_in_bytes. Any unexpected growth on production data is a ticket: events are being permanently rejected and diverted. Two caveats. First, the DLQ is disabled by default (dead_letter_queue.enable: false), and without it, permanently failed events are logged and silently lost. Second, teams that enable it often never monitor or replay it, which turns the safety net into silent data loss with extra disk usage.
  • File descriptor pressure. process.open_file_descriptors / process.max_file_descriptors from /_node/stats/process. Alert above 80% sustained. File inputs hold one FD per tailed file, so wildcards matching thousands of files can exhaust the limit; connection leaks show up as slow FD growth over weeks.

Level 3: mature

Everything above, plus the signals that surface degradation before it becomes an outage.

  • Per-plugin performance breakdown. plugins.filters[].flow.worker_utilization, plugins.filters[].events.duration_in_millis, and the output equivalents. Logstash problems are usually plugin-local; aggregate metrics hide the one bad grok pattern consuming 80% of pipeline time. Assign explicit id values to filters in config so stats map back to config without guesswork.
  • Queue fill rate and runway. flow.queue_persisted_growth_bytes gives you the PQ fill rate directly. Runway is (max_queue_size_in_bytes - queue_size_in_bytes) / current_fill_rate. This converts “PQ is growing” into “inputs block in 40 minutes.” The growth signal moves in chunks as pages are allocated and freed, not smoothly.
  • Queue backpressure. flow.queue_backpressure, the fraction of time input threads spend blocked pushing into the queue. This is the earliest backpressure signal, visible before the queue visibly fills. Alert on a sustained rise above the pipeline’s own baseline.
  • Event processing duration trend. Compute per-event average as delta(events.duration_in_millis) / delta(events.filtered). A sustained 2x rise without a workload change points at filter cost, enrichment latency, or pathological regex backtracking.
  • Configuration reload state. reloads.successes, reloads.failures, reloads.last_error. Any failure is a ticket: the old config keeps running, which is safe but creates invisible drift between deployed and running configuration. Teams routinely believe a fix shipped when it did not.
  • Composite failure pattern detection. The three big cascades have distinct signatures. Backpressure: output errors rise, queue grows, CPU stays moderate. Compute bottleneck (“grok hell”): CPU pegged, worker utilization pinned, no output errors. GC death spiral: post-GC floor rising, old-gen GC frequent, API flaky, throughput wobbling while the process looks alive. Alerting on the combination is far safer than any single leg.
  • Incident-time hot threads. GET /_node/hot_threads. Not an alert signal; take repeated snapshots during incidents to separate filter burn from output waits. Store snapshots from peak load for forensics.

Level 4: expert

Deep signals for catching subtle or chronic issues. Adopt these after the corresponding incident has cost you once.

  • Post-GC floor monitoring. The heap signal that actually works: the heap level after garbage collection, not the peak. A rising floor plus old-gen pool above 85% of max plus GC overhead above 20%, sustained with visible throughput impact, is the GC death spiral composite worth paging on. The raw heap_used_percent > 80% rule fires on every normal sawtooth peak and gets silenced before the real event.
  • End-to-end event freshness. Logstash exposes processing duration, not freshness. Derive it by comparing event @timestamp against destination indexing time. Throughput can stay flat while latency grows, and for alerting and security use cases stale data is as bad as missing data.
  • Elasticsearch per-document bulk failure detection. Logstash counts a bulk request as “out” when ES returns 200, even if individual documents inside it were rejected for mapping conflicts. The event counter looks healthy while data is lost. Catch this with documents.non_retryable_failures in the Elasticsearch output stats and ES-side rejection metrics.
  • Recovery-path monitoring. Watch queue drain rate after a downstream outage, not just failure detection. PQ drain can take far longer than fill, and high I/O plus reduced throughput during drain is expected, not a new incident.
  • PQ runway paging. The full page condition: PQ occupancy above 90%, smoothed positive growth over 5 to 15 minutes, output below input, runway under 30 minutes, active input, JVM uptime over 600 seconds. All legs together mean inputs block imminently with no self-recovery in sight.
  • Configuration drift detection. Compare running config against source-controlled config. File mtimes under /etc/logstash outside deploy windows plus reload log entries catch both accidents and unauthorized changes.
  • Sincedb health for file inputs. Sincedb corruption causes Logstash to re-read files from the beginning, producing a spike in events.in and duplicates downstream that look like a traffic surge.

What this checklist deliberately excludes

Three traps show up in almost every Logstash monitoring setup. Absolute throughput thresholds instead of baseline-relative ones. Heap percentage alerts instead of post-GC floor trends. And process liveness treated as health. If your current setup is built on those, fixing them is worth more than adding any new signal.

How Netdata helps

  • Netdata’s Logstash collector scrapes the monitoring API on port 9600 and charts pipeline event rates, queue depth, JVM heap, GC, and process stats per second, so output-rate drops and queue growth are visible without hand-rolled polling scripts.
  • Per-pipeline charts make multi-pipeline deployments readable: one dead pipeline stands out instead of being averaged away in a global aggregate.
  • JVM heap and GC charts side by side make the death-spiral signature (rising floor, rising old-gen time, falling throughput) a visual correlation rather than a log-diving exercise.
  • File descriptor usage against the configured limit is charted continuously, which catches slow FD leaks over weeks before the “too many open files” cliff.
  • Anomaly detection on throughput metrics handles the baseline-relative problem: deviations from the learned pattern for that time of day stand out without hand-tuned thresholds per pipeline.