Logstash multi-pipeline monitoring: why aggregate stats hide a failed pipeline

You migrated to pipelines.yml to isolate workloads: one pipeline per team, per source, or per destination. Each pipeline got its own queue, its own workers, its own failure domain. Right call for fault isolation. But if your monitoring still polls the node once and looks at global event counts, you have given back the visibility the isolation bought you.

The concrete scenario: five pipelines, roughly equal traffic, one of them wedges. An output stalls, a config reload fails, a Kafka consumer group gets stuck rebalancing. That pipeline’s throughput goes to zero; the other four keep flowing. Aggregate node throughput drops by about 20 percent. If your alert is “throughput drops more than 50 percent” or “events per second below N”, nothing fires. The failed pipeline can sit dead for hours while node-level graphs show a normal dip.

This is the most common multi-pipeline monitoring mistake: the node is green, one pipeline is dead, and nobody knows until a downstream consumer asks where their data went. Below: why the math works against you, how to check per-pipeline state right now, and how to restructure collection and alerting so a single failed pipeline pages.

What this means

Multiple pipelines in pipelines.yml run inside one JVM but keep their state separate. Each pipeline has its own queue (memory or persistent), its own worker pool, its own event counters, its own flow metrics, and its own reload state. Persistent queues and dead letter queues are namespaced on disk by pipeline ID. A stall in one pipeline does not directly block the others. It also does not show up in their stats.

The aggregate view, the one most dashboards and quick curl /_node/stats checks surface, sums or averages across all pipelines. That view is fine for JVM-level concerns (heap, GC, process CPU, file descriptors), because those are genuinely shared. It is misleading for pipeline-level health, because pipeline failure is localized by design.

The failure pattern:

flowchart TD
  A[Five pipelines in one JVM] --> B[Pipeline 3 output stalls]
  B --> C[Pipeline 3 queue fills, throughput 0]
  C --> D[Aggregate node throughput drops ~20%]
  D --> E{Alert threshold: -50%?}
  E -->|No| F[No alert fires]
  F --> G[Pipeline 3 PQ fills to max_bytes]
  G --> H[Pipeline 3 inputs block, upstream backs up]
  A --> I[Per-pipeline stats: /_node/stats/pipelines]
  I --> J[output_throughput = 0 while input > 0]
  J --> K[Alert fires per pipeline]

The fix is not a new tool. It is a monitoring topology change: query each pipeline’s stats individually and alert on each pipeline individually.

Common causes

The localized failures aggregate stats most often hide:

The last row matters: not everything needs per-pipeline scoping. Heap, GC, file descriptors, and process CPU are shared across the JVM and belong at the node level. The mistake is applying node-level scoping to per-pipeline things.

Quick checks

Safe, read-only. Assumes the monitoring API is on the default port 9600 on localhost.

# List every pipeline the node is actually running
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | grep -E '^    "[^"]+": \{$'

# Compare against what you configured
grep -E 'pipeline.id' /etc/logstash/pipelines.yml

# Per-pipeline throughput and queue state (the core check)
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | \
  python3 -c "
import sys, json
ps = json.load(sys.stdin)['pipelines']
for pid, p in ps.items():
    flow = p.get('flow', {})
    inp = flow.get('input_throughput', {}).get('current', 'n/a')
    out = flow.get('output_throughput', {}).get('current', 'n/a')
    q = p.get('queue', {})
    print(f'{pid}: in={inp} out={out} queue_type={q.get(\"type\")} queue_events={q.get(\"events\")}')
"

# Check for failed reloads per pipeline
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | \
  python3 -c "
import sys, json
ps = json.load(sys.stdin)['pipelines']
for pid, p in ps.items():
    r = p.get('reloads', {})
    print(f'{pid}: successes={r.get(\"successes\")} failures={r.get(\"failures\")} last_error={r.get(\"last_error\")}')
"

# In Logstash 8.x, structured pipeline health
curl -sS http://127.0.0.1:9600/_health_report?pretty

Two caveats. First, the flow metrics (input_throughput, output_throughput, queue_backpressure) are per-pipeline flow rates available in Logstash 8.x; on 7.x you compute rates from the cumulative events.in / events.out counters by sampling twice. Second, do not poll faster than every 10 seconds; the stats API shares the JVM with the pipelines and adds load on a stressed node.

(/_node/stats/pipelines/<pipeline_id> is a supported per-pipeline path and returns 404 for a missing pipeline id; the commands above use the full pipelines endpoint and filter client-side, which always works.)

How to diagnose it

When you suspect a pipeline is silently dead, or when auditing whether your monitoring would catch one:

1. Enumerate expected pipelines. Read pipelines.yml and list the pipeline IDs that should exist. This is your ground truth. Note that if Logstash was started with -e or -f, pipelines.yml is ignored entirely and a warning is logged; confirm how the service actually starts.
2. Enumerate actual pipelines. Query /_node/stats/pipelines and compare the response keys to the expected list. A missing pipeline ID means the pipeline failed to start or was terminated, and aggregate counters will not tell you that.
3. Check per-pipeline output throughput. For each pipeline, look at flow.output_throughput.current (or delta events.out over a sampling interval). Zero output while flow.input_throughput.current is positive means the pipeline is alive but not delivering: the “living dead” state, scoped to one pipeline.
4. Check per-pipeline queue state. For the suspect pipeline, look at queue.events and, for persistent queues, queue.capacity.queue_size_in_bytes / queue.capacity.max_queue_size_in_bytes. A growing queue on one pipeline while others are flat confirms a localized output or worker problem, not a node-wide one.
5. Check reload state. Non-zero reloads.failures on the suspect pipeline means the running config may not match the deployed config. The pipeline can be “running” and still be wrong.
6. Check per-plugin stats within the pipeline. If the pipeline is flowing but slowly, plugins.filters[].events.duration_in_millis and plugins.outputs[].events.duration_in_millis tell you which stage is the bottleneck.
7. Correlate with node-level signals only for shared resources. If multiple pipelines degrade simultaneously, pivot to /_node/stats/jvm (heap, GC) and /_node/stats/process (CPU, FDs). Simultaneous degradation across pipelines points at the shared JVM, not at individual pipeline configs.

Metrics and signals to monitor

Every row here should be collected and alerted per pipeline, with the pipeline ID as a label or dimension.

Keep these node-level, not per-pipeline: jvm.mem.heap_used_percent (post-GC floor), jvm.gc.collectors.old.collection_time_in_millis, process.cpu.percent, process.open_file_descriptors. Pipelines share the JVM, so these are genuinely global.

Fixes

Restructure collection to be per-pipeline

Whatever collects Logstash stats must query /_node/stats/pipelines and explode the response into per-pipeline series, preserving the pipeline ID as a dimension. If your collector emits a single “logstash events out” metric with no pipeline label, that is the gap. Some aggregation layers have historically flattened this: Metricbeat’s logstash module, for example, has been reported to aggregate queue stats across pipelines in default dashboards, and pipeline-level data has been slow to appear in Stack Monitoring. Verify what your stack actually stores, not what the API returns.

Alert per pipeline, not per node

Rewrite throughput alerts so evaluation is per pipeline ID. “Output throughput zero while input non-zero, for 5 minutes, for any pipeline” catches the single dead pipeline. “Node throughput below 1000 eps” never will. Use baseline-relative thresholds per pipeline, because pipelines legitimately have different traffic volumes and different transformation ratios (clone, split, drop filters all change the in/out relationship).

Add a pipeline-presence check

Alert if the set of pipeline IDs in the stats response differs from the set in pipelines.yml. This catches the hardest case: a pipeline that failed to start and emits nothing at all. In Logstash 8.x, the /_health_report endpoint gives structured pipeline health and is preferable to inferring state from stats presence.

Isolate runaway pipelines operationally

If one pipeline’s PQ is filling because its destination is down, you can reduce blast radius by shedding that pipeline’s non-critical input or temporarily stopping it while the others continue. This is only possible because queues are isolated per pipeline. The same isolation means a PQ max_bytes sized for the whole node’s disk is wrong: each pipeline’s queue competes for the same underlying volume, so sum the configured maxima and compare against actual disk.

Account for shared resource contention

Default settings are tuned for a single pipeline: each pipeline defaults to one worker per CPU core. Five pipelines on an 8-core host means 40 workers competing for 8 cores. Per-pipeline CPU attribution is not exposed, so when you see node CPU saturation, check per-pipeline flow.worker_utilization to find which pipeline is burning it.

Prevention

• Treat pipeline ID as a mandatory label. No Logstash throughput, queue, or error metric should exist in your monitoring system without it. Make this a review item for any new pipeline.
• Dashboard per pipeline, aggregate only for JVM. Node-level dashboards show heap, GC, CPU, FDs. Pipeline-level dashboards show throughput, queue, backpressure, reloads, DLQ. Do not mix the scopes.
• Alert on baseline deviation per pipeline. Absolute thresholds fail because pipelines differ and workloads drift. Compare against each pipeline’s own rolling baseline, and gate zero-throughput alerts on jvm.uptime_in_millis > 300000 to avoid cold-start noise.
• Watch for version-specific stats bugs. The pipelines stats endpoint has had regressions: 7.3.x returned stats for only one pipeline (fixed in 7.3.2, elastic/logstash#11021), and 8.17.3 returned an empty pipelines object with X-Pack monitoring enabled (fixed in 8.17.4 and 8.18.0, elastic/logstash#17185). If per-pipeline graphs suddenly go empty after an upgrade while aggregate counters still move, suspect the endpoint, not the pipelines. Pin this as an upgrade checklist item.
• Remember stats reset on reload. A hot config reload resets pipeline counters, producing artificial dips to zero. Do not confuse a reload dip with a pipeline death; check reloads.successes timestamps.
• Include pipeline isolation in capacity reviews. Queue isolation means one pipeline’s max_bytes is not the whole story. Sum per-pipeline PQ maxima against the shared disk, and sum per-pipeline worker counts against cores.

How Netdata helps

• Per-pipeline series, not just node totals: Netdata collects Logstash node stats and keeps pipeline-level dimensions, so a single stalled pipeline shows as its own series rather than being averaged into a node-wide line.
• Throughput correlation per pipeline: input throughput, output throughput, and queue depth side by side per pipeline makes the “input flowing, output zero, queue growing” signature visible in one view instead of three curl calls.
• Queue backpressure and worker concurrency: per-pipeline flow.* metrics distinguish “this pipeline is throttled because its queue is full” from “this pipeline is starved because the JVM is busy”, which determines whether you fix the pipeline or the node.
• Reload failure visibility: reload successes and failures per pipeline surface configuration drift that otherwise stays invisible until behavior changes.
• Node-level shared resources in the same view: JVM heap, GC time, CPU, and file descriptors alongside per-pipeline data, so simultaneous multi-pipeline degradation is immediately attributable to the shared JVM.
• Anomaly detection per series: per-pipeline ML anomaly scoring catches the “one pipeline deviates from its own baseline” case that static node-wide thresholds structurally miss.

Related guides

• Logstash monitoring checklist: the signals every production pipeline needs
• Logstash monitoring maturity model: from survival to expert
• How Logstash actually works in production: a mental model for operators
• Logstash config reload failed: reloads.failures and invisible configuration drift
• Logstash memory queue vs persistent queue: durability, visibility, and failure modes
• Logstash persistent queue full: max_bytes reached and inputs blocked
• Logstash flow.queue_backpressure: the input-throttling metric explained
• Logstash Kafka input: consumer group lag and rebalances