$ guides / logstash / logstash-pipeline-not-started ▌

Operations Guides

Logstash pipeline not started: a healthy JVM with a missing pipeline

Systemd says Logstash is active. The monitoring API on port 9600 returns 200. Heap looks fine, GC is quiet, and yet one of your data sources has gone silent at the destination. When you query /_node/stats/pipelines, the pipeline ID you expected is not there, or its event counters have not moved in hours.

This is a partial outage that process-level checks cannot see. Since Logstash 7.11, a pipeline that crashes no longer takes the JVM down with it: the process stays alive, serving the API, with fewer pipelines running than you configured. A failed initial config load, a failed reload that stopped the old pipeline without starting the new one, or a worker error that terminated a single pipeline all produce the same shape: healthy JVM, missing pipeline.

This guide covers how to confirm which pipelines are actually running, why they stop, and how to get them back without masking the underlying cause.

What this means

A Logstash process is a container for one or more pipelines. Each pipeline has its own inputs, queue, workers, filters, and outputs, and each has a lifecycle: it loads, it runs, and it can finish or terminate. The JVM can be perfectly healthy while any individual pipeline is dead.

The states that matter:

loading: the pipeline is initializing. Plugins are connecting to external systems, compiling patterns, warming up. Normal for tens of seconds at startup; suspicious if it persists.
running: the pipeline is processing events. The only state that means “working.”
finished: the pipeline completed a finite input (a one-shot file import). Normal for batch jobs, abnormal for streaming pipelines.
terminated: the pipeline died. A worker error, an unrecoverable plugin failure, or a reload that stopped the old pipeline and failed to start the new one.

Two API endpoints tell you which state you are in:

GET /_node/stats/pipelines shows which pipeline IDs exist and their event counters. It is a throughput view. It does not reliably expose lifecycle state, and a terminated pipeline’s entry can linger with stale counters, so presence alone does not prove “running.”
GET /_health_report (Logstash 8.16 and later) reports per-pipeline state explicitly and aggregates an overall status. On supported versions this is the canonical pipeline health check. Prefer it over inferring state from stats.

flowchart TD
  A[JVM up, API 200] --> B{Expected pipeline IDs
in /_node/stats/pipelines?}
  B -- "missing" --> C[Failed initial load:
check logs, pipelines.yml,
central management]
  B -- "present" --> D{/_health_report state
or counters moving?}
  D -- "loading" --> E[Plugin init blocked:
unreachable dependency]
  D -- "terminated / stale counters" --> F[Pipeline crashed or
reload killed it:
check logs + reloads.failures]
  D -- "running, counters moving" --> G[Pipeline fine:
look downstream]

Common causes

Cause	What it looks like	First thing to check
Failed initial config load	Pipeline ID never appears in stats after a restart; JVM healthy	Logstash log for config errors at startup; `pipelines.yml` content
Failed reload that stopped the old pipeline	Pipeline was running, then vanished after a config change; `reloads.failures` incremented	Log timestamps around the config change; `reloads.last_error`
Crashed pipeline (worker/plugin error)	Pipeline present with stale counters, or reported `terminated` by the health report; JVM uptime unaffected	Log for the exception that terminated the pipeline
Empty or misconfigured `pipelines.yml`	JVM starts, zero pipelines running, often on first setup	Log line about the pipelines YAML file; the file itself
Centralized pipeline management misconfiguration	Local `pipelines.yml` and `path.config` are ignored; zero or wrong pipelines running	Whether central management is enabled in `logstash.yml`
Pipeline stuck in `loading`	Health report shows `loading` for minutes; an input plugin pre-connects to an unreachable service	Which plugin initializes first; reachability of its dependency
Port conflict or plugin init failure in multi-pipeline setups	Some pipelines start, others do not; JVM healthy	Log for per-pipeline start failures

Quick checks

All read-only. Run them in order; the first three answer most incidents.

# 1. Is the JVM up and how long has it been running?
systemctl status logstash
curl -sS http://127.0.0.1:9600/_node/stats/jvm?pretty | grep uptime

Short uptime means a recent restart, which reframes the problem: did the pipeline ever come up after this start?

# 2. Which pipeline IDs exist, and are counters moving?
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty

Compare the pipeline IDs in the response against the IDs you expect from pipelines.yml or your deployment manifest. For any pipeline that is present, sample events.in and events.out twice, 30 seconds apart. Counters that do not move on a pipeline that should have traffic mean the pipeline is not doing work, regardless of what the process check says.

# 3. Structured pipeline health (Logstash 8.16+)
curl -sS http://127.0.0.1:9600/_health_report?pretty

The health report gives each pipeline an explicit state (loading, running, finished, terminated) and, for terminated pipelines, a diagnosis such as PIPELINE_TERMINATED with a cause. This is the fastest way to confirm the failure mode on 8.16 and later. On older versions you are inferring state from counter movement and logs.

# 4. Reload state per pipeline
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | grep -A 10 reloads

Any non-zero reloads.failures you have not seen before is a lead. Check reloads.last_error if present.

# 5. What did the log say when the pipeline stopped?
grep -Ei '(pipeline.*terminated|pipeline.*started|failed to execute action|reload)' /var/log/logstash/logstash-plain.log | tail -n 100

# 6. Did the config change recently?
find /etc/logstash -maxdepth 2 -type f -printf '%TY-%Tm-%Td %TT %p\n' | sort | tail

# 7. Is pipelines.yml non-empty and does it define what you expect?
cat /etc/logstash/pipelines.yml

An empty pipelines YAML file produces a JVM with zero pipelines and an explicit error in the log. Also confirm centralized pipeline management is not silently overriding your local files: when it is enabled, pipelines.yml and path.config are ignored.

How to diagnose it

Establish the expected set. List the pipeline IDs that should be running: parse pipelines.yml, your configuration management, or your central management console. You cannot detect a missing pipeline without knowing what “complete” looks like.
Diff expected against actual. Query /_node/stats/pipelines and compare IDs. A missing ID means the pipeline never loaded or was removed. A present ID means go to step 3.
Get the real state. On 8.16+, query /_health_report and read the per-pipeline state and diagnosis. On older versions, take two samples of events.in / events.out 30 to 60 seconds apart for the suspect pipeline. Static counters on a pipeline with active sources is functionally terminated.
Correlate the stop time with events. Use the log to find when the pipeline last logged activity, then look for what happened at that moment: a config file mtime change (quick check 6), a reload attempt, an exception, a restart. Pipeline death almost always has a proximate cause in the log.
Classify the cause. The log line tells you which branch of the cause table you are in: a config compile error points at the config; Failed to execute action with a reload message points at a bad reload; a plugin exception with a worker thread name points at a crash; nothing at all around the stop time combined with a recent restart points at initial load failure.
Check for the stuck-loading variant. If the health report shows loading for more than a couple of minutes, identify which plugin the pipeline initializes first and test its dependency directly. Some inputs (for example, database or Elasticsearch-backed inputs) establish connections before the pipeline is allowed to start, and an unreachable dependency can hold the pipeline in loading indefinitely.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Pipeline ID presence in `/_node/stats/pipelines`	A missing pipeline is invisible to process and aggregate checks	Any expected ID absent from the response
Health report per-pipeline state (8.16+)	Explicit `loading` / `running` / `finished` / `terminated` per pipeline, with diagnoses	Any production pipeline not `running`; `PIPELINE_TERMINATED` diagnosis
`pipelines.<name>.events.out` delta	The real proof a pipeline is doing work	Counter not incrementing while sources are active
`pipelines.<name>.reloads.failures` and `reloads.last_error`	Failed reloads are the most common path to a stopped pipeline and to config drift	Any new non-zero failure count
`jvm.uptime_in_millis`	Short uptime reframes the incident as a startup failure	Uptime far lower than expected between deployments
Aggregate vs per-pipeline throughput	In multi-pipeline setups, one dead pipeline drops aggregate throughput only fractionally, below most alert thresholds	Per-pipeline output at zero while the aggregate looks “a bit low”

The last row is the trap. One failed pipeline out of five drops aggregate throughput by roughly 20 percent. If your alerts are only on aggregate events per second, this incident class pages no one. Per-pipeline presence and state checks are the specific fix.

Fixes

Failed initial config load

Fix the config error reported in the log, then restart Logstash or trigger a reload. Before restarting, validate the config offline if your deployment supports it, so you do not bounce the process into the same failure. If pipelines.yml is empty or points at the wrong path.config, correct it directly.

Failed reload

Two sub-cases, and they need different responses:

Old pipeline still running on stale config. This is the safer failure mode: the reload failed, the previous config kept running, and you have configuration drift rather than an outage. Fix the config and let the next reload apply it. See Logstash config reload failed.
Old pipeline stopped, new one never started. This is the outage case. Reverting the config to the last known-good version and forcing a reload is usually the fastest recovery. Restarting the whole process also works but takes every other pipeline on the node down with it, so prefer the reload if only one pipeline is affected.

Crashed pipeline (terminated)

By default, a crashed pipeline does not restart itself. Your options:

Trigger a config reload. With config.reload.automatic: true, a terminated pipeline is restarted only if its definition changed; a crashed pipeline with no config change stays down. Touching the config to force a reload is a deliberate, if inelegant, recovery path.
Restart the Logstash process. This recovers everything but interrupts all healthy pipelines on the same JVM. Weigh blast radius.
Enable pipeline auto-recovery. Newer 8.x releases add a pipeline.recoverable setting in pipelines.yml with values false (default, never recover), auto (recover only when queue.type: persisted), and true (always recover, with data-loss risk on a memory queue because in-flight events are gone). auto with a persistent queue is the conservative production choice.

Before re-enabling anything, read the exception that killed the pipeline. Auto-recovery on a pipeline that crashes deterministically (a poison event, a broken dependency) just gives you a crash loop.

Pipeline stuck in loading

Fix the dependency the input plugin is waiting on, or correct the plugin configuration pointing at the wrong endpoint. There is no Logstash-side timeout that rescues a pipeline whose plugin blocks during initialization; the pipeline waits as long as the plugin waits.

Centralized management misconfiguration

Confirm whether xpack.management.enabled (or your version’s equivalent) is set in logstash.yml. If central management is on, the local pipelines.yml is documentation, not configuration. Fix the pipeline definitions in the management console. If it was enabled by accident, disable it and restart.

Prevention

Alert on pipeline presence and state, not process liveness. For every expected pipeline ID, assert it appears in /_node/stats/pipelines and, on 8.16+, that the health report says running. This single check covers most of this article.
Monitor per-pipeline, not aggregate, throughput in multi-pipeline deployments. Aggregates average away single-pipeline death.
Alert on reloads.failures so a bad config push is noticed when it happens, not when someone notices missing data.
Consider pipeline.recoverable: auto with persistent queues on versions that support it, so transient crashes self-heal without a human round trip.
Gate config deployments on validation. A reload that stops the old pipeline and fails to start the new one is a self-inflicted outage; catching the config error before rollout is the cheapest fix in this guide.
Retain the Logstash log long enough to matter. The exception that terminated a pipeline is the evidence you need at 3 a.m.; if the log has rotated away, you are guessing.

How Netdata helps

Netdata collects the Logstash monitoring API per pipeline, so a missing pipeline ID or a pipeline whose events.out stops incrementing shows up as a visible gap rather than a diluted aggregate.
Correlating per-pipeline throughput with reloads.failures and JVM uptime on one dashboard collapses the diagnostic sequence above: the reload failure, the pipeline stopping, and the counters going flat appear on the same timeline.
Anomaly detection on per-pipeline output throughput catches the “20 percent aggregate drop” case that static aggregate thresholds miss in multi-pipeline setups.
Alerting on pipeline presence and state closes the gap between “process is up” and “pipeline is working,” which is the entire failure mode of this article.