Logstash pipeline stalled: output rate at zero while the process looks alive

The Logstash JVM is running. systemctl status logstash says active. The monitoring API on port 9600 returns 200. Every liveness check you have is green. But events.out has not moved in twenty minutes, and data stopped arriving downstream at about the same time.

This is the “living dead” state: the process exists, but the pipeline is functionally dead. It is one of the most common ways Logstash fails in production, and it is invisible to any monitoring that only checks process existence. Output throughput, not liveness, is the health signal.

This article walks through confirming the state, telling the usual causes apart, and recovering without losing more data than necessary.

What this means

Logstash is a queue-backed, multi-threaded batch processor. Inputs push events into a queue, worker threads pull batches through the filter chain, and each worker blocks until its output acknowledges the batch. That design means a stall anywhere downstream of the queue propagates backward: workers block, the queue fills, inputs block, and eventually the whole pipeline produces zero output while the JVM looks perfectly healthy.

The specific signature of this incident:

• pipelines.<name>.events.out (or flow.output_throughput.current) is zero for minutes to hours.
• pipelines.<name>.events.in is non-zero, or was until the queue filled and inputs got backpressured.
• The API responds, systemctl reports active, and process checks pass.

Before treating this as an incident, rule out the two benign lookalikes: an idle server (input rate is also zero, so zero output is correct) and cold start (the first 30-60 seconds after restart have legitimately low throughput while the JVM warms up). Gate any alert or diagnosis on jvm.uptime_in_millis > 300000 and events.in > 0 (or flow.input_throughput.current > 0). Without those gates you will page people for quiet test servers.

flowchart TD
    A[events.out = 0, events.in > 0] --> B{API responsive?}
    B -->|No or very slow| C[GC death spiral or JVM distress]
    B -->|Yes| D{CPU high?}
    D -->|Yes, GC time high| C
    D -->|Yes, GC normal| E[Worker burn: pathological filter]
    D -->|Low| F{Queue full or growing?}
    F -->|Yes| G[Backpressure wedge: workers blocked on output]
    F -->|No| H[Pipeline missing or reload failure]

Common causes

A less common variant is a compute-side stall: a pathological grok pattern (catastrophic backtracking) pins workers at 100% CPU while nothing completes. The diagnostic flow above separates it: high CPU with normal GC points at filter burn, not at output blocking.

Quick checks

All read-only. Run them in order; the first three usually identify the cause class.

# 1. Confirm the stall: pipeline counters and flow rates
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty
# Look at events.in, events.out, events.filtered per pipeline,
# and flow.output_throughput.current / flow.input_throughput.current

# 2. Confirm this is not cold start or an idle box
curl -sS http://127.0.0.1:9600/_node/stats/jvm?pretty
# jvm.uptime_in_millis should be > 300000 before you treat zero output as an incident

# 3. Check pipeline presence and reload state
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | grep -A 10 reloads
# In Logstash 8.x, also:
curl -sS http://127.0.0.1:9600/_health_report?pretty

# 4. Queue state: is backpressure building?
# From the pipelines response: queue.type, queue.events_count,
# queue.queue_size_in_bytes vs queue.max_queue_size_in_bytes (PQ),
# flow.queue_backpressure, flow.worker_utilization

# 5. JVM pressure
# From the jvm response: mem.heap_used_percent, mem.pools.old.*,
# gc.collectors.old.collection_count and collection_time_in_millis

# 6. Process CPU and FDs
curl -sS http://127.0.0.1:9600/_node/stats/process?pretty
# process.cpu.percent, process.open_file_descriptors vs max_file_descriptors

# 7. What are threads actually doing (take 2-3 samples a few seconds apart)
curl -sS 'http://127.0.0.1:9600/_node/hot_threads?pretty'

# 8. Output-side errors in the log
grep -Ei '(retry|error|exception|failed|reject|unavailable|timeout|429|503)' \
  /var/log/logstash/logstash-plain.log | tail -n 200

# 9. Recent reload or shutdown-watcher messages
grep -Ei '(reload|reloading|stalled|shutdown)' \
  /var/log/logstash/logstash-plain.log | tail -n 100

If the API itself is unresponsive or extremely slow on a running process, that is already a strong signal: the JVM is in severe distress, most often a GC death spiral. jstack $(pgrep -f org.logstash.Logstash) still works in that state when the Logstash API does not.

How to diagnose it

1. Verify the signature, not the vibe. Pull /_node/stats/pipelines twice, 60 seconds apart. Confirm the events.out delta is zero (or near zero) while the box has uptime over 300 seconds and events.in was recently non-zero. Note whether events.in is still arriving or has also collapsed; if input has dropped to zero too, the queue is likely full and inputs are blocked, which points downstream.

2. Classify by CPU and GC. From /_node/stats/jvm and /_node/stats/process: if CPU is high and old-gen collection_time_in_millis is consuming a large fraction of wall time (compute delta(collection_time) / delta(wall_time); over 20% is severe), you are in GC collapse. If CPU is low and the queue is full or growing, workers are waiting on I/O, which is the backpressure wedge. If CPU is high with normal GC, suspect a pathological filter.

3. Look at the workers. Take two or three /_node/hot_threads samples a few seconds apart. Workers parked in output plugin code (connection waits, retry loops) confirm output blocking. Workers burning in regex or filter code confirm compute stall. A single snapshot can mislead; persistent state across samples is what matters.

4. Check the output side directly. Grep the log for retries, 429s, timeouts, auth and TLS failures. Verify the destination independently of Logstash (for Elasticsearch, _cluster/health; for other outputs, whatever health check applies). If the destination has been down or rejecting, you have your root cause and the queue occupancy tells you how much runway is left: (max_queue_size_in_bytes - queue_size_in_bytes) / current fill rate.

5. Rule out the reload failure. If queue is not full, CPU is normal, GC is normal, and hot threads show nothing interesting, check whether the pipeline you think is running is actually there. Compare the pipeline IDs in /_node/stats/pipelines against pipelines.yml, and check reloads.failures and reloads.last_error. A failed reload normally keeps the old pipeline running, but a reload attempted while outputs were blocked can leave the pipeline wedged: still registered, processing nothing, and refusing subsequent reloads. The log will show shutdown or stalled-reload messages around the time throughput stopped.

Metrics and signals to monitor

Fixes

Match the fix to the cause class. Restarting first destroys the evidence and, in the backpressure case, does not fix anything.

Downstream outage or rejection (backpressure wedge)

Fix the destination, not Logstash. Restore Elasticsearch health, rotate the expired credential, clear the network fault. Once the output recovers, a healthy Logstash drains the queue on its own; monitor the drain rate rather than intervening. If the queue is near full and the destination will not recover soon, reduce incoming load or shed non-critical pipelines to buy runway.

One important exception: there are known scenarios, especially with a full persistent queue plus a blocked output, where Logstash does not resume cleanly even after the downstream recovers. Workers sit in the output plugin’s reconnect loop indefinitely. If the destination has been healthy for several minutes and events.out is still zero with workers parked in hot threads, a restart is the remaining option.

GC death spiral

This is the one case where restarting promptly is correct: a JVM deep in a death spiral will not recover on its own. Before restarting, capture /_node/stats/jvm and, if possible, hot threads for forensics. If the memory queue is in use, events in the queue are lost on restart; with PQ they survive. After the restart, address the root cause: raise -Xms/-Xmx (set them equal) if the heap is undersized, reduce pipeline.batch.size or worker count to shrink in-flight events, and hunt the leak if the post-GC floor was ratcheting. Enabling -XX:+HeapDumpOnOutOfMemoryError gives you evidence if it recurs.

Workers blocked on output with no downstream fault

If the destination is healthy but workers are stuck (connection pool exhaustion, a plugin-level deadlock, a DNS lookup with no timeout), hot threads tells you what they are waiting on. Resolve that specific wait: resize the connection pool, add timeouts to DNS or HTTP-dependent filters, or split the pipeline to reduce contention. Increasing pipeline.workers only helps when the box has CPU headroom and workers are compute-bound; it makes an I/O-blocked pipeline worse by queueing more waiters.

Failed reload left the pipeline wedged

Fix the config error named in reloads.last_error first. Then check whether the pipeline is present and processing. If the pipeline is absent or registered but dead after the failed reload, a restart is usually required to get back to a clean state; reload attempts against a wedged pipeline tend to keep failing. Afterward, verify reloads.successes increments and throughput resumes, and confirm the running config matches what you deployed.

Prevention

• Alert on output throughput, gated correctly. Zero output for > 5 minutes with events.in > 0 (or flow.input_throughput.current > 0) and jvm.uptime_in_millis > 300000. This one rule catches every cause class in this article and does not fire on idle servers or restarts.
• Watch the wedge forming, not just the stall. flow.queue_backpressure, queue occupancy trend, and output retry activity rise minutes to hours before output hits zero.
• Monitor reload state. Any new reloads.failures is a ticket. Failed reloads create invisible config drift and are the precondition for the wedged-pipeline case.
• Track the post-GC heap floor and old-gen GC rate, not raw heap percentage. Raw percentage alert-fires on every normal sawtooth peak and gets silenced before the real event.
• Size the persistent queue deliberately and monitor runway: flow.queue_persisted_growth_bytes tells you the fill rate directly, so “how long until full” is a computed answer, not a guess.
• Assign explicit IDs to plugins in pipeline configs. When a stall happens, per-plugin stats and hot-thread output map back to config lines instead of opaque generated IDs.
• In multi-pipeline deployments, alert per pipeline. One dead pipeline out of five drops aggregate throughput 20%, which hides under most thresholds.

How Netdata helps

• Netdata collects the Logstash node stats directly, so events.in, events.out, and flow rates are graphed per pipeline at high resolution; the living-dead signature (out flat at zero, in non-zero) is visible at a glance instead of requiring two manual API samples.
• Queue occupancy, queue_backpressure, and worker_utilization are charted together, which separates the backpressure wedge (queue growing, CPU low) from compute stalls (CPU high) without running hot threads first.
• JVM heap pools and GC collector times are collected alongside pipeline stats, so a GC death spiral shows up as rising old-gen time and a climbing post-GC floor on the same dashboard as the throughput collapse.
• Reload success and failure counters are tracked over time, surfacing the failed-reload wedge and config drift that otherwise only appear when someone greps the API by hand.
• Because Netdata also monitors the host and common downstreams, output stalls can be correlated with Elasticsearch health, disk pressure on PQ volumes, and network errors in one view, which is the correlation this failure mode demands.

Related guides

• Logstash monitoring checklist: the signals every production pipeline needs
• Logstash monitoring maturity model: from survival to expert
• Logstash queue full: inputs blocked and the backpressure wedge