$ guides / logstash / logstash-input-specific-failure-multi-input ▌

Operations Guides

Logstash one input stopped: per-input failures masked by aggregate metrics

One of three inputs in your Logstash pipeline stops receiving events. The pipeline-level events.in counter drops by a third. But because the other two inputs keep flowing, the aggregate rate never hits zero, and your threshold-based alert stays silent. The failed input’s upstream source starts accumulating: Kafka consumer lag grows, file tails fall behind, or Beats agents buffer locally. By the time someone notices, hours or days of data from that source are delayed or lost.

The Logstash Node Stats API exposes per-input counters and, in recent versions, per-input throughput metrics that make this failure immediately visible. Most monitoring setups collect only pipeline-level aggregates.

What this means

When a single input plugin fails or stalls in a multi-input pipeline, the symptoms are subtle:

Pipeline status shows running. The process is alive. The monitoring API responds normally.
Aggregate events.in continues to grow, just more slowly. Without baseline-relative rate tracking, the partial drop is invisible.
Queue depth stays normal or low because the remaining inputs produce events the pipeline can comfortably process.
Output rate looks healthy. Events are being delivered. No output errors.
No alerts fire because everything is “up.”

The specific data source feeding the failed input is no longer being ingested. Depending on the source type:

Kafka: consumer group lag grows continuously. Data accumulates in Kafka partitions within the retention window, but if retention expires before the input recovers, data is permanently lost.
Beats: agents buffer events in their local memory and spool queue. Once those fill, Beats begins dropping events at the source.
JDBC: scheduled queries stop executing. No database polling occurs, so data accumulates in the source tables unprocessed.
File: file tails stop advancing. Sincedb position freezes, and once the source rotates or purges logs, the missed data is gone.

flowchart TD
    A["Multi-input pipeline"] --> B["Input A: healthy"]
    A --> C["Input B: failed"]
    A --> D["Input C: healthy"]
    B --> E["Aggregate events.in still growing"]
    C --> E
    D --> E
    E --> F["Absolute threshold: not crossed"]
    F --> G["No alert fires"]
    C --> H["Upstream data accumulates
Kafka lag, Beats buffer, files pile up"]

Common causes

Cause	What it looks like	First thing to check
Kafka consumer thread death	One Kafka input shows zero `events.out` while topic partitions have messages. Consumer group may show the Logstash instance as inactive. No error in some cases.	Check consumer group lag on the Kafka broker and `plugins.inputs[].events.out` for the Kafka input.
Beats input port conflict	Beats input fails to bind its listen port because another process (or a previous Logstash instance) holds it. Other inputs start normally.	`ss -tlnp \| grep <port>` and Logstash log for bind errors.
JDBC connection lost	JDBC input stops running scheduled queries. No error log in some cases. Last successful query timestamp is stale.	Check JDBC connection string, database reachability, and `plugins.inputs[].events.out` for the JDBC input.
Credential or certificate expiry	Input fails TLS handshake or authentication. Error appears in logs but pipeline keeps running on other inputs.	`grep -Ei '(SSL\|TLS\|certificate\|handshake\|authentication\|unauthorized)' /var/log/logstash/logstash-plain.log`
Source-side failure	The upstream system stopped sending. Logstash input is healthy but idle because no data arrives.	Check source system health independently: Kafka topic production rate, Beats agent status, database row counts.
Network partition to one source	One input cannot reach its source while others on different network paths are fine.	Check network connectivity to the specific source host and port.

Quick checks

# Check per-input event counters - identify which input stopped
curl -sS http://127.0.0.1:9600/_node/stats/pipelines | python3 -c "
import sys,json
data = json.load(sys.stdin)
for pname, pdata in data.get('pipelines',{}).items():
    inputs = pdata.get('plugins',{}).get('inputs',[])
    for inp in inputs:
        evts = inp.get('events',{})
        print(f\"pipeline={pname} input={inp.get('id','?')} type={inp.get('name','?')} events_out={evts.get('out',0)} failures={evts.get('failures',0)}\")
"

This shows the cumulative events.out counter per input plugin. Take two samples 30 to 60 seconds apart. An input whose counter is not advancing has stopped processing.

# Check per-input throughput (Logstash 8.6+ with flow metrics)
curl -sS http://127.0.0.1:9600/_node/stats/pipelines | python3 -c "
import sys,json
data = json.load(sys.stdin)
for pname, pdata in data.get('pipelines',{}).items():
    inputs = pdata.get('plugins',{}).get('inputs',[])
    for inp in inputs:
        flow = inp.get('flow',{})
        thr = flow.get('throughput',{})
        print(f\"pipeline={pname} input={inp.get('id','?')} throughput_current={thr.get('current','N/A')} throughput_lifetime={thr.get('lifetime','N/A')}\")
"

The flow.throughput metric provides a pre-computed events/second rate per input, eliminating the need to sample counters manually. An input with throughput.current at zero while others show non-zero values is the failed input. This metric is available in Logstash 8.6 and later. On earlier versions, rely on delta sampling of plugins.inputs[].events.out.

# Search logs for errors from the specific input plugin
grep -Ei '(error|exception|failed|timeout|refused|unauthorized)' /var/log/logstash/logstash-plain.log | tail -n 200

# Check TLS/auth failures specifically
grep -Ei '(SSL|TLS|certificate|handshake|authentication|forbidden|unauthorized)' /var/log/logstash/logstash-plain.log | tail -n 200

# Verify pipeline status and aggregate event counts
curl -sS http://127.0.0.1:9600/_node/stats/pipelines | python3 -c "
import sys,json
data = json.load(sys.stdin)
for pname, pdata in data.get('pipelines',{}).items():
    print(f\"pipeline={pname} status={pdata.get('status','?')} events_in={pdata.get('events',{}).get('in',0)}\")
"

How to diagnose it

Identify the stopped input. Pull per-input stats from the Node Stats API and compare events.out counters across inputs. The input with a flat counter is the one that failed. On Logstash 8.6+, use flow.throughput.current for an instant rate comparison without sampling.
Determine whether the failure is in Logstash or upstream. Check the source system independently:
- Kafka: check consumer group lag and whether the Logstash consumer is still a member of the group.
- Beats: check whether agents are connected and sending. If Logstash backpressured the Beats input, agents may have disconnected.
- JDBC: check whether the database is reachable and the connection string is valid.
- File: check whether the source files still exist and are being written to.
Check the Logstash log for that plugin’s errors. The log file is often the only place where plugin-level errors appear. The API gives you counts. The log gives you the cause. Search for the input plugin name or type in the log.
Check whether the input’s thread is alive. Use the hot threads API to inspect thread state. A dead or blocked input thread may not produce an error in the log.

# Check hot threads for blocked or dead input threads
curl -sS 'http://127.0.0.1:9600/_node/hot_threads?threads=20&human=true'

Check config reload state. A failed reload may have changed which inputs are loaded, or left the pipeline running with a stale configuration that no longer includes the affected input.

curl -sS http://127.0.0.1:9600/_node/stats/pipelines | python3 -c "
import sys,json
data = json.load(sys.stdin)
for pname, pdata in data.get('pipelines',{}).items():
    r = pdata.get('reloads',{})
    print(f\"pipeline={pname} reload_failures={r.get('failures',0)} last_error={r.get('last_error','N/A')}\")
"

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Per-input `events.out` counter	Cumulative events pushed by each input. A flat counter means that input stopped.	One input’s counter stops advancing while others continue.
Per-input `flow.throughput.current` (8.6+)	Pre-computed events/second per input. Instant detection without manual sampling.	Zero throughput on one input while others are non-zero.
Per-input `failures` counter	Plugin-level error count. Non-zero values indicate the input encountered errors.	Sudden increase in failures for one input.
Pipeline-level `flow.input_throughput`	Aggregate input rate. Useful as a baseline, but cannot identify which input failed.	Partial drop from baseline without corresponding output or filter change.
Source-side metrics (Kafka lag, Beats queue depth)	The upstream accumulation is the downstream symptom of this failure.	Growing lag or queue depth at the source for one specific data feed.
`reloads.failures` counter	A failed config reload can silently change which inputs are active.	Non-zero failure count with no corresponding investigation.

Fixes

Kafka consumer thread death

Kafka input threads can die without restarting the plugin. This has been observed when a Kafka topic is deleted or a commit fails. The consumer thread dies but Logstash continues running other inputs without restarting the failed one.

Immediate fix: Restart the Logstash pipeline to reinitialize the Kafka consumer. This is one of the cases where a pipeline restart is necessary because the plugin does not self-heal.

Preventive measures:

Set enable_auto_commit => true in the Kafka input configuration so commits happen in the background rather than depending on the consumer thread remaining alive.
Know there is no auto-restart for dead input threads. Logstash has no mechanism to restart a failed input plugin thread; the logstash-input-kafka issue #287 thread documents consumer-thread death (for example after a topic deletion) where the plugin never self-heals and only a pipeline or process restart recovers it. pipeline.recoverable (Logstash 9.4 and later) restarts crashed pipelines, not dead input threads.

Beats input port conflict

If another process binds the Beats listen port before Logstash starts, the Beats input fails to initialize. Other inputs start normally, masking the failure.

Fix: Identify and stop the conflicting process:

# Find what holds the port
ss -tlnp | grep <port>
# Or with lsof
lsof -i :<port>

Then restart Logstash. If the conflict is from a previous Logstash instance that did not shut down cleanly, ensure the process is fully stopped before starting a new one. See Logstash address already in use for deeper coverage of port conflicts.

JDBC connection lost

JDBC input can stop executing scheduled queries without producing error logs. The input appears configured but silently goes idle. The pod or process stays in a healthy state.

Fix: Verify database connectivity from the Logstash host:

# Test raw connectivity to the database
nc -zv <db_host> <db_port>

If connectivity is fine, restart the pipeline to reinitialize the JDBC connection. Review the JDBC input’s schedule and jdbc_connection_string settings. If the database has connection timeouts or idle session limits, the JDBC driver may silently lose its connection without the input noticing.

Credential or certificate expiry

TLS certificate expiry on an input (for example, mutual TLS with a Kafka broker or HTTPS source) causes the connection to fail while other inputs on different trust chains continue.

Fix: Rotate the expired certificate or credential. Check the keystore or truststore configuration:

# Check certificate expiry
keytool -list -v -keystore <path> -storepass <password> | grep -A2 'Valid from'

For credential-based authentication, verify the credential is valid at the source and that the Logstash keystore still holds the correct value. See Logstash certificate expiry for a deeper treatment of this failure mode.

Source-side failure

If the input plugin is healthy but no data is arriving, the problem is upstream. Logstash cannot fix this. Work with the source system owners to restore the data feed. This produces per-input symptoms identical to a Logstash-side failure: a flat events.out counter with no plugin errors.

Prevention

Monitor per-input stats, not just pipeline aggregates. The Node Stats API exposes per-input events.out counters at plugins.inputs[].events.out. On Logstash 8.6+, flow.throughput provides per-input event rates. Alert when any individual input’s throughput drops to zero or deviates significantly from its rolling baseline.
Use baseline-relative alerting, not absolute thresholds. An alert like “events.in below 5000/sec” fires during low-traffic periods and stays silent when one input out of five fails during peak. Instead, alert when an individual input’s rate deviates more than 50% from its rolling average for that time window.
Enable pipeline.separate_logs: true for multi-pipeline deployments. This isolates log output per pipeline, making it easier to correlate log errors with the affected pipeline and its inputs.
Assign explicit id values to every input plugin in your configuration. Without explicit IDs, Logstash auto-generates opaque identifiers that make it difficult to map stats API output back to configuration blocks.
Monitor source-side signals alongside Logstash metrics. Kafka consumer group lag, Beats agent queue depth, and JDBC query timestamps provide the upstream view that Logstash’s internal metrics cannot. Correlating Logstash per-input stats with source-side metrics confirms whether a stopped input is a Logstash problem or a source problem.
Track config reload state. A failed reload can change which inputs are active without your knowledge. Monitor reloads.failures and reloads.last_error to catch configuration changes that silently drop or misconfigure an input.

How Netdata helps

Per-input throughput visibility. Netdata collects per-input plugin stats from the Logstash Node Stats API. A stopped input shows as a flat line on one series while others continue trending.
Correlation with source-side metrics. When a Kafka input stops, Netdata can display the Logstash input throughput and the Kafka consumer group lag side by side. Rising lag with flat input throughput confirms the input failure and quantifies the data accumulation.
Anomaly detection on per-input rates. Netdata’s ML-based anomaly detection flags unusual deviations in individual input throughput, catching partial drops that absolute thresholds miss. This is particularly useful for inputs with variable traffic patterns where static thresholds are noisy.
Log correlation. Netdata surfaces plugin errors from the Logstash log alongside metric anomalies, so you can see the input thread death error in context with the throughput drop on the same timeline.
Config reload monitoring. Reload failures that silently change input configuration are caught by tracking the reloads.failures counter over time, surfacing configuration drift that could cause an input to stop.