Logstash input rate dropped to zero: upstream failure vs blocked inputs

The Logstash input rate metric (flow.input_throughput or the events.in counter delta) has dropped to zero. Before restarting anything, answer one question: is the queue empty or full?

An input rate of zero with an empty queue means events are not arriving from upstream. An input rate of zero with a queue at capacity means Logstash has applied backpressure to its own inputs because it cannot drain events fast enough. These conditions require opposite responses, and treating one as the other wastes time during an incident.

One subtlety: with persistent queues enabled, events.in counts events written to the queue, not raw network receipt. If the PQ disk is full, the rate drops to zero even while traffic is still arriving at the network layer. The input plugin may be receiving bytes, but Logstash cannot enqueue them, so the counter does not advance.

Input plugins run in their own threads, receive raw bytes from external sources, pass them through a codec (line, json, multiline), and push Event objects into the central queue. events.in increments when an event enters the queue. flow.input_throughput is the pre-computed derivative of that counter.

Before treating a zero input rate as an incident, gate against the idle-server false positive. If no traffic should be flowing, a zero rate is correct behavior.

Common causes

Quick checks

All commands are read-only and safe to run during an incident.

# Check input throughput, output throughput, queue state, and backpressure together
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | python3 -c "
import sys, json
data = json.load(sys.stdin)['pipelines']
for name, s in data.items():
    flow = s.get('flow', {})
    q = s.get('queue', {})
    it = flow.get('input_throughput', {}).get('current', 'N/A')
    ot = flow.get('output_throughput', {}).get('current', 'N/A')
    qb = flow.get('queue_backpressure', {}).get('current', 'N/A')
    wu = flow.get('worker_utilization', {}).get('current', 'N/A')
    ec = q.get('events_count', 'N/A')
    qt = q.get('type', 'N/A')
    print(f'{name}: in={it} out={ot} backpressure={qb} workers={wu} queue_events={ec} queue_type={qt}')
    if q.get('max_queue_size_in_bytes'):
        used = q.get('queue_size_in_bytes', 0)
        mx = q['max_queue_size_in_bytes']
        print(f'  PQ: {used} / {mx} bytes ({used/mx*100:.1f}%)')
"

# Check per-input event counts to isolate a single failed input
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | python3 -c "
import sys, json
data = json.load(sys.stdin)['pipelines']
for name, s in data.items():
    for inp in s.get('plugins', {}).get('inputs', []):
        ev = inp.get('events', {})
        print(f\"{name}/{inp.get('id','?')} ({inp.get('name','?')}): out={ev.get('out', 'N/A')}\")
"

# Check JVM uptime to gate against cold-start false positives
curl -sS http://127.0.0.1:9600/_node/stats/jvm?pretty | python3 -c "
import sys, json
up = json.load(sys.stdin)['jvm']['uptime_in_millis']
print(f'Uptime: {up // 3600000}h {(up % 3600000) // 60000}m ({up} ms)')
"

# Check disk space on the PQ partition (default path shown; adjust for your install)
df -h /var/lib/logstash

# Check for output errors in the Logstash log
grep -Ei '(retry|error|exception|failed|reject|unavailable|timeout|429|503)' /var/log/logstash/logstash-plain.log | tail -n 100

How to diagnose it

The decision flow is built around queue state. Every step assumes you have confirmed the API is reachable and the process is running.

flowchart TD
    A["input_throughput = 0"] --> B{"Queue state?"}
    B -->|"Empty or draining"| C["UPSTREAM FAILURE"]
    B -->|"At capacity"| D["BACKPRESSURE"]
    C --> C1["Check source health"]
    C --> C2["Check network path"]
    C --> C3["Check per-input stats"]
    D --> D1["Output errors or retries?"]
    D --> D2["Worker CPU saturated?"]
    D --> D3["PQ disk full?"]

Step 1: Gate against false positives.

Check JVM uptime. If it is under 300 seconds, the pipeline may still be warming up. JIT compilation, filter compilation, and PQ replay all cause temporary throughput dips during startup. Wait for uptime to exceed 5 minutes before treating a zero rate as an incident.

If uptime is healthy, confirm that traffic should be flowing. An idle server with no active sources sending data is not a failure. Cross-reference with the source system or upstream monitoring to confirm events are being generated and sent.

Step 2: Read queue state.

Query the pipeline stats API and look at queue.events_count alongside queue capacity.

• Queue empty or draining (events_count trending toward zero, output throughput positive): the problem is upstream. Logstash is ready to process but no events are arriving.
• Queue at or near capacity (events_count high, or queue_size_in_bytes approaching max_queue_size_in_bytes for PQ): the problem is downstream backpressure. Inputs are blocked because the queue cannot accept more events.

Step 3a: Queue empty – investigate upstream.

Check the source system directly. For Beats inputs, verify Beats agents are running and connected. For Kafka inputs, check consumer group lag and partition assignment. For file inputs, check whether the source files exist and are growing. For JDBC inputs, check database connectivity.

Check the network path. A firewall rule change, DNS failure, or routing problem can silently sever the connection without killing the input plugin.

Check per-input stats. In a multi-input pipeline, one input may have failed while others continue. The aggregate events.in drops proportionally, masking the failed input. Query plugins.inputs[].events.out per input to identify which one is flat.

Check the Logstash log for input plugin errors: connection refused, authentication failure, TLS handshake errors, or plugin exceptions. All appear in /var/log/logstash/logstash-plain.log.

Step 3b: Queue full – investigate backpressure.

Determine whether the cause is output failure, worker saturation, or PQ disk exhaustion.

Output failure: check output plugin error counts and logs. Look for HTTP 429 (bulk rejection), 503, timeouts, connection refused, or TLS errors. The downstream system (typically Elasticsearch) may be under disk pressure, experiencing mapping conflicts, or rejecting bulk requests due to thread pool saturation.

Worker saturation: check flow.worker_utilization. If it is near 100% sustained, workers are CPU-bound. Check per-plugin stats to identify which filter is consuming the most time. This is the “grok hell” pattern: expensive regex evaluation consumes all worker capacity, the queue fills, and inputs block.

PQ disk exhaustion: check queue_size_in_bytes against max_queue_size_in_bytes, and check actual disk capacity with df. If the PQ partition is full from PQ pages, DLQ growth, or competing log files, Logstash cannot write new queue pages and inputs block. This can happen even when the PQ is within its configured byte limit, if the disk itself is exhausted by other consumers.

Memory queue: memory queue capacity is small by design and not exposed as a byte ratio. Hitting capacity happens quickly and blocks inputs immediately.

Metrics and signals to monitor

Fixes

Upstream failure

Source system down. Fix the source. There is nothing to change in Logstash. Verify connectivity restored by watching events.in resume.

Network partition. Identify and fix the network path. Check firewall rules, DNS resolution, routing tables, and intermediate load balancers. For Beats inputs behind a load balancer, check LB health check configuration: an overly aggressive health check can deregister Logstash during slow startups.

Input plugin failure. Check the log for the specific error. Common causes include expired TLS certificates, credential rotation failures, and plugin bugs under specific conditions. For Kafka inputs, consumer group rebalancing can cause temporary input drops. For file inputs, sincedb corruption can cause Logstash to believe it has already read all available data.

Port conflict. If an input plugin cannot bind its listen port (Beats, TCP, or HTTP inputs), the input fails to start. Check ss -tlnp for the expected port and verify no other process holds it.

Backpressure from output failure

Downstream unavailable or slow. Fix the downstream system. For Elasticsearch, check cluster health, bulk rejection rate, and disk watermarks. Logstash self-heals once the destination recovers, but monitor the queue drain rate. If the queue was very full, drain may take significantly longer than fill.

Reduce incoming load. If downstream recovery time exceeds queue runway, shed non-critical traffic before the queue fills completely. Stop specific inputs, pause Kafka consumers, or route lower-priority pipelines to an alternate sink. Doing this early preserves runway. Doing it after the queue is already full buys nothing.

Backpressure from worker saturation

Increase workers (if CPU headroom exists). Raise pipeline.workers if host CPU has spare capacity. More workers help only until CPU cores, lock contention, or downstream blocking becomes the limit.

Optimize expensive filters. If per-plugin stats show one filter dominating processing time (typically grok), optimize the pattern, switch to dissect for simple formats, or cache external lookups. See the related guide on CPU-bound filters.

PQ disk exhaustion

Free disk space. Remove old PQ pages, DLQ entries, or log files from the shared partition. The PQ does not release disk space immediately when events are consumed. Pages are freed only when fully drained and checkpointed, so there is a lag between events leaving the queue and disk space recovering.

Increase PQ max_bytes. If the PQ is genuinely undersized for expected outage durations, increase queue.max_bytes in logstash.yml. Ensure the disk partition has room for the new limit.

Isolate PQ on its own partition. If PQ, DLQ, and logs share a partition, they compete for space. Moving PQ to a dedicated volume prevents log storms or DLQ growth from exhausting queue capacity.

Stuck pipeline after PQ full

A known issue exists where Logstash becomes stuck after the PQ fills and does not resume sending to output even after downstream recovers. If output errors have stopped but throughput remains zero and the queue stays full, a pipeline restart may be necessary. This is one of the few cases where a restart is the correct immediate action. Back up the PQ directory first if data preservation is critical.

Prevention

Alert on queue state, not just input rate. A zero input rate alone is ambiguous. Alert on the composite condition: input rate zero AND queue empty signals upstream failure. Input rate zero AND queue at capacity signals backpressure. These are different incidents requiring different responders.

Monitor PQ runway. Calculate time-to-full from the current fill rate: (max_queue_size_in_bytes - queue_size_in_bytes) / current_fill_rate_bytes_per_second. Alert before runway drops below 30 minutes, not after the queue is already full.

Gate alerts against uptime. Any throughput-based alert should include a condition on jvm.uptime_in_millis > 300000 to suppress cold-start noise from JIT compilation and filter warmup.

Gate alerts against expected traffic. Use baseline-relative thresholds (percent deviation from rolling average) rather than absolute events-per-second thresholds. An absolute threshold fires continuously during off-hours and never fires after traffic doubles.

Monitor per-input stats in multi-input pipelines. Aggregate events.in masks a single failed input. Per-input monitoring catches partial failures that aggregate thresholds miss.

How Netdata helps

• Per-second input and output throughput lets you see the exact moment input rate drops and correlate it with output rate, queue depth, and worker utilization in the same time window.
• Queue occupancy and backpressure metrics (flow.queue_backpressure, queue.events_count, PQ byte ratio) provide the empty-vs-full disambiguation without manual curl commands during an incident.
• ML-based anomaly detection on input throughput establishes baseline-relative expectations, separating idle-server false positives and cold-start noise from genuine deviations without static thresholds.
• Per-pipeline visibility isolates which pipeline has the stalled input in multi-pipeline deployments, preventing aggregate metrics from masking a single failed pipeline.
• Disk space monitoring on the PQ partition catches PQ disk exhaustion before it blocks inputs, including space consumed by DLQ and log files sharing the partition.
• Correlated JVM, CPU, and output error signals in a single view shorten the path from “input rate is zero” to “because the Elasticsearch output is returning 429s and the queue has 8 minutes of runway left.”

Related guides

• Logstash Beats input: Filebeat backpressure and connection health
• Logstash CPU-bound filters (grok hell): high CPU, saturated workers, growing queue
• Logstash disk full: PQ, DLQ, and log volumes competing for space
• Logstash dead letter queue growing: DLQ diversion, replay, and disabled-by-default risk
• Logstash API unreachable on port 9600: crash, GC pause, or startup
• Logstash address already in use: input port conflicts on Beats, TCP, and HTTP