Logstash downstream backpressure cascade: when a slow output stalls the whole pipeline

Logstash is up, the API returns 200, and your dashboards are going dark. Beats agents are buffering, Kafka consumer lag is climbing, and throughput collapsed while CPU sits at 15%. The process looks healthy from the outside. It is not.

A slow or failing output (Elasticsearch, Kafka, HTTP endpoint, syslog receiver) raises output duration. Worker threads block waiting for acknowledgment. The queue fills. The persistent queue absorbs the gap for a while, sometimes hours. When it hits max_bytes, inputs are blocked. Upstream systems back up or drop events.

The defining signature is low CPU with a growing queue and output errors. This distinguishes it from CPU-bound filter saturation (grok hell), where high CPU is the dominant symptom. Confusing the two leads to the wrong fix: adding workers or CPU to a pipeline whose real problem is downstream.

What this means

The cascade follows a predictable propagation path. Each stage blocks the one before it, and the queue is the shock absorber that delays the visible failure.

flowchart TD
    A[Output slows or rejects] --> B[Output duration rises]
    B --> C[Workers block on acknowledgment]
    C --> D[Queue consumption stalls]
    D --> E[Queue grows]
    E --> F{PQ enabled?}
    F -->|Yes| G[PQ absorbs for hours]
    G --> H[PQ hits max_bytes]
    H --> I[Inputs blocked]
    F -->|No, memory queue| I
    I --> J[Beats/Kafka back up]
    J --> K[Upstream monitoring goes dark]

Output duration rises first, before throughput drops. Workers block next, still before throughput visibly collapses. The queue then grows, and if a persistent queue is enabled, it can mask the problem for hours. Only when the queue fills do inputs finally block, and that is when upstream systems start failing in ways operators notice.

This delay is both a feature and a trap. The PQ buys time to fix the downstream issue. But if nobody is watching the fill rate, that time runs out silently.

Workers blocked on output I/O are not computing. They are waiting. CPU stays low because the bottleneck is network or destination latency, not filter processing. If your first instinct on seeing a growing queue is to add more workers or CPU, you will waste time and make nothing better.

Common causes

Quick checks

These are safe, read-only commands. Run them in order during the first minutes of a suspected cascade.

# Check API responsiveness and basic liveness
curl -sS --connect-timeout 5 http://127.0.0.1:9600/

# Check pipeline throughput and queue state
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | \
  python3 -c "
import sys, json
d = json.load(sys.stdin)
for name, p in d.get('pipelines', {}).items():
    ev = p.get('events', {})
    q = p.get('queue', {})
    flow = p.get('flow', {})
    print(f'Pipeline: {name}')
    print(f'  events.in={ev.get(\"in\",0)} out={ev.get(\"out\",0)} filtered={ev.get(\"filtered\",0)}')
    print(f'  queue.events_count={q.get(\"events_count\",0)} type={q.get(\"type\",\"?\")}')
    if q.get('type') == 'persisted':
        used = q.get('queue_size_in_bytes', 0)
        mx = q.get('max_queue_size_in_bytes', 1)
        print(f'  PQ: {used/1024/1024:.0f}MB / {mx/1024/1024:.0f}MB ({used/mx*100:.1f}%)')
    ot = flow.get('output_throughput', {})
    it = flow.get('input_throughput', {})
    print(f'  flow.input_throughput={it.get(\"current\",\"N/A\")} output_throughput={ot.get(\"current\",\"N/A\")}')
"

# Check per-output plugin duration and errors
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | \
  python3 -c "
import sys, json
d = json.load(sys.stdin)
for name, p in d.get('pipelines', {}).items():
    for o in p.get('plugins', {}).get('outputs', []):
        ev = o.get('events', {})
        dur = ev.get('duration_in_millis', 0)
        out = ev.get('out', 1)
        print(f'{o.get(\"name\",\"?\")} ({o.get(\"id\",\"?\")}): {dur}ms total, {out} events, {dur/max(out,1):.2f} ms/event')
"

# Capture hot threads to confirm output-wait pattern
curl -sS 'http://127.0.0.1:9600/_node/hot_threads?threads=10&human=true'

# Check JVM stats for GC interference
curl -sS http://127.0.0.1:9600/_node/stats/jvm?pretty | \
  python3 -c "
import sys, json
j = json.load(sys.stdin)['jvm']
gc = j['gc']['collectors']
for gen, s in gc.items():
    print(f'{gen}: {s[\"collection_count\"]} collections, {s[\"collection_time_in_millis\"]}ms total')
print(f'heap_used_percent: {j[\"mem\"][\"heap_used_percent\"]}')
"

# Check for output errors in the log
grep -Ei '(retry|error|exception|failed|reject|unavailable|timeout|429|503)' \
  /var/log/logstash/logstash-plain.log | tail -n 200

# Check PQ disk space (if PQ enabled)
df -h /var/lib/logstash

How to diagnose it

1. Confirm the pattern. You are looking for three things together: output rate below input rate, queue growing, and CPU low relative to worker count. If CPU is high and there are no output errors, you are dealing with CPU-bound filter saturation, not a downstream cascade. See the related guide on grok hell for that path.

2. Verify destination health. Check the downstream system independently. For Elasticsearch, query _cluster/health and _cat/thread_pool/write. For Kafka, check broker status. If the destination is impaired, that is your root cause. Logstash is the victim, not the culprit.

3. Estimate PQ runway. Calculate how long until the queue fills: (max_queue_size_in_bytes - queue_size_in_bytes) / current_fill_rate_bytes_per_second. If the growth rate is measured in megabytes per minute and you have gigabytes of headroom, you have hours. If the fill rate exceeds available space by large margins, you may have minutes.

4. Identify which output is blocking. In multi-output pipelines, one slow output blocks all others by design. Logstash guarantees at-least-once delivery, so workers cannot skip a blocked output and send to a healthy one. Check per-output plugin duration_in_millis to find the culprit.

5. Check for auth or TLS failures. Grep the log for SSL, TLS, certificate, handshake, and authentication errors. An expired certificate on the output side produces the same cascade as a slow destination, but the fix is cert renewal, not capacity.

6. Assess upstream impact. Check Beats agent buffers, Kafka consumer group lag, or any upstream source that depends on Logstash accepting input. If inputs are already blocked, upstream data is at risk.

7. Take multiple hot threads samples. A single snapshot is noisy. Take 2-3 samples a few seconds apart. Workers stuck in output wait show thread states like BLOCKED or TIMED_WAITING with traces through output plugin code, not filter or regex code.

Metrics and signals to monitor

Fixes

Fix the downstream root cause

This is almost always the correct first action. The cascade is a symptom. If Elasticsearch is rejecting bulk requests, scale the cluster or address thread pool pressure. If a network path is degraded, fix routing or connectivity. If a certificate expired, renew it. Logstash will self-heal once the destination recovers, and the PQ will drain naturally as events are acknowledged.

Estimate runway before the queue fills

If the downstream fix will take longer than the PQ runway, you need to buy time. The PQ sizing formula from Elastic documentation is: Required Queue Capacity = (Bytes Received Per Hour * Tolerated Hours of Downtime) * 1.10. If your existing PQ is undersized for this outage duration, inputs will block before the fix lands.

Shed non-critical load

If runway is short, reduce incoming traffic before the queue fills. Options:

• Temporarily stop or throttle non-critical input sources. Stop Beats agents on low-priority hosts, or pause Kafka consumers for non-critical topics. This extends runway for critical pipelines.
• Isolate critical pipelines. In multi-pipeline deployments, you can stop less important pipelines to free worker threads and queue capacity for the ones that matter.
• Reduce batch size or worker count. This does not fix the cascade, but it can reduce memory pressure if the system is also approaching heap limits. Use cautiously.

Address multi-output blocking

In a pipeline with multiple outputs, one slow output blocks all others. This is by design for at-least-once delivery. The workaround is the pipeline-to-bus (isolator) pattern: route events to separate downstream pipelines, each with its own persistent queue. One slow destination no longer blocks delivery to healthy ones.

Note a limitation: if any downstream pipeline’s PQ fills, backpressure still propagates upstream. The isolator buys time and isolation, but it does not eliminate the cascade under sustained downstream failure.

Watch for the stuck-after-recovery edge case

A known issue (GitHub #14740, filed against Logstash 7.15.2) describes Logstash getting permanently stuck when the PQ fills and the output remains blocked. Even after the downstream recovers, Logstash may not resume processing. The issue is intermittent. If your pipeline does not self-heal after the destination comes back, a restart may be necessary.

Prevention

Monitor PQ runway, not just occupancy. Alerting on PQ at 90% full is late. Alert on the fill rate trend. Track the delta of queue_size_in_bytes over time. Calculate estimated time-to-full and alert when it drops below your team’s response threshold (for example, 30 minutes).

Set baseline-relative throughput alerts. Absolute thresholds fail when workload changes. Compare flow.output_throughput.current against a rolling baseline for the same time window. Alert when output rate drops significantly while input rate remains positive.

Correlate CPU with queue growth. The single most useful diagnostic correlation is: queue growing + low CPU = downstream problem. Queue growing + high CPU = compute problem. Build this into your alerting or dashboard so the distinction is visible at a glance.

Size the PQ for realistic outage durations. The PQ should be large enough to absorb the longest downstream outage your team expects to survive without data loss. If your Elasticsearch cluster takes 2 hours to recover from a worst-case event, the PQ needs to hold 2+ hours of data. The default 1GB max_bytes is often too small for production.

Monitor destination health alongside Logstash. The cascade crosses system boundaries. Elasticsearch cluster health, bulk rejection rates, and thread pool saturation are leading indicators for Logstash output problems. Monitoring only the Logstash side means you see the symptom but not the cause.

Use per-plugin stats. Aggregate pipeline metrics hide which output is blocking. In multi-output pipelines, always monitor per-output duration_in_millis and error counts individually.

How Netdata helps

The cascade develops in stages, and the transitions between stages can happen faster than typical 60-second polling intervals capture.

• Correlate output throughput, queue depth, and CPU in a single view. The low-CPU-with-growing-queue signature is immediately visible when these metrics share the same timeline, without switching between dashboards.
• Track PQ fill rate at per-second resolution. Finer granularity gives a more accurate growth trend for predicting time-to-full before occupancy crosses a static threshold.
• Per-pipeline and per-plugin breakdowns. In multi-pipeline or multi-output deployments, Netdata surfaces which specific output is blocking, not just that something is slow.
• Anomaly-aware baselines on flow metrics. queue_backpressure, worker_concurrency, and output_throughput baselines are workload-dependent. Netdata learns each pipeline’s normal pattern and flags deviations without manually tuned thresholds that drift.
• JVM and OS-level signals in context. Heap usage, GC time, CPU throttling (in containers), and disk I/O on the PQ volume alongside pipeline metrics eliminate the need to correlate across separate tools during an incident.

Related guides

• Logstash CPU-bound filters (grok hell): high CPU, saturated workers, growing queue
• Logstash disk full: PQ, DLQ, and log volumes competing for space
• Logstash Beats input: Filebeat backpressure and connection health
• Logstash API unreachable on port 9600: crash, GC pause, or startup
• Logstash certificate expiry: the silent, total outage no built-in metric shows
• Logstash config reload failed: reloads.failures and invisible configuration drift
• Logstash file descriptor pressure: leaks, tailed files, and reconnection churn