Logstash flow.queue_backpressure: the input-throttling metric explained

You opened the Logstash node stats API during an incident, found flow.queue_backpressure sitting at 0.9, and now you need to know what that number is actually telling you. The short answer: it is the fraction of time your input threads spend blocked trying to push events into the pipeline queue. It measures ingestion throttling, not worker-side slowness, and that distinction drives the entire diagnosis.

This article explains what the metric counts, how it is derived, why its absolute value is less meaningful than its movement, and how to combine it with worker utilization and queue depth to decide whether you are looking at a downstream outage, a CPU-bound filter chain, or simple capacity exhaustion.

For the broader pipeline architecture this metric sits inside, see How Logstash actually works in production.

What flow.queue_backpressure measures

Every Logstash input plugin runs in its own thread. That thread receives events, runs them through a codec, and pushes the resulting Event objects into the pipeline’s central queue. Worker threads pull batches off the other side and run them through filters and outputs.

When workers cannot drain the queue as fast as inputs fill it, the queue fills up. Once the queue has no room, the input thread’s push operation blocks. The input is still connected, still receiving data, but it cannot hand events off. That blocked time is exactly what flow.queue_backpressure captures: the share of wall-clock time input threads spend stuck on the queue push.

Two consequences follow:

• It is an input-side signal. A high value tells you ingestion is being throttled. It says nothing directly about why the queue is not draining. That is a separate question you answer with other metrics.
• It is the leading edge of the backpressure wedge. Rising queue backpressure is what happens right before upstream systems (Beats agents, Kafka consumer lag, TCP send buffers) start absorbing the stall. If your inputs have no acknowledgement or buffering mechanism, this is also the point where data loss begins upstream.

How it is computed

The metric is derived from a cumulative counter the pipeline has always exposed: events.queue_push_duration_in_millis. That counter totals the milliseconds input threads have spent pushing events into the queue, including time blocked waiting for space. The flow metric divides that push time by wall-clock time, giving a unitless ratio instead of a raw counter you would have to differentiate yourself.

flowchart LR
    A[Input threads] -->|push events| B[Queue]
    B -->|pull batches| C[Worker threads]
    C --> D[Filters]
    D --> E[Outputs]
    E -->|slow or failing| F[Downstream]
    B -.->|no room: push blocks| A
    B --- G[flow.queue_backpressure
= blocked push time / wall time]

The metric path is pipelines.<name>.flow.queue_backpressure in the node stats API:

# Read queue backpressure and its neighbors for every pipeline
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty

You will find it under each pipeline’s flow object alongside input_throughput, output_throughput, and worker_utilization. On multi-pipeline deployments, read it per pipeline. Aggregates hide the one pipeline that is actually wedged.

Version availability and time windows

Flow metrics, including queue_backpressure, were introduced in Logstash 8.5.0 and do not exist on 7.x. If you are on 7.x, the only way to approximate the same signal is to sample events.queue_push_duration_in_millis twice and compare its delta against elapsed wall time:

# Manual backpressure estimate on versions without flow metrics
Q1=$(curl -sS http://127.0.0.1:9600/_node/stats/pipelines/main | jq '.pipelines.main.events.queue_push_duration_in_millis')
T1=$(date +%s%3N)
sleep 30
Q2=$(curl -sS http://127.0.0.1:9600/_node/stats/pipelines/main | jq '.pipelines.main.events.queue_push_duration_in_millis')
T2=$(date +%s%3N)
echo "backpressure ~= $(( (Q2 - Q1) * 100 / (T2 - T1) ))% of wall time"

The flow object reports each metric across several rolling windows: current (roughly the last 10 seconds), last_1_minute, last_5_minutes, last_15_minutes, last_1_hour, last_24_hours, and lifetime. Operationally:

• Use current and last_1_minute during an active incident to see what is happening now.
• Use last_5_minutes or last_15_minutes to separate a sustained wedge from a bursty blip.
• Treat lifetime with suspicion. It is diluted by every quiet hour since the process started, and the underlying counter resets on pipeline reload, so a mid-incident config reload can make lifetime misleading. The short windows recover within seconds; lifetime does not.

Do not poll the stats API faster than about every 10 seconds on a loaded instance. The API shares the JVM with the pipeline.

How to read it during an incident

The metric’s value comes from what you correlate it with. A high backpressure reading always means inputs are throttled, but the correct response depends entirely on the drain side. The three patterns that matter:

The second row is the classic downstream backpressure cascade: Elasticsearch slows down or starts rejecting bulk requests, workers block on output acknowledgement, the queue stops draining, and the blockage propagates back to the inputs. Host CPU is usually low in this mode because the workers are waiting, not computing. If that is what you see, the fix is downstream, not in Logstash. The full failure cascade is covered in Logstash queue full: inputs blocked and the backpressure wedge.

The third row is the compute bottleneck: grok backtracking, heavy JSON parsing, a Ruby filter. Workers are pegged and the queue grows behind them. CPU is high and output health is fine.

If you cannot tell which mode you are in, hot threads settles it quickly:

# See whether workers are burning CPU in filters or parked on output I/O
curl -sS 'http://127.0.0.1:9600/_node/hot_threads?pretty'

Take two or three samples a few seconds apart. Threads consistently inside output plugin code point downstream. Threads consistently inside grok or Ruby point at filters.

Common misreadings

Treating it as an absolute threshold. The useful interpretation is baseline-relative, not absolute. The magnitude depends on pipeline shape, input concurrency, and queue type. A pipeline that pulls from a source (Kafka, for example) can legitimately sit at a nonzero backpressure value indefinitely, because that backpressure is what paces the pull to a rate the downstream pipeline can tolerate. Alert on a sustained rise above the pipeline’s own baseline, not on crossing a fixed number.

Assuming a 0.0-1.0 ceiling. Blocked time aggregates across input threads, so with highly concurrent inputs (a beats input with hundreds of inbound connections, for example) the value can exceed 1.0. Do not treat any fixed bound as meaningful.

Comparing values across pipelines. The value is pipeline-specific. Two pipelines on the same host can have completely different normal ranges depending on input type and connection count. Never rank pipelines by this number.

Reading it as worker health. queue_backpressure measures input threads blocked on the queue push. It is the mirror image of worker_utilization, not a synonym for it. You need both to know which side of the queue is the problem.

Assuming throttled inputs means safe inputs. Backpressure only protects you if the input can propagate it. Beats and HTTP inputs can acknowledge or pace senders. Inputs without an acknowledgement mechanism, such as UDP, cannot slow their senders down; when the queue blocks them, data is lost at the source. A rising queue_backpressure on a UDP syslog pipeline is a data-loss-in-progress signal, not a graceful-degradation signal.

Panicking over brief spikes. Short queue spikes during bursty input are normal buffering. So is transient pressure during cold start while the JVM warms up, and during PQ drain after a downstream recovery. Gate interpretation on JVM uptime and on the last_5_minutes or last_15_minutes windows before treating it as an incident.

Signals to watch alongside it

How Netdata helps

flow.queue_backpressure is only decisive in combination with the metrics around it, which is where per-second collection and correlation pay off:

• Netdata’s Logstash collector polls the node stats API and charts queue_backpressure per pipeline alongside worker_utilization, input and output throughput, and queue depth, so the three diagnostic patterns above are visible on one screen instead of three curl commands.
• Rolling-window flow values (current, last_1_minute, last_5_minutes) are collected continuously, giving you a per-pipeline baseline so “rising above baseline” becomes observable rather than guesswork.
• JVM heap, GC time, and process CPU are collected from the same node, letting you rule GC pressure in or out without switching tools.
• Because Netdata also monitors the host and the downstream side (Elasticsearch nodes, Kafka, system disk and network), you can follow the cascade from output blockage to queue growth to input throttling in one place.
• Anomaly detection on throughput and backpressure flags the deviation-from-baseline condition the metric actually calls for, rather than relying on absolute thresholds that do not transfer across pipelines.

Related guides

• How Logstash actually works in production: a mental model for operators
• Logstash monitoring checklist: the signals every production pipeline needs
• Logstash monitoring maturity model: from survival to expert
• Logstash queue full: inputs blocked and the backpressure wedge