$ guides / logstash / logstash-persistent-queue-runway ▌

Operations Guides

Logstash persistent queue runway: how long until the PQ fills

A persistent queue changes the failure shape of a Logstash pipeline. When the output destination goes down, nothing looks broken for a while: inputs keep accepting events, throughput counters keep moving, the process stays green, and the queue absorbs the gap between arrival and delivery. The outage exists the whole time; the PQ just delays the moment anyone feels it.

Once queue_size_in_bytes reaches max_queue_size_in_bytes, inputs block and upstream systems (Beats agents, Kafka consumers, TCP senders) back up or drop data. The question that matters during a downstream impairment is not “is the queue growing?” but “how long until it is full at this rate?” This guide covers how to compute that number from the node stats API, how to read it against expected recovery time, and the pitfalls that produce misleading answers.

Why runway matters more than occupancy

Occupancy alone is a weak signal. A queue at 60% full that is draining is healthy. A queue at 20% full growing at 500 MB per minute is an emergency. The same byte count means opposite things depending on the trend sign and magnitude.

Treat PQ occupancy as a resilience budget, not a steady state. Every byte written while the output is impaired is budget consumed. Runway tells you how much budget is left in minutes. You compare it against expected downstream recovery time, and that comparison drives the decision: if recovery takes 20 minutes and runway is 4 hours, you wait. If recovery takes 2 hours and runway is 35 minutes, you start shedding load now, while there is still time for that decision to matter.

The runway formula

runway_seconds = (max_queue_size_in_bytes - queue_size_in_bytes) / fill_rate_bytes_per_second

Three inputs, all from the pipeline stats API:

queue_size_in_bytes: current on-disk queue occupancy.
max_queue_size_in_bytes: the configured queue.max_bytes cap (default 1 GB).
Fill rate: how fast occupancy is changing, in bytes per second.

For the fill rate you have two options:

Read it directly from flow.queue_persisted_growth_bytes (verified: exposed since Logstash 8.5.0; example in the official Node Stats API reference). Positive means growing, negative means draining. Prefer a smoothed window (last_5_minutes or similar) over current, which is burst-sensitive.
Derive it from two samples: (size_t2 - size_t1) / (t2 - t1).

If the fill rate is zero or negative, runway is infinite: the queue is not filling, so there is nothing to budget. Runway only has meaning when growth is positive.

Gathering the inputs

All inputs come from one endpoint. These are read-only queries.

# PQ capacity and occupancy for one pipeline
curl -sS http://127.0.0.1:9600/_node/stats/pipelines/main?pretty

# Relevant fields:
#   pipelines.main.queue.type                    (must be "persisted")
#   pipelines.main.queue.queue_size_in_bytes
#   pipelines.main.queue.max_queue_size_in_bytes
#   pipelines.main.queue.data.free_space_in_bytes
#   pipelines.main.flow.queue_persisted_growth_bytes

# Filesystem free space on the queue volume (independent check)
df -h /var/lib/logstash

Two things to confirm before trusting the numbers:

queue.type must be persisted. On a memory queue there is no disk runway to compute; the buffer is small and backpressure is immediate.
In multi-pipeline deployments, query each pipeline individually. Each pipeline has its own queue, and aggregate views mask the one that is filling.

Computing runway: a worked procedure

Step 1: check the direct flow metric

# Read the PQ growth rate from flow metrics
curl -sS http://127.0.0.1:9600/_node/stats/pipelines/main | python3 -c "
import sys, json
p = json.load(sys.stdin)['pipelines']['main']
q = p['queue']
g = p.get('flow', {}).get('queue_persisted_growth_bytes', {})
print('queue_size_in_bytes:    ', q.get('queue_size_in_bytes'))
print('max_queue_size_in_bytes:', q.get('max_queue_size_in_bytes'))
print('growth current:         ', g.get('current'))
print('growth last_5_minutes:  ', g.get('last_5_minutes'))
"

If last_5_minutes is positive, the queue is filling at roughly that many bytes per second on a 5-minute smoothed basis. If negative, the queue is draining.

Step 2: compute runway

# Compute runway in minutes from the flow metric
curl -sS http://127.0.0.1:9600/_node/stats/pipelines/main | python3 -c "
import sys, json
p = json.load(sys.stdin)['pipelines']['main']
q = p['queue']
size = q['queue_size_in_bytes']
cap  = q['max_queue_size_in_bytes']
g = p.get('flow', {}).get('queue_persisted_growth_bytes', {})
rate = g.get('last_5_minutes') or g.get('current') or 0
if rate <= 0:
    print('Queue is stable or draining; runway is unbounded.')
else:
    runway_min = (cap - size) / rate / 60
    print(f'Occupancy: {size/cap*100:.1f}%  Fill rate: {rate/1024/1024:.1f} MB/s  Runway: {runway_min:.0f} min')
"

Step 3: cross-check with the two-sample method

If the flow metric is unavailable or you want an independent check, sample the counter twice:

# Two-sample fill rate and runway (60-second window)
API=http://127.0.0.1:9600
readq() {
  curl -sS "$API/_node/stats/pipelines/main" | python3 -c "
import sys, json
q = json.load(sys.stdin)['pipelines']['main']['queue']
print(q['queue_size_in_bytes'], q['max_queue_size_in_bytes'])
"
}
read S1 MAX <<< "$(readq)"
sleep 60
read S2 MAX <<< "$(readq)"
python3 -c "
size1, size2, cap = $S1, $S2, $MAX
rate = (size2 - size1) / 60.0
if rate <= 0:
    print(f'Queue stable or draining ({rate/1024/1024:.1f} MB/s). No runway concern.')
else:
    print(f'Fill rate: {rate/1024/1024:.1f} MB/s')
    print(f'Runway:    {(cap - size2) / rate / 60:.0f} minutes')
"

A 60-second window is the minimum worth trusting. A 5-15 minute window is better: batch flushes and input bursts make short windows noisy.

Step 4: decide based on the comparison

The decision rule is a comparison of two durations:

flowchart TD
  A[PQ growth positive?] -->|no| B[Stable or draining: monitor only]
  A -->|yes| C[Compute runway minutes]
  C -->D{Runway vs expected downstream recovery time}
  D -->|runway much larger| E[Monitor: recompute every few minutes]
  D -->|runway comparable| F[Prepare mitigation: reduce ingest, isolate pipelines]
  D -->|runway under 30 min| G[Act now: shed load and escalate downstream fix]

The PAGE combination: runway under 30 minutes with positive smoothed growth, occupancy above 90%, output rate below input rate, and active input. Everything short of that is a TICKET with a recomputation loop.

Reading the result correctly

Recompute frequently. Fill rate is not constant. It changes as backpressure propagates, the downstream degrades further, or retry storms kick in. A runway of 90 minutes computed once can be 25 minutes ten minutes later. During an incident, recompute every few minutes.

Compare against realistic recovery time, not hopeful recovery time. The headroom definition that matters: enough free PQ capacity to absorb the longest realistic downstream outage you expect to survive. If your Elasticsearch cluster has historically taken 2 hours to recover from a full outage, a 1 GB queue that fills in 45 minutes under normal ingest does not provide that headroom, no matter how healthy it looks today.

Watch the drain side. After downstream recovery, the queue drains, but drain often takes longer than fill. Logstash now processes live traffic plus the backlog, and the output destination may still be fragile. High occupancy that is declining is expected post-recovery behavior. What deserves investigation is occupancy that stays flat after the downstream is healthy again: that points at disk I/O or drain rate as the bottleneck.

Common pitfalls

Cold start replay. After a restart, Logstash replays queued events from the prior session. Occupancy starts high and events.out can exceed events.in while the backlog drains. A runway calculation in the first minutes after startup is meaningless. Gate any alerting on JVM uptime (at least 300-600 seconds) and check the trend sign before reacting.
Burst sensitivity of current. The current window of flow metrics reacts to short bursts. A single 30-second ingest spike can show a frightening fill rate that does not persist. Use smoothed 5-15 minute windows for decisions.
Page-granular disk usage. The PQ allocates disk in pages (64 MB by default) and frees pages only when fully drained and checkpointed. Occupancy moves in jumps, and disk usage can stay high after events are consumed. Do not confuse page-release lag with continued growth.
Filesystem limits beat queue limits. max_queue_size_in_bytes caps the queue, but the queue lives on a real filesystem. If the PQ shares a partition with logs, the DLQ, or the OS, the disk can fill before the queue cap is reached. A full disk can crash the process. Always check queue.data.free_space_in_bytes and df alongside the queue math, and compute a second runway against filesystem free space if the partition is shared.
Multi-pipeline blind spots. In pipelines.yml deployments, each pipeline has its own queue. One pipeline’s PQ can be at 95% while others sit at 5%, and any aggregate view averages that away. Query each pipeline’s stats individually.
Treating “no data loss yet” as healthy. This is the core mistake. The PQ is designed to make a downstream outage invisible to data-loss checks. The only advance warning is the occupancy trend and the runway computed from it. If you are not computing runway, the first symptom you see will be blocked inputs.

Signals to monitor

Signal	Why it matters	Warning sign
`queue.queue_size_in_bytes` / `queue.max_queue_size_in_bytes`	Occupancy, the raw input to runway	Sustained above 80%; above 90% with positive growth is critical
`flow.queue_persisted_growth_bytes`	Direct fill rate; positive = consuming budget, negative = draining	Positive on smoothed windows for more than a few minutes while output lags input
`flow.output_throughput` vs `flow.input_throughput`	Confirms the queue is filling because delivery is impaired	Output persistently below input during the growth period
`queue.data.free_space_in_bytes`	Filesystem runway, which can expire before queue runway	Declining free space on the PQ volume
Output errors and retries in logs	Corroborates that the fill is downstream-caused	429s, timeouts, connection failures coinciding with PQ growth

The correlation pattern: PQ growing, output errors present, output rate below input rate, CPU moderate (workers waiting on I/O, not computing). That combination says the fill is a downstream outage being absorbed, and the runway math tells you how much absorption is left.

Monitoring with Netdata

Netdata collects the metrics needed for continuous runway tracking:

queue_size_in_bytes and max_queue_size_in_bytes per pipeline, making occupancy trend a continuous curve.
flow.queue_persisted_growth_bytes, providing fill rate directly for continuous runway derivation instead of manual two-sample calculation.
PQ growth correlated with output throughput and worker utilization, to distinguish downstream-caused fills from compute bottlenecks.
Filesystem free space on the queue volume alongside queue metrics, surfacing the “disk fills before max_bytes” failure path.
Per-pipeline visibility, so a single filling queue is not averaged away by healthy neighbors.