$ guides / logstash / logstash-persistent-queue-full-inputs-blocked ▌

Operations Guides

Logstash persistent queue full: max_bytes reached and inputs blocked

The persistent queue has hit queue.max_bytes, input threads are blocked trying to push events in, and the pipeline has stopped accepting new data. A full PQ is almost never the root cause. It is the terminal stage of a downstream outage or capacity failure that the queue was absorbing. Fix the cause, drain the queue, prevent recurrence.

This guide covers confirming the state, finding the downstream cause, calculating how much time you have, and recovering without making things worse. For the underlying throttling metric, see Logstash flow.queue_backpressure: the input-throttling metric explained.

What this means

The persistent queue is a page-based on-disk buffer: events are written to fixed-size page files (default queue.page_capacity is 64mb per page) and checkpointed, so they survive restarts. The default queue.max_bytes is 1GB per pipeline. When queue_size_in_bytes reaches max_queue_size_in_bytes, inputs block instantly with no slow degradation. Input threads stall on queue push, and what happens next depends on the input plugin: the Beats input stops accepting new connections, Kafka consumer lag starts growing, UDP-style inputs drop events outright.

Two things operators regularly confuse here:

PQ capacity vs filesystem space. max_queue_size_in_bytes is a Logstash-internal limit. queue.data.free_space_in_bytes is the free space on the partition holding the queue. A full PQ (Logstash stops accepting events) is a different failure from a full disk (Logstash can crash). You can hit either one first.
Full vs filling. A queue at 95% that is draining is post-recovery behavior and is healthy. A queue at 95% with positive growth is minutes from wedging. The trend matters as much as the level.

flowchart TD
  A[Downstream slows or fails] --> B[Output throughput drops]
  B --> C[Workers blocked on output]
  C --> D[PQ fills: queue_size_in_bytes rises]
  D --> E{max_bytes reached?}
  E -->|no, cause fixed in time| F[Queue drains, recovery]
  E -->|yes| G[Inputs block instantly]
  G --> H[Upstream backs up or drops]
  H --> I[Beats refuses connections, Kafka lag grows, UDP drops]

Common causes

Cause	What it looks like	First thing to check
Downstream outage (Elasticsearch, Kafka, HTTP endpoint)	Output rate near zero, retries and errors in the log, low CPU	Destination health and output error logs
Downstream saturation (bulk rejections, 429s, timeouts)	Output duration rising, retry storms, queue filling steadily	Log lines with retry, reject, 429, timeout patterns
Auth/TLS failure on the output	Sudden output stop after a cert or credential change, SSL/handshake errors in logs	`grep -Ei ‘(SSL
Worker starvation on blocking output calls	Worker utilization high but CPU low, hot threads show output waits	`/_node/hot_threads`
`queue.max_bytes` sized too small for realistic outages	Queue fills during every downstream hiccup, even short ones	Compare max_bytes to your longest expected outage at peak ingest rate
Filesystem full before max_bytes reached	PQ below max but Logstash erroring on writes, free space near zero	`df -h /var/lib/logstash` and `queue.data.free_space_in_bytes`
PQ stuck after downstream recovery	Destination healthy again but queue is not draining	Drain rate, disk I/O, and hot threads

The first three causes dominate. A full PQ with low CPU and output errors in the log is the classic downstream backpressure cascade: outputs slow or fail, queue fills, inputs block, upstream backs up. CPU is low because workers are waiting on I/O, not computing.

Quick checks

All of these are read-only and safe to run during an incident.

# 1. Confirm queue state, type, and occupancy per pipeline
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty
# Look at pipelines.<name>.queue: type, events_count,
# queue_size_in_bytes vs max_queue_size_in_bytes,
# and queue.data.free_space_in_bytes

# 2. Check flow metrics: is the queue still growing?
# In the same response, look at:
#   flow.input_throughput vs flow.output_throughput
#   flow.queue_persisted_growth_bytes (positive = filling, negative = draining)
#   flow.queue_backpressure (fraction of time inputs are blocked)

# 3. Check filesystem space on the PQ and log volumes
df -h /var/lib/logstash /var/log/logstash

# 4. Look for output-side errors and retries
grep -Ei '(retry|error|exception|failed|reject|unavailable|timeout|429|503)' /var/log/logstash/logstash-plain.log | tail -n 200

# 5. Look for auth/TLS failures (sudden output stop after rotation)
grep -Ei '(SSL|TLS|certificate|handshake|authentication|unauthorized)' /var/log/logstash/logstash-plain.log | tail -n 100

# 6. Capture hot threads: are workers blocked on output I/O?
curl -sS 'http://127.0.0.1:9600/_node/hot_threads?pretty'
# Take 2-3 snapshots a few seconds apart. Output wait paths, not
# filter code, confirm a downstream cause.

# 7. Check disk I/O on the queue volume (drain bottleneck)
iostat -xz 1 5

Interpreting the results:

If queue.type is persisted and queue_size_in_bytes equals max_queue_size_in_bytes, the queue is full. Inputs are blocked right now.
flow.queue_persisted_growth_bytes tells you the direction of travel directly. Positive means still filling. Negative means draining.
queue.data.free_space_in_bytes is filesystem free space, not remaining PQ capacity. Check both.
Hot threads showing workers in output wait states (blocked on the Elasticsearch or network output path) confirm the cause is downstream. Hot threads showing filter code would point at a compute bottleneck instead, which is a different incident.

How to diagnose it

Confirm the wedge. Query /_node/stats/pipelines. Verify queue_size_in_bytes is at or near max_queue_size_in_bytes, flow.output_throughput is below flow.input_throughput (or zero), and flow.queue_backpressure is elevated. That combination is a full queue with blocked inputs, not a display artifact.
Rule out the disk. Run df -h /var/lib/logstash and compare against queue.data.free_space_in_bytes. If the filesystem is full, the queue limit is irrelevant: Logstash cannot write pages and may crash. Freeing disk (rotating logs, moving unrelated data) is the first move, and it is a different fix from anything below.
Identify the downstream cause. Read the output error patterns from the log. Timeouts and connection failures point at network or destination availability. 429s and rejections point at destination saturation. Auth and TLS errors point at credentials or certificates. Check the destination’s own health independently, because Logstash’s view of it is only “my writes are failing.”
Check what workers are doing. Hot threads showing all workers waiting on output confirms the cascade and tells you the pipeline itself is fine. Low CPU alongside this is expected and is the key differentiator from a filter-bound pipeline.
Calculate runway or confirm you are past it. If the queue is not yet at max:
runway = (max_queue_size_in_bytes - queue_size_in_bytes) / fill_rate_bytes_per_second
Use the smoothed fill rate (flow.queue_persisted_growth_bytes over 5-15 minutes), not a single spike. If runway is under 30 minutes, treat it as a page-level emergency even before the queue is technically full.
Decide: fix downstream, shed load, or expand capacity. If downstream recovery is expected within the runway, hold and monitor drain. If not, reduce ingest (pause non-critical shippers, drop lower-value inputs) to stretch the runway while the downstream fix lands. Raising queue.max_bytes mid-incident only helps if the filesystem has the space and the cause will outlast the new runway.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`queue.queue_size_in_bytes / queue.max_queue_size_in_bytes`	PQ occupancy, the headline signal	Above 80% sustained; above 90% with growth is page territory
`flow.queue_persisted_growth_bytes`	Fill or drain direction and rate	Positive and sustained over 5-15 minutes
`flow.output_throughput` vs `flow.input_throughput`	Whether the pipeline is keeping up	Output persistently below input while queue grows
`flow.queue_backpressure`	Fraction of time inputs are blocked on the queue	Sustained rise above baseline
`events.queue_push_duration_in_millis`	Earliest backpressure signal, before the queue visibly fills	Non-trivial values appearing when it was near zero
`queue.data.free_space_in_bytes`	Filesystem runway under the queue	Declining toward zero independently of PQ occupancy
Output errors and retries in logs	Root cause evidence	Sustained nonzero retry, reject, 429, or timeout patterns
`process.cpu.percent`	Distinguishes downstream wait from compute saturation	Low CPU during a full queue confirms I/O wait, not grok hell

Fixes

Fix the downstream cause first

This is the fix in almost every case. Restore the Elasticsearch cluster, clear the bulk rejections, fix the credential or certificate, repair the network path. Until output throughput recovers, nothing you do on the Logstash side drains the queue.

Tradeoff: none. This is the only fix that resolves the incident rather than buying time.

Reduce ingest to stretch runway

If downstream recovery will take longer than the runway, shed load deliberately before the queue decides for you. Pause non-critical Beats shippers, disable lower-value inputs, or route less important pipelines elsewhere. Every byte you do not ingest is runway.

Tradeoff: you are choosing which data arrives late or is lost, instead of letting the wedge choose arbitrarily. Chosen loss beats arbitrary loss.

Raise queue.max_bytes (only with disk to back it)

If the filesystem has headroom and the cause will outlast current runway, increasing queue.max_bytes extends the buffer. This requires a pipeline restart to apply, and with a PQ that restart is safe for queued data.

Tradeoffs: a bigger queue means a longer drain after recovery and more disk consumed. Size max_bytes against the partition, never larger than the space you can actually afford, and remember the PQ shares its partition with logs and possibly the DLQ.

Recover from the wedge

Once downstream is healthy, the queue should drain on its own. Watch flow.queue_persisted_growth_bytes go negative and queue_size_in_bytes fall. Expect drain to take longer than fill, sometimes much longer, and expect high disk I/O and a burst of output load during drain. That is normal recovery behavior, not a second incident.

If the queue does not drain after downstream recovery, check disk I/O on the queue volume first, then hot threads. There are reports of Logstash staying wedged with a full PQ even after the output recovers on some versions. If you are genuinely stuck with a healthy downstream and no drain, a restart with the PQ intact is the pragmatic move: queued events survive and will replay.

Do not delete PQ page or checkpoint files as a first resort. That trades a recoverable wedge for guaranteed data loss. If you suspect queue corruption after an unclean shutdown (OOM kill, kill -9), that is a different incident with its own procedure, and Logstash ships queue inspection and repair utilities (bin/pqcheck and bin/pqrepair) for exactly that case.

Prevention

Alert on occupancy with trend, not just level. Ticket at PQ occupancy above 80% sustained. Page only when all of these hold: occupancy above 90%, smoothed growth positive over 5-15 minutes, output rate below input rate, runway under 30 minutes, and JVM uptime past warmup. The full-queue event itself is too late to be your first alert.
Track fill rate as a first-class metric. flow.queue_persisted_growth_bytes answers the only question that matters during a downstream outage: how long until full. Run the runway calculation continuously.
Size max_bytes against realistic outages. Your PQ should absorb the longest downstream outage you expect to survive at peak ingest rate, with margin, and still fit on its partition alongside logs and DLQ.
Watch the filesystem separately. queue.data.free_space_in_bytes and df on the PQ volume are independent failure paths. Alert on both.
Monitor the output side, not just the queue. Output errors, retries, and per-output duration are the early warnings that fire hours before the queue fills.
Watch the drain path too. After any downstream incident, alert if the queue is not draining. Recovery that stalls is a second incident.

How Netdata helps

Netdata’s Logstash collector scrapes /_node/stats per second, so PQ occupancy, throughput, and flow metrics are charted on the same host as disk space and disk I/O. This matters operationally:

The fill-to-wedge timeline is visible in one view: output throughput drops, queue grows, backpressure rises, inputs block. You see the cascade instead of inferring it from separate tools.
PQ occupancy and filesystem free space on the queue volume are on the same dashboard, making the “PQ full vs filesystem full” distinction immediate.
Anomaly detection on output throughput and queue growth flags the slow fill during the masking window, when the queue is absorbing a downstream outage but nothing has failed yet.