$ guides / logstash / logstash-persistent-queue-not-draining ▌

Operations Guides

Logstash persistent queue not draining: page-release lag after downstream recovery

The downstream outage is over. Elasticsearch is green, output errors have stopped, and Logstash is delivering events again. But the persistent queue is shrinking far slower than it filled, disk I/O is pinned, and df shows the same used space it did an hour ago even though queue_size_in_bytes has clearly dropped.

Most of the time, nothing is stuck. A persistent queue (PQ) draining after recovery looks unhealthy from almost every angle: high disk I/O, reduced effective throughput, and disk usage that refuses to go down. This is expected behavior driven by how PQ pages are released. The failure modes you actually need to rule out are narrower: page or checkpoint corruption, poison events pinning pages, and disk I/O saturation becoming the new bottleneck.

What this means

The PQ stores events in append-only page files on disk. During a downstream outage, inputs keep writing pages while outputs acknowledge nothing. When the downstream recovers, workers resume pulling batches, outputs start acknowledging, and the queue begins to drain. Three properties of the design make this drain look worse than it is:

Drain is I/O-bound, fill was not. Filling the queue was sequential writes. Draining it is random reads of page files plus the normal write path for newly arriving events, on top of the output’s own network latency. On slow storage, and especially on network-attached storage, drain takes far longer than fill.
Pages are released only when fully drained and checkpointed. A page file counts against queue_size_in_bytes until every event in it has been acknowledged by the output and the checkpoint records that fact. One unacknowledged event in a page keeps the whole page on disk.
Logical queue size and physical disk usage diverge. queue_size_in_bytes drops as pages become releasable, but the filesystem does not show free space returning until page files are actually deleted. So the API says the queue is shrinking while df says nothing changed.

The practical consequence: judge drain progress by queue_size_in_bytes and flow.queue_persisted_growth_bytes (negative means draining), not by df. Judge disk health separately, because the filesystem can still fill while the queue is logically shrinking if pages are not being released.

flowchart TD
    A[Downstream recovered, PQ draining] --> B{queue_size_in_bytes decreasing?}
    B -- No --> C[Check output errors and worker state]
    B -- Yes --> D{flow.queue_persisted_growth_bytes negative?}
    D -- No --> C
    D -- Yes --> E{Drain rate reasonable for storage?}
    E -- Yes --> F[Normal recovery drag - monitor]
    E -- No, I/O saturated --> G[Disk I/O is now the bottleneck]
    B -- Stall after restart --> H[Check checkpoint/page corruption - pqcheck]
    C --> I{Output errors or retries rising?}
    I -- Yes --> J[Downstream not actually healthy]
    I -- No --> H

Common causes

Cause	What it looks like	First thing to check
Normal page-release lag	`queue_size_in_bytes` dropping, `df` flat, high read I/O, drain slower than fill	`flow.queue_persisted_growth_bytes` negative and steady
Disk I/O saturation is the new bottleneck	Drain rate far below what the output could absorb, `%util` near 100 on the PQ device, low CPU	`iostat -xz 1 5` on the PQ volume
Slow or network storage under the PQ	Drain takes many times longer than fill; latency on every page read	Whether `path.data` sits on NFS (not supported) or other slow storage
Checkpoint or page corruption after unclean shutdown	Logstash restarts but queue does not drain, or drain stalls partway; PQ-related errors in the log	PQ errors in `logstash-plain.log`; run `pqcheck` on the queue directory
One poison event pinning a page	Queue drains then stops at a specific size; one page never released because one event in it keeps failing	Output errors and DLQ growth correlated with the stall point
Downstream not actually healthy	Output throughput below input, retries or 429s still appearing	Per-output plugin stats and downstream cluster health

Quick checks

All read-only. Run them in this order.

# 1. Queue state: is it actually draining?
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty
# Look at: queue.events_count, queue.queue_size_in_bytes,
# queue.max_queue_size_in_bytes, queue.data.free_space_in_bytes,
# flow.queue_persisted_growth_bytes (negative = draining),
# flow.input_throughput vs flow.output_throughput

# 2. Logical queue vs physical disk
# Note: du can be slow on a very large queue directory.
du -sh /var/lib/logstash/queue/main/
df -h /var/lib/logstash
# du larger than queue_size_in_bytes and df not moving = page-release lag, expected

# 3. Is disk I/O the bottleneck?
iostat -xz 1 5
# Watch %util and r_await on the device holding the PQ

# 4. Is the output actually healthy now?
grep -Ei '(retry|error|exception|failed|reject|unavailable|timeout|429|503)' /var/log/logstash/logstash-plain.log | tail -n 200

# 5. What are workers doing?
curl -sS 'http://127.0.0.1:9600/_node/hot_threads?pretty'
# Output-wait stacks = downstream still limiting; PQ read paths = disk-bound drain

# 6. DLQ growth (events permanently failing pin their pages)
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty
# Look at dead_letter_queue.queue_size_in_bytes

Two notes while checking. First, do not poll the stats API faster than every 10 seconds on a loaded instance; the API competes with the pipeline for JVM resources. Second, if queue.drain: true is set and Logstash is draining during a shutdown, the Node Stats API largely stops updating and only events.in/events.out keep moving, so drain progress read from the API during shutdown is unreliable.

How to diagnose it

Confirm the drain is real. Take two samples of queue.queue_size_in_bytes a minute apart. If it is decreasing and flow.queue_persisted_growth_bytes is negative, the queue is draining. Nothing is stuck. Go to step 4 to estimate how long it will take.
Confirm the downstream is genuinely healthy. Check output throughput against input throughput, per-output plugin durations, and the log for retries and 429s. A downstream that recovered “mostly” produces exactly this symptom: a queue that drains at a trickle because outputs are still slow. Hot threads showing workers parked in output-wait paths confirms this.
If the queue is not draining, check for a pinned page. A page is released only when every event in it is acknowledged. If the drain stops at a specific size and stays there while outputs are otherwise flowing, suspect one event (or a small set of events) that the output permanently fails: mapping conflicts, schema mismatches, poison-pill payloads. These land in the DLQ if it is enabled, or are logged and dropped if it is not, but only after retries are exhausted. Until then, the event’s page stays on disk. Check DLQ growth and the log for non-retryable failures.
Measure the drain rate against your storage. Compute the drain rate from the queue_size_in_bytes delta per minute. Compare it to what the device can do in iostat. If %util is pinned near 100 and read latency is high, disk I/O has become the bottleneck and the drain rate you see is all you are going to get. If neither I/O nor CPU is saturated, standard tuning (batch size, worker counts) often does not help much; splitting into parallel pipelines per output has been reported to improve throughput where single-pipeline tuning did not.
Rule out corruption. If the queue refuses to drain after a restart, or Logstash failed to start cleanly, check the log for checkpoint and page errors, then run the bundled checker against the queue directory:

# Read-only check of PQ checkpoint and page state (path is the pipeline's queue dir)
bin/pqcheck /var/lib/logstash/queue/main/

pqrepair is the companion repair tool. Treat it as disruptive: stop Logstash first, and back up the queue directory before running it, since repair can mean losing unacknowledged events. On older versions, unexpected shutdowns can leave checkpoint.<n>.tmp files that made these tools crash; this was reportedly fixed in Logstash 8.3. There was also a pre-8.3 crash scenario where a head checkpoint pointed at a purged page and neither restart nor repair could recover the queue; on current versions this specific failure is reportedly fixed, but unclean shutdowns (OOM kill, kill -9, force-killed pods) remain the primary corruption source.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`queue.queue_size_in_bytes` / `max_queue_size_in_bytes`	Occupancy and whether it is falling	Flat or rising after downstream recovery
`flow.queue_persisted_growth_bytes`	Direct fill/drain rate; negative means draining	Positive or near-zero during supposed recovery
`flow.output_throughput` vs `flow.input_throughput`	Whether output can absorb the backlog plus live traffic	Output at or below input during drain
`queue.data.free_space_in_bytes`	Filesystem free space on the PQ volume, distinct from logical queue size	Falling while `df` was expected to recover
Disk `%util` and `r_await` on the PQ device (OS-level)	Whether I/O saturation now caps the drain	`%util` near 100 for the whole drain
`dead_letter_queue.queue_size_in_bytes`	Permanently failing events that pin pages	Any growth during drain
`flow.worker_utilization` and hot threads	Whether workers are I/O-waiting on output or on PQ reads	All workers blocked on output while queue is full

Fixes

Let normal drag finish

If the queue is draining, the correct action is usually no action. Do not restart Logstash to “speed up” a healthy drain: a restart forces PQ replay of unacknowledged events and can make things slower. Set expectations: df will lag queue_size_in_bytes, potentially by a long time, because pages are deleted only when fully drained and checkpointed.

Disk I/O is the bottleneck

Move the PQ to faster local storage. The PQ documentation is explicit that NFS is not supported for the queue, for both integrity and performance reasons. If path.data is on network storage, that is the finding. Slow drain on NFS is not a tunable; it is the architecture.
Reduce competing I/O on the volume. PQ pages, DLQ files, and Logstash’s own logs compete for the same disk. If the PQ shares a partition with verbose logs, drop the log level back from debug and check log rotation.
Reduce concurrent pressure. If input rate is high during the drain, the head page keeps being written while tail pages are being read. Shedding non-critical inputs during recovery shortens total drain time.

Downstream still limiting

If output throughput is the cap, fix the downstream, not Logstash: cluster health, bulk rejections, thread pool saturation. Logstash-side, increasing pipeline.workers can help only if workers are genuinely CPU-starved; when workers are blocked on output I/O, more workers mostly means more waiters. Check flow.worker_utilization and per-plugin output utilization before touching worker counts.

Pinned pages from permanently failing events

Find the failing event class in the log (mapping conflicts, schema mismatches), fix the data or the destination mapping, and let the output acknowledge the events so their pages release. If the DLQ is enabled, the failed events are preserved for replay; if it is not, they are logged and dropped once retries exhaust, which also releases the page but loses the event. Decide which tradeoff you want before the incident, because during the incident you get whichever one is configured.

Corruption

Stop Logstash, back up the queue directory, run pqcheck, then pqrepair if warranted. If repair fails and the pipeline must come back, moving the queue directory aside and starting with a fresh queue loses everything the old queue held; assess that impact explicitly before doing it. On versions before 8.3, stale .tmp checkpoint files can block the tools themselves; removing those files manually is the documented workaround.

Prevention

Size queue.max_bytes relative to the disk, not to wishful thinking. PQ max_bytes limits the queue but not the filesystem. Keep the PQ well under the partition’s capacity and leave headroom for logs and the DLQ. Also keep queue.page_capacity below queue.max_bytes; a page capacity larger than max_bytes stalls the pipeline with no error in the log.
Use local disk for path.data. No NFS. Drain time on network storage is measured in multiples of fill time.
Monitor the recovery path, not just the failure path. Alerting on PQ occupancy catches the fill; you also want flow.queue_persisted_growth_bytes and drain rate tracked after incidents so a stalled drain pages someone instead of being discovered the next morning.
Enable and monitor the DLQ so permanently failing events are visible instead of silently pinning pages and then silently vanishing.
Shut down cleanly. Most PQ corruption follows kill -9, OOM kills, and orchestrators force-killing pods before drain. Give termination grace periods room for the queue. queue.drain: true waits for an empty queue before shutdown, but the official guidance warns against it unless the queue is small enough to drain quickly. Treat it as a small-queue tool, not a general safety net.

How Netdata helps

Queue occupancy and fill/drain direction in one view. Netdata charts queue_size_in_bytes, max_queue_size_in_bytes, and flow.queue_persisted_growth_bytes per pipeline, so you can see whether the queue is draining and how fast without hand-computing counter deltas.
Logical queue vs physical disk correlation. Plotting PQ size next to filesystem usage on /var/lib/logstash makes the page-release lag visible instead of alarming: the queue line falls first, df follows later.
Disk I/O saturation alongside drain rate. Per-second %util, read latency, and throughput on the PQ device, correlated with drain rate, answers “is the disk now the bottleneck” without a separate iostat session.
Output health correlation. Output throughput, per-plugin durations, and downstream error signals next to queue depth show whether a slow drain is the downstream’s fault or the storage’s fault.
Recovery-path alerting. Alerts on queue_persisted_growth_bytes staying non-negative after downstream recovery catch a stalled drain, which pure occupancy alerts miss.
DLQ growth during drain. DLQ size charted against the drain curve surfaces the pinned-page scenario early.