$ guides / logstash / logstash-disk-space-io-pressure ▌

Operations Guides

Logstash disk full: PQ, DLQ, and log volumes competing for space

Logstash is down, the logs show disk write failures, and the persistent queue metrics look innocent: queue_size_in_bytes is well under max_queue_size_in_bytes. The queue never filled. The disk did.

This is a distinct outage path from queue fullness. queue.max_bytes limits how much the persistent queue itself allocates. It says nothing about total disk consumption on the partition. When the PQ shares a filesystem with the dead letter queue, Logstash’s own log files, or the OS, any of those consumers can fill the partition while the PQ stays comfortably within its configured limit. Once the filesystem returns no-space errors, page writes fail, checkpoint writes fail, and the process dies.

The other trap is metric interpretation. The Node Stats API exposes queue.data.free_space_in_bytes, and operators routinely read it as “space left in the queue.” It is not. It is raw filesystem free space on the PQ path. That is actually the right number to watch for this failure mode, but only if you know what you are looking at.

What this means

Three independent writers typically converge on one filesystem under /var/lib/logstash (or wherever path.data points):

PQ page files: written into page files capped at 64MB each by default (queue.page_capacity). Space is not released as events drain; a page file is freed only when every event in it has been read and the head checkpoint has moved past it.
DLQ segments: events that permanently failed output delivery. The DLQ has its own size cap (default 1GB via dead_letter_queue.max_bytes), and on versions before 8.4 consumed segments are never deleted automatically.
Log files: Logstash’s own logs. An output retry storm or a debug log level can grow these fast, independently of event volume.

Any one of them, or all three together, can exhaust the partition. The PQ’s configured cap protects against exactly one of the three.

flowchart TD
  PQ[PQ page files] --> FS[Shared filesystem]
  DLQ[DLQ segments] --> FS
  LOGS[Logstash log files] --> FS
  OTHER[OS and other processes] --> FS
  FS --> FULL{Partition at 100 percent}
  FULL --> CRASH[Page and checkpoint writes fail - process crashes]
  FULL --> DRAIN[Slow drain and high I/O wait before the crash]

Version-specific behaviors that matter here:

On Logstash 8.4.0 and later, the dead_letter_queue input plugin supports clean_consumed: true (with commit_offsets: true), which deletes consumed DLQ segments automatically. On earlier versions, segments sit on disk until you delete them manually.
On versions before 8.4, the DLQ writer has a known stuck behavior: once it hits its internal max_bytes accounting, it stops writing permanently, even if you manually delete segment files. Only a restart resets the counter. The disk pressure is gone, but the DLQ silently does nothing until the restart.
At startup, Logstash checks whether the sum of queue.max_bytes across pipelines fits in the available disk space. It logs a warning if not, but starts anyway. That warning is easy to miss and is your earliest signal that the sizing math is wrong.
PQ on NFS is explicitly unsupported. Slow or network-attached storage under the PQ amplifies I/O wait, slows checkpoint writes, and stretches drain times during exactly the incident where you need fast recovery.

Common causes

Cause	What it looks like	First thing to check
PQ grew during a downstream outage and the partition was too small for `max_bytes`	`queue_size_in_bytes` high, `queue.data.free_space_in_bytes` near zero, output errors in logs	`df -h` on the PQ volume; compare `max_queue_size_in_bytes` to partition size
DLQ accumulated over weeks and nobody replayed or cleaned it	`dead_letter_queue.queue_size_in_bytes` large and monotonically growing	`du -sh` on the DLQ directory
Log volume filled by an error/retry storm or debug logging	`logstash-plain.log` or rotated logs consuming the partition; grep shows a repeating exception	Log file sizes and rate of new log lines
PQ on a shared partition with OS or other apps	PQ within limits, but the filesystem is full from non-Logstash data	`du` on the largest directories on that mount
PQ page-release lag after recovery	Downstream is healthy, events flow, but disk usage stays high	Watch whether `queue_size_in_bytes` declines after queue events drain
`queue.drain=true` shutdown hanging on a full disk	Shutdown stalls indefinitely, disk full	Free space first, then allow drain to finish

Quick checks

All read-only. Run them before touching anything.

# Filesystem state on the data and log volumes
df -h /var/lib/logstash /var/log/logstash

# Per-consumer usage on the data volume (adjust paths to your path.data)
du -sh /var/lib/logstash/queue/* 2>/dev/null
du -sh /var/lib/logstash/dead_letter_queue/* 2>/dev/null

# I/O pressure on the volume
iostat -xz 1 5

# What Logstash sees: PQ size, max, and real filesystem free space
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty
# Read: queue.queue_size_in_bytes, queue.max_queue_size_in_bytes,
#       queue.data.free_space_in_bytes, dead_letter_queue.queue_size_in_bytes

# PQ fill or drain direction
# In the same response: flow.queue_persisted_growth_bytes
# positive = filling, negative = draining

# What is flooding the logs right now
tail -n 200 /var/log/logstash/logstash-plain.log
grep -Ei '(retry|error|exception|failed|reject|unavailable|timeout)' /var/log/logstash/logstash-plain.log | tail -n 100

# Biggest space consumers on the partition (top 20)
du -x -h /var/lib/logstash 2>/dev/null | sort -rh | head -20

If path.data or path.logs are customized in logstash.yml, or you run in containers with bind-mounted volumes, adjust the paths. With a different mount layout, the PQ and logs may sit on separate partitions from the defaults above.

How to diagnose it

Confirm which partition is full. df -h on every mount Logstash touches: PQ, DLQ, and logs. Do not assume they share one. If they are split, the fix scope shrinks to one consumer.
Attribute the growth. du -sh per subdirectory of path.data. You are looking for which of the three consumers (queue pages, dead_letter_queue, logs) owns the growth. This determines the fix; freeing the wrong thing buys nothing.
Check the PQ’s own accounting. From the Node Stats API, compare queue_size_in_bytes to max_queue_size_in_bytes, then look at queue.data.free_space_in_bytes. If the queue is at 40% of its cap but filesystem free space is near zero, your max_bytes is larger than the partition can ever honor. The queue limit will never save you.
Check direction. flow.queue_persisted_growth_bytes tells you whether the PQ is still filling (downstream still impaired) or draining (recovery in progress). If it is draining but disk usage is not dropping, that is the page-release lag: pages are freed only when fully drained and checkpointed, so disk reclaim trails event drain. Give it time before deleting anything.
Check I/O wait. iostat -xz showing high %util and await on the PQ device means the disk is now the bottleneck for drain. On NFS or other network storage this gets dramatically worse. A full disk on slow storage extends the incident well past the point where the downstream recovered.
Correlate with the cause of the queue growth. The disk full event is usually the tail end of a downstream backpressure cascade: output errors rose, the queue grew for hours, and the partition ran out. Check output errors and retries in the logs, and confirm downstream health. Fixing the disk without fixing the downstream just restarts the clock. See the PQ masking a real outage pattern and the backpressure metric explainer.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`queue.data.free_space_in_bytes`	Real filesystem free space on the PQ path; this is the crash predictor, not the queue occupancy	Falling steadily; low absolute headroom relative to PQ fill rate
`queue_size_in_bytes / max_queue_size_in_bytes`	PQ occupancy; how much resilience budget is consumed	Sustained growth over 5-15 minutes with output rate below input rate
`flow.queue_persisted_growth_bytes`	Direct fill/drain rate of the PQ in bytes	Positive during a downstream outage; use it to compute runway
`dead_letter_queue.queue_size_in_bytes`	DLQ disk consumption and silent data loss indicator	Any unexpected growth; monotonic increase over days
`df` usage on PQ/DLQ/log mounts	The ground truth the API cannot see (other consumers on the partition)	Usage climbing regardless of what PQ metrics say
`iostat` await / `%util` on the PQ device	I/O saturation slows checkpoint writes and PQ drain	High await during drain after a downstream recovery
Log volume growth rate	Error storms and debug logging can fill a partition faster than the PQ	Log file sizes growing at a rate inconsistent with event volume

Runway math for the shared partition: free_space_in_bytes / (pq_growth_rate + dlq_growth_rate + log_growth_rate). All three consumers count, not just the queue.

Fixes

Emergency: free space without making things worse

Do not restart Logstash as a first move. A restart with queue.drain=true on a full disk can hang shutdown indefinitely, and an unclean kill risks PQ page/checkpoint inconsistency.

Free the cheapest space first. Compress or move old rotated log files off the partition. Usually the fastest safe win.
Delete consumed or stale DLQ segments if the DLQ is the consumer. On 8.4+, prefer replaying through a dead_letter_queue input with clean_consumed: true and commit_offsets: true so segments are deleted as they are consumed. On older versions, manual deletion frees disk but the DLQ writer stays stuck until a restart.
Do not delete live PQ page files under a running process unless you accept losing their contents. If the installation is unrecoverable and you must wipe the queue, stop Logstash cleanly first, move (not delete) the queue directory aside, then start. Events in the moved queue are lost unless you reprocess them by other means.

Fix the sizing mistake

The total of queue.max_bytes across all pipelines on a filesystem should be well under the partition’s capacity, with room left for DLQ, logs, and OS overhead. If the startup logs show the PQ space-check warning, that is this problem announcing itself. Reduce queue.max_bytes or grow the volume. Size max_bytes to the longest downstream outage you intend to survive, then verify the partition can hold that plus the other consumers.

Isolate the consumers

Put the PQ, DLQ, and logs on separate filesystems. This converts one shared cliff edge into three independent capacity problems, each with its own headroom. It also stops an error-storm log flood from killing the queue.

Cap the log consumer

Return log level to info after any debug session. Debug logging is extremely verbose and can itself cause disk I/O problems. Configure log rotation with hard size or age limits so a retry storm cannot fill the volume, and alert on log growth rate, not just log errors.

Address slow storage

If the PQ sits on NFS, move it. NFS under the PQ is unsupported and amplifies drain time and I/O wait exactly when you can least afford it. Local SSD is the reference configuration; if you are on anything slower, test drain rate under load before you need it.

Prevention

Separate volumes for PQ, DLQ, and logs so no single consumer can starve the others.
Headroom rule: keep the partition under roughly 70% at peak PQ utilization, with queue.max_bytes (summed across pipelines) sized against real partition capacity, not aspirations.
Alert on queue.data.free_space_in_bytes trend and absolute value, not only on PQ occupancy percentage. Occupancy can look fine up to the moment the filesystem dies.
Monitor DLQ growth as a correctness signal and a capacity signal. Any growth above zero deserves investigation; unreplayed DLQ is silent data loss plus a disk liability.
Rotate and cap logs, and alert on abnormal log growth rate.
Include drain rate in recovery monitoring. After a downstream outage, watch flow.queue_persisted_growth_bytes go negative and confirm disk usage follows. Recovery is not done when the downstream is healthy; it is done when the queue is drained and space is reclaimed.
Rehearse the full-disk path. Know in advance which consumer you would sacrifice first and how, before the 3 a.m. version of this decision.

How Netdata helps

Netdata charts queue_size_in_bytes, max_queue_size_in_bytes, and queue.data.free_space_in_bytes per pipeline at per-second resolution, so you see the divergence between “queue within limits” and “filesystem nearly full” as it develops, not after the crash.
Disk space and disk I/O (df-equivalent usage, iostat-equivalent await and utilization) for every mount sit on the same dashboard as the Logstash pipeline stats, which is exactly the correlation this incident requires.
flow.queue_persisted_growth_bytes alongside filesystem free space gives a live runway estimate: fill rate versus remaining bytes, for the queue and the partition at once.
DLQ size tracking turns silent DLQ accumulation into a visible trend, catching both the data-loss angle and the capacity angle before either bites.
Log growth anomalies and output error bursts show up on the same timeline as disk usage, so the “error storm filled the log volume” variant is diagnosable in one view instead of three terminals.