Logstash memory queue vs persistent queue: durability, visibility, and failure modes

Every Logstash pipeline has a queue between its inputs and its worker threads. The default is the memory queue: a small, bounded, in-memory buffer with no durability. The alternative is the persistent queue (PQ): a page-based, checkpointed, on-disk buffer that survives restarts.

This choice changes three things that matter operationally: what you lose when the process dies, what you can see in the metrics API, and how the pipeline fails when the queue fills. Teams enable PQ for durability and then discover it added disk I/O saturation, page corruption, and a false sense of safety to their failure catalogue. Teams running the memory queue often have no written answer for what a crash costs them.

For the broader pipeline model, see How Logstash actually works in production.

The two queues at a glance

Neither queue is strictly better. They are different failure-mode packages.

How each queue works

The memory queue is a bounded buffer between input threads and the worker pool. Its effective size is the in-flight count: pipeline.workers (default: CPU core count) times pipeline.batch.size (default: 125 events). There is no direct queue size knob. On an 8-core host with defaults, the buffer holds on the order of a thousand events. When it is full, input threads block on queue push, and backpressure propagates to whatever is upstream: Beats agents, Kafka consumers, TCP senders.

The persistent queue writes events to append-style page files on disk, defaulting to 64MB per page and 1GB total (queue.max_bytes). Writes are checkpointed: the checkpoint records which events have been fully processed and acknowledged. On restart after an abnormal termination, unacknowledged events are replayed. This buys durability, and it is also why PQ is at-least-once: events acknowledged at the output but not yet checkpointed in the queue can be delivered twice.

flowchart LR
    IN[Input threads] --> Q{Queue}
    Q -->|memory: small in-flight buffer| W[Worker pool]
    Q -->|persisted: 64MB pages + checkpoints| W
    W --> F[Filter chain]
    F --> OUT[Output plugins]
    OUT -.->|slow or failing: backpressure| Q
    Q -.->|full: inputs block| IN
    P[(Disk)] -.->|PQ pages, checkpoints, free space| Q

The shared mechanic: a worker holds its batch until the output acknowledges it. A slow output blocks workers, which stops queue drain, which fills the queue, which blocks inputs. That cascade is identical for both queue types. What differs is how much time the queue buys you before inputs block, and what state survives if the process dies in the middle.

What the choice changes about durability

Memory queue: crash means loss. Anything in the queue or in a worker batch when the JVM dies is gone. The loss is bounded (the in-flight count is small) but silent. After a restart there is a gap in your data that nothing in Logstash will tell you about. You detect it, if at all, by tracking event continuity at the destination.

Persistent queue: crash means replay, with caveats. A JVM or process crash loses nothing already written to a page file; unacknowledged events are replayed on startup. The caveats:

• Duplicates. At-least-once delivery means replay can re-deliver events that were already sent but not yet checkpointed. Your destination or downstream consumers must tolerate duplicates.
• OS-level failure is weaker than process-level failure. A kernel panic or power loss can lose recently written events that had not reached a checkpoint boundary. PQ is not fsync-per-event by default.
• Hardware failure is out of scope. PQ provides no replication. If the disk dies, the queue dies with it.
• Unclean shutdown is itself a failure mode. After an OOM kill, kill -9, or forced pod termination, page files and checkpoints can become inconsistent. Logstash may refuse to start, start but fail to drain, or silently drop events in unacknowledged pages. Elastic ships bin/pqcheck to inspect checkpoint files for corruption and pqrepair to remove corrupt segments. pqrepair is destructive and can lose data: run it only with Logstash stopped, after backing up the queue directory.

The operational summary: the memory queue converts crashes into small, silent data gaps. PQ converts crashes into replays with duplicates, plus a new class of corruption incidents that can take the whole pipeline down at startup.

What the choice changes about visibility

The two queues are not equally observable, and teams underestimate this.

The memory queue exposes queue.events_count, but that number is inherently tiny because the buffer is tiny. There is no queue_size_in_bytes and no max_queue_size_in_bytes for a memory queue: those fields only appear for PQ. You cannot compute occupancy percentage or runway. Your practical signals are indirect:

• events.queue_push_duration_in_millis rising above its normal near-zero baseline, which is input threads blocking on queue push.
• flow.queue_backpressure (Logstash 8.x flow metrics) climbing, which measures the fraction of time inputs spend throttled.
• events.in dropping while upstream sources are known to be active.

PQ is well instrumented. You get queue.queue_size_in_bytes, queue.max_queue_size_in_bytes, queue.data.free_space_in_bytes (filesystem free space on the PQ partition), and flow.queue_persisted_growth_bytes for fill rate. Occupancy and time-to-full are directly computable:

runway_seconds = (max_queue_size_in_bytes - queue_size_in_bytes) / fill_rate_bytes_per_second

The paradox: the memory queue fails fast and tells you little; the PQ fails slowly and tells you a lot, but only if you watch it. An unmonitored PQ is worse than a memory queue in one specific way: it converts an obvious immediate outage into a delayed outage that looks healthy for hours. Downstream is broken, output rate is below input rate, every health check is green, and the only advance warning is the PQ occupancy trend.

How the same “queue full” symptom differs

“Inputs are blocked” is the terminal state for both queue types, and the composite backpressure cascade (output stalls, workers block, queue fills, inputs block) is the same either way. The differences that matter during an incident:

With a memory queue:

• The wedge forms in seconds to minutes. There is no runway to calculate; by the time you see queue growth, inputs are already throttled.
• The signal to watch is flow.queue_backpressure and queue push duration, not occupancy.
• Restarting Logstash mid-incident discards the backlog. Sometimes that is the right call (shed load to recover), but make it a decision, not an accident.

With a persistent queue:

• The wedge forms over hours. You have runway, and estimating it is the first response action: (max_bytes - current_size) / fill_rate.
• Fill rate can change as backpressure propagates upstream, so recalculate frequently rather than trusting one estimate.
• Drain is slower than fill. After downstream recovers, the PQ replays at whatever the workers and outputs can sustain, plus disk read I/O. High occupancy that declines slowly after recovery is normal, not a new incident.
• A full disk and a full queue are different failure modes. queue.max_bytes caps the queue, but PQ pages, DLQ files, and Logstash’s own logs can share a partition. If the filesystem fills first, the process can crash regardless of queue configuration. Watch queue.data.free_space_in_bytes, not just queue occupancy.
• Disk saturation can corrupt the queue. Operators have hit unstartable pipelines after disk-full events, requiring pqrepair or manual checkpoint cleanup to recover. Keep max_bytes comfortably below partition capacity.

For the full triage of the blocked-inputs state, see Logstash queue full: inputs blocked and the backpressure wedge.

Signals to watch for each queue type

The full signal catalogue, including severity composition rules, is in the Logstash monitoring checklist.

Choosing between them

Use the memory queue when upstream systems already provide durability or buffering (Kafka in front of Logstash is the common case, since consumer lag replaces the internal queue as your backlog signal), when the data is cheap to lose (metrics-like telemetry, sampled logs), or when you want failures to be loud and immediate rather than absorbed and delayed.

Use the persistent queue when events arrive over protocols with no upstream durable buffer (raw TCP, UDP syslog, HTTP posts that would otherwise be rejected), when you must survive Logstash restarts and downstream outages without data loss, and when you can commit to monitoring occupancy, fill rate, and the health of the disk underneath it. Size max_bytes for the longest downstream outage you expect to survive at your normal ingest rate, and keep it well under partition capacity.

The mistake is not picking either one. It is picking PQ and treating it as set-and-forget, or picking the memory queue and never writing down what a crash costs you. Where your team sits on this spectrum tends to track the levels in the Logstash monitoring maturity model.

How Netdata helps

• Netdata collects the Logstash node stats API per pipeline, so queue metrics (events_count, queue_size_in_bytes, max_queue_size_in_bytes, data.free_space_in_bytes) are graphed continuously rather than sampled during an incident.
• Flow metrics such as flow.queue_backpressure and flow.queue_persisted_growth_bytes are plotted alongside input and output throughput, which is the correlation that separates a backpressure wedge from an upstream failure.
• Because Netdata also monitors the host, PQ occupancy can be correlated with disk space, disk I/O latency, and I/O wait on the same dashboard. This catches the “filesystem fills before the queue limit” failure mode that queue metrics alone miss.
• Per-pipeline views keep multi-pipeline deployments honest: one pipeline’s PQ filling does not get averaged away by healthy neighbors.
• Anomaly detection on output throughput and queue growth surfaces the slow-burn pattern (PQ masking a downstream outage) before occupancy reaches page-worthy levels.

Related guides

• How Logstash actually works in production: a mental model for operators
• Logstash monitoring checklist: the signals every production pipeline needs
• Logstash monitoring maturity model: from survival to expert
• Logstash queue full: inputs blocked and the backpressure wedge