Logstash won't start after a crash: persistent queue corruption and checkpoint errors

Logstash was killed uncleanly (OOM kill, kill -9, SIGKILL after a pod termination grace period, power loss) and now refuses to start. The log at /var/log/logstash/logstash-plain.log shows java.io.IOException and checkpoint-related errors during pipeline initialization, and the process exits before any events flow.

The persistent queue (PQ) is a page-based on-disk queue: events live in page files (default 64MB each, up to queue.max_bytes which defaults to 1GB), and checkpoint files record which events have been acknowledged as delivered. Both are updated continuously while the pipeline runs. An unclean shutdown can leave the checkpoint files and page files inconsistent with each other, and Logstash’s startup code refuses to open a queue it cannot reconcile.

Every recovery path involves a tradeoff between re-delivery (duplicates downstream) and loss (gaps downstream). There is no zero-impact repair, because the corrupted state is precisely the record of what has been delivered. Decide which side of that tradeoff your pipeline can tolerate before deleting or moving files.

If this deployment uses the default memory queue instead of PQ, this failure mode does not exist. A crash with a memory queue loses the in-flight events silently and Logstash starts cleanly. If you are seeing startup failure with these errors, PQ is enabled.

What this means

The PQ has two kinds of state on disk under the queue directory (typically /var/lib/logstash/queue/<pipeline_id>/):

• Page files (page.N) hold the serialized events, written sequentially.
• Checkpoint files (checkpoint.head, checkpoint.N) record which pages and events have been fully processed and acknowledged. Checkpoints are written periodically (by default after 1024 writes or 1024 acks, or every 1000ms), not on every event.

On a clean shutdown, the final checkpoint reflects the true state of the queue. On an unclean shutdown, the last checkpoint write may be torn, stale, or missing entirely. Logstash validates this on startup and fails closed rather than guessing.

The direction of the inconsistency determines the failure consequence:

• Checkpoint behind actual state: events that were already delivered look unacked. Recovery replays them, so downstream sees duplicates. PQ is at-least-once by design, so this is the “safe” direction.
• Corrupted or unreadable pages: events in those pages cannot be recovered. They are gone, and repair means discarding the damaged segments.
• Corrupted or zero-byte head checkpoint: Logstash cannot determine queue state and aborts startup. This is the most common crash artifact.

For the broader model of how the queue sits between inputs and workers, see How Logstash actually works in production and Logstash memory queue vs persistent queue.

Common causes

One non-corruption case worth ruling out first: if you recently upgraded to Logstash 9.2.0 and set queue.max_bytes to 2147483648 (2 GiB) or more, the process will not start even with a perfectly healthy queue. That is a documented known issue with no workaround other than downgrading. If the log shows checkpoint or IOException errors rather than a config validation failure, you are dealing with real corruption instead.

Quick checks

All read-only. Do not delete anything yet.

# 1. Confirm the failure reason from the log
grep -Ei '(IOException|checkpoint|queue|persistent)' /var/log/logstash/logstash-plain.log | tail -n 50

# 2. Confirm how Logstash died last time (root cause matters later)
journalctl -u logstash --since "2 hours ago" | tail -n 100
dmesg -T | grep -i -E '(oom|killed process)' | tail -n 20

# 3. Check queue type and PQ settings
grep -E 'queue\.(type|max_bytes|page_capacity|checkpoint)' /etc/logstash/logstash.yml

# 4. Inspect the queue directory contents (sizes, zero-byte files)
ls -la /var/lib/logstash/queue/*/

# 5. Check disk state on the PQ volume
df -h /var/lib/logstash
du -sh /var/lib/logstash/queue/*/

# 6. Verify Logstash is actually stopped before any repair
pgrep -f org.logstash.Logstash

If running in a container, replace journalctl -u logstash with your orchestrator’s pod event log or kubectl describe pod / docker inspect.

In step 4, look for a zero-byte checkpoint.head. That alone is enough to block startup. A checkpoint.head.tmp alongside checkpoint.head indicates a checkpoint write was interrupted mid-rename, and the .tmp file may be the newer, intact copy.

How to diagnose it

1. Identify the exact error. The log line tells you which file is bad. A checkpoint checksum mismatch points at checkpoint.head. Errors referencing page numbers point at page files. The pipeline ID in the error tells you which queue subdirectory is affected in multi-pipeline setups.

2. Run pqcheck. Logstash ships a diagnostic utility that reads the queue offline. Logstash must be stopped. For package installs (RPM/DEB), the tool is at /usr/share/logstash/bin/pqcheck; for tarball installs, it is under bin/ in the extraction directory.

# Point pqcheck at the affected pipeline's queue dir
/usr/share/logstash/bin/pqcheck /var/lib/logstash/queue/main

pqcheck walks the checkpoint files and reports each page’s state: page numbers, the first unacknowledged page and event, and whether pages are fully acknowledged.

A page whose size reports as “NOT FOUND” is corrupted. This tells you whether you have a checkpoint-only problem (cheap to fix, possible duplicates) or damaged pages (data loss is already real, repair just bounds it).

3. Map damage to impact. If only the head checkpoint is bad and all pages check out, recovery risks re-delivery of recently acknowledged events but no loss. If pqcheck reports corrupted pages, the events in those pages are unrecoverable; the decision is how much surrounding state you discard to get a bootable queue.

4. Check the root cause before recovering. If the OOM killer or a full disk caused the crash, recovering the queue without fixing that guarantees a repeat. Confirm heap sizing, container memory limits, and free space on the PQ volume now, not after the next crash.

The recovery decision tree:

flowchart TD
    A[Logstash fails to start with PQ errors] --> B[Back up the queue directory]
    B --> C[Run pqcheck on the queue dir]
    C --> D{What is damaged?}
    D -->|Head checkpoint only| E[Restore checkpoint.head from checkpoint.head.tmp, or delete checkpoint files]
    D -->|Corrupted pages| F[Run pqrepair to remove corrupt segments]
    D -->|pqrepair fails or damage too broad| G[Move queue dir aside, start fresh]
    E --> H[Start Logstash, expect some re-delivery]
    F --> H2[Start Logstash, expect gaps in corrupt pages]
    G --> H3[Start Logstash, queued events in old dir are lost]
    H --> I[Fix unclean-shutdown root cause]
    H2 --> I
    H3 --> I

Metrics and signals to monitor

You cannot watch these during the outage (the stats API on port 9600 is down with the process), but they are the signals that tell you the next crash is coming, and they confirm recovery afterwards.

Note the PQ sizing subtlety: max_bytes limits the queue’s logical size, but pages are allocated in fixed increments (default 64MB) and freed only when fully drained and checkpointed, so on-disk usage can exceed the configured limit. Size max_bytes against the partition, not against your comfort level.

Fixes

Every step below is destructive in different ways. Back up the queue directory before any of them, with Logstash stopped. Ensure the backup volume has enough free space for a copy of the full queue:

# Stop Logstash, then back up the queue for forensics or a second attempt
systemctl stop logstash
# In a container: docker stop / kubectl delete pod instead
cp -a /var/lib/logstash/queue /var/lib/logstash/queue.backup-$(date +%Y%m%d-%H%M)

Fix 1: repair with pqrepair (preferred when pages are corrupt)

# Removes corrupt queue segments. Run per pipeline queue dir. Use the logstash user
# so file ownership stays consistent. Path shown is for package installs.
sudo -u logstash /usr/share/logstash/bin/pqrepair /var/lib/logstash/queue/main

pqrepair produces no output on success. It discards corrupted segments, which means the events in those pages are lost; the rest of the queue survives and will drain normally. This is the least-bad option when pqcheck shows real page damage, because it preserves everything readable.

Fix 2: restore or remove the head checkpoint (checkpoint-only corruption)

If pqcheck shows healthy pages and the log points at the head checkpoint:

• If a checkpoint.head.tmp exists, it may be the newer intact copy. Logstash writes each checkpoint through a temporary .tmp file that is then atomically renamed over the target, so an interrupted write can leave the old checkpoint.head beside a newer, intact .tmp. Restoring the .tmp over checkpoint.head is a reasonable recovery when pqcheck confirms the page files are healthy and only the head checkpoint is at fault.

• If there is no usable .tmp file, or the head checkpoint is zero bytes, delete the checkpoint files. Logstash will rebuild checkpoint state from the page files on startup.

The tradeoff: rebuilt checkpoints treat already-delivered events as unacknowledged, so downstream will see some duplicates. Most Elasticsearch-destined pipelines tolerate this (duplicate documents overwrite by ID or appear as dupes); measure it against your destination’s dedup behavior.

Fix 3: move the queue aside (last resort)

# WARNING: queued events in the old directory will NOT be delivered.
mv /var/lib/logstash/queue/main /var/lib/logstash/queue/main.corrupt
systemctl start logstash

Logstash creates a fresh empty queue and starts. Everything that was queued is lost from Logstash’s perspective. Whether that data is truly gone depends on your sources: Kafka inputs can re-consume (consumer offsets are in Kafka, not the PQ), Beats sources resend from their own registry, but fire-and-forget sources (UDP syslog, some HTTP) are gone for good.

Fix 4: fix the root cause

Recovery without this step is a scheduled repeat incident:

• OOM kill: raise container/host memory or lower JVM heap so RSS fits with headroom. Heap plus off-heap overhead should sit comfortably under the limit; heap alone is not the whole footprint.
• Force kills: give Logstash a real shutdown window. For planned maintenance on PQ deployments, queue.drain: true drains the queue before shutdown so nothing is in flight. In Kubernetes, size the termination grace period around drain time, not the default 30s.
• Disk full: separate the PQ volume from log and DLQ storage, or lower max_bytes relative to the partition.
• Config trigger: if you set queue.max_bytes at or above 2 GiB on Logstash 9.2.0, downgrade; that version will not start regardless of queue health.

Prevention

• Eliminate unclean shutdowns. They are the only common trigger for this failure. That means memory limits with headroom, graceful stops with adequate grace periods, and no kill -9 in runbooks.
• Alert on the pre-crash signals from the table above: rising post-GC heap floor, GC overhead, RSS near limit, PQ volume free space. The crash is the last step of a visible trend.
• Keep PQ on local, reliable storage. NFS and slow network storage under the PQ amplify both latency and corruption risk.
• Size max_bytes against the partition with room for page-allocation overshoot, and keep the PQ partition dedicated.
• Know your source replay capability before an incident. If sources cannot replay (UDP, transient HTTP), the blast radius of a queue wipe is permanent loss, which should factor into how aggressively you protect the queue volume.
• Expect re-delivery after checkpoint recovery. Make sure downstream consumers tolerate duplicates, because at-least-once is the PQ’s design contract even in the best-case recovery.

How Netdata helps

• Netdata’s Logstash collector polls the node stats API, so heap sawtooth, old-gen pool usage, and GC collection time are charted per second. A rising post-GC floor is visible hours before the OOM kill that corrupts the queue.
• PQ occupancy, queue_size_in_bytes against max_queue_size_in_bytes, and persisted growth rate show how much in-flight state a crash would put at risk, and how full the queue was when it happened.
• Correlating JVM metrics with host-level signals (RSS against cgroup limits, disk free space and I/O on the PQ volume, OOM kill events) connects the unclean shutdown to its cause in one view instead of three tools.
• JVM uptime resets and API reachability gaps make restart loops and crash timing obvious, which helps line the crash up with orchestrator events.
• After recovery, output throughput against input throughput confirms the queue is draining and the pipeline is actually processing, not just running.

Related guides

• Logstash flow.queue_backpressure: the input-throttling metric explained
• How Logstash actually works in production: a mental model for operators
• Logstash memory queue vs persistent queue: durability, visibility, and failure modes
• Logstash monitoring checklist: the signals every production pipeline needs
• Logstash monitoring maturity model: from survival to expert
• Logstash pipeline stalled: output rate at zero while the process looks alive
• Logstash queue events count growing: reading the in-flight backlog
• Logstash queue full: inputs blocked and the backpressure wedge