$ guides / logstash / logstash-silent-data-loss ▌

Operations Guides

Logstash silent data loss: events dropped with no DLQ and green dashboards

The first sign is usually a question from downstream: “Where are last Wednesday’s logs?” Your Logstash dashboards show green. Process liveness is fine. Events-in and events-out curves track each other. The queue is empty. GC is quiet. Nothing paged.

Silent data loss happens when events leave the Logstash accounting system as “delivered” but never reach their destination in usable form. The events.out counter increments, throughput looks normal, and no alert fires. By the time anyone notices, the loss window may be hours or days old and the evidence is gone.

This guide covers the loss vectors that produce this pattern, the signals that expose them (not the standard health metrics), and how to instrument for silent loss before someone asks.

What this means

There are several distinct vectors:

Output rejection with DLQ disabled. When DLQ is off (the default in every Logstash version) and an event permanently fails at the output, Logstash logs a warning and drops it. The event leaves the pipeline. No counter captures the loss.
Partial Elasticsearch bulk failure. The Elasticsearch Bulk API returns HTTP 200 even when individual documents fail. The ES output plugin tracks per-document failures in its own stats, but the pipeline-level events.out counter still increments for the entire batch. Without DLQ, rejected documents are gone.
DLQ drop_newer eviction. When the DLQ reaches its size limit (default 1 GB) and storage_policy is drop_newer (the default), new failed events are silently discarded. The only signal is a dropped-event counter that most teams do not monitor.
Memory queue loss on crash. With the default memory queue, any unclean shutdown (OOM kill, SIGKILL, node failure) loses all in-flight events. After restart, throughput resumes from the last input checkpoint with no metric indicating the gap.
Persistent queue checkpoint gap. Even with PQ enabled, events written to the head page but not yet checkpointed are lost on crash. With the default queue.checkpoint.writes of 1024, up to 1024 events can vanish per crash (the default is 1024 on both Logstash 7.x and 8.x; 0 means no event-count-based checkpointing).
Codec and filter drops. A multiline codec that never sees its termination condition holds a partial event indefinitely and never emits it. Parse failures tag events with _grokparsefailure and deliver them with raw, unstructured data. The event “arrives” but is useless.
Sincedb re-reads (duplication). If the file input’s sincedb state is corrupted or lost, Logstash re-reads files from an earlier position. This produces duplicates rather than loss, but it contributes to the “where are my logs?” confusion.

flowchart TD
    A[Input plugin] --> B{Codec parse}
    B -- fail --> C[Silent drop or stuck buffer]
    B -- ok --> D[Queue]
    D -- memory mode, crash --> E[Lost on crash]
    D -- PQ mode, uncheckpointed --> F[Lost on crash]
    D -- drained --> G[Filter chain]
    G -- parse fail --> H[Tagged and delivered unusable]
    G -- ok --> I[Output plugin]
    I -- bulk 200, per-doc fail --> J[Counted as out, lost]
    I -- permanent fail, no DLQ --> K[Silently dropped]
    I -- permanent fail, DLQ on --> L[DLQ]
    L -- full, drop_newer --> M[Silently dropped]

Common causes

Cause	What it looks like	First thing to check
DLQ disabled, permanent output failures	Events-out steady, no DLQ growth, warning lines in logstash-plain.log	Grep log for output rejection warnings; confirm `dead_letter_queue.enable` in logstash.yml
ES partial bulk failure	Bulk request returns 200, document-level failures in output plugin stats	Per-output `documents` and `bulk_requests` stats in `_node/stats/pipelines`
DLQ drop_newer eviction	DLQ at max size, `dropped_events` counter rising	DLQ `queue_size_in_bytes` vs `max_queue_size_in_bytes`
Memory queue crash loss	Recent restart (short JVM uptime), no gap in throughput metrics	JVM uptime vs known incident window; compare source counts to destination
PQ checkpoint gap	Recent unclean shutdown, small count discrepancy	`queue.checkpoint.writes` setting; recent OOM or SIGKILL in dmesg
Codec or filter drop	Throughput steady, but destination has fewer usable events	Parse failure tag count at destination; multiline codec config
Event stats inconsistency (pre-8.0.0)	`events.in` equals `events.out` even with active drop filters	Logstash version; check if below 8.0.0 or 7.17.0

Quick checks

All read-only and safe to run during an incident.

# Check DLQ status in pipeline stats (stats appear even when DLQ is disabled)
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | grep -A 15 dead_letter_queue

# Check ES output document-level stats for partial failures
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | grep -A 30 '"outputs"'

# Check grok filter failures counter
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | grep -A 10 '"failures"'

# Check for output rejection warnings in the log
grep -Ei '(retry|error|exception|failed|reject|unavailable|timeout|429|503)' /var/log/logstash/logstash-plain.log | tail -n 200

# Check JVM uptime for recent restarts (memory queue loss indicator)
curl -sS http://127.0.0.1:9600/_node/stats/jvm?pretty | grep uptime_in_millis

# Check DLQ size on disk
du -sh /var/lib/logstash/dead_letter_queue 2>/dev/null

# Check for recent OOM kills
dmesg -T 2>/dev/null | grep -i 'killed process' | tail

# Count parse failure tags at the destination (Elasticsearch example)
# NOTE: adjust auth/credentials for your ES cluster
curl -s 'localhost:9200/<index>/_count?q=tags:_grokparsefailure'

How to diagnose it

Confirm DLQ is enabled. Check dead_letter_queue.enable in logstash.yml or per-pipeline config. The pipeline stats API shows DLQ metrics even when disabled, so verify the config directly. If DLQ is disabled, every permanent output failure is a silent drop. This is the single most common cause.
Check ES output document-level stats. Look at the Elasticsearch output plugin stats in _node/stats/pipelines. The plugin tracks per-document outcomes separately from the pipeline-level events.out counter. documents.non_retryable_failures counts events that were rejected and, without DLQ, lost. bulk_requests.with_errors counts batches that returned 200 but had per-document failures. The exact paths under _node/stats/pipelines/<pipeline> are plugins.outputs[].bulk_requests.{successes,failures,with_errors} and plugins.outputs[].documents.{successes,non_retryable_failures,retryable_failures}.
Check DLQ growth and eviction counters. Compare queue_size_in_bytes to max_queue_size_in_bytes. If the DLQ is at capacity, look for a dropped_events counter. With drop_newer (the default), the DLQ silently discards new failures when full. The dropped counter was exposed in the monitoring API starting in Logstash 8.3.0 (elastic/logstash PR #14058); earlier versions offer no API signal for DLQ eviction.
Check for recent restarts. A short JVM uptime means a recent restart. With the memory queue, all in-flight events were lost. With PQ, events not yet checkpointed were lost. Check dmesg and journalctl for OOM kills or SIGKILL around the restart time.
Compare input and output counts against your transformation ratio. Know how many events your filters drop, clone, or split. A pipeline with no transforms should see events.in equal events.out. A pipeline that drops 10% of events should see a stable 10% gap. A growing gap that exceeds queue depth plus in-flight batches indicates unaccounted events. Note: on Logstash versions before 8.0.0 (or 7.17.0), the events.out counter was incremented even for events dropped by drop filters, making this comparison unreliable; fixed in Logstash 8.0.0 and backported to 7.17.0 (elastic/logstash PR #13593).
Check parse failure indicators. Query the destination for _grokparsefailure, _jsonparsefailure, and _dateparsefailure tags. A rising rate means events arrive in unusable form. The grok filter’s failures counter in per-plugin stats is the most direct signal.
Check log warnings for output rejections. Grep the Logstash log for rejection, retry, and mapping error patterns. These lines are often the only record of dropped events when DLQ is off.
Reconcile source-side counts to destination counts. The definitive test. Compare Filebeat registry offsets, Kafka consumer lag, or source log line counts against Elasticsearch document counts for the same time window.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`dead_letter_queue.queue_size_in_bytes`	Growing DLQ means events are failing delivery	Any sustained growth above zero
`dead_letter_queue.dropped_events`	DLQ at capacity is silently discarding failures	Any non-zero value (8.3.0+)
ES output `documents.non_retryable_failures`	Per-document rejections that bypassed DLQ	Any non-zero value
ES output `bulk_requests.with_errors`	Partial bulk failures hidden by batch-level 200	Sustained non-zero rate
`plugins.filters[].failures` (grok)	Parse failures producing unusable events	Rate increase above baseline
`events.in` vs `events.out` divergence	Unaccounted events entering but not leaving	Growing gap beyond what your transformation ratio explains
JVM uptime	Recent restart means potential memory queue loss	Unexpected reset
Parse failure tag count at destination	Data quality loss invisible to throughput metrics	Sudden spike, especially after source format change

Fixes

Enable and size the DLQ

Set dead_letter_queue.enable: true in logstash.yml or per-pipeline config. Set dead_letter_queue.max_bytes high enough to absorb the longest realistic output outage. Consider storage_policy: drop_older if losing the newest failures is worse than evicting old ones, but understand that drop_older discards the oldest unreplayed events instead.

This stops the most common silent drop but does not fix the root cause. Monitor the DLQ and replay it.

Address partial ES bulk failures

Partial bulk failures usually indicate mapping conflicts (field type mismatches) or index write blocks. Check the Elasticsearch mapping for conflicting field types across indices. Fix the mapping or route conflicting events to a separate index.

Replay the DLQ

DLQ events do not auto-retry. Run a separate pipeline with the dead_letter_queue input plugin to replay them. The clean_consumed option auto-removes consumed segments. Without it, DLQ files accumulate and must be cleared manually by stopping the pipeline and deleting the files.

Switch to persistent queue for crash durability

Enable PQ (queue.type: persisted) to survive clean restarts. For unclean shutdowns, reduce queue.checkpoint.writes from the default 1024 to a lower value to shrink the checkpoint gap, at the cost of disk I/O. Setting it to 1 eliminates the gap but significantly increases write overhead.

Fix codec and parse failure drops

For multiline codec issues, ensure the timeout and max_lines settings cover your log patterns. For grok failures, update patterns to handle the new source format, or add a catch-all pattern that tags but preserves raw events. Route parse failures to a separate index for investigation rather than dropping them.

Prevention

Enable DLQ on every production pipeline. The default is off in every Logstash version.
Monitor DLQ size, growth, and dropped-event counters. A DLQ that grows and then stops growing may have started evicting.
Monitor ES output document-level stats. documents.non_retryable_failures and bulk_requests.with_errors expose partial failures that events.out hides.
Track the in/out ratio per pipeline. Know your expected transformation ratio. Alert on divergence.
Reconcile source counts against destination counts periodically. This is the only definitive test for silent loss. Automate it if possible.
Upgrade past 8.0.0 (or 7.17.0). Earlier versions have an event stats bug that made events.out count events dropped by drop filters as delivered; fixed in Logstash 8.0.0 and backported to 7.17.0 (elastic/logstash PR #13593).

How Netdata helps

Per-second metrics on events.in, events.out, and events.filtered let you compute the in/out ratio at high resolution. A growing divergence that 60-second polling would smooth over becomes visible.
DLQ queue_size_in_bytes tracking with anomaly detection surfaces DLQ growth before it hits the size limit and starts evicting.
Grok filter failures counter collected per-plugin catches parse failure spikes at the source, before they propagate to the destination as unusable events.
JVM uptime and restart detection flags memory queue loss windows. Correlating a restart timestamp with a throughput gap confirms crash-related loss.
Per-pipeline stats, not just aggregates, prevent one failing pipeline from being averaged away in a multi-pipeline deployment.
Correlation across the stack (Logstash throughput, Elasticsearch indexing rate, Filebeat registry progress) makes end-to-end count reconciliation faster during an investigation.