$ guides / logstash / logstash-dead-letter-queue-growing ▌

Operations Guides

Logstash dead letter queue growing: DLQ diversion, replay, and disabled-by-default risk

A growing dead_letter_queue.queue_size_in_bytes means an output is permanently rejecting events and diverting them to the DLQ instead of delivering them downstream. This is a data-integrity failure, not just a metric ticking up. Every event in the DLQ is data your destination never received.

Two design properties compound the risk. The DLQ is disabled by default in all Logstash versions; many production deployments run without it, meaning permanently failed events are logged and silently lost with no on-disk record. And the DLQ only captures specific output failure classes. Filter and parse failures (grok, json, date) do not go to the DLQ. They pass through the pipeline with _grokparsefailure or similar tags and reach the output as malformed events.

When enabled, the DLQ defaults to a 1 GB maximum. Once full, storage_policy controls behavior: drop_newer (default) stops accepting new events; drop_older evicts the oldest entries. In both cases, a full DLQ means the safety net is itself discarding data.

What this means

The metric pipelines.<pipeline_id>.dead_letter_queue.queue_size_in_bytes reflects on-disk DLQ size. A rising value means events are accumulating because an output permanently rejected them after exhausting retries. The DLQ is not a retry queue. Events sit there until a human replays them, deletes them, or the queue fills and the storage policy kicks in.

What lands in the DLQ:

Events rejected by the Elasticsearch output with non-retriable HTTP response codes (400, 404). These are permanent failures: mapping conflicts, type mismatches, malformed documents the destination cannot accept.
Events that fail conditional statement evaluation in the pipeline configuration.

What does NOT land in the DLQ:

Filter and parse failures. A grok pattern that fails to match tags the event with _grokparsefailure and sends it through the normal output path. The DLQ never sees it.
Transient output errors. Connection timeouts, HTTP 429 rate limiting, and temporary unavailability are retried by the output plugin. Only permanent rejections after retries are exhausted go to the DLQ.

Official docs confirm DLQ support is limited to the Elasticsearch output and conditional-statement evaluation; other output types (Kafka, S3, HTTP) never write to the DLQ.

flowchart LR
    A[Input] --> B[Queue]
    B --> C[Workers + Filters]
    C -->|parse fail| D[Tagged event
bypasses DLQ]
    C -->|parsed OK| E[Output plugin]
    D --> E
    E -->|accepted| F[Destination]
    E -->|permanent reject
HTTP 400 or 404| G{DLQ enabled?}
    G -->|no| H[Logged and
silently lost]
    G -->|yes| I[DLQ on disk]
    I -->|full| J[drop_newer or
drop_older]
    I -->|manual replay| K[Replay pipeline]
    K --> C

The diagram shows the two paths that matter most. Parse failures bypass the DLQ entirely, reaching the destination as malformed data. If the DLQ is disabled, permanently rejected events vanish with only a log entry.

Common causes

Cause	What it looks like	First thing to check
Elasticsearch mapping conflict	Bulk request returns 400 with per-document errors; DLQ grows steadily	ES output `documents.non_retryable_failures` counter
Schema or type mismatch at source	New field appears with wrong type (string vs integer); specific events always fail	Sample DLQ content and compare against current index mapping
Poison-pill events	Small number of specific events repeatedly fail; DLQ grows in small bursts	Inspect a few DLQ entries for common field patterns
Index write block	ES index set to read-only (disk watermark exceeded); all writes fail	ES cluster health and index settings
Version incompatibility	After ES upgrade, some documents fail due to breaking mapping changes	ES version and recent upgrade history

Quick checks

# Check DLQ size and max via the stats API
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | grep -A 5 dead_letter_queue

# Check DLQ size on disk
du -sh /var/lib/logstash/dead_letter_queue/*/

# Check if DLQ is enabled and its settings
grep -E 'dead_letter_queue\.(enable|max_bytes|storage_policy)' /etc/logstash/logstash.yml

# Check ES output document-level failure counts
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty \
  | python3 -c "
import sys,json
outputs = json.load(sys.stdin)['pipelines']['main']['plugins']['outputs']
for o in outputs:
    name = o.get('name','?')
    docs = o.get('documents',{})
    print(f'{name}: non_retryable_failures={docs.get(\"non_retryable_failures\",\"N/A\")}')
"

# Check for rejection and mapping errors in logs
grep -Ei '(reject|mapping|error|exception|failed)' /var/log/logstash/logstash-plain.log | tail -n 100

# Check Elasticsearch cluster health
curl -s localhost:9200/_cluster/health?pretty

# Check the grok failures counter (parse failures bypass the DLQ)
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty \
  | python3 -c "
import sys,json
filters = json.load(sys.stdin)['pipelines']['main']['plugins']['filters']
for f in filters:
    if f.get('name') == 'grok':
        print(f\"grok ({f.get('id','?')}): failures={f['events'].get('failures', 0)}\")
"

How to diagnose it

Confirm DLQ is enabled and growing. Query the stats API for dead_letter_queue.queue_size_in_bytes. Take two samples 60 seconds apart. If the value is rising, events are actively being diverted. If DLQ is not enabled, check logs for output rejection messages: those events were silently lost.
Check the Elasticsearch output for non-retryable failures. In the stats API response, look at plugins.outputs[] for documents.non_retryable_failures or bulk_requests.failures. These tell you the destination is permanently rejecting events.
Sample DLQ content. The DLQ stores events in a binary format at <path.data>/dead_letter_queue/<pipeline_id>/. Replay a few events through a temporary pipeline using the dead_letter_queue input plugin to inspect what is failing. Each DLQ entry includes the original event, the reason for failure, and the failure timestamp.
Identify the pattern. Are all events failing (systemic issue like mapping conflict or index write block), or only some (poison-pill events or specific field values)?
Check the destination mapping. Compare fields in failing events against the Elasticsearch index mapping. A field defined as long in the mapping receiving a string value is a classic mapping conflict.
Check for recent source format changes. If the source application changed its log format, new fields or changed field types may conflict with the existing mapping.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`dead_letter_queue.queue_size_in_bytes`	Primary DLQ growth indicator	Any sustained increase above zero
`dead_letter_queue.max_queue_size_in_bytes`	Configured DLQ capacity	Queue size approaching this limit means data will be discarded
ES output `documents.non_retryable_failures`	Events the destination permanently rejected	Any non-zero and growing value
Pipeline `events.out`	Whether events are flowing at all	Normal output rate with DLQ growth equals silent partial failure
Grok filter `failures` counter	Parse failures that bypass DLQ entirely	Rising rate means data quality is degrading without DLQ involvement
Disk usage on DLQ partition	DLQ competes with PQ and logs for space	Shared partition filling up can crash Logstash

Fixes

Fix the root cause

The DLQ is a symptom, not the problem. The problem is that specific events cannot be accepted by the destination. Fix options depend on the cause:

Elasticsearch mapping conflict. Update the index mapping to accept the new field type, or transform the event in a Logstash filter to match the expected type. If the conflict comes from a new field the source should not be sending, drop or rename it before the output.

Index write block. If Elasticsearch set the index to read-only due to disk watermarks, free disk space on the data nodes. The write block lifts automatically once the disk watermark recovers.

Poison-pill events. If a small number of malformed events are causing rejections, add a filter condition to catch and route them before they reach the output. Quarantine them to a separate index for investigation.

Replay events from the DLQ

DLQ events do not automatically retry. To replay them, create a temporary pipeline that reads from the DLQ using the dead_letter_queue input plugin:

input {
  dead_letter_queue {
    path => "/var/lib/logstash/dead_letter_queue"
    pipeline_id => "main"
    commit_offsets => true
  }
}

The commit_offsets option (default true) records the read position so events are not re-read on restart. Since Logstash 8.4.0, the clean_consumed option automatically deletes consumed segments, keeping the DLQ from growing during replay.

The queue_size_in_bytes accounting fix for clean_consumed replays shipped in Logstash 8.15.0 (elastic/logstash#16195); earlier versions can report a stale DLQ size while consumed segments are deleted.

Replay risks. Replay re-sends events to the output. If the root cause is not fixed, the same events will fail again and re-enter the DLQ. Always confirm the root cause is resolved before replaying. Replay also adds load to the destination, which may already be under pressure.

Clear the DLQ

If the DLQ contains events you choose not to replay, you can clear it manually:

There is no online clearing path: the official docs require stopping the pipeline before deleting the DLQ directory, and no newer version has added an API to clear it while running.

Stop the Logstash pipeline or the entire Logstash process.
Delete the DLQ directory: rm -rf <path.data>/dead_letter_queue/<pipeline_id>
WARNING: This is destructive and unrecoverable. Confirm the pipeline is fully stopped before deleting. Verify <path.data> resolves correctly before running the command.
Restart Logstash.

The DLQ directory cannot be deleted while the pipeline is running. The path follows the pattern <path.data>/dead_letter_queue/<pipeline_id>/.

Enable the DLQ if it is disabled

If you discover DLQ is disabled on a production pipeline, permanently failed events have been silently lost. There is no way to recover them retroactively. Enable the DLQ going forward:

In logstash.yml:

dead_letter_queue.enable: true
dead_letter_queue.max_bytes: 1024mb
dead_letter_queue.storage_policy: drop_newer

The storage_policy option requires Logstash 8.3.0 or later (verified: the setting was added in 8.3.0). On earlier versions, the behavior is always drop_newer. Restart Logstash for the change to take effect.

Prevention

Monitor DLQ growth even when throughput looks healthy. A pipeline can show normal events.out while events are silently diverted. The output counter increments for successful documents in a partial bulk response, masking rejected ones.
Alert on any DLQ growth. Any non-zero growth rate on a production pipeline warrants investigation. The question is not “is the DLQ full?” but “why is even one event being diverted?”
Track DLQ capacity, not just size. Monitor queue_size_in_bytes relative to max_queue_size_in_bytes. A DLQ at 90% capacity with drop_newer is one event away from permanent data loss.
Do not confuse DLQ with parse failure monitoring. The DLQ catches output rejections. Parse failures are invisible to the DLQ. Monitor the grok filter failures counter and parse failure tags at the destination separately.
Prepare a replay pipeline before you need it. Most teams never replay DLQ events because they have no replay pipeline ready. Build and test the configuration in staging, and document the procedure.
Watch disk space. The DLQ competes with the persistent queue, log files, and the OS for the same partition. A growing DLQ can fill a disk and crash Logstash even if the queue is within its configured limit.

How Netdata helps

Per-second collection of dead_letter_queue.queue_size_in_bytes catches growth the moment it starts, before the DLQ fills and the storage policy begins discarding events.
Correlating DLQ growth with Elasticsearch output error rates (documents.non_retryable_failures, bulk request failures) pinpoints whether the destination is rejecting events.
ML anomaly detection on DLQ size flags unexpected growth patterns that static thresholds miss, especially on pipelines with intermittent DLQ activity.
Disk space monitoring on the DLQ partition warns when the queue competes with PQ and log volumes for the same filesystem.
Pipeline throughput metrics alongside DLQ size reveal the silent partial failure pattern where throughput looks normal but events are being diverted.
Grok filter failures counter monitoring catches parse failures that bypass the DLQ, covering the correctness gap the DLQ does not.