Logstash Elasticsearch partial bulk failure: HTTP 200 with per-document errors and silent data loss

Your Logstash pipeline looks healthy. The process is up, the API responds on port 9600, events.out is climbing steadily, and the queue is empty. But someone asks why last Tuesday’s application logs are missing from Elasticsearch. You run a count query against the index and the numbers do not add up. Logstash says it delivered 50 million events. Elasticsearch shows 48 million documents. Two million vanished.

The Elasticsearch bulk API returns HTTP 200 as long as it successfully received and processed the request at the transport level. Individual documents within that batch may still fail with mapping conflicts, type errors, or version conflicts. Logstash counts the batch as delivered. The events.out counter goes up. The queue stays flat. Every standard monitoring signal reads green. But some documents were rejected at the index level and either went to the Dead Letter Queue (disabled by default), were logged and dropped, or were silently discarded.

The only way to detect this is to look at signals most teams never instrument: per-document success and failure counts in the Elasticsearch output plugin stats, DLQ growth, per-document error lines in the Logstash log, and ES-side indexing metrics. Throughput metrics will not help you here.

What this means

The Elasticsearch output plugin sends events in bulk batches. Each bulk request is a single HTTP POST containing multiple indexing actions. Elasticsearch processes each document independently and returns a response with per-item status codes. The overall HTTP response status is 200 when the bulk endpoint accepts the request, even if some documents within it failed.

From Logstash’s perspective, a 200 response means the batch was delivered. The pipeline increments events.out and moves on. But the response body may contain per-item errors. What happens to failed documents depends on the error type and your configuration:

• Retryable errors (429, 503): The plugin retries the individual failed actions. Logstash logs “retrying failed action with response code: 429” or similar. These usually self-heal when the ES-side pressure subsides.
• Non-retryable errors (400, 404): These include mapping conflicts, type mismatches, and missing-index errors. If DLQ is enabled, the event is written to the DLQ. If DLQ is disabled, the event is logged and silently dropped.
• Version conflicts (409): Logged as a warning and dropped. Not retried.

The critical insight: events.out does not distinguish between fully successful batches and partially successful ones. The output plugin exposes granular counters (bulk_requests.with_errors, documents.non_retryable_failures) but these are rarely monitored.

flowchart TD
    A["Logstash sends bulk batch"] --> B["Elasticsearch processes request"]
    B --> C{"HTTP 200 response"}
    C -->|"All docs succeed"| D["events.out increments - accurate"]
    C -->|"Per-doc failures in batch"| E["events.out increments - misleading"]
    E --> F{"Error type?"}
    F -->|"429 or 503"| G["Plugin retries individual docs"]
    F -->|"400, 404 non-retryable"| H{"DLQ enabled?"}
    F -->|"409 version conflict"| I["Logged and dropped"]
    H -->|"Yes"| J["Event sent to DLQ"]
    H -->|"No"| K["Event logged and dropped"]
    K --> L["Silent data loss - no metric fires"]
    J --> M["DLQ grows - still needs monitoring"]

Common causes

Quick checks

# Check per-output document-level success and failure counters
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | \
  python3 -c "
import sys, json
data = json.load(sys.stdin)
for pname, pdata in data.get('pipelines', {}).items():
    for o in pdata.get('plugins', {}).get('outputs', []):
        name = o.get('name', '?')
        oid = o.get('id', '?')
        bulk = o.get('bulk_requests', {})
        docs = o.get('documents', {})
        print(f'Pipeline={pname} Output={name}({oid})')
        print(f'  bulk_requests: successes={bulk.get(\"successes\",0)} failures={bulk.get(\"failures\",0)} with_errors={bulk.get(\"with_errors\",0)}')
        print(f'  documents: successes={docs.get(\"successes\",0)} non_retryable_failures={docs.get(\"non_retryable_failures\",0)}')
"

If bulk_requests.with_errors is non-zero and growing, partial failures are happening right now. If documents.non_retryable_failures is rising, events are being permanently rejected.

# Search Logstash log for per-document error lines
grep -Ei '(mapper_parsing_exception|failed to parse|illegal_argument|rejected|retrying failed action)' \
  /var/log/logstash/logstash-plain.log | tail -n 50

# Check DLQ size and growth
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | \
  python3 -c "
import sys, json
data = json.load(sys.stdin)
for pname, pdata in data.get('pipelines', {}).items():
    dlq = pdata.get('dead_letter_queue', {})
    if dlq:
        print(f'Pipeline={pname} DLQ size={dlq.get(\"queue_size_in_bytes\",0)} max={dlq.get(\"max_queue_size_in_bytes\",\"N/A\")}')
"

# Check DLQ on disk
du -sh /var/lib/logstash/dead_letter_queue/*/ 2>/dev/null

# Compare Logstash events.out against ES indexed document count.
# Logstash side:
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | \
  python3 -c "
import sys, json
d = json.load(sys.stdin)['pipelines']
for k, v in d.items():
    ev = v.get('events', {})
    print(f'{k}: in={ev.get(\"in\", 0)} out={ev.get(\"out\", 0)}')
"

# ES side. Scope to a time range matching the Logstash window, not a full index count.
# A raw _count against an entire index pattern is not a meaningful comparison.
curl -sS 'http://localhost:9200/<index-pattern>-*/_count' -H 'Content-Type: application/json' -d '
{"query": {"range": {"@timestamp": {"gte": "now-1h", "lt": "now"}}}}' | \
  python3 -c "import sys, json; print(json.load(sys.stdin)['count'])"

A growing gap between Logstash events.out and ES document count is your evidence of silent loss.

How to diagnose it

1. Confirm the gap. Pull events.out from the Logstash pipeline stats API and compare it against the document count in Elasticsearch for the same time window. A small gap is normal (in-flight events, timing differences). A persistent or growing gap means events are being lost.

2. Check bulk_requests.with_errors. This counter tracks batches where at least one document failed but the overall request returned 200. Any non-zero value that increases over time confirms partial failures are occurring.

3. Check documents.non_retryable_failures. This is the most direct signal. It counts individual documents that failed permanently and will never succeed regardless of retries. Every increment is a lost event (or a DLQ entry, if enabled).

4. Inspect the Logstash log for error details. The log contains the actual ES error messages. Search for mapper_parsing_exception, failed to parse, illegal_argument_exception, and similar. When a single document in a batch fails, Logstash may log the entire bulk request including successful events, making it difficult to isolate the failing document.

5. Check DLQ status. If DLQ is enabled (dead_letter_queue.enable: true), check whether queue_size_in_bytes is growing. Non-zero DLQ growth confirms events are being diverted instead of delivered. If DLQ is disabled, every non-retryable failure is a permanent loss with only a log line as evidence.

6. Check ES-side metrics. Verify cluster health and thread pool rejection rates. Use _cluster/health for cluster status. Check _cat/thread_pool or _nodes/stats for write thread pool rejections. ES thread pool saturation returns 429 responses that trigger Logstash retries.

7. Sample failing events. If DLQ is enabled, use the dead_letter_queue input plugin in a temporary pipeline to read and inspect failed events. Each DLQ entry includes the original event, the failure reason, and the timestamp. This identifies which field or value caused the rejection.

8. Identify the root cause. The most common causes are mapping conflicts (a field arrives as a string when the mapping expects a number) and dynamic field limit exhaustion. Compare the failing event’s fields against the ES index mapping. Check whether a source-side change introduced new fields or changed field types.

Metrics and signals to monitor

Fixes

Fix the mapping conflict

The most common cause is a field arriving with a type that conflicts with the ES index mapping. For example, a field mapped as float receives a string value, or a field mapped as keyword receives an object.

1. Identify the conflicting field from the ES error message in the Logstash log.
2. Update the ES index template to accommodate the new field type, or normalize the field in the Logstash filter chain before it reaches the output.
3. For existing indices, you may need to reindex with a corrected mapping. New events will use the updated template.

Enable the Dead Letter Queue

If DLQ is not enabled, every non-retryable failure is permanent and silent. Enable it per-pipeline:

# In logstash.yml or pipelines.yml:
dead_letter_queue.enable: true
dead_letter_queue.max_bytes: 1024mb

DLQ does not fix the root cause, but it captures failed events for later inspection and replay instead of discarding them. DLQ growth then becomes an observable, alertable signal.

Route specific error codes to DLQ

The dlq_custom_codes option (added in plugin v11.8.0; verified against the plugin changelog) lets you route specific HTTP status codes to DLQ that would otherwise not go there. For example, routing 413 (Payload Too Large) responses to DLQ instead of retrying them indefinitely.

Use drop_error_types for known-unfixable errors

The drop_error_types option (added in plugin v12.1.0; verified against the plugin changelog) lets you list ES error types for which individual bulk actions will not be retried. This prevents Logstash from getting stuck retrying errors that will never succeed. Events matching these types are not added to DLQ.

Use silence_errors_in_log for expected errors

The silence_errors_in_log option (renamed from failure_type_logging_whitelist in plugin v11.7.0) suppresses log output for specific error types. Use this when certain errors are expected and acceptable, such as version_conflict_engine_exception in update scenarios. This reduces log noise without hiding unexpected errors.

Replay DLQ events

After fixing the mapping or filter, replay DLQ events using a temporary pipeline with the dead_letter_queue input plugin. This recovers previously lost data. Replay is a manual operation, so plan for it as part of your incident runbook.

Prevention

• Monitor documents.non_retryable_failures and bulk_requests.with_errors on every Elasticsearch output. These are the only counters that distinguish partial failure from success.
• Enable DLQ on all production pipelines. It is disabled by default. Without it, non-retryable failures are permanent and invisible.
• Alert on DLQ growth. Any unexpected growth on production data requires investigation. Set dead_letter_queue.max_bytes high enough to buffer during incidents, and monitor dropped_events (the counter that increments when DLQ itself is full and discarding entries).
• Compare Logstash events.out against ES document counts periodically. A reconciliation job that checks for divergence catches slow data loss that no real-time metric detects.
• Validate field types in the filter chain before sending to ES. Adding type checks or coercion in Logstash filters catches mapping conflicts before they reach the output.
• Track source schema changes. Most mapping conflicts originate from upstream format changes. Monitor parse failure tag rates and field cardinality drift.

How Netdata helps

• Per-second metric collection from the Logstash stats API means you can track documents.non_retryable_failures and bulk_requests.with_errors at high resolution, catching partial failure bursts that coarser polling intervals miss.
• ML-based anomaly detection on DLQ size and events.out rate flags unexpected changes without static thresholds that drift as workloads evolve.
• Correlation across layers lets you overlay Logstash output plugin stats against Elasticsearch indexing metrics and thread pool rejections in a single view, making the causal chain visible during incidents.
• DLQ growth monitoring with per-pipeline granularity catches silent data diversion before the DLQ fills and starts dropping entries.
• Log anomaly detection on Logstash log files surfaces new error patterns like mapper_parsing_exception as they appear, without requiring you to pre-define alert rules for every possible ES error string.

Related guides

• Logstash disk full: PQ, DLQ, and log volumes competing for space
• Logstash API unreachable on port 9600: crash, GC pause, or startup
• Logstash configuration drift: when the running config no longer matches the deployed one
• Logstash Beats input: Filebeat backpressure and connection health
• Logstash CPU-bound filters (grok hell): high CPU, saturated workers, growing queue
• Logstash certificate expiry: the silent, total outage no built-in metric shows