Logstash Elasticsearch mapping conflict: field type mismatches and rejected documents

A mapping conflict occurs when Logstash sends a document whose field type does not match the index mapping. A field mapped as long arrives as a string, or an object appears where a scalar was expected. Elasticsearch rejects the document at the bulk API level.

The rejection is often invisible. Elasticsearch returns HTTP 200 for bulk requests that contain per-document failures. Logstash counts the batch as events.out and moves on. If the Dead Letter Queue (DLQ) is enabled, rejected documents divert there. If it is not (the default), documents are logged at WARN and silently lost. Pipeline throughput stays green while data disappears.

flowchart TD
    A[Logstash output sends bulk request] --> B{Elasticsearch bulk API}
    B -->|HTTP 200, all docs succeed| C[Events indexed normally]
    B -->|HTTP 200, per-doc failure| D{DLQ enabled?}
    D -->|Yes| E[Rejected doc written to DLQ]
    D -->|No| F[Doc logged at WARN and silently lost]
    E --> G{DLQ full?}
    G -->|No| H[Doc retained for manual replay]
    G -->|Yes - drop_newer| I[New entries permanently discarded]

Elasticsearch enforces a schema per index. Once a field is mapped, via dynamic mapping on the first document or through an explicit index template, all subsequent values must be compatible. Incompatible documents are rejected with mapper_parsing_exception or illegal_argument_exception. Mapping conflicts do not cause backpressure or queue growth. They cause silent data loss. The only evidence lives in the DLQ, in the Logstash log file, or in Elasticsearch’s document-level rejection metrics.

Common causes

Quick checks

All commands are read-only and safe for production.

# Check for mapping-related errors in Logstash logs
grep -Ei 'mapper_parsing|mapping|illegal_argument|failed.*parse' /var/log/logstash/logstash-plain.log | tail -n 100

# Check DLQ size and growth on disk
du -sh /var/lib/logstash/dead_letter_queue/*/

# Check DLQ metrics via the stats API
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | grep -A 5 dead_letter_queue

# Check Elasticsearch output document-level failure counters
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | grep -A 10 '"outputs"'

# Check whether DLQ is enabled
grep dead_letter_queue /etc/logstash/logstash.yml

# Inspect the current mapping for the target index
curl -sS 'localhost:9200/<target-index>/_mapping?pretty'

# Check the installed index template
curl -sS 'localhost:9200/_index_template/<template-name>?pretty'

# Check field count against the limit
curl -sS 'localhost:9200/<target-index>/_field_caps?fields=*' | jq '[.fields | keys | length]'

(_field_caps on a concrete index returns that index’s fields; on an index pattern it aggregates across all matched indices — use the concrete index for a per-index field count.)

How to diagnose it

1. Confirm DLQ growth. Check dead_letter_queue.queue_size_in_bytes in the pipeline stats API. Any unexpected growth means documents are being rejected. If DLQ is disabled, check the log file for WARN entries containing rejection reasons.

2. Inspect DLQ entries for the rejection reason. DLQ entries include metadata fields: reason, plugin_id, plugin_type, and entry_time. The reason field contains the Elasticsearch error string, which names the specific field and the type conflict. Use the dead_letter_queue input plugin in a temporary pipeline to read and inspect entries.

3. Identify the conflicting field. The reason typically names the field, for example: object mapping for [host] tried to parse field [host] as object, but found a concrete value. This tells you which field is mismatched and in what direction.

4. Compare the incoming event against the index mapping. Query the target index mapping for the named field. Then inspect a sample event from the source. The mismatch should be apparent: the field is mapped as one type and arrives as another.

5. Check for recent upstream changes. A deployment that altered a log format, added a nested field, or changed a numeric field to a string is the most common trigger. If the issue appeared after a Logstash 7.x to 8.x upgrade, verify pipeline.ecs_compatibility (8.x default is enabled). ECS changes field names and types; templates generated for 7.x may not match ECS-shaped events.

6. Check for template installation failures. Look for log entries about template installation. If the template failed to install (format mismatch between template_api setting and actual template structure, or incompatible field types for the target ES version), the index falls back to dynamic mapping, which may conflict with the data.

Metrics and signals to monitor

(Metric names confirmed in the Elasticsearch output plugin: bulk_requests.successes / bulk_requests.with_errors, and documents.successes / documents.non_retryable_failures.)

Fixes

Reconcile the index template or mapping

The most direct fix: update the index template so the mapping matches the data. Identify the correct template (composable templates in ES 8.x via _index_template, legacy via _template), update the field mapping for the conflicting field, and apply it.

New templates only apply to new indices. Existing indices keep their current mapping. To fix existing data, reindex into a new index with the corrected mapping.

Coerce or mutate the field in Logstash

If the upstream change is legitimate and the mapping is correct, transform the field before it reaches Elasticsearch:

filter {
  mutate {
    convert => { "response_code" => "integer" }
  }
}

convert uses Ruby semantics. Converting to float or float_eu produces a double-precision value. Note that add_field within the same mutate block executes after convert, so converting a field that does not exist yet silently does nothing. Split into separate mutate blocks if you need to add then convert.

Fast and non-disruptive, but requires knowing exactly which fields need conversion.

Enable Elasticsearch ignore_malformed

Setting ignore_malformed: true (default false) tells Elasticsearch to index the rest of the document when a field value cannot be parsed. The malformed field is not indexed but remains in _source.

"properties": {
  "response_code": {
    "type": "integer",
    "ignore_malformed": true
  }
}

The field value becomes silently unsearchable. Use this when the field is non-critical and the upstream format is unstable.

Use dynamic mapping settings to control field creation

• true (default): new fields are dynamically mapped and indexed.
• false: new fields are kept in _source but not indexed or searchable.
• strict: documents with unknown fields are rejected.

Setting dynamic: false on specific sub-objects prevents mapping explosion. Setting dynamic: strict forces producers to update templates before sending new fields, but causes more rejections.

Route conflicts to a quarantine index

For pipelines where data must not be lost, route events that may conflict to a separate index with permissive dynamic mapping:

output {
  if [tags] and "_mapping_conflict_risk" in [tags] {
    elasticsearch {
      index => "quarantine-%{+YYYY.MM.dd}"
      template => "/etc/logstash/quarantine-template.json"
    }
  } else {
    elasticsearch {
      index => "logs-%{+YYYY.MM.dd}"
    }
  }
}

Requires a way to identify at-risk events before they reach Elasticsearch. Most useful when you know a specific source produces potentially conflicting data.

Replay from DLQ after fixing the mapping

Once the template or mapping is corrected, replay DLQ entries using the dead_letter_queue input plugin:

input {
  dead_letter_queue {
    pipeline_id => "main"
    clean_consumed => true
  }
}
output {
  elasticsearch {
    index => "%{[@metadata][_dlq_doc_index]}"
  }
}

clean_consumed => true removes fully consumed segments from the DLQ. Without it, replayed events remain on disk and the DLQ grows unbounded (age-based retention, dead_letter_queue.retain.age, purges expired segments on writes and on the DLQ’s periodic flush check even without replay).

Prevention

• Enable DLQ on all production pipelines. Set dead_letter_queue.enable: true in logstash.yml. Without it, mapping conflicts cause silent data loss.
• Monitor DLQ growth. Alert on any non-zero queue_size_in_bytes growth rate. DLQ growth is always a data integrity signal.
• Monitor per-document failure counters. Track documents.non_retryable_failures and bulk_requests.with_errors. These catch partial bulk failures that events.out masks.
• Use explicit index templates. Do not rely on dynamic mapping for production indices.
• Set field count limits proactively. Increase index.mapping.total_fields.limit deliberately or use dynamic: false on sub-objects to prevent mapping explosion.
• Test template changes before deployment. Validate that custom templates match the format required by the selected template_api (composable for ES 8.x, legacy for ES 7.x).
• Track ECS compatibility during upgrades. When upgrading from 7.x to 8.x, decide whether to keep ecs_compatibility: disabled for existing pipelines or update templates and mappings. Mixed states cause conflicts.

How Netdata helps

Netdata surfaces the signals that reveal mapping conflicts before they cause prolonged data loss:

• DLQ growth correlation. Netdata monitors dead_letter_queue.queue_size_in_bytes per pipeline. Sudden growth, correlated with Elasticsearch output error counters, pinpoints when a field shape changed.
• Per-second output plugin metrics. The Elasticsearch output’s bulk_requests.with_errors and documents.non_retryable_failures counters are visible at per-second resolution. Partial bulk failures show up as non-zero with_errors even when events.out looks healthy.
• Events in vs out divergence. Netdata tracks the gap between input and output rates. A growing divergence combined with DLQ growth confirms events are being rejected rather than delayed.
• Anomaly detection on rejection rates. ML-based anomaly detection flags unusual spikes in log error patterns or DLQ growth that baseline threshold alerts would miss, especially when the conflict affects a small percentage of total traffic.
• Log correlation. Netdata’s log collection surfaces mapper_parsing_exception entries from logstash-plain.log alongside metrics, showing the exact field and conflict type at the moment the anomaly fired.

Related guides

• Logstash address already in use: input port conflicts on Beats, TCP, and HTTP
• Logstash API unreachable on port 9600: crash, GC pause, or startup
• Logstash Beats input: Filebeat backpressure and connection health
• Logstash certificate expiry: the silent, total outage no built-in metric shows
• Logstash configuration drift: when the running config no longer matches the deployed one
• Logstash configuration integrity: detecting unexpected changes to pipeline files
• Logstash config reload failed: reloads.failures and invisible configuration drift
• Logstash CPU-bound filters (grok hell): high CPU, saturated workers, growing queue
• Logstash could not be started: another instance is using the configured data.dir
• Logstash disk full: PQ, DLQ, and log volumes competing for space
• Logstash downstream backpressure cascade: when a slow output stalls the whole pipeline
• Logstash Elasticsearch 429: retrying failed action with response code 429