Elasticsearch mapper_parsing_exception: type conflicts and failed document indexing

Document counts do not match what your pipeline sent. The _bulk endpoint returns HTTP 200, yet documents are missing from queries. In the Elasticsearch logs you see mapper_parsing_exception with messages like failed to parse field [fieldname] of type [typename] in document with id [id]. The indices.indexing.index_failed counter is climbing. This is a per-document schema rejection, not a cluster outage, and it silently drops data.

The error means the primary shard refused a document because a field value does not match the mapping declared for that field, or because the index uses dynamic: strict and the document contains an unknown field. The document is rejected in its entirety; other fields in the same document are not indexed. The bulk API returns HTTP 200 whenever the request body is valid NDJSON, so failures hide inside the per-item response array.

flowchart TD
    A[Document arrives at coordinating node] --> B[Routed to primary shard]
    B --> C[Mapping validation]
    C -->|Value matches type| D[Indexed to translog and buffer]
    C -->|Value mismatches type| E[mapper_parsing_exception]
    C -->|Unknown field with dynamic strict| F[strict_dynamic_mapping_exception]
    E --> G[Document rejected; index_failed increments]
    F --> G
    D --> H[Bulk item status 201]
    G --> I[Bulk item status 400]
    H --> J[HTTP 200 with errors true]
    I --> J

Common causes

CauseWhat it looks likeFirst thing to check
Value type does not match mappingfailed to parse field [X] of type [long] in document with id [Y] with a nested number_format_exception or illegal_argument_exceptionThe index mapping for the field against the JSON value being sent
Strict mapping blocks unknown fieldsstrict dynamic mapping exception for [fieldname]Whether the index has dynamic: strict and whether the document contains new fields
Bulk API item failures hidden by HTTP 200Client sees HTTP 200 but documents are missing; index_failed counter risingThe errors flag and per-item status codes in the bulk response body
Reindex into incompatible destination mappingThe same mapper_parsing_exception reoccurs in the destination index during _reindexThe destination index mapping before running the reindex operation

Quick checks

These commands are read-only and safe to run during an incident.

# Check document-level indexing failures across nodes
curl -s 'http://localhost:9200/_nodes/stats/indices?filter_path=nodes.*.indices.indexing.index_failed'

# Compare failed indexing against total indexing per node
curl -s 'http://localhost:9200/_cat/nodes?v&h=name,indexing.index_total,indexing.index_failed'

# Inspect the mapping for the target index
curl -s "http://localhost:9200/<index>/_mapping?pretty"

# Check if the index enforces strict dynamic mapping
curl -s "http://localhost:9200/<index>/_settings/index.mapping.dynamic?pretty"

# List indices and document counts to spot unexpectedly low counts
curl -s 'http://localhost:9200/_cat/indices?v&h=index,pri,rep,docs.count,store.size'

# Count documents with ignored fields if ignore_malformed is already enabled
curl -s "http://localhost:9200/<index>/_search?size=0&q=_ignored:*&pretty"

# Search logs for the exception pattern
grep -E "mapper_parsing_exception|failed to parse field" /var/log/elasticsearch/*.log | tail -n 20

How to diagnose it

  1. Confirm that bulk responses are inspected item by item. The _bulk API returns HTTP 200 if the NDJSON payload parses correctly, even if every item fails. Check the top-level errors boolean and iterate the items array for non-2xx status codes. If you saved the response, extract failures with jq:

    jq '.items[] | select(.index.status >= 400) | {id: .index._id, error: .index.error.reason}' response.json

  2. Parse the exception message. mapper_parsing_exception includes the field name, expected type, document id, and a nested caused_by block. Note the root cause (for example, number_format_exception or illegal_argument_exception).

  3. Compare the mapping to the source value. Retrieve the current mapping for the field. If dynamic mapping created the field, the first-seen value may have established an unintended type. A string such as "179.152.62.82" mapped dynamically becomes text, not ip.

  4. Determine if strict mapping is the blocker. If the index uses dynamic: strict, any field not explicitly defined in the mapping triggers rejection, even if the value itself is well-formed.

  5. Quantify the scope with index_failed. Sample indices.indexing.index_failed before and after the suspected incident window. Correlate the spike with an application deployment, a Logstash filter change, or a new data source.

  6. Check reindex pipelines. If the error appeared during _reindex, remember that reindex copies values as-is. It does not coerce types. Inspect the destination mapping and verify that the source data conforms to it before starting the operation.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
indices.indexing.index_failedIncrements on every rejected document, including mapper_parsing_exceptionSudden spike from near-zero or sustained nonzero rate
indices.indexing.index_totalBaseline to compare failed against successful indexingindex_failed sustained above 0.1% of indexing rate
Bulk API errors flagPer-item failures are invisible to HTTP-level monitoringerrors: true in any bulk response
Index field countUnexpected growth indicates dynamic mapping runaway or pipeline changesField count growing without a corresponding mapping update
write thread pool rejectionsRejections can rise if clients retry failed documents aggressivelySustained nonzero rejected count

Fixes

Correct the mapping or the data

The durable fix is to align the data pipeline with the index mapping. Update the producer to send the correct type. If you need to ingest documents that occasionally contain malformed values for a numeric or date field, set ignore_malformed: true on the field mapping. Elasticsearch then skips the offending field and indexes the rest of the document. You can later find those documents by querying the _ignored meta-field:

curl -s "http://localhost:9200/<index>/_search?q=_ignored:<fieldname>&pretty"

Do not apply ignore_malformed to field types that do not support it, such as keyword or text, or the mapping update itself will be rejected.

If dynamic mapping established the wrong type because the first document contained a string where a number was expected, you cannot change the type on an existing field. You must create a new index with the correct explicit mapping and reindex the data.

Handle strict mode violations

If the index uses dynamic: strict, you have two options. Either update the index mapping or template to pre-define the new field before indexing documents that contain it, or modify the ingest pipeline to drop or rename the unexpected field using a remove or rename processor. Changing dynamic: strict to true or false on a live production index is possible via the settings API, but it allows dynamic mapping immediately, which can lead to mapping explosion if the data source is untrusted.

Reindex safely across type changes

Reindexing is resource-intensive and generates disk I/O and CPU load proportional to source size. Do not run it against large indices during peak traffic without prior testing.

Do not assume that _reindex resolves type conflicts automatically. It copies the source _source as-is into the destination. If the source index mapped a field as text and the destination declares it as long, the reindex operation will reproduce the same mapper_parsing_exception in the destination.

Before reindexing, define the destination mapping explicitly. If the source contains mixed types, attach an ingest pipeline with a convert processor or a script processor to sanitize values during the reindex. Handle nulls, empty strings, and unparseable values explicitly so they do not trigger the same exception in the new index.

Prevention

  • Use explicit mappings or index templates for production indices. Do not rely on dynamic mapping for structured or machine-generated data.
  • Set index.mapping.total_fields.limit to cap mapping growth and prevent cluster state bloat.
  • Validate document schemas upstream, in the application or ingest pipeline, before sending them to Elasticsearch.
  • Always inspect the errors flag and per-item status codes in bulk responses. Never treat HTTP 200 as unconditional success.
  • Test reindex operations and ingest pipelines against a representative sample before applying them to large indices.
  • Monitor indices.indexing.index_failed as a first-class metric alongside indexing rate.

How Netdata helps

  • Netdata collects indices.indexing.index_failed per node and surfaces spikes in real time, so you do not need to poll /_nodes/stats manually during an incident.
  • Correlate indexing failures with JVM heap pressure, write thread pool rejections, and disk watermark breaches on the same charts to distinguish mapping errors from resource exhaustion.
  • Indexing rate and latency are exposed together. A divergence between the two, with a rising failure count, points to document-level rejections rather than cluster saturation.
  • Alert on nonzero deltas for index_failed to detect mapper_parsing_exception before downstream consumers notice missing documents.