Logstash _jsonparsefailure: malformed JSON and codec mismatches

When events arrive at your Elasticsearch indices carrying the _jsonparsefailure tag, they contain raw, unparsed data instead of the structured fields your downstream consumers expect. Throughput metrics look healthy, events-out counts keep climbing, and no alerts fire. But the data is wrong.

The _jsonparsefailure tag is added by either the json codec or the json filter when it receives input it cannot parse as valid JSON. The event is not dropped. It flows through to the output carrying whatever raw data was received, plus the failure tag. This is a silent correctness failure, not an availability failure.

The critical diagnostic question: is the source data genuinely malformed, or is Logstash configured to parse data that was never JSON in the first place? Correcting a codec mismatch takes one line of config. Chasing a producer that occasionally truncates JSON objects may require upstream changes.

What this means

Both the json codec and the json filter can emit _jsonparsefailure. Their behavior on failure differs:

• json codec: Falls back to plain text. The raw payload is stored in the message field, and the event continues through the pipeline with the _jsonparsefailure tag.
• json filter: Leaves the event untouched (the source field retains its original value) and adds the _jsonparsefailure tag.

In both cases the event reaches the output. It is not diverted to the Dead Letter Queue. The DLQ captures output delivery failures, not filter or codec parse errors. An event tagged _jsonparsefailure still increments the events-out counter, so standard throughput monitoring stays green.

Because throughput metrics do not reflect this failure, it commonly goes unnoticed. Dashboards querying structured fields return fewer results, and the degradation can persist for weeks.

The json filter has a skip_on_invalid_json option (default false). When set to true, the filter returns without parsing or tagging the event on failure. This does not fix the data quality problem. It hides it.

flowchart TD
    A["_jsonparsefailure on events"] --> B["Sample failed events
from destination"]
    B --> C{"Raw message
is valid JSON?"}
    C -->|"No - plain text"| D["Codec mismatch:
fix input codec"]
    C -->|"No - truncated"| E["Source issue:
TCP drops or
multiline needed"]
    C -->|"Yes"| F{"Both json codec
and json filter
on same path?"}
    F -->|"Both"| G["Double-parsing:
remove filter
or change codec"]
    F -->|"Filter only"| H{"Keys have brackets
or metadata
non-object?"}
    H -->|"Yes"| I["Field-reference
or metadata bug"]
    H -->|"No"| J["Encoding issue:
check charset"]

Common causes

Quick checks

# Check Logstash log for JSON parse errors and related exceptions
grep -iE 'JSON parse|_jsonparsefailure|Invalid FieldReference|ClassCastException' \
  /var/log/logstash/logstash-plain.log | tail -n 50

# Count _jsonparsefailure events in Elasticsearch.
# If `tags` is mapped as text, use `tags.keyword` instead.
curl -s 'http://localhost:9200/<index>-*/_count' -H 'Content-Type: application/json' -d '{
  "query": { "term": { "tags": "_jsonparsefailure" } }
}'

# Sample three failed events to inspect raw message content
curl -s 'http://localhost:9200/<index>-*/_search?size=3' -H 'Content-Type: application/json' -d '{
  "query": { "term": { "tags": "_jsonparsefailure" } }
}'

# Check pipeline config for double-parsing: json codec AND json filter on same path
grep -rn 'codec.*json' /etc/logstash/conf.d/
grep -rn 'json {' /etc/logstash/conf.d/

# Verify a sample raw message is valid JSON
echo '<paste raw message here>' | python3 -m json.tool

# Check json filter settings including skip_on_invalid_json
grep -B2 -A10 'json {' /etc/logstash/conf.d/*.conf

How to diagnose it

1. Sample the failed events. Query your destination for events with the _jsonparsefailure tag. Look at the raw message field. This is your primary diagnostic signal. The raw content tells you what Logstash actually received.

2. Determine if the raw message is valid JSON. If it is valid, the problem is on the Logstash side (double-parsing, field-reference issues, encoding). If it is not, the problem is at the source (producer sending non-JSON, truncated data, encoding mismatch).

3. Check for double-parsing. If the input uses codec => json, the JSON is already deserialized before filters run. The message field no longer contains the original JSON string. A downstream json { source => "message" } filter then fails because message is empty or contains a non-JSON value. Fix: remove the json filter, or switch the input codec to plain and keep only the filter.

4. Check for codec mismatch. If the raw message is not JSON (plain text, syslog, key=value), the input codec is wrong. Change the codec to plain or line and parse in a filter, or fix the producer to send JSON.

5. Check for special key issues. If the raw message looks like valid JSON but events are still failing, inspect the keys. JSON keys containing [ or ] cause failures in the json filter path due to Logstash’s field-reference parser. A @metadata field set to a non-object type (null, string, number) causes a ClassCastException that also results in _jsonparsefailure. Neither is fixed by skip_on_invalid_json.

6. Check for encoding issues. If the raw message contains non-ASCII characters and the source may not be UTF-8, set the charset option on the json codec to match the source encoding.

7. Check for truncated or multiline JSON. If failures are intermittent and correlate with high traffic or network instability, the source may be sending partial JSON. Use a multiline codec on the input to reassemble fragmented messages, then parse with a json filter instead of a json codec. An input can only have one codec, so you cannot chain multiline and json codecs on the same input.

Metrics and signals to monitor

Fixes

Fix double-parsing

The most common cause of spurious _jsonparsefailure. If the input already uses codec => json, the JSON is deserialized at ingestion time. The original message field no longer contains the raw JSON string. Adding filter { json { source => "message" } } then tries to parse the now-empty or modified message field and fails.

Pick one path:

• Option A: Use codec => json on the input. Remove the json filter entirely. The fields are already parsed into the event.
• Option B: Use codec => plain (or line) on the input. Keep the json filter. The raw JSON arrives in message and the filter parses it.

Do not do both.

Fix codec mismatch

If the source sends non-JSON data to an input configured with codec => json, every event will be tagged. Check what the producer actually sends:

# Capture raw data hitting a TCP input without interfering with the pipeline.
# Run on the Logstash host. Adjust the port to match your input.
tcpdump -i any -A -s0 'tcp port <input_port>' -c 50

# If the source exposes its own TCP listener, you can connect directly.
# WARNING: this consumes data from the source. Do not run this while
# Logstash is actively reading from the same source.
nc <source_host> <source_port> | head -5

Either fix the producer to emit valid JSON, or change the codec to match the actual format and parse with appropriate filters.

Handle mixed formats conditionally

If the same input receives both JSON and non-JSON events (common with syslog inputs where some applications format as JSON and others as plain text), use conditional parsing:

filter {
  if [message] =~ /^\s*{/ {
    json {
      source => "message"
      target => "json_data"
    }
  } else {
    # Handle non-JSON format with grok, dissect, or kv
  }
}

This prevents non-JSON events from being tagged. The regex check avoids the parse attempt entirely for input that does not start with a JSON object delimiter.

Fix encoding issues

If the source emits data in an encoding other than UTF-8 (common with nxlog on Windows sending CP1252), set the charset option on the json codec:

input {
  tcp {
    port => 5000
    codec => json {
      charset => "CP1252"
    }
  }
}

The codec converts from the specified charset to UTF-8 before attempting JSON parsing.

Handle @metadata non-object type

This is a known Logstash bug (issue #13630). If the source JSON contains @metadata as null, a string, or a number, Logstash throws ClassCastException and tags the event _jsonparsefailure. Setting skip_on_invalid_json => true on the json filter does not prevent this.

The tracking issue (elastic/logstash#13630) remains open as of 2026 with no fix released; the sanitization workarounds below still apply in current Logstash.

Workaround: sanitize the @metadata field upstream at the producer, or use a ruby filter to remove or convert it before the json filter processes the event.

Handle square brackets in field names

JSON keys containing [ or ] conflict with Logstash’s field-reference syntax. In Logstash 8.3.0+, the --field-reference-escape-style flag was added, but it only works for the json codec path, not the json filter path. As of Logstash 8.5.3, the filter path still fails, and the tracking issue (elastic/logstash#14821) remains open as of 2026: the field-reference escape style covers event creation from codecs, not the Event API calls the json filter uses, so the workarounds below still apply.

Options:

• Use codec => json instead of the json filter, combined with --field-reference-escape-style ampersand.
• Use a ruby filter to sanitize keys with gsub before parsing. Note: this has severe performance impact on high-throughput pipelines (reported degradation from 4k events/sec to 150 events/sec).

Avoid re-ingesting Logstash’s own logs through a json codec

The json codec’s “Falling back to plain-text” message is logged at INFO level (it was logged at ERROR only in codec versions 2.1.0 through 2.1.3, released in 2016; there is no 7.15 log-level change). Even at INFO, if Logstash logs to syslog and syslog is re-ingested by a pipeline that uses the json codec, the fallback message can itself fail JSON parsing and reappear as a _jsonparsefailure event, feeding back into the same pipeline.

Check whether Logstash’s own logs are being re-ingested through the same pipeline that parses your data. Route Logstash’s own logs separately, or filter them out at the input level before they reach the json codec.

Prevention

• Monitor the _jsonparsefailure tag rate. Do not wait to discover parse failures weeks later when a dashboard stops showing data. Query your destination regularly for failure tag counts, or add a metrics filter in the pipeline to count tagged events.
• Route failed events to a separate index. This preserves raw data for debugging and keeps malformed events from polluting your primary data store. It also makes the failure rate trivially queryable.
• Validate config changes for double-parsing. Before deploying a config that includes a json filter, verify the input codec is not also json. This is the single most common preventable cause.
• Gate source format changes. Coordinate with application teams before they change log output format. A format change at the source that breaks JSON structure will silently degrade data quality until someone notices missing fields downstream.
• Set skip_on_invalid_json deliberately. If you set it to true on the json filter, failed events pass through without the tag. You lose the ability to detect parse failures at all. Only use this when you have another mechanism to validate data quality.

How Netdata helps

• Correlate parse failure spikes with deployment events. A sudden increase in _jsonparsefailure events that coincides with a config reload or application deploy pinpoints the change that introduced the mismatch.
• Per-pipeline throughput visibility. In multi-pipeline setups, aggregate metrics mask parse failures in one pipeline. Per-pipeline event rates help identify which pipeline is producing tagged events.
• Config reload state monitoring. Tracking reloads.failures alongside parse failure rates catches the scenario where a failed reload leaves old config running against a new data format.
• Log-based anomaly detection. JSON parse error patterns in Logstash logs, especially after the 7.15 log-level change to ERROR, can be correlated with throughput and pipeline health metrics to distinguish data quality issues from availability issues.

Related guides

• Logstash address already in use: input port conflicts on Beats, TCP, and HTTP
• Logstash API unreachable on port 9600: crash, GC pause, or startup
• Logstash Beats input: Filebeat backpressure and connection health
• Logstash certificate expiry: the silent, total outage no built-in metric shows
• Logstash configuration drift: when the running config no longer matches the deployed one
• Logstash configuration integrity: detecting unexpected changes to pipeline files
• Logstash config reload failed: reloads.failures and invisible configuration drift
• Logstash CPU-bound filters (grok hell): high CPU, saturated workers, growing queue
• Logstash could not be started: another instance is using the configured data.dir
• Logstash disk full: PQ, DLQ, and log volumes competing for space
• Logstash downstream backpressure cascade: when a slow output stalls the whole pipeline
• Logstash Elasticsearch 429: retrying failed action with response code 429