When events arrive at your Elasticsearch indices carrying the _jsonparsefailure tag, they contain raw, unparsed data instead of the structured fields your downstream consumers expect. Throughput metrics look healthy, events-out counts keep climbing, and no alerts fire. But the data is wrong.

The _jsonparsefailure tag is added by either the json codec or the json filter when it receives input it cannot parse as valid JSON. The event is not dropped. It flows through to the output carrying whatever raw data was received, plus the failure tag. This is a silent correctness failure, not an availability failure.

The critical diagnostic question: is the source data genuinely malformed, or is Logstash configured to parse data that was never JSON in the first place? Correcting a codec mismatch takes one line of config. Chasing a producer that occasionally truncates JSON objects may require upstream changes.

What this means

Both the json codec and the json filter can emit _jsonparsefailure. Their behavior on failure differs:

  • json codec: Falls back to plain text. The raw payload is stored in the message field, and the event continues through the pipeline with the _jsonparsefailure tag.
  • json filter: Leaves the event untouched (the source field retains its original value) and adds the _jsonparsefailure tag.

In both cases the event reaches the output. It is not diverted to the Dead Letter Queue. The DLQ captures output delivery failures, not filter or codec parse errors. An event tagged _jsonparsefailure still increments the events-out counter, so standard throughput monitoring stays green.

Because throughput metrics do not reflect this failure, it commonly goes unnoticed. Dashboards querying structured fields return fewer results, and the degradation can persist for weeks.

The json filter has a skip_on_invalid_json option (default false). When set to true, the filter returns without parsing or tagging the event on failure. This does not fix the data quality problem. It hides it.

flowchart TD
    A["_jsonparsefailure on events"] --> B["Sample failed events
from destination"]
    B --> C{"Raw message
is valid JSON?"}
    C -->|"No - plain text"| D["Codec mismatch:
fix input codec"]
    C -->|"No - truncated"| E["Source issue:
TCP drops or
multiline needed"]
    C -->|"Yes"| F{"Both json codec
and json filter
on same path?"}
    F -->|"Both"| G["Double-parsing:
remove filter
or change codec"]
    F -->|"Filter only"| H{"Keys have brackets
or metadata
non-object?"}
    H -->|"Yes"| I["Field-reference
or metadata bug"]
    H -->|"No"| J["Encoding issue:
check charset"]

Common causes

CauseWhat it looks likeFirst thing to check
Codec mismatchProducer sends plain text, syslog strings, or key=value pairs to an input with codec => json. Every event gets tagged.Inspect raw message field on failed events. Is it actually JSON?
Double-parsing (codec + filter)Input has codec => json and a filter does json { source => "message" }. The codec already parsed the JSON into fields. The filter then tries to parse the message field, which is now empty or contains a non-JSON string, and fails.Check pipeline config for both json codec and json filter on the same data path.
Truncated or partial JSONTCP connection drops mid-message, or a producer sends incomplete JSON objects. Failures are intermittent rather than constant.Check whether failures correlate with network events or source-side issues.
Concatenated JSON objectsProducer sends multiple JSON objects on one line without a delimiter (e.g., {"a":1}{"b":2}). The codec fails because trailing content after the first object makes the line invalid JSON.Inspect raw messages for multiple JSON objects on a single line.
Character encoding mismatchSource emits non-UTF-8 data (e.g., CP1252 from nxlog or Windows producers). The json codec default charset is UTF-8.Check the charset option and the source’s actual encoding.
@metadata set to non-object typeIncoming JSON contains {"@metadata": null}, {"@metadata": "string"}, or {"@metadata": 0}. Logstash throws ClassCastException and tags the event. Known bug (issue #13630). skip_on_invalid_json does not prevent this.Inspect failed events for @metadata field values.
Square brackets in field namesJSON keys containing [ or ] trigger Invalid FieldReference errors in the json filter path. The --field-reference-escape-style flag (added in Logstash 8.3.0) only fixes the codec path, not the filter path.Check if failed events have keys containing brackets.

Quick checks

# Check Logstash log for JSON parse errors and related exceptions
grep -iE 'JSON parse|_jsonparsefailure|Invalid FieldReference|ClassCastException' \
  /var/log/logstash/logstash-plain.log | tail -n 50

# Count _jsonparsefailure events in Elasticsearch.
# If `tags` is mapped as text, use `tags.keyword` instead.
curl -s 'http://localhost:9200/<index>-*/_count' -H 'Content-Type: application/json' -d '{
  "query": { "term": { "tags": "_jsonparsefailure" } }
}'

# Sample three failed events to inspect raw message content
curl -s 'http://localhost:9200/<index>-*/_search?size=3' -H 'Content-Type: application/json' -d '{
  "query": { "term": { "tags": "_jsonparsefailure" } }
}'

# Check pipeline config for double-parsing: json codec AND json filter on same path
grep -rn 'codec.*json' /etc/logstash/conf.d/
grep -rn 'json {' /etc/logstash/conf.d/

# Verify a sample raw message is valid JSON
echo '<paste raw message here>' | python3 -m json.tool

# Check json filter settings including skip_on_invalid_json
grep -B2 -A10 'json {' /etc/logstash/conf.d/*.conf

How to diagnose it

  1. Sample the failed events. Query your destination for events with the _jsonparsefailure tag. Look at the raw message field. This is your primary diagnostic signal. The raw content tells you what Logstash actually received.

  2. Determine if the raw message is valid JSON. If it is valid, the problem is on the Logstash side (double-parsing, field-reference issues, encoding). If it is not, the problem is at the source (producer sending non-JSON, truncated data, encoding mismatch).

  3. Check for double-parsing. If the input uses codec => json, the JSON is already deserialized before filters run. The message field no longer contains the original JSON string. A downstream json { source => "message" } filter then fails because message is empty or contains a non-JSON value. Fix: remove the json filter, or switch the input codec to plain and keep only the filter.

  4. Check for codec mismatch. If the raw message is not JSON (plain text, syslog, key=value), the input codec is wrong. Change the codec to plain or line and parse in a filter, or fix the producer to send JSON.

  5. Check for special key issues. If the raw message looks like valid JSON but events are still failing, inspect the keys. JSON keys containing [ or ] cause failures in the json filter path due to Logstash’s field-reference parser. A @metadata field set to a non-object type (null, string, number) causes a ClassCastException that also results in _jsonparsefailure. Neither is fixed by skip_on_invalid_json.

  6. Check for encoding issues. If the raw message contains non-ASCII characters and the source may not be UTF-8, set the charset option on the json codec to match the source encoding.

  7. Check for truncated or multiline JSON. If failures are intermittent and correlate with high traffic or network instability, the source may be sending partial JSON. Use a multiline codec on the input to reassemble fragmented messages, then parse with a json filter instead of a json codec. An input can only have one codec, so you cannot chain multiline and json codecs on the same input.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
_jsonparsefailure tag rate at destinationPrimary data quality indicator. Parse failures mean events arrive with raw data instead of structured fields.Sudden spike after a source-side change. Sustained non-zero rate on a stream that should be all JSON.
Filter failures counterThe Logstash stats API exposes per-filter metrics at _node/stats/pipelines. If the pipeline uses grok alongside json, correlated failures point to a common source format change.Counter rate increasing above baseline alongside jsonparsefailure spikes.
Output throughput (events.out)Stays green during parse failures. Events with failure tags still count as successfully delivered.Normal throughput with rising failure tags equals silent correctness degradation.
Config reload state (reloads.failures)A failed config reload can leave old config running that no longer matches the data format.Non-zero reloads.failures after a config deploy.
Log error rate for JSON parse messagesThe Logstash log records codec and filter parse errors. Since Logstash 7.15, these are logged at ERROR level instead of INFO, which can create feedback loops if logs are re-ingested.Spike in ERROR-level log lines mentioning JSON or parse, especially after an upgrade.

Fixes

Fix double-parsing

The most common cause of spurious _jsonparsefailure. If the input already uses codec => json, the JSON is deserialized at ingestion time. The original message field no longer contains the raw JSON string. Adding filter { json { source => "message" } } then tries to parse the now-empty or modified message field and fails.

Pick one path:

  • Option A: Use codec => json on the input. Remove the json filter entirely. The fields are already parsed into the event.
  • Option B: Use codec => plain (or line) on the input. Keep the json filter. The raw JSON arrives in message and the filter parses it.

Do not do both.

Fix codec mismatch

If the source sends non-JSON data to an input configured with codec => json, every event will be tagged. Check what the producer actually sends:

# Capture raw data hitting a TCP input without interfering with the pipeline.
# Run on the Logstash host. Adjust the port to match your input.
tcpdump -i any -A -s0 'tcp port <input_port>' -c 50

# If the source exposes its own TCP listener, you can connect directly.
# WARNING: this consumes data from the source. Do not run this while
# Logstash is actively reading from the same source.
nc <source_host> <source_port> | head -5

Either fix the producer to emit valid JSON, or change the codec to match the actual format and parse with appropriate filters.

Handle mixed formats conditionally

If the same input receives both JSON and non-JSON events (common with syslog inputs where some applications format as JSON and others as plain text), use conditional parsing:

filter {
  if [message] =~ /^\s*{/ {
    json {
      source => "message"
      target => "json_data"
    }
  } else {
    # Handle non-JSON format with grok, dissect, or kv
  }
}

This prevents non-JSON events from being tagged. The regex check avoids the parse attempt entirely for input that does not start with a JSON object delimiter.

Fix encoding issues

If the source emits data in an encoding other than UTF-8 (common with nxlog on Windows sending CP1252), set the charset option on the json codec:

input {
  tcp {
    port => 5000
    codec => json {
      charset => "CP1252"
    }
  }
}

The codec converts from the specified charset to UTF-8 before attempting JSON parsing.

Handle @metadata non-object type

This is a known Logstash bug (issue #13630). If the source JSON contains @metadata as null, a string, or a number, Logstash throws ClassCastException and tags the event _jsonparsefailure. Setting skip_on_invalid_json => true on the json filter does not prevent this.

Workaround: sanitize the @metadata field upstream at the producer, or use a ruby filter to remove or convert it before the json filter processes the event.

Handle square brackets in field names

JSON keys containing [ or ] conflict with Logstash’s field-reference syntax. In Logstash 8.3.0+, the --field-reference-escape-style flag was added, but it only works for the json codec path, not the json filter path. As of Logstash 8.5.3, the filter path still fails.

Options:

  • Use codec => json instead of the json filter, combined with --field-reference-escape-style ampersand.
  • Use a ruby filter to sanitize keys with gsub before parsing. Note: this has severe performance impact on high-throughput pipelines (reported degradation from 4k events/sec to 150 events/sec).

Manage the Logstash 7.15 log-level change

Logstash 7.15 changed the json codec error log level from INFO to ERROR. If Logstash logs to syslog and syslog is re-ingested by Logstash, the error messages themselves can create a feedback loop: error logs fail JSON parsing, generate more error logs, and fill disk.

If you encounter this after upgrading to 7.15+, check whether Logstash error logs are being re-ingested through the same pipeline. Route Logstash’s own logs separately, or filter them out at the input level before they reach the json codec.

Prevention

  • Monitor the _jsonparsefailure tag rate. Do not wait to discover parse failures weeks later when a dashboard stops showing data. Query your destination regularly for failure tag counts, or add a metrics filter in the pipeline to count tagged events.
  • Route failed events to a separate index. This preserves raw data for debugging and keeps malformed events from polluting your primary data store. It also makes the failure rate trivially queryable.
  • Validate config changes for double-parsing. Before deploying a config that includes a json filter, verify the input codec is not also json. This is the single most common preventable cause.
  • Gate source format changes. Coordinate with application teams before they change log output format. A format change at the source that breaks JSON structure will silently degrade data quality until someone notices missing fields downstream.
  • Set skip_on_invalid_json deliberately. If you set it to true on the json filter, failed events pass through without the tag. You lose the ability to detect parse failures at all. Only use this when you have another mechanism to validate data quality.

How Netdata helps

  • Correlate parse failure spikes with deployment events. A sudden increase in _jsonparsefailure events that coincides with a config reload or application deploy pinpoints the change that introduced the mismatch.
  • Per-pipeline throughput visibility. In multi-pipeline setups, aggregate metrics mask parse failures in one pipeline. Per-pipeline event rates help identify which pipeline is producing tagged events.
  • Config reload state monitoring. Tracking reloads.failures alongside parse failure rates catches the scenario where a failed reload leaves old config running against a new data format.
  • Log-based anomaly detection. JSON parse error patterns in Logstash logs, especially after the 7.15 log-level change to ERROR, can be correlated with throughput and pipeline health metrics to distinguish data quality issues from availability issues.