Logstash _grokparsefailure: why grok stops matching and how to fix it

Events tagged with _grokparsefailure pass through your pipeline unstructured. Grok could not match any of its configured patterns against the event’s input field, so it appended the failure tag and moved on. The event is not dropped. It is not counted as filtered. Unless you conditionally route it, it reaches your output destination carrying raw, unparsed data alongside the failure tag.

Throughput metrics stay green because events are still flowing. The process is up, workers are busy, queue depth is normal, and output counts look healthy. But the data in your indices is wrong: fields that downstream dashboards, alerts, and searches depend on are missing or empty because grok never extracted them. The grok filter’s per-plugin failures counter is the most direct signal for this problem and rarely has dedicated monitoring.

The most common cause is source log-format drift: an application deployed a new version, changed its log structure, added a field, or switched a timestamp format. The pattern that worked yesterday no longer matches today. A secondary cause is a bad filter deployment where a config change broke pattern matching, either through a syntax error that reloaded with unintended behavior, or through an ECS compatibility mode change that altered field names.

What this means

When grok fails to match, the filter applies its tag_on_failure setting, which defaults to ["_grokparsefailure"]. The event passes through the rest of the filter chain and reaches output with this tag and without any of the fields the grok pattern was supposed to extract.

Key behaviors:

• Events are not dropped. They continue through the pipeline to outputs. If your output does not filter on the tag, failed events land in the same index as successfully parsed events, polluting search results and dashboards.
• Events are not counted as filtered. The events.filtered counter tracks events intentionally dropped or routed by conditionals. Unless you add an explicit drop {} or conditional routing for tagged events, they are counted as normal output events.
• The dead letter queue does not capture them. DLQ only intercepts output-side failures such as Elasticsearch rejects or mapping conflicts. Filter-level parse failures are invisible to DLQ.
• The failures counter is per-plugin. Each grok filter instance in your pipeline has its own counter in plugins.filters[].failures. In multi-grok pipelines, one filter may be failing while others succeed.

Using conditionals like if "_grokparsefailure" in [tags] is the standard approach for routing failed events. Tags remain the mechanism grok uses for failure signaling.

Common causes

Quick checks

# Check grok filter failures counter in per-plugin stats
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | grep -A 20 '"filters"'

# Check whether a recent config reload failed (stale config running)
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | grep -A 10 '"reloads"'

# Query downstream index for failure-tagged events (Elasticsearch)
curl -s "localhost:9200/<index-pattern>/_count?q=tags:_grokparsefailure"

# Sample a failed event to inspect the raw message
curl -s "localhost:9200/<index-pattern>/_search?q=tags:_grokparsefailure&size=1&sort=@timestamp:desc"

# Check for pattern timeout tags alongside parse failures
curl -s "localhost:9200/<index-pattern>/_count?q=tags:_groktimeout"

# Check Logstash logs for grok-related errors or warnings
grep -Ei 'grok|timeout|pattern' /var/log/logstash/logstash-plain.log | tail -n 100

# Extract per-plugin failures for all grok filters
# Adjust 'main' if your pipeline has a different name
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | python3 -c "
import sys,json
filters = json.load(sys.stdin)['pipelines']['main']['plugins']['filters']
for f in filters:
    if f.get('name') == 'grok':
        fid = f.get('id','?')
        failures = f.get('failures', 0)
        print(f'grok ({fid}): failures={failures}')
"

How to diagnose it

flowchart TD
    A["grok failures counter rising"] --> B{"Config changed
recently?"}
    B -->|Yes| C["Check reloads.failures
and last_error"]
    B -->|No| D["Source format
likely changed"]
    C --> E{"Reload
succeeded?"}
    E -->|Failed| F["Running stale config
- invisible drift"]
    E -->|Succeeded| G["New pattern
does not match data"]
    D --> H["Sample failed events
from destination"]
    H --> I["Compare raw message
to current grok pattern"]
    I --> J["Fix pattern or add
catch-all fallback"]
    G --> I
    F --> K["Fix and redeploy
config successfully"]

Step 1: Confirm the failure rate and onset time.

Query the grok filter’s failures counter twice with a gap to compute the rate. If you have destination-side access, count _grokparsefailure-tagged events over the last hour versus the previous hour. Note when the rate changed. A sudden spike points to a discrete event: a source deploy or a config change. A gradual increase points to slow format drift.

Step 2: Correlate with config changes.

Check reloads.failures and reloads.last_error in the pipeline stats API. If a reload failed, Logstash is running the previous config, which may not match the data being sent. Check file modification times on pipeline config files:

find /etc/logstash -maxdepth 2 -type f -printf '%TY-%Tm-%Td %TT %p\n' | sort | tail

Step 3: Sample actual failed events.

Query your destination for recent events tagged with _grokparsefailure. Look at the raw message field. Compare it to the pattern in your grok filter config. The mismatch is usually visible: a new field inserted, a timestamp format changed, a delimiter switched, or extra whitespace added.

Step 4: Test the pattern against the sample.

Use a grok debugger (Kibana’s Grok Debugger, or an online grok tester) to run your current pattern against the sampled failed message. This confirms the mismatch and helps you iterate on a fix before deploying to production.

Step 5: Check for timeout-related failures.

If events are tagged with _groktimeout rather than (or alongside) _grokparsefailure, the pattern may be experiencing catastrophic backtracking on large messages. Multiple nested or overlapping GREEDYDATA captures, or ambiguous alternation, can trigger exponential backtracking and exhaust the 30-second default timeout_millis. Check message sizes of failed events and inspect patterns for ambiguous regex constructs.

Step 6: Check ECS compatibility mode.

If your grok filter uses built-in patterns such as COMMONAPACHELOG, SYSLOGLINE, or IPORHOST, the ecs_compatibility setting changes which field names those patterns produce. In ECS v8 mode, COMMONAPACHELOG produces ECS-style fields instead of the legacy clientip and verb names. If the mode changed (for example, from a Logstash upgrade that altered the default), downstream references to old field names break even though grok itself succeeds.

Metrics and signals to monitor

Fixes

Fix patterns for format drift

When the source changes its log format, update the grok pattern to match. Sample several failed events to identify the exact change. Common drift patterns:

• Added fields: A new field appears between existing fields. Extend the pattern to capture or skip it.
• Timestamp format change: From dd/MMM/yyyy:HH:mm:ss Z to ISO 8601, or similar. Update the date pattern in your grok match.
• Delimiter change: Pipes to commas, spaces to tabs. Update the literal characters in the pattern.
• Removed fields: A field your pattern expects is no longer present. Make that portion optional or remove it from the pattern.

Deploy the fix, verify the reload succeeded, and confirm the failures counter stops rising.

Add a catch-all fallback pattern

Add a final pattern in your match array that matches anything, so events that fail specific patterns are still captured with minimal structure:

grok {
  id => "parse_app_logs"
  match => {
    "message" => [
      "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:raw_message}",
      "%{GREEDYDATA:raw_message}"
    ]
  }
  tag_on_failure => ["_grokparsefailure"]
}

With break_on_match defaulting to true, grok tries patterns in order and stops at the first match. The final GREEDYDATA catch-all ensures every event matches something. You can distinguish catch-all matched events from fully parsed ones by checking whether specific extracted fields are populated, or by adding a mutate filter with add_tag conditioned on the presence of those fields.

Route failures to a separate index

Conditional routing isolates failed events for investigation without polluting your primary index:

output {
  if "_grokparsefailure" in [tags] {
    elasticsearch {
      index => "logstash-parse-failures-%{+YYYY.MM.dd}"
    }
  } else {
    elasticsearch {
      index => "logstash-%{+YYYY.MM.dd}"
    }
  }
}

Set up alerts on the failure index document count to catch new format drift early.

Fix character encoding issues

If failures correlate with events containing non-ASCII characters, verify the codec on your input. The plain codec assumes a charset; the json codec handles encoding internally. If the source sends Latin-1 but the codec expects UTF-8, certain bytes produce malformed events that grok cannot match. Set the codec charset explicitly:

input {
  tcp {
    port => 5000
    codec => plain { charset => "ISO-8859-1" }
  }
}

Address pattern timeout

If you are seeing _groktimeout tags, the pattern is hitting the 30-second default timeout_millis. Two approaches:

1. Optimize the pattern. Remove ambiguous alternation, unbounded GREEDYDATA in the middle of patterns, and nested optional groups that cause exponential backtracking. Use the dissect filter for simple delimiter-based parsing instead of grok where possible. dissect does not use regex and is significantly faster for predictable formats.

2. Adjust timeout settings. If optimization is not immediately possible, you can disable timeouts with timeout_millis => 0. Warning: this means a pathological pattern can hang a worker thread indefinitely. Use this only as a temporary measure while you fix the pattern.

Handle anchored patterns

Patterns using ^ and $ anchors fail when the input has trailing whitespace, carriage returns (\r), or other unexpected characters at the start or end. Remove the anchors, or add a trailing %{GREEDYDATA} after the meaningful content to absorb anything unexpected:

# Instead of:
match => { "message" => "^%{IP:clientip} %{GREEDYDATA:message}$" }

# Use:
match => { "message" => "%{IP:clientip} %{GREEDYDATA:message}" }

Prevention

• Monitor the grok failures counter. This is the most direct signal and is available in the pipeline stats API under per-plugin stats. Alert on sustained rate increases above baseline.
• Baseline your parse failure rate. A low background rate (under 1%) is common for mixed log formats. Know your normal rate so you can detect a sudden increase.
• Route failures to a separate index by default. This makes drift immediately visible and prevents unstructured events from corrupting dashboards and alerts.
• Add catch-all patterns. Ensure no event passes through grok without matching at least a minimal pattern.
• Prefer dissect for simple formats. It is faster, does not use regex, and is immune to backtracking.
• Assign explicit IDs to grok filters. Without an id, Logstash auto-generates opaque identifiers, making it hard to map per-plugin stats to your config. Always set id => "parse_syslog" or similar.
• Watch for ECS compatibility changes during upgrades. A Logstash version upgrade may change the default ecs_compatibility mode, silently altering field names produced by built-in patterns.
• Validate patterns before deployment. Use the Kibana Grok Debugger or a staging pipeline to test new patterns against representative log samples.

How Netdata helps

Netdata collects several signals that shorten diagnosis of _grokparsefailure incidents:

• Per-plugin failures counter: Netdata collects the grok filter’s failures counter per pipeline and per filter instance. A sudden rate increase triggers an anomaly alert before downstream users notice missing data.
• Reload state monitoring: Netdata tracks reloads.successes and reloads.failures, making config drift from failed reloads immediately visible.
• Throughput correlation: Correlating grok failures with events-in, events-out, and events-filtered rates confirms whether throughput is normal while data quality degrades.
• Processing duration trends: Per-plugin duration_in_millis trends reveal when grok patterns are getting slower, which may precede timeout failures.
• ML-based anomaly detection: Netdata flags unusual changes in the grok failures rate, even when absolute values remain low, catching format drift that static thresholds miss.

Related guides

• Logstash address already in use: input port conflicts on Beats, TCP, and HTTP
• Logstash API unreachable on port 9600: crash, GC pause, or startup
• Logstash Beats input: Filebeat backpressure and connection health
• Logstash certificate expiry: the silent, total outage no built-in metric shows
• Logstash configuration drift: when the running config no longer matches the deployed one
• Logstash configuration integrity: detecting unexpected changes to pipeline files
• Logstash config reload failed: reloads.failures and invisible configuration drift
• Logstash CPU-bound filters (grok hell): high CPU, saturated workers, growing queue
• Logstash could not be started: another instance is using the configured data.dir
• Logstash disk full: PQ, DLQ, and log volumes competing for space
• Logstash downstream backpressure cascade: when a slow output stalls the whole pipeline
• Logstash Elasticsearch 429: retrying failed action with response code 429