Fluentd pattern not matched: the parser is silently dropping log lines

You found this in the Fluentd log:

[warn]: #0 pattern not match: "2026-07-21T22:58:11.123456789Z stdout F some application log line"

One warning line per dropped record. Or worse: someone turned the warnings off to silence the noise, and now log lines are vanishing with no signal at all. This is input-level data loss. The events never enter the filter chain, never reach the buffer, and never appear in your destination. Pipeline metrics can look healthy because most counters only count what the parser accepted.

The most common trigger is an application log-format change: a team ships a new log layout, a JSON serializer gains or loses a field, or the container runtime underneath Kubernetes changes (Docker JSON logs to containerd plain text), and the parser regex that matched yesterday stops matching today.

This guide covers how to confirm which input is affected, quantify what you are losing, find the format mismatch, and the three fixes: correct the pattern, add a fallback parser, or route unmatched lines somewhere visible. It also covers how to tell this apart from a poison pill, where a single malformed line crashes the whole process instead of being dropped.

What this means

Fluentd’s path for every event is: input, parser, filter chain, buffer, output. The parser stage runs inside the input plugin (for example in_tail with a <parse> section) or in a filter_parser later in the chain. When the configured format or regex does not match a line, what happens depends on where the parse occurs:

• In an input plugin like in_tail: the line is dropped by default. Fluentd logs pattern not match once per line and moves on. The record never becomes an event. Since v0.14.0, in_tail has an emit_unmatched_lines escape hatch (default false) that emits the raw line as a record under the key unmatched_line instead of dropping it.
• In a filter_parser: the record already exists, so the behavior is controlled by emit_invalid_record_to_error (default true), which routes the failed record to the @ERROR label. Set it to false and unmatched records are silently dropped.

The visible symptom is the warning. The actual damage is that the affected input’s emit_records counter stops reflecting what the source is writing. If you alert on “is Fluentd running” or even “is output flowing,” you will not catch this, because other inputs and outputs keep working normally.

Distinguish this from a poison pill. A poison pill is a malformed line that makes the parser itself crash or hang (catastrophic regex backtracking, encoding failure, malformed JSON with pathological nesting). That kills the process, the supervisor restarts it, in_tail resumes from the same pos_file position, and it crashes again: a crash loop. “Pattern not matched” is the opposite failure mode: the process stays healthy and quietly discards data. If you see restart cycling instead of warnings, see Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes.

flowchart LR
  A[Log line read] --> B{Parser format matches?}
  B -- yes --> C[Record emitted into pipeline]
  B -- no --> D[pattern not match warning logged]
  D --> E{emit_unmatched_lines?}
  E -- false, default --> F[Line silently dropped]
  E -- true --> G[Record with key unmatched_line]

Common causes

The containerd case deserves emphasis because it is the number-one cause of this warning at scale in Kubernetes. Docker writes container logs as JSON, so the standard daemonset config uses @type json. Containerd writes plain text in the format <timestamp> <stream> <logtag> <log>. Migrate the node runtime and every line fails the JSON parser at once. Input emit_records for that source falls toward zero while the host keeps producing logs.

Quick checks

All read-only and safe to run during an incident. Paths shown are for the td-agent package; for fluent-package use /var/log/fluent/fluentd.log and /etc/fluent/fluentd.conf, and in Kubernetes use kubectl logs on the daemonset pod.

# 1. Confirm the warnings and see the actual failing lines
grep "pattern not match" /var/log/td-agent/td-agent.log | tail -20

# 2. Count total dropped lines in the current log file (rotate-aware: resets on rotation)
grep -c "pattern not match" /var/log/td-agent/td-agent.log

# 3. Check input emit_records per input plugin (needs monitor_agent on :24220)
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="input") | {id: .plugin_id, type: .type, emit_records: .emit_records}'

# 4. Sum output emit_records (cumulative; run twice a minute apart and diff)
curl -s http://localhost:24220/api/plugins.json | \
  jq '[.plugins[] | select(.plugin_category=="output") | .emit_records // 0] | add'

# 5. Look at the raw source lines Fluentd is trying to parse
tail -5 /var/log/containers/<failing-pod>.log   # or your application log path

# 6. Check the pos_file: is in_tail still advancing on the file?
grep pos_file /etc/td-agent/td-agent.conf
cat /var/log/td-agent/*.pos | head -5

Two caveats. First, on Fluentd older than v1.19.0, input emit_records reads zero unless enable_input_metrics true is set in <system>. From v1.19.0 input metrics are on by default. If your input counter is permanently zero on an older version, that is an instrumentation gap, not proof of health; see Fluentd input emit_records stuck at zero: enable_input_metrics on older versions. Second, checks 3 and 4 are cumulative counters, so a single sample tells you nothing. Take two samples a minute apart and compare the deltas per input.

How to diagnose it

1. Identify the affected input. The warning line is tagged with the plugin id (#0 is the default when no @id is set). If you run several inputs, give each one an explicit @id so warnings map to sources unambiguously. Cross-reference with the emit_records delta from check 3: the input whose rate collapsed is your suspect.

2. Capture a failing raw line. Take the quoted line from the warning message itself, verbatim. This is your test fixture. Do not work from a line you think the application emits; work from the line Fluentd actually read.

3. Reproduce the parse failure outside Fluentd. Extract the regex or format from the input’s <parse> section and test the failing line against it. For a regexp format, use any regex tester with the named capture groups ((?<time>...), (?<message>...)). For json, run the line through jq. The mismatch is usually obvious once you stare at the real line: a field order changed, a level name changed case, the timestamp gained or lost fractional digits.

4. Check the timestamp separately. A line can match the overall regex and still fail if time_format does not match the captured time field. Classic mismatches: the log has no fractional seconds but time_format includes %N, or the log ends in Z but the format expects a numeric offset.

5. Rule out the containerd case if you are on Kubernetes. If the failing source is /var/log/containers/*.log and your <parse> section says @type json, look at the raw file. If lines look like 2026-07-21T... stdout F message instead of {"log":...}, the runtime is containerd and the JSON parser will never match. This is the runtime-migration failure, not an application change.

6. Quantify the loss window. The first pattern not match timestamp in the log is when the format diverged (usually a deploy time). Everything from that point to now, for that source, is gone. Query the destination for that tag and confirm the gap.

7. Confirm it is a drop, not a crash. Steady process uptime, no restarts, warnings streaming: silent drop. Process restarting, crash at the same pos_file position, CPU spike at startup: poison pill. Different incident, different fix.

Metrics and signals to monitor

Note the trap: a full buffer with the default overflow_action (throw_exception) also drops data silently and also shows as an input/output gap. The discriminator is the buffer: if buffer_available_buffer_space_ratios is near zero, suspect overflow; if the buffer is fine and warnings are streaming, suspect the parser. See Fluentd buffer queue length growing for the overflow side.

Fixes

Fix the pattern to match the new format

This is the correct fix when the format change is permanent. Update the <parse> section against the real failing line you captured, test the regex offline, then reload. For the containerd case specifically, switch the parser from json to a regexp matching the CRI format:

<parse>
  @type regexp
  expression /^(?<time>.+) (?<stream>stdout|stderr) (?<logtag>[FP]) (?<log>.*)$/
  time_format %Y-%m-%dT%H:%M:%S.%N%:z
</parse>

In the standard fluentd Kubernetes daemonset images, this same fix is exposed as the FLUENT_CONTAINER_TAIL_PARSER_TYPE and FLUENT_CONTAINER_TAIL_PARSER_TIME_FORMAT environment variables. If you reload via SIGHUP, verify the new config actually applied; a partially applied reload leaves the old parser running. See Fluentd config reload failed: SIGHUP that partially applies.

Add a fallback parser

When formats are mixed or in transition (some pods on the old runtime, some on the new; some apps emitting the old layout), the fluent-plugin-multi-format-parser gem lets you list several formats tried in order, ending with format none as a catch-all that passes the line through unparsed instead of dropping it. Two limitations: it does not work with multiline parsers, and a catch-all that never drops anything can push garbage downstream, so pair it with a filter that tags or separates the none-parsed records.

Route unmatched lines instead of dropping them

On in_tail, set emit_unmatched_lines true. Unmatched lines become records with the key unmatched_line, tagged like everything else from that input. Route them to a dedicated match stanza (a file, a separate index, anywhere visible) so format drift becomes a queryable stream instead of silent loss. Tradeoff: malformed lines now flow through your pipeline and cost buffer and network, so do this deliberately, not as a blanket default. This option exists on in_tail; do not assume it is available on socket-based inputs like in_syslog or in_udp.

For filter_parser, the control is emit_invalid_record_to_error (default true), which sends parse failures to the @ERROR label. Keep it true and route @ERROR to a visible destination. If you set it to false, records are dropped silently, which recreates the original blind spot. One warning: do not route @ERROR back through the same filter_parser; that can loop. Also note that suppress_parse_error_log from v0.12 is gone in v1 for filter_parser; emit_invalid_record_to_error is the replacement, and silencing the warning by other means just hides the loss.

Do not fix it by silencing the warning

Setting log levels high enough to hide pattern not match, or dropping unmatched records intentionally, converts a detectable problem into an invisible one. If some sources genuinely produce lines you never want, exclude them upstream (narrower glob, or filter by path), not by letting them fail the parser.

Prevention

• Alert on the warning itself. Fluentd’s own log is the ground truth here, and it is frequently the one log nobody collects. Ship the Fluentd log somewhere and alert on any sustained rate of pattern not match.
• Alert on per-input emit_records deviation. A greater than 50% drop from the rolling baseline on one input, sustained, catches format changes even if the warnings are suppressed. On versions older than v1.19.0 this requires enable_input_metrics true.
• Give every input an explicit @id. Otherwise every warning says #0 and triage starts with guesswork.
• Treat log format as an interface. Application teams changing log layout without telling the logging pipeline is the root organizational cause. A contract test that parses a sample line from CI against the production Fluentd parser config catches drift before deploy.
• Pin the runtime/parser pairing in Kubernetes. When migrating nodes between Docker and containerd, migrate the daemonset parser config in the same change window, or use a multi-format parser that handles both.
• Verify end to end. Periodically confirm that what the source writes and what the destination holds actually reconcile for each tag. Parser drops are exactly the failure that per-component health checks miss. For the broader pipeline view, see How Fluentd actually works in production: a mental model for operators.

How Netdata helps

• Netdata collects the Fluentd monitor_agent endpoint and turns the cumulative emit_records counters into per-second rates per input plugin, so a parser-induced collapse on one input shows up as a visible cliff rather than a number you had to diff by hand.
• Input-vs-output rate correlation on one dashboard separates parser drops (input falls, buffer stays empty) from buffer overflow drops (input stays high, buffer fills, output stalls). Same downstream symptom, opposite cause.
• Anomaly detection on per-input rates flags the slow version of this failure: a parser that fails on a growing fraction of lines as a rollout progresses, before the input hits zero.
• Historical retention on the affected input’s rate gives you the loss window for free: the timestamp of the cliff is when the format diverged, which is what you need for the postmortem and for scoping the downstream data gap.
• Combined with emit_error_count and buffer metrics on the same host, you can rule the neighboring silent-loss modes (overflow exception, drop_oldest_chunk) in or out without switching tools.

Related guides

• Fluentd broken pipe / connection reset: dropped output connections and LB timeouts
• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss
• Fluentd emit_error_count: the number-one under-monitored data-loss signal
• Fluentd input emit_records stuck at zero: enable_input_metrics on older versions
• Fluentd failed to flush the buffer: the output cannot deliver and retries begin
• Fluentd average flush time rising: the earliest sign of destination slowdown
• How Fluentd actually works in production: a mental model for operators