Fluentd poison pill crash loop: one bad log line that kills the process on every restart

Fluentd is restarting every few seconds. The service comes up, CPU spikes to 100 percent almost immediately, the process dies, systemd (or Kubernetes) restarts it, and the cycle repeats. Input throughput is effectively zero, but the process is “running” most of the time, so a naive process-alive check flaps between green and red without telling you anything.

This is the poison pill pattern: a single malformed log line that hangs or crashes the parser. Fluentd dies, the supervisor restarts it, in_tail resumes from the last recorded offset in its pos_file, reads the same bad line again, and crashes again. Because the crash happens before Fluentd advances its read position past the offending line, the process is trapped at one byte offset forever.

The distinguishing feature versus a generic crash loop is determinism: the crash happens at the same pos_file position every time, within seconds of startup, with an immediate CPU spike. This article covers how to confirm that signature, how to find the bad line, and how to get the pipeline moving without losing everything behind it.

What this means

Fluentd’s in_tail plugin tracks its read position per watched file in a pos_file, so restarts do not re-read or skip data. That durability is normally a feature. Here it becomes the trap: the parser crashes before the position is updated past the bad line, so every restart replays the crash.

flowchart LR
  A[Application writes bad log line] --> B[in_tail reads line]
  B --> C{Parser handles it?}
  C -->|no: regex hang or crash| D[Process dies]
  D --> E[Supervisor restarts Fluentd]
  E --> F[in_tail resumes from pos_file offset]
  F --> B
  C -->|yes| G[Event emitted, pos advances]

The crash itself takes one of three forms:

• Hang: catastrophic regex backtracking. The Ruby regex engine tries exponentially many match combinations against a malformed line and never returns. CPU pins at 100 percent on one core. Depending on your supervisor and any watchdogs, the process is eventually killed or just stops making progress.
• Crash: an unhandled exception in parsing, such as a stack-level error on deeply nested JSON or an encoding error on invalid UTF-8 bytes written into the log file by a misbehaving application.
• Silent stall: a multi-line regex waiting for a terminator that never comes on an extremely long “line”, blocking the read loop.

In Kubernetes, this surfaces as CrashLoopBackOff with very short run durations. On systemd-managed hosts, it surfaces as a rapidly incrementing restart counter. Either way, downstream log delivery for the whole node stops, because the poisoned input blocks the worker.

Common causes

Quick checks

These are read-only. Run them in order; each narrows the picture.

# 1. Confirm the rapid restart cycle
systemctl status td-agent          # td-agent package
systemctl status fluentd           # fluent-package
journalctl -u td-agent --since "10 minutes ago" | grep -iE "started|stopped|failed"

A restart every few seconds to a couple of minutes is the poison pill cadence. A one-off crash is something else.

# 2. Find the crash reason in Fluentd's own log
grep -iE "error|exception|invalid byte|stack level|backtrace" /var/log/td-agent/td-agent.log | tail -30
# fluent-package:
grep -iE "error|exception|invalid byte|stack level|backtrace" /var/log/fluent/fluentd.log | tail -30

Look for the same exception class repeating once per restart. Encoding errors and deep-nesting stack errors are the classic signatures.

# 3. Rule out OOM as the killer
dmesg | grep -i oom | tail -5

If the OOM killer is doing the killing, you are in the memory-bloat archetype, not the poison pill archetype. See Fluentd monitoring checklist: the signals every production log pipeline needs for the memory pattern.

# 4. Watch CPU on startup: poison pill spikes immediately
top -b -n 3 -d 2 -p $(pgrep -f fluentd | head -1)

CPU pinning to 100 percent of one core within seconds of startup, before any real throughput, is the backtracking signature. Because of the Ruby GVL, one busy thread blocks the whole worker.

# 5. Locate the pos_file and find the stuck position
grep pos_file /etc/td-agent/td-agent.conf     # or /etc/fluent/fluentd.conf
cat /var/log/td-agent/<your>.pos

The pos_file is a plain text file: one line per watched file, containing the file path, the read position, and the inode, with the position and inode recorded in hex. If the recorded position is identical after every restart and the crash happens while parsing, that offset is where your poison pill lives.

# 6. Read the bytes at the stuck offset (substitute path and hex position)
dd if=/var/log/app/app.log bs=1 skip=$((0x<hexpos>)) count=4096 2>/dev/null | cat -v | head -40

cat -v makes non-printing and invalid-UTF-8 bytes visible. You are looking for the line that differs from everything around it: binary garbage, an abnormally long line, or JSON nesting that runs off the screen.

How to diagnose it

1. Confirm the loop is deterministic. From the checks above, verify three things hold simultaneously: restart interval is short and regular, the pos_file position is unchanged across restarts, and the same error (or the same 100-percent-CPU hang) appears each cycle. If any of these fails, you have a different failure archetype.

2. Extract the poison line. Use the dd command from check 6, or open the source log file and jump to the recorded offset. The poison line is at or immediately after the saved position, since Fluentd records the position after successfully processing a line.

3. Identify the failure mode. Match the log evidence:

   • Repeated invalid byte sequence in UTF-8 or similar encoding errors: encoding poison.
   • No exception at all, just a hang at 100 percent CPU: regex backtracking. Take the parser regex from your <parse> block and test it against the extracted line with Rubular or fluentular. If the tester hangs, confirmed.
   • Stack-level or JSON parse errors: malformed or pathologically nested JSON.

4. Check blast radius. If this Fluentd instance handles multiple inputs, the crashed worker takes all of them down, not just the poisoned one. In multi-worker mode, in_tail must be pinned to one worker via <worker N> (each worker runs its own copy otherwise, and duplicates reads of the same files); the poisoned input, its monitor_agent port, and its log lines all belong to that worker.

5. Decide the recovery strategy (next section): skip the line or filter it, then fix the parser so the class of line can never kill the process again.

Metrics and signals to monitor

Fixes

Do these in order. The first two get logs flowing again; the rest make sure it cannot recur.

Advance the pos_file past the bad line

This is the direct unblocking move and it deliberately sacrifices the one poisoned line. Stop Fluentd before editing the pos_file, or the running process may overwrite your change on shutdown.

1. Stop the service: systemctl stop td-agent (or fluentd).
2. Back up the pos_file: cp file.pos file.pos.bak.
3. Edit the hex position for the affected file to a value just past the end of the bad line. You can compute the target offset from the dd inspection: offset of the bad line plus its byte length.
4. Start the service and watch input emit_records and the Fluentd log to confirm it is processing past that point.

An alternative that avoids hand-editing hex: temporarily move the source log file aside (mv app.log app.log.poisoned), restart Fluentd so it starts a fresh file, then re-ingest or archive the quarantined file after the parser is fixed. The tradeoff is the same: events written while the file is moved are handled differently depending on how the application reopens its log, so prefer the pos_file edit when the application holds the file open.

Data-loss note: either approach skips the bad line and possibly a few lines around it. That is almost always the right trade against an indefinitely down pipeline, but record the skipped offset range in the incident notes.

Exclude the pattern with a grep filter

If the poison line matches a recognizable pattern (a specific error signature, a known binary prefix), add a grep filter that drops it before it reaches a fragile downstream parse step. The deprecated excludeN style is gone in modern configs; use directive blocks:

<filter app.**>
  @type grep
  <exclude>
    key message
    pattern /KNOWN_BAD_SIGNATURE/
  </exclude>
</filter>

Tradeoff: this is a targeted drop rule, and it only protects stages after the filter. If the crash happens in the input’s own <parse> block, the grep filter cannot help, because parsing happens before filtering. In that case the pos_file advance plus the parser fix is the only path.

Fix the regex

If backtracking is the cause, the regex is the bug. Rewrite the offending alternation or nested quantifier so match time stays linear, and test the new pattern against the actual poison line plus a sample of normal lines before deploying. Never deploy a parser regex change untested. Long term, prefer structured formats (JSON) over regex parsing for sources you control, and cap line length for sources you do not.

Bound the read loop

Two guardrails limit how much damage one pathological input can do per cycle:

• read_lines_limit caps how many lines in_tail processes per I/O operation (default 1000). Lowering it reduces how hard a backtracking hang bites per cycle, at the cost of slightly slower catch-up on healthy files.
• max_line_size (available since v1.14.4) skips lines longer than a set threshold. This neutralizes the “extremely long line” class of poison pill outright, at the cost of silently dropping legitimately huge log lines if you set it too low.

Prevention

• Test parser regexes against hostile input. Any regex with nested quantifiers or ambiguous alternation should be exercised against truncated, concatenated, and binary-corrupted samples, not just well-formed lines.
• Prefer structured logging at the source. Applications that emit JSON remove most of the regex attack surface. Keep a regex parser only where you cannot change the emitter.
• Set max_line_size on high-volume tail inputs where supported, sized above your largest legitimate line.
• Alert on restart cadence, not just liveness. A process that restarts every 20 seconds is “alive” to most checks. Track the restart rate and the input emit_records flatline so the loop pages someone in minutes, not after a downstream gap is noticed.
• Protect the pos_file. It is a plain text file with no checksums. Keep it on a reliable filesystem, and treat unexplained position resets as an incident, since corruption produces duplicate reads or gaps that look nothing like a crash loop.
• Watch input emit rates per source. A flat input counter on one source while others flow is the earliest indicator of a stuck read loop, even when the process has not crashed yet.

How Netdata helps

• Per-process CPU and uptime curves make the loop visible at a glance: sawtooth uptime and a 100 percent single-core spike on every start, which is the backtracking signature.
• Process restart counting distinguishes a crash loop from a one-off OOM or deploy restart without log spelunking.
• Correlating Fluentd’s input throughput against the process lifecycle shows the flatline between restarts, confirming ingestion is stalled at a fixed position.
• RSS trends alongside restart events separate the poison pill pattern (flat memory, CPU-driven death) from the memory-bloat OOM pattern (monotonic growth before death).
• Alerts on process liveness with sustained-failure gating catch the loop without paging on every transient restart.

Related guides

• Fluentd monitoring checklist: the signals every production log pipeline needs
• Fluentd monitoring maturity model: from survival to expert