$ guides / fluentd / fluentd-worker-died ▌

Operations Guides

Fluentd worker died: a partial outage the supervisor hides

systemctl status td-agent says active (running). The supervisor PID exists. Host-level checks are green. Meanwhile, one of your four Fluentd workers died twenty minutes ago, and a quarter of the log pipeline is gone or degraded.

This is the trap of Fluentd multi-worker mode. When you set workers N in <system>, Fluentd starts a supervisor process and N independent Ruby workers. Each worker has its own event loop, buffers, output threads, and, if configured, monitor_agent endpoint. Workers do not share in-memory state.

When one worker dies, the others keep running and the supervisor stays alive. Process-liveness checks, the systemd unit state, and the pidfile still report success. The result is a partial outage: reduced throughput, worker-specific buffer growth or loss, and no alert from monitoring designed around a single process.

What this means

The supervisor spawns workers, watches them, and restarts them when they exit. It is not in the data path. It can be healthy while the pipeline is degraded.

Two properties create the blind spot:

Supervisor liveness is not worker liveness. A dead worker does not change the supervisor’s state, the systemd unit state, or the pidfile.
Workers are independent. Each has its own buffers and monitor_agent HTTP server. With the usual port allocation, worker 0 listens on 24220, worker 1 on 24221, and so on. A check that only polls worker 0 will not notice that worker 2 is gone.

flowchart TD
  S[Supervisor process alive] --> W0[Worker 0: running]
  S --> W1[Worker 1: running]
  S --> W2[Worker 2: dead]
  S --> W3[Worker 3: running]
  W2 --> B2[Memory buffer and in-flight events at risk]
  W2 --> M2[monitor_agent port 24222: connection refused]
  S -.->|systemctl status: active| OK[Host checks report green]

By default, the supervisor restarts a dead worker immediately. The common failure is therefore often not “worker missing for hours” but “worker crash-looping” or “worker restarted and lost memory-buffered events.” Fast restarts leave little evidence in process-count monitoring unless you also track process age, worker exit logs, or per-worker metrics.

Common causes

Cause	What it looks like	First thing to check
OOM kill of one worker	Worker restarts; kernel log contains an OOM entry for a Ruby process; RSS was high or climbing	`dmesg -T \| grep -i oom`
Unhandled plugin exception	The same worker exits repeatedly, often with a Ruby stack trace	Fluentd log for worker exit and plugin error lines
Poison-pill log line	Worker crashes, restarts, reads the same malformed line from its pos_file, and crashes again	Whether the pos_file offset repeats across restarts
`in_tail` outside a worker pin	`in_tail` is not multi-worker safe; unpinned workers can contend for the same files or fail at startup	Every `in_tail` source has an explicit `<worker N>` section
Per-worker resource exhaustion	One worker hits a per-process FD limit while the others remain healthy	`/proc/<worker_pid>/fd` count versus that worker’s limit
Failed reload	A worker is stopped for reload and its replacement exits or crash-loops	Fluentd log around the SIGHUP or reload timestamp

For OOM kills in Kubernetes, also inspect the container’s last state for OOMKilled; kernel log access from the node may be restricted. Unflushed events in a memory buffer are at risk. File buffers on persistent storage can be drained after restart, but expect a delivery gap and replay burst.

Quick checks

All checks are read-only. Adjust paths, unit names, and ports for your deployment.

# 1. Compare live Fluentd processes with the configured worker count
pgrep -af fluentd
grep -R 'workers' /etc/td-agent/ 2>/dev/null

For workers N, you should see one supervisor plus N workers for the relevant Fluentd instance. Fewer than N workers means one is down now.

# 2. Probe each worker's monitor_agent endpoint
for p in 24220 24221 24222 24223; do
  code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 http://localhost:$p/api/plugins.json)
  echo "port $p: $code"
done

With the usual base-24220 allocation, 000 or a timeout on one port while the others return 200 identifies a dead or wedged worker. Adjust the range to your worker count and monitor_agent configuration.

# 3. Check for OOM kills
dmesg -T | grep -iE "oom|killed process" | tail -20

# 4. Find worker exits in the Fluentd log
grep -iE "finished unexpectedly|worker.*(signal|exit|dead)" \
  /var/log/td-agent/td-agent.log | tail -20

The supervisor records worker deaths and their exit condition. A common message contains finished unexpectedly. Repeats for the same worker ID indicate a crash loop rather than a one-off restart.

# 5. Count recent worker-death events when Fluentd logs to the journal
journalctl -u td-agent --since "1 hour ago" |
  grep -ciE "finished unexpectedly|worker.*(signal|exit|dead)"

A nonzero or rising count with a green unit status confirms that supervisor restarts are masking worker failures. If Fluentd logs to a file instead, apply the same pattern to that file.

# 6. Compare worker age and RSS
ps -o pid,etimes,rss,comm --sort=-rss -p $(pgrep -f fluentd)

One worker with RSS far above its siblings is an OOM candidate. One worker with a much lower etimes value has restarted recently.

How to diagnose it

Establish the expected shape. Read workers N from the effective configuration, including included files. On Fluentd before v1.19.0, set enable_input_metrics true if you need per-worker input emit_records; otherwise those counters can read as zero.
Map live processes to workers. If pgrep -af fluentd shows fewer than N workers, one is down now. If the count is correct, compare process ages. A replacement worker can restore the count before a periodic process check notices the gap.
Read the supervisor log around the death. A SIGKILL with no preceding application error points toward the kernel OOM killer. SIGABRT, a Ruby exception, or a plugin stack trace points toward application or plugin code. Correlate the timestamp with the kernel OOM log.
Compare per-worker throughput. Poll every monitor_agent endpoint and compare input and output counters. A restarting worker shows counters that reset. A live but wedged worker keeps its endpoint or process alive while counters remain flat. These require different fixes: crash loop versus hang.
If the worker crash-loops, check for a poison pill. Compare the relevant pos_file offset across restarts. A repeated offset suggests that the worker dies on the same malformed line before advancing its cursor.
Determine what can be replayed. Treat unflushed memory-buffer contents as lost unless the input can replay them. Persistent file buffers should be drained by the replacement worker. A replay burst and some duplicate events are possible after restart because output delivery is at least once.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Running worker count versus `workers N`	Directly detects this failure	Count stays below N beyond the restart or reload window
Per-worker monitor_agent responsiveness	A dead worker’s HTTP server dies with it	One port times out or returns non-200 while siblings return 200
Per-worker RSS	OOM kills target individual processes	One worker trends upward or approaches its limit
Supervisor worker-exit messages	The death is explicitly recorded here	Any exit line; repeated exits for the same worker ID
Per-worker emit rates	Restarts reset counters and hangs flatten them	Counter resets or a flat rate on one worker only
Per-worker `buffer_queue_length`	Buffers are worker-local	Queue growth isolated to one worker
Per-worker process age	Exposes restarts hidden by a recovered process count	One worker is much younger than its siblings

Evaluate these signals per worker. Aggregates hide the failure: average RSS across four workers can look normal while one worker is near its limit.

Fixes

Worker was OOM-killed

Use file-backed buffers on persistent storage so a worker death does not destroy the buffered backlog. Set explicit chunk_limit_size and total_limit_size values instead of allowing an unbounded backlog. Increase the memory limit if the worker’s normal workload no longer fits, and leave headroom above its normal RSS plateau.

Ruby RSS often plateaus at a high value; a sustained upward trend is the leak signal. If one worker consistently uses more memory than the others, check whether its routing, watched files, or pinned input gives it more work.

Worker crash-loops on a poison pill

Do not let the supervisor restart it indefinitely. Identify the bad source line and pos_file offset first.

Do not edit a live pos_file. Stop Fluentd or otherwise ensure the cursor is no longer being written before changing it; cursor edits can skip or duplicate data. Advance the cursor past the offending line only after recording the skipped range. Then fix the parser. Catastrophic regex backtracking and malformed JSON are common causes.

Worker died during a reload

During a graceful reload, worker stop and start churn is expected. If a replacement exits immediately, the unit can remain active while one worker is missing or crash-looping. Treat the new configuration as the suspect and validate it before reloading:

td-agent --dry-run -c /etc/td-agent/td-agent.conf

A dry run does not catch every runtime-only plugin failure, so correlate the reload timestamp with the worker exit log. If the new configuration only partially applied, use the reload-failure guide linked below.

Reduce the blast radius of the next death

Since v1.15.0, restart_worker_interval in <system> can delay worker restarts. The default is 0, which restarts immediately. A non-zero interval reduces tight crash-loop churn at the cost of a longer partial outage. Set your worker-count alert threshold longer than this interval and the normal reload window.

Pin in_tail to a specific worker with <worker N>. This makes the failure domain explicit and keeps its metrics on a known monitor_agent port.

Monitoring fix

Alert when the live worker count differs from workers N for longer than the expected restart or reload window. Add an HTTP check for every worker’s monitor_agent endpoint. Pair those checks with process age or supervisor worker-exit logs so an immediate restart cannot hide a crash loop.

Prevention

Alert on per-worker health. The running worker count must equal workers N. This closes the blind spot that systemctl status leaves open.
Use file-backed buffers on persistent storage. A dead worker with memory buffers risks confirmed loss; a dead worker with persistent file buffers usually causes a delivery delay.
Monitor per-worker RSS against limits. OOM kills select one process, so aggregate host memory is the wrong granularity.
Alert on supervisor worker-exit messages. Apply sustained-failure logic so a normal single restart does not page you.
Pin in_tail to a worker. Monitor that worker’s port for tail and rotation metrics; in_tail does not support ordinary multi-worker operation.
Load-test parser chains with real samples. Regex blowups and parser crashes often appear only under production log volume.

How Netdata helps

Netdata process and application charts expose Fluentd process count, CPU, memory, and FD usage. A count below the expected supervisor-plus-worker total catches a missing worker even when systemd remains green. If workers are grouped together, use process grouping or per-endpoint metrics to attribute the failure.
Process uptime and restart indicators expose frequent worker resets that a recovered process count would otherwise hide.
If you use Netdata’s Fluentd collector, configure one scrape job per monitor_agent endpoint. Per-worker buffer_queue_length and retry_count make a dead, restarting, or wedged worker visible as a missing, reset, or flat series. Track emit_records where it is exposed.
Put process count, RSS, uptime, and per-worker buffer charts side by side. That makes it possible to distinguish an OOM restart from an output stall without waiting for a postmortem.