Fluentd process not running: the log pipeline is dead and the host has gone dark

The Fluentd process is gone. No logs are being collected, parsed, buffered, or forwarded from this host. Your downstream systems (Elasticsearch, S3, a SIEM, an aggregator tier) are now receiving nothing from here, and most of them will not tell you that. Log pipelines fail silently at the consumer side: the absence of data looks identical to a quiet host.

This is a full observability blackout for the host and a page-worthy condition in most environments. The one nuance: brief absences are normal. Fluentd restarts during config reloads, rolling updates, package upgrades, and container rescheduling. Gate your page on sustained absence (more than about 2 minutes) so a routine restart at 3 a.m. does not wake anyone up.

This guide covers confirming the process is really dead (not just hidden behind a live supervisor), finding the crash reason, and recovering without losing more data than necessary.

What this means

A default Fluentd installation runs two Ruby processes: a supervisor and a worker. The supervisor owns the worker lifecycle and restarts it after a crash. In multi-worker mode (workers N in <system>), there are N worker processes under one supervisor. This creates a failure mode that fools naive checks:

• The supervisor can be alive while every worker is dead. systemctl status td-agent reports “active” because systemd tracks the supervisor. Meanwhile, zero events are flowing.
• A worker killed by the OOM killer may be restarted by the supervisor within seconds. The outage is real but invisible unless you track restarts or crash log lines.
• In Kubernetes, CrashLoopBackOff produces rapid up/down oscillation. Sustained absence shows up as repeated brief absences rather than one long gap, so a naive “down for 2 minutes” check may never fire even though the pipeline is effectively dead.

The crash reason matters for data loss. File-backed buffers survive a restart and replay on recovery. Memory-backed buffers are gone the instant the process dies. If in_tail position files are intact, collection resumes where it left off; if the pos_file was lost or corrupted, you get duplicates or gaps.

flowchart TD
  A[Fluentd process check fails] --> B{Supervisor alive?}
  B -- No --> C[Check systemd unit and journal for boot failure]
  B -- Yes --> D{Workers alive?}
  D -- No --> E[Check supervisor log for worker exit signal]
  D -- Yes --> F[Process exists: check monitor_agent responsiveness]
  C --> G[Config error or plugin load failure at boot]
  E --> H{Exit signal}
  H -- SIGKILL --> I[OOM killer: check dmesg]
  H -- SIGSEGV --> J[Crash in Ruby or C extension: check core logs]
  H -- non-zero exit code --> G
  F -- Unresponsive --> K[Process hung, not dead: treat as separate incident]

Common causes

Quick checks

Run these read-only checks in order. They take under a minute and localize the failure.

# 1. Service-level status (adjust unit name for your package)
systemctl status td-agent   # td-agent package
systemctl status fluentd    # fluent-package

# 2. Actual process list: you should see a supervisor AND at least one worker
pgrep -af fluentd

# 3. For td-agent: two ruby processes expected by default (supervisor + worker)
ps w -C ruby -C td-agent --no-heading

# 4. Kernel OOM events
dmesg | grep -i oom | tail -20

# 5. Recent unit logs: crash reason, restart loop, config errors
journalctl -u td-agent --since "30 minutes ago" | tail -50

# 6. Fluentd's own log: worker exit lines and plugin errors
grep -iE "finished unexpectedly|LoadError|uncaught|error" /var/log/td-agent/td-agent.log | tail -30
# fluent-package path: /var/log/fluent/fluentd.log

# 7. Monitor agent responsiveness (only if configured; default port 24220)
curl -s -o /dev/null -w "%{http_code}\n" --max-time 5 http://localhost:24220/api/plugins.json

Interpretation notes:

• Steps 2 and 3 are the checks that matter. A live supervisor with no workers means step 1 lies to you.
• Step 6 looks for the supervisor’s “Worker N finished unexpectedly with signal …” lines. SIGKILL almost always means the OOM killer. SIGSEGV means a crash in Ruby or a C extension. A clean non-zero exit code points at config or plugin failure.
• Step 7 distinguishes “dead” from “hung”. A process that exists but does not answer the monitor agent within 5 seconds is a different incident (deadlock, GC storm, blocked event loop), not this one.

How to diagnose it

1. Confirm the blast radius. Is this one host or many? If a DaemonSet or a whole aggregator tier lost Fluentd at once, suspect a shared cause: a bad config push, a package upgrade, a destination outage that cascaded into OOM, or a certificate expiry. Check a second host before debugging the first in depth.

2. Establish the process topology. Count supervisors and workers. Expected: one supervisor plus workers N workers (default N=1). If the supervisor is missing entirely, the failure is at the unit level (systemd gave up restarting, or the boot sequence never got there). If the supervisor is present but workers are missing, read the supervisor log for worker exit lines.

3. Get the crash reason from the journal, not just Fluentd’s log. A Fluentd worker that dies from a Ruby segfault or an OOM kill may never write its final error to /var/log/td-agent/td-agent.log. The kernel and systemd records survive: journalctl -u td-agent for exit codes and restart counts, dmesg for OOM kills. Check whether systemd has been restarting the unit repeatedly; that masks the outage as flapping rather than a clean “down”.

4. Correlate with change. Did a config deploy, package upgrade, or logrotate run happen just before the death? Config syntax errors and plugin load failures show up immediately at boot and produce a tight restart loop. Plugin load errors look like LoadError or “cannot load” lines in the journal at startup.

5. Check the resource ceiling. If the evidence points at SIGKILL/OOM: was it the host OOM killer or a container memory limit? In Kubernetes, kubectl describe pod shows OOMKilled as the last state and the restart count. On a bare host, dmesg shows which process was chosen and the memory state at the time. Ruby RSS grows and plateaus; a plateau is normal, a monotonic rise is a leak or unbounded memory-backed buffers.

6. Rule out the poison pill. If the process restarts and dies again within seconds, repeatedly, at roughly the same input position, suspect a malformed log line crashing a parser. The crash recurs because in_tail resumes from the saved pos_file position and re-reads the same bad line. Look for parse exceptions or regex errors in the Fluentd log immediately before each death.

7. Verify recovery actually restored flow. After the process is back, “running” is not enough. Confirm the monitor agent returns 200, that input emit_records is incrementing, and that output write_count is incrementing. A live process with a full buffer and a dead destination is still a dark host.

Metrics and signals to monitor

Two gating rules for paging:

• Page on sustained absence, not instantaneous absence. More than 2 minutes covers restarts, rolling updates, and container rescheduling without delaying real pages materially.
• In Kubernetes, also page on CrashLoopBackOff or a restart count climbing past a handful within a short window. Sustained-absence logic alone can miss a tight crash loop, because the process is technically “up” for a few seconds each cycle.

Fixes

OOM kill

1. Confirm with dmesg or the pod’s last state, then decide whether the limit is wrong or the usage is wrong.
2. If buffers are memory-backed, switch to file-backed (@type file in the buffer section). This removes the largest memory consumer and makes buffered data survive the next crash.
3. Increase chunk_limit_size so fewer, larger chunks exist. Millions of small chunks create millions of Ruby objects and heavy GC pressure.
4. Raise the container or systemd memory limit with headroom above the observed RSS plateau. Ruby fragmentation means the plateau sits higher than the configured buffer sizes suggest; leave at least 20-30% above normal RSS.
5. If RSS grows monotonically even with file buffers and stable throughput, suspect a plugin leak. Check whether the growth correlates with a recent plugin addition or upgrade.

Tradeoff: file-backed buffers cost disk I/O and need disk headroom on the buffer path. That trade is almost always worth it for crash durability.

Config syntax error or plugin load failure at boot

1. Pull the exact error from journalctl -u td-agent or the Fluentd log. Syntax errors name the file and line.
2. Roll back the offending config change if it came from a deploy. Do not leave the unit crash-looping while you edit in place on the host.
3. Validate before the next reload: fluentd --dry-run -c /etc/td-agent/td-agent.conf parses the config without starting the pipeline. Make this a CI or deploy-pipeline step so the next bad config never reaches a host.
4. For LoadError-style failures, check for a missing gem or an incompatible plugin version after a package upgrade. Plugin load errors at startup appear in the journal even when the unit looks merely “flapping”.

Poison pill crash loop

1. Confirm by matching crash times to the same pos_file position and to parse exceptions in the log.
2. Stop the loop by moving Fluentd past the bad line: advance the position in the pos_file, or temporarily move the offending source log file aside. This is disruptive and skips data; do it deliberately. The bad line is lost either way, so the choice is between losing one line and losing everything behind it.
3. Fix the parser: eliminate catastrophic regex backtracking, cap line sizes, or add a filter that excludes the malformed pattern.
4. Set read_lines_limit on in_tail to bound per-cycle processing.

Supervisor alive, workers dead

1. Treat this as a monitoring gap first: your process check was watching the parent PID. Fix the check to count workers.
2. Restart the unit (systemctl restart td-agent) to get a clean supervisor/worker pair. This is disruptive to any in-flight in-memory state, but the workers are already dead, so there is little left to lose. If workers die again immediately, you have one of the causes above; the supervisor restart loop is a symptom, not the disease.
3. If you run with --no-supervisor under an external supervisor (systemd, runit), verify that the external supervisor is actually restarting the worker and that you have not silently lost the auto-restart behavior the built-in supervisor would have provided.

After any recovery

• Expect a brief throughput spike from file-buffer replay and a burst of duplicates. File-backed buffers replay unflushed chunks by design; exactly-once delivery is not guaranteed. Do not page on the replay burst.
• If buffers were memory-backed, accept the gap and quantify it: compare input and output emit_records at the destination for the outage window.
• Verify the pos_file survived. If it was on volatile storage (tmpfs, container ephemeral layer), in_tail will either re-read everything (duplicates, if read_from_head true) or skip history (gaps).

Prevention

• Worker-aware liveness checks. Alert on the expected worker process count, not the unit status and not the parent PID. In multi-worker mode, check each worker’s monitor_agent port: each worker binds its own port by adding its worker id to the configured port (worker 0 is 24220, worker 1 is 24221, and so on).
• Track restart counts. The single most reliable early signal of a crash loop. A unit that restarted 5 times in an hour is telling you the next OOM or poison pill is already scheduled.
• File-backed buffers in production. Crash durability plus lower memory pressure. Verify with /api/plugins.json?with_config=true that each output’s buffer section actually says @type file.
• Config validation in the deploy pipeline. Dry-run the config before it ships. Boot-time syntax errors are the cheapest class of Fluentd outage to eliminate entirely.
• Memory headroom discipline. Set container limits at least 20-30% above the observed RSS plateau, and alert on the RSS trend, not the absolute value. Plateaus are normal for Ruby; slopes are not.
• Persistent pos_file storage. Keep position files off tmpfs and off the container ephemeral filesystem so restarts resume instead of replaying or skipping.
• Watch Fluentd’s own log. The log pipeline’s own error log is routinely the one log nobody collects. Ship it somewhere that survives the host going dark.

How Netdata helps

• Process liveness and restart tracking. Netdata’s process and systemd-unit monitoring show the Fluentd process disappearing and, critically, the restart count climbing, which is what turns a masked crash loop into a visible signal.
• Fluentd collector via monitor_agent. Netdata polls the monitor_agent endpoint and charts buffer queue length, retry counts, and emit rates per plugin, so you can confirm after recovery that replay is draining and flow is actually restored rather than just “process is up”.
• Memory correlation. Per-process RSS alongside cgroup memory limits shows the monotonic growth trend that precedes an OOM kill, which is the difference between preventing the next crash and explaining the last one.
• OOM and log context on one timeline. Kernel OOM events, RSS, Fluentd restarts, and buffer depth land on the same dashboard timeline, so the SIGKILL-to-OOM-to-restart chain takes minutes to confirm instead of a dmesg archaeology session.
• Sustained-absence alerting. Netdata alert hysteresis lets you require the process to be absent for a sustained window before paging, matching the 2-minute gating rule and filtering out routine restarts.

Related guides

• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• Fluentd poison pill crash loop: one bad log line that kills the process on every restart
• Fluentd monitor_agent not responding: a process that is up but hung
• Fluentd plugin load error at startup: LoadError and missing gems
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd monitoring checklist: the signals every production log pipeline needs
• How Fluentd actually works in production: a mental model for operators
• Fluentd monitoring maturity model: from survival to expert
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer stage vs queue: telling healthy batching from backpressure
• Fluentd buffer available space low: computing time-to-overflow before it fires