Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes

A Fluentd DaemonSet pod in CrashLoopBackOff is not “down.” It is oscillating: the container starts, runs for seconds or minutes, dies, and kubelet restarts it with an exponentially growing backoff. kubectl get pods shows the pod flapping between Running, Error, and CrashLoopBackOff while the restart count climbs.

Because kubelet keeps resurrecting the process, you rarely see one long outage. Instead you see repeated brief absences: buffers drain partially and refill, and downstream destinations see gaps and duplicate windows as file-backed buffers replay on each restart. The auto-restart that keeps the node “mostly covered” is also what masks the root cause.

The fix is never “restart it again.” Kubelet is already doing that. The job is to find why the process keeps dying, using the previous container’s logs, kernel OOM records, and restart timing, then remove the specific crash trigger.

What this means

CrashLoopBackOff is a Kubernetes pod status, not a Fluentd error. It means the container’s main process exited with a non-zero status or was killed, and kubelet’s restart policy (Always, for a DaemonSet) is restarting it with backoff that grows from seconds up to a cap. The “BackOff” part is kubelet waiting before the next attempt.

Three properties matter operationally:

• Restart count is the real signal. A pod that is Running right now with 47 restarts is sicker than a pod in CrashLoopBackOff with 2 restarts after a deploy. Track the count and its rate of increase, not the current state.
• The previous container’s logs are the primary evidence. Once the container restarts, kubectl logs shows the fresh, healthy-looking startup of the new attempt. The crash reason lives in the prior instance.
• Each restart has side effects. File-backed buffers replay unflushed chunks, causing duplicate delivery downstream. in_tail position files usually survive (they are on a hostPath or persistent volume), so collection resumes, but every crash window is a collection gap.

flowchart TD
  A[Fluentd process exits] --> B{Exit reason?}
  B -->|SIGKILL, no app error| C[OOM kill - check dmesg and pod events]
  B -->|Non-zero exit, exception in logs| D{Where in logs?}
  D -->|During startup| E[Config error or plugin load failure]
  D -->|After running, same pos each time| F[Poison pill log line]
  D -->|After running, varied| G[Plugin exception or destination fatal error]
  C --> H[Fix memory limit or buffer config]
  E --> I[Fix config or plugin]
  F --> J[Advance pos_file past bad line]
  G --> K[Fix plugin or destination]

Common causes

The OOM case deserves emphasis: container memory limits are hard ceilings. There is no gradual degradation; the cgroup OOM killer terminates the process immediately. Fluentd’s own log may show nothing more than the worker finishing unexpectedly, because a SIGKILL leaves no time to log anything. If you only read Fluentd’s logs, an OOM kill looks like the process vanished for no reason.

Quick checks

All read-only.

# 1. Pod status, restart count, and how long restarts have been happening
kubectl get pods -n logging -l app=fluentd -o wide

# 2. Termination reason, exit code, memory limit, recent events
kubectl describe pod -n logging <fluentd-pod>

# 3. The previous container's logs - this is where the crash reason lives
kubectl logs -n logging <fluentd-pod> --previous --tail=100

# 4. Current attempt's logs (often just a clean startup - compare timing)
kubectl logs -n logging <fluentd-pod> --tail=50

# 5. Kernel OOM records on the node where the pod runs
ssh <node> 'dmesg -T | grep -i -E "oom|killed process" | tail -20'

# 6. Kernel journal on the node, if dmesg is rotated
ssh <node> 'journalctl -k --since "2 hours ago" | grep -i oom'

# 7. If the pod is briefly up: is the monitor agent responsive?
kubectl exec -n logging <fluentd-pod> -- \
  curl -s -o /dev/null -w "%{http_code}" --max-time 5 \
  http://localhost:24220/api/plugins.json

# 8. While up: buffer state per output (is it dying under buffer pressure?)
kubectl exec -n logging <fluentd-pod> -- \
  curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, queue: .buffer_queue_length, avail_pct: .buffer_available_buffer_space_ratios, retries: .retry_count}'

How to diagnose it

1. Confirm the restart pattern. From kubectl get pods, note RESTARTS and AGE. Divide restarts by age to get the crash rate. Fast cycles (every 30 seconds) point at startup failures: config, plugin load, corrupt state. Slow cycles (every few hours) point at runtime failures: OOM from memory bloat, poison pill, destination fatal error.

2. Read the termination reason. kubectl describe pod shows Last State: Terminated with a Reason and Exit Code. OOMKilled settles the primary question immediately. An exit code without OOMKilled means Fluentd exited on its own, usually via an unhandled exception.

3. Pull the previous container’s logs. This is the single most important step. Look at the last lines before exit:

   • An exception during startup, with the same error every cycle, is a config or plugin problem. It will never self-heal.
   • A parse exception or stack overflow after some uptime, recurring at the same point, is a poison pill: Fluentd restarts, in_tail resumes from the saved pos_file position, reads the same malformed line, and crashes again.
   • No error at all, just an abrupt end, strongly suggests SIGKILL from the OOM killer. Cross-check with step 4.

4. Check the kernel for OOM kills. On the node, dmesg -T | grep -i oom shows which process was killed and its RSS at the time. Match timestamps to pod restart times. If OOM is confirmed, the question shifts from “why does it crash” to “why does memory grow”: memory-backed buffers growing unbounded, the Ruby fragmentation plateau sitting too close to the limit, a leaking plugin, or tag explosion.

5. Correlate crash timing with log sources. For suspected poison pills, check which files Fluentd was tailing and whether the crash time matches a specific application’s output. The pos_file records byte offsets per file; if each crash cycle dies at the same offset in the same file, you have found the bad line.

6. Check buffer and plugin state on the host. If crashes follow a node reboot or unclean shutdown, inspect the buffer directory (buffer_path) and any plugin storage files on the hostPath. Corrupted or empty state files can make Fluentd fail during startup with Invalid contents (not object) in plugin storage file (a Fluent::ConfigError). Storage plugin state defaults to <path>/worker<worker_id>/storage.json, or <plugin-root-dir>/storage.json when no explicit path is set.

7. Verify the destination independently. If the previous logs show the output plugin failing during startup (connection refused, TLS errors, auth failures), test the destination from the node directly. Some output plugins raise during start rather than buffering gracefully, so an unreachable destination becomes a crash loop instead of a retry loop. Which plugins do this varies by version and plugin; for example, the S3 output plugin has long raised on invalid credentials at flush time (fluent-plugin-s3 issue #417, confirmed as expected behavior), while core out_http/out_forward enter retry instead.

Metrics and signals to monitor

Fixes

OOM kill

• Switch buffers to file-backed (@type file) if you are running memory-backed buffers. File buffers trade disk I/O for durability and move the growth off the Ruby heap. Memory buffer data is also lost on every crash, so file-backed buffers reduce both the crash frequency and the blast radius of each crash.
• Raise the container memory limit with real headroom. Ruby RSS grows and plateaus due to fragmentation; the plateau is normal, but it is usually higher than teams expect. Set the limit with at least 20% headroom above observed steady-state RSS. Do not size the limit from RSS right after startup.
• Increase chunk_limit_size. Bigger chunks mean fewer Ruby objects and less GC overhead, which lowers both CPU and fragmentation pressure.
• Check for tag explosion. Dynamically generated tags (per-pod, per-container labels expanded into routing keys) create unbounded routing state. Count distinct tags; if the number grows with pod churn, cap the tag cardinality.

Config or plugin load failure

• Fix the configuration error shown in the previous container’s logs, and validate the config before rolling it out. If the failure followed an upgrade, check that all plugins in the config are present in the new image; a missing gem after an image bump fails at startup every cycle.
• Note for the official container images: since the v1.17.0-1.3 / v1.16.5-1.3 image tags, jemalloc is disabled by default in the docker and kubernetes-daemonset images, because the combination of the systemd plugin and jemalloc caused crashes (free(): invalid pointer). The toggle is image-wide; if you re-enabled jemalloc via LD_PRELOAD on an image that uses the systemd plugin, that is a candidate crash cause (the images document enabling it via LD_PRELOAD when you do not use the systemd plugin).

Poison pill

• Advance the pos_file past the bad line. Edit the byte offset in the pos_file for the affected file (each line is filepath, read position, inode — position and inode in hex, position before inode), or temporarily move the offending source log so Fluentd cannot re-read it. This breaks the loop immediately. Take a copy of the pos_file before editing it.
• Add a filter to drop the offending pattern so recurrence does not re-trigger the loop.
• Fix the parser. Catastrophic regex backtracking on malformed input is the classic cause. Simplify the regex or switch to a structured format where possible. Setting read_lines_limit caps per-cycle processing and limits how far one bad file can run away.

Destination fatal at startup

• Restore or fix the destination, then let the pod restart on its own. If the destination will be down for a while and the output plugin crashes at startup instead of buffering, temporarily route to a secondary output rather than leaving the node without collection.
• Longer term, prefer output plugin configurations that tolerate destination startup failure and buffer. Verify retry and secondary behavior in a staging cluster before you need it.

Corrupt buffer or plugin state

• After confirming from logs that startup fails reading a state file, remove the corrupted buffer chunk or storage file from the host. This is destructive to the buffered data in that chunk: you are trading that backlog for a working agent. Check first whether the file is empty or truncated after an unclean shutdown; those are safe-ish to delete, but anything with real queued data is a judgment call.

Prevention

• Alert on restart rate, not pod state. A restart-count derivative alerts through the flapping. Gate process-down pages with a sustained-absence condition (for example, absent more than 2 minutes) so rolling updates and single restarts do not page. In a crash loop, sustained absence shows up as repeated brief absences, so pair the page with restart-rate detection.
• Use file-backed buffers in production. Every crash loses memory-buffer contents. File buffers survive restarts and turn a crash loop from repeated data loss into repeated brief gaps plus a replay burst.
• Size memory limits against the Ruby plateau, not startup RSS. Give the container at least 2-3x the expected buffer memory footprint, and keep 20% headroom above observed steady-state RSS.
• Validate config before rollout. A DaemonSet config error takes down collection on every node simultaneously. Dry-run the config in CI or on a canary node.
• Test with hostile input. Replay malformed, huge, and binary log lines against your parser chain in staging. Poison pills are found in production by default; find them earlier.
• Keep the previous-logs reflex. Make kubectl logs --previous the first command in the runbook entry, not an afterthought.

How Netdata helps

• Per-second pod and container metrics show the restart oscillation directly: container uptime sawtoothing, restart count climbing, and RSS climbing to the limit before each kill, which separates OOM loops from exception loops at a glance.
• Memory trend correlation puts container RSS next to the cgroup limit over the pod’s lifetime, so you can see whether the plateau is healthy or a monotonic leak heading for the next OOM.
• Fluentd monitor agent collection tracks buffer_queue_length, buffer_total_queued_size, retry_count, and input vs output emit_records per plugin, so you can tell whether each crash follows backpressure building or arrives with buffers empty.
• Node-level visibility surfaces the context kubelet hides: host memory pressure, disk usage on the buffer hostPath, and file descriptor consumption, all on the same timeline as the restart events.
• Restart-rate alerting fires on the derivative of restart count rather than instantaneous pod state, catching crash loops that look “Running” in any single sample.

Related guides

• Fluentd monitoring checklist: the signals every production log pipeline needs
• Fluentd monitoring maturity model: from survival to expert