$ guides / fluentd / fluentd-oom-killed ▌

Operations Guides

Fluentd OOM killed: memory bloat, the OOM killer, and lost buffers

Fluentd disappeared. The process is gone, systemd or Kubernetes restarted it, and there is a gap in your log storage covering the last few minutes or hours. A check of dmesg shows the OOM killer picked Fluentd as its victim. Memory grew for hours, GC fought harder and harder, and then the kernel ended it.

This failure mode is expensive because of what dies with the process. If your outputs use memory-backed buffers, every staged and queued chunk that had not yet been flushed is gone. The OOM kill is not a graceful shutdown. Ruby cleanup code does not run, so flush_at_shutdown never gets a chance to fire. In Kubernetes the same mechanism shows up as CrashLoopBackOff: the pod restarts, memory climbs back to the cgroup limit, and the kubelet kills it again.

The fix is usually not “add more memory”. It is identifying which of a handful of known causes is inflating RSS, then attacking that cause directly.

What this means

The Linux OOM killer terminates a process when the system (or the cgroup, in Kubernetes) runs out of memory. Fluentd is a frequent victim because its memory profile combines three things: a Ruby heap that fragments and rarely returns memory to the OS, memory-backed buffer chunks that consume RAM directly, and event objects that multiply with tiny chunk sizes or dynamic tags.

Two distinct patterns produce the kill:

Buffer-driven growth: an output stalls, memory-backed buffers fill toward total_limit_size, and buffer_total_queued_size climbs in lockstep with RSS. The first cliff is buffer overflow; the second cliff, right behind it, is the OOM kill.
Leak or fragmentation-driven growth: RSS rises monotonically over hours or days while buffer metrics stay flat. This points at Ruby heap fragmentation, a leaking plugin, or object proliferation from small chunks or tag explosion.

A high but stable RSS is normal for Ruby. The plateau from fragmentation is expected. What kills Fluentd is a trend that never plateaus.

flowchart TD
  A[RSS rising] --> B{Buffer metrics growing too?}
  B -->|Yes| C[Output stall filling memory buffer]
  B -->|No| D{Growth pattern}
  D -->|Steady over days| E[Plugin leak or Ruby fragmentation]
  D -->|Scales with tag count| F[Tag explosion]
  D -->|Scales with chunk count| G[chunk_limit_size too small]
  C --> H[OOM kill - memory chunks lost]
  E --> H
  F --> H
  G --> H
  H --> I[In Kubernetes: CrashLoopBackOff]

Common causes

Cause	What it looks like	First thing to check
Memory buffer growing unbounded	RSS tracks `buffer_total_queued_size`; output retrying or stalled	`buffer_available_buffer_space_ratios` and output `retry_count` in monitor_agent
Ruby heap fragmentation	RSS climbs then plateaus high; plateau creeps upward over weeks	RSS trend over days, not minutes; compare after restarts
Plugin memory leak	RSS grows even with stable throughput and flat buffers	Correlate growth start with a recent plugin or config change
Tiny `chunk_limit_size`	Millions of small chunks; high CPU from GC; memory grows under load	`buffer_queue_length` (chunk count) vs `buffer_total_queued_size` (bytes)
Tag explosion	Chunk count grows without bound; unique tags keep increasing	Count distinct tags hitting the output
Container limit too tight	OOM at a stable RSS that looks reasonable on bare metal	Compare normal RSS plateau against the cgroup limit

Quick checks

These are read-only and safe to run during an incident.

# Confirm the OOM killer fired and which process it chose
dmesg | grep -i oom | tail -20

# In Kubernetes, check for OOMKilled as the last termination reason
kubectl describe pod <fluentd-pod> | grep -A5 "Last State"

# Current RSS of the Fluentd process
ps -o rss= -p $(pgrep -f fluentd | head -1) | awk '{print $1/1024 " MB"}'

# Buffer saturation per output: queue, bytes, and remaining headroom
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, queue: .buffer_queue_length, bytes: .buffer_total_queued_size, avail_pct: .buffer_available_buffer_space_ratios}'

# Is the output actually delivering, or stuck in retry?
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, writes: .write_count, retries: .retry_count}'

# Verify the buffer type actually running on each output
curl -s "http://localhost:24220/api/plugins.json?with_config=true" | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, buffer: .config.buffer}'

The last check matters more than people expect. Some output plugins default to the memory buffer even when the team believes file buffers are in use. Config intent and running config can diverge, especially after a partial reload.

How to diagnose it

Confirm the mechanism. Check dmesg | grep -i oom for the kill, and in Kubernetes look for Reason: OOMKilled in the pod’s last state. This separates an OOM death from a crash loop caused by a config error or a poison pill log line, which look similar from the outside.
Reconstruct the RSS trend. You need the growth shape, not a point sample. Look at per-process RSS history for the 24-72 hours before the kill. Linear growth to the limit means an active consumer (buffer fill or leak). A sawtooth that resets only on restarts points to fragmentation or a leak.
Correlate RSS with buffer metrics. Pull buffer_total_queued_size and buffer_available_buffer_space_ratios history for the same window. If buffer bytes and RSS rose together, the output stalled and the memory buffer absorbed the backlog. This is the buffer-driven pattern.
If buffers were flat, suspect objects, not bytes. Flat buffers plus rising RSS means the memory is in the Ruby heap: fragmentation, a leaking plugin, or object proliferation. Check whether the growth started after a plugin update or config change.
Check chunk economics. Divide buffer_total_queued_size by buffer_queue_length to get average queued chunk size. If chunks are far below chunk_limit_size (8MB default for memory buffers), you are paying Ruby object overhead per chunk. Very small chunks mean large numbers of live objects and heavy GC.
Check for tag explosion. If chunk keys include dynamic values (container IDs, request IDs, user IDs), every distinct value spawns separate chunks that each carry overhead. Count distinct tags over a window and compare against the chunk count.
Quantify the loss. For memory-backed buffers, everything staged or queued at the kill was lost. Use the gap between input emit_records and output emit_records at the destination to estimate how much data disappeared.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Process RSS	The direct input to the OOM killer	Monotonic rise over hours; above 80% of the container limit
`buffer_total_queued_size`	Buffer bytes held in RAM when memory-backed	Rising in lockstep with RSS
`buffer_available_buffer_space_ratios`	Headroom before overflow fires	Below 20% and still falling
`buffer_queue_length` vs `buffer_total_queued_size`	Reveals tiny-chunk object overhead	Large chunk count with small byte total
Output `retry_count` and `write_count`	A stalled output is what fills the buffer	Retries rising, writes flat
CPU per process	GC pressure rises as the heap fills	CPU climbing while throughput is stable
OOM events in kernel log	Confirms the kill and its frequency	Any occurrence; repeats indicate underprovisioning or a leak

Fixes

Switch to file-backed buffers

This is the highest-leverage fix for the buffer-driven pattern. Set @type file in each output’s <buffer> section so backpressure spills to disk instead of the Ruby heap. You trade disk I/O for durability: file buffers survive restarts and OOM kills, and the process no longer dies holding your undelivered data. Verify the running config with ?with_config=true after the change, because a partially applied reload can leave an output on its old buffer type. Watch filesystem free space on the buffer directory afterward; the failure moves from RAM to disk.

Raise chunk_limit_size

If chunks are tiny, increase chunk_limit_size (8MB is the memory buffer default). Bigger chunks mean fewer chunk objects, less per-chunk overhead, and less GC work. The tradeoff is coarser flush granularity: each flush moves more data, and a failed flush retries a larger unit.

Cap tag cardinality

Remove high-cardinality values (container IDs, request IDs) from tag composition and chunk keys. Each unique chunk key multiplies the number of live chunk objects. If you need those values for routing, route on a stable prefix and keep the dynamic value inside the record, not the tag.

Tune the Ruby GC

Adjust RUBY_GC_* environment variables to reduce fragmentation and cap heap growth. For memory-constrained containers, RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR=1.2 (down from Ruby’s default 2.0, defined in the Ruby runtime’s gc/default.c) limits how much the old-generation heap can balloon before GC is forced. Fluentd does not document this value; 1.2 is a community-tuned starting point, so measure against your own workload. GC tuning reduces the plateau height; it does not fix a genuine leak.

Fix or isolate a leaking plugin

If RSS grows with flat buffers and stable throughput, find the plugin that changed when the growth started. Check Fluentd and plugin release notes for known memory-leak fixes; upgrading is often the actual fix. As a stopgap, a supervised restart on a schedule or on memory threshold bounds the damage, but treat that as mitigation, not resolution.

Set the container memory limit correctly

In Kubernetes the cgroup limit is a hard ceiling with no graceful degradation. Set the limit 20-30% above the observed steady-state RSS plateau, not above the bare-minimum RSS you see right after startup. Ruby’s fragmentation plateau is usually higher than teams expect, and a limit sized to the fresh-process value guarantees a kill under sustained load. With no limit set, Fluentd can consume the whole node before the OOM killer intervenes.

Prevention

File buffers for production outputs. Make @type file the standard, with the buffer directory on a filesystem with headroom at least 2x the configured total_limit_size. Reserve memory buffers for cases where losing buffered data is genuinely acceptable.
RSS alerting on trend, not level. Alert on sustained monotonic growth and on RSS above 80% of the container limit. A single high reading is normal for Ruby; a rising line is not.
Buffer headroom alerting. Alert when buffer_available_buffer_space_ratios drops below 20% with positive growth, so the output stall gets fixed while there is still runway.
Tag cardinality review. Treat tag design as a capacity decision. Reject dynamic tag segments in review the same way you would reject an unbounded label in Prometheus.
Right-size the limit after observing the plateau. Measure steady-state RSS over at least a week of normal load, then set the container limit 20-30% above it.
Track restarts, not just liveness. A supervisor that restarts Fluentd in 30 seconds makes repeated OOM kills nearly invisible. Alert on restart count and check dmesg whenever it increments.

How Netdata helps

Per-process RSS at one-second granularity shows the growth shape leading into an OOM kill, including the fragmentation plateau versus a true linear leak, which point-in-time checks cannot distinguish.
Fluentd buffer metrics (buffer_queue_length, buffer_total_queued_size, buffer_available_buffer_space_ratios) collected from the monitor_agent API can be overlaid with RSS on one timeline, which is exactly the correlation that separates buffer-driven growth from heap-driven growth.
Output retry_count and write_count alongside buffer growth reveal the stalled destination that started the cascade, so you fix the cause rather than just the memory symptom.
Process restart events and uptime tracking expose the CrashLoopBackOff pattern that systemd auto-restart otherwise hides.
Per-container memory usage against cgroup limits in Kubernetes shows how much headroom remains before the next OOMKill, turning the 20-30% headroom rule into a live signal.