Fluentd memory growing: leak versus the normal Ruby fragmentation plateau

Fluentd RSS has been climbing for hours or days and you are trying to decide whether you have a memory leak or whether this is just what Ruby does. The answer matters because the two cases have completely different responses: one requires no action at all, the other ends in an OOM kill and, if your buffers are memory-backed, permanent loss of buffered log data.

Ruby’s garbage collector almost never returns memory to the operating system. After a traffic spike, Fluentd’s heap is mostly free slots internally, but the RSS stays at the peak level. This produces the characteristic pattern: RSS climbs during load, reaches a high level, and then sits there. That plateau is normal. A truly leaking process never plateaus; it keeps climbing at a roughly constant rate until the kernel or the container runtime kills it.

This article gives you the decision procedure: how to read the RSS trend correctly, how to use buffer metrics to split buffer-driven growth from plugin leaks, and what to fix in each case.

What this means

Fluentd memory consumption has three distinct drivers, and they need different responses:

1. Ruby fragmentation plateau. Ruby allocates memory in arenas and rarely gives it back. RSS grows to a level determined by your peak workload and then flattens. High but stable. Not a leak, not fixable by tuning, not dangerous as long as the plateau fits within your memory limit with headroom.
2. Buffer-driven growth. If you use memory-backed buffers and the output cannot keep up, buffered events accumulate in RAM. RSS tracks buffer_total_queued_size almost one-to-one. This is a capacity or destination problem, not a leak. It ends in either buffer overflow or OOM, whichever limit hits first.
3. Genuine leak. A plugin, a C extension, or tag explosion allocates memory that is never reused. RSS climbs monotonically regardless of traffic, buffer metrics stay flat, and the process eventually dies. In Kubernetes this shows up as OOMKilled restarts; see Fluentd CrashLoopBackOff if you are already in the restart cycle.

flowchart TD
  A[RSS climbing over hours-days] --> B{Buffer total queued size also rising?}
  B -- yes --> C[Buffer-driven growth: output cannot keep pace]
  B -- no --> D{RSS trend shape over 24-72h}
  D -- flattens at high level --> E[Ruby fragmentation plateau: normal]
  D -- monotonic rise, no plateau --> F{Correlates with traffic or restarts?}
  F -- no, constant rate --> G[Plugin or runtime leak]
  F -- grows with unique tags --> H[Tag explosion]
  C --> I[Fix output or move buffer to file]
  E --> J[Set memory limit with headroom, no action needed]
  G --> K[Identify plugin, upgrade or remove]
  H --> K

Common causes

Quick checks

All of these are read-only and safe to run during an incident.

# Current RSS of the Fluentd process (MB)
ps -o rss= -p $(pgrep -f fluentd | head -1) | awk '{print $1/1024 " MB"}'

# Detail from /proc: RSS, virtual size, thread count
grep -E "VmRSS|VmSize|Threads" /proc/$(pgrep -f fluentd | head -1)/status

# Sample RSS every 60s for an hour to start building the trend
for i in $(seq 1 60); do
  echo "$(date +%s) $(ps -o rss= -p $(pgrep -f fluentd | head -1))" >> /tmp/fluentd-rss.log
  sleep 60
done

One snapshot tells you nothing. The diagnostic value is entirely in the trend, so start sampling now if you do not have historical RSS data. In containers, read RSS from cgroup memory stats instead; in Kubernetes the working set is what the OOM killer acts on.

# Buffer metrics from monitor_agent: is the buffer growing with RSS?
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, total_bytes: .buffer_total_queued_size, queue_chunks: .buffer_queue_length, avail_pct: .buffer_available_buffer_space_ratios}'

# Evidence of previous OOM kills (may require root)
dmesg | grep -i oom | tail -20

# How long has this process been alive? A recent restart resets RSS.
ps -o pid,etime,rss -p $(pgrep -f fluentd | head -1)

In multi-worker mode each worker is a separate Ruby process with its own RSS. Check every worker, not just the first PID pgrep returns, and remember each worker exposes its own monitor_agent port (24220, 24221, …).

How to diagnose it

1. Establish the trend, not the value. Collect RSS every 1-5 minutes for at least 24 hours, ideally 72. One reading, or even one hour of readings, cannot distinguish a plateau from a leak. Plot it or eyeball the samples from the loop above.
2. Classify the shape. Flattening at a high level: fragmentation plateau. Steady climb with no sign of flattening: leak, unbounded buffer, or tag explosion. Sawtooth with drops only at restarts: growth between restarts is the thing to explain.
3. Correlate with buffer metrics. Pull buffer_total_queued_size over the same window. If buffer bytes and RSS rise together, this is buffer-driven: the output is not keeping pace and events are accumulating in a memory buffer. Treat it as a capacity problem first. See Fluentd buffer queue length growing and Fluentd buffer available space low.
4. If buffer is flat while RSS climbs, suspect a leak. Check what changed: plugin installs, Fluentd upgrades, new outputs, new parsing rules. A constant growth rate independent of traffic volume points at a leak in a plugin or its native extension. Some packaged versions have shipped with known leaks in dependencies, so check release notes for your exact package version before assuming it is your config.
5. Check for tag explosion. If your config builds tags dynamically (for example embedding pod names, request IDs, or timestamps in tags), each unique tag creates routing state that is never freed. Count distinct tags seen over an hour. If the number is in the tens of thousands and still climbing, that is your leak.
6. Factor in process age. Compare RSS against elapsed time (etime). If RSS is always “high” but the process restarts daily for unrelated reasons, you may be looking at the normal ramp, not growth at all.
7. Decide. Plateau inside your limit with headroom: no action, tune alerts. Buffer-driven: fix the output path or move to file buffers. Monotonic leak with flat buffers: isolate the plugin or version and remediate.

Metrics and signals to monitor

If you find the output is the root cause (buffer-driven growth), the failure mode continues in Fluentd failed to flush the buffer.

Fixes

If it is the fragmentation plateau

Do nothing to Fluentd itself. Set your container memory limit or host alerting around the observed plateau plus roughly 20-30% headroom, and alert on the growth trend rather than the absolute value. A high but flat RSS line is the steady state of a healthy Ruby process. Restarting Fluentd to “fix” it just restarts the ramp and, with memory-backed buffers, loses buffered data on every restart.

If it is buffer-driven growth

The memory is a symptom; the disease is an output that cannot keep pace. Two directions:

• Fix the output path. Check retry_count, write_count, and average flush time (flush_time_count / write_count) to find the slow or failing destination. Until the output recovers, the buffer keeps filling.
• Move to file-backed buffers. Switching the buffer @type from memory to file moves queued data from the Ruby heap to disk, converting a memory problem into a much more survivable disk problem. File buffers survive restarts; memory buffers do not. The tradeoff is disk I/O and disk capacity, so watch the buffer directory’s filesystem.

If it is a plugin or runtime leak

• Isolate by elimination. Disable suspect plugins one at a time (or run a canary instance with a reduced config) and watch which change flattens the RSS curve.
• Upgrade. Leaks in Fluentd core and its bundled dependencies get fixed in releases. If your growth started right after a downgrade or is pinned to an old package, moving to the current stable package is often the whole fix. Check the release notes for memory-related fixes for your exact version.
• Increase chunk sizes. If you run small chunk_limit_size values, you create millions of small Ruby objects, which inflates both memory and GC CPU. Larger chunks mean fewer objects and less overhead. The tradeoff is larger, less frequent flushes.
• Ruby GC tuning. Environment variables such as the RUBY_GC_HEAP_* family can reduce the plateau level and GC pressure. These tune the symptom; they do not fix a genuine leak.

If it is tag explosion

Remove high-cardinality values from tag construction. Tags should route events, not identify them; move pod names, request IDs, and timestamps into record fields instead of the tag. This requires a config change and a reload, so verify the new config in a canary first, and be aware that a botched reload can partially apply; see Fluentd config reload failed.

Prevention

• Alert on the RSS trend, not the value. A static “RSS > X” alert either fires on the normal plateau or misses slow leaks. Alert on monotonic growth over hours, and in containers add a hard alert at RSS above 80% of the memory limit.
• Size limits from the plateau. Allocate container memory at 2-3x the expected buffer footprint, or the observed plateau plus headroom, whichever is larger. A 512MB limit on a Ruby log shipper is an OOM schedule, not a limit.
• Default to file-backed buffers in production. This removes the tightest coupling between buffer backlog and process memory, and it makes restarts survivable.
• Keep cardinality out of tags. Review any config that interpolates values into tags during code review.
• Track versions. Keep Fluentd and its plugins current, and read release notes for memory-leak fixes before deploying upgrades.
• Sample RSS continuously. You cannot diagnose a trend you did not record. Per-worker RSS should be a standard time series everywhere Fluentd runs.

How Netdata helps

• Netdata collects per-process RSS at the host level, per second, which is exactly the resolution you need to see whether the curve flattens or keeps climbing. The monitor_agent API does not expose RSS at all, so host-level collection is the only option.
• Long retention lets you overlay 24-72 hours of RSS against buffer metrics and confirm the plateau versus monotonic growth distinction from real data instead of guesswork.
• Netdata’s Fluentd collector pulls the monitor_agent buffer signals (buffer_total_queued_size, buffer_queue_length, available space ratio), so you can correlate RSS with buffer fill on one dashboard and immediately split buffer-driven growth from leaks.
• Per-process CPU alongside RSS makes GC-driven pressure visible: rising CPU at constant throughput while RSS climbs is the classic small-chunk or leak signature.
• In Kubernetes, container memory working set versus the configured limit is charted directly, making the 80%-of-limit alert straightforward to express.

Related guides

• Fluentd broken pipe / connection reset: dropped output connections and LB timeouts
• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd buffer_oldest_timekey lag: how far behind the oldest buffered data is
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss
• Fluentd emit_error_count: the number-one under-monitored data-loss signal
• Fluentd input emit_records stuck at zero: enable_input_metrics on older versions
• Fluentd end-to-end pipeline latency: stale logs during an incident
• Fluentd failed to flush the buffer: the output cannot deliver and retries begin