Fluentd Ruby GC pressure: garbage-collection pauses that stall event processing

Fluentd runs on CRuby. Every parsed record, buffered chunk, and serialized payload creates Ruby objects that the garbage collector eventually has to reclaim. Under memory pressure, GC stops being background work and starts blocking the pipeline: inputs stop reading, flush threads miss their windows, chunks roll back into the queue, and retries allocate even more objects.

The outside symptom is easy to misread. The process is alive, CPU is high, and throughput dips in pulses or sags steadily. Buffer metrics can make it look like a slow destination; CPU metrics can make it look like parser load. The distinguishing pattern is Ruby GC activity rising with memory pressure while destination health remains otherwise explainable.

This failure is most common with memory-backed buffers, undersized chunks that create large numbers of short-lived objects, and long-running processes with fragmented Ruby heaps.

What this means

CRuby uses a generational garbage collector. Minor collections are frequent and relatively cheap. Major collections examine more of the object space and create longer pauses. During a GC pause, no Ruby thread can execute Ruby code. Native work that has released the GVL may continue, but Fluentd’s Ruby-level inputs, routing, and flush logic all wait.

There is no universal safe GC-time percentage. Compare GC rates and pause time with the process’s normal baseline. A sustained increase, especially alongside rising retries and RSS, indicates a feedback loop:

flowchart LR
  A[Memory pressure: many objects or fragmented heap] --> B[Major GC runs more often or takes longer]
  B --> C[Ruby threads wait during GC pauses]
  C --> D[Flush threads miss deadlines]
  D --> E[Chunks roll back and retry]
  E --> F[Retries allocate new objects]
  F --> A
  B --> G[Throughput dips and API responses stall]

Retried chunks are serialized and sent again, allocating fresh objects and increasing GC work. The loop continues until the destination drains the backlog faster than GC stalls it, or RSS reaches a container or system limit and the OOM killer terminates the process. In Kubernetes, that can appear as OOMKilled restarts; see Fluentd CrashLoopBackOff.

A high but stable RSS is normal for Ruby. Fragmentation can keep memory mapped to the process even after objects are collected. A continuously rising RSS, a rising major-GC rate, or a plateau far above the expected buffered-data volume is the concern.

Common causes

Quick checks

These checks are read-only. Checks 2 and 3 require monitor_agent to be enabled; adjust the bind address and port for your configuration.

# 1. Identify the Fluentd worker. Repeat checks per worker in multi-worker setups.
ps -eo pid,ppid,etime,rss,%cpu,args | grep -E '[f]luentd|[t]d-agent'
FLUENTD_PID=<fluentd-worker-pid>

# 2. Confirm process age, RSS, and CPU.
ps -o pid,etime,rss,%cpu,cmd -p "$FLUENTD_PID"

# 3. Check whether monitor_agent is responsive. GC storms can stall it.
time curl --max-time 5 -sS -o /dev/null -w "%{http_code}\n" \
  http://localhost:24220/api/plugins.json

# 4. Inspect output buffers and retry state.
curl --max-time 5 -sS http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") |
      {id: .plugin_id, queue: .buffer_queue_length,
       bytes: .buffer_total_queued_size, retries: .retry_count,
       rollbacks: .rollback_count}'

# 5. Look for one thread dominating CPU. This suggests Ruby-level serialization,
# but does not by itself prove GC.
ps -T -p "$FLUENTD_PID" -o spid,%cpu,comm

# 6. Inspect the GC environment the process inherited. May require root.
tr '\0' '\n' < "/proc/$FLUENTD_PID/environ" | grep -i '^RUBY_GC_'

# 7. Check chunk sizing. Adjust paths for your package.
grep -n -A8 '<buffer' /etc/td-agent/td-agent.conf /etc/fluent/fluentd.conf 2>/dev/null

# 8. Rule out recent OOM kills. May require root.
dmesg -T 2>/dev/null | grep -Ei 'oom|out of memory' | tail -5
journalctl -k --no-pager 2>/dev/null | grep -Ei 'oom|out of memory' | tail -5

A monitor_agent request that takes seconds while CPU is high and buffers remain stable is a strong GC-storm signal. If the stalls affect only one output’s network connections, compare with Fluentd broken pipe / connection reset.

How to diagnose it

1. Establish the timeline. Compare the throughput change with deploys, config reloads, traffic bursts, and Fluentd or Ruby upgrades. Pressure starting immediately after a change points to configuration or version behavior. Pressure building over days points more toward fragmentation or slow object growth.

2. Separate GC stalls from destination stalls. Check retry_count, rollback_count, and slow_flush_count where exposed. If retries rise while the destination is independently slow or unavailable, restore the destination first. In that case GC pressure may be a downstream effect of a full buffer, not the root cause. See Fluentd buffer queue length growing.

3. Measure GC directly. Use the built-in in_gc_stat input and route its records somewhere queryable:

   <source>
     @type gc_stat
     emit_interval 10s
   </source>
   

   Watch the rates of major_gc_count and minor_gc_count, not just their absolute values (each record is the full GC.stat hash, so the available keys depend on the Ruby version; use_symbol_keys defaults to true). Compare them with the normal baseline and lower emit_interval temporarily during an incident; the default 60-second interval can hide short storms.

4. Use gdb only as an intrusive last resort. Attaching gdb stops the process, requires ptrace permission, and can fail on binaries without usable Ruby symbols. Do not run this on a production pipeline unless a full pause is acceptable.

   # INTRUSIVE: stops the process and evaluates Ruby code in it.
   gdb -batch -ex 'call (void)rb_eval_string("$stderr.puts GC.stat.inspect")' \
     -p "$FLUENTD_PID"
   

5. Check chunk geometry. Small chunk_limit_size values create more chunks and more per-chunk objects. Memory buffers commonly default to an 8 MB chunk_limit_size and 512 MB total_limit_size. If chunk size was reduced to lower flush latency, the change also multiplied object lifecycle overhead.

6. Check the Fluentd version and package. Fluentd v1.19.0 includes PR #4845 (“memory_chunk: clear internal memory use in buffer string”), which fixes issue #1657 (excessive retained memory after traffic peaks); older releases retain that memory until a full GC. Confirm that the package you actually run contains the fix before treating an upgrade as the cure.

7. Check the allocator. td-agent v4.2.0 moved its bundled jemalloc from 5.2.1 back to 3.6.0 because jemalloc 4.x and 5.x consumed excessive memory for Ruby workloads. A self-built Fluentd or container image linked against jemalloc 5.x can carry substantially more RSS for the same traffic, increasing the heap that GC must manage.

Metrics and signals to monitor

Fixes

Reduce object churn first

• Increase chunk_limit_size. Larger chunks reduce chunk-object and per-flush allocation overhead. The tradeoff is longer flushes, more data per retry, and a larger failure unit.
• Use file-backed buffers where appropriate. File chunks move payload data out of the Ruby heap and onto disk, trading I/O for a smaller object graph. Plan disk capacity before switching; see Fluentd buffer disk full.
• Reduce allocation-heavy parsing and filtering. Complex regular expressions, per-record Ruby code, and heavy record_transformer use allocate objects for every event. Prefer simpler filters and structured input formats where the source supports them.

Break the feedback loop during an incident

• Restore or bypass the failing destination. Draining the queue stops retries from generating more allocation work. If the destination will remain unavailable, choose overflow_action deliberately. Options such as dropping old chunks and throwing an exception have very different data-loss and backpressure behavior; see Fluentd BufferOverflowError.
• Restart as a reset, not a fix. A restart clears fragmented heap state and retry state. If pressure rebuilds over hours, investigate fragmentation or retention. If it returns immediately, investigate configuration, traffic, or destination health.

Tune Ruby GC env vars, carefully

The main documented Fluentd tuning knob is RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR, which defaults to 2.0. It controls when the old-object count triggers another full GC. Lower values favor a smaller heap with more frequent full collections; higher values favor fewer collections with more memory retained.

Two warnings apply:

• Do not set it below 1.0. Older v0.12-era advice sometimes used 0.9 or 1.2. Current v1.x documentation warns that values below 1.0 degrade performance and can delay startup.
• GC tuning trades memory for pause frequency. Change one variable at a time, record the old value, and compare RSS, GC rates, retries, and throughput across at least one full traffic cycle.

Upgrade or change the substrate

• Upgrade only after verifying the package changelog. If the symptoms match post-burst buffer retention on an older Fluentd release, confirm that the target package contains the relevant fix rather than relying only on the upstream version number.
• Check jemalloc in custom images. A Ruby linked against jemalloc 4.x or 5.x can use substantially more RSS than the td-agent allocator baseline. Every extra retained byte increases the heap GC has to traverse.
• Use multiple workers for CPU headroom, not as a fragmentation fix. Each worker has an independent heap and GVL, so splitting traffic reduces per-process object load. It also multiplies total memory use and does not solve fragmentation that follows each worker over time.

Prevention

• Set explicit buffer limits on every production output. Prefer file-backed buffers where disk capacity allows, and define both chunk_limit_size and total_limit_size. See Fluentd buffer available space low for capacity planning.
• Collect GC baseline data. The official docs classify in_gc_stat as a diagnostic tool and advise against leaving it enabled in production; run it during investigations (or collect GC.stat another way) and retain enough history to establish a normal major- and minor-GC baseline.
• Alert on retry growth with a healthy destination. That combination is often the first external sign of GC-induced flush timeouts.
• Keep RUBY_GC_HEAP_* settings under configuration review. Do not inherit values from v0.12-era templates without testing them against the deployed Ruby and Fluentd versions.
• Leave container memory headroom above the RSS plateau. Ruby memory commonly grows and remains resident. Limits too close to steady state turn fragmentation headroom into an OOM kill.
• Review configs for allocation-heavy patterns. Watch for tiny chunks, very short flush_interval values, complex regular expressions, and per-record Ruby code.

How Netdata helps

• Per-second process metrics: RSS, CPU, and thread activity make GC-driven CPU burn and memory plateaus visible without waiting for long-interval averages.
• Monitor agent correlation: Netdata can collect Fluentd buffer_queue_length, retry_count, rollback_count, flush_time_count, and emit_records alongside host metrics when monitor_agent is exposed.
• API responsiveness context: A live process with a stalling monitor_agent endpoint is easier to identify when endpoint behavior, CPU, and RSS appear together.
• Anomaly detection on throughput: Periodic output-rate dips from major GC pauses can stand out even when static thresholds are not crossed.
• OOM context: Kernel OOM events next to the Fluentd RSS trend show when GC pressure escalated into a kill.

Related guides

• Fluentd broken pipe / connection reset: dropped output connections and LB timeouts
• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd file buffer filling the disk: when the buffer partition runs out
• Fluentd buffer_oldest_timekey lag: how far behind the oldest buffered data is
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss
• Fluentd duplicate events: why the same log shows up twice downstream
• Fluentd emit_error_count: the number-one under-monitored data-loss signal
• Fluentd input emit_records stuck at zero: enable_input_metrics on older versions