Fluentd CPU bottleneck: the Ruby GVL caps a single worker at one core

Fluentd throughput has flatlined. The host has idle cores, the destination is healthy, retry_count is zero, and yet the buffer is growing and in_tail is falling behind the files it watches. top shows the Fluentd process pinned at 100% CPU, but 100% of exactly one core.

This is the CRuby Global VM Lock (GVL) doing what it is designed to do. Within a single Fluentd worker process, only one Ruby thread can execute Ruby code at a time. The event router, parsers, filters, and flush threads all compete for that lock. The moment one thread does sustained CPU-bound work, such as regex parsing or JSON serialization, everything else in the process waits.

The failure is easy to misdiagnose as an output problem, because the visible symptom is backpressure: the buffer grows and delivery slows. But the destination is fine. The bottleneck is inside the process, and no amount of destination tuning will fix it.

What this means

Fluentd’s internal path is Input, Parser, Filter chain, Buffer, Output. In a single-worker deployment, all of these stages run as Ruby threads inside one process, and all Ruby threads share one GVL. I/O-bound work (network writes to Elasticsearch, S3, Kafka) releases the lock, so flush threads waiting on sockets do not block each other. CPU-bound work does not release it. A thread running a complex regex over every incoming line holds the GVL while the flush threads sit idle.

The practical ceiling: a single-worker Fluentd process can consume at most one core, regardless of host size. Once parsing and filtering alone approach that core’s capacity, throughput plateaus. Inputs cannot read fast enough, outputs cannot flush fast enough, and the pipeline backs up even though every external system is healthy.

flowchart TD
  A[Buffer growing, throughput flat] --> B{retry_count or rollback_count rising?}
  B -->|yes| C[Output/destination failure - see related guides]
  B -->|no, both zero| D{Destination healthy? flush time normal?}
  D -->|no| C
  D -->|yes| E{Fluentd process CPU at ~100% of one core?}
  E -->|no| F[Check input side: source stopped, tail lost file]
  E -->|yes| G[Per-thread CPU: one thread dominant?]
  G -->|yes| H[GVL contention: CPU-bound parsing or serialization]
  H --> I[Fix: simplify parsers or add workers]

Common causes

Quick checks

All read-only. Adjust package paths for your install (td-agent vs fluent-package).

# 1. Confirm the process is pinned near one core
top -b -n 3 -d 2 -p $(pgrep -f fluentd | head -1)

# 2. Per-thread CPU: the decisive GVL check
ps -T -p $(pgrep -f fluentd | head -1) -o spid,%cpu,comm

# 3. Confirm this is NOT an output failure
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, retries: .retry_count, rollbacks: .rollback_count, writes: .write_count}'

# 4. Check queue vs stage: GVL starvation keeps the queue small
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, stage: .buffer_stage_length, queue: .buffer_queue_length}'

# 5. Input side: is Fluentd consuming what sources produce?
curl -s http://localhost:24220/api/plugins.json | \
  jq '[.plugins[] | select(.plugin_category=="input") | .emit_records // 0] | add'

# 6. in_tail falling behind: compare recorded positions to file sizes
cat /var/log/td-agent/td-agent.pos   # format: filepath, inode, position
ls -la /var/log/containers/*.log 2>/dev/null | head

# 7. How many workers are actually running?
pgrep -af fluentd

# 8. Rule out a known version bug
fluentd --version

Reading the results:

• Check 2 is the confirmation step. Under GVL contention, one thread (typically the input/parser thread) sits near 100% while flush threads show near-zero CPU. Under an output failure, flush threads are active but stuck on I/O, and retry_count climbs.
• Check 4 distinguishes starvation from backpressure. GVL starvation: stage and queue stay modest because data never reaches the buffer fast enough. Output failure: queue grows toward its limit.
• Check 6: if positions trail file sizes by growing margins while the destination is idle, Fluentd cannot read fast enough. That is CPU starvation at the input stage.

How to diagnose it

1. Establish the symptom shape. Throughput plateaued below expected volume, buffer growing slowly or input lagging, destination healthy. If retry_count is non-zero, stop here and treat it as an output failure instead.

2. Measure process CPU against core count. If the Fluentd process sits at roughly 100% and the host has N cores, the process is using 1/N of available CPU. top on a 16-core host may show total system CPU at 8% while the pipeline is completely starved. This is the single most misleading aspect of this failure.

3. Profile per-thread. ps -T on the Fluentd PID. One dominant thread confirms GVL contention. Note which thread: if it is an input thread, the cost is parsing; if it is a flush thread, the cost is serialization (JSON generation, compression) on the output side.

4. Correlate with buffer signals. Small queue, flat write_count increments, normal flush times, zero retries: the output stage is healthy and starved of work. Growing in_tail position lag confirms the input stage cannot keep up either.

5. Identify the expensive stage. Review parser configs for complex format /.../ regexes, multi-line regexes, and filters doing Ruby-level record mutation. Regex parsers are the usual suspect; JSON or LTSV formats parse at a fraction of the cost.

6. Rule out version bugs before tuning. On Fluentd v1.12.x, a Ruby resolv.rb leak pins one thread at 100% CPU in DNS resolution after enough lookups, with flush threads stuck in resolv.rb. The fix is an upgrade (v1.13.0+), not parser work. Similarly, v1.8.0 through v1.14.4 carried a deprecated-warning loop that raised CPU roughly 50% over v1.7.4. Check the changelog before blaming your config.

Metrics and signals to monitor

Fixes

Simplify the parsers

This is the cheapest fix and often sufficient. Complex regex is the dominant CPU cost in most pipelines.

• Prefer structured log formats at the source: JSON or LTSV instead of regex-parsed plaintext. The parse cost difference is large and grows with line complexity.
• Remove or rewrite regexes with nested quantifiers or alternation over long strings; these are backtracking hazards as well as throughput sinks.
• Audit filters. record_transformer blocks doing Ruby-level string work per record run on the same single core.
• If you offload compression to the destination or disable it, you trade bandwidth for CPU. On a GVL-bound pipeline, that trade is often correct.

Tradeoff: changing log formats requires upstream application changes, which may not be on your timeline. Parser rewrites need testing against real traffic, including malformed lines.

Do not expect flush_thread_count to help

Raising flush_thread_count (default 1) parallelizes I/O-bound flushes but does nothing for CPU-bound work. The Fluentd documentation states this explicitly: it does not improve processing performance. More threads fighting for the same GVL can make contention worse, not better. This knob is for slow destinations, not for CPU saturation.

Enable multi-worker mode

This is the structural fix. In <system>, set workers N where N matches your available cores. Each worker is an independent OS process with its own GVL, its own buffers, and its own event loop, so N workers can use N cores. Multi-worker has been built in since Fluentd v0.14.x and replaces the old fluent-plugin-multiprocess gem. The official guidance is that single-process tuning suffices up to roughly 5,000 messages per second; beyond that, go multi-worker.

Operational caveats:

• in_tail does not support multi-worker. Pin it with a <worker N> directive, or you get Plugin 'tail' does not support multi workers configuration. The pinned worker becomes your ingestion bottleneck; size the other workers for output.
• Buffer path collisions. Output plugins writing to file or S3 paths must include ${worker_id} in the path, or workers overwrite each other and lose data.
• Per-worker monitoring. The monitor_agent port auto-increments (worker 0 = 24220, worker 1 = 24221, and so on). Scrape every worker; aggregates from one port hide imbalance.
• Memory multiplies. Each worker carries its own Ruby heap and buffers. Set container memory limits accordingly.

Tradeoff: multi-worker adds operational complexity (per-worker metrics, plugin compatibility checks, buffer path hygiene). For modest volumes, parser simplification is the better first move.

Prevention

• Track per-core CPU, not host CPU. Alert when the Fluentd process sustains more than about 60 to 70% of one core in single-worker mode. That is your runway indicator; throughput cliffs follow as traffic grows.
• Watch input consumption, not just output health. A flat input emit_records rate against growing source volume is the earliest starvation signal. On Fluentd before v1.19.0, this requires enable_input_metrics true in <system>.
• Benchmark with production configs. Complex regex parsers routinely cut achievable throughput by an order of magnitude versus simple configs. Test with real parsers, real filters, and realistic line distributions, including the malformed ones.
• Prefer structured logging upstream. Every service that emits JSON instead of regex-parsed text buys CPU headroom on every Fluentd node that touches it.
• Plan the multi-worker migration before you need it. Verify plugin compatibility (in_tail pinning, ${worker_id} paths) in staging, and wire per-worker scrape targets into monitoring first.

How Netdata helps

• Per-process and per-thread CPU charts for the Fluentd process make the one-core ceiling visible directly, instead of being masked by low host-wide CPU averages.
• Fluentd’s monitor_agent metrics (buffer queue length, stage vs queue split, retry_count, write_count) are collected per plugin, so you can see a small queue with zero retries alongside rising input lag, the exact GVL-starvation signature.
• Correlating process CPU against input and output emit_records rates on one dashboard separates “CPU-bound and starved” from “output failing and backing up” in seconds.
• Per-worker views let you spot imbalance in multi-worker mode, where one worker pinned with in_tail saturates while the others idle.
• Anomaly detection on input throughput catches the slow plateau of GVL saturation as traffic grows, before the buffer starts paying for it.

Related guides

• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd buffer_oldest_timekey lag: how far behind the oldest buffered data is
• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd emit_error_count: the number-one under-monitored data-loss signal
• Fluentd input emit_records stuck at zero: enable_input_metrics on older versions
• Fluentd end-to-end pipeline latency: stale logs during an incident
• Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss
• Fluentd broken pipe / connection reset: dropped output connections and LB timeouts
• Fluentd duplicate events: why the same log shows up twice downstream
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes