How Fluentd actually works in production: a mental model for operators

Most Fluentd incidents are not random. They are the predictable output of a small set of internal mechanisms: a single-threaded event router, a chunked buffer with a hard limit, Ruby threads fighting over one GVL, and a retry engine with exponential backoff. If you understand those four, you can predict almost every Fluentd failure mode before you open a dashboard.

This article is the mental model, not a runbook. It explains what Fluentd is doing between the log file and the destination, so that when a runbook says “check buffer_queue_length” or “watch retry.next_time,” you know which piece of machinery those numbers describe.

Everything here applies to Fluentd v1.x on CRuby (MRI), which is what td-agent and fluent-package ship. Fluent Bit is a separate C-based project with a different architecture; none of this applies to it.

What Fluentd is and why the internals matter

Fluentd is an event router: it ingests log events, optionally transforms them, and delivers them to destinations. Operationally it sits in the middle of your pipeline, and when it degrades it degrades silently. The process stays alive, metrics look plausible, and data quietly stops arriving, arrives duplicated, or arrives six hours late.

Each of those outcomes maps to a specific internal component. Operators who treat Fluentd as a black box guess. Operators who know the event path go straight from symptom to component.

The event path

Every event flows through the same path:

Input -> Parser -> Filter chain -> Event Router -> Buffer -> Flush threads -> Output

flowchart LR
  IN[Input plugin
own thread] --> P[Parser
tag + record]
  P --> F[Filter chain
transform / enrich / drop]
  F --> R[Event router
single thread per worker]
  R -->|tag matches match block| B[Buffer
staged to queued]
  B --> FT[Flush threads
GVL-bound Ruby threads]
  FT --> O[Output plugin]
  O -->|success| PURGE[Chunk purged]
  O -->|failure| RETRY[Retry engine
exponential backoff]
  RETRY -->|exhausted| SEC[Secondary
or discard]

Inputs: one thread per plugin

Each input plugin (tail, forward, TCP/UDP sockets, HTTP, syslog) runs in its own thread. Inputs parse raw data into timestamped records and attach a routing tag, for example apache.access. The tag is the only thing the router looks at later, so tagging decisions at the input determine where data can go downstream.

For file ingestion, in_tail maintains a position file (.pos) tracking the byte offset and inode of each watched file, and holds an open file descriptor per file. This matters later: position file corruption and file descriptor exhaustion are input-side failures that surface as mysteriously missing data.

Filters: ordered, optional, and CPU-hungry

Filters (grep, record_transformer, parser) run in configuration order; each can transform, enrich, or drop records. Two things operators underestimate: filters run on the same GVL-bound Ruby runtime as everything else, so an expensive filter throttles the entire pipeline, and a filter that silently drops records produces no error anywhere. The pipeline looks healthy while data vanishes.

The event router: single-threaded per worker

The router matches each event’s tag against <match> directives and hands the event to the matching output. This loop is single-threaded per worker process, so routing complexity directly costs throughput: dozens of wildcard <match> blocks and heavy tag manipulation burn CPU on every event.

This is also where misrouted events happen. If a tag matches no <match> block, events flow to a null or unintended destination with no error raised. Input and output rates look healthy while data goes to the wrong index or nowhere at all.

The buffer: where most incidents are born

The buffer sits between the router and the outputs, and it is the most operationally significant component in Fluentd. Most production failures are buffer failures or buffer-adjacent failures.

Chunks and their lifecycle

Buffers do not hold individual events; they accumulate them into chunks. Each chunk moves through a lifecycle:

1. Staged: the chunk is open and accumulating events.
2. Queued: the chunk is full (or its time slice expired) and is waiting for a flush thread.
3. Flushing: a flush thread has dequeued it and is writing it to the output.
4. Purged: delivery succeeded, the chunk is deleted. On failure, the chunk rolls back to the queue and the retry engine takes over.

The staged-versus-queued distinction is critical and frequently missed. A high staged count is normal: batching is working. A high queued count is backpressure: chunks are ready but the output cannot drain them. The monitor_agent reports these separately as buffer_stage_length and buffer_queue_length; treating “buffer usage” as one number hides the difference between health and trouble.

Memory versus file buffers

• Memory-backed: fast, volatile. Contents live in the Ruby heap and contribute directly to process RSS. All unflushed data is lost on crash or restart. Defaults: chunk_limit_size 8MB, total_limit_size 512MB.
• File-backed: slower, persistent across restarts. Chunks are append-only binary files on disk. Defaults: chunk_limit_size 256MB, total_limit_size 64GB.

Take the file buffer in production. With file buffers, a restart replays unflushed chunks, causing a brief burst of duplicates (Fluentd does not guarantee exactly-once delivery) but no loss. With memory buffers, a restart or OOM kill silently destroys everything in flight.

Two version-specific behaviors for file buffers: since v1.16.0, corrupted chunk files found at startup are reportedly moved to a backup directory instead of deleted , and since v1.19.0, chunks that exceed their retry limit are reportedly evacuated to ${root_dir}/buffer/${plugin_id}/ instead of discarded . Both changes convert silent loss into recoverable artifacts.

Overflow: the decision most teams never make

When buffered data reaches total_limit_size, overflow_action decides what happens next:

The default is the dangerous one. Teams assume “buffer full means blocking” and get “buffer full means dropping” instead. Choose this deliberately for every output, and know each output’s behavior before an incident forces the question.

Flush threads and the Ruby GVL

Each output plugin flushes chunks using flush_thread_count threads (default 1). These are Ruby threads, and on CRuby the GVL allows only one thread per process to execute Ruby code at a time.

The practical consequence: I/O-bound work (network writes to Elasticsearch, S3, Kafka) releases the GVL and genuinely parallelizes, but CPU-bound work (regex parsing, JSON serialization, compression) does not. A single Fluentd worker at “100% CPU” is saturating exactly one core, no matter how many flush threads you configure. Worse, CPU-intensive parsing starves the flush threads, so events accumulate in buffers even when the destination is healthy. It looks like an output failure and is actually an input-side CPU bottleneck.

The fixes follow from the mechanism: raise flush_thread_count only for I/O-bound slowness, offload compression to an external process (gzip_command) so it escapes the GVL, simplify regex parsers, and use multiple workers for real CPU parallelism.

Multi-worker mode

Setting workers N in <system> launches N independent Ruby processes, each with its own GVL, event loop, buffers, and output threads. This is the only way to use more than one core.

Operational implications:

• No shared state. Workers do not share buffers or queues. One worker can drown in backpressure while the others idle, and aggregate metrics mask it.
• Per-worker monitoring. The monitor_agent port auto-increments per worker: worker 0 on 24220, worker 1 on 24221, and so on. You must query each port to see the whole system.
• in_tail is single-worker. in_tail does not support multi-worker and must be pinned to a specific worker with <worker N>. That worker becomes your file-ingestion bottleneck by design.

The retry engine and the secondary output

When a flush fails, the chunk rolls back and the retry engine schedules another attempt with exponential backoff (retry_wait starting at 1s, randomized by default via retry_randomize). Retries continue until retry_max_times or retry_timeout (default 72 hours) is hit, after which the chunk goes to the configured <secondary> output or is discarded (or evacuated, per the v1.19.0 note above).

Two things operators get wrong:

• retry_count is not the retry state. The counter tells you failures happened. The retry object in the monitor_agent response (retry.steps, retry.next_time) tells you how bad it is. With exponential backoff, retry.next_time can be 30 minutes out: the pipeline looks like it is “retrying” but is effectively stalled.
• The secondary threshold. retry_secondary_threshold defaults to 0.8, meaning chunks fall through to the secondary after 80% of retry_timeout has elapsed, not after the full timeout. write_secondary_count incrementing means the primary pipeline has failed exhaustively for that chunk.

The monitor_agent: your only real window in

Almost everything above is observable through one interface: the monitor_agent input plugin, which exposes per-plugin metrics as JSON on port 24220 (auto-incrementing per worker). It is not enabled by default in all distributions. Without it, you are limited to process liveness checks and log grepping.

# Enable in config, then query plugin state
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | {id: .plugin_id, cat: .plugin_category, queue: .buffer_queue_length, retries: .retry_count}'

Key fields: emit_records (cumulative, derive the rate), write_count (successful chunk flushes), retry_count and rollback_count, slow_flush_count , and the buffer gauges buffer_stage_length, buffer_queue_length, buffer_total_queued_size, and buffer_available_buffer_space_ratios.

Version notes: on older Fluentd versions, input plugin emit_records may require enable_input_metrics true in <system> or the counter stays at zero . In recent v1.19.x releases, include_config, include_retry, and include_debug_info reportedly default to false, so older queries that relied on retry state or config being visible need those parameters enabled explicitly . That change was security hardening, because include_config can expose credentials from plugin configuration.

Where this shows up in production

The characteristic failure archetypes all map back to the machinery:

Signals worth watching continuously:

How Netdata helps

The mental model is only useful if you can see the machinery moving. Netdata helps on this specific axis:

• It collects Fluentd’s host-level signals (per-process RSS, CPU, file descriptors) continuously, which is where memory-bloat and FD-exhaustion failures become visible long before the OOM kill or the “too many open files” error.
• Per-second process metrics expose GVL starvation directly: one core pinned at 100% while the rest of the host idles is the signature.
• Disk metrics on the buffer directory’s mount point catch file-buffer growth against real filesystem capacity, which is the actual limit when total_limit_size is set optimistically.
• Correlating process restarts with memory and CPU history turns a “mysterious crash loop” into a readable poison-pill or OOM narrative.
• Alerting on trends rather than absolutes (RSS growth rate, queue growth rate) matches how Fluentd actually fails: gradually, then on a cliff edge.

Related guides

• Fluentd monitoring checklist: the signals every production log pipeline needs
• Fluentd monitoring maturity model: from survival to expert
• Fluentd process not running: the log pipeline is dead and the host has gone dark
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• Fluentd poison pill crash loop: one bad log line that kills the process on every restart
• Fluentd monitor_agent not responding: a process that is up but hung
• Fluentd plugin load error at startup: LoadError and missing gems
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer stage vs queue: telling healthy batching from backpressure
• Fluentd buffer available space low: computing time-to-overflow before it fires