Fluentd Operations Guides on Netdata

Fluentd average flush time rising: the earliest sign of destination slowdown

Mon, 01 Jan 0001 00:00:00 +0000

Your Fluentd process is alive. retry_count is zero. The buffer queue looks flat. And yet, if you are computing average flush time from flush_time_count and write_count, you can see the destination getting slower, sometimes hours before the first retry fires. This is the earliest signal of destination or network degradation in a Fluentd pipeline, and most teams never look at it because the monitor_agent API does not expose it as a ready-made field.

Fluentd broken pipe / connection reset: dropped output connections and LB timeouts

Mon, 01 Jan 0001 00:00:00 +0000

Your Fluentd logs show broken pipe or Connection reset by peer during flush, retries start climbing, and chunks roll back into the buffer queue. The destination is not down. It answers health checks, other clients reach it fine, and the errors come and go in a pattern that looks almost random.

The usual explanation: Fluentd’s output plugin is holding a long-lived TCP connection to the destination, something in the middle (a load balancer, NAT gateway, firewall, or the destination itself) has an idle timeout, and it silently drops the connection after a period of inactivity. Fluentd does not find out until the next flush writes into the dead socket. The kernel returns EPIPE (“broken pipe”) or the peer returns RST (“connection reset by peer”), and the chunk goes back for retry.

Fluentd buffer available space low: computing time-to-overflow before it fires

Mon, 01 Jan 0001 00:00:00 +0000

The monitor_agent metric buffer_available_buffer_space_ratios tells you what percentage of an output plugin’s configured buffer capacity is still free. When it drops, the buffer is filling. When it hits zero, the overflow_action fires: with the default throw_exception, new events are rejected at the input with a BufferOverflowError; with block, input threads stall; with drop_oldest_chunk, your oldest undelivered data is thrown away. None of those are outcomes you want to discover after the fact.

Fluentd buffer queue length growing: the output cannot keep pace with the input

Mon, 01 Jan 0001 00:00:00 +0000

You are looking at a graph of buffer_queue_length for one of your Fluentd outputs and it has been climbing for twenty minutes. No alerts have fired, nothing has crashed, but the trend only goes one direction. This is the earliest visible stage of the most common Fluentd failure mode: the output destination is falling behind the input, and the buffer is absorbing the difference.

The buffer is not the problem. The buffer is the symptom. Every queued chunk is data accepted from your inputs but not yet delivered. If the queue keeps growing, you are on a clock: when the buffer reaches total_limit_size, the configured overflow_action fires, and depending on how it is set you will either block your inputs, drop new events at the input, or start discarding the oldest buffered data.

Fluentd buffer stage vs queue: telling healthy batching from backpressure

Mon, 01 Jan 0001 00:00:00 +0000

A common mistake in Fluentd operations is treating “the buffer” as a single number. Teams graph one buffer metric, see it climb, and either panic over normal batching or ignore a real backpressure signal because “the buffer number looks like it always does.” The buffer is not one thing. It is two distinct states with two distinct gauges, and they mean opposite things.

buffer_stage_length counts chunks still being filled with incoming events. That is normal batching. buffer_queue_length counts chunks that are full and waiting to be flushed to the destination. A queue that stays deep is backpressure: the output cannot keep up. Both are gauges, reported separately per output plugin in the monitor_agent API, and the relationship between them tells you more about pipeline health than either does alone.

Fluentd buffer_oldest_timekey lag: how far behind the oldest buffered data is

Mon, 01 Jan 0001 00:00:00 +0000

buffer_oldest_timekey is a timestamp: the timekey of the oldest chunk still sitting in an output plugin’s buffer, waiting to be delivered. Subtract it from the current time and you get a number that answers the question operators care about during an incident: how old is the oldest data that has not reached its destination yet?

That number is more honest than the alternatives. Comparing input and output emit_records rates works for steady streams, but it misleads you for time-sliced outputs (which legitimately hold chunks until the slice expires) and for bursty workloads (where rates oscillate and any short window looks alarming). The oldest timekey does not care about rates. It tells you directly how stale the data is.

Fluentd BufferOverflowError: buffer space has too many data

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd just started logging BufferOverflowError: buffer space has too many data and your destination stopped receiving events. The buffer has reached total_limit_size (512MB by default for memory buffers, 64GB for file buffers) and the default overflow_action, throw_exception, is rejecting every new event at the input. Those events are gone. They do not retry, they do not queue, and no error counter reliably tracks them.

This is a cliff-edge failure, not gradual degradation. One moment the pipeline is absorbing backpressure normally; the next, every incoming event is discarded. buffer_available_buffer_space_ratios sits at or near 0%, input emit rates diverge from output rates, and the only evidence in your logs is the repeated overflow exception.

Fluentd config integrity: leaked credentials and silent tampering

Mon, 01 Jan 0001 00:00:00 +0000

A Fluentd config file is not just routing logic. In most production deployments it is also a credential store: Elasticsearch passwords, S3 access keys, Kafka SASL secrets, TLS client certificates, and forward shared keys all live in the same file that tells Fluentd where your logs go. If that file is world-readable, every local user and every compromised process on the host can read those credentials. If it is writable outside the deploy pipeline, an attacker can redirect your log stream to their own endpoint, quietly drop collection for the services they are touching, or add an output you never approved.

Fluentd config reload failed: SIGHUP that partially applies

Mon, 01 Jan 0001 00:00:00 +0000

You edited the Fluentd config, sent SIGHUP, watched the log line saying the config reloaded, and moved on. Hours later you notice a new output never started receiving data, or an old filter is still dropping events you told it to keep. The process never crashed. No error fired. But the pipeline running in memory is not the pipeline in the config file.

This is the partial reload failure: some plugins reloaded with the new configuration, others are still running the old one, and one or more plugins you expected are absent from the running process. Because Fluentd stays up and keeps processing events through whatever plugins did load, this failure is silent unless you verify the loaded plugin inventory after every reload.

Fluentd CPU bottleneck: the Ruby GVL caps a single worker at one core

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd throughput has flatlined. The host has idle cores, the destination is healthy, retry_count is zero, and yet the buffer is growing and in_tail is falling behind the files it watches. top shows the Fluentd process pinned at 100% CPU, but 100% of exactly one core.

This is the CRuby Global VM Lock (GVL) doing what it is designed to do. Within a single Fluentd worker process, only one Ruby thread can execute Ruby code at a time. The event router, parsers, filters, and flush threads all compete for that lock. The moment one thread does sustained CPU-bound work, such as regex parsing or JSON serialization, everything else in the process waits.

Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

A Fluentd DaemonSet pod in CrashLoopBackOff is not “down.” It is oscillating: the container starts, runs for seconds or minutes, dies, and kubelet restarts it with an exponentially growing backoff. kubectl get pods shows the pod flapping between Running, Error, and CrashLoopBackOff while the restart count climbs.

Because kubelet keeps resurrecting the process, you rarely see one long outage. Instead you see repeated brief absences: buffers drain partially and refill, and downstream destinations see gaps and duplicate windows as file-backed buffers replay on each restart. The auto-restart that keeps the node “mostly covered” is also what masks the root cause.

Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss

Mon, 01 Jan 0001 00:00:00 +0000

drop_oldest_chunk_count is not a warning signal. It is a receipt for data that no longer exists. Every increment means Fluentd discarded the oldest buffered chunk to make room for incoming events, and those events are permanently gone. No retry, no secondary output, no replay will bring them back.

This counter only moves when you have explicitly configured overflow_action drop_oldest_chunk in an output’s <buffer> section. That setting is a deliberate trade: keep the pipeline alive under sustained output failure by sacrificing the oldest data first. The failure mode is that nothing else in the system complains when it fires. Fluentd keeps running, input keeps flowing, output keeps writing. The only evidence of loss is this counter and a warning line in Fluentd’s own log, which most pipelines never scrape.

Fluentd duplicate events: why the same log shows up twice downstream

Mon, 01 Jan 0001 00:00:00 +0000

You query Elasticsearch, S3, or your log backend and the same event appears twice. Sometimes the duplication is a burst that lines up exactly with a Fluentd restart or a pod reschedule. Sometimes it is a slow, persistent trickle that inflates dashboards and breaks counts. The two situations have different root causes and different fixes.

Fluentd does not guarantee exactly-once delivery. It guarantees at-least-once for file-backed buffers, and the brief duplication window after a restart is by design. Persistent or large-scale duplication almost always traces back to one of a small set of mechanisms: position tracking resets at the input, or full-chunk retries at the output after a partial write.

Fluentd emit_error_count: the number-one under-monitored data-loss signal

Mon, 01 Jan 0001 00:00:00 +0000

Most Fluentd monitoring setups watch retry_count, buffer_queue_length, and process liveness. Almost none watch emit_error_count. The usual discovery is postmortem: hours of logs are missing, and the missing window is the exact window the incident needed.

emit_error_count counts emit transactions that failed inside the pipeline: events that could not be handed off and will never be delivered. Any nonzero rate means data loss is happening right now. This is not a degradation signal or a leading indicator. It is confirmation that events are gone.

Fluentd end-to-end pipeline latency: stale logs during an incident

Mon, 01 Jan 0001 00:00:00 +0000

Your dashboards show errors spiking at 02:00, but when you query the log store for the same window, the newest events are 40 minutes old. Fluentd is running, the process is healthy, no alerts fired. The pipeline is alive but the data is stale, and you are debugging an incident on a delay you did not know existed.

Fluentd end-to-end pipeline latency is the time from event generation to arrival at the destination. Fluentd does not expose this as a native metric. There is no pipeline_latency_seconds field in the monitor_agent API. You have to derive it: compare event timestamps to arrival time at the destination, or inject synthetic events with known timestamps and measure round-trip time.

Fluentd failed to flush the buffer: the output cannot deliver and retries begin

Mon, 01 Jan 0001 00:00:00 +0000

Your Fluentd log starts emitting lines like this, over and over:

[warn]: #0 failed to flush the buffer. retry_time=3 next_retry_seconds=2026-07-21 22:50:11 +0000 chunk="5e1a2b..." error_class=Net::OpenTimeout error="execution expired"

An output plugin tried to flush a buffer chunk to its destination and failed. Fluentd did not lose the chunk; it rolled the chunk back into the queue and scheduled a retry with exponential backoff. The warning repeats once per failed attempt, with retry_time climbing and next_retry_seconds drifting further into the future.

Fluentd file buffer filling the disk: when the buffer partition runs out

Mon, 01 Jan 0001 00:00:00 +0000

A file-backed buffer is supposed to be the durable option: the output stalls, chunks pile up on disk, the destination recovers, the backlog drains. That story only holds while the partition underneath the buffer directory has free space. When it does not, the failure is abrupt. At 100% full, Fluentd cannot stage new chunks, incoming events are lost, and anything else writing to the same partition fails at the same time.

Fluentd in_tail not reading: the file is growing but no events are emitted

Mon, 01 Jan 0001 00:00:00 +0000

The application is writing logs. ls -la shows the file growing. But Fluentd’s input emit_records for that in_tail source has been flat for minutes or hours, and nothing is arriving downstream. The daemon runs, the buffers stay empty, and the only evidence is missing data. This is an input-side stall, and it is one of the quieter Fluentd failures because nothing page-worthy breaks.

A restart forces file re-discovery and usually restores flow temporarily, which is exactly why this failure gets misdiagnosed: the restart masks the cause, and the stall returns at the next rotation or the next burst of new files.

Fluentd in_tail with multi-worker: pinning to a single worker

Mon, 01 Jan 0001 00:00:00 +0000

You added workers 4 to <system> to get past the single-core ceiling the Ruby GVL imposes on one Fluentd process, and now Fluentd refuses to start. The log says: Plugin 'tail' does not support multi workers configuration (Fluent::Plugin::TailInput). Or worse: it starts, and you discover weeks later that some log files were read twice by different workers while others were never read at all.

This is a design constraint, not a bug. in_tail is explicitly not multi-worker-ready: it maintains per-file position state, open file descriptors, and rotation tracking that cannot be shared or split across independent worker processes. The fix is to pin every in_tail source to exactly one worker with the <worker N> directive, and to adjust monitoring to match, because only that worker’s monitor_agent port exposes the tail metrics.

Fluentd input emit_records dropped to zero: ingestion has stopped

Mon, 01 Jan 0001 00:00:00 +0000

The emit_records counter for an input plugin is the number of records that plugin has handed to the Fluentd router since process start. It is cumulative and monotonically increasing. If its computed rate drops to zero and holds there while the upstream source is still writing logs, ingestion is broken: events are being generated somewhere and silently going nowhere.

This is one of the few Fluentd conditions that justifies paging immediately. A dead process is obvious. A live process that has stopped ingesting is a silent observability blackout, and it will not announce itself anywhere except this counter.

Fluentd input emit_records stuck at zero: enable_input_metrics on older versions

Mon, 01 Jan 0001 00:00:00 +0000

You open the monitor_agent API (or a dashboard built on top of it) and every input plugin reports emit_records: 0. First read: ingestion has stopped. Then you notice output plugins are delivering records, buffers are cycling, and downstream log storage is receiving fresh data. The pipeline is fine. The metric is blind.

On Fluentd versions before v1.19.0, input plugin metrics are disabled by default. Unless you set enable_input_metrics true in the <system> block, every input plugin’s emit_records counter stays at 0 forever, no matter how much data flows through it. This is a common false alarm, and it also cuts the other way: teams running real ingestion stalls on older versions have no input-side visibility at all because they never turned the metrics on.

Fluentd input spike: a log storm that overwhelms buffers and outputs

Mon, 01 Jan 0001 00:00:00 +0000

Your dashboards show input emit_records suddenly at 5x or 50x baseline. Within minutes, buffer_queue_length starts climbing, average flush time rises, and you are watching the buffer fill in real time. This is a log storm: an input spike large enough that the output side of the pipeline cannot drain it.

The danger is not the spike itself; Fluentd is built to absorb bursts, that is what the buffer is for. The danger is the cascade: the buffer fills to total_limit_size, overflow_action fires, and depending on configuration you either drop new events (throw_exception, the default), stall all input threads (block), or discard your oldest buffered data (drop_oldest_chunk). None of these outcomes is obvious unless you know which counters to watch.

Fluentd log rotation loss: copytruncate, rotate_wait, and missed lines

Mon, 01 Jan 0001 00:00:00 +0000

Log rotation is the most common source of quiet data loss in a Fluentd deployment. Pipeline metrics look healthy, the destination keeps receiving data, and yet there is a gap in the log stream at exactly 00:00 every night, or a burst of duplicate records right after logrotate runs. The cause is almost always the interaction between the rotation method used by logrotate (or the container runtime) and the assumptions Fluentd’s in_tail plugin makes about how files change on disk.

Fluentd memory growing: leak versus the normal Ruby fragmentation plateau

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd RSS has been climbing for hours or days and you are trying to decide whether you have a memory leak or whether this is just what Ruby does. The answer matters because the two cases have completely different responses: one requires no action at all, the other ends in an OOM kill and, if your buffers are memory-backed, permanent loss of buffered log data.

Ruby’s garbage collector almost never returns memory to the operating system. After a traffic spike, Fluentd’s heap is mostly free slots internally, but the RSS stays at the peak level. This produces the characteristic pattern: RSS climbs during load, reaches a high level, and then sits there. That plateau is normal. A truly leaking process never plateaus; it keeps climbing at a roughly constant rate until the kernel or the container runtime kills it.

Fluentd memory vs file buffer: why the default buffer loses data on restart

Mon, 01 Jan 0001 00:00:00 +0000

You restarted Fluentd for a config change, or the OOM killer restarted it for you, or Kubernetes rescheduled the pod. The process came back healthy. Every metric looks normal. But downstream there is a gap in the logs covering the minutes before the restart, and no error anywhere explains it.

The explanation is almost always the same: the output was using the memory buffer, and every chunk that had not been flushed at the moment the process died was deleted with it. This is not a bug. It is the documented behavior of the memory buffer. The Fluentd troubleshooting documentation itself lists “change buffer type from memory to file” as a standard remediation for exactly this symptom.

Fluentd monitor_agent not responding: a process that is up but hung

Mon, 01 Jan 0001 00:00:00 +0000

Your liveness check says Fluentd is fine. The PID exists, systemctl status shows active, but curl http://localhost:24220/api/plugins.json hangs until it times out, or returns something other than 200, and no logs are moving.

This is the zombie state: a process that passes every cheap liveness check but is functionally dead. The monitor_agent endpoint is the cheapest honest probe you have for this. If it does not answer, the internal runtime is not making progress, regardless of what the process table says.

Fluentd monitoring checklist: the signals every production log pipeline needs

Mon, 01 Jan 0001 00:00:00 +0000

Most Fluentd monitoring setups answer one question: is the process running? That is necessary and nowhere near sufficient. A Fluentd process can be alive, responsive, and completely idle because the buffer is full and the default overflow_action (throw_exception) is dropping every new event at the input. No error counter reliably increments for that path. The process looks healthy while the pipeline loses data.

This checklist covers what to actually watch, in two parts: the core signal set every production deployment needs regardless of size, and the maturity levels that tell you what to add as the pipeline becomes more critical. Everything here comes from the monitor_agent API (default port 24220), Fluentd’s own logs, and a small number of OS-level signals.

Fluentd monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Most teams monitoring Fluentd stop at “is the process running?” and discover the gap during an incident, when the SIEM is missing the exact logs they need for a postmortem. Fluentd can be alive, responsive, and green on every dashboard while silently dropping events, accumulating a buffer that will overflow in forty minutes, or retrying into a backoff so deep the pipeline is effectively dead.

This is a maturity model for Fluentd monitoring: four levels, each with the specific signals, collection commands, and failure modes it catches. Use it as a self-assessment. Find the highest level where you have every signal covered, with alerting, in production. Everything above that is your roadmap.

Fluentd OOM killed: memory bloat, the OOM killer, and lost buffers

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd disappeared. The process is gone, systemd or Kubernetes restarted it, and there is a gap in your log storage covering the last few minutes or hours. A check of dmesg shows the OOM killer picked Fluentd as its victim. Memory grew for hours, GC fought harder and harder, and then the kernel ended it.

This failure mode is expensive because of what dies with the process. If your outputs use memory-backed buffers, every staged and queued chunk that had not yet been flushed is gone. The OOM kill is not a graceful shutdown. Ruby cleanup code does not run, so flush_at_shutdown never gets a chance to fire. In Kubernetes the same mechanism shows up as CrashLoopBackOff: the pod restarts, memory climbs back to the cgroup limit, and the kubelet kills it again.

Fluentd output authentication errors: 401/403 and rejected credentials

Mon, 01 Jan 0001 00:00:00 +0000

A 401, 403, unauthorized, forbidden, or TLS certificate error in the Fluentd log means the destination is actively rejecting the output plugin’s connection. The pipeline does not crash. Fluentd keeps accepting input, buffering events, and attempting flushes that keep failing, so from the outside the agent looks alive while no data reaches that destination.

The operational risk is the backlog that builds behind the failure. Every rejected flush pushes chunks back into the buffer, and if credentials stay broken long enough the buffer fills and the configured overflow_action decides what data you lose. With the default throw_exception, new events are silently discarded. With drop_oldest_chunk, the oldest buffered chunks are destroyed. Neither emits an obvious alert unless you are watching the right counters.

Fluentd output rate lower than input rate: the deficit that is quietly dropping logs

Mon, 01 Jan 0001 00:00:00 +0000

Your Fluentd process is up. The monitor agent responds. retry_count is zero. And yet, when you query the destination, whole time windows of logs are missing. The cause is usually the same: output emit_records has been running below input emit_records for hours or days, and nobody was comparing them.

Most teams chart input rate and output rate independently and never compute the ratio. A sustained 5% deficit on a host doing 200 events per second is over 6 million events lost per week. The two rates should converge over any reasonable window. When they do not, data is either being dropped or piling up in a buffer that will eventually overflow and drop it anyway.

Fluentd overflow_action: throw_exception, block, and drop_oldest_chunk

Mon, 01 Jan 0001 00:00:00 +0000

When a Fluentd output buffer reaches total_limit_size, the pipeline does not degrade gradually. It hits a binary transition: the buffer was accepting events, and now it is not. What happens in that moment is decided by a single parameter in the <buffer> section: overflow_action. It accepts three values: throw_exception (the default), block, and drop_oldest_chunk.

Most teams assume “buffer full = pipeline blocks and waits.” That is only true if you explicitly configure block. The default, throw_exception, means new events are rejected when the buffer is full, and depending on the input plugin, those events are simply never ingested. There is no reliable metric that counts these losses.

Fluentd pattern not matched: the parser is silently dropping log lines

Mon, 01 Jan 0001 00:00:00 +0000

You found this in the Fluentd log:

[warn]: #0 pattern not match: "2026-07-21T22:58:11.123456789Z stdout F some application log line"

One warning line per dropped record. Or worse: someone turned the warnings off to silence the noise, and now log lines are vanishing with no signal at all. This is input-level data loss. The events never enter the filter chain, never reach the buffer, and never appear in your destination. Pipeline metrics can look healthy because most counters only count what the parser accepted.

Fluentd per-worker imbalance: one struggling worker hidden by aggregate metrics

Mon, 01 Jan 0001 00:00:00 +0000

Your Fluentd dashboards look fine. Input rate is steady, summed output rate roughly matches, and average buffer usage across the instance is comfortably below the limit. Then someone notices a three-hour gap in the destination for a subset of logs, or the node starts flapping on disk pressure, and the aggregate metrics still insist nothing is wrong.

This is the defining trap of multi-worker Fluentd. When you set workers N in <system>, Fluentd spawns N independent Ruby processes. Each worker has its own event router, its own buffers, its own flush threads, its own retry state, and its own memory footprint. Workers do not share buffer state. If worker 1’s output is stuck in a retry storm while workers 0, 2, and 3 are draining normally, any metric you sum or average across workers will report “mostly healthy” right up until worker 1 starts losing data.

Fluentd plugin load error at startup: LoadError and missing gems

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd fails to start, or starts with part of the pipeline missing, and the log shows LoadError, cannot load such file, or a load_plugin failure. If the failed plugin is an output, that destination silently receives no data while everything else looks healthy. If the plugin is critical to the config, the process refuses to start entirely and you find out via your process-alive check, or via a gap in downstream logs.

Fluentd poison pill crash loop: one bad log line that kills the process on every restart

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd is restarting every few seconds. The service comes up, CPU spikes to 100 percent almost immediately, the process dies, systemd (or Kubernetes) restarts it, and the cycle repeats. Input throughput is effectively zero, but the process is “running” most of the time, so a naive process-alive check flaps between green and red without telling you anything.

This is the poison pill pattern: a single malformed log line that hangs or crashes the parser. Fluentd dies, the supervisor restarts it, in_tail resumes from the last recorded offset in its pos_file, reads the same bad line again, and crashes again. Because the crash happens before Fluentd advances its read position past the offending line, the process is trapped at one byte offset forever.

Fluentd pos_file corruption: duplicates and gaps after a restart

Mon, 01 Jan 0001 00:00:00 +0000

You restarted Fluentd (a deploy, an OOM kill, a pod reschedule) and now one of two things is wrong downstream: the same log lines appear twice, or a window of logs never arrived. Both symptoms point at the same component: the in_tail position file.

The pos_file is how in_tail remembers where it stopped reading. It is a plain text file with one line per tailed file, recording the file path, a byte offset in hexadecimal, and an inode number in hexadecimal. There are no checksums and no integrity metadata. On startup, Fluentd reads it and trusts it completely.

Fluentd process not running: the log pipeline is dead and the host has gone dark

Mon, 01 Jan 0001 00:00:00 +0000

The Fluentd process is gone. No logs are being collected, parsed, buffered, or forwarded from this host. Your downstream systems (Elasticsearch, S3, a SIEM, an aggregator tier) are now receiving nothing from here, and most of them will not tell you that. Log pipelines fail silently at the consumer side: the absence of data looks identical to a quiet host.

This is a full observability blackout for the host and a page-worthy condition in most environments. The one nuance: brief absences are normal. Fluentd restarts during config reloads, rolling updates, package upgrades, and container rescheduling. Gate your page on sustained absence (more than about 2 minutes) so a routine restart at 3 a.m. does not wake anyone up.

Fluentd read_from_head replay: a memory and CPU spike that re-reads whole files

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd has just started (or restarted) and the process is immediately pinned: CPU at or near 100% of one core, RSS climbing fast, and the pipeline emitting a burst of events with old timestamps. Alerts on process CPU, memory growth, or “events older than X arriving at the destination” are probably firing.

In most cases this is read_from_head true doing exactly what you configured. The in_tail plugin is reading each watched file from byte zero, parsing every historical line, and pushing those events through the filter chain, buffer, and outputs. The spike is real resource consumption, but it is one-time per file per position entry, and it stops when the read catches up.

Fluentd retry backoff: a pipeline that is 'retrying' but effectively dead

Mon, 01 Jan 0001 00:00:00 +0000

The Fluentd process is up. The monitor agent responds. retry_count is non-zero, which you already knew, because the destination had a bad night. What retry_count does not tell you is that the next retry attempt is scheduled 4 hours from now. No data is flowing, none will flow for hours, and every dashboard that only tracks the retry counter shows the same flat number it showed an hour ago. The pipeline is technically retrying and operationally dead.

Fluentd retry storm: thundering-herd resonance that keeps a destination down

Mon, 01 Jan 0001 00:00:00 +0000

Your log destination had a hiccup. It recovered. Then it fell over again. And again. Each time it comes back, it survives for a minute or two, gets hit by a wall of buffered log traffic, and falls over. Fluentd’s own metrics look confusing: retries intermittently succeed, the buffer queue drains a little, then grows again. The destination team insists their service “works fine when we test it.”

This is a retry storm with resonance. After a destination failure, many Fluentd output plugins, or many Fluentd pods in a DaemonSet, enter retry at roughly the same time. Their backoff timers are aligned because they all started failing at the same moment. When the backoff expires, they all retry at once. The just-recovering destination, which has no capacity headroom yet, gets hammered by the synchronized burst and fails again. Retries reset, timers realign, and the cycle repeats. The destination is not the problem anymore. The retry behavior is.

Fluentd retry_count climbing: the destination is rejecting or unreachable

Mon, 01 Jan 0001 00:00:00 +0000

You pulled /api/plugins.json from the monitor_agent and one of your output plugins shows a nonzero retry_count, and it keeps going up. At the same time, write_count has stopped incrementing and buffer_queue_length is growing. Fluentd itself is alive and inputs are still collecting.

This is the destination-unavailable failure pattern: the output plugin cannot deliver chunks, the retry engine has taken over, and Fluentd is now in exponential backoff against a destination that is rejecting connections, rejecting data, or simply gone. Every event that arrives from now on accumulates in the buffer. The clock you are racing is buffer capacity, not the retry count itself.

Fluentd rollback_count: chunks recycling back into the queue after failed flushes

Mon, 01 Jan 0001 00:00:00 +0000

rollback_count is one of the output-plugin counters exposed by Fluentd’s monitor_agent. It answers a specific question: how many times has a buffer chunk been taken out of the queue for flushing, failed, and been put back? If buffer_queue_length sits stubbornly non-zero while write_count refuses to move, rollback_count tells you chunks are actively cycling through the failure path rather than sitting idle.

This article explains what the counter measures, the mechanism behind it, how it differs from retry_count (the two are easy to conflate), and how to alert on it without paging on noise.

Fluentd Ruby GC pressure: garbage-collection pauses that stall event processing

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd runs on CRuby. Every parsed record, buffered chunk, and serialized payload creates Ruby objects that the garbage collector eventually has to reclaim. Under memory pressure, GC stops being background work and starts blocking the pipeline: inputs stop reading, flush threads miss their windows, chunks roll back into the queue, and retries allocate even more objects.

The outside symptom is easy to misread. The process is alive, CPU is high, and throughput dips in pulses or sags steadily. Buffer metrics can make it look like a slow destination; CPU metrics can make it look like parser load. The distinguishing pattern is Ruby GC activity rising with memory pressure while destination health remains otherwise explainable.

Fluentd silent data loss: logs missing downstream with no error at all

Mon, 01 Jan 0001 00:00:00 +0000

Someone queries the log store for an incident window and the logs are not there. Not delayed, not misindexed: missing. You check Fluentd. The process is running, retry_count is zero, no error logs, buffer metrics look unremarkable. Everything is green, and the data is gone.

This is the hardest Fluentd failure to detect because it is designed into the defaults. When the buffer fills and overflow_action is throw_exception (the default), new events are rejected at the input and lost. When overflow_action is drop_oldest_chunk, chunks are discarded with only a log warning and one counter that almost nobody alerts on. Parse failures can drop records with no visible trace if the error stream is not routed anywhere. In all three cases the pipeline looks healthy from every angle except the one that matters: the destination.

Fluentd slow flush: buffer flush took longer than slow_flush_log_threshold

Mon, 01 Jan 0001 00:00:00 +0000

You see this in the Fluentd log:

[warn]: buffer flush took longer time than slow_flush_log_threshold: elapsed_time=... slow_flush_log_threshold=20.0 plugin_id="..."

This is a performance warning, not a data-loss signal. Fluentd delivered (or is still retrying) a buffer chunk to the destination, and the write took longer than slow_flush_log_threshold, which defaults to 20 seconds. Each occurrence also increments the slow_flush_count counter on that output plugin.

Slow flushes are the earliest indicator of destination degradation. A flush that takes 30 seconds but succeeds is a destination that is one bad day away from rejecting writes entirely. While flushes are slow, effective output throughput drops, chunks accumulate in the queue, and the buffer marches toward total_limit_size. What happens at the limit depends on your overflow_action, and the default (throw_exception) loses data. So the right response to this warning is not “make the log line go away” but “find out why writes to the destination got slow, before slow becomes failed.”

Fluentd throttled_log_count: in_tail rate limiting is dropping lines at the source

Mon, 01 Jan 0001 00:00:00 +0000

throttled_log_count is a cumulative counter exposed by the in_tail input plugin. Every increment means Fluentd deferred log lines at the source because a configured rate limit was exceeded. Unlike buffer-side loss, this happens before the event enters the pipeline: no filter sees it, no buffer holds it, and no output will ever deliver it.

The counter only moves when you have configured throttling in in_tail, either via the <group> section with rate rules or via byte-rate limiting on reads. If you never configured throttling, this counter stays at zero forever, and a zero value tells you nothing. If you did configure it, any non-zero increment rate is a decision point: either the throttle is doing what you designed it to do, or it is silently eating log volume you expected to keep.

Fluentd TLS certificate expiry: a cliff-edge that stops every output at once

Mon, 01 Jan 0001 00:00:00 +0000

Every TLS output on a Fluentd node was working. Then, at one exact second, all of them started failing with the same handshake error. retry_count is climbing on every output that uses TLS, write_count has flatlined, and the buffer is filling. Nothing was deployed. Nothing changed on the network. The only thing that changed is the wall clock: the certificate crossed its notAfter timestamp.

Certificate expiry is one of the few failures that is both perfectly predictable and a total outage. There is no gradual degradation, no partial failure, no per-connection luck. TLS connections to a destination succeed right up to the expiry instant and then every new handshake fails identically, for every output plugin, on every Fluentd node presenting or validating that certificate.

Fluentd too many open files: file descriptor exhaustion stalls the pipeline

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd logs Errno::EMFILE: Too many open files and the pipeline stops making progress. Log files keep growing on disk, the destination sees nothing new, and the Fluentd process is still alive. It is not crashed; it is wedged against its file descriptor limit, and almost every operation it needs to do next requires opening something.

What makes this failure nasty is the cliff edge. Everything works until the limit is reached, and then several things fail at once: in_tail cannot open newly created log files (and may stop watching them without a loud error), the buffer cannot create new chunk files, and outputs cannot establish new connections. A single resource limit takes out input, buffer, and output simultaneously.

Fluentd unauthorized in_forward connections: log injection on the aggregator tier

Mon, 01 Jan 0001 00:00:00 +0000

You ran ss -tn on your Fluentd aggregator and saw connections to port 24224 from IP addresses you do not recognize. Or a security review turned up the fact that your aggregator-tier Fluentd accepts forwarded events from anything that can reach it. Either way: your in_forward input is unauthenticated, and any network-reachable host can inject events directly into your log pipeline.

An in_forward source without a <security> section performs zero authentication. Any host that can open a TCP connection to port 24224 can submit events with arbitrary tags and arbitrary record contents. The Fluentd project stated this plainly when authentication was introduced in v0.14.5: anyone who can connect to the TCP port of in_forward can inject events into the Fluentd process.

Fluentd worker died: a partial outage the supervisor hides

Mon, 01 Jan 0001 00:00:00 +0000

systemctl status td-agent says active (running). The supervisor PID exists. Host-level checks are green. Meanwhile, one of your four Fluentd workers died twenty minutes ago, and a quarter of the log pipeline is gone or degraded.

This is the trap of Fluentd multi-worker mode. When you set workers N in <system>, Fluentd starts a supervisor process and N independent Ruby workers. Each worker has its own event loop, buffers, output threads, and, if configured, monitor_agent endpoint. Workers do not share in-memory state.

Fluentd write_secondary_count: the primary output has failed to its backup

Mon, 01 Jan 0001 00:00:00 +0000

write_secondary_count is a per-output counter in Fluentd’s monitor_agent API. A nonzero value means the primary output plugin exhausted its retries for at least one chunk, and Fluentd wrote that chunk to the configured <secondary> backup destination instead. The primary pipeline for that output is broken.

This is a ticket-level signal. Data is not lost yet (that is the point of the secondary), but it is no longer flowing where downstream systems expect it. Dashboards, SIEM rules, and alerts that read from the primary destination are now working from a gap.

How Fluentd actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

Most Fluentd incidents are not random. They are the predictable output of a small set of internal mechanisms: a single-threaded event router, a chunked buffer with a hard limit, Ruby threads fighting over one GVL, and a retry engine with exponential backoff. If you understand those four, you can predict almost every Fluentd failure mode before you open a dashboard.

This article is the mental model, not a runbook. It explains what Fluentd is doing between the log file and the destination, so that when a runbook says “check buffer_queue_length” or “watch retry.next_time,” you know which piece of machinery those numbers describe.