$ guides / fluentd ▌

FLUENTD · OPERATIONS PLAYBOOK

Fluentd's buffer is the fulcrum: fill it, and the default is to drop your logs in silence

A Ruby event router where every log flows input to parser to filter to buffer to output, a buffer of chunks absorbs every destination hiccup in between, and when that buffer fills the out-of-the-box behavior is to discard new events without a single counter moving. We trace how the pipeline behaves under backpressure, where 'the process is up' stops meaning 'logs are flowing', and what to do when the buffer, the parser, or the memory wall gives way.

> Start with the monitoring checklist → # Jump to the full guide list

Fluentd's defaults get you collecting logs in an afternoon, then hand you a set of cliff-edges that most teams only discover during an incident — usually the one where they need the logs that were lost.

The defaults work. Until a destination slows down, chunks pile up in a buffer that often defaults to memory, and the worker is OOM-killed with every buffered log gone. Until that buffer hits total_limit_size and — with overflow_action at its throw_exception default — new events are dropped silently, no counter you are watching moving. Until a log line stops matching your parser, Fluentd logs pattern not matched, and quietly discards it. Until a destination flaps, the log fills with failed to flush the buffer, retries back off to thirty minutes out, and the pipeline is effectively dead while still 'retrying'. Until in_tail runs the host out of descriptors with too many open files and stops watching new logs without a word.

These guides are written for engineers who already run Fluentd, not for people learning what a log router is. The goal is the mental model of how events actually move from input to output, why 'the process is up' is not 'logs are flowing', the failure patterns that keep recurring — backpressure, silent drops, OOM, poison pills, rotation loss — the monitoring story that catches them before a postmortem needs the data you dropped, and the runbooks you wish someone had handed you before your last gap in the logs.

How Fluentd actually runs in production

Fluentd is not just a log forwarder. It is a Ruby event router where inputs, a parser, filters, and a tag-matching router feed a buffer of chunks, a pool of flush threads drains that buffer to outputs, and a retry engine sits behind every delivery. Most production failures live in the buffer between ingestion and delivery — or in the input tracking upstream of it — not inside any single plugin.

inputs / sources

Each input plugin runs in its own thread: <code>in_tail</code> follows files, <code>in_forward</code> and <code>in_syslog</code> and <code>in_http</code> take the network. <code>in_tail</code> holds an open file descriptor per watched file and tracks its byte offset in a <code>pos_file</code>. Inputs turn raw data into timestamped, tagged records.

INPUT

parser

Each record is parsed by a format (regex, JSON, LTSV). A line that does not match logs <code>pattern not matched</code> and is dropped; a pathological regex can backtrack catastrophically and hang the process on one bad line.

PARSE

filter chain

An optional ordered pipeline of filters (<code>grep</code>, <code>record_transformer</code>, <code>parser</code>) that transform, enrich, or drop records. Ruby code blocks here run under the GVL and are a common CPU bottleneck.

FILTER

event router

The core dispatch loop matches each event's tag against <code><match></code> directives. Single-threaded per worker. Misrouted tags send events to a null or wrong output — every rate looks healthy while the data never lands where it is needed.

ROUTE

buffer: stage → queue

The most operationally significant component. Events accumulate in chunks that move staged → queued → flushing → purged (or retried). Memory-backed (fast, lost on crash) or file-backed (durable). At <code>total_limit_size</code>, <code>overflow_action</code> decides: drop, block, or throw.

BUFFER

flush threads + output

Each output uses <code>flush_thread_count</code> threads to drain queued chunks to the destination. Network writes release the GVL and parallelize; serialization does not. <code>write_count</code>, <code>emit_records</code>, and <code>flush_time_count</code> live here.

OUTPUT

retry engine + secondary

When a flush fails, chunks retry with exponential backoff. After <code>retry_timeout</code> (default 72h) or <code>retry_max_times</code>, a chunk is discarded or written to a <code><secondary></code>. The backoff interval, not the retry count, tells you whether the pipeline is really recovering.

RETRY

Ruby VM + workers

Everything runs on CRuby under one GVL per worker process. GC pauses block event processing; RSS grows and plateaus. Multi-worker spreads load across cores, each worker with its own buffers and its own <code>monitor_agent</code> port (24220 + worker_id).

RUNTIME

Why this matters: 'Fluentd is dropping logs' can come from a parser that stopped matching, a buffer full under a throw_exception overflow, an OOM-killed worker that lost its memory buffer, in_tail that lost a rotated file, a destination stuck in exponential-backoff retry, or file-descriptor exhaustion. The symptom rhymes but each layer has a different signal — and a different fix.

The failures you'll actually see

Most Fluentd incidents fall into a small set of recurring patterns. Recognise the shape, and triage gets dramatically faster.

CRITICAL

The backpressure cascade

A destination goes slow or unreachable. Flushes fail, retries begin with exponential backoff, and chunks accumulate because new events keep arriving but nothing drains. write_count stops incrementing while the buffer climbs toward its limit. Left alone, it ends at the overflow wall — and what happens there depends entirely on overflow_action.

failed to flush the buffer in the Fluentd log
retry_count incrementing while write_count is flat
buffer_queue_length growing, buffer_stage_length steady
buffer_available_buffer_space_ratios falling toward zero

Investigate →

CRITICAL

Silent data loss

Logs are missing downstream and nothing paged. The buffer filled under overflow_action: throw_exception (the default) and dropped events at the input, or drop_oldest_chunk discarded them, or the parser rejected them. No single reliable counter increments. The only proof is a gap between what went in and what arrived — usually found hours later, in the postmortem.

input emit_records exceeds output emit_records over a window
buffer_available_buffer_space_ratios at or near 0%
emit_error_count or drop_oldest_chunk_count nonzero
Downstream store missing a whole time range or source

Investigate →

CRITICAL

The OOM kill

A memory-backed buffer grows unbounded, or a plugin leaks, or tiny chunks spawn millions of Ruby objects — RSS climbs until the OOM killer terminates the process. Every log still in a memory buffer is gone. In Kubernetes the cgroup limit is a hard ceiling with no warning, and the pod enters CrashLoopBackOff.

Process RSS climbing monotonically, not plateauing
oom-kill / Out of memory entries in dmesg
Process uptime resetting; buffered data missing after restart
CrashLoopBackOff on a Fluentd DaemonSet pod

Investigate →

ACTIVE

File descriptor exhaustion

Tailed files plus buffer chunk files plus output connections cross the process ulimit. The next open() fails with EMFILE: in_tail silently stops watching new files, the buffer cannot create chunks, and outputs cannot connect. The default limit of 1024 is nowhere near enough for a real deployment.

too many open files in the Fluentd log
Open FD count near ulimit -Sn
tracked_file_count dropping unexpectedly
New file watches or output connections failing

Investigate →

ACTIVE

The parse-failure blackout

An application changes its log format and the parser regex no longer matches. Fluentd logs pattern not matched and drops the record — silent, input-level data loss. Input emit_records for that source falls or flatlines while the file itself keeps growing, and a whole source quietly vanishes from downstream.

pattern not matched warnings in the log
input emit_records dropping after a deploy or format change
Log file still growing while its events stop arriving
One source missing downstream, others fine

Investigate →

IMMINENT

The buffer overflow wall

The buffer reaches total_limit_size and Fluentd raises BufferOverflowError — 'buffer space has too many data'. This is the terminal state of the cascade: a cliff-edge where new events can no longer be accepted, and with the default overflow action they are simply lost. Everything upstream stalls behind a buffer that has nowhere left to put anything.

BufferOverflowError / buffer space has too many data
buffer_available_buffer_space_ratios at 0%
buffer_total_queued_size pinned at total_limit_size
input emit_records stalling as events are rejected

Investigate →

Choosing a tool

Best Fluentd Monitoring Tools: 6 Top Picks for 2026

A ranked review of the tools teams actually shortlist here, what each one is genuinely good at, and how the pricing behaves as you scale.

Read the buyer's guide →

Fluentd monitoring maturity levels

Fluentd observability works in four practical levels. Each is a complete operation, not a stepping stone. Pick the level that matches how much your log pipeline matters. Most production pipelines should land at the second level.

Level 1: Survival

Know that something is wrong

Survival monitoring is the floor. With these signals you can answer one question: is Fluentd alive and is it actively losing data? You will not learn where the pipeline is degrading, but you will learn that it broke. Survival is enough for dev collectors and non-critical pipelines.

Process alive (all workers) A dead collector is a blind spot in every downstream system.
emit_error_count Any nonzero rate means events were permanently dropped.
retry_count sustained nonzero The output has been failing repeatedly, not just once.
Process RSS trend Rising toward the memory limit is the OOM precursor.
Fluentd's own error log The log collector's logs are the last place teams look.

Level 2: Operational

Diagnose most incidents on your own

Operational monitoring is what most production pipelines should target. Survival tells you something is wrong; operational tells you what. With this coverage your team can usually diagnose an incident on its own: backpressure, retries, buffer pressure, and the input/output imbalance that quietly loses data.

monitor_agent responsive (24220) A process that is up but hung answers a PID check, not this.
buffer_queue_length + total_queued_size Per output. Growing queue is the backpressure signal.
retry_count per output Which destination is failing, not just that one is.
buffer_available_buffer_space_ratios The runway to overflow, per output plugin.
input vs output emit_records A sustained deficit is data loss, in real numbers.
Process RSS vs the memory limit Alert at 80% in containers; the OOM ceiling is hard.
Open FDs vs ulimit Cliff-edge: at the limit, in_tail and outputs fail.
File-buffer disk usage A full buffer partition takes system logs down with it.

Level 3: Mature

Catch problems before they become incidents

Mature monitoring catches problems before they wake anyone. Flush latency creeping, in_tail falling behind, a retry backed off to thirty minutes, a certificate a week from expiry. None of these page you on day one. They become the incident on day thirty.

in_tail position lag pos_file offset trailing the file size means it can't keep up.
Average flush time flush_time_count / write_count rising precedes retries.
slow_flush_count Individual flushes past slow_flush_log_threshold (20s).
buffer_oldest_timekey Age of the oldest buffered data, not just its volume.
Buffer stage vs queue Distinguishes healthy batching from real backpressure.
drop_oldest_chunk / write_secondary Confirmed loss, and primary-output failover.
Retry backoff state retry.next_time tells you if it is really recovering.
Output TLS certificate expiry An expired cert stops every flush at the same instant.

Level 4: Expert

Reactive instrumentation after real incidents

Expert signals enter your stack the day after a specific incident proved you needed them. Ruby GC pauses, GVL contention, kernel UDP drops, per-worker imbalance, pos_file inode drift. Most teams never need every signal here. Add the ones your incident history says you do.

Ruby GC statistics Major-GC frequency and pause time under memory pressure.
Per-thread CPU (GVL contention) One thread pegged at a core while flush threads wait.
Kernel UDP buffer drops /proc/net/udp drops for syslog/forward before Fluentd sees it.
inotify watch count vs limit Exhausted watches force in_tail into slow polling.
Per-worker metric decomposition One struggling worker hidden by aggregate numbers.
pos_file inode validation Inode in the pos_file vs the current file after rotation.
Output connection-state distribution CLOSE_WAIT buildup means the destination is dropping us.
Config file integrity Tampering can redirect or silence logs; not a metric.

Operating mistakes worth avoiding

The traps Fluentd teams keep falling into. Each has a clear, well-known fix. Most teams only learn it after an incident.

Assuming 'buffer full' means 'blocking'

The default <code>overflow_action</code> is <code>throw_exception</code>, which means a full buffer silently <em>drops</em> new events, not blocks them. Most teams never realise this because no reliable counter tracks the drops. Choose <code>overflow_action</code> deliberately for each output, and watch <code>buffer_available_buffer_space_ratios</code> as an early warning.

Monitoring process-alive and nothing else

A running Fluentd with a full buffer and a stalled output is as useless as a dead one — logs are not flowing. 'Is the process up?' is a survival check, not an operational one. Monitor <code>output emit_records</code> and <code>buffer_queue_length</code>, and compare input to output rates.

Treating the buffer as a single number

Teams watch 'buffer usage' without splitting <code>buffer_stage_length</code> (chunks filling — normal batching) from <code>buffer_queue_length</code> (chunks waiting to flush — backpressure). High stage is fine; high queue is the problem. Collapsed into one number, the distinction that matters disappears.

Never watching emit_error_count

Both playbooks call this the single most under-monitored signal. With <code>overflow_action: drop_oldest_chunk</code>, data loss is completely invisible except through this counter. Teams discover they lost hours of logs during the very incident those logs would have explained.

Running memory-backed buffers in production

Memory buffers are fast and often the default, so teams never switch — until the output has a bad day, the buffer fills, and the OOM killer takes the process and every buffered event with it. File-backed buffers trade some I/O for durability across restarts. In production that trade is almost always worth it.

Leaving the file-descriptor limit at 1024

The default <code>ulimit -n</code> of 1024 is inadequate for any non-trivial deployment. Teams chase mysterious <code>in_tail</code> failures and intermittent output errors for hours before finding FD exhaustion — and the errors are not always explicit, because <code>in_tail</code> often just stops watching files. Set 65536+ in the systemd unit or the pod security context.

Never comparing input and output rates

Teams graph input rate and output rate on separate panels and never subtract one from the other. A sustained 5% deficit is 5% of every log lost — millions of events a week. The input/output balance is the single most telling health metric of the entire pipeline.

Setting up Fluentd and never testing rotation

Teams verify Fluentd works, then never test what happens when <code>logrotate</code> runs. The first rotation may work; the second may not, because it is timing-dependent. <code>copytruncate</code> is inherently racy with <code>in_tail</code>; <code>create</code> rotation with an adequate <code>rotate_wait</code> is safer — but only if it was actually tested.

Fluentd runbooks in this section

Each guide is a focused runbook for one symptom or topic. Pick one when you have an incident, or use the categories to learn the area.

▸

Start here

▸

Process liveness, crashes, and the monitor agent

▸

Buffers, backpressure, and overflow

▸

Silent data loss: overflow, drops, and gaps

▸

Output retries, backoff, and destination failures

▸

Throughput balance and pipeline flow

▸

Flush latency and delivery freshness

▸

Inputs, in_tail, and log rotation

▸

Memory, OOM, file descriptors, and CPU

▸

Multi-worker and deployment topology

▸

Authentication, TLS, and pipeline integrity

WHERE TO GO NEXT

Setting up Fluentd monitoring, or putting out a fire?

If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, jump straight to the symptom that matches what you're seeing.

> Start with the checklist > Back to Operations Guides