$ guides / fluentd / fluentd-overflow-action ▌

Operations Guides

Fluentd overflow_action: throw_exception, block, and drop_oldest_chunk

When a Fluentd output buffer reaches total_limit_size, the pipeline does not degrade gradually. It hits a binary transition: the buffer was accepting events, and now it is not. What happens in that moment is decided by a single parameter in the <buffer> section: overflow_action. It accepts three values: throw_exception (the default), block, and drop_oldest_chunk.

Most teams assume “buffer full = pipeline blocks and waits.” That is only true if you explicitly configure block. The default, throw_exception, means new events are rejected when the buffer is full, and depending on the input plugin, those events are simply never ingested. There is no reliable metric that counts these losses.

This article explains what each mode actually does, what data loss looks like under each one, how to detect overflow for each mode, and how to choose deliberately per output instead of inheriting the default everywhere.

What overflow_action controls

Every output plugin’s buffer accumulates events into chunks. Chunks move through a lifecycle: staged (accumulating), queued (waiting to flush), flushing, then purged or retried. When staged plus queued data reaches total_limit_size, the next write to the buffer triggers the overflow path. The overflow_action parameter decides what happens to the events that no longer fit.

The three modes answer one question: when the buffer is full, who loses?

throw_exception: the new events lose. The buffer raises an exception to the input, and the incoming events are not stored.
block: nobody loses inside Fluentd, but the input thread stops, and whatever is upstream of Fluentd absorbs the pressure instead.
drop_oldest_chunk: the oldest buffered data loses. New events always get in; the oldest queued chunk is discarded to make room.

None of these is universally correct. Each one moves the pain somewhere different.

flowchart TD
  A[Buffer reaches total_limit_size] --> B{overflow_action}
  B -->|throw_exception - default| C[BufferOverflowError raised to input plugin]
  B -->|block| D[Input thread sleeps until space frees]
  B -->|drop_oldest_chunk| E[Oldest queued chunk discarded]
  C --> F[in_tail stops reading - new events never buffered]
  D --> G[Backpressure moves upstream - kernel drops UDP syslog]
  E --> H[Old data lost - drop_oldest_chunk_count increments]

throw_exception: the default, and silent data loss

When the buffer is full, the buffer write raises BufferOverflowError back to the input plugin. What happens next depends on the input. For in_tail, the plugin stops reading new lines from the file. The log lines still get written to the file by the application, so they are not gone from disk, but they are not in Fluentd either. If the buffer never drains enough for in_tail to catch up, and log rotation deletes the old file in the meantime, those lines are permanently lost.

For network inputs, the behavior is worse. A forward or HTTP input that cannot buffer an event has nowhere to put it. The event is rejected at ingestion.

The operational trap is that this loss is nearly invisible:

No counter tracks it. The monitor_agent API exposes no metric that reliably counts events rejected by throw_exception. The only visible signs are buffer_available_buffer_space_ratios sitting at or near 0%, and gaps in downstream data.
Logs are the only direct evidence. Fluentd logs BufferOverflowError warnings, so grep on the Fluentd log is your ground truth during an incident.
Input emit_records can mislead. The input’s emit_records counter may stall because events are being rejected, which looks like “the source went quiet” when the source is actually producing normally.
Sustained overflow is expensive. With a persistently full buffer, Fluentd spends significant CPU repeatedly raising and handling exceptions, which slows processing for other outputs too.

The default is defensible for streaming pipelines where backpressure at the source is impossible anyway, but only if you actually monitor buffer fill level. Running the default without watching buffer_available_buffer_space_ratios is how teams lose hours of logs and find out during the postmortem.

block: backpressure, and its blast radius

With overflow_action block, the input thread that tried to write to the full buffer sleeps until space becomes available. No exception, no drops inside Fluentd. The pipeline exerts backpressure toward the source.

This sounds like the safe option, and for some sources it is. in_tail blocking is mostly benign: the file keeps growing on disk and Fluentd catches up when the output recovers. But two failure modes make block dangerous in production:

Upstream loss moves to the kernel. For UDP syslog inputs, a blocked Fluentd stops draining its socket receive buffer. The kernel then drops incoming UDP packets silently. You have traded “Fluentd drops events with a log line” for “the kernel drops packets with no log line at all.” You can detect this only at the host level, via the drops column in /proc/net/udp.

One full buffer can stall unrelated inputs. There are documented cases where a single output with overflow_action block going full stopped ingestion from all input sources, including sources that route to completely different, healthy outputs. With throw_exception, the other sources continued processing. This means block can convert one destination’s outage into a pipeline-wide ingestion stall.

There are also reported deadlock scenarios: when an output plugin re-emits failed chunks back through the router, that re-emit writes into the same full buffer, and the write thread loops forever waiting for space that its own flush is supposed to free. If you use block, test the failure path with your actual output plugin, not just the happy path.

Use block when: the source can tolerate backpressure (file-based inputs), destination outages are short, and you have verified that a blocked output does not starve unrelated inputs in your topology.

drop_oldest_chunk: bounded loss, newest data wins

With drop_oldest_chunk, a full buffer discards the oldest queued chunk to make room for the new write. The pipeline never blocks and never raises. Data loss is continuous but bounded: you always keep the most recent total_limit_size worth of events.

This is the right mode for destinations where freshness matters more than completeness: metrics-flavored logs, monitoring feeds, dashboards where a gap in old data is acceptable but a delay in current data is not.

This mode is also the most observable. The monitor_agent API exposes drop_oldest_chunk_count, a cumulative counter of discarded chunks. Any increment is confirmed data loss, which makes drop_oldest_chunk the only overflow mode with a first-class loss metric.

Two caveats:

A full stage defeats it. Chunk dropping operates on queued chunks. If the buffer is full but everything sits in the staged chunk (not yet queued), there is nothing to drop, and the write fails with BufferOverflowError and a “no queued chunks to be dropped” error. Small chunk keys and long flush intervals make this more likely.
Zero queue length is ambiguous. With drop_oldest_chunk, a buffer queue that stays near zero can mean the output is healthy, or it can mean chunks are being discarded as fast as they arrive. Only the counter tells you which.

Choosing per output

Do not set one global policy. Different outputs in the same Fluentd usually have different loss tolerances. A workable decision frame:

Question	If yes	Suggested mode
Is this source a file that persists on disk?	Yes	`block` or `throw_exception` are both survivable; the data stays on disk
Is the source UDP syslog or another non-replayable stream?	Yes	Never `block` without watching `/proc/net/udp` drops; prefer `drop_oldest_chunk`
Is this a monitoring or metrics destination where freshness beats completeness?	Yes	`drop_oldest_chunk`
Is this an audit or compliance destination where every event must arrive?	Yes	None of these save you; use file buffers, size `total_limit_size` generously, add a `<secondary>`, and alert on buffer fill
Do you have monitoring on `buffer_available_buffer_space_ratios`?	No	Fix that first, before touching overflow_action

A representative configuration:

<match app.**>
  @type forward
  <buffer>
    @type file
    path /var/log/td-agent/buffer/app
    total_limit_size 64GB
    overflow_action drop_oldest_chunk
  </buffer>
  <server>
    host aggregator.internal
  </server>
</match>

Paths above follow the td-agent packaging; fluent-package uses /var/log/fluent/ and /etc/fluent/ instead. Adjust for your install.

Note for v0.12-era configs: the old parameter name buffer_queue_full_action with value exception is still mapped by the compatibility layer to overflow_action throw_exception. If you are migrating configs, rename it explicitly rather than relying on the mapping.

Detecting overflow under each mode

The detection strategy differs per mode because the loss mechanism differs. All commands assume monitor_agent is enabled on port 24220 (add one per worker port in multi-worker mode).

# Buffer fill level: the universal early warning, regardless of mode
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, avail_pct: .buffer_available_buffer_space_ratios}'

# drop_oldest_chunk loss counter (any increment = confirmed data loss)
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, dropped: .drop_oldest_chunk_count}'

# throw_exception evidence: BufferOverflowError in Fluentd's own log
grep -c "BufferOverflowError" /var/log/td-agent/td-agent.log

# block evidence: kernel-level UDP drops for syslog inputs
grep . /proc/net/udp /proc/net/udp6 2>/dev/null | awk '{print $NF}'

Signal	Why it matters	Warning sign
`buffer_available_buffer_space_ratios`	The only mode-independent early warning; overflow fires at 0%	Below 20% and still falling
`drop_oldest_chunk_count`	Direct data-loss counter for `drop_oldest_chunk`	Any increment
`BufferOverflowError` log lines	Only direct evidence of `throw_exception` loss	Any occurrence outside a known destination outage
Input `emit_records` rate	Stalls when inputs are blocked (`block`) or rejecting (`throw_exception`)	Flat while the log source file keeps growing
`retry_count` + `write_count`	Tells you whether the overflow is caused by a failing destination	Retries climbing, writes flat, buffer not draining

Common misuses

Assuming the default blocks. It does not. throw_exception is the default, and under sustained destination failure it drops new events with no counter. Explicitly set overflow_action on every production output.
Using block to “fix” BufferOverflowError. The Fluentd documentation itself recommends against this. Blocking treats the symptom by moving pressure upstream; it does not fix a slow destination. Size the buffer and fix the destination instead, or route overflow through @ERROR or a <secondary>.
Using drop_oldest_chunk without monitoring the counter. The loss is bounded but silent without drop_oldest_chunk_count. On older versions the counter may not exist; alert on buffer fill instead.
Setting the mode but never sizing the buffer. overflow_action only matters at total_limit_size. A file buffer defaults to 64GB total; a memory buffer to 512MB. If your limits are wrong, the best overflow policy just delays the same incident.
Verifying config by reading the file only. Confirm what Fluentd actually loaded with curl -s "http://localhost:24220/api/plugins.json?with_config=true" and inspect each output’s buffer section, especially after a reload.

For the broader picture of how the buffer subsystem fits into the pipeline, see How Fluentd actually works in production. If you are already in an incident with a full buffer, start with Fluentd BufferOverflowError and Fluentd buffer queue length growing.

How Netdata helps

Netdata collects the Fluentd monitor_agent output plugin metrics, including buffer_queue_length, buffer_total_queued_size, and buffer_available_buffer_space_ratios, so you can see a buffer trending toward total_limit_size long before any overflow mode fires.
Correlating drop_oldest_chunk_count against buffer fill level distinguishes “healthy pipeline” from “pipeline silently discarding old chunks,” which is the difference between a green dashboard and a real one for drop_oldest_chunk outputs.
Comparing input emit_records against output emit_records over time surfaces the divergence that throw_exception loss creates, since no dedicated counter exists for it.
Retry and rollback metrics (retry_count, rollback_count, flat write_count) identify the destination failure that is driving the buffer toward overflow, which is the root cause in nearly every overflow incident.
Host-level signals close the gaps Fluentd cannot report: /proc/net/udp drop counters for block-induced kernel loss, process RSS for memory-buffer growth, and disk usage of the buffer directory for file buffers.