Fluentd just started logging BufferOverflowError: buffer space has too many data and your destination stopped receiving events. The buffer has reached total_limit_size (512MB by default for memory buffers, 64GB for file buffers) and the default overflow_action, throw_exception, is rejecting every new event at the input. Those events are gone. They do not retry, they do not queue, and no error counter reliably tracks them.

This is a cliff-edge failure, not gradual degradation. One moment the pipeline is absorbing backpressure normally; the next, every incoming event is discarded. buffer_available_buffer_space_ratios sits at or near 0%, input emit rates diverge from output rates, and the only evidence in your logs is the repeated overflow exception.

The error itself is a symptom. The root cause is almost always downstream: the destination is slow, down, or rejecting data, and the buffer filled while you were not watching. Restarting Fluentd clears retry state and buys you time, but if the destination problem persists, the buffer refills and the error returns.

What this means

Fluentd buffers sit between the filter chain and the output plugins. Events accumulate in chunks, chunks move from staged to queued, and flush threads deliver queued chunks to the destination. When delivery fails or falls behind, queued chunks pile up against total_limit_size. When the limit is reached, overflow_action decides what happens next:

  • throw_exception (the default): the buffer raises BufferOverflowError and the event is dropped at the input. Silent data loss.
  • block: the input thread waits and backpressure propagates upstream. For file-tail inputs this pauses reading; for UDP or socket inputs, the kernel may drop packets instead.
  • drop_oldest_chunk: the oldest queued chunk is discarded to make room. Bounded, visible data loss (tracked by drop_oldest_chunk_count).
flowchart TD
  A[Destination slow or down] --> B[Flush fails, retries with backoff]
  B --> C[Chunks accumulate in queue]
  C --> D{buffer_total_queued_size reaches total_limit_size}
  D -->|throw_exception default| E[BufferOverflowError, events dropped at input]
  D -->|block| F[Input threads stall, upstream pressure]
  D -->|drop_oldest_chunk| G[Oldest chunks discarded, drop_oldest_chunk_count rises]
  E --> H[Silent data loss, gap in downstream]
  F --> H
  G --> H

The important detail: with the default configuration, the failure is nearly invisible to metrics. retry_count may be flat, drop_oldest_chunk_count stays at zero, and the process looks alive and healthy. The only reliable signal is buffer_available_buffer_space_ratios pinned at 0% combined with a gap between what inputs emit and what arrives downstream.

Common causes

CauseWhat it looks likeFirst thing to check
Destination down or unreachableretry_count rising, write_count flat, connection errors in Fluentd logCurl the destination endpoint directly
Destination slow but upflush_time_count / write_count climbing, slow_flush_count incrementingCompare average flush time against flush_interval
Auth or TLS failure at destination401/403 or certificate errors in log, retries with no successesGrep Fluentd log for auth/TLS errors
Buffer undersized for the workloadOverflow fires during normal traffic peaks, destination healthyCompare peak input rate against total_limit_size and drain rate
Memory buffer on a busy nodeRSS climbing alongside buffer_total_queued_size, overflow at 512MBCheck which @type the buffer section uses
Retry storm after long outageretry.steps high, retry.next_time far in the future, queue full of old dataInspect the retry object in monitor_agent output
File buffer disk fullOverflow fires below total_limit_size because the filesystem is out of spacedf -h on the buffer directory partition

Quick checks

These are read-only and safe to run during an incident.

# Confirm the overflow errors and when they started
grep -c "BufferOverflowError" /var/log/td-agent/td-agent.log
grep "BufferOverflowError" /var/log/td-agent/td-agent.log | head -5
grep "BufferOverflowError" /var/log/td-agent/td-agent.log | tail -5
# fluent-package logs to /var/log/fluent/fluentd.log instead

# Buffer state per output plugin: how full, how deep the queue
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, avail_pct: .buffer_available_buffer_space_ratios, queue: .buffer_queue_length, total_bytes: .buffer_total_queued_size}'

# Is the output actually delivering? write_count should be incrementing
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, writes: .write_count, retries: .retry_count, rollbacks: .rollback_count}'
# Run twice, 60s apart. Flat write_count with rising retry_count means the destination is failing.

# Retry state: how deep in backoff are we?
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, retry: .retry}'
# If retry.next_time is minutes or hours out, recovery will be slow even after the destination is fixed.

# Destination health, checked independently of Fluentd
curl -sv --max-time 5 https://your-destination-endpoint/ 2>&1 | tail -20

# If file-backed buffer: is the partition full?
grep -E "path" /etc/td-agent/td-agent.conf | grep -v "#"
df -h /var/log/td-agent/buffer/   # adjust to your buffer path

# Recent output errors: the why behind the retries
grep -iE "failed to flush|temporarily failed|broken pipe|connection refused|401|403|429|certificate" \
  /var/log/td-agent/td-agent.log | tail -20

How to diagnose it

  1. Confirm which output is overflowing. The BufferOverflowError log lines name the plugin. Match that against the monitor_agent output to see which output’s buffer_available_buffer_space_ratios is at 0%. In multi-output configs, usually only one destination is the problem.

  2. Check destination health independently. Do not trust Fluentd’s view. Curl the endpoint, check the destination’s own monitoring (Elasticsearch cluster state, S3 errors, Kafka broker health, Splunk HEC status). Most overflow incidents trace back here.

  3. Classify the failure: dead, slow, or rejecting. Flat write_count plus rising retry_count means dead or unreachable. Incrementing write_count with rising average flush time (flush_time_count / write_count) and slow_flush_count means slow. 401/403 in the logs means rejecting. Each has a different fix.

  4. Check the retry state. Pull the retry object from the API. If retry.steps is large and retry.next_time is far out, exponential backoff has pushed the next attempt well into the future. Even a fixed destination will not drain the buffer quickly. Restarting Fluentd resets retry state, but only do this after the destination is confirmed healthy, and only if you accept the data-loss implications for memory buffers.

  5. Quantify the data loss. With throw_exception, dropped events are not counted anywhere reliable. Estimate the gap by comparing input emit_records rate against what arrived downstream during the incident window. Note: input emit_records requires enable_input_metrics true in <system>; without it the counter reads 0.

  6. Verify buffer configuration. Confirm @type (memory or file), total_limit_size, chunk_limit_size, and overflow_action for the affected output. Use curl -s "http://localhost:24220/api/plugins.json?with_config=true" to see the running configuration rather than trusting the config file on disk.

  7. For file buffers, rule out disk. If the partition holding buffer chunks is full, overflow behavior can fire below the configured total_limit_size. Also confirm the buffer directory is not sharing a partition with system or application logs, since a full buffer can cascade into host-wide logging failure.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
buffer_available_buffer_space_ratiosDistance to overflow. This is the cliff-edge gauge.Below 20% and still falling; below 5% means overflow is imminent
buffer_total_queued_sizeAbsolute bytes buffered (staged + queued)Approaching total_limit_size with positive growth rate
buffer_queue_lengthQueued (not staged) chunks. High queue = backpressureSustained growth; queue not draining between flush intervals
write_count (rate)Chunks actually deliveredFlat for more than one flush interval while input is active
retry_count and retry.next_timeDestination failing; backoff state tells you recovery speedNon-zero sustained; next_time minutes or hours out
flush_time_count / write_countAverage flush latency. Earliest destination-degradation signalRising trend; flush time approaching flush_interval
slow_flush_countFlushes exceeding slow_flush_log_threshold (default 20s)Incrementing steadily
Input vs output emit_records ratesThe only practical way to detect silent dropsSustained divergence over a 15-minute window
drop_oldest_chunk_countOnly if overflow_action drop_oldest_chunk is setAny increment is confirmed data loss

Fixes

Restore the destination first

Nothing else matters until data can drain. Fix the Elasticsearch cluster, rotate the expired credentials, replace the certificate, unblock the network path. Every minute spent resizing buffers while the destination is down just defers the same failure.

After recovery, force the buffer to drain

If retry.next_time is far in the future due to accumulated exponential backoff, a Fluentd restart resets retry state and lets a healthy destination start draining immediately. Warning: with memory-backed buffers, a restart discards everything buffered. With file-backed buffers, chunks persist and replay, so restart is safe but expect a flush burst and possible duplicates. Restart only after the destination is confirmed healthy, or you refill the buffer against a still-broken output.

Resize the buffer, deliberately

If the destination is healthy but the buffer is too small for your peak-to-drain ratio, raise total_limit_size (and ensure chunk_limit_size still divides it sensibly). For memory buffers, remember that buffered bytes are process RSS: a larger memory buffer moves you closer to an OOM kill, which is a worse failure mode than overflow. For file buffers, verify the partition actually has the space you are configuring, plus headroom for the OS.

Choose overflow_action explicitly

The default throw_exception is the worst option for most production pipelines because the loss is silent. Pick per output:

  • drop_oldest_chunk: when pipeline liveness matters more than complete delivery, and you can tolerate losing the oldest data first. Monitor drop_oldest_chunk_count. Caveat: if a chunk’s retries are exhausted (and retry_forever is false), chunks in the queue can be discarded wholesale, not just the oldest.
  • block: when losing events is worse than slowing producers. Best suited to in_tail workloads where the input can safely pause. Socket and UDP inputs do not absorb backpressure well; the kernel drops what Fluentd does not read.
  • Route overflow to a backup: instead of changing overflow_action, configure a <secondary> output or an @ERROR label so failed or rejected events land somewhere durable (local file, object storage) rather than vanishing.

Fix forward-chain data loss

In collector-to-aggregator topologies, a BufferOverflowError on the aggregator can cause the collector’s out_forward to mishandle the failure and purge its own chunk as if delivery succeeded. If you run chained Fluentd and see gaps on the collector side during aggregator overflows, check whether your versions are affected (reported in the v1.12.x to v1.14.x range) and upgrade.

Prevention

  • Alert on the leading edge, not the cliff. Page on buffer_available_buffer_space_ratios below 20% with positive growth, not on the overflow error itself. By the time BufferOverflowError logs, data is already gone. On Fluentd < v1.10.0 the ratio metric is broken (always 0 or 100 due to a rounding bug); upgrade or fall back to buffer_total_queued_size against total_limit_size.
  • Track average flush time. Rising flush_time_count / write_count is the earliest destination-degradation signal, appearing before retries start. Alert when average flush time exceeds roughly half of flush_interval.
  • Use file-backed buffers in production. Memory buffers trade durability for speed. A crash or OOM kill destroys everything buffered, and the default 512MB limit arrives fast under a stalled destination.
  • Size for drain, not for steady state. Compute runway as (total_limit_size - current usage) / buffer growth rate during a destination outage, and make sure that runway exceeds your realistic mean time to repair.
  • Compare input and output rates continuously. A sustained divergence is the only practical detector for silent drops under throw_exception. On older versions, set enable_input_metrics true so input counters exist at all.
  • Set retry_randomize true (default in v1.x) so many agents recovering simultaneously do not hammer a just-recovered destination into another failure.

How Netdata helps

  • Netdata collects Fluentd’s monitor_agent metrics per output plugin, so buffer_available_buffer_space_ratios, buffer_queue_length, and buffer_total_queued_size are graphed continuously rather than sampled by hand during the incident.
  • Correlating buffer fill rate against write_count and retry_count on one dashboard shows the classic cascade shape (writes flat, retries climbing, buffer draining to zero) in seconds instead of requiring three manual API polls.
  • Because buffer fill is a cliff-edge, per-second collection catches fast-moving overflows that minute-resolution scraping misses entirely, especially with the 512MB memory buffer default.
  • Anomaly detection on average flush time and slow-flush rate flags destination degradation before retries begin, which is the earliest point you can act.
  • Alerting on the ratio below 20% with positive growth gives you runway to fix the destination before BufferOverflowError ever fires.