Fluentd silent data loss: logs missing downstream with no error at all

Someone queries the log store for an incident window and the logs are not there. Not delayed, not misindexed: missing. You check Fluentd. The process is running, retry_count is zero, no error logs, buffer metrics look unremarkable. Everything is green, and the data is gone.

This is the hardest Fluentd failure to detect because it is designed into the defaults. When the buffer fills and overflow_action is throw_exception (the default), new events are rejected at the input and lost. When overflow_action is drop_oldest_chunk, chunks are discarded with only a log warning and one counter that almost nobody alerts on. Parse failures can drop records with no visible trace if the error stream is not routed anywhere. In all three cases the pipeline looks healthy from every angle except the one that matters: the destination.

You cannot detect this from any single metric. You detect it by comparing input emit_records against output emit_records over a window, watching buffer space ratios, and verifying end-to-end at the destination.

What this means

Fluentd’s delivery guarantee is not “every event arrives.” It is “every event the buffer accepts will be retried until delivered or exhausted.” Data loss happens at the boundaries that guarantee does not cover:

1. At the buffer boundary. When buffer_total_queued_size reaches total_limit_size, the overflow_action fires. With the default throw_exception, an exception is raised back into the input plugin and the event is lost. With drop_oldest_chunk, the oldest queued chunk is discarded to make room. Neither path produces a retry or an error-level log.

2. At the parser boundary. A record that fails to parse can be dropped before it ever reaches the buffer. Whether it lands anywhere depends on whether error events are routed (the @ERROR label) and whether anything is listening there.

3. At the routing boundary. A misconfigured <match> can send events to a null output or the wrong destination. Input and output rates look perfectly healthy. The data simply arrives somewhere nobody looks.

flowchart TD
  A[Events arrive at input] --> B{Buffer has space?}
  B -->|yes| C[Chunk staged, queued, flushed]
  C --> D[Destination]
  B -->|no, throw_exception default| E[Exception at input - event lost - no counter]
  B -->|no, drop_oldest_chunk| F[Oldest chunk discarded - drop_oldest_chunk_count increments]
  B -->|no, block| G[Input thread stalls - backpressure upstream]
  A --> H{Parse OK?}
  H -->|no, no @ERROR label route| I[Record dropped silently]
  C --> J{Match routes correctly?}
  J -->|no| K[Null or wrong output - metrics stay green]

The common thread: the process stays alive, retry metrics stay flat, and the only evidence is a gap between what went in and what came out, or a hole in the destination.

Common causes

Quick checks

All of these are read-only. They assume monitor_agent is enabled on port 24220 (per-worker ports auto-increment in multi-worker mode: 24220, 24221, …).

# 1. Compare input vs output record totals (the core check)
curl -s http://localhost:24220/api/plugins.json | \
  jq '[.plugins[] | select(.plugin_category=="input") | .emit_records // 0] | add'
curl -s http://localhost:24220/api/plugins.json | \
  jq '[.plugins[] | select(.plugin_category=="output") | .emit_records // 0] | add'

These are cumulative counters since process start, so compare deltas over a window, not raw totals. A sustained gap where input exceeds output, beyond what batching explains, is your loss signal. On older Fluentd versions, input-side emit_records may require enable_input_metrics true in <system>; without it the counter reads 0 and this check is impossible.

# 2. Buffer space remaining per output
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, avail_pct: .buffer_available_buffer_space_ratios}'

Anything at or near 0% means the overflow_action is firing or about to fire.

# 3. Dropped chunks (only nonzero when overflow_action is drop_oldest_chunk)
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, dropped: .drop_oldest_chunk_count}'

Any nonzero value is confirmed data loss.

# 4. Writes falling through to the secondary output
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, secondary: .write_secondary_count}'

# 5. Grep Fluentd's own log for overflow and discard evidence
grep -iE "BufferOverflowError|drop_oldest_chunk|discard" /var/log/td-agent/td-agent.log | tail -30
# fluent-package path: /var/log/fluent/fluentd.log

# 6. Check which overflow_action each output is actually running
grep -B2 -A8 "overflow_action\|<buffer" /etc/td-agent/td-agent.conf
# fluent-package path: /etc/fluent/fluentd.conf

If overflow_action does not appear at all, you are running the default: throw_exception.

# 7. Verify end-to-end: query the destination for a known recent event
TIMESTAMP=$(date +%s)
echo "{\"silent_loss_check\":\"$TIMESTAMP\"}" | fluent-cat debug.silentloss
# Then search the destination for that timestamp value

If the synthetic event arrives but historical application events from the same window do not, the loss was real and bounded in time.

How to diagnose it

1. Bound the loss window. At the destination, find the first and last timestamp of the gap. Everything else is easier once you know when it happened.

2. Correlate the window with Fluentd’s own log. Look for BufferOverflowError warnings, “no queued chunks to be dropped”, retry exhaustion messages, or a restart/OOM at the start of the gap. Fluentd’s own log is the highest-signal artifact here because the metrics often show nothing.

3. Check buffer state history. If you have time series on buffer_available_buffer_space_ratios, did it sit at or near 0% during the gap? If yes, you lost data at the buffer boundary regardless of which overflow_action is configured.

4. Identify the configured overflow_action. throw_exception means new events were rejected at the input during the full period. drop_oldest_chunk means the oldest data was discarded, and drop_oldest_chunk_count tells you how many chunks. block means the loss, if any, happened upstream (for example UDP syslog packets dropped by the kernel while the input thread was stalled).

5. Rule out parse drops. If the gap affects only specific log sources or formats, compare the volume of raw lines in the source files against input emit_records for that input over the same window. A shortfall with a healthy buffer points at the parser. Check whether the config has a <label @ERROR> section; without one, parse-failed records have nowhere to go.

6. Rule out misrouting. If all Fluentd metrics look healthy for the window, inject a synthetic event on the affected tag and watch where it lands. If it arrives at an unexpected destination (or none), the problem is the match configuration, not delivery.

7. Quantify the loss. For a full-buffer event, the lost volume is roughly input rate x duration of the overflow window. For drop_oldest_chunk, the chunk count times average records per chunk gives an estimate. You need this number for the incident review even though the data itself is unrecoverable.

One caution on interpretation: retry_count is a cumulative error counter, not a current-state gauge, and in some versions it does not return to zero after a successful flush. Do not treat a stale nonzero retry_count as evidence of an active problem, and do not treat retry_count of zero as proof nothing was lost.

Metrics and signals to monitor

Fixes

Fix the buffer overflow path

The root fix is always the destination or the buffer sizing: a buffer that fills is a symptom of an output that cannot keep pace. See Fluentd buffer queue length growing for the throughput side and Fluentd BufferOverflowError for the acute overflow event.

For the loss behavior itself, choose the overflow_action deliberately per output instead of inheriting the default:

• throw_exception (default): rejects new events at the input. Acceptable only if you monitor buffer space ratios tightly and treat approach-to-full as an incident.
• block: stalls input threads, converting data loss into backpressure. Safer for data, but pushes the problem upstream (sender timeouts, kernel UDP drops). The official docs do not recommend block as the general answer; a secondary output or the @ERROR label is usually the better safety net.
• drop_oldest_chunk: keeps the pipeline moving by discarding oldest data. Defensible for metrics-like logs, unacceptable for audit or security logs. If you use it, alerting on drop_oldest_chunk_count is mandatory, not optional.

Sizing changes that buy runway: increase total_limit_size (defaults are 512MB for memory buffers, 64GB for file buffers), and prefer file-backed buffers in production so a restart does not destroy the backlog. Memory buffers lose everything unflushed on crash or restart.

Fix the parse-drop path

Route parse failures somewhere visible. Ensure error events are captured by a <label @ERROR> section that writes to a file or a dead-letter destination, so a format change produces artifacts instead of silence. When an upstream application changes its log format, the volume into @ERROR is your detection signal.

Fix the misrouting path

Verify tag routing end-to-end after every config change: inject a synthetic event per tag with fluent-cat and confirm arrival at the intended destination. Treat “all metrics green but wrong destination” as a standard post-change check, not an exotic edge case.

Prevention

• Alert on the input/output comparison. Compute both rates from monitor_agent, sum across plugins, and alert on sustained divergence over a window of at least max(2 x flush_interval, 10 minutes). Require input rate > 0 so idle hosts do not false-positive. This is the single most valuable alert in this entire article.
• Alert on buffer space, not just queue depth. buffer_available_buffer_space_ratios below 20% and falling is a ticket; below 5% with a stalled write_count is a page. This fires before the loss starts.
• Alert on drop_oldest_chunk_count and write_secondary_count at any increment. Both mean data already went somewhere other than the primary destination.
• Enable input metrics. If input-side emit_records reads zero, set enable_input_metrics true in <system>, or the input side of the comparison is blind forever.
• Make overflow_action an explicit, reviewed choice in every output’s buffer config. Never ship the default by accident.
• Monitor Fluentd’s own log. BufferOverflowError and chunk-discard warnings appear there before any counter moves. The log pipeline’s own logs are often the first place this failure shows up.
• Verify end-to-end routinely. Periodic synthetic events per tag, checked at the destination, catch the misrouting and parse-drop cases that no internal metric can see.
• Use file-backed buffers with adequate disk headroom so restarts and OOM kills do not convert backpressure into permanent loss.

How Netdata helps

• Netdata collects the monitor_agent fields that matter here (emit_records per plugin, buffer_available_buffer_space_ratios, drop_oldest_chunk_count, write_count, retry_count) as per-second time series, so the input-vs-output divergence becomes a visible, alertable comparison instead of a manual curl during an incident.
• Per-second buffer space ratios show the approach to overflow minutes before the overflow_action fires, which is the only window where you can still prevent the loss.
• Because every counter is retained as history, bounding the loss window after the fact (when did the gap start, how long did the buffer sit at 0%) is a dashboard query rather than log archaeology.
• Correlating Fluentd’s buffer and emit metrics with host-level signals (process restarts, OOM kills, disk fill on the buffer partition) distinguishes “buffer overflow drop” from “memory buffer lost on restart” without switching tools.
• Anomaly detection on per-plugin emit rates flags the shape of silent loss (input steady, output subtly lower) that fixed thresholds tend to miss.

Related guides

• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd buffer stage vs queue: telling healthy batching from backpressure
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• How Fluentd actually works in production: a mental model for operators
• Fluentd monitor_agent not responding: a process that is up but hung
• Fluentd monitoring checklist: the signals every production log pipeline needs
• Fluentd monitoring maturity model: from survival to expert
• Fluentd plugin load error at startup: LoadError and missing gems
• Fluentd poison pill crash loop: one bad log line that kills the process on every restart
• Fluentd process not running: the log pipeline is dead and the host has gone dark