Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss

drop_oldest_chunk_count is not a warning signal. It is a receipt for data that no longer exists. Every increment means Fluentd discarded the oldest buffered chunk to make room for incoming events, and those events are permanently gone. No retry, no secondary output, no replay will bring them back.

This counter only moves when you have explicitly configured overflow_action drop_oldest_chunk in an output’s <buffer> section. That setting is a deliberate trade: keep the pipeline alive under sustained output failure by sacrificing the oldest data first. The failure mode is that nothing else in the system complains when it fires. Fluentd keeps running, input keeps flowing, output keeps writing. The only evidence of loss is this counter and a warning line in Fluentd’s own log, which most pipelines never scrape.

The question this article answers: when you see the counter move, how do you tell a contained one-time overflow from active, ongoing loss?

What this means

Fluentd organizes buffered events into chunks. A chunk moves through a lifecycle: staged (accumulating events), queued (ready to flush), flushing (being written to the destination), then purged on success or rolled back on failure. The buffer has a hard ceiling, total_limit_size, covering staged and queued chunks together. When a new event arrives and the buffer has no room, overflow_action decides what happens:

• throw_exception (the default): the new event is rejected at the input. Data is lost through a different path, and there is no dedicated counter for it.
• block: the input thread waits. Data is preserved but backpressure propagates upstream.
• drop_oldest_chunk: the oldest queued chunk is discarded to free space. That is what increments drop_oldest_chunk_count.

Two properties make this signal operationally important. First, it is cumulative and only resets on process restart, so the absolute value tells you total loss since startup and the rate tells you whether loss is happening now. Second, it only fires under a specific configuration, so a zero value does not mean “no loss” unless you have confirmed the overflow action in the running config. With the default throw_exception, this counter stays at zero while data is dropped elsewhere. Check the config before trusting the zero.

When a chunk is dropped, Fluentd logs a line like [warn]: #0 dropping oldest chunk to make space after buffer overflow. Repeated warnings mean repeated increments.

Common causes

One nasty variant: when the buffer is full but the data is still in staging chunks rather than queued chunks, there is nothing to drop, and Fluentd logs an error about having no queued chunks to drop instead of cleanly shedding load. This happens when many low-volume tag streams each hold a staging chunk. The workaround is a lower chunk_full_threshold or a higher total_limit_size.

Watch for the compound failure too: if retry limits (retry_timeout or retry_max_times) are hit while overflow is active, Fluentd drops the entire queue, not just the oldest chunk. That shows up as a large step in effective loss, logged as Hit limit for retries. dropping all chunks in the buffer queue.

Quick checks

All of these are read-only. Paths assume td-agent packaging; adjust for fluent-package or your Kubernetes deployment.

# Which outputs are dropping, and how much
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, dropped: .drop_oldest_chunk_count}'

# Is the buffer still under pressure right now
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, avail_pct: .buffer_available_buffer_space_ratios, queue: .buffer_queue_length, total_bytes: .buffer_total_queued_size}'

# Is the output recovering (writes happening) or stalled (retries only)
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, writes: .write_count, retries: .retry_count, rollbacks: .rollback_count}'

# Confirm the overflow action actually configured
grep -B2 -A8 'overflow_action' /etc/td-agent/td-agent.conf

# Find the drop warnings and any retry-exhaustion queue purges
grep -E "dropping oldest chunk|dropping all chunks|no queued chunks" /var/log/td-agent/td-agent.log | tail -20

# How far behind is the oldest buffered data
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, oldest: .buffer_oldest_timekey, newest: .buffer_newest_timekey}'

In multi-worker mode, repeat the API checks against each worker’s port (24220, 24221, and so on). Workers have independent buffers, and one can be dropping while the others are fine.

How to diagnose it

1. Quantify the loss window. Sample drop_oldest_chunk_count twice, a few minutes apart, per output plugin. If the delta is zero, the loss event is over and you are in postmortem mode: note the total count and find the root cause from logs. If the delta is positive, loss is ongoing and the clock is running.

2. Check buffer headroom. Read buffer_available_buffer_space_ratios for the same plugin. Below 5% while drops continue means the buffer cannot absorb anything and every new burst of input immediately costs you old data. Above 20% with drops stopped means the event has passed.

3. Determine whether the output is recovering. Sample write_count twice. Flat write_count with rising retry_count means the destination is still rejecting everything and drops will continue. Incrementing write_count with a draining queue means recovery is underway and the remaining buffered data will be delivered.

4. Find why the output failed. Check Fluentd’s log for connection refused, auth failures (401/403), rate limiting (429), or TLS errors around the time drops started. Then verify the destination independently from the Fluentd host.

5. Estimate the data that was lost. Each dropped chunk held up to chunk_limit_size bytes (256MB default for file buffers, 8MB for memory buffers). Multiply the drop count by your typical chunk fill for an upper bound. buffer_oldest_timekey before and after the event tells you which time range of logs is missing downstream.

The diagnostic flow:

flowchart TD
  A[drop_oldest_chunk_count delta > 0] --> B{drops sustained over 5 min?}
  B -- no --> C[TICKET: contained event, postmortem the cause]
  B -- yes --> D{buffer_available space < 5%}
  D -- no --> E[TICKET: buffer absorbing, watch closely]
  D -- yes --> F{write_count flat OR retry_count rising}
  F -- no --> E
  F -- yes --> G[PAGE: active loss, no recovery, intervene now]

Metrics and signals to monitor

Alerting guidance that matches the severity model: ticket on any single increment of drop_oldest_chunk_count, because confirmed loss always deserves a human look even if the event self-resolved. Page only on the combination of sustained drops for more than 5 minutes, buffer available space under 5%, and a stalled output (write_count flat or retry_count rising). That combination is active loss with no recovery path, and it does not self-resolve.

Fixes

Fixes group by root cause. Do not restart Fluentd first: a restart resets the counter and the retry state, which destroys the evidence you need and, with memory-backed buffers, destroys the remaining buffered data too.

Destination down or failing. Restore the destination. While it is down, the buffer is your only protection. If the outage will be long and disk allows, raise total_limit_size to buy time, and confirm a <secondary> output exists so retry-exhausted chunks go somewhere instead of being discarded. Check the retry object: if next_time is far in the future due to backoff, recovery will lag even after the destination is healthy.

Destination permanently slower than input. Drops in periodic bursts with a healthy destination mean the output path is undersized. Increase flush_thread_count, verify average flush time (flush_time_count / write_count) is well under flush_interval, and consider scaling the destination or adding workers. Until output throughput exceeds input throughput, drops will recur at every peak.

Buffer undersized. Raise total_limit_size to cover your longest expected destination outage at peak ingest rate. For file buffers, verify the filesystem actually has that much free space; the configured limit and the disk are independent ceilings. For memory buffers, every buffered byte is RSS, so plan container memory limits with headroom for Ruby overhead.

Staging-chunk deadlock (“no queued chunks to be dropped”). Lower chunk_full_threshold so staging chunks are queued sooner and become droppable, or reduce tag fragmentation so fewer concurrent staging chunks exist. Raising total_limit_size also relieves it.

Config-level decision. Revisit whether drop_oldest_chunk is still the right choice for this output. If losing the oldest data is unacceptable, block preserves data at the cost of upstream backpressure. There is no setting that makes a full buffer free; you are choosing which data is at risk.

Prevention

• Alert on the counter, not just the buffer. Any increment tickets. The combined condition (sustained drops, buffer under 5%, stalled output) pages. This is in the mature tier of the monitoring checklist and most teams never wire it up.
• Watch the leading indicator. buffer_available_buffer_space_ratios gives you time-to-overflow before a single chunk is lost. See computing time-to-overflow.
• Size for the outage, not the average. Set total_limit_size from peak ingest rate multiplied by your tolerable destination outage window, with filesystem headroom to match.
• Keep input and output rates compared. A persistent gap between input and output emit_records is the earliest sign you are trending toward overflow. See buffer queue length growing.
• Choose overflow_action deliberately per output. Understand the tradeoffs before an incident picks one for you: overflow_action: throw_exception, block, and drop_oldest_chunk.
• Add a secondary output for important streams so retry exhaustion writes to a fallback instead of discarding the queue.
• Watch Fluentd’s own log for dropping oldest chunk and dropping all chunks lines. The metric tells you loss happened; the log tells you when and which plugin.

How Netdata helps

• Netdata collects the monitor agent fields per output plugin, including drop_oldest_chunk_count, and charts the delta so you see the rate of loss, not just a cumulative number that hides when it happened.
• Correlating the drop counter with buffer_available_buffer_space_ratios on one dashboard distinguishes a contained overflow (drops fired, headroom recovered) from an active bleed (drops firing, headroom near zero).
• Overlaying write_count and retry_count against the drop rate answers the recovery question visually: writes resuming while drops stop means the pipeline healed; retries rising while drops continue means it has not.
• Per-worker breakdowns expose the multi-worker case where one worker’s buffer is dropping while the aggregate looks acceptable.
• Historical retention of buffer_oldest_timekey and queue depth around the event lets you scope the lost time window for the postmortem without having been online when it fired.

Related guides

• Fluentd overflow_action: throw_exception, block, and drop_oldest_chunk
• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd monitoring checklist: the signals every production log pipeline needs
• How Fluentd actually works in production: a mental model for operators