Fluentd output rate lower than input rate: the deficit that is quietly dropping logs

Your Fluentd process is up. The monitor agent responds. retry_count is zero. And yet, when you query the destination, whole time windows of logs are missing. The cause is usually the same: output emit_records has been running below input emit_records for hours or days, and nobody was comparing them.

Most teams chart input rate and output rate independently and never compute the ratio. A sustained 5% deficit on a host doing 200 events per second is over 6 million events lost per week. The two rates should converge over any reasonable window. When they do not, data is either being dropped or piling up in a buffer that will eventually overflow and drop it anyway.

What this means

Every event flows through Input, Parser, Filter chain, Buffer, Output. Input and output plugins each maintain a cumulative emit_records counter. In a healthy pipeline, the output rate tracks the input rate. There are only three ways output can stay below input:

1. Events are accumulating in the buffer. The output cannot keep pace. buffer_queue_length grows and buffer_available_buffer_space_ratios declines. This is debt that must be repaid, either by draining or by overflow.
2. Events are being dropped deliberately. overflow_action drop_oldest_chunk is discarding old chunks (drop_oldest_chunk_count increments), or retries have been exhausted and chunks were discarded or sent to a secondary (write_secondary_count increments).
3. Events are being dropped silently. The buffer is full and overflow_action is the default throw_exception. New events are rejected at the buffer and lost. No counter reliably tracks these drops. The only evidence is the rate gap itself, plus buffer_available_buffer_space_ratios pinned near 0%.

There is also one legitimate cause: time-sliced outputs (daily S3 files, for example) hold chunks until the slice expires, so input and output rates diverge by design and converge only when the slice rolls over.

flowchart TD
  A[Output rate below input rate sustained] --> B{Buffer queue growing?}
  B -- Yes --> C[Output cannot keep pace]
  C --> C1{write_count incrementing?}
  C1 -- No --> C2[Destination down or retrying: check retry_count and logs]
  C1 -- Yes, slowly --> C3[Destination slow: check flush_time_count per write_count]
  B -- No, buffer full --> D{overflow_action?}
  D -- drop_oldest_chunk --> E[Confirmed loss: drop_oldest_chunk_count increments]
  D -- throw_exception --> F[Silent loss at buffer: no counter, gap is the signal]
  B -- No, buffer healthy --> G{Time-sliced output?}
  G -- Yes --> H[Expected: rates converge at slice boundary]
  G -- No --> I[Misrouting or filter dropping: verify end to end]

Common causes

Quick checks

All of these are read-only. They assume the monitor agent is enabled on the default port 24220. In multi-worker mode, each worker has its own port (24220 + worker_id), so check each one.

# Sum input emit_records and output emit_records
curl -s http://localhost:24220/api/plugins.json | \
  jq '{input: [.plugins[] | select(.plugin_category=="input") | .emit_records // 0] | add,
       output: [.plugins[] | select(.plugin_category=="output") | .emit_records // 0] | add}'

# Run it twice, 60 seconds apart, and compute per-second rates from the deltas.
# These are cumulative counters; raw values are meaningless without a delta.

# Per-output breakdown: writes, retries, queue, available space
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") |
      {id: .plugin_id, writes: .write_count, retries: .retry_count,
       rollbacks: .rollback_count, queue: .buffer_queue_length,
       avail_pct: .buffer_available_buffer_space_ratios,
       dropped: .drop_oldest_chunk_count, secondary: .write_secondary_count}'

# Average flush time per output (run twice, divide deltas)
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") |
      {id: .plugin_id, flush_ms: .flush_time_count, writes: .write_count,
       slow: .slow_flush_count}'

# Buffer data age: how far behind is the oldest chunk
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") |
      {id: .plugin_id, oldest: .buffer_oldest_timekey, newest: .buffer_newest_timekey}'

# Recent output errors from the Fluentd log (path varies by package)
grep -E "failed to flush|retry|BufferOverflowError|could not connect" \
  /var/log/td-agent/td-agent.log | tail -20

One prerequisite that bites people constantly: on Fluentd versions before v1.19.0, input emit_records requires enable_input_metrics true in the <system> section. Without it, the input counter is always 0 and the comparison is impossible. If your input sum comes back as zero while logs are clearly flowing, check that setting first.

How to diagnose it

1. Confirm the deficit is real, not sampling noise. Both counters are cumulative. Take two samples at least max(2 * flush_interval, 10 minutes) apart, compute per-second rates, and compare. Require input rate greater than 0 before computing any ratio, or you will divide by zero on idle hosts. Batching causes short-term spikes; only a sustained gap over that window matters.

2. Check the buffer to classify the deficit. Look at buffer_queue_length and buffer_available_buffer_space_ratios per output. Growing queue with declining available space means accumulation: the output cannot keep pace. Available space pinned near 0% with a stable queue means the buffer is full and the overflow action is firing: active loss.

3. If accumulating, find out why the output is slow. Flat write_count with rising retry_count and rollback_count means the destination is failing; read the Fluentd error log for the actual error (connection refused, 401/403, 429, TLS). Writes incrementing but too slowly, with rising flush_time_count / write_count and slow_flush_count increments, means the destination is degraded, not dead.

4. If the buffer is full, check which loss path you are on. drop_oldest_chunk_count incrementing means confirmed, counted loss of the oldest chunks. If it is zero and overflow_action is unset, you are on the default throw_exception: new events are being rejected at the buffer with no counter. Grep the Fluentd log for BufferOverflowError to confirm.

5. Check retry exhaustion. If retry_forever is false and retry_max_times is unset, retries are governed by retry_timeout (default 72 hours), after which the chunk is discarded. If a <secondary> is configured, write_secondary_count tells you chunks have already fallen through to the backup destination. Also check the retry object: if retry.next_time is far in the future, exponential backoff has pushed recovery out even though the process looks alive.

6. Rule out the legitimate cause. For time-sliced outputs, compare buffer_oldest_timekey to the current time and the configured timekey and timekey_wait. A gap that closes at each slice boundary is expected behavior, not loss.

7. If buffers and outputs look healthy, suspect routing. Events can flow through Fluentd and land in a null output or the wrong destination. Everything looks green, but the data never arrives where you query it. Verify end to end: inject a test event with fluent-cat and confirm it appears at the destination.

Metrics and signals to monitor

Fixes

Destination slow or failing

Fix the destination first. No Fluentd-side tuning compensates for an Elasticsearch cluster in red, an expired credential, or a throttled S3 bucket. Check the destination independently, then read the Fluentd error log for the specific failure. If retry backoff has run away (retry.next_time far in the future), a Fluentd restart resets retry state, but only do this after the destination is healthy. A restart also resets all cumulative counters, including the ones you were just comparing. With file-backed buffers, unflushed chunks survive the restart and will replay; with memory-backed buffers, everything buffered is lost on restart.

Buffer full with throw_exception

You are losing new events silently. Decide explicitly which overflow semantics you want per output: block stops the loss but exerts backpressure on inputs (upstream senders may drop instead), drop_oldest_chunk keeps the pipeline flowing but loses old data in a countable way. There is no zero-loss option once the buffer is full; the real fix is capacity. Raise total_limit_size if disk allows, and treat buffer_available_buffer_space_ratios as the early warning you should have had.

Output cannot keep pace with input

If average flush time is approaching flush_interval, the output has no headroom. Options: increase flush_thread_count (helps for I/O-bound outputs, since network writes release the GVL; it does not help if the bottleneck is CPU-bound serialization), increase chunk_limit_size so each flush carries more records, or add workers in multi-worker mode for true parallelism. Each option trades memory, destination load, or operational complexity. If the destination itself is undersized for the event volume, scale the destination or shed input volume at the source.

Retry exhaustion discarding chunks

Set explicit retry policy instead of inheriting the 72-hour retry_timeout default, and configure a <secondary> output so exhausted chunks land somewhere recoverable instead of being discarded with only a log line. Monitor write_secondary_count so you know when the fallback has engaged.

Prevention

• Compare the rates, always. Alert when output rate stays below input rate over a window of at least max(2 * flush_interval, 10 minutes), with the input > 0 guard. This one check catches nearly every loss path in this article, including the silent ones no error counter exposes.
• Enable input metrics. On Fluentd before v1.19.0, add enable_input_metrics true to <system>. Without it the comparison is impossible.
• Choose overflow_action explicitly per output. Never inherit the default throw_exception without understanding that it means silent loss under a full buffer.
• Use file-backed buffers in production. Memory buffers lose all unflushed data on restart or OOM kill. The deficit you detect today becomes unrecoverable loss the moment the process dies.
• Alert on the confirmation signals too. drop_oldest_chunk_count any increment, write_secondary_count any increment, buffer_available_buffer_space_ratios below 20% and declining. The rate ratio tells you something is wrong; these tell you what.
• Account for time-sliced outputs in the alert. Exclude or widen the window for outputs whose gap closes at slice boundaries, or you will train the team to ignore the alert.

How Netdata helps

• Netdata collects per-plugin emit_records for both inputs and outputs and derives rates automatically, so the input/output comparison is a first-class chart rather than a manual jq exercise.
• Buffer gauges (buffer_queue_length, buffer_available_buffer_space_ratios, stage vs queue byte sizes) are graphed alongside the throughput rates, so you can see in one view whether a deficit is accumulating, full, or draining.
• Loss-confirmation counters (drop_oldest_chunk_count, write_secondary_count, retry_count, rollback_count) are tracked per output plugin, letting you move from “there is a gap” to “this output, this cause” without SSHing in.
• Flush latency signals (flush_time_count, slow_flush_count) let you spot destination degradation as a rising average flush time before the rate deficit even appears.
• In multi-worker deployments, per-worker visibility surfaces imbalances that aggregate metrics hide, since each worker has independent buffers and counters.

Related guides

• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss
• Fluentd emit_error_count: the number-one under-monitored data-loss signal
• Fluentd failed to flush the buffer: the output cannot deliver and retries begin
• Fluentd memory vs file buffer: why the default buffer loses data on restart
• Fluentd monitor_agent not responding: a process that is up but hung
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• How Fluentd actually works in production: a mental model for operators
• Fluentd monitoring checklist: the signals every production log pipeline needs