Fluentd average flush time rising: the earliest sign of destination slowdown

Your Fluentd process is alive. retry_count is zero. The buffer queue looks flat. And yet, if you are computing average flush time from flush_time_count and write_count, you can see the destination getting slower, sometimes hours before the first retry fires. This is the earliest signal of destination or network degradation in a Fluentd pipeline, and most teams never look at it because the monitor_agent API does not expose it as a ready-made field.

There is no last_flush_duration field in the API. What you get are two cumulative counters: flush_time_count (total milliseconds spent flushing buffer chunks) and write_count (total successful chunk writes). Both only reset on process restart. If you divide the raw cumulative values, you get a lifetime average that barely moves even when the destination is actively degrading. The only meaningful number is a windowed average: delta(flush_time_count) / delta(write_count) over your scrape interval.

This guide covers how to compute that number correctly, what a rising value means, what to check first, and how to act before it escalates into retries, queue growth, and data loss. For how buffers, flush threads, and retry mechanics fit together, see How Fluentd actually works in production.

What this means

Every output plugin flushes buffer chunks to its destination using flush_thread_count threads (default 1). Each successful flush increments write_count by one and adds the elapsed time of that flush to flush_time_count. A single flush can carry thousands of records, so flush time is dominated by network round trips, TLS handshakes, destination-side indexing or ingestion time, and chunk size.

When the average flush time rises, one of three things is happening:

1. The destination is slower (Elasticsearch under indexing pressure, S3 throttling, a Kafka broker struggling).
2. The network path is degraded (latency, packet loss, an LB closing or delaying connections).
3. Chunks got bigger or compression/serialization got more expensive on the Fluentd side.

What matters is where this signal sits in the failure cascade. Rising flush time comes first. Slow flushes that cross slow_flush_log_threshold come next. Only when flushes actually fail do retry_count and rollback_count move. Queue growth and buffer pressure come last, when flush time approaches or exceeds flush_interval and the output can no longer drain chunks as fast as they stage.

flowchart LR
  A[Destination or network degrades] --> B[Average flush time rises]
  B --> C[slow_flush_count increments]
  C --> D[Flushes start failing: retry_count, rollback_count]
  D --> E[buffer_queue_length grows]
  E --> F[buffer_available_buffer_space_ratios falls]
  F --> G[overflow_action fires: drops, block, or exception]

If you only alert on retries or buffer fill, you are alerting at step 4 or 5 of a 7-step cascade. Average flush time is step 2. Alert on a sustained increase beyond roughly 2x your baseline. Because both counters are cumulative and never reset between restarts, all of this must be computed from deltas, never raw values.

Common causes

The last row deserves emphasis: time-sliced outputs (for example daily or hourly S3 files) hold chunks until the time slice expires. A chunk held for an hour by timekey 3600 is working as designed. Do not page anyone for it. Either exclude time-sliced outputs from flush-time alerting or compare against the expected timekey interval.

Quick checks

All read-only. Paths below use the td-agent package layout; adjust for fluent-package or your container image.

# 1. Snapshot flush counters per output (the raw material for the average)
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, flush_ms: .flush_time_count, writes: .write_count, slow: .slow_flush_count, retries: .retry_count}'

# 2. Confirm the monitor_agent itself is responsive (rules out a hung event loop)
curl -s -o /dev/null -w "%{http_code}\n" --max-time 5 http://localhost:24220/api/plugins.json

# 3. Check queue state: is the backlog growing yet, or is this still early?
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, queue: .buffer_queue_length, avail_pct: .buffer_available_buffer_space_ratios}'

# 4. Slow-flush warnings in Fluentd's own log (one warning line per slow flush)
grep -i "slow flush" /var/log/td-agent/td-agent.log | tail -20

# 5. Look for connection-level problems that inflate flush time
grep -iE "broken pipe|connection reset|timed out|tls|ssl" /var/log/td-agent/td-agent.log | tail -20

# 6. Rule out CPU-side flush starvation (GVL contention on a single worker)
ps -o pid,%cpu,comm -p $(pgrep -f fluentd | head -1)

How to diagnose it

1. Compute the windowed average correctly. Take two snapshots of flush_time_count and write_count separated by a fixed interval (60 seconds works well), then compute delta(flush_time_count) / delta(write_count). Do not divide the raw cumulative values: they are lifetime counters since process start, and a lifetime average will hide a doubling of recent flush time for days on a long-lived process.

2. Establish the baseline. You need the host’s own historical average for the same output plugin, at comparable load. There is no universal threshold: an Elasticsearch output with 64 MB chunks has a completely different normal from a small forward output. Use a relationship-based rule: sustained rise beyond about 2x baseline.

3. Confirm the rise is real, not sampling noise. A single window with a small delta(write_count) produces a noisy average (one big flush dominates). Require the elevated average to persist across several consecutive windows before treating it as a signal.

4. Check whether slow flushes have started. Look at delta(slow_flush_count) and the “slow flush” warnings in the Fluentd log. Individual flushes that cross slow_flush_log_threshold (configurable; default 20 seconds per the playbook, verify against your version ) increment that counter. If the average is rising but slow_flush_count is flat, degradation is still mild and uniform; if slow_flush_count is climbing, some flushes are already severely delayed.

5. Localize the slowness. Compare Fluentd’s flush time against the destination’s own view. If the destination (Elasticsearch indexing latency, broker produce latency, S3 response codes) shows the same degradation, the problem is downstream. If the destination looks healthy but flush time is high, suspect the network path, connection churn, or Fluentd-side serialization cost.

6. Check for the false positive. If the affected output uses timekey-based chunking, verify whether “slow” flushes align with time-slice boundaries. Held chunks are not a latency fault.

7. Project the runway. Compare the current average flush time to flush_interval. Average flush time should stay under 50% of flush_interval. When it approaches the interval, a single flush thread can no longer keep pace and queue growth becomes inevitable, even with zero failures. Effective output capacity is roughly flush_thread_count / avg_flush_time chunks per second.

Metrics and signals to monitor

Fixes

Fixes depend on where the slowness lives. Do not restart Fluentd as a first move; the counters reset on restart and you lose the trend data you need to confirm the fix.

Destination-side slowness

• Fix the destination. Scale the Elasticsearch cluster, resolve the throttling, replace the struggling broker. Fluentd-side tuning only buys buffer time; it cannot outrun a slow destination forever.
• If the destination is undergoing maintenance, verify your buffer headroom (buffer_available_buffer_space_ratios) and compute time-to-overflow at the current fill rate: (total_limit_size - buffer_total_queued_size) / growth_rate. If the runway is shorter than the maintenance window, intervene early. See buffer available space low.

Network path degradation

• Look for LB idle timeouts closing long-lived connections, forcing reconnection (and TLS handshakes) on every flush. Keepalive behavior and LB timeout alignment are covered in broken pipe / connection reset.
• Check for path changes: new hops, cross-AZ or cross-region routing shifts, MTU problems. A modest latency add multiplies across every flush.

Fluentd-side causes

• Increase flush_thread_count on the affected output if single-flush latency is high but the destination has spare capacity. Network writes release the GVL, so parallel flush threads genuinely parallelize I/O. This does not help if the destination itself is saturated; it makes destination overload worse.
• Reduce chunk size (chunk_limit_size) if chunks have grown with traffic and per-chunk write time is the problem. Tradeoff: more chunks means more flush operations and more per-request overhead at the destination.
• Shorten flush_interval if chunks are sitting queued between flushes. Tradeoff: smaller, more frequent writes.
• Reduce parser/filter cost if per-thread CPU shows GVL starvation rather than destination slowness.

What not to do

• Do not treat a restart as a fix. A restart masks the symptom, resets the counters, and with file-backed buffers adds a replay burst that temporarily makes flush times look even worse.
• Do not alert on raw flush_time_count or raw write_count. They are monotonic counters; an alert on the absolute value is meaningless.

Prevention

• Alert on the windowed average, not the counters. Scrape flush_time_count and write_count per output plugin, compute the delta ratio, and page on a sustained rise beyond ~2x baseline. This single alert fires earlier than anything else in the cascade.
• Pair it with the slow-flush ratio. Track delta(slow_flush_count) / delta(write_count) so a rising average is immediately qualified by how many flushes are severely slow.
• Keep flush headroom. Enforce the capacity rule in config reviews: expected average flush time under 50% of flush_interval, and flush_thread_count sized so the output can drain at peak input rate.
• Segment your alerting by output type. Time-sliced outputs need different expectations (compare against the timekey interval) than streaming outputs like Elasticsearch or forward.
• Correlate with destination metrics in dashboards. Flush time next to Elasticsearch indexing latency or S3 error rates turns a 30-minute localization step into a 30-second glance.

How Netdata helps

• Netdata collects the Fluentd monitor_agent counters, including flush_time_count, write_count, and slow_flush_count, per output plugin, so the windowed average flush time can be charted and alerted on as a derived ratio rather than a hand-rolled cron job.
• Because collection is per-second, the delta computation is fine-grained enough to catch flush-time drift within minutes instead of waiting for a 5-minute scrape to smear it.
• Per-plugin breakdowns let you see one output degrading (the Elasticsearch match) while others (S3, forward) stay flat, which immediately localizes the problem to a destination rather than the host.
• Netdata charts flush-time trends alongside buffer_queue_length, retry_count, and buffer space ratios on the same dashboard, so the full cascade from early slowdown to queue pressure is visible in one view.
• Process-level CPU and RSS for the Fluentd workers are collected on the same host, which makes the destination-versus-GVL-starvation distinction a single correlation instead of two separate investigations.

Related guides

• Fluentd failed to flush the buffer: the output cannot deliver and retries begin
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd broken pipe / connection reset: dropped output connections and LB timeouts
• Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss
• Fluentd emit_error_count: the number-one under-monitored data-loss signal
• How Fluentd actually works in production: a mental model for operators