Fluentd slow flush: buffer flush took longer than slow_flush_log_threshold

You see this in the Fluentd log:

[warn]: buffer flush took longer time than slow_flush_log_threshold: elapsed_time=... slow_flush_log_threshold=20.0 plugin_id="..."

This is a performance warning, not a data-loss signal. Fluentd delivered (or is still retrying) a buffer chunk to the destination, and the write took longer than slow_flush_log_threshold, which defaults to 20 seconds. Each occurrence also increments the slow_flush_count counter on that output plugin.

Slow flushes are the earliest indicator of destination degradation. A flush that takes 30 seconds but succeeds is a destination that is one bad day away from rejecting writes entirely. While flushes are slow, effective output throughput drops, chunks accumulate in the queue, and the buffer marches toward total_limit_size. What happens at the limit depends on your overflow_action, and the default (throw_exception) loses data. So the right response to this warning is not “make the log line go away” but “find out why writes to the destination got slow, before slow becomes failed.”

What this means

Every output plugin flushes buffer chunks using flush_thread_count threads (default: 1). A flush is the network write of one chunk to the destination: an Elasticsearch bulk request, an S3 put, a Kafka produce, a forward send. Fluentd times each flush, and when one exceeds slow_flush_log_threshold, it logs the warning and bumps slow_flush_count.

Two things follow:

1. Throughput drops. With one flush thread (the default), a 40-second flush means that output delivers at most one chunk per 40 seconds, regardless of how fast chunks fill up. If a flush takes longer than flush_interval, the output structurally cannot keep pace.
2. The buffer absorbs the difference. Chunks move from staged to queued and sit there. buffer_queue_length grows, buffer_available_buffer_space_ratios shrinks, and you start approaching the conditions described in Fluentd buffer queue length growing and Fluentd BufferOverflowError.

Occasional slow flushes on low-volume outputs are harmless. A single flush that coincides with a destination GC pause or a network retransmit will trip the threshold once and never again. The signal that matters is the ratio delta(slow_flush_count) / delta(write_count): if a meaningful fraction of all writes are slow, the destination has a real performance problem.

flowchart LR
  A[Flush exceeds 20s threshold] --> B[slow_flush_count increments]
  B --> C{Ratio of slow flushes to all writes}
  C -->|occasional, low-volume output| D[Harmless, watch only]
  C -->|sustained high ratio| E[Destination degraded]
  E --> F[Flush throughput drops]
  F --> G[buffer_queue_length grows]
  G --> H[Buffer approaches total_limit_size]
  H --> I[overflow_action fires: data loss or backpressure]

Common causes

Quick checks

All of these are read-only. Paths shown are for the td-agent package; for fluent-package use /var/log/fluent/fluentd.log, and in Kubernetes use kubectl logs.

# 1. How often is the warning firing, and for which plugin?
grep "slow_flush_log_threshold" /var/log/td-agent/td-agent.log | tail -20

# 2. Current slow_flush_count and write_count per output
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, slow: .slow_flush_count, writes: .write_count}'

# 3. Cumulative flush time (ms) per output; delta ratio with write_count is average flush time
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, flush_time_ms: .flush_time_count, writes: .write_count}'

# 4. Is the queue growing while flushes are slow?
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, queue: .buffer_queue_length, avail_pct: .buffer_available_buffer_space_ratios}'

# 5. Any retries or rollbacks yet (slow is tipping into failing)?
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, retries: .retry_count, rollbacks: .rollback_count}'

# 6. Is Fluentd CPU-bound (GVL starvation stalling flush threads)?
ps -p $(pgrep -f fluentd | head -1) -o pid,%cpu,%mem,comm

# 7. Destination reachable and fast from this host?
time curl -s -o /dev/null -w "%{http_code}\n" https://your-destination-endpoint/

# 8. If file-backed buffer: is the disk slow?
iostat -x 2 3

Checks 2 through 5 require the monitor_agent to be configured (<source> @type monitor_agent</source>, default port 24220). In multi-worker mode, each worker has its own port (24220 + worker id), and slow flushes may be concentrated on one worker.

How to diagnose it

1. Quantify the ratio, not the count. Take two snapshots of slow_flush_count and write_count a few minutes apart. Compute delta(slow_flush_count) / delta(write_count). Under 1-2% on a low-volume output is background noise. Above roughly 10%, or any sustained upward trend, is a destination performance problem.
2. Compute average flush time. From the same snapshots, delta(flush_time_count) / delta(write_count) gives average flush latency in milliseconds. Compare it to your baseline and to flush_interval. If average flush time approaches flush_interval, the output is at capacity and the queue will grow on any further slowdown.
3. Check whether the queue is actually growing. If buffer_queue_length is flat near zero and buffer_available_buffer_space_ratios is healthy, the output still keeps pace despite slow flushes; you have headroom but a degrading destination. If the queue is growing, treat this as early-stage backpressure and move to the destination investigation immediately.
4. Isolate Fluentd vs destination. time curl the destination endpoint from the Fluentd host. If that is also slow, the problem is downstream (destination load or network). If the endpoint is fast but flushes are slow, suspect Fluentd-side causes: GVL starvation from heavy parsing, oversized chunks, or a slow buffer disk.
5. Check CPU. A Fluentd process pinned near 100% of one core with a healthy destination means CPU-bound work (regex parsing, serialization, GC) is starving the flush threads. Per-thread CPU (ps -T) will show one thread doing all the work.
6. Check the destination’s perspective. For Elasticsearch, look at indexing pressure and bulk rejection rates; bulk queue saturation on the ES side shows up in Fluentd as exactly this warning. For S3, check throttling. For Kafka, broker health.
7. Look for correlation with restarts or bursts. Slow flushes only right after a restart with file-backed buffers are the normal replay backlog draining. Slow flushes only at the top of the hour suggest a scheduled destination-side job (snapshot, merge, backup).

Metrics and signals to monitor

Fixes

Destination is overloaded

Fix the destination, not Fluentd. Scale the Elasticsearch cluster, relieve indexing pressure, or address S3/Kafka throttling. As a temporary measure you can raise buffer headroom (total_limit_size, if disk allows) to buy time, but that only delays overflow if the destination never recovers. For Elasticsearch outputs specifically, reducing the per-request bulk size (smaller chunks via chunk_limit_size) reduces per-flush latency at the cost of more requests.

Too few flush threads

If per-flush latency is inherently high (cross-region writes, large chunks), one flush thread caps throughput at one chunk per latency. Raise flush_thread_count in the output’s <buffer> section. Note the tradeoff: flush threads are Ruby threads under the GVL, so this parallelizes I/O-bound waits well but does nothing for CPU-bound serialization. Extremely high thread counts hit diminishing returns for the same reason.

Chunk size mismatch

Very large chunks make every flush a long write and make the 20s threshold easy to trip. Very small chunks create per-chunk overhead and GC pressure. If your average flush time is dominated by payload size, reduce chunk_limit_size; if it is dominated by fixed per-request overhead, increase it. Check buffer_total_queued_size / buffer_queue_length to see your actual average chunk size before changing anything.

GVL starvation from CPU-bound parsing

If the destination is healthy but Fluentd is pinned on one core, the flush threads are waiting on the GVL while parsing holds it. Simplify heavy regex parsers, move to structured log formats where possible, and consider multi-worker mode (workers N in <system>) for true CPU parallelism. Remember that in_tail must be pinned to a specific worker in multi-worker setups.

Slow buffer disk

For file-backed buffers, high disk latency on the buffer filesystem stalls the flush cycle independently of the destination. Move the buffer directory to faster storage and keep it off the same spindle as the logs being tailed.

Adjusting the threshold itself

You can raise slow_flush_log_threshold for a legitimately slow destination (for example, a time-sliced S3 output where large multipart writes are expected). This quiets the log but changes nothing operationally. Do it only after you understand the baseline flush time, and keep monitoring slow_flush_count and average flush time regardless. Never tune the threshold as the first response.

Prevention

• Alert on the ratio, not the log line. Track delta(slow_flush_count) / delta(write_count) per output and ticket when it stays elevated. The log warning is a symptom; the ratio is the signal.
• Baseline average flush time. flush_time_count / write_count should be a standing dashboard panel per output. Destination degradation shows up here days before it becomes an incident.
• Keep flush time well under flush_interval. Average flush time below 50% of flush_interval is healthy. Above 70%, any load spike tips you into queue growth.
• Size flush threads for latency. If your destination is remote or latency-variable, set flush_thread_count deliberately rather than leaving the default of 1.
• Watch buffer headroom as a composite. Slow flushes plus declining buffer_available_buffer_space_ratios plus rising queue length is the backpressure cascade in progress. Correlate these rather than alerting on any one alone.

How Netdata helps

• Netdata collects the Fluentd monitor_agent output per plugin, so slow_flush_count, write_count, and flush_time_count are graphed together, making the slow-flush ratio and average flush time visible without manual delta math.
• Buffer gauges (buffer_queue_length, buffer_total_queued_size, buffer_available_buffer_space_ratios) are tracked alongside flush latency, so you can see slow flushes translating into queue growth in one view.
• retry_count and rollback_count on the same output let you catch the transition from “slow but succeeding” to “failing and retrying” without log scraping.
• Process-level CPU and RSS for the Fluentd process are collected from the host, which makes the GVL-starvation case (destination healthy, Fluentd pinned on one core) a quick correlation instead of a guess.
• Per-second collection catches short flush-latency spikes that a 60-second scrape interval would average away, which matters when slow flushes are bursty rather than sustained.

Related guides

• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd failed to flush the buffer: the output cannot deliver and retries begin
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss
• Fluentd emit_error_count: the number-one under-monitored data-loss signal
• Fluentd broken pipe / connection reset: dropped output connections and LB timeouts
• Fluentd memory vs file buffer: why the default buffer loses data on restart
• Fluentd input emit_records dropped to zero: ingestion has stopped
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• How Fluentd actually works in production: a mental model for operators