Fluentd buffer queue length growing: the output cannot keep pace with the input

You are looking at a graph of buffer_queue_length for one of your Fluentd outputs and it has been climbing for twenty minutes. No alerts have fired, nothing has crashed, but the trend only goes one direction. This is the earliest visible stage of the most common Fluentd failure mode: the output destination is falling behind the input, and the buffer is absorbing the difference.

The buffer is not the problem. The buffer is the symptom. Every queued chunk is data accepted from your inputs but not yet delivered. If the queue keeps growing, you are on a clock: when the buffer reaches total_limit_size, the configured overflow_action fires, and depending on how it is set you will either block your inputs, drop new events at the input, or start discarding the oldest buffered data.

This guide covers reading the signal correctly, finding why the output is behind, and fixing it before the overflow decision gets made for you.

What this means

Fluentd buffers organize events into chunks. A chunk moves through a lifecycle: staged (filling with incoming events), queued (full and waiting for a flush thread), flushing (being written to the destination), then purged on success or rolled back to the queue on failure.

buffer_queue_length counts the chunks sitting in the queued state. A few things about this number trip people up:

• It is a gauge, not a counter. It goes up and down. Brief non-zero values are normal during every flush cycle.
• It is per output plugin, per worker. In multi-worker mode, each worker has its own independent buffer. An aggregate view can hide one worker drowning while the others are fine.
• It is a count, not a size. Chunks vary in size. buffer_total_queued_size gives you the byte-level view, which is what actually matters against total_limit_size.
• Growth rate matters more than level. A queue of 200 chunks that is draining (write_count incrementing) is a backlog being worked off. A queue of 20 chunks that has not moved in ten minutes (write_count flat) is a stalled output.

flowchart LR
  IN[Input plugins] --> ST[Stage: filling chunk]
  ST --> Q[Queue: buffer_queue_length]
  Q --> FL[Flush threads]
  FL -->|success| OUT[Destination]
  FL -->|failure| RB[Rollback + retry backoff]
  RB --> Q
  Q -->|total_limit_size reached| OV[overflow_action fires]

A growing queue means the loop above is unbalanced: chunks enter the queue faster than flush threads can purge them. Your job is to find which side of that imbalance is broken.

Common causes

Quick checks

All of these are read-only. They assume the monitor agent is enabled on port 24220 (auto-incrementing per worker in multi-worker mode). Paths use the td-agent package layout; adjust for fluent-package or your container image.

# 1. Queue length and stage length per output plugin
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, queue: .buffer_queue_length, stage: .buffer_stage_length}'

High stage with low queue is normal batching. High queue with low or normal stage is backpressure. That distinction is the first fork in the diagnosis.

# 2. Is the queue draining or stuck? Sample write_count twice, 60s apart
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, writes: .write_count}'
sleep 60
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, writes: .write_count}'

Incrementing means the output is alive but behind. Flat means it is fully stalled. These demand different responses.

# 3. Retry state, not just the counter
# (v1.19.3+ ignores the ?with_retry query parameter; set include_retry true
#  in the monitor_agent source instead)
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, retries: .retry_count, retry: .retry}'

retry.steps and retry.next_time tell you how deep the backoff hole is. A retry.next_time 20 minutes out means the pipeline is effectively dead even though it is “retrying”.

# 4. How much room is left
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, avail_pct: .buffer_available_buffer_space_ratios, total_bytes: .buffer_total_queued_size}'

Below 20% available and still growing: investigate now. Below 5%: overflow is imminent.

# 5. Input vs output record rates (sample twice and compare deltas, not raw counters)
curl -s http://localhost:24220/api/plugins.json | \
  jq '{input: ([.plugins[] | select(.plugin_category=="input") | .emit_records // 0] | add),
       output: ([.plugins[] | select(.plugin_category=="output") | .emit_records // 0] | add)}'

On Fluentd older than v1.19.0, input emit_records requires enable_input_metrics true in <system>; without it the input counter is always 0.

# 6. What the logs say about why
grep -E "failed to flush|retry|temporarily failed|could not connect|broken pipe" \
  /var/log/td-agent/td-agent.log | tail -20

# 7. Auth and TLS failures specifically
grep -iE "(tls|ssl|auth|401|403|unauthorized|forbidden|certificate)" \
  /var/log/td-agent/td-agent.log | tail -20

# 8. Slow flushes (flush_time_total_ms is cumulative; use deltas for average flush time)
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, slow: .slow_flush_count, flush_time_total_ms: .flush_time_count, writes: .write_count}'

How to diagnose it

1. Confirm the signal. Is buffer_queue_length actually growing over 10 to 15 minutes, or oscillating? Oscillation around a flush cycle is normal. Time-sliced outputs (daily S3 files, for example) hold chunks until the slice expires and create legitimate large queues that drain on schedule. Sustained one-direction growth is the real signal.

2. Split draining from stuck. Run check 2. If write_count is incrementing, the destination works but cannot keep up: go to step 4. If write_count is flat, the output is stalled: go to step 3.

3. For a stalled output, find the failure reason. Check retry_count and the retry object (check 3), then the logs (checks 6 and 7). The pattern is usually one of: connection refused (destination down), 401/403 or TLS errors (credentials or certificates), or nothing at all with retry.next_time far in the future (backoff exhaustion). Verify destination health independently: curl the endpoint from the Fluentd host. Do not trust Fluentd’s view alone; DNS, firewall, and proxy changes all sit between them.

4. For a draining-but-losing output, quantify the deficit. Compare input and output emit rates over a 15-minute window (check 5). Compute average flush time as delta(flush_time_count) / delta(write_count). If average flush time is approaching flush_interval, the output is at capacity: each chunk takes almost as long to write as the interval between flushes, so any burst accumulates.

5. Check whether Fluentd itself is the bottleneck. Look at per-core CPU. A single worker pegged at 100% of one core with a healthy destination is GVL starvation: parsing and filtering are starving the flush threads. pgrep -f fluentd lists the worker processes; ps -T -p <busy_pid> -o spid,%cpu,comm shows whether one thread is doing all the work. The fix here is parser simplification or multi-worker mode, not destination work.

6. Estimate time to overflow. Compute runway as (total_limit_size - buffer_total_queued_size) / growth_rate. For file-backed buffers, also check filesystem free space on the buffer directory: the disk, not total_limit_size, may be the real limit, and a full buffer partition can break more than Fluentd if the filesystem is shared.

7. Check per-worker. In multi-worker mode, repeat the checks against each worker’s monitor agent port (24220, 24221, …). One worker behind on a hot tag is invisible in any aggregate.

Metrics and signals to monitor

Alert on sustained growth, not a static queue-level threshold. Batch and time-sliced workloads legitimately produce large temporary queues. “Queue growing for 15 minutes and write_count not keeping pace” catches real incidents; “queue > 100” pages you every night at the batch window.

Fixes

Destination is down or failing

Fix the destination first; nothing on the Fluentd side substitutes for that. While it is down, your lever is runway: verify buffer headroom and time-to-overflow (step 6 above), and decide consciously what happens at the limit rather than discovering it. If the destination is failing on auth or TLS, rotate the credentials or certificates in the Fluentd config and reload. If backoff has exhausted (retry.next_time far out), restarting Fluentd resets retry state. Do this only once the destination is healthy again, and only with file-backed buffers: a restart with memory buffers loses every unflushed chunk.

Destination is slow, not down

The cheapest lever is flush concurrency: raise flush_thread_count so multiple chunks flush in parallel. Network writes release the Ruby GVL, so I/O-bound flush threads genuinely parallelize within one worker. If flush latency itself is rising (check 8), the problem is on the destination side: Elasticsearch under indexing pressure, S3 throttling, an overloaded Kafka broker. Fluentd tuning cannot fix a saturated destination; it can only buy buffer time.

Input exceeds output capacity structurally

If input rate has permanently outgrown what one output path can flush, options are raising flush_thread_count, moving to multi-worker mode (each worker gets independent buffers and event loops, and it is the only way to get true CPU parallelism), or scaling the destination. Note that in_tail does not support multi-worker and must be pinned to a specific worker, so plan the topology change rather than flipping workers N blindly.

Fluentd is CPU-bound (GVL starvation)

Simplify parsing: replace complex regex with structured formats where the source allows it. Increase chunk sizes so fewer, larger chunks reduce per-chunk overhead and Ruby object churn. Move to multi-worker mode to use more cores.

Choosing overflow_action deliberately

This is a policy decision, not an incident fix, but the incident is when you find out what you chose:

• throw_exception (default): new events are rejected at the input when the buffer is full. Some inputs log the drop, but no metric tracks the loss, so it is easy to miss. Worst option for auditability.
• block: inputs stall and backpressure propagates upstream. Protects Fluentd’s data but can drop data further up (UDP syslog at the kernel, for example).
• drop_oldest_chunk: bounded loss of the oldest data, tracked by drop_oldest_chunk_count. Often the right default for metrics-like logs, wrong for audit or security logs.

Prevention

• Alert on growth, not level. Sustained positive slope on buffer_queue_length over 10 to 15 minutes, confirmed by write_count lagging, is the alert that fires early enough to act.
• Watch average flush time. flush_time_count / write_count rising is the earliest destination-degradation signal, appearing before retries start.
• Keep headroom. Target buffer_available_buffer_space_ratios above 30% in normal operation, and keep filesystem free space on the buffer partition well above the configured total_limit_size.
• Use file-backed buffers in production. Memory buffers lose everything unflushed on crash or OOM. File buffers survive restart and give you disk-sized runway.
• Size flush threads to latency. Average flush time should stay under half of flush_interval; if it does not, add threads before you add buffer.
• Enable input metrics. On versions before v1.19.0, set enable_input_metrics true in <system> so the input/output rate comparison is possible at all.
• Test the overflow path. Know what your overflow_action does and verify the corresponding counter is monitored before an incident demonstrates it for you.

How Netdata helps

• Netdata collects the Fluentd monitor agent metrics per output plugin, so buffer_queue_length, buffer_stage_length, and buffer_total_queued_size are visible as time series rather than point-in-time curl samples.
• Stage versus queue is charted separately, which makes the healthy-batching versus backpressure distinction immediate instead of something you reconstruct by hand.
• Because write_count, retry_count, rollback_count, and flush_time_count are collected alongside the buffer gauges, you can correlate a climbing queue with flat writes or climbing retries on one dashboard and jump straight to the stalled-versus-draining fork.
• Anomaly detection on the queue length trend flags sustained abnormal growth without a static threshold, which avoids paging on legitimate batch windows.
• Per-worker breakdowns in multi-worker deployments keep one drowning worker from hiding inside a healthy aggregate.

Related guides

• Fluentd monitoring checklist: the signals every production log pipeline needs
• How Fluentd actually works in production: a mental model for operators
• Fluentd monitoring maturity model: from survival to expert
• Fluentd process not running: the log pipeline is dead and the host has gone dark
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• Fluentd poison pill crash loop: one bad log line that kills the process on every restart