Fluentd buffer stage vs queue: telling healthy batching from backpressure

A common mistake in Fluentd operations is treating “the buffer” as a single number. Teams graph one buffer metric, see it climb, and either panic over normal batching or ignore a real backpressure signal because “the buffer number looks like it always does.” The buffer is not one thing. It is two distinct states with two distinct gauges, and they mean opposite things.

buffer_stage_length counts chunks still being filled with incoming events. That is normal batching. buffer_queue_length counts chunks that are full and waiting to be flushed to the destination. A queue that stays deep is backpressure: the output cannot keep up. Both are gauges, reported separately per output plugin in the monitor_agent API, and the relationship between them tells you more about pipeline health than either does alone.

What stage and queue actually are

Every event in Fluentd passes through a buffer on its way to an output plugin. Buffers organize events into chunks: containers that accumulate events until a flush trigger fires (size limit, time interval, or record count depending on configuration). Each chunk moves through a lifecycle:

Staged: the chunk is open and accumulating events. This is batching working as designed.
Queued: the chunk is closed and waiting for a flush thread to pick it up.
Flushing: a flush thread is actively writing the chunk to the destination.
Purged or retried: the chunk was delivered, or the write failed and it goes back for retry.

The two gauges map to the first two states:

Gauge	What it counts	What a high value means
`buffer_stage_length`	Chunks still being filled	Input is arriving and batching up. Usually healthy.
`buffer_queue_length`	Chunks waiting to flush	Output is behind. Sustained depth is backpressure.

The same split exists in bytes: buffer_stage_byte_size and buffer_queue_byte_size, with buffer_total_queued_size covering the total across both. The byte breakdown matters because chunk counts alone hide size variation. Queue bytes growing while stage bytes stay stable is the same backpressure signal at higher resolution.

One version note: the stage-side fields (buffer_stage_length, buffer_stage_byte_size, buffer_queue_byte_size) were added to the monitor_agent output in Fluentd v1.6.0. On older versions only buffer_queue_length and buffer_total_queued_size exist, so the stage/queue split is not observable.

How the chunk lifecycle produces the two gauges

flowchart LR
  A[Input events] --> B[Staged chunk
buffer_stage_length]
  B -->|chunk full or flush trigger| C[Queued chunk
buffer_queue_length]
  C -->|flush thread picks up| D[Flushing]
  D -->|success| E[Purged]
  D -->|failure| C
  E --> F[Destination]

Two things in this loop explain almost every reading you will see:

Chunks move from stage to queue continuously during normal operation. A briefly non-zero queue is healthy; it just means a flush cycle is in progress.
The arrow from “Flushing” back to “Queued” is the failure path. When the destination rejects a write, the chunk returns to the queue and the queue grows even though input and staging look normal.

A high queue length that is draining (write_count incrementing) is a different situation from a high queue length that is stuck (write_count flat). Always read the queue alongside the write rate.

Reading the combinations

The rule of thumb: monitor buffer_stage_length and buffer_queue_length separately, and treat their ratio as the diagnostic.

Stage	Queue	Reading	What to do
High	Low	Healthy batching. Input is flowing, chunks are being filled and flushed promptly.	Nothing.
Low	High	Backpressure. Chunks close, enter the queue, and sit there because the output is failing or slow.	Investigate the destination. Check `retry_count`, `write_count`, and average flush time.
High	High	Both sides under pressure. Input volume is high and the output cannot drain it.	Investigate the destination and check headroom: `buffer_available_buffer_space_ratios`.
Low	Low	Quiet pipeline, or time-sliced output between flushes.	Verify input is actually producing events.

A useful threshold: when buffer_queue_length exceeds buffer_stage_length by a factor of 5 and stays there, the output is struggling. The absolute numbers matter less than the shape. There is no universal magic number for queue length, because the danger level depends on your configured limits (total_limit_size relative to chunk_limit_size) and on whether the queue is growing or draining.

Where this shows up in production

Normal flush cycles. Queue briefly goes non-zero, then returns to zero as chunks flush. Alerting on “queue > 0” will page you for healthy behavior; alert on sustained growth instead.

Restart with file-backed buffers. On startup, Fluentd replays unflushed chunks from disk. A large queue right after restart is the backlog draining, not a new problem. Give it a few minutes before reacting, and confirm the queue is trending down with write_count incrementing.

Time-sliced outputs. Outputs that chunk by time window (daily S3 files, for example) hold chunks until the slice expires. Stage and queue can sit elevated for long, legitimate periods, and input/output rates diverge by design. For these outputs, raw rate comparisons mislead; buffer_oldest_timekey is the safer freshness check.

Destination failure. The classic backpressure cascade: retry_count and rollback_count rise, write_count stalls, chunks roll back into the queue, and buffer_queue_length grows while stage stays low or normal. Input rate is unaffected until the buffer fills. What happens at the limit depends on overflow_action: throw_exception (the default) raises BufferOverflowError back at the input, which for most input plugins means new events are dropped; block stalls the input threads; drop_oldest_chunk discards the oldest data.

Output slower than input. Same shape as above but without errors: retry_count stays zero, average flush time (flush_time_count / write_count) creeps up, and the queue grows steadily. Rising flush latency is the earliest indicator of destination degradation; it appears before any retries.

Checking it by hand

All of these are read-only and safe to run against the monitor_agent API (default port 24220; in multi-worker mode each worker gets its own port, 24220 + worker ID).

# Stage vs queue per output plugin
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, stage: .buffer_stage_length, queue: .buffer_queue_length}'

# Byte-level split: stage, queue, and total
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, total_bytes: .buffer_total_queued_size, stage_bytes: .buffer_stage_byte_size, queue_bytes: .buffer_queue_byte_size}'

# Is the queue draining? write_count should be incrementing; rollbacks mean failures
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, writes: .write_count}'

# How close to the limit: percentage of buffer space still available
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, avail_pct: .buffer_available_buffer_space_ratios}'

Take two samples a minute apart. A single snapshot cannot tell you whether a deep queue is draining or stuck; the delta in write_count and in buffer_queue_length between samples is the answer.

One caveat: buffer_total_queued_size has been reported to behave counterintuitively in some versions (appearing to grow without decreasing). If your total-bytes graph looks monotonic, prefer buffer_queue_byte_size for queue monitoring.

Signals to watch in production

Signal	Why it matters	Warning sign
`buffer_queue_length`	Depth of the flush backlog	Sustained upward trend, not a static value
`buffer_stage_length`	Normal batching volume	Only meaningful relative to queue; high alone is fine
Queue vs stage ratio	Separates batching from backpressure	Queue greater than 5x stage, sustained
`buffer_queue_byte_size`	Queue depth in bytes, immune to chunk-size variation	Growing while stage bytes are flat
`buffer_available_buffer_space_ratios`	Distance to overflow and its action	Below 20% and actively filling; below 5% is imminent overflow
`write_count` (rate)	Confirms whether the queue is draining	Flat while queue grows
`retry_count` / `rollback_count`	Confirms destination failures behind the queue	Any sustained non-zero rate
`flush_time_count / write_count`	Average flush latency, earliest destination-degradation signal	Rising above 2x baseline, or approaching `flush_interval`
`buffer_oldest_timekey`	Age of the oldest undelivered data	Exceeds roughly 2x `flush_interval` for non-time-sliced outputs

Alert on sustained queue growth and on space-ratio decline combined with positive growth rate, not on static queue thresholds. Legitimate bursts and time-sliced outputs create temporary deep queues that drain on their own.

How Netdata helps

Netdata collects the Fluentd monitor_agent metrics as separate dimensions, so buffer_stage_length and buffer_queue_length stay visible as independent gauges per output plugin instead of collapsing into one “buffer usage” number.
Plotting stage and queue together per output makes the healthy-batching pattern (high stage, low queue) visually distinct from backpressure (queue climbing while stage stays flat).
Correlating queue depth with write_count and rollback_count on the same dashboard answers the first diagnostic question instantly: is this queue draining or stuck.
buffer_available_buffer_space_ratios tracked over time gives a time-to-overflow estimate, which is what actually decides urgency when the queue is deep.
In multi-worker deployments, per-worker breakdown keeps one struggling worker from hiding inside healthy aggregate numbers, since each worker’s buffers are independent.