$ guides / fluentd / fluentd-buffer-available-space-low ▌

Operations Guides

Fluentd buffer available space low: computing time-to-overflow before it fires

The monitor_agent metric buffer_available_buffer_space_ratios tells you what percentage of an output plugin’s configured buffer capacity is still free. When it drops, the buffer is filling. When it hits zero, the overflow_action fires: with the default throw_exception, new events are rejected at the input with a BufferOverflowError; with block, input threads stall; with drop_oldest_chunk, your oldest undelivered data is thrown away. None of those are outcomes you want to discover after the fact.

The trap is that the raw percentage is a bad alerting signal on its own. Twenty percent free on a default 64 GB file buffer is still roughly 12.8 GB of runway, which might be days of headroom or eight minutes, depending entirely on how fast the buffer is growing. A static “alert below 20%” rule pages you for a stable, healthy batch workload and stays silent on a fast-filling buffer that blows through 40% to zero in one bad hour.

The number that actually matters is time-to-overflow: how long until the buffer is full at the current fill rate. This article shows how to compute it from monitor_agent data, which thresholds to act on, and what to do at each level. For the broader buffer health picture, see Fluentd buffer queue length growing and the mental model in How Fluentd actually works in production.

What this means

Every output plugin in Fluentd has a buffer with a configured total_limit_size. The defaults are 512 MB for memory-backed buffers and 64 GB for file-backed buffers. Events accumulate in chunks, which move from staged (filling up) to queued (waiting to flush) to flushed. The monitor_agent reports the fill state per output plugin:

buffer_available_buffer_space_ratios: percentage of total_limit_size still free, 0 to 100. It counts both staged and queued chunks against the limit.
buffer_total_queued_size: total bytes currently held, staged plus queued. The breakdown is available as buffer_stage_byte_size and buffer_queue_byte_size.

Available space in bytes is total_limit_size - buffer_total_queued_size. The ratio field is just that expressed as a percentage.

flowchart TD
  A[buffer_available_buffer_space_ratios low] --> B{Is buffer_total_queued_size still growing?}
  B -- "No: stable level" --> C[Not urgent. Batch backlog or drained burst. Watch, do not page.]
  B -- "Yes: shrinking headroom" --> D[Compute time-to-overflow = available_bytes / growth_rate]
  D --> E{Ratio and runway}
  E -- "< 20% and growing" --> F[Investigate now: find why output is not draining]
  E -- "< 5% and growing" --> G[Overflow minutes away: act before overflow_action fires]
  D --> H{File buffer?}
  H -- "Yes" --> I[Check filesystem free space on buffer mount: it may be the real limit]

One version caveat before you trust any of this: on Fluentd versions before v1.10.0, buffer_available_buffer_space_ratios could only report 0 or 100 due to an integer rounding bug (fixed in v1.10.0). If you are on an older version, ignore the ratio field entirely and compute the percentage yourself from buffer_total_queued_size against your configured total_limit_size.

Why a static percentage alert misleads

Two failure modes:

False positives. A time-sliced output (for example, hourly S3 uploads) legitimately holds large buffers between flushes. The ratio can sit at 15% for fifty minutes and then drain to 95% when the slice expires. A static threshold pages you every cycle.

False negatives. A destination that just went down on a busy aggregator can fill gigabytes per minute. By the time a 20% threshold crosses, you might have ten minutes of runway left, and your paging loop plus escalation time eats most of it.

The fix is to combine the level with the direction and speed: alert on low ratio AND positive growth, and escalate on short computed time-to-overflow.

Quick checks

All of these are read-only.

# Per-output available space ratio, queue, and stage sizes
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") |
      {id: .plugin_id,
       avail_pct: .buffer_available_buffer_space_ratios,
       total_bytes: .buffer_total_queued_size,
       stage_bytes: .buffer_stage_byte_size,
       queue_bytes: .buffer_queue_byte_size,
       queue_chunks: .buffer_queue_length}'

# Is the output actually delivering? write_count should be incrementing.
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") |
      {id: .plugin_id, writes: .write_count, retries: .retry_count, rollbacks: .rollback_count}'

# Retry state: if retry.next_time is far in the future, recovery is slow
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, retry: .retry}'

# For file buffers: filesystem free space on the buffer mount (find buffer_path in your config first)
grep -E "buffer_path|@type file" /etc/td-agent/td-agent.conf
df -h /var/log/td-agent/buffer/   # adjust to your buffer_path
du -sh /var/log/td-agent/buffer/

In multi-worker mode, each worker has its own monitor_agent port (24220 + worker_id) and its own independent buffer. Check each one; aggregates hide a single worker that is about to overflow.

Computing time-to-overflow

Fluentd does not expose a time-to-overflow metric. You compute it externally from two samples of buffer_total_queued_size:

Sample buffer_total_queued_size for the output plugin at time t0.
Sample again after a fixed interval, for example 60 seconds, at t1.
Growth rate = (size_t1 - size_t0) / (t1 - t0) in bytes per second. If the value is zero or negative, the buffer is draining and there is no overflow risk right now.
Available bytes = total_limit_size - size_t1 (or total_limit_size * ratio / 100 if you trust the ratio field on v1.10.0+).
Time-to-overflow = available_bytes / growth_rate.

# Two-sample growth rate and time-to-overflow for one output plugin
PLUGIN="output_plugin_id"          # your plugin_id from plugins.json
LIMIT_BYTES=68719476736            # total_limit_size: 64GB file buffer default; use your configured value

read S0 T0 < <(curl -s http://localhost:24220/api/plugins.json | \
  jq -r --arg p "$PLUGIN" '.plugins[] | select(.plugin_id==$p) | "\(.buffer_total_queued_size) \(now|floor)"')
sleep 60
read S1 T1 < <(curl -s http://localhost:24220/api/plugins.json | \
  jq -r --arg p "$PLUGIN" '.plugins[] | select(.plugin_id==$p) | "\(.buffer_total_queued_size) \(now|floor)"')

RATE=$(( (S1 - S0) / (T1 - T0) ))
AVAIL=$(( LIMIT_BYTES - S1 ))
if [ "$RATE" -le 0 ]; then
  echo "buffer draining or stable (rate=${RATE} B/s), no overflow risk"
else
  echo "growth=${RATE} B/s, available=${AVAIL} bytes, time-to-overflow=$(( AVAIL / RATE )) seconds"
fi

A worked example: file buffer, default 64 GB total_limit_size, ratio at 20%. Available space is about 12.8 GB. Your two samples show buffer_total_queued_size grew by 3.2 GB in 60 seconds, so the growth rate is roughly 53 MB/s. Time-to-overflow is 12.8 GB / 53 MB/s, about 4 minutes. That is a page-now situation even though 20% free sounds comfortable. The same 20% with a growth rate of 1 MB/s gives you over 3.5 hours, which is a ticket during business hours.

Sampling caveats:

Flush cycles make the size sawtooth. A chunk flush between your two samples can make the growth rate look negative even while the destination is slowly failing. Use at least a 60 second window, and prefer a rolling average over several windows for alerting.
Bursty inputs (batch jobs, log storms) produce short-lived growth spikes. Require the growth condition to persist across consecutive windows before escalating.
For file buffers, run the same computation against filesystem free space on the buffer mount: df bytes free divided by the same growth rate. Fluentd’s accounting limit and the disk limit are independent ceilings, and the disk one can hit first, especially if the buffer shares a partition with other data. The smaller of the two runways is your real one.

Thresholds and what to do at each level

Condition	Meaning	Action
Ratio low but stable, `delta(buffer_total_queued_size) <= 0`	Drained burst or time-sliced backlog	Watch only. No page.
Ratio < 20% and growing	Output is not keeping pace; runway is being consumed	Investigate now: check retry_count, write_count, destination health
Ratio < 5% and growing	Overflow is minutes away	Page. Act before `overflow_action` fires
Computed time-to-overflow < 15 min	Same as above, rate-adjusted	Page regardless of the raw percentage
Filesystem runway < buffer runway (file buffers)	Disk will fill before `total_limit_size`	Treat disk exhaustion as the overflow deadline

Keep buffer_available_buffer_space_ratios above roughly 30% during normal operation. If your baseline sits below that, the buffer is undersized for the workload or the output is chronically slow.

Common causes

Cause	What it looks like	First thing to check
Destination down or unreachable	retry_count rising, write_count flat, queue bytes growing	Destination health directly (curl the endpoint), Fluentd error log
Destination slow, not down	Average flush time (flush_time_count / write_count) rising, slow_flush_count incrementing	Average flush time vs flush_interval
Retry storm with long backoff	retry.steps high, retry.next_time far in the future, queue high but static	The retry object in the API; restart resets retry state after the cause is fixed
Buffer undersized for the workload	Ratio chronically low even with healthy output, drops correlate with peak hours	Baseline ratio over a week; compare against total_limit_size defaults
Filesystem filling (file buffer)	df free space falling faster than Fluentd’s accounting suggests	df and du on the buffer path; check what else shares the partition
Silent drops already happening	Ratio pinned near 0, input emit_records stalled or gapped vs downstream	overflow_action config; BufferOverflowError warnings in the Fluentd log

Fixes

Restore the output drain rate

This is the root cause in most incidents. Confirm whether the destination is down (retry_count rising, write_count flat) or slow (average flush time climbing). Fix the destination first; everything else buys time. If retries have backed off so far that retry.next_time is many minutes out, restarting Fluentd after the destination recovers resets retry state and speeds up the drain. Restarting is disruptive: it interrupts in-flight flushes and, with memory buffers, discards everything buffered. Queued chunks in file buffers survive and resume on restart. Do it only after the underlying cause is fixed.

Buy runway

For file buffers, raising total_limit_size only helps if the filesystem has the space. Keep filesystem free space at least 2x the configured total_limit_size so Fluentd’s limit, not ENOSPC, is what fires first. If the disk is the binding constraint, free space or move the buffer path. Do not run file buffers on remote filesystems (NFS and similar); the upstream documentation warns that the file buffer must not be used on remote file systems (NFS, GlusterFS, HDFS, etc.) because major data loss has been observed there.

Choose overflow behavior deliberately

If overflow is genuinely unavoidable, know what your overflow_action does. throw_exception (default) rejects new events with a BufferOverflowError, and there is no reliable built-in counter for how much was lost. block exerts backpressure on inputs, which can stall the whole process and push loss upstream. drop_oldest_chunk loses old data but keeps the pipeline moving and increments drop_oldest_chunk_count (exposed by monitor_agent since v1.19.0). A <secondary> output gives exhausted chunks somewhere to go; write_secondary_count tells you when it engaged.

Reduce intake

If the destination will be down for a while and the buffer cannot hold the gap, shedding load at the input (for example, in_tail group rate limiting, which surfaces as the throttled_log_count in_tail stat in monitor_agent, v1.17.1+) is a controlled loss instead of an uncontrolled one.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`buffer_available_buffer_space_ratios`	Percentage of capacity left	< 20% and shrinking; < 5% is imminent overflow
`buffer_total_queued_size` (delta)	Growth rate for time-to-overflow	Positive slope sustained across windows
`buffer_stage_byte_size` vs `buffer_queue_byte_size`	Stage high is normal batching; queue high is backpressure	Queue bytes growing while stage stays flat
`write_count`	Proof chunks are draining	Flat while queue grows
`retry_count` and `retry.next_time`	Destination failing; backoff state shows real recovery time	Non-zero retries; next_time far out
`drop_oldest_chunk_count`	Confirmed data loss with drop_oldest_chunk	Any increment
Filesystem free space on buffer mount	Real ceiling for file buffers	Falling; below 2x total_limit_size

Prevention

Alert on level plus direction. ratio < 20% AND delta(buffer_total_queued_size) > 0 for a ticket; < 5% AND growing for a page. Never alert on the static percentage alone.
Track time-to-overflow as a first-class metric. Compute it in your monitoring pipeline from successive samples, not by hand during incidents.
Monitor the buffer mount separately. Filesystem free space on the buffer path is an independent overflow clock for file buffers.
Size for the worst outage you will tolerate. If you want the buffer to absorb a 2 hour destination outage at peak ingest rate, total_limit_size (and the disk behind it) must hold 2 hours of peak bytes.
Watch the drain signals, not just the fill signals. write_count, average flush time, and retry state degrade before the ratio moves. See the Fluentd monitoring checklist for the full signal set.

How Netdata helps

Netdata collects Fluentd monitor_agent metrics per output plugin, including buffer_available_buffer_space_ratios, queue and stage byte sizes, so the fill level and its trend are on one dashboard instead of two curl commands.
Growth-rate based alerting is native: you can alert on the derivative of buffer_total_queued_size combined with the ratio level, which is exactly the “low AND shrinking” condition this article describes, instead of a static percentage.
Retry count, write count, and rollback count sit next to the buffer charts, so when the ratio drops you can see in the same view whether the output stalled, is retrying, or is draining slowly.
Host filesystem metrics for the buffer mount are collected alongside the Fluentd metrics, which makes the “disk is the real limit” comparison a visual correlation rather than a separate df check.
Per-second granularity catches the sawtooth flush pattern, so you can tell a draining sawtooth from a monotonic fill that a 60 second sample would hide.