$ guides / fluentd / fluentd-buffer-disk-full ▌

Operations Guides

Fluentd file buffer filling the disk: when the buffer partition runs out

A file-backed buffer is supposed to be the durable option: the output stalls, chunks pile up on disk, the destination recovers, the backlog drains. That story only holds while the partition underneath the buffer directory has free space. When it does not, the failure is abrupt. At 100% full, Fluentd cannot stage new chunks, incoming events are lost, and anything else writing to the same partition fails at the same time.

What makes this incident nasty is the blast radius. Buffer directories frequently live on the same filesystem as journald, syslog, and application logs. When the buffer fills that partition, you do not just lose Fluentd’s backlog. The host’s own logging stops, applications that block on log writes stall, and the evidence you need for the postmortem stops being written.

This guide covers how to confirm the buffer is what filled the disk, why the configured buffer limits did not protect you, and how to size and isolate the buffer so it cannot happen again.

What this means

Fluentd’s file buffer organizes events into chunks on disk with a lifecycle of staged, queued, flushing, then purged or retried. The configured guardrail is total_limit_size per output (default 64 GB for file buffers, with 256 MB chunks). When buffered bytes reach that limit, overflow_action fires: throw_exception (the default, which drops the incoming event), block (input threads wait), or drop_oldest_chunk (oldest chunks are discarded).

The critical gap: total_limit_size is a number in a config file. It knows nothing about actual free space on the partition. Three situations put the disk at 100% before or regardless of that limit:

total_limit_size is set larger than the partition (the 64 GB default on a 20 GB root volume is a classic).
The output is failing, so chunks accumulate until the disk, not the configured limit, becomes the binding constraint.
Other writers share the partition. System logs, application logs, container logs, and the buffer all compete, and the buffer loses its headroom to someone else’s growth.

When the filesystem returns ENOSPC, Fluentd does not crash or block. It logs a warning and discards data. A long-standing behavior (tracked in fluent/fluentd#1698) is a log line like:

unexpected error while checking flushed chunks. ignored. error_class=Errno::ENOSPC

Meanwhile journald, syslog, and any application writing to the same mount start failing too. Note that overflow_action block does not save you here: it gates on total_limit_size, not on real filesystem free space, so it cannot prevent ENOSPC caused by other writers or by a limit set above the partition size.

flowchart TD
  A[Output destination slow or down] --> B[Chunks accumulate on disk]
  C[total_limit_size larger than partition] --> B
  D[System and app logs share the partition] --> E[Free space shrinks independently]
  B --> F[Partition at 100 percent]
  E --> F
  F --> G[ENOSPC: new chunks cannot be staged]
  G --> H[Incoming events discarded]
  F --> I[journald, syslog, app logs fail on same mount]

Common causes

Cause	What it looks like	First thing to check
Output failing, buffer draining slower than it fills	`retry_count` rising, `write_count` flat, buffer directory growing steadily	`df -h` on the buffer mount plus retry/write counters from the monitor agent
`total_limit_size` exceeds real free space	Buffer grows past what the partition can hold; overflow never fires because the configured limit is never reached	Compare the sum of `total_limit_size` across outputs against `df` free space
Shared partition with system or app logs	Disk fills even though buffer usage is well under `total_limit_size`; other files are growing too	`du -sh` on the buffer dir vs. the largest other consumers on the same mount
Orphaned chunk files from old configs	Buffer usage reported by Fluentd is low but the directory is large; stale `.buf`/`.log` chunk files remain after a buffer path or chunk-key change	`find` for chunk files older than any plausible backlog, in current and previous buffer paths
Small root volume with default limits	A 20 GB root disk meets a 64 GB default `total_limit_size`	Check whether `total_limit_size` was ever explicitly set

Quick checks

All read-only. Paths shown use td-agent conventions; fluent-package uses /var/log/fluent/fluentd.log and /etc/fluent/fluentd.conf. Adjust for your install.

# 1. Find the buffer path(s) from the config
grep -E "buffer_path|path" /etc/td-agent/td-agent.conf | grep -v "#"

# 2. Check free space on the partition holding the buffer dir
df -h /var/log/fluent/buffer/

# 3. Measure the buffer directory itself
du -sh /var/log/fluent/buffer/

# 4. See who else is consuming the partition (largest top-level dirs)
du -x -h --max-depth=1 /var/log 2>/dev/null | sort -rh | head -15

# 5. Count and date buffer chunk files (backlog vs. orphans)
find /var/log/fluent/buffer/ \( -name "*.buf" -o -name "*.log" \) | wc -l
find /var/log/fluent/buffer/ -type f -name "*.buf*" -mtime +1 -ls | head

# 6. Look for ENOSPC evidence in Fluentd's own log
grep -i "ENOSPC\|No space left" /var/log/td-agent/td-agent.log | tail -20

# 7. Check what Fluentd thinks the buffer holds vs. what the disk shows
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, total_bytes: .buffer_total_queued_size, avail_pct: .buffer_available_buffer_space_ratios, queue: .buffer_queue_length}'

# 8. Confirm the output is the reason the buffer is not draining
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, retries: .retry_count, writes: .write_count, rollbacks: .rollback_count}'

Check 7 is the interesting comparison. If Fluentd reports modest buffer_total_queued_size but du shows a huge directory, suspect orphaned chunks (check 5). If Fluentd reports the buffer near full and retry_count is climbing, the disk is a symptom and the output is the disease.

How to diagnose it

Confirm the partition is actually full and identify the mount. df -h on the buffer directory. Note the filesystem. If Fluentd is still running, it may be logging ENOSPC warnings; check its own log.
Attribute the space. du -sh the buffer directory, then du the rest of the mount. Three outcomes: the buffer dir dominates (Fluentd’s backlog did it), other directories dominate (someone else filled it and the buffer is collateral), or both are large (shared-partition contention).
Reconcile Fluentd’s view with the filesystem’s view. Compare buffer_total_queued_size from the monitor agent with the du total. A large gap means files on disk that Fluentd is not tracking: orphaned chunks from a previous buffer path, changed chunk keys, or old configs. These are never cleaned up and consume space indefinitely.
Check whether the configured limits could ever have protected the disk. Sum total_limit_size across all outputs using this partition. If the sum exceeds partition size, the limit was decorative. Remember the file-buffer default is 64 GB.
Determine why the buffer grew. If the buffer legitimately filled, the output stalled first. Check retry_count, rollback_count, and whether write_count is incrementing. Then read Fluentd’s error log for the underlying destination failure (connection refused, auth errors, 429s). The disk-full event is the end of the backpressure cascade: destination fails, retries back off, queue grows, buffer fills, disk fills. See Fluentd buffer queue length growing for the earlier stages.
Assess the data loss window. Grep for ENOSPC and BufferOverflowError in the Fluentd log to bracket when discards started. With the default throw_exception overflow action, there is no reliable drop counter; the log is the only record. If overflow_action is drop_oldest_chunk, check drop_oldest_chunk_count instead. See Fluentd BufferOverflowError.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Filesystem free space on the buffer mount	The real limit, regardless of what the config says	Declining trend; below 2x the sum of `total_limit_size`
`buffer_total_queued_size` per output	Absolute buffered bytes, comparable to both the configured limit and disk free space	Growing faster than `write_count` drains it
`buffer_available_buffer_space_ratios`	Percentage of configured buffer remaining; cliff-edge when it hits 0	Below 20% and still falling
`buffer_queue_length` vs. `buffer_stage_length`	Queue growth with stable staging means the output cannot flush	Sustained queue growth
`retry_count` / `rollback_count`	The reason the buffer is filling in the first place	Any sustained non-zero rate
`drop_oldest_chunk_count`	Confirmed data loss if using that overflow action	Any increment
Fluentd log ENOSPC lines	The only direct evidence of disk-full discards	Any occurrence

The important habit: alert on the filesystem, not just on Fluentd’s internal ratio. buffer_available_buffer_space_ratios is measured against total_limit_size, and it will sit comfortably at 80% while the partition dies underneath it. Keep the buffer directory under roughly 60% of partition free space, and the sum of total_limit_size values under about 70% of the partition. For time-to-overflow math on the configured limit, see Fluentd buffer available space low.

Fixes

Restore free space immediately

Options, in order of safety:

Delete orphaned chunk files identified in diagnosis step 3. Files Fluentd is not tracking are pure waste. Verify they are in a path no current config references before deleting.
Fix the output so the buffer drains naturally. This is the real fix when retries are the cause. Once write_count resumes, purged chunks release space on their own.
Free space from non-Fluentd consumers on the same mount: rotate or vacuum oversized system logs, clean old container logs. This buys time without touching Fluentd state.

Deleting chunks Fluentd is tracking is destructive: those events are permanently lost, and deleting files out from under a running Fluentd can produce errors. If you must sacrifice backlog to save the host, stop Fluentd cleanly first, remove the oldest chunk files, then restart. Treat this as a data-loss decision, not a cleanup step. Avoid restarting Fluentd as a first response: with a file buffer, a restart triggers replay of the backlog, which is fine, but it does nothing about a full disk.

Right-size `total_limit_size` against the partition

Set each output’s total_limit_size explicitly, and make the sum across all outputs comfortably smaller than the partition. A workable headroom rule: filesystem free space should exceed 2x the configured limit, and total configured limits should stay under ~70% of the partition. On a small root volume, the 64 GB default is a trap.

Isolate the buffer on its own mount

The structural fix for the cascade is a dedicated partition or volume for buffer directories. When the buffer fills its own filesystem, only the buffer suffers: journald, syslog, and applications keep writing, and the incident stays a logging-pipeline problem instead of a host problem. In Kubernetes, give the buffer dir its own volume rather than sharing the node’s /var/log. Do not put file buffers on remote filesystems (NFS, GlusterFS, HDFS); the Fluentd docs warn this causes major data loss.

Choose `overflow_action` deliberately

None of the options prevent ENOSPC from a shared partition, but they decide what happens when the configured limit is reached. throw_exception (default) drops incoming events with no reliable counter. block pushes backpressure to inputs, which can drop upstream (for example, UDP syslog). drop_oldest_chunk discards old data with a countable metric. Pick per-output based on whether fresh or historical data matters more, and monitor accordingly. See Fluentd drop_oldest_chunk_count incrementing.

Prevention

Explicit limits everywhere. Never run a production file buffer on the default total_limit_size without checking it against the partition.
Dedicated buffer mount. One filesystem for buffers, separate from system and application logs.
Alert on the filesystem, not just Fluentd internals. Page below 10% free on the buffer mount; ticket at 60% buffer-dir-to-free-space ratio.
Time-to-overflow alerting. free_space / buffer_growth_rate during incidents; buffer_available_buffer_space_ratios declining with positive delta(buffer_total_queued_size) as the early warning.
Orphan sweeps after config changes. Any change to buffer path or chunk keys strands old chunk files. Audit buffer directories after config rollouts and after upgrades (v0.12 buffer files are incompatible with v1 and will sit forever).
Watch the output health signals that precede all of this: retry_count, rollback_count, average flush time (flush_time_count / write_count). Disk-full is the last stage of a cascade that starts at the destination. See Fluentd end-to-end pipeline latency and Fluentd broken pipe / connection reset for the upstream causes.

How Netdata helps

Filesystem free space per mount, collected every second, so the buffer partition’s decline is visible long before Fluentd’s internal ratios react. This is the signal Fluentd itself cannot give you.
Fluentd buffer metrics from the monitor agent (buffer_total_queued_size, buffer_queue_length, buffer_available_buffer_space_ratios, retry and rollback counters) charted per output plugin, letting you overlay buffer growth against disk consumption on one timeline.
Correlation of cause and symptom: rising retry_count and flat write_count alongside falling filesystem free space confirms the backpressure cascade in one view, instead of three separate tools.
Growth-rate-based alerting: ML anomaly detection on disk-usage and buffer-size trends catches a steadily filling partition hours before a static threshold fires, which is where time-to-overflow math becomes actionable.
Fluentd’s own log monitored for ENOSPC and BufferOverflowError lines, bracketing the data-loss window for the postmortem.