$ guides / fluentd / fluentd-memory-vs-file-buffer ▌

Operations Guides

Fluentd memory vs file buffer: why the default buffer loses data on restart

You restarted Fluentd for a config change, or the OOM killer restarted it for you, or Kubernetes rescheduled the pod. The process came back healthy. Every metric looks normal. But downstream there is a gap in the logs covering the minutes before the restart, and no error anywhere explains it.

The explanation is almost always the same: the output was using the memory buffer, and every chunk that had not been flushed at the moment the process died was deleted with it. This is not a bug. It is the documented behavior of the memory buffer. The Fluentd troubleshooting documentation itself lists “change buffer type from memory to file” as a standard remediation for exactly this symptom.

This article covers how to confirm which buffer type each of your outputs is actually using, why the data is unrecoverable once the process is gone, and how to move to a file buffer with explicit limits.

What this means

Fluentd buffers events in chunks between the filter chain and the output plugin. Chunks move from staged (accumulating events) to queued (ready for flush) to flushed. Where those chunks live depends on the buffer type:

Memory buffer: chunks are Ruby objects in the process heap. Fast, but they exist only as long as the process does. When Fluentd shuts down, buffered logs that cannot be written quickly are deleted. On a SIGKILL, OOM kill, or crash, there is no shutdown path at all: the heap simply ceases to exist.
File buffer: chunks are written to disk as append-only binary files. They survive restarts and are reloaded on startup, then flushed. Delivery becomes at-least-once, so a brief duplication window after a restart is expected.

The trap is the default. The @type parameter in <buffer> is not mandatory. If you omit it, the output plugin may specify its own buffer plugin; otherwise Fluentd falls back to the memory buffer. A config that never mentions buffers can still be running memory buffers everywhere.

flowchart TD
  A[Events staged in buffer chunks] --> B{Process stops: crash, restart, OOM}
  B -->|"@type memory"| C[Chunks exist only in RAM]
  C --> D[All unflushed data lost]
  B -->|"@type file"| E[Chunks written to disk]
  E --> F[Reloaded on startup and flushed]
  F --> G[At-least-once: brief duplicates possible]

One nuance: flush_at_shutdown defaults to true for non-persistent buffers like memory and false for persistent buffers like file. So on a clean, graceful shutdown the memory buffer tries to flush before exit. That helps with planned restarts when the destination is healthy. It does nothing for crashes and OOM kills, and it does not save you when the destination is the reason you restarted, because the flush attempt will fail and the remaining chunks are deleted anyway.

Common causes

Cause	What it looks like	First thing to check
Memory buffer in use (explicit or fallback default)	Log gap in the destination matching the restart window, no Fluentd errors	`with_config=true` on the monitor API, look at each output’s buffer section
Plugin default overrode team intent	Same gap, but the config file says nothing about buffers at all	The running config from the API, not the file on disk
OOM kill during backpressure	Gap plus a process restart nobody initiated	`dmesg` for OOM entries, container restart count
Graceful restart while destination was down	Gap even though shutdown was clean, retry warnings in the log before restart	Fluentd log for retry and flush failures around the shutdown
File buffer configured but path lost	Gap despite `@type file`, usually in Kubernetes	Whether the buffer path is on a persistent volume or ephemeral container storage

Quick checks

All read-only.

# 1. Confirm the running buffer config for every output plugin
curl -s "http://localhost:24220/api/plugins.json?with_config=true" | \
  jq '.plugins[] | select(.plugin_category=="output")'

Look at each output’s buffer section. If there is no @type file, you are on the plugin default, which for a bare output plugin is memory. This shows the running config, which is what matters; the file on disk may differ after a partial reload.

# 2. Check the on-disk config for explicit buffer sections
grep -n -A5 "<buffer" /etc/fluent/fluentd.conf
# td-agent: /etc/td-agent/td-agent.conf

An absent <buffer> section, or one without @type, means the default decided for you.

# 3. Confirm the process actually restarted (and why)
systemctl status fluentd          # fluent-package; td-agent: systemctl status td-agent
dmesg | grep -i oom               # OOM kill evidence
kubectl get pods -l app=fluentd   # Kubernetes: check RESTARTS count

# 4. See whether chunks were waiting when the process died
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, queue: .buffer_queue_length, bytes: .buffer_total_queued_size}'

A queue that was non-trivial before the restart is, roughly, the size of your loss if the buffer was memory-backed.

# 5. If a file buffer exists, check what is on disk
du -sh /var/log/fluent/buffer/    # or wherever your buffer path points

Chunk files present after a restart will be replayed. An empty buffer directory after a restart with a memory buffer is exactly what you would expect.

How to diagnose it

Establish the timeline. Find the restart time (systemd status, pod restart timestamp, Fluentd log start). Find the gap window in the destination. If they align, buffer loss on restart is the working theory.
Determine the restart type. Graceful (deploy, config change) versus hard (OOM kill, crash, node failure, pod eviction). SIGHUP is a reload, not a restart. Hard stops with a memory buffer lose everything queued, no exceptions. Graceful stops only flush what the destination accepts before shutdown.
Verify the effective buffer type from the API, not the config file. ?with_config=true shows what is actually running. This catches the case where the file on disk was edited but a reload only partially applied, or where a plugin’s own default silently chose memory.
Estimate the blast radius. The pre-restart buffer_total_queued_size for that output is the upper bound of lost bytes. If you were not capturing it, use input emit rate times the gap duration as a rough estimate.
Rule out the lookalikes. A gap can also come from overflow_action throw_exception dropping events at a full buffer while Fluentd stayed up, or from a pos_file desync on the input side. The distinguishing feature here is that the gap brackets the restart itself and Fluentd metrics show no overflow or retry anomaly during it.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`buffer_total_queued_size` per output	Bytes at risk if the process dies on a memory buffer	Sustained growth; any high value with `@type memory` is pure exposure
`buffer_queue_length` per output	Chunks waiting to flush; on memory buffer, this is the loss inventory	Non-zero and growing, especially before a planned restart
`buffer_available_buffer_space_ratios`	Proximity to overflow, which compounds restart loss with drop loss	Below 20% and falling
`write_count` rate	Confirms chunks are draining before you restart	Flat while queue is non-zero: do not restart yet
`retry_count` and retry state	Destination failing; restarting into a down destination maximizes loss	Non-zero with `retry.next_time` far in the future
Process RSS and restart count	OOM kills are ungraceful by definition; each one on a memory buffer is a loss event	Any OOM entry in `dmesg`, any unexpected restart
`buffer_oldest_timekey`	Age of oldest buffered data; tells you how far back a loss would reach	Oldest timekey hours behind current time

Fixes

Confirm exposure, then restart deliberately (if you must)

Before any planned restart of a memory-buffered Fluentd, check buffer_queue_length and write_count. If the queue is non-zero, wait for it to drain or fix whatever is blocking the output first. A restart with a stalled output and a full memory buffer is a deliberate data loss event. For ungraceful events (OOM, crash), the data is already gone; the fix is making sure it cannot happen again.

Switch production outputs to the file buffer

Set the type and an explicit size limit per output:

<match **>
  @type forward
  <buffer>
    @type file
    path /var/log/fluent/buffer/forward
    total_limit_size 8GB
  </buffer>
  ...
</match>

Defaults if you do not set them: chunk_limit_size 256MB and total_limit_size 64GB for file buffers, versus 8MB and 512MB for memory. The 64GB default is almost never what you want. Set total_limit_size deliberately, size the filesystem to hold it with headroom, and keep the partition’s free space above roughly 2x the configured limit.

Two constraints from the file buffer documentation that bite in production:

Local disk only. Do not put the buffer path on remote filesystems (NFS, GlusterFS, HDFS). Major data loss has been observed with remote filesystems.
Path characters. On versions before v1.19.3, a path containing [ or ] could cause buffer chunks to be ignored after a restart (they are special characters in Dir.glob, and resume scanned the buffer directory with a glob); fixed in v1.19.3 (fluent/fluentd PR #5305), which resumes buffers with such paths correctly.

In Kubernetes, the buffer path must live on a persistent volume to survive pod replacement. A file buffer on ephemeral container storage is a memory buffer with extra steps: the container dies, the filesystem goes with it.

Handle the version-specific behaviors

v1.16.0 and later: corrupted chunk files found at startup are moved to a backup directory instead of silently deleted. On older versions they were just deleted, so a crash could lose file-buffered data too.
v1.19.0 and later: file buffer plugins evacuate chunk files when retry limits are exceeded instead of discarding the queue; they are written to ${root_dir}/buffer/${plugin_id}/ (root_dir is system_config.root_dir, or the default backup dir /tmp/fluent, overridable via FLUENT_BACKUP_DIR). The memory buffer does not support this.

Accept the tradeoff honestly

File buffers cost disk I/O and disk capacity, and restart replay produces a burst of flushes and possible brief duplicates. That is the price of durability, and the official recommendation for usual workloads is the file buffer for exactly this reason. The legitimate use for memory buffers is high-throughput paths where losing a minute of data on restart is acceptable, decided on purpose, per output.

Prevention

Explicit @type file on every production output. Never rely on the fallback default or a plugin’s own default; write the buffer section deliberately.
Explicit total_limit_size per output. The 64GB file default and 512MB memory default are both wrong for most deployments; size from your actual log rate and acceptable backlog.
Pre-restart queue check in the runbook. No restart proceeds while buffer_queue_length is non-zero or write_count is flat.
Alert on unexpected restarts. Process liveness with sustained-failure gating, plus restart count tracking, because each ungraceful restart is a potential loss event even after you move to file buffers.
Periodic config audit via the API. ?with_config=true in CI or a scheduled check, diffing effective buffer config against intent, catches plugins that default to memory and reloads that partially applied.
Filesystem capacity headroom. Buffer partition free space above 2x total configured limits, monitored, because a full disk converts your durable buffer back into a data loss path via overflow.

How Netdata helps

Netdata’s Fluentd collector polls the monitor agent API and charts buffer_queue_length, buffer_total_queued_size, and buffer_available_buffer_space_ratios per output plugin, so the size of the data at risk on any given restart is visible before you pull the trigger.
Correlating write_count rate against retry_count on the same dashboard tells you whether a restart would land on a draining pipeline or a stalled one.
Process RSS and restart events on the host side, viewed next to buffer queue growth, expose the OOM-kill-on-memory-buffer loop that produces repeated silent gaps.
After switching to file buffers, buffer_oldest_timekey trending back toward current time after a restart confirms the replay drained and the durability fix actually works.