Fluentd read_from_head replay: a memory and CPU spike that re-reads whole files

Fluentd has just started (or restarted) and the process is immediately pinned: CPU at or near 100% of one core, RSS climbing fast, and the pipeline emitting a burst of events with old timestamps. Alerts on process CPU, memory growth, or “events older than X arriving at the destination” are probably firing.

In most cases this is read_from_head true doing exactly what you configured. The in_tail plugin is reading each watched file from byte zero, parsing every historical line, and pushing those events through the filter chain, buffer, and outputs. The spike is real resource consumption, but it is one-time per file per position entry, and it stops when the read catches up.

Two failure modes matter: the replay is large enough to OOM-kill the process before it finishes (common in containers with tight memory limits), and the spike gets misdiagnosed as a leak, a poison pill, or an output problem, burning incident time. This guide covers confirming the replay, ruling out lookalikes, and making it survivable.

What this means

The in_tail input decides where to start reading a file based on two things: the read_from_head parameter and the position file (pos_file).

• With read_from_head false (the default), a newly seen file is tailed from the end. Only new data is read.
• With read_from_head true, a file with no recorded position is read from the beginning. Every line is parsed and emitted.

The replay happens once per file per position entry. The triggers are:

1. First ever startup: the pos_file is empty or missing, so every matched file is “new.”
2. New files appearing under a watched glob (for example /var/log/containers/*.log after pods are scheduled).
3. The pos_file being lost: deleted, on tmpfs, on an ephemeral container filesystem, or corrupted. On the next start every file is “new” again and the entire history replays.

Two version-dependent behaviors change the picture:

• pos_file takes precedence. If a position already exists for a file, read_from_head true does not re-read from the start; reading resumes from the recorded offset. Operators often expect read_from_head true to force a full re-read on every restart and are confused when it does not. That is expected behavior, not a bug.
• v1.14.3 changed the semantics for newly discovered files. From v1.14.3, in_tail reads newly added files from head automatically even when read_from_head is false. The parameter now only controls startup behavior for files already present at launch. On v1.14.3 and later, read_from_head false no longer protects you from head-reads of files that appear after startup.

During the replay, in_tail reads as fast as it can. Every line becomes a parsed event object in the Ruby heap, flows through filters, and lands in buffers. Because CRuby’s GVL serializes CPU-bound work, the parse-and-emit path saturates one core, and RSS climbs with the volume of in-flight events and buffered chunks. If the historical files are enormous, the replay can exhaust the container memory limit before it finishes. The process dies, restarts, and if the pos_file was not persisted or the crash prevented position updates from being written, the replay starts over.

flowchart TD
  A[Startup CPU and RSS spike] --> B{read_from_head true in config?}
  B -- no --> C[Not replay: check poison pill, GC storm, output stall]
  B -- yes --> D{pos_file exists and persisted?}
  D -- no --> E[Full replay of all matched files: expected but expensive]
  D -- yes --> F[Replay limited to files without a position entry]
  E --> G{RSS climbing toward limit?}
  F --> G
  G -- yes --> H[OOM risk: reduce replay size or raise limits before restarting]
  G -- no --> I[Wait it out: spike ends when read catches up]

Common causes

Quick checks

These are read-only and safe to run during the event.

# 1. Confirm read_from_head is set
grep -n "read_from_head" /etc/td-agent/td-agent.conf /etc/fluent/fluentd.conf 2>/dev/null

# 2. Find the pos_file path
grep -n "pos_file" /etc/td-agent/td-agent.conf /etc/fluent/fluentd.conf 2>/dev/null

# 3. Check pos_file freshness and contents
# Line format is: filepath<TAB>position (hex)<TAB>inode
ls -la /var/log/td-agent/*.pos 2>/dev/null
cat /var/log/td-agent/td-agent.pos 2>/dev/null

# 4. Compare recorded positions to actual file sizes.
# Position is stored in hex; a position climbing steadily toward
# the file size means a replay is in progress and healthy.
while IFS=$'\t' read -r filepath pos_hex inode; do
  pos=$((16#$pos_hex))
  actual_size=$(stat -c%s "$filepath" 2>/dev/null || echo "MISSING")
  echo "$filepath pos=$pos actual_size=$actual_size"
done < /var/log/td-agent/td-agent.pos

# 5. Watch RSS and CPU of the Fluentd process
ps -p $(pgrep -f fluentd | head -1) -o pid,rss,%cpu,etime,comm

# 6. Check whether a previous OOM kill already happened
dmesg | grep -i -E "oom|killed process" | tail -5

# 7. Confirm events flowing are historical (input emit rate high)
curl -s http://localhost:24220/api/plugins.json | \
  jq '[.plugins[] | select(.plugin_category=="input") | .emit_records // 0] | add'

# 8. Check buffer pressure from the replay burst
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, queue: .buffer_queue_length, avail_pct: .buffer_available_buffer_space_ratios}'

Note on check 7: on Fluentd versions older than v1.19.0, input emit_records is always 0 unless enable_input_metrics true is set in <system>. If you get zeros, that is an instrumentation gap, not proof the replay is not happening. Also note the pos_file path and filename vary by deployment; adjust checks 3 and 4 to whatever pos_file your config declares.

How to diagnose it

1. Confirm the configuration. Verify read_from_head true exists on an in_tail source and note which paths or globs it watches. If it watches /var/log/containers/*.log on a busy node, the replay volume can be very large.

2. Check whether this is first run or pos_file loss. If the pos_file is missing, empty, or on a filesystem that does not persist (tmpfs, container writable layer without a volume), every restart is a “first run.” This is the difference between a one-time event and a recurring incident. Check your Kubernetes volumes or systemd unit for where the pos_file actually lives.

3. Watch the positions move. Re-run check 4 above a minute apart. Positions advancing steadily toward file sizes mean the replay is progressing normally and will end. A position stuck on one file with CPU pegged suggests something else, for example a pathological line triggering catastrophic regex backtracking (the poison pill pattern).

4. Project the memory runway. Watch RSS growth rate and compare against the container or host limit. If (limit - current RSS) / growth rate is shorter than the estimated time to finish the replay, the process will not survive. Check dmesg for prior OOM kills; a process that already died once during replay is in a loop if the pos_file was not updated before the kill.

5. Check downstream impact. The replay floods outputs with historical events. Watch buffer_queue_length, buffer_available_buffer_space_ratios, and retry_count on the outputs. If the destination (Elasticsearch, S3, a forward receiver) cannot absorb the burst, the replay pushes the buffer toward overflow, turning an expected input-side spike into real backpressure. Old data arriving can also pollute time-based dashboards and trigger stale-data or duplicate-data alarms downstream.

6. Rule out the lookalikes. A real memory leak keeps RSS rising after positions reach end-of-file. A poison pill shows CPU pegged with a stationary position. An output stall shows rising retry_count and a queue that does not drain after the replay burst passes. The distinguishing feature of read_from_head replay is that it is bounded: positions advance, then the spike ends.

Metrics and signals to monitor

Fixes

Reduce or eliminate the replay

• Drop read_from_head true if you do not need historical data. The default (false) tails from the end on startup. You lose the history in existing files but avoid the spike entirely. On v1.14.3 and later, files discovered after startup are still read from head automatically, so this only controls the startup burst for pre-existing files.
• Scope it narrowly. If only one low-volume source needs head-reading, split your <source> blocks so the flag applies only where needed. Multiple in_tail sources where the topmost uses read_from_head true block each other on startup; other plugins cannot start until the head-read finishes.

Make the replay survivable

• skip_refresh_on_startup true avoids the startup blocking behavior: in_tail defers the initial file scan so other plugins can start instead of waiting for large files to be read. This reduces the blast radius of the replay window but does not reduce total work.
• max_line_size skips lines longer than the given size. During a head-read of old files, one unexpectedly huge line can balloon memory or trip BufferChunkOverflowError. Cheap insurance.
• Raise the container memory limit temporarily for the first start, then lower it once positions are recorded. Crude but effective when you cannot avoid one large replay.
• Pre-position the pos_file. If you are deploying to a host with large existing logs and do not need history, start once with read_from_head false so positions are recorded at end-of-file, then enable read_from_head true for future new files.

Fix pos_file persistence

This is the highest-value fix for recurring replays:

• Put the pos_file on a persistent volume in Kubernetes (hostPath or PVC), not the container’s writable layer.
• Never put it on tmpfs unless you genuinely want a full replay on every boot.
• Ensure the Fluentd process can write it; a read-only pos_file means positions never advance durably.

Version hygiene

• If you run v1.12.x, upgrade: the v1.14.3 release notes call out serious in_tail bugs in that series and recommend at least v1.12.4.
• If you use follow_inodes true with read_from_head true and wildcard paths, log duplication on rotation was addressed in v1.16.2; partial duplication during the rotate_wait window can still occur.

Prevention

• Persist the pos_file. Treat it as state, not cache. Losing it converts a one-time cost into a per-restart incident.
• Size for the replay, not the steady state. If read_from_head true is intentional, the startup memory ceiling must absorb parsing the largest matched file plus normal buffer headroom. Measure the replay peak once in staging and set the limit above it with margin.
• Alert on RSS trend, not startup spikes. A high but plateauing RSS is normal for Ruby. A monotonic rise after positions reach end-of-file is the leak signal. Tune alerts to fire on sustained growth over tens of minutes, not on the first minute after a restart.
• Watch positions, not just metrics. The cheapest confirmation that a replay is healthy is positions advancing in the pos_file. Build that check into your runbooks for startup incidents.
• Keep destination headroom for replay bursts. A replay is a synthetic traffic spike at the output. If your Elasticsearch cluster or forward receiver runs hot at steady state, the first restart with read_from_head true will push it into retries. Verify the output can absorb a burst of historical volume.

How Netdata helps

• Per-second process CPU and RSS charts make the replay signature obvious: a bounded single-core CPU plateau and a linear RSS climb that flattens when catch-up completes. That shape distinguishes replay from a leak at a glance.
• Netdata’s Fluentd collector polls the monitor_agent API, so input emit_records rate, output emit_records, buffer_queue_length, buffer_available_buffer_space_ratios, and retry_count sit on the same timeline as the process resource spike. Input rate spike plus stable outputs plus a draining queue confirms a healthy replay; input spike plus rising retries warns the destination is buckling.
• Restart and OOM context sits next to the memory charts, so a CrashLoopBackOff caused by replay-induced OOM kills shows up as repeated short-lived process lifetimes rather than a mystery gap in log flow.
• Anomaly detection on the input rate flags replays you did not schedule, such as a new file under a glob or a pos_file loss after a node replacement. That is how silent pos_file persistence problems surface.

Related guides

• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss
• Fluentd emit_error_count: the number-one under-monitored data-loss signal
• Fluentd input emit_records stuck at zero: enable_input_metrics on older versions
• Fluentd end-to-end pipeline latency: stale logs during an incident
• Fluentd failed to flush the buffer: the output cannot deliver and retries begin
• Fluentd buffer_oldest_timekey lag: how far behind the oldest buffered data is
• Fluentd broken pipe / connection reset: dropped output connections and LB timeouts
• Fluentd config reload failed: SIGHUP that partially applies