Fluentd pos_file corruption: duplicates and gaps after a restart

You restarted Fluentd (a deploy, an OOM kill, a pod reschedule) and now one of two things is wrong downstream: the same log lines appear twice, or a window of logs never arrived. Both symptoms point at the same component: the in_tail position file.

The pos_file is how in_tail remembers where it stopped reading. It is a plain text file with one line per tailed file, recording the file path, a byte offset in hexadecimal, and an inode number in hexadecimal. There are no checksums and no integrity metadata. On startup, Fluentd reads it and trusts it completely.

If the pos_file is missing, stale, corrupt, or was sitting on tmpfs, the restart degrades to one of two bad outcomes: re-reading from offset 0 (duplicates) or resuming from an offset past the current end of file (a permanent gap). This guide walks through confirming the symptom, isolating which failure you have, recovering with the least collateral damage, and preventing recurrence.

What this means

While running, in_tail tracks a byte offset per watched file and persists offsets to the pos_file. On startup, for each pos_file line, Fluentd checks whether the recorded inode still matches the file on disk. If it does, reading resumes at the recorded offset. Everything that can go wrong after a restart is a corruption of one of those three fields, or the absence of the line entirely:

• Pos_file missing (tmpfs, deleted, fresh container filesystem): there is no entry for the file. With read_from_head true, the whole file is re-read: duplicates of every line still on disk, plus a CPU and memory spike while the backlog is parsed. With read_from_head false, reading starts at the current end of file: every line written before startup is skipped, which is a gap.
• Corrupt or stale offset: an offset of 0 forces a full re-read (duplicates). An offset beyond the current end of file means the lines between the real end and the recorded offset are never read (gap).
• Inode mismatch: the recorded inode no longer matches the current file, which means rotation handling failed. The current file is treated as untracked or the wrong file is followed.

One distinction before you start: a brief burst of duplicate deliveries right after a restart is normal with file-backed buffers, because unflushed chunks are replayed. Buffer replay duplicates are bounded to the chunks that had not been flushed. Pos_file-driven duplicates cover entire files and keep arriving until the re-read catches up. The volume and duration of the duplicate window tells you which mechanism you are looking at.

flowchart TD
  A[Fluentd restarts] --> B{pos_file intact?}
  B -- yes --> C{inode matches file on disk?}
  C -- yes --> D[Resume at recorded offset: clean]
  C -- no --> E[Rotation follow failed: skip or re-read]
  B -- no --> F{what is recorded or missing}
  F --> G[No entry or offset 0: full re-read, duplicates]
  F --> H[Offset past current EOF: lines skipped, gap]

Common causes

The in_tail documentation explicitly warns against sharing one pos_file between in_tail configurations: concurrent updates from two plugins corrupt the content.

Quick checks

All of these are read-only.

Find where the pos_file lives. Paths vary by package: td-agent uses /etc/td-agent/, fluent-package uses /etc/fluent/, Kubernetes deployments usually mount config from a ConfigMap.

# Locate pos_file directives
grep -rn "pos_file" /etc/td-agent/ /etc/fluent/ 2>/dev/null

Check whether the pos_file is on durable storage. If the filesystem type is tmpfs, the file is lost on every reboot; in a container, the default writable layer is lost on every reschedule.

# Identify the filesystem backing the pos_file
findmnt -T /var/log/td-agent/td-agent.pos
df -T /var/log/td-agent/td-agent.pos

Inspect the file itself. A healthy pos_file has one clean line per tailed file and an mtime that advances while Fluentd runs. Truncated lines, merged hex fields, or lines referencing files that no longer exist are all signs of trouble.

# Check freshness and contents
ls -la /var/log/td-agent/td-agent.pos
cat /var/log/td-agent/td-agent.pos

Compare each pos_file entry against the file it references. The recorded inode should match the live inode, and the recorded offset should sit near (slightly behind) the current file size.

# Show live state for every file referenced in the pos_file
awk '{print $1}' /var/log/td-agent/td-agent.pos | while read -r f; do
  if [ -f "$f" ]; then
    stat -c 'inode=%i size=%s %n' "$f"
  else
    echo "MISSING $f"
  fi
done

The offset is stored in hexadecimal. Convert it to decimal before comparing with the size from stat:

# Convert a recorded hex offset to decimal bytes
printf '%d\n' 0x0000000000014bfa

Check the input-side counters on the monitor agent (default port 24220). A post-restart spike in input emit_records confirms a re-read; a drop to zero while source files keep growing confirms skipped files.

# Total input emit_records (cumulative counter, derive the rate)
curl -s http://localhost:24220/api/plugins.json | \
  jq '[.plugins[] | select(.plugin_category=="input") | .emit_records // 0] | add'

On Fluentd versions before v1.19.0, input emit_records is always 0 unless enable_input_metrics true is set in <system>. If you get zeros everywhere, check that first.

Check the in_tail-specific counters. rotated_file_count (v1.14.1+) tells you rotation detection is firing; tracked_file_count (v1.19.0+) tells you how many files are currently watched.

# in_tail tracking and rotation counters
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.type=="tail") | {id: .plugin_id, tracked: .tracked_file_count, rotated: .rotated_file_count}'

If in_tail is pinned to a specific worker with <worker N> (it does not support multi-worker), query that worker’s monitor_agent port; worker 0 is 24220, worker 1 is 24221, and so on.

Establish when the process actually started, and whether it was killed rather than stopped cleanly. An OOM kill or SIGKILL means the pos_file was last flushed some time before death, so offsets are stale by that window.

# Process start time and recent OOM kills
ps -o lstart= -p $(pgrep -f fluentd | head -1)
dmesg | grep -i oom | tail -5

How to diagnose it

1. Confirm the symptom at the destination. Query your log store for a known line from the affected host around the restart. Duplicates mean a re-read; a missing time window means skipped data. Do not trust Fluentd-side metrics alone for this.
2. Fix the timeline. Get the restart time from ps or the pod’s restart count, and the kill cause from dmesg or the Fluentd log. An unclean kill implies stale offsets; a clean stop implies the pos_file should have been flushed on shutdown.
3. Check whether the pos_file survived. Existence, mtime, and the filesystem it sits on. If it is on tmpfs, an emptyDir, or the container’s writable layer, the diagnosis ends here: positions were lost, and the restart behavior is governed by read_from_head.
4. For each affected file, compare entry to reality. Inode in the pos_file versus stat -c %i, recorded offset versus current size. Offset at 0 with a large live file means full re-read. Offset beyond the live size means a gap. Inode mismatch means the rotation follow failed.
5. Confirm with input emit_records. A sharp spike starting at the restart time is the re-read in progress. A flat line while stat shows the source file growing means the file is not being read at all.
6. Correlate with rotated_file_count. If the symptom window lines up with a rotation event and the rotation counter did not increment as expected, rotation handling, not the restart itself, is the primary cause.
7. Choose a recovery path based on whether you are containing duplicates (usually tolerable, dedup downstream) or recovering a gap (only possible if the skipped bytes still exist in a file Fluentd can be pointed at).

Metrics and signals to monitor

No API metric directly exposes pos_file integrity; this is a known blind spot. The host-level checks above are the only way to cover it, which is why mature setups include pos_file freshness and inode validation as scripted checks.

Fixes

Recover a lost or corrupt pos_file

Warning: the blunt recovery, deleting the pos_file and restarting, re-reads or skips every tailed file on the host. With read_from_head true, every file is re-read in full: massive duplicates and a startup CPU and memory spike proportional to on-disk log volume. With read_from_head false, reading starts at the current end of each file: everything written before startup is skipped. Since v1.14.3 (PR #3542, also stated in the in_tail documentation), files discovered after startup are read from head regardless; read_from_head false only changes startup behavior, which is exactly the case you are in.

The surgical option is usually better. Stop Fluentd first, because a running in_tail periodically rewrites the pos_file and will clobber your edit. Then edit the hex offset on the affected file’s line: set it to 0 to force a full re-read of that one file (accepting duplicates of just that file), or to a known-good byte offset you computed from the file’s current content. Restart Fluentd and watch input emit_records for the expected step.

For gaps: if the skipped bytes are still in the live file, the same edit recovers them by moving the offset backwards. If the file was rotated away, the data only exists in the archived rotated files, and in_tail will not touch them unless your path pattern covers them. In that case, recovery means re-ingesting the archived files through a separate path or accepting the gap.

Move the pos_file to durable storage

This is the fix for the most common root cause. The pos_file must survive reboots, container restarts, and pod reschedules. On hosts, put it on a persistent local filesystem, not tmpfs. In Kubernetes DaemonSets, mount a hostPath volume or a PVC for the pos_file directory; an emptyDir is lost when the pod leaves the node, and a memory-backed emptyDir is tmpfs. Until this is fixed, every restart reproduces the incident.

Fix rotation follow failures

• Prefer rename/create rotation over copytruncate. copytruncate has an inherent race: lines written between the copy and the truncate can be missed or read twice.
• Set follow_inodes true when using wildcard paths. Available since v1.12.0, this tracks files by inode across rotation instead of by path, which is what prevents re-reads when the path pattern matches both the old and new file. Residual duplication risk in the rotate_wait window has been documented even after later fixes, so verify at the destination after rotations.
• Tune rotate_wait (default 5s) and refresh_interval. Versions before v1.16.3 could silently stop tailing a file when rotate_wait exceeded refresh_interval; v1.16.2 and v1.16.3 fixed the watcher-stall bugs for both follow_inodes modes. If you are on an older release, upgrading is the real fix.
• Kubernetes note: container log paths are symlinks, and the pos_file tracks the symlink target’s inode, which changes on rotation. An inode mismatch after rotation on a node is expected to resolve via follow_inodes; if it persists, the watcher is stuck.

Bound pos_file growth and avoid known bugs

When tailing many files with dynamic paths, the pos_file grows until restart because entries for unwatched files are only cleaned at startup. pos_file_compaction_interval (v1.9.2+) periodically removes unwatched, unparsable, and duplicated lines. All supported releases write the current line format (path, offset, inode as hex fields), but if you inherit an ancient td-agent pos_file, inspect it for malformed lines such as merged or truncated hex fields and compact or recreate the file. There is also an open issue (fluent/fluentd #4843, still open as of v1.19.x) where limit_recently_modified can cause an all-ones hex offset to be recorded for unwatched files, forcing a full re-read from head when the file is re-watched; if you use that parameter and see periodic duplicate bursts, this is a candidate cause.

Prevention

• Durable pos_file storage. Positions must survive reboots and pod reschedules, or every restart is a duplicates-or-gaps incident.
• One pos_file per in_tail source. Sharing corrupts the file and produces erratic resume positions.
• Rotation method audit. Use rename/create where possible, follow_inodes true with wildcards, and rotate_wait long enough for your rotation tooling.
• Version floor. Run v1.16.3 or later to carry the in_tail watcher fixes; add pos_file_compaction_interval if you tail dynamic paths.
• Pos_file integrity checks. Scripted host-level checks for mtime freshness and inode match catch the failure mode no API metric exposes.
• Post-restart input rate alerting. An alert on input emit_records deviation from baseline turns every future occurrence into a detection instead of a downstream surprise.

How Netdata helps

• Netdata collects the monitor_agent counters (emit_records, rotated_file_count, tracked_file_count) at per-second granularity, so the restart-time step change is visible instead of averaged away.
• Correlating the input emit_records spike with process restarts and OOM kills on the same host separates a pos_file re-read storm from a genuine application log storm.
• Comparing input and output emit rates over a rolling window quantifies how much data was duplicated or lost, which decides whether downstream dedup or gap recovery is worth doing.
• Tracking rotated_file_count against the expected rotation schedule catches follow failures before they compound across rotations.
• Process uptime, OOM events, and file descriptor usage sit next to the Fluentd metrics in one view, so the kill-cause-to-symptom link takes minutes instead of a log archaeology session.

Related guides

• Fluentd broken pipe / connection reset: dropped output connections and LB timeouts
• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd buffer_oldest_timekey lag: how far behind the oldest buffered data is
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss
• Fluentd emit_error_count: the number-one under-monitored data-loss signal
• Fluentd input emit_records stuck at zero: enable_input_metrics on older versions
• Fluentd end-to-end pipeline latency: stale logs during an incident
• Fluentd failed to flush the buffer: the output cannot deliver and retries begin