Fluentd log rotation loss: copytruncate, rotate_wait, and missed lines

Log rotation is the most common source of quiet data loss in a Fluentd deployment. Pipeline metrics look healthy, the destination keeps receiving data, and yet there is a gap in the log stream at exactly 00:00 every night, or a burst of duplicate records right after logrotate runs. The cause is almost always the interaction between the rotation method used by logrotate (or the container runtime) and the assumptions Fluentd’s in_tail plugin makes about how files change on disk.

The two rotation strategies behave completely differently from in_tail’s perspective. With copytruncate, the file is copied and then truncated in place, which creates an inherent race: lines written between the copy and the truncate are never in the copied file and are erased from the original. With rename/create rotation, the old file is renamed and a new one is created, which is safer, but only if Fluentd finishes reading the old file before it detaches and discovers the new file promptly. This article covers how to tell which failure you have, how to confirm it with the monitor agent and the pos_file, and how to configure rotation and in_tail so the loss stops.

What this means

in_tail tracks each watched file by inode and byte offset, persisted in the pos_file. Under normal operation it reads appended lines and advances the offset. Rotation breaks both of the invariants that make this work:

• rename/create: the path now points to a new inode. Fluentd keeps reading the old file for rotate_wait seconds (default 5s) to catch lines flushed to it just before rotation, then switches to the new file. If rotate_wait is too short, the last lines written to the old file are skipped. If refresh_interval is too long, the new file is discovered late and its early lines arrive late, showing up as a delivery gap right after each rotation.
• copytruncate: the inode stays the same but the file size drops. Fluentd detects the truncation via file size decrease, but there is a window during the copy phase where the application is still writing to the original file. Anything written after Fluentd’s current offset but before the truncate is lost. The truncation can also trigger a re-read of the file, producing duplicates in the destination.

With wildcard paths (the norm in Kubernetes, e.g. /var/log/containers/*.log), there is a third failure mode: without follow_inodes true (available since v1.12.0), the rotated file can be re-discovered as a new match on the glob and re-read from the beginning, producing a large block of duplicates on every rotation.

flowchart TD
  R[logrotate fires] --> M{Rotation method?}
  M -->|copytruncate| C1[copy phase: app still writing]
  C1 --> C2[truncate: lines since last read erased]
  C2 --> L1[Missed lines + possible re-read duplicates]
  M -->|rename/create| N1[old file renamed, new inode at path]
  N1 --> Q{rotate_wait long enough?}
  Q -->|no| L2[tail of old file skipped]
  Q -->|yes| Q2{new file discovered promptly?}
  Q2 -->|no| L3[gap until refresh_interval fires]
  Q2 -->|yes| OK[clean handoff]

Common causes

Quick checks

# 1. Which rotation method is in play for the affected file?
grep -rE "copytruncate|nocreate|create|delaycompress" /etc/logrotate.d/ /etc/logrotate.conf

# 2. Is rotation being detected at all? (v1.14.1+)
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.type=="tail") | {id: .plugin_id, rotated: .rotated_file_count, opened: .opened_file_count, closed: .closed_file_count}'

# 3. Input emit rate for the tail source - drops to zero or spikes at rotation?
#    (requires enable_input_metrics true on Fluentd < v1.19.0)
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.type=="tail") | {id: .plugin_id, emit_records: .emit_records}'

# 4. Compare pos_file state to reality: inode and offset per tracked file
cat /var/log/td-agent/td-agent.pos   # adjust path to your pos_file
while IFS=$'\t' read -r filepath pos inode rest; do
  echo "$filepath pos=$pos pos_inode=$inode cur_inode=$(stat -c %i "$filepath" 2>/dev/null || echo MISSING) size=$(stat -c %s "$filepath" 2>/dev/null)"
done < /var/log/td-agent/td-agent.pos

# 5. What do Fluentd's own logs say around the rotation window?
grep -iE "rotation|following tail|detected" /var/log/td-agent/td-agent.log | tail -30

# 6. Current in_tail rotation-related configuration
grep -A20 "@type tail" /etc/td-agent/td-agent.conf | grep -E "rotate_wait|refresh_interval|follow_inodes|read_from_head|pos_file|path"

# 7. Fluentd version - several in_tail rotation bugs are version-specific
fluentd --version

The pos_file format is <filepath>\t<offset>\t<inode> (older versions may show a different field order; check yours before scripting against it). An inode in the pos_file that does not match the current file’s inode, persisting well beyond rotate_wait, means in_tail has lost the file.

How to diagnose it

1. Pin down the loss window. In the destination, find a gap or duplicate burst and note the timestamps. Compare them against the logrotate schedule (/etc/cron.daily/logrotate timing, or the container runtime’s rotation settings). Rotation loss correlates tightly with the rotation schedule; a gap at a random time points elsewhere.

2. Identify the rotation method. Read the logrotate stanza for the affected file. copytruncate means a race condition you cannot fully eliminate, only shrink. create (rename/create) means a timing problem you can fix with rotate_wait and refresh_interval. nocreate means in_tail has nothing to follow after rotation and collection for that file stops; the official documentation calls this combination out as broken.

3. Classify the symptom: gap or duplicates. A gap at rotation with rename/create points to rotate_wait being too short, or the pos_file being wrong. Duplicates with a wildcard path point to missing follow_inodes true, or to the rotated filename still matching the glob after rotation (a known duplicate-read hazard: once rotate_wait expires and the watcher detaches, the rotated file can be re-discovered by the glob and re-read).

4. Verify detection with rotated_file_count. Sample it before and after the next scheduled rotation. It should increment on the rotation schedule. If it does not, Fluentd never noticed the rotation, which implicates the watcher (inotify vs stat watcher, wildcard handling, or a version bug) rather than the timing knobs.

5. Check the pos_file at the moment of loss. If the pos_file position is ahead of the current file size, or the inode is stale long after rotate_wait, in_tail is reading from the wrong place. Positions reset to zero produce full re-reads (duplicates); positions beyond end of file produce gaps.

6. Check the version against known bugs. v1.13.3 had a regression where in_tail could stop processing after detecting rotation. v1.16.2 and v1.16.3 fixed significant bugs where in_tail wrongly stopped tailing after rotation. If you are on anything before v1.16.2 and see tailing stop after rotation, upgrade first and tune second.

Metrics and signals to monitor

Fixes

If you can change the rotation method: drop copytruncate

copytruncate is inherently racy. Lines written between the copy and the truncate are gone no matter how you tune Fluentd, and the truncation can trigger a re-read. Switch the logrotate stanza to rename/create (create without copytruncate) and let in_tail follow inodes. This is the single highest-leverage fix.

If copytruncate is forced on you (an application holds the file open and cannot reopen it), accept that a small loss window exists and size it honestly: the window is the time between Fluentd’s last read and the truncate. Keeping Fluentd’s read loop fast shrinks the window but never closes it. Do not build compliance-sensitive pipelines on copytruncate.

Tune rename/create rotation properly

For rename/create rotation, two knobs determine whether the handoff is clean:

• rotate_wait (default 5s): how long Fluentd keeps reading the old file after rotation. Raise it until it comfortably exceeds the time the application needs to flush its last writes to the old file. There is deliberately no “wait until EOF” option, because for a streaming file EOF is indistinguishable from a pause. Large files rotated with delaycompress have documented data loss when rotate_wait is too short; the maintainer answer on that issue is exactly “set enough rotate_wait.”
• refresh_interval: how often in_tail scans for files matching the path. If this is long, the new file is discovered late. Keep it short enough that the new file is picked up well within your latency budget.

Wildcard paths: set follow_inodes true

For any path containing * with rotation in play, set follow_inodes true (v1.12.0+). Without it, the rotated file may be re-read from the beginning when the glob re-matches it. Two cautions:

• If your rotated filenames still match the glob (e.g. rotating app.log to app.log.1 while watching app.log*), the rotated file can be re-discovered and re-read after rotate_wait expires even with follow_inodes. Exclude rotated names from the pattern or rotate to a directory outside the glob.
• On versions with pos_file bugs, wildcard plus rotation has produced corrupt pos entries (zero inodes) requiring a restart. Keep versions current.

Protect the pos_file

• Put pos_file on a persistent, reliable filesystem. On tmpfs or an ephemeral container layer, every restart replays or skips everything.
• Never delete the pos_file while Fluentd is running; positions are lost on the next restart.
• If you must recover from a corrupted position, stop Fluentd, fix or remove the pos_file entry for the affected file, and start it. Editing offsets by hand is delicate; the safer recovery is usually to delete the single bad entry so that file alone re-syncs.

Version hygiene

If you are on v1.13.3, or anything before v1.16.2, and rotation misbehaves: upgrade before tuning. Multiple stop-tailing-after-rotation bugs were fixed in v1.16.2/v1.16.3, and v1.14.3 changed behavior so newly discovered files are read from head automatically. Tuning rotate_wait around a watcher bug is wasted effort.

Prevention

• Standardize on rename/create for every file Fluentd tails, and audit logrotate stanzas on a schedule. New services appear with default copytruncate configs constantly.
• Set follow_inodes true and an explicit rotate_wait in every in_tail source, chosen from observed application flush behavior rather than the default.
• Monitor rotated_file_count against the rotation calendar. Rotation that silently stops being detected is the earliest warning of watcher trouble.
• Alert on input emit_records anomalies at rotation windows: a drop to zero or a re-read spike at the same time every day is rotation loss until proven otherwise.
• Keep pos_file on persistent disk and include it in any host or volume provisioning checklist for Fluentd nodes and DaemonSets.
• Test rotation before trusting a new pipeline. Force a rotation (logrotate -f on a test stanza) and verify in the destination that no lines are missing or duplicated. Do this on a test stanza only: logrotate -f forces rotation of every file in the config you point it at, so never run it against the production /etc/logrotate.conf just to test one source.
• Track your Fluentd version against in_tail fixes. Rotation handling is one of the most actively fixed areas of the codebase.

How Netdata helps

• Netdata collects Fluentd monitor agent metrics, so rotated_file_count, opened_file_count, and per-plugin emit_records become time series you can align with the rotation schedule instead of one-off curl samples.
• Correlating input emit_records with output emit_records on one dashboard makes the post-rotation gap or duplicate spike visible as a divergence at a specific minute, not a vague complaint about missing logs.
• Anomaly detection on the input rate catches rotation windows where the rate profile deviates from the host’s own history, which is more reliable than a static threshold for this symptom.
• Process and file-descriptor metrics alongside Fluentd’s plugin metrics help rule out the common lookalike: FD exhaustion stopping new file opens at rotation time.
• Historical retention lets you compare last night’s rotation against the previous 30 to confirm a gap is periodic (rotation) rather than random (network or destination).

Related guides

• Fluentd broken pipe / connection reset: dropped output connections and LB timeouts
• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss
• Fluentd emit_error_count: the number-one under-monitored data-loss signal
• Fluentd input emit_records stuck at zero: enable_input_metrics on older versions
• Fluentd end-to-end pipeline latency: stale logs during an incident
• Fluentd failed to flush the buffer: the output cannot deliver and retries begin
• Fluentd average flush time rising: the earliest sign of destination slowdown