Fluentd duplicate events: why the same log shows up twice downstream

You query Elasticsearch, S3, or your log backend and the same event appears twice. Sometimes the duplication is a burst that lines up exactly with a Fluentd restart or a pod reschedule. Sometimes it is a slow, persistent trickle that inflates dashboards and breaks counts. The two situations have different root causes and different fixes.

Fluentd does not guarantee exactly-once delivery. It guarantees at-least-once for file-backed buffers, and the brief duplication window after a restart is by design. Persistent or large-scale duplication almost always traces back to one of a small set of mechanisms: position tracking resets at the input, or full-chunk retries at the output after a partial write.

What this means

Every event in Fluentd moves through Input -> Parser -> Filter chain -> Buffer -> Output. Duplicates are introduced at the two ends of that path:

• At the input: in_tail re-reads data it already read because its position tracking (pos_file) was reset, corrupted, deleted, or confused by log rotation and wildcard paths.
• At the output: a chunk that was partially delivered is retried in full, so the destination stores the already-delivered records a second time.

The buffer in the middle is mostly a messenger: file-backed buffers replay unflushed chunks after a restart, which re-sends anything that was flushed to disk but not confirmed delivered. That replay is the at-least-once guarantee working as intended.

The diagnostic question is always: does the duplication correlate with restarts and rotation events (input-side or replay), or does it correlate with output retries and destination instability (output-side)?

flowchart TD
  A[Duplicates seen downstream] --> B{Correlates with Fluentd restart or pod reschedule?}
  B -- yes --> C[File buffer replay of unflushed chunks - expected brief window]
  B -- no --> D{Correlates with logrotate schedule or rotation events?}
  D -- yes --> E[in_tail re-read: pos_file reset, wildcard path without follow_inodes, or copytruncate race]
  D -- no --> F{retry_count or rollback_count incrementing on output?}
  F -- yes --> G[Partial write then full-chunk retry - classic with Elasticsearch bulk API]
  F -- no --> H[Check pos_file integrity, pos_file on tmpfs, read_from_head config]

Common causes

Quick checks

These are read-only and safe to run during an incident. Paths shown are for td-agent; adjust for fluent-package (/var/log/fluent/, /etc/fluent/) or your container layout.

# 1. How long has the process been up? Restart correlation is the fastest triage.
ps -o pid,etime,cmd -p $(pgrep -f fluentd | head -1)

# 2. Are output retries happening? Non-zero retry/rollback points at output-side duplication.
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, retries: .retry_count, rollbacks: .rollback_count, writes: .write_count}'

# 3. Inspect the pos_file: does it exist, where does it live, are offsets sane?
grep pos_file /etc/td-agent/td-agent.conf
ls -la /var/log/td-agent/*.pos 2>/dev/null
df -T /var/log/td-agent/   # tmpfs here means positions die on reboot

# 4. Compare pos_file offsets against actual file sizes.
while IFS=$'\t' read -r filepath pos inode rest; do
  actual=$(stat -c%s "$filepath" 2>/dev/null || echo MISSING)
  echo "$filepath pos=$pos actual=$actual inode=$inode"
done < /path/to/your.pos

# 5. Is rotation happening, and does it line up with duplicates? (v1.14.1+)
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.type=="tail") | {id: .plugin_id, rotated: .rotated_file_count}'

# 6. Check Fluentd's own log for flush failures and retries around the duplicate window.
grep -E "failed to flush|retry|temporarily failed" /var/log/td-agent/td-agent.log | tail -30

For check 4, the pos_file column order is path, position (stored in hex), inode. If positions reset to zero after every restart, the pos_file is not persisting.

How to diagnose it

1. Bound the duplication in time. Pull the timestamps of duplicated events downstream. Clustered in a short window points at restart replay or a rotation event; spread continuously points at ongoing output retries or a persistent input misconfiguration.

2. Correlate with restarts. Compare the duplicate window against process uptime, deploy times, and (in Kubernetes) pod restart events. If every restart produces a duplicate burst whose size roughly matches the pre-restart buffer backlog, you are looking at file buffer replay. That is expected behavior. See Fluentd CrashLoopBackOff if restarts themselves are the problem: frequent restarts turn a small design-window duplication into a constant one.

3. Check whether replay volume is abnormal. Right after a restart with file-backed buffers, a high buffer_queue_length that drains as write_count increments is the backlog being re-sent, and it is healthy. If duplicates vastly exceed the plausible backlog, move on to the input side.

4. Verify pos_file persistence and integrity. If the pos_file is on tmpfs, in a container writable layer wiped on reschedule, or was deleted, in_tail loses all positions. What happens next depends on read_from_head: with true it re-reads everything (massive duplicates); with false on older versions it skips to the end (gaps). Also confirm you are not sharing one pos_file between multiple in_tail configurations, which corrupts position tracking.

5. Check the rotation interaction. If duplicates align with the logrotate schedule, check: (a) does the in_tail path use * or strftime? By default, wildcard paths combined with rotation cause duplication, and the documented fix is follow_inodes true; (b) is rotation using copytruncate? The copy-then-truncate sequence has an inherent race window where in_tail can re-read or miss lines; (c) is rotate_wait (default 5s) long enough for the last writes to be read before the old file is closed?

6. Check the output side for partial-write retries. If duplicates are continuous and the destination is Elasticsearch, look at retry_count and rollback_count on the ES output and at ES bulk rejections. The bulk API can return success for the request while rejecting individual documents (for example during cluster congestion with 429s). Fluentd retries the whole chunk, and the documents that succeeded get indexed again under new generated IDs. The signature: duplicate records with identical content but different _id values.

7. Check versions for known duplication bugs. On Fluentd before v1.12.0, an out_forward ack-reading race at shutdown caused duplicate delivery on restart; fixed in v1.12.0. follow_inodes itself had a duplication bug (wrongly unwatching files) that was fixed in v1.16.2, so enabling it on v1.16.1 or earlier may not fully solve wildcard duplication.

Metrics and signals to monitor

Input emit_records requires enable_input_metrics true in <system> on Fluentd before v1.19.0; without it the counter reads zero. See Fluentd input emit_records stuck at zero.

Fixes

Buffer replay duplicates after restart

There is no config that makes file-buffer replay exactly-once; at-least-once is the design. What you can do:

• Fix the restart frequency. If duplicates are constant because Fluentd restarts constantly, treat the restarts as the incident (OOM, crash loop, config reload failures). The duplication is a symptom.
• Deduplicate downstream. Give records a deterministic document ID at the destination so replays overwrite instead of duplicating. For Elasticsearch, the elasticsearch_genid filter (bundled with fluent-plugin-elasticsearch) generates a unique _hash per record; setting id_key _hash on the output uses it as the document _id, making both restart replays and chunk retries idempotent.
• Accept memory buffers only where loss beats duplication. Memory buffers drop unflushed data on restart instead of replaying it. That trades duplicates for gaps, which is rarely the right trade.

pos_file resets

• Put the pos_file on persistent, reliable storage. In containers, mount a persistent volume for it; never leave it on tmpfs or the container layer.
• One pos_file per in_tail source. Never share.
• On versions with pos_file_compaction_interval (v1.9.2+), enable it for dynamic paths so stale entries do not accumulate.

Wildcard paths and rotation

• Set follow_inodes true when path contains * or strftime patterns and rotation is in play. Be aware of the pre-v1.16.2 bug if you are on an older release.
• Prefer rename/create rotation over copytruncate. If you are stuck with copytruncate, a small duplicate-or-gap window at each rotation is a known limitation.
• Increase rotate_wait beyond the default 5s if the application or rotator writes slowly after the rename.

Elasticsearch partial-success retries

• Add the elasticsearch_genid filter plus id_key _hash as described above. This is the standard fix for bulk partial-success duplication.
• Reduce the retry pressure that causes partial failures in the first place: watch flush_time_count / write_count (average flush time) and slow_flush_count for early signs of ES congestion before rejections start. See Fluentd failed to flush the buffer.

Prevention

• Pin versions deliberately. Run v1.12.0 or later for the forward ack fix, and v1.16.2 or later if you rely on follow_inodes. Test rotation behavior after any Fluentd upgrade; teams routinely verify basic flow and never test what happens when logrotate runs.
• Make idempotency a destination property. Wherever the destination supports it (document IDs in Elasticsearch, object keys in S3), derive the ID deterministically from record content so any retry or replay converges instead of duplicating.
• Alert on retry and rollback counters, not just buffer depth. Retries are the duplication engine on the output side; catching them early bounds the duplicate volume.
• Treat restart count as a first-class metric. Every restart has a duplicate cost. Crash loops and OOM cycling are also duplication incidents.
• Audit pos_file placement in every new deployment, especially Kubernetes DaemonSets where a missing volume mount silently turns every pod reschedule into a full re-read or a gap.

How Netdata helps

• Netdata collects the Fluentd monitor_agent counters per output plugin: retry_count, rollback_count, write_count, and buffer depth, so the output-side duplication signature (retries climbing while duplicates appear) shows up directly on a timeline.
• Process uptime and restart events sit next to pipeline metrics, making the restart-replay correlation a visual check instead of a log dig.
• Input vs output emit_records rates side by side expose re-read spikes (input surge after a position reset) and re-delivery surges (output exceeding input during backlog drain).
• rotated_file_count and in_tail metrics on supported versions let you overlay rotation events with the duplicate timestamps you found downstream.
• Anomaly detection on these counters flags the slow-trickle case (a low, steady duplicate rate from partial-write retries) that threshold alerts tend to miss.

Related guides

• Fluentd broken pipe / connection reset: dropped output connections and LB timeouts
• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd buffer_oldest_timekey lag: how far behind the oldest buffered data is
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss
• Fluentd emit_error_count: the number-one under-monitored data-loss signal
• Fluentd input emit_records stuck at zero: enable_input_metrics on older versions
• Fluentd end-to-end pipeline latency: stale logs during an incident
• Fluentd failed to flush the buffer: the output cannot deliver and retries begin