Logstash file input and sincedb: re-read loops, duplicates, and FD pressure

Your Logstash process looks healthy. The API answers, the pipeline is running, workers are busy. Then someone downstream asks why the same log lines appear three times in Elasticsearch, or why events.in just spiked to ten times its baseline, or why Logstash is logging “too many open files” while traffic looks normal.

All three symptoms usually trace back to the same component: the file input plugin and its position-tracking file, the sincedb. The file input holds one file descriptor per tailed file and records how far it has read in a sincedb file on disk. When that tracking breaks (corruption, inode reuse after rotation, a remounted filesystem changing device numbers, a wildcard that suddenly matches thousands of files) the failure shows up as duplicate events, unexpected events.in spikes, or climbing process.open_file_descriptors.

What this means

The file input tails files by tracking read position per file. Each tracked file is identified by a triple key: inode number, major device number, and minor device number. The sincedb file on disk maps that key to a byte offset. On a clean shutdown, and periodically while running, Logstash flushes its in-memory positions to sincedb so it can resume where it left off after a restart.

Two consequences follow from this design:

1. Identity is positional, not path-based. If the inode behind a path changes (rotation that deletes and recreates, rsync copying a new file over an old one, a SAN/NFS remount changing device numbers), Logstash can no longer reliably connect “the file at this path” to “the position I recorded.” Depending on timing, it either re-reads from the beginning (duplicates) or seeks to a stale offset in a completely different file (lost or garbled data).
2. Every tailed file costs a file descriptor. A path pattern like /var/log/**/*.log that matches thousands of files means thousands of open FDs, until the plugin’s own cap or the OS limit stops it. FD exhaustion is a cliff-edge failure, not a graceful one.

flowchart TD
  A[File input tails files] --> B{sincedb position valid?}
  B -->|yes| C[Resume from recorded offset]
  B -->|corrupt / inode reused / device changed| D[Re-read from start or seek stale offset]
  D --> E[events.in spikes]
  E --> F[Duplicate events downstream]
  A --> G{How many files match path pattern?}
  G -->|thousands| H[One FD per tailed file]
  H --> I[open_file_descriptors climbs toward max]
  I --> J[New opens and connections fail]

The spike in events.in combined with downstream duplicates is the signature of sincedb corruption. If you see that combination, you are almost certainly in this failure mode, not in a backpressure or filter problem.

Common causes

Quick checks

All read-only.

# 1. Pipeline event counters: is events.in spiking beyond what sources should send?
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty

# 2. FD pressure right now
curl -sS http://127.0.0.1:9600/_node/stats/process?pretty
# Compare open_file_descriptors against max_file_descriptors

# 3. FD count from the OS side
PID=$(pgrep -f org.logstash.Logstash)
ls /proc/$PID/fd | wc -l
grep "Max open files" /proc/$PID/limits

# 4. What is the file input actually holding open?
ls -l /proc/$PID/fd | grep -v socket | awk '{print $NF}' | sort | uniq -c | sort -rn | head -30

# 5. Find the sincedb files and inspect them (inode, device major/minor, offset columns)
find /var/lib/logstash \( -name '.sincedb*' -o -name 'sincedb*' \) 2>/dev/null
head -5 $(find /var/lib/logstash -name '.sincedb*' | head -1)

# 6. Count files matched by your configured path patterns
# Adjust the glob to match your input's path setting
ls /var/log/myapp/*.log 2>/dev/null | wc -l

# 7. Look for the FD-limit warning and rotation-time errors
grep -Ei 'open files limit|too many open files|sincedb' /var/log/logstash/logstash-plain.log | tail -n 50

# 8. Check uptime and recent restarts (unclean shutdown loses unflushed sincedb state)
curl -sS http://127.0.0.1:9600/_node/stats/jvm?pretty | grep uptime
journalctl -u logstash --since '24 hours ago' | grep -Ei 'start|stop|kill|oom' | tail -n 20

The sincedb file format is plain text, one line per tracked file: inode, device major, device minor, byte offset, and timestamp columns. If you see the same inode on multiple lines, or offsets that exceed the current file size, tracking state is already inconsistent.

How to diagnose it

1. Confirm the symptom class. Pull /_node/stats/pipelines twice, 60 seconds apart. Compute the events.in rate. Compare it to what the tailed files can actually be producing (check file growth with du or by watching a file’s size). If events.in is far above real file growth, Logstash is re-reading, not tailing new data.
2. Check for the restart correlation. Get jvm.uptime_in_millis. If uptime is short and the spike started at startup, suspect lost sincedb state: an unclean shutdown (OOM kill, kill -9, SIGKILL after container grace period) means positions held in memory were never flushed to disk, so the last chunk of every file is re-read. If the config sets sincedb_path => "/dev/null", every restart re-reads everything by design.
3. Check for the rotation correlation. If duplicates arrive on a schedule (hourly, daily), line the timing up with your rotation tool. Rotation schemes that copy-and-truncate or that delete and recreate files change inodes. Since identity is (inode, device major, device minor), a recreated file is either a brand-new file (read from start_position) or, worse, an inode that was recycled from a file Logstash already tracked (seek to a stale offset in unrelated content).
4. Inspect sincedb on disk. Match sincedb lines against live files: stat -c '%i %D %s' /path/to/file gives inode, device, and size. Note that %D prints the device number in hex while sincedb records major and minor device numbers as separate decimal values, so convert before comparing. A sincedb entry whose inode no longer exists, or whose offset exceeds the file size, is stale. Multiple entries for the same logical file across a remount indicates device-number drift.
5. Quantify FD pressure. Divide open_file_descriptors by max_file_descriptors. Above 80% sustained, you are in the warning zone. Then count how many files your path patterns actually match. If the count is in the thousands, FD exhaustion is structural, not a leak.
6. Rule out the other spike causes. An events.in spike can also come from upstream catch-up after a queue blockage. Check queue.events_count and flow.queue_backpressure: if the queue was full just before the spike, the input was throttled and is now draining buffered data. Sincedb re-reads show up without a preceding full queue.

Metrics and signals to monitor

Fixes

Repair sincedb state

If sincedb is corrupted and causing re-reads, the blunt fix is to stop Logstash, move the sincedb file aside, and start Logstash so it rebuilds positions. Understand the tradeoff: with default start_position => "end", rebuilt tracking starts at the end of each file, so you skip unprocessed backlog but avoid duplicates. With start_position => "beginning", you re-read everything, which is exactly the duplicate storm you are trying to escape. Do not delete sincedb while Logstash is running; the input holds state in memory and will write the stale view back over your fix.

Verify sincedb_path is a real, writable file path on persistent storage, unique per file input. Sharing one sincedb path across multiple inputs corrupts both inputs’ tracking. Setting it to /dev/null disables persistence entirely, which is fine for one-shot imports and wrong for production tailing.

Survive rotation and inode changes

• Prefer rotation schemes that rename rather than delete-and-recreate where you control the log producer. The file input has fixes for copy/truncate and rename cascading schemes in plugin 4.1.x (rotation start-position handling since 4.1.7), but delete-and-recreate with inode reuse remains the hardest case.
• Tune sincedb_clean_after (default 14 days) down toward your rotation cadence. Expiring stale sincedb records protects against inode recycling: a recycled inode whose record has expired is treated as a new file instead of being seeked to a stale offset. For daily rotation, 14 days is far too conservative. Note that plugin versions before 4.1.15 had a duration-conversion bug that made cleanup take vastly longer than configured (fixed in 4.1.15); check your plugin version if cleanup does not seem to happen.
• Be aware that virtualized or network storage (SAN, NFS) can change device minor numbers across remounts, which breaks the identity triple even when the inode is stable. If you tail from such mounts, expect duplicate sincedb entries after remount events and plan cleanup accordingly.
• Prevent unclean shutdowns from losing state. The plugin updates positions in memory and flushes periodically (sincedb_write_interval, default 15 seconds). A kill -9 or OOM kill loses up to that interval of progress per file, which is re-read on restart. Fix the OOM cause (heap sizing, container memory limits) and ensure your service manager sends SIGTERM and waits for shutdown, so sincedb flushes.

Contain FD pressure

• Narrow the path patterns. Match what you actually need to tail. /var/log/**/*.log across a busy host can match thousands of files; per-application patterns with explicit directories rarely do.
• Respect the plugin’s cap. The file input enforces max_open_files (default 4095) and logs “Reached open files limit” with a count of files yet to open. If you see this, the input is not even tailing everything you asked for; some files are silently deferred. Raising the cap alone just shifts pressure to the OS limit.
• Raise OS limits only alongside scope reduction. If you genuinely need to tail thousands of files, raise the systemd LimitNOFILE for the Logstash unit and confirm max_file_descriptors via /_node/stats/process. But FD exhaustion is cliff-edge: also watch close_older (default 1 hour) so inactive files release their FDs instead of being held forever.
• Watch for the hard ceiling at extreme scale. The file input’s codec identity map has a documented capacity of 20,000 identities per input (Elastic’s file-input engineering post), with idle identities evicted after roughly an hour; beyond the cap, new identities are not tracked. Split files across multiple file input stanzas, or move high-cardinality tailing to a dedicated shipper such as Filebeat.

Prevention

• Monitor FD ratio, not just count. Alert on open_file_descriptors / max_file_descriptors above 80% sustained, and trend it weekly. FD problems build slowly and fail suddenly.
• Alert on input-rate anomalies relative to baseline. An events.in spike is as meaningful as a drop. Baseline-relative alerting catches re-read loops that absolute thresholds miss.
• Track event cardinality. For pipelines without clone/split/drop, out should track in. A sustained mismatch after restarts or rotation windows points at re-reads.
• Treat sincedb as state that matters. Put it on persistent, local disk, one path per input, and include sincedb health (stale entries, offset vs file size) in periodic checks.
• Align sincedb_clean_after with rotation cadence and verify your plugin version is past the known duration-conversion and rotation-handling bugs.
• Count your globs in CI or config review. A path pattern is a capacity decision. If a pattern can match more files than your FD budget allows, that is a design defect, not an operational surprise.

How Netdata helps

• Netdata collects the Logstash node stats API signals that matter here: events.in per pipeline, process.open_file_descriptors against max_file_descriptors, and JVM uptime, so a re-read spike shows up as an input-rate anomaly correlated with a recent restart.
• Correlating events.in against host-side log file growth (disk and file activity on the tailed directories) separates real source volume from re-read amplification without manual sampling.
• FD ratio trending over weeks catches the slow climb toward the cliff edge long before “too many open files” appears in logs.
• Per-pipeline views keep a file-input re-read loop visible even in multi-pipeline deployments where aggregate stats would dilute the spike.
• Anomaly detection on events.in flags rotation-time duplicate bursts that recur too infrequently for static thresholds to catch reliably.

Related guides

• How Logstash actually works in production: a mental model for operators
• Logstash flow.queue_backpressure: the input-throttling metric explained
• Logstash memory queue vs persistent queue: durability, visibility, and failure modes
• Logstash monitoring checklist: the signals every production pipeline needs
• Logstash monitoring maturity model: from survival to expert
• Logstash pipeline stalled: output rate at zero while the process looks alive
• Logstash queue events count growing: reading the in-flight backlog
• Logstash queue full: inputs blocked and the backpressure wedge