$ guides / logstash / logstash-event-duplication ▌

Operations Guides

Logstash event duplication: in/out ratio drift, clones, and re-read sources

Logstash exposes three cumulative event counters at the pipeline level: events.in, events.out, and events.filtered. Their relationship reflects the pipeline’s intended transformation. A passthrough pipeline should see out approximately equal to in over any sustained window. A pipeline with drop {} filters should see out plus filtered approximately equal to in. Clone and split filters legitimately produce more output events than input events.

When this ratio drifts from what the configuration intends, the pipeline is duplicating events, losing events, or both. The worst case is silent duplication: the process is up, throughput looks healthy, and the only evidence is downstream indices containing more documents than the source produced. Clone, split, aggregate, and drop filters all change the ratio by design, so the diagnostic question is not whether the ratio differs from 1:1 but whether it matches what this specific pipeline’s configuration should produce.

What this means

Event duplication manifests as events.out exceeding what the pipeline’s filter logic should produce for a given events.in volume. The divergence may be constant (a filter always cloning or splitting) or sudden (a source re-read spike after a restart or sincedb corruption).

The counters work as follows. events.in counts events after codec processing, before filters. For persistent queues, this counts events written to the queue, not raw network receipt. events.filtered counts events removed from the pipeline, primarily by drop {} filters. events.out counts events successfully emitted by output plugins: sent to the destination, not necessarily acknowledged or persisted.

A pipeline without clone, split, drop, or conditional routing should see out approximately equal to in over any sustained window. Persistent divergence beyond normal in-flight buffering (queue events count plus batch_size times pipeline.workers) means events are being created, lost, or re-sent unexpectedly.

flowchart TD
    A["Unexplained in/out ratio drift"] --> B{"Sudden spike in events.in?"}
    B -->|Yes| C["Source re-ingestion"]
    B -->|No| D{"events.out consistently exceeds events.in?"}
    D -->|Yes| E["Filter fan-out or output resend"]
    D -->|No| F{"Output retries in logs?"}
    C --> G["Check sincedb state and restart history"]
    E --> H["Check clone/split per-plugin stats"]
    F --> I["Check output error and retry patterns"]
    G --> J["sincedb corruption, deletion, or inode change"]
    H --> K["Intentional clone/split or accidental?"]
    I --> L["PQ replay, destination rejects, or retry loop"]

Common causes

Cause	What it looks like	First thing to check
Clone or split filter	`events.out` consistently exceeds `events.in` by a fixed multiplier	Per-plugin `events.out` for clone and split filter instances
Sincedb corruption or deletion	Sudden spike in `events.in` after restart, downstream duplicates	Sincedb file existence, timestamps, and inode references
PQ replay after crash	Burst of `events.out` at startup exceeding recent `events.in`	JVM uptime and recent unclean restart history
Output retry loop	`events.out` rising with retry/error patterns in logs; destination has duplicate documents	Log file for retry, error, reject, 429, 503 patterns
Multiple config files merged	Duplicate writes to same destination; `events.out` roughly 2x expected	Count of `.conf` files in the pipeline config directory
Multiline codec misassembly	Inconsistent event counts; partial or merged events at destination	Whether multiline codec is used with Beats input

Quick checks

# Check pipeline-level event counters for all pipelines
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty \
  | python3 -c "
import sys,json
data = json.load(sys.stdin)
for pname, pdata in data.get('pipelines',{}).items():
    ev = pdata.get('events',{})
    print(f\"{pname}: in={ev.get('in',0)} out={ev.get('out',0)} filtered={ev.get('filtered',0)}\")
"

# Check per-plugin event counters for clone/split filters
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty \
  | python3 -c "
import sys,json
data = json.load(sys.stdin)
for pname, pdata in data.get('pipelines',{}).items():
    for f in pdata.get('plugins',{}).get('filters',[]):
        evts = f.get('events',{})
        name = f.get('name','?')
        if name in ('clone','split','aggregate'):
            print(f\"{pname}/{f.get('id','?')} ({name}): in={evts.get('in',0)} out={evts.get('out',0)}\")
"

# Check for output retry and error patterns in logs
grep -Ei '(retry|error|exception|failed|reject|unavailable|timeout|429|503)' \
  /var/log/logstash/logstash-plain.log | tail -n 200

# Check JVM uptime to correlate duplication with restarts
curl -sS http://127.0.0.1:9600/_node/stats/jvm?pretty | grep uptime_in_millis

# Check how many config files exist in the pipeline directory
find /etc/logstash/conf.d/ -name '*.conf' -type f

# Check sincedb file state for file inputs
ls -la $(grep -r 'sincedb_path' /etc/logstash/conf.d/ 2>/dev/null \
  | head -1 | awk -F'"' '{print $2}')/ 2>/dev/null || echo "sincedb_path not explicitly set"

# Check DLQ size to rule out DLQ-diverted events
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty \
  | grep -A 5 'dead_letter_queue'

# Sample two data points 10 seconds apart to compute live event rates
E1=$(curl -s http://127.0.0.1:9600/_node/stats/pipelines/main \
  | python3 -c "import sys,json; d=json.load(sys.stdin)['pipelines']['main']['events']; print(f\"{d['in']} {d['out']} {d['filtered']}\")")
sleep 10
E2=$(curl -s http://127.0.0.1:9600/_node/stats/pipelines/main \
  | python3 -c "import sys,json; d=json.load(sys.stdin)['pipelines']['main']['events']; print(f\"{d['in']} {d['out']} {d['filtered']}\")")
echo "T0: $E1"
echo "T1: $E2"
echo "Compute deltas to see live ratio"

How to diagnose it

Establish the expected transformation ratio. Read the pipeline configuration. Count clone filters and the entries in each clones array. Identify split filters and what fields they operate on. Note any drop {} conditionals. This ratio is your baseline.
Sample the live ratio. Poll events.in, events.out, and events.filtered twice with a known interval between samples. Compute the delta for each counter. The ratio of delta(out) to delta(in) over the window is the live transformation ratio. Compare it to the expected ratio from step 1.
Classify the divergence. If delta(out) consistently exceeds delta(in) by a factor matching the clone or split configuration, the filter is working as configured and the duplication is intentional. If the factor does not match, or no clone/split filter exists, investigate further.
Check for source re-ingestion if events.in spiked. A sudden increase in events.in that does not correspond to a known upstream volume change suggests the file input is re-reading files from the beginning. Check the sincedb file for the affected pipeline. Look for missing entries, stale inode references, or filenames with whitespace that break sincedb matching.
Check restart history for PQ replay. If the divergence appeared immediately after a restart and the pipeline uses persistent queues, the burst may be PQ replay. On abnormal termination (OOM kill, SIGKILL, container force-kill), all persisted and unacknowledged in-flight events are replayed on restart. This is by-design at-least-once delivery. The burst should be temporary and self-correcting.
Check output error and retry patterns. If the destination (typically Elasticsearch) rejects events and Logstash retries, the same events may be sent multiple times. Look for sustained retry patterns in the log file. Correlate with output plugin error stats from the stats API. An infinite retry loop on certain failure types, such as write blocks on Time Series Data Streams, can resend the same events repeatedly.
Inspect per-plugin stats to localize fan-out. The per-plugin breakdown in /_node/stats/pipelines/<id> shows events.in and events.out for each filter and output plugin instance. A clone filter with three entries in its clones array should show out equal to in times four (original plus three clones). A split filter operating on a field with N elements should show proportional fan-out. If per-plugin numbers do not match expectations, the filter configuration needs review.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`events.in` vs `events.out` ratio	Primary duplication and loss indicator	Sustained deviation from the pipeline’s expected transformation ratio
Per-plugin `events.out` for clone/split	Isolates where fan-out occurs	Plugin `out` exceeds `in` by an unexpected factor
Sincedb file state	Detects file re-read risk for file inputs	Missing entries, stale inode references, or unexpected file deletion
Output retry and error count	Identifies output-driven duplication	Sustained nonzero retry pattern correlated with `events.out` growth
DLQ size and growth	Rules out DLQ diversion as cause of unexpected count changes	Growth when output rejections are active
JVM uptime	Correlates duplication with restart events and PQ replay	Recent unclean restart followed by output burst

Fixes

Clone or split filter fan-out

If the duplication comes from a clone or split filter, verify it is intentional. Check the clones array length in each clone filter. For split filters, identify the field being split and the typical number of elements.

Clone filter behavior changes with ECS compatibility. When ECS is disabled, clones receive a type field. When ECS is enabled (v1 or v8), clones receive a tags array entry instead. Downstream conditionals that check type may break after an ECS migration, causing clones to follow unexpected routing paths.

If a split or clone filter was added accidentally (for example, by a config merge from multiple .conf files in the same pipeline directory), remove it or restructure the config directory so each pipeline has a single, explicit configuration source.

Sincedb corruption causing file re-reads

Sincedb corruption causes the file input to lose its read position and re-process files from the beginning. The result is a spike in events.in and downstream duplicates for every previously processed event.

Common triggers include deleting the sincedb file, inode changes from in-place file modification (the new inode is treated as a new file), and filenames containing whitespace that the file input does not escape correctly when writing sincedb entries.

To prevent recurrence, set an explicit sincedb_path to a persistent, backed-up location. Avoid modifying watched files in-place; rotate them instead. If sincedb corruption has already occurred, you cannot undo the duplicates already written to the destination. Deduplicate downstream or accept the duplicates and fix the sincedb path to prevent the next occurrence.

PQ replay after crash

PQ replay is by-design at-least-once delivery. When Logstash crashes or is force-killed, all unacknowledged in-flight events in the persistent queue are replayed on restart. This produces a burst of events.out that temporarily exceeds events.in.

To make replay idempotent at the destination, use the fingerprint filter to generate a consistent hash from one or more event fields, and set that hash as the document_id in the Elasticsearch output. On replay, the re-sent event overwrites the existing document with the same ID instead of creating a duplicate.

This approach requires a field or combination of fields that uniquely identifies each event. If events lack a natural unique key, the fingerprint approach cannot deduplicate them.

Output retry loops

When the Elasticsearch output retries failed bulk requests, events may be delivered multiple times. This is especially dangerous with write blocks on certain index types, where Logstash may enter an infinite retry loop resending the same events.

Check for sustained retry patterns in the log file. If the destination is rejecting events due to resource pressure (HTTP 429, bulk rejections, thread pool saturation), address the destination capacity first. See Logstash downstream backpressure cascade for the full diagnostic flow.

For idempotent writes, apply the same fingerprint-plus-document_id approach described for PQ replay. Retried events overwrite existing documents rather than creating duplicates.

Multiple config files merged

If multiple .conf files exist under one pipeline’s config directory, Logstash merges them. All filters and outputs from all files apply to all inputs. If two files each define an Elasticsearch output pointing to the same cluster, every event is written twice.

Consolidate the configuration into a single file per pipeline, or use pipelines.yml to assign separate config directories to separate pipelines. Verify the fix by checking that events.out returns to the expected ratio.

Multiline codec with Beats input

The multiline codec should not be used with the Beats input plugin or any input supporting multiple hosts. Doing so can mix streams from different hosts and corrupt event assembly. Configure multiline processing in Filebeat before it sends to Logstash.

If you see inconsistent event counts with partial or merged events at the destination, and the pipeline uses a multiline codec on a Beats input, move the multiline logic upstream to Filebeat.

Prevention

Document the intended transformation ratio for every pipeline. Store it alongside the configuration or in a runbook. Without this baseline, ratio drift is invisible until downstream users report duplicates.
Alert on ratio deviation, not absolute counts. Compare the live delta(out) / delta(in) ratio against the documented expected ratio. A pipeline that legitimately clones 1:4 should not alert at 4x, but it should alert if the ratio suddenly changes to 5x.
Protect sincedb files. Set an explicit sincedb_path on a persistent volume. Include sincedb health in your monitoring for file-input pipelines.
Use fingerprint plus document_id for idempotent Elasticsearch writes. This converts at-least-once delivery into effectively-once delivery at the destination, eliminating duplicates from PQ replay and output retries.
Audit config directories for accidental duplicate outputs. A single misplaced .conf file can silently double-index every event.

How Netdata helps

Per-second event counter collection makes ratio drift visible within seconds. Transient spikes such as PQ replay bursts or sincedb re-read floods show up immediately, not at the next polling interval.
Anomaly detection on event flow rates surfaces ratio changes without requiring you to pre-define the expected multiplier for every pipeline. The learned baseline flags deviations that static thresholds miss.
Correlation between JVM restarts and event counter resets helps distinguish PQ replay bursts (temporary, startup-correlated) from persistent duplication bugs (constant, config-correlated).
Per-plugin breakdown visibility narrows the search from “the pipeline duplicates” to “the clone filter with ID parse_syslog is fanning out 5x instead of 3x.”