Logstash configuration drift: when the running config no longer matches the deployed one

You pushed a config fix to /etc/logstash/conf.d/ an hour ago. The pipeline is running, throughput is normal, no alert fired. But the fix never took effect: the reload failed validation, Logstash kept the old pipeline running, and nothing in your standard monitoring noticed. The running config and the deployed config are now two different things.

This is Logstash configuration drift, and it is dangerous because it is invisible by default. Logstash has no built-in drift detection: there is no API endpoint that returns the currently active config text or a hash of it. The reload machinery is deliberately safe (a failed reload keeps the old pipeline alive rather than dropping it), but that safety creates a silent gap between what you think is deployed and what is actually processing your events.

Worse, drift does not require a failed reload. A syntactically valid config that is logically wrong reloads successfully and reports success: a conditional that routes events to the wrong output, a grok pattern that no longer matches the source format. Throughput stays flat. Every health check stays green. The pipeline is working fine, it is just doing the wrong thing.

What this means

When config.reload.automatic is enabled, Logstash polls the config files (default interval 3 seconds), and on detecting a change it stops the current pipeline’s inputs, validates the new config, verifies that plugins can initialize, and swaps pipelines. If validation fails, the old pipeline continues running and the error is logged. Two drift paths follow:

1. Failed reload drift. The file on disk changed, the reload failed, and the old pipeline keeps running. The config management system reports the file as deployed. Operators believe the fix is live. It is not. No alarm fires unless someone is watching reloads.failures.
2. Successful-but-wrong drift. The new config validates and the reload succeeds, but the logic is wrong: a conditional routes events to the wrong output, a drop rule catches legitimate traffic, a grok pattern tags everything with _grokparsefailure. Throughput metrics look identical. This is the silent correctness failure pattern from the playbook: process up, throughput normal, data wrong.

A third, subtler case: Logstash does not watch files referenced by inputs, filters, or outputs. A grok pattern file, a translate filter dictionary, or a GeoIP database only reloads when the main config file itself triggers a reload. Editing the pattern file alone changes nothing until something touches the pipeline config.

flowchart TD
  A[Config change deployed to disk] --> B{Reload attempt}
  B -->|validation fails| C[Old pipeline keeps running]
  B -->|valid, reload succeeds| D{Logic correct?}
  C --> E[Drift: running config is stale]
  D -->|no| F[Drift: valid but wrong behavior]
  D -->|yes| G[Converged]
  E --> H[reloads.failures +1, last_error set]
  F --> I[No reload signal fires, detect via correctness signals]

How drift happens in practice

Quick checks

All read-only. The API defaults to port 9600 on localhost.

# Per-pipeline reload state: the primary drift signal
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | grep -A 10 reloads

Look at successes, failures, last_success_timestamp, last_failure_timestamp, and last_error for each pipeline. Any failure count you have not seen before is drift until proven otherwise.

# Config file modification times, newest last
find /etc/logstash -maxdepth 2 -type f -printf '%TY-%Tm-%Td %TT %p\n' | sort | tail

Compare the newest mtime against your deployment pipeline’s change window and against last_success_timestamp. A config file newer than the last successful reload is the textbook drift signature.

# Reload activity in the Logstash log
grep -Ei '(reload|reloading|pipeline.*started|pipeline.*terminated)' /var/log/logstash/logstash-plain.log | tail -n 100

The log carries the actual validation error that last_error may truncate, and shows reload attempts the counters aggregate away.

# Which pipelines exist and are running
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | grep -E '"[a-zA-Z-]+":'
# Logstash 8.x structured health view:
curl -sS http://127.0.0.1:9600/_health_report?pretty

A missing pipeline after a deploy is drift of the worst kind: in edge cases a failed reload can leave the old pipeline stopped and the new one never started.

# Correctness spot-check for successful-but-wrong reloads
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty
# Compare events.in vs events.out vs events.filtered per pipeline,
# and check plugins.filters[].failures for grok filters

An in/out ratio that no longer matches the pipeline’s intended transformation ratio after a reload is evidence the new logic is dropping, cloning, or misrouting events.

How to diagnose it

1. Establish the timeline. Get the config file mtime from the find command above, the deployment timestamp from your config management system, and last_success_timestamp / last_failure_timestamp from the stats API. Lay them in order. If mtime is newer than last success and a failure timestamp sits between them, you have failed-reload drift.

2. Read the actual error. Pull reloads.last_error from the API and the surrounding log lines. Typical causes: syntax errors, missing plugins, missing files or credentials, permission problems, or plugin options incompatible with the installed version.

3. Check whether the failure timestamp is lying to you. There is a long-standing known bug (Logstash issue #6149, still open): if a reload fails and you then revert the config file to exactly what was previously running, Logstash detects no change and does not reload. last_success_timestamp is never updated, so the failure timestamp stays newer than the success timestamp even though the pipeline is now running the correct config. Naive alerting on “last_failure newer than last_success” produces false positives. Reconcile against file mtime and content, not timestamps alone.

4. If the reload succeeded, verify behavior, not the counter. Compare the pipeline’s event in/out/filtered ratio against its intended transformation ratio. Check the grok filter failures counter for an upward step at the reload time. Query the destination for _grokparsefailure / _jsonparsefailure tag rates. A successful reload with a rising parse failure rate or a shifted in/out ratio means the new config is valid but wrong.

5. Check referenced files. If the change was to a grok patterns file, a translate dictionary, or another plugin-referenced file, confirm a reload actually fired after that file changed. If only the referenced file was edited, touch or redeploy the pipeline config to force the reload, then confirm via reloads.successes.

6. Reconcile and redeploy. Fix the config (or roll back), let the reload fire, and confirm: reloads.successes incremented, last_success_timestamp is newer than the file mtime, and correctness signals (parse failures, in/out ratio) return to baseline. Do not consider drift closed on a green deploy pipeline alone.

One instrumentation gotcha while diagnosing: pipeline stats counters reset on reload, which creates artificial zero-dips in throughput graphs. A dip at the reload timestamp is the reset, not an outage. Confirm with output rate before and after the window.

Signals to monitor

Alerting guidance: any reload failure is a TICKET during business hours. It does not page on its own because the old pipeline keeps serving, but it must never age unnoticed, because nothing else will tell you.

Fixing and preventing drift

Fix the immediate state. Correct the config error from last_error, redeploy, and confirm the reload counters and timestamps as in step 6 above. If the pipeline is in a genuinely bad state that reload cannot recover from (rare, but possible with plugin initialization problems), a Logstash restart will load the on-disk config cleanly. Warning: with a persistent queue, queued events survive a restart; with a memory queue, in-flight events are lost. Treat restart as the last resort, not the first fix.

Validate before deploy. Run config.test_and_exit in CI so syntactically broken configs never reach a host. This kills the most common failed-reload drift at the source. It does not catch logically-wrong-but-valid configs; those need the correctness signals above.

Alert on reload failures, not just deploy success. The most common operational mistake here is seeing the reload counter increment, assuming success, and never checking that the reload actually succeeded. reloads.failures is the simplest check and almost never monitored. Add it.

Build a reconciliation loop. Expert-level (maturity level 4) practice is explicit drift detection: periodically compare config file mtimes and content hashes under /etc/logstash against the source-controlled versions and against last_success_timestamp. Logstash will not do this for you; a small scheduled job or your config management’s audit mode will. Elastic’s own guidance for eliminating drift is GitOps-style pipeline management, where config only reaches hosts through version control and CI, which also gives you the change window to correlate mtimes against.

Watch correctness after every config change. Parse failure tags, grok failures, in/out ratio, and DLQ growth are the only signals that catch a successful-but-wrong reload. Make “check correctness signals for N minutes after any config deploy” a standard part of the deploy runbook.

Remember what reload does not cover. Referenced files (pattern files, dictionaries) do not trigger reloads on their own. A reload does not restart the JVM, so plugin-level static state can persist across reloads. And per-pipeline config.reload.* overrides in pipelines.yml are silently ignored: reload is an agent-level function, so set it at the agent level only.

How Netdata helps

• Netdata’s Logstash collector scrapes the monitoring API continuously, so per-pipeline reloads.successes and reloads.failures are graphed over time rather than polled ad hoc during incidents. A new failure shows up as a step in the chart, correlated with everything else.
• Because reload failures and throughput share a timeline, you can see the key correlation directly: reload failure followed by a throughput or parse-failure change means the old config is handling traffic it was not meant to handle.
• Per-pipeline event in/out/filtered rates make successful-but-wrong reloads visible as ratio shifts at the reload moment, which aggregate metrics would hide in multi-pipeline setups.
• Grok filter failure counters and DLQ size sit on the same dashboards as reload state, so the correctness half of drift detection does not require a separate tool.
• Alerting on any increase in reload failures, plus anomalies on output throughput and parse failures, covers both drift paths without hand-built polling scripts.

Related guides

• Logstash config reload failed: reloads.failures and invisible configuration drift
• How Logstash actually works in production: a mental model for operators
• Logstash monitoring checklist: the signals every production pipeline needs
• Logstash monitoring maturity model: from survival to expert
• Logstash memory queue vs persistent queue: durability, visibility, and failure modes
• Logstash flow.queue_backpressure: the input-throttling metric explained
• Logstash persistent queue not draining: page-release lag after downstream recovery
• Logstash file input and sincedb: re-read loops, duplicates, and FD pressure