$ guides / logstash / logstash-config-integrity-changes ▌

Operations Guides

Logstash configuration integrity: detecting unexpected changes to pipeline files

Logstash has no built-in file integrity monitoring. It will load, reload, and run whatever it finds in /etc/logstash/conf.d/, pipelines.yml, and logstash.yml, and it will not tell you that those files changed outside your deployment pipeline. A hand edit at 02:00, a config-management agent fighting your last deploy, or an unauthorized change all look identical to the process: new config, reload attempt, keep running.

The risk is not just that something changed. It is that the change is silent and the effects are subtle. A modified output destination routes data somewhere it should not go. A tweaked conditional drops fields you depend on downstream. A new input opens a listen port nobody approved. Throughput metrics stay green, because the pipeline is working fine. It is just doing the wrong thing.

This guide covers detecting unexpected config changes, correlating them with reload behavior and runtime state, and separating authorized deployments from everything else. It is the security-side counterpart to operational config drift: drift is the running config no longer matching the deployed one (see Logstash configuration drift); integrity is the files on disk changing without authorization.

What this means

Logstash loads .conf files from its pipeline config path (commonly /etc/logstash/conf.d/), plus the agent-level logstash.yml and the multi-pipeline definition file pipelines.yml. With config.reload.automatic: true, Logstash polls the pipeline config files on an interval and reloads when it detects a change. SIGHUP also forces a reload on platforms that support signals.

Three properties of this mechanism create the integrity blind spot:

Logstash does not care who changed the file or why. Any write to a .conf file is a candidate reload.
Reload success is announced only in the log and an API counter. If the new config validates, the old pipeline is replaced quietly. If it fails, the old pipeline keeps running and only reloads.failures increments.
Logstash does not watch referenced files. Grok pattern files, translation dictionaries, and similar plugin-referenced files are only re-read when a config file change triggers a reload. Someone can edit a pattern file, see no immediate effect, and have it picked up by the next unrelated reload.

So the integrity question splits into two: did the files on disk change outside an approved window, and did the running process pick up that change.

flowchart TD
  A[Config file mtime changed] --> B{Change inside approved deploy window?}
  B -->|yes| C[Verify reload succeeded: reloads.successes incremented]
  B -->|no| D[Integrity event: investigate]
  D --> E{Reload log lines present?}
  E -->|yes, success| F[Running config changed: diff against source control, assess routing impact]
  E -->|yes, failure| G[reloads.failures incremented: running config is old config, files are dirty]
  E -->|no reload lines| H[Check pipelines.yml edits and referenced files: not reliably hot-reloaded]
  F --> I[Check for new listen ports and changed output destinations]

Common causes

Cause	What it looks like	First thing to check
Manual hotfix or console edit	Single file mtime outside any deploy window; no CI/CD record	`find /etc/logstash -printf` mtimes against the deploy calendar
Config management loop	Repeated mtimes at fixed intervals; alternating reload successes and failures	`reloads.failures` trend plus mtime pattern frequency
Unauthorized modification	Mtime with no corresponding change ticket; new output destination or listen port	Diff files against source control; `ss -tlnp` for unexpected ports
Partial or failed deploy	Files changed but `reloads.failures` incremented; running config is stale	`pipelines.<name>.reloads` counters and `last_error`
Edit to `pipelines.yml` or referenced files	File changed but no reload occurred; behavior unchanged until restart	Compare file mtimes against reload log lines

Quick checks

All read-only. Run these on the Logstash host.

# List config files by modification time, newest last
find /etc/logstash -maxdepth 2 -type f -printf '%TY-%Tm-%Td %TT %p\n' | sort | tail -n 20

# Find reload and pipeline lifecycle events in the log
grep -Ei '(reload|reloading|pipeline.*started|pipeline.*terminated)' /var/log/logstash/logstash-plain.log | tail -n 100

# Check reload counters and last error per pipeline
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | grep -A 10 reloads

# List the pipeline IDs the JVM actually has loaded (requires jq)
curl -sS http://127.0.0.1:9600/_node/stats/pipelines | jq -r '.pipelines | keys[]'

# Enumerate listen ports owned by the Logstash process
ss -tlnp | grep "$(pgrep -f org.logstash.Logstash)"

Compare the port list against what your deployed configs should open (Beats, TCP, HTTP, syslog inputs). A port you do not recognize is a strong integrity signal.

# Check who can write to the config directory
ls -la /etc/logstash /etc/logstash/conf.d/

World-writable or group-writable config directories convert every local account compromise into a pipeline compromise.

How to diagnose it

Establish the change timeline. Run the find -printf command above and note every mtime newer than your last approved deployment. Include logstash.yml, pipelines.yml, and every file under conf.d/. Do not forget plugin-referenced files (grok patterns, translate dictionaries) if they live outside conf.d/; their mtimes matter even though Logstash does not watch them.
Match mtimes to reload events in the log. Each automatic reload produces log lines around the mtime window. If a file changed and no reload lines exist, either auto-reload is disabled, the change was to pipelines.yml (which is not reliably hot-reloaded; operators report reload failures when adding or removing pipeline definitions at runtime), or the change was to a referenced file that only takes effect on the next reload. Treat any of these as “files and runtime may disagree.”
Check the reload outcome via the API. For each pipeline, look at pipelines.<name>.reloads.successes, reloads.failures, and reloads.last_error. A success counter that incremented at the suspicious mtime means the running pipeline adopted the change. A failure means the old config is still running and you now have both an integrity problem and a drift problem.
Diff the files against source control. git diff or your config-management system’s drift report against /etc/logstash. Categorize every hunk: input changes (new sources, new listen ports), filter changes (dropped or renamed fields, changed conditionals), output changes (new hosts, new indices, new credentials references). Output destination changes are the highest-severity class because they determine where your data goes.
Verify runtime behavior matches the files. Confirm the pipeline list from the API matches pipelines.yml, and the listen ports from ss match the inputs in the configs. A pipeline present in stats but absent from pipelines.yml, or vice versa, indicates a failed reload sequence or manual pipeline manipulation.
Assess blast radius. If a suspicious change was picked up, find the window between the reload timestamp and now, and check what the pipeline did during it: output throughput per pipeline, _grokparsefailure rates downstream, DLQ growth, and event in/out ratios. A malicious or broken filter change often shows up first as a correctness anomaly, not a throughput one.
Identify the writer. Correlate the mtime with auth logs (/var/log/auth.log or equivalent), config-management agent runs, and CI/CD job history. If nothing authorized touched the file in that window, escalate as a security incident, not an ops ticket.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Config file mtimes (`find -printf` vs deploy schedule)	The raw integrity signal; Logstash will not generate it for you	Any mtime outside an approved change window
`pipelines.<name>.reloads.successes` / `reloads.failures`	Tells you whether a detected change was adopted or rejected by the runtime	Failure increment, or a success with no matching deploy
`reloads.last_error`	Captures why a reload failed, confirming a bad file reached the runtime	Non-empty error after an unexpected mtime
Reload/restart log lines	The only narrative record of what the agent did with the new config	Reload activity with no corresponding change record
Pipeline presence in `/_node/stats/pipelines`	Catches pipelines added or removed outside `pipelines.yml` review	Pipeline ID you do not recognize, or a missing expected pipeline
Listen ports owned by the Logstash process	New inputs mean new ingestion surface	A port not present in the reviewed configs
Downstream correctness signals (parse failure tags, DLQ growth, in/out ratio)	A hostile or broken config change often degrades data, not throughput	Correctness anomaly beginning at a reload timestamp

Fixes

Unauthorized or unexplained change confirmed

Treat it as an incident. Snapshot the current state of /etc/logstash (preserve mtimes and contents for forensics) before restoring anything. Then redeploy the known-good config from source control and verify the reload succeeds via the API counters. Rotate any credentials that appeared in the affected config files, and review where data was routed during the exposure window: output destination changes mean data may have left your perimeter.

Config management loop

Symptom: mtimes churn at regular intervals and reloads.failures climbs steadily. Your config-management agent and your deployment pipeline are fighting, or the agent’s rendered template is invalid and every convergence attempt fails validation. Fix the template or disable the agent’s management of Logstash configs; do not just restart Logstash, since the loop will resume. Repeated failed reloads are safe for availability (the old pipeline keeps running) but leave you permanently drifted from what you think is deployed.

`pipelines.yml` changed but runtime did not pick it up

Pipeline definition changes are the weak spot of hot reload: .conf file edits reload cleanly, but adding or removing pipelines in pipelines.yml at runtime is not reliably applied on every release (Logstash re-reads pipelines.yml on each automatic reload cycle, but pipeline-set changes are not a documented reload path; the official reloading-config page covers only path.config files and SIGHUP). If the files and the loaded pipeline set disagree, the reliable path is a controlled restart of Logstash during a low-traffic window. With a persistent queue, in-flight events survive; with a memory queue, queued events are lost, so drain traffic first if possible.

Referenced file changed silently

If a grok pattern file or dictionary changed but no reload has occurred since, the running pipeline is still using the old content, and the next unrelated reload will silently adopt the new content. Force the issue on your terms: validate the referenced file, then trigger a reload deliberately (or restart), and confirm success. Do not leave a pending surprise in the reload path.

Prevention

Version-control every config file and referenced file. conf.d/*.conf, logstash.yml, pipelines.yml, pattern files, and dictionaries. If it is not in git, it is not auditable.
Deploy through one path only. Manual edits on the host should be technically hard (read-only mounts where feasible, tight file permissions) and procedurally forbidden.
Alert on mtime change outside deploy windows. A scheduled check comparing find -printf output against the deployment calendar catches both config-management loops and unauthorized writes. This is cheap and catches the cases API metrics cannot.
Alert on reloads.failures increments. Any non-zero failure not tied to a known deploy is a drift and integrity event. See Logstash config reload failed for the operational side.
Lock down the config directory. Root-owned, not group- or world-writable. Audit write access.
Use the Logstash keystore for secrets. Plaintext credentials in pipeline configs raise the stakes of every config exposure. Reference secrets via the keystore instead.
Log reload events centrally. Ship logstash-plain.log reload lines to your log platform so the reload narrative survives host-level tampering.
Reconcile ports and pipelines periodically. A scheduled comparison of ss -tlnp output and the API pipeline list against the reviewed config catches additions that file-mtime checks miss (for example, config restored from an unauthorized backup).

How Netdata helps

Netdata collects the Logstash Node Stats API, so reloads.successes and reloads.failures per pipeline become time series. A failure increment correlated with a file mtime immediately separates “change adopted” from “change rejected.”
Per-pipeline event counters (events.in, events.out, events.filtered) show what the pipeline did after a suspicious reload timestamp: throughput shifts, ratio drift, or a pipeline that quietly disappeared.
DLQ size and grok failure counters surface the correctness fallout of a bad or hostile filter change, which is usually the first visible symptom.
Process-level metrics (open file descriptors, CPU, uptime) corroborate the timeline: an unexpected uptime reset alongside config mtimes points to a manual restart, not a reload.
Because Netdata samples at per-second granularity and retains history on the host, you can reconstruct the exact window between an unauthorized mtime and detection, which is the window that matters for assessing data exposure.