Logstash config reload failed: reloads.failures and invisible configuration drift

You pushed a config change. The reload counter incremented, the process never restarted, and nothing paged. Two days later someone notices the new filter logic is not in the data. The deploy “worked” in every system that tracks deploys, but Logstash is still running the old pipeline.

With config.reload.automatic enabled, Logstash polls config files, validates changes, and hot-swaps the pipeline. When validation fails, it keeps the old pipeline running. That is the safe choice, but it means the config on disk and the config in memory diverge with no process-level symptom. The only evidence lives in three counters almost nobody graphs: reloads.failures, reloads.last_error, and reloads.successes.

The trap is watching for “a reload happened” rather than “a reload succeeded.” A config management run that touches the file looks like progress. It is not.

What this means

Logstash’s automatic reload is a validation gate, not an apply operation. The sequence is:

1. Config file changes on disk.
2. Logstash detects the change and validates the new config (syntax, plugin availability, option compatibility).
3. On success, the old pipeline is torn down and the new one starts. reloads.successes increments.
4. On failure, the error is logged, reloads.failures increments, reloads.last_error captures the error text, and the old pipeline keeps running untouched.

Step 4 is the drift. From the outside, everything looks healthy: the process is up, the API returns 200, throughput is normal because the old pipeline still works. Meanwhile your source-controlled config says one thing and the running pipeline does another. Every subsequent change builds on a config that was never applied, so when someone finally restarts Logstash, months of “deployed but not running” changes activate at once, or the process fails to start on a config nobody remembers writing.

flowchart TD
    A[Config file changes on disk] --> B[Logstash validates new config]
    B -->|valid| C[Old pipeline stops, new starts]
    C --> D[reloads.successes increments]
    B -->|invalid| E[Old pipeline keeps running]
    E --> F[reloads.failures increments, last_error set]
    F --> G[Deployed config != running config]
    G --> H[Silent drift: throughput normal, no page]

A failed reload where the old pipeline keeps running is the common case, but there is a worse variant: the pipeline disappears from the stats API entirely because the old pipeline stopped and the new one failed to start. Check for pipeline presence, not just the reload counters.

Common causes

The config management loop deserves emphasis: if Ansible or Puppet keeps pushing the same bad config, Logstash attempts and fails a reload on every poll cycle. That generates log noise, masks other errors, and buries the original failure under hundreds of identical retries.

Quick checks

All read-only. The monitoring API defaults to port 9600 on localhost.

# 1. Reload state per pipeline: the core check
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | grep -A 10 reloads

# 2. Full reload block including last_error and timestamps
curl -sS http://127.0.0.1:9600/_node/stats/pipelines/main | python3 -c "
import sys,json
r = json.load(sys.stdin)['pipelines']['main']['reloads']
print('successes:', r.get('successes'))
print('failures:', r.get('failures'))
print('last_error:', r.get('last_error'))
print('last_success:', r.get('last_success_timestamp'))
print('last_failure:', r.get('last_failure_timestamp'))
"

# 3. Confirm the expected pipelines still exist (top-level keys under "pipelines")
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | grep -E '"[a-zA-Z_-]+":'

# 4. Reload-related log lines with the actual error detail
grep -Ei '(reload|reloading|pipeline.*started|pipeline.*terminated)' /var/log/logstash/logstash-plain.log | tail -n 100

# 5. Which config files changed recently, and when
find /etc/logstash -maxdepth 2 -type f -printf '%TY-%Tm-%Td %TT %p\n' | sort | tail

# 6. Verify reload is actually enabled (agent level, not pipeline level)
grep -E 'config\.reload' /etc/logstash/logstash.yml
ps aux | grep '[o]rg.logstash.Logstash' | grep -o 'config.reload.automatic'

Check 6 matters more than it looks. config.reload.* settings are agent-level. Putting them inside a pipeline definition in pipelines.yml is silently ignored, so a team can believe automatic reload is on when it is not. Likewise, SIGHUP only triggers a reload if config.reload.automatic was enabled at startup. If your manual reload signal “does nothing,” that is why.

How to diagnose it

1. Establish whether a failure actually occurred. Compare reloads.failures against its last known value. Any new increment you did not already know about is drift. If you have never recorded a baseline, treat the current count as unknown and investigate the last failure timestamp.

2. Read reloads.last_error. This is the fastest path to the cause: the validation error text for the most recent failure. Cross-reference with the log file, which carries the full detail, including which config file and line caused it.

3. Verify reloads.successes advanced after your last good deploy. This is the step people skip. The reload counter incrementing means a reload was attempted. Only reloads.successes advancing proves the new config is live. If you deployed at 14:00 and last_success_timestamp predates that, you are running the old config.

4. Check the failure rate, not just the count. Take two samples a minute apart. If failures is climbing on every reload interval, you have a config management loop pushing a bad config repeatedly. Stop the config management run before fixing the config, or your fix gets overwritten.

5. Confirm the pipeline is still present. A failed reload normally leaves the old pipeline running, but if a pipeline is missing from /_node/stats/pipelines after a reload attempt, the old pipeline stopped and the new one never started. That is an outage, not drift. In Logstash 8.x, GET /_health_report gives a structured per-pipeline health view.

6. Correlate with downstream symptoms. If the failed config was supposed to fix something (a parse failure, a routing change), that symptom will still be present. Parse failures continuing after “the fix deployed” is often the first real-world hint that the reload never applied.

One known timestamp trap: there is a long-standing upstream issue where reverting to the exact previously-working config after a failure does not update last_success_timestamp. The failure timestamp stays newer than the success timestamp even though the pipeline is running valid config again. Do not use last_failure_timestamp > last_success_timestamp as your sole health check. Prefer: last_error is null (or unchanged) AND reloads.successes advanced since your last known-good deploy. This revert behavior is community-reported and no upstream issue tracks it; the Health Report API (Logstash 8.16 and later) gives per-pipeline state but does not expose config validity directly, so the timestamp caveat stands.

Metrics and signals to monitor

Fixes

Fix the config, then prove it applied

Correct the error identified by last_error and the log detail. Then watch the next reload cycle: reloads.successes must increment and last_error should clear. Do not declare the incident closed on “file saved.” Declare it closed on successes advancing.

If you cannot find the error quickly, revert to the last known-good config and let the reload succeed, then iterate on the new config in a non-production pipeline.

Break the config management loop first

If failures is climbing every few seconds, your config management tool is the active fault. Pause its runs against this host, then fix the config. Fixing the file while the tool is mid-loop means your fix gets reverted by the next push of the same bad content.

If the pipeline is gone entirely

When the old pipeline stopped and the new one never started, a valid config on disk followed by a successful reload will bring it back. If reload cannot recover it, a Logstash restart will load the on-disk config directly. Restart is disruptive: with a memory queue, in-flight events are lost; with a persistent queue they survive and replay. Treat restart as the fallback, not the first move, and only after the on-disk config validates.

Reconcile the drift backlog

Once the pipeline is running the correct config, audit what happened during the drift window. Every change deployed between the first failure and the recovery was never live. Data processed in that window used the old logic: check for parse failure tags, routing mistakes, or missing fields that the unapplied changes were supposed to handle. If any of the skipped changes were correctness fixes, downstream data may need reprocessing.

Prevention

• Alert on any new reloads.failures increment. Any failure means deployed and running config diverge. One alert rule catches the entire failure class.
• Gate deploys on reloads.successes. Make your deployment pipeline poll the stats API after each config push and fail the deploy if successes does not advance within a couple of reload cycles. This turns invisible drift into a visible deploy failure.
• Rate-alert on repeated failures. A rapid climb signals a config management loop, which is a different incident than a single bad push.
• Monitor pipeline presence, not just process liveness. A missing pipeline ID in the stats API is a partial outage invisible to process checks.
• Track config file mtimes against approved change windows to catch out-of-band edits.
• Keep config.reload.* at the agent level (CLI flag or logstash.yml), never per-pipeline in pipelines.yml, where it is silently ignored.
• Know what blocks reloads. The stdin input prevents reloading entirely, and changes to files referenced by configs (grok pattern files, for example) are not independently watched; they are only picked up when a config file change triggers a reload. If your team edits pattern files and expects them to take effect, they do not.
• Be aware stats reset on reload. Pipeline counters reset on a successful hot reload, which creates artificial zero-dips in graphs. Baseline your reload counters as monotonic JVM-lifetime counters at the agent level where possible, or annotate reload events.

How Netdata helps

• Netdata collects the Logstash node stats API per pipeline, so reloads.successes and reloads.failures are graphed as trends rather than discovered by hand after the fact.
• An alert on any positive delta in reloads.failures converts invisible drift into a notification the moment a bad config lands.
• Correlating a reload failure against per-pipeline flow.output_throughput tells you immediately whether the old pipeline is still carrying traffic (drift) or the pipeline is down (outage).
• Overlaying reload failures with grok filter failures and DLQ growth shows whether the unapplied config was a correctness fix, and how much bad data the drift window produced.
• Rate-of-change views on the failures counter expose config management loops as a staircase pattern, distinct from a single one-off failure.
• In multi-pipeline deployments a failed reload on one pipeline is invisible in aggregates; the per-pipeline breakdown keeps it isolated.

Related guides

• Logstash flow.queue_backpressure: the input-throttling metric explained
• How Logstash actually works in production: a mental model for operators
• Logstash memory queue vs persistent queue: durability, visibility, and failure modes
• Logstash monitoring checklist: the signals every production pipeline needs
• Logstash monitoring maturity model: from survival to expert
• Logstash won’t start after a crash: persistent queue corruption and checkpoint errors
• Logstash persistent queue full: max_bytes reached and inputs blocked
• Logstash persistent queue not draining: page-release lag after downstream recovery
• Logstash persistent queue runway: how long until the PQ fills
• Logstash pipeline stalled: output rate at zero while the process looks alive
• Logstash queue events count growing: reading the in-flight backlog
• Logstash queue full: inputs blocked and the backpressure wedge