Fluentd config reload failed: SIGHUP that partially applies

You edited the Fluentd config, sent SIGHUP, watched the log line saying the config reloaded, and moved on. Hours later you notice a new output never started receiving data, or an old filter is still dropping events you told it to keep. The process never crashed. No error fired. But the pipeline running in memory is not the pipeline in the config file.

This is the partial reload failure: some plugins reloaded with the new configuration, others are still running the old one, and one or more plugins you expected are absent from the running process. Because Fluentd stays up and keeps processing events through whatever plugins did load, this failure is silent unless you verify the loaded plugin inventory after every reload.

The defining check: the plugin set reported by the monitor agent at /api/plugins.json must exactly match the plugin set implied by the config file. If a plugin is missing after a reload, the reload did not fully apply, and you are running a hybrid of the old and new configuration.

What this means

On SIGHUP, Fluentd does not patch configuration in place. The supervisor kills the worker process and respawns it with the new config. Plugins that use shared server sockets (via server_helper) keep their listen sockets across the respawn, so there is no connection downtime, but everything else is rebuilt: buffers, filters, outputs, parsers. A CPU spike and a brief throughput dip during this window is normal.

The respawn can fail partway. A plugin that fails to initialize during the respawn does not necessarily kill the process. Depending on where the failure lands, you can end up with a worker that loaded most of the new config, kept some old state, or silently dropped a plugin that could not start. Fluentd logs may show a reload message even when individual plugin initialization failed.

Since v1.9 there is also SIGUSR2 graceful reload, which reloads config in-process without restarting the worker, with its own limitations (changes to <system> are ignored, and any plugin using class variables makes the whole reload fail with “Failed to reload config file: Unreloadable plugin”). Since v1.18, zero-downtime restart (SIGUSR2 to the supervisor) is the recommended approach. Which mechanism you used changes what “partially applied” looks like, but the verification step is the same: diff the running plugin inventory against the config.

Common causes

Quick checks

These are all read-only.

# 1. Full plugin inventory the worker actually loaded
curl -s http://localhost:24220/api/plugins.json | jq '.plugins[] | {id: .plugin_id, type: .type, category: .plugin_category}'

# 2. How many plugins of each category are running
curl -s http://localhost:24220/api/plugins.json | jq '[.plugins[].plugin_category] | group_by(.) | map({(.[0]): length}) | add'

# 3. What config the process thinks it has
curl -s http://localhost:24220/api/config.json | jq .

# 4. Process tree: supervisor plus expected worker count, nothing else
ps aux | grep '[f]luentd'

# 5. Reload-related log lines around the time you sent the signal
grep -iE "reload|sighup|unreloadable|failed to (configure|start)" /var/log/td-agent/td-agent.log | tail -30
# fluent-package: /var/log/fluent/fluentd.log

# 6. Plugin load errors since the reload
journalctl -u td-agent --since "30 minutes ago" | grep -iE "(load_plugin|LoadError|uncaught|cannot load)"

Notes on these:

• Port 24220 assumes a single worker. In multi-worker mode, query each worker’s port (24220, 24221, …). Each worker has an independent plugin inventory.
• /api/plugins.json requires <source> @type monitor_agent </source> in the config. If it was never configured, you have no in-band way to verify plugin state, which is itself a gap worth fixing.
• Since v1.19.3, the config and retry fields are no longer included by default, and the old ?with_config=true / ?with_retry=true query parameters stopped working in the same release. Set include_config true or include_retry true in the monitor_agent <source> section to get them back.

How to diagnose it

flowchart TD
  A[SIGHUP or SIGUSR2 sent] --> B{Reload message in log?}
  B -- no --> C[Signal never reached supervisor: check container init, PID, signal routing]
  B -- yes --> D[Fetch /api/plugins.json]
  D --> E{Plugin set matches config?}
  E -- yes --> F[Reload applied; investigate elsewhere]
  E -- plugin missing --> G[Check log for plugin init failure at reload time]
  E -- old config still active --> H{SIGUSR2 used?}
  H -- yes --> I[Unreloadable plugin or ignored system section; restart required]
  H -- no --> J[Worker respawn failed partway; check for plugin load errors, worker_id collisions]
  G --> K[Fix plugin, then full restart, not another reload]
  I --> K
  J --> K

1. Snapshot the expected plugin set from the config. Count the <source>, <filter>, <match>, and <label> plugin blocks in the effective config file. Include plugins that other plugins start implicitly only if you know they exist; the point is a baseline of the plugins you declared.

2. Fetch the actual running set. Pull /api/plugins.json and list every plugin_id, type, and plugin_category. Do this per worker port in multi-worker mode.

3. Diff expected versus actual. Any plugin in the config but not in the API output failed to load during the reload. Any plugin in the API output with stale behavior (old routing, old match rules) kept old state while others reloaded: the partial apply.

4. Check the log at the reload timestamp. Look for plugin configure errors, LoadError, “Unreloadable plugin”, or “specified worker_id collisions is detected”. The reload log line and the failure line are usually close together but easy to miss if you only grepped for “reload”.

5. Confirm which reload mechanism ran. SIGHUP respawns the worker; SIGUSR2 reloads in process. If you sent SIGUSR2 and one plugin is unreloadable, the entire reload fails atomically and everything stays on the old config. If you sent SIGHUP, the failure is per-plugin during respawn, which is what produces the true partial state.

6. Check the process tree for anomalies. More workers than configured, a worker with a much older start time than its siblings, or leftover processes from before the reload all indicate the respawn did not complete cleanly.

7. Decide: another reload will not fix it. Once the running state and the config diverge, the only reliable convergence is a full process restart. Repeated SIGHUPs against a wedged worker have produced SIGKILLed workers and leaked processes in reported issues.

Metrics and signals to monitor

Fixes

Plugin failed to initialize during the reload

Find the init error in the log, fix the plugin configuration or the missing gem, then do a full restart of Fluentd rather than another SIGHUP. A restart guarantees every plugin initializes from a clean process. Tradeoff: a restart drops memory-backed buffer contents and resets in_tail to its pos_file positions, so expect a brief gap or duplicate window with file-backed buffers. That replay is normal.

Unreloadable plugin blocks SIGUSR2

The check is mechanical: Fluentd refuses to reload if any plugin class defines class variables. The known offender was fluent-plugin-prometheus before v1.8.2. Upgrade the plugin. If no upgrade exists, stop using graceful reload for that deployment and use SIGHUP or a full restart instead. Any plugin can be unreloadable; there is no maintained list, so treat the log error as the discovery mechanism.

A plugin breaks Fluentd core at load time

Plugins that require 'yajl/json_gem' replace Ruby’s JSON module globally, which breaks Fluentd’s config parser (it depends on JSON.parse raising on incomplete input; the yajl compatibility layer returns {} instead). This has broken reloads in the wild (splunkhec before v2.1). Remove or upgrade the offending plugin, then restart. This class of bug is why plugin upgrades deserve the same change control as config changes.

Container signal routing

In Docker with dumb-init, docker kill -s HUP <container> delivered SIGHUP to the entire process group, so the worker got the signal directly and died with SignalException instead of the supervisor handling a reload. Set DUMB_INIT_SETSID=0, or send the signal to the supervisor PID from inside the container, or use an orchestrator-native rolling restart instead of in-place reload.

Diverged state that will not converge

If the running plugin set and the config disagree and the cause is not obvious, do not keep sending signals. Capture the evidence (/api/plugins.json output, /api/config.json output, log excerpt, process list), then restart the process. Reload is a convenience; restart is the convergence mechanism.

Prevention

• Make post-reload verification a step, not an afterthought. Any automation that sends SIGHUP should immediately fetch /api/plugins.json, compare the plugin set to the rendered config, and fail the pipeline on mismatch.
• Baseline the plugin inventory. Record the expected plugin IDs per worker in your deployment tooling. Alert when the live set deviates, not just when the process dies.
• Enable the monitor agent everywhere. Without in_monitor_agent, this failure class is undetectable except by noticing missing data downstream.
• Prefer restart-friendly reload paths on modern versions. Since v1.18, zero-downtime restart (SIGUSR2 to the supervisor) is the documented recommendation over graceful reload, and it supports in_udp, in_tcp, and in_syslog via shared sockets.
• Keep plugins current and few. Unreloadable plugins and core-breaking requires are third-party plugin behaviors. Fewer, newer plugins means fewer reload failure modes.
• Test reloads in staging with the exact production plugin set. A config that reloads cleanly without your output plugins tells you nothing.

How Netdata helps

• Netdata collects Fluentd’s monitor agent metrics continuously, so you have a per-second history of emit_records and write_count per plugin across the reload boundary. A plugin that vanished in a partial reload shows up as an abrupt stop in its series at the reload timestamp, even if nobody checked the API at the time.
• Because counters are per plugin ID, you can correlate “new plugin never emitted” against “reload happened at 14:03” without parsing logs first.
• Process-level metrics (process count, RSS, CPU per process) expose leaked or mixed-generation workers after a bad respawn, which pure Fluentd metrics miss.
• Alerting on plugin throughput dropping to zero while its input source is still producing catches the silent half of a partial reload: the config that applied to some plugins but not the one that moves your data.
• Per-worker history matters in multi-worker deployments, where one worker can fail a reload while its siblings apply it cleanly.

Related guides

• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• How Fluentd actually works in production: a mental model for operators
• Fluentd monitoring checklist: the signals every production log pipeline needs
• Fluentd monitoring maturity model: from survival to expert
• Fluentd poison pill crash loop: one bad log line that kills the process on every restart
• Fluentd process not running: the log pipeline is dead and the host has gone dark