Fluentd config integrity: leaked credentials and silent tampering

A Fluentd config file is not just routing logic. In most production deployments it is also a credential store: Elasticsearch passwords, S3 access keys, Kafka SASL secrets, TLS client certificates, and forward shared keys all live in the same file that tells Fluentd where your logs go. If that file is world-readable, every local user and every compromised process on the host can read those credentials. If it is writable outside the deploy pipeline, an attacker can redirect your log stream to their own endpoint, quietly drop collection for the services they are touching, or add an output you never approved.

Both failure modes are silent. Fluentd keeps running, events flow in and out, and the only way to notice that a <match> block was narrowed to exclude your auth logs, or that a second output was added pointing at an unfamiliar host, is to watch the config itself and the shape of the pipeline it produces.

What this protects against

flowchart TD
  A[Tampered fluentd config] --> B[Add unauthorized output - log exfiltration]
  A --> C[Remove or narrow a match block - collection silently stops]
  A --> D[Change endpoint or credentials - reroute or break delivery]
  E[World-readable config file] --> F[Credential leak: ES password, S3 keys, Kafka SASL]
  G[monitor_agent exposed on 0.0.0.0:24220] --> F

• Credential disclosure. Anyone who can read the config file (or scrape an exposed monitor agent endpoint) harvests destination credentials in plaintext.
• Log exfiltration. An added <match> with an attacker-controlled out_forward or out_http destination duplicates your log stream out of the environment. Input and output rates still look healthy because the pipeline is working exactly as configured.
• Covering tracks. Removing or narrowing a match block stops collection for specific tags. The rest of the pipeline looks fine; you only notice when you go looking for logs that were never shipped.

Prerequisites

• Shell access to the hosts running Fluentd, with sudo for permission changes.
• Knowledge of which package variant you run. Paths differ: td-agent uses /etc/td-agent/td-agent.conf and /var/log/td-agent/td-agent.log; fluent-package uses /etc/fluent/fluentd.conf and /var/log/fluent/fluentd.log. Examples below use td-agent paths; adjust for your install.
• A record of your last known-good deploy (timestamp, config hash, or both). Without a baseline, “changed outside the pipeline” is undetectable.
• In Kubernetes, access to the ConfigMaps carrying Fluentd config and whatever you use to watch object changes.

Procedure

1. Inventory every config file and check permissions

Do not stop at the main config. @include directives pull in conf.d/ directories, and credentials are just as likely to sit in an included fragment.

# Find the config files and check permissions and ownership
ls -la /etc/td-agent/td-agent.conf
ls -la /etc/td-agent/conf.d/

# See what credentials are actually sitting in them
grep -rE "password|secret|key|token" /etc/td-agent/

Anything world-readable here is an active leak. Group-readable is acceptable only if the group is tightly controlled.

2. Lock down ownership and permissions

Config files should be mode 600 or 640, owned by root with the group set to the Fluentd service account (or owned by the service account directly). These are write operations; get the ownership wrong and Fluentd fails to read its config on next start or reload, so verify immediately after.

# Restrict permissions (verify the Fluentd user can still read these afterward)
chown root:td-agent /etc/td-agent/td-agent.conf /etc/td-agent/conf.d/*.conf
chmod 640 /etc/td-agent/td-agent.conf /etc/td-agent/conf.d/*.conf

The group name must match your package variant and service account (td-agent, fluent, or whatever the unit runs as). Default permissions set at package install time vary by distribution and package version, so do not assume a fresh install is safe. Check.

3. Move credentials out of the file

Fluentd supports embedded Ruby evaluation in config files, so credentials can be read from the process environment instead of being hardcoded:

<match app.**>
  @type elasticsearch
  host es-internal.example.com
  user fluentd
  password #{ENV['ES_PASSWORD']}
</match>

The environment variables then live in the systemd unit’s EnvironmentFile, a Kubernetes Secret mounted as env vars, or your secrets manager’s injection mechanism, all of which have better access control and rotation stories than a flat file. Keep step 2 anyway: the config still reveals your topology, but a leaked config no longer hands over working credentials.

4. Stop Fluentd from re-leaking the config into its own logs

On startup, Fluentd dumps the full effective configuration to its log. If you ship Fluentd’s own logs anywhere (and you should), the credentials you just protected end up in your log storage. Suppress the dump in <system>:

<system>
  suppress_config_dump true
</system>

5. Restrict the monitor agent to localhost

The monitor agent is the standard way to get Fluentd’s internal metrics, and the commonly copied example config binds it to every interface:

<source>
  @type monitor_agent
  bind 0.0.0.0
  port 24220
</source>

Two problems with that. First, /api/plugins.json?with_config=true returns plugin configuration over unauthenticated HTTP. Second, the monitor agent response includes internal plugin state, and in Fluentd versions up to 1.19.2 this can expose credentials stored in plugin instance variables to anyone who can reach port 24220 (CVE-2026-44025, fixed in 1.19.3).

Bind it to loopback and scrape it locally, or firewall the port:

<source>
  @type monitor_agent
  bind 127.0.0.1
  port 24220
</source>

In multi-worker mode the port auto-increments per worker (24220, 24221, …), so the firewall rule needs to cover the range, not just the first port.

6. Baseline the config and put integrity monitoring on it

File integrity monitoring is the control that turns “silent tampering” into a page. At minimum, record a hash after every deploy and compare on a schedule:

# Record a baseline after each deploy
sha256sum /etc/td-agent/td-agent.conf /etc/td-agent/conf.d/*.conf > /var/lib/fim/fluentd-config.baseline

# Later, on schedule or on demand
sha256sum -c /var/lib/fim/fluentd-config.baseline

A dedicated FIM tool (auditd watches, or a commercial agent) gives you who-changed-it attribution; the hash check only tells you it changed. A lightweight fallback that catches many cases is comparing the file mtime against your deploy log:

# Modification time as epoch - compare against your last recorded deploy time
stat -c '%Y' /etc/td-agent/td-agent.conf

In Kubernetes, watch the ConfigMap object itself. ConfigMap updates propagate to pods asynchronously, so there is a window where the object changed but the running Fluentd has not reloaded yet. Alert on the object change, not on the pod’s file.

7. Add network-level tripwires

Config integrity tells you the file changed. Network observation tells you what a changed (or long-ago-misconfigured) Fluentd is actually doing, and it catches exfiltration even if the file was reverted afterward.

# Where is Fluentd actually connecting?
ss -tnp | grep "$(pgrep -f fluentd | head -1)" | awk '{print $5}'

# Who is talking to the forward input?
ss -tn | grep ":24224" | awk '{print $5}' | sort -u

With multiple workers, repeat the first check per PID; head -1 only covers the first match. Any outbound destination outside your known allowlist of log backends deserves immediate review: either the config was tampered with or a plugin is misbehaving. Any inbound source on 24224 outside your known senders means someone can inject events with crafted tags into your pipeline, and on aggregator-tier instances in_forward without a <security> shared key means any network-reachable host can write to any tag.

Verifying it works

After the lockdown, prove each control instead of assuming it:

1. Read test. As an unprivileged user (or a throwaway shell as the web/app service account), try cat /etc/td-agent/td-agent.conf. It must fail.
2. Service read test. Confirm Fluentd itself can still read the config: fluentd --dry-run -c /etc/td-agent/td-agent.conf validates the config without starting plugins. A permission mistake shows up immediately.
3. Monitor agent exposure test. From a different host on the same network, curl --max-time 2 http://<fluentd-host>:24220/api/plugins.json. It must time out or be refused.
4. FIM test. Append a harmless comment to the config, confirm the integrity check fires, then revert.
5. Pipeline shape test. Pull /api/plugins.json and diff the plugin list against your expected inventory. An unauthorized output shows up as a plugin you do not recognize, and this is also how you catch a reload that only partially applied (see Fluentd config reload failed).

Common pitfalls

• FIM that fires on every deploy. If your pipeline legitimately rewrites the config, alerts that ignore deploy events train everyone to ignore the alert. Feed deploy timestamps into the alerting logic so only out-of-band changes page.
• Protecting the main file but not conf.d. Included fragments are config. Permissions, hashing, and credential greps must cover the whole include tree.
• Copy-pasting the 0.0.0.0 monitor_agent example. It is in documentation and blog posts everywhere. Audit every running instance; do not assume yours is bound to loopback.
• ConfigMaps treated as secret stores. ConfigMaps are not a secrets mechanism. Keep credentials in Secrets referenced as environment variables, not inline in the ConfigMap.
• Assuming a revert means safety. An attacker who adds an exfiltration output and later removes it leaves a clean config file and a clean hash. The network tripwires in step 7 and the pipeline-shape check observe behavior, not just file state.
• Reload versus restart surprises. A tampered config only takes effect after a reload, and a SIGHUP reload can partially apply, leaving the running pipeline different from both the old and new files. Verify the live plugin list after any reload, expected or not.

Signals to monitor

How Netdata helps

• Netdata’s Fluentd collector scrapes the monitor agent, so per-output emit_records, retry_count, write_count, and buffer gauges are already time-series. A match block silently removed by tampering shows up as one output’s rate going to zero while the rest stay healthy, and ML anomaly detection flags that divergence without a static threshold.
• Per-process network connection visibility makes step 7 continuous instead of manual: a Fluentd process holding a connection to a destination it has never talked to before stands out against its historical baseline.
• Per-process uptime charts reveal the restarts and reloads a tampered config requires to take effect, a second tripwire independent of the file itself.
• Correlating the timestamp of a config change (from FIM or deploy events) against the moment output rates shifted turns a forensic exercise into a single look at one dashboard.

Related guides

• Fluentd broken pipe / connection reset: dropped output connections and LB timeouts
• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd file buffer filling the disk: when the buffer partition runs out
• Fluentd buffer_oldest_timekey lag: how far behind the oldest buffered data is
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd CPU bottleneck: the Ruby GVL caps a single worker at one core
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss
• Fluentd duplicate events: why the same log shows up twice downstream
• Fluentd emit_error_count: the number-one under-monitored data-loss signal