Fluentd unauthorized in_forward connections: log injection on the aggregator tier

You ran ss -tn on your Fluentd aggregator and saw connections to port 24224 from IP addresses you do not recognize. Or a security review turned up the fact that your aggregator-tier Fluentd accepts forwarded events from anything that can reach it. Either way: your in_forward input is unauthenticated, and any network-reachable host can inject events directly into your log pipeline.

An in_forward source without a <security> section performs zero authentication. Any host that can open a TCP connection to port 24224 can submit events with arbitrary tags and arbitrary record contents. The Fluentd project stated this plainly when authentication was introduced in v0.14.5: anyone who can connect to the TCP port of in_forward can inject events into the Fluentd process.

The aggregator tier is where this hurts most. A node agent tailing local files only trusts the host it runs on. An aggregator exists to accept events from the network, so its exposure surface is the network itself.

What this means

The forward protocol is how Fluentd instances ship events to each other. A node-level agent runs out_forward, the aggregator runs in_forward (default port 24224), and events flow over MessagePack. Without a <security> block, the aggregator does not verify who the sender is, what tags it uses, or what the records contain. The routing engine matches injected tags against your <match> directives and pushes forged events through filters, buffers, and outputs as legitimate traffic.

That single misconfiguration opens three attack modes:

flowchart LR
  A[Any reachable host] -->|TCP 24224, no auth| B[in_forward on aggregator]
  B --> C[Log injection: forged events, false audit trails]
  B --> D[Flooding: buffer fills, overflow fires]
  B --> E[Downstream poisoning: crafted payloads hit ES, SIEM]
  E --> F[tag-based path traversal in file outputs]

1. Log injection. The attacker picks tags that match your routing rules and writes forged events. Audit trails, access logs, and security events can be fabricated wholesale. If you use logs for forensics or compliance, their evidentiary value is gone the moment an unauthenticated sender can write to them.

2. Flooding and buffer-overflow DoS. A hostile sender pushes events faster than your outputs can drain. The buffer queue grows, buffer_available_buffer_space_ratios drops toward zero, and the overflow_action fires. With the default throw_exception, legitimate events are discarded. This is the backpressure cascade from buffer queue length growing, triggered deliberately.

3. Downstream poisoning. Crafted record contents are delivered to systems that trust log data: Elasticsearch, SIEM correlation rules, alerting pipelines. A specific and severe variant: if any output plugin builds file paths with ${tag} placeholders, injected tags containing ../ sequences can write outside the intended directory. (CVE-2026-44024, GHSA-44hj-4m45-frj3 — arbitrary file write via ${tag} path validation, CVSS 9.8 critical, fixed in v1.19.3, official Fluentd security advisory). A related decompression issue lets a small gzip-compressed payload expand and exhaust memory, because older versions limited compressed payload size but not decompressed size. This is CVE-2026-44160, GHSA-j9cw-hwqf-85w7 — a gzip decompression bomb in in_http and in_forward (CVSS 7.5 high, fixed in v1.19.3; affected versions up to v1.19.2).

There is also a quieter exposure: in_http (default port 9880) has no built-in authentication mechanism. If it is enabled and reachable, it accepts events from anyone, with no <security> option available. Network-level controls are the only defense.

Common causes

Quick checks

All read-only and safe to run on a production aggregator.

# 1. Who is connected to the forward port right now? (peer address is column 5)
ss -tn src :24224 | awk 'NR>1 {print $5}' | sort | uniq -c | sort -rn

# 2. Every unique peer seen on the port
ss -tnp | grep ":24224" | awk '{print $5}' | sort -u

# 3. Does the forward source have a security section?
grep -A20 '@type forward' /etc/fluent/fluentd.conf | grep -iE 'security|shared_key|allow_anonymous|transport'
# (Use /etc/td-agent/td-agent.conf for td-agent installs.)

# 4. Is in_http also listening?
grep -B2 -A5 '@type http' /etc/fluent/fluentd.conf
ss -tlnp | grep -E '9880|24224'

# 5. Authentication and TLS errors in the Fluentd log
grep -iE "(tls|ssl|auth|unauthorized|forbidden|certificate)" \
  /var/log/fluent/fluentd.log | tail -20

# 6. Fluentd version (CVE exposure depends on it)
fluentd --version

# 7. Is the port reachable from outside the expected sender network?
# Run this FROM a host that should not be allowed:
nc -zv <aggregator-ip> 24224

Interpretation notes:

• Checks 1 and 2 compare reality against your sender inventory. Every peer IP should map to a known node agent, forwarder, or load balancer. Anything else is an incident.
• Check 3 returning nothing means no authentication at all. Returning shared_key without allow_anonymous_source false means partial protection only.
• Check 7 is the one teams skip. Internal firewalls, security groups, and Kubernetes NetworkPolicies drift. Assume the config file lies and test the path.

How to diagnose it

1. Inventory your legitimate senders. List the nodes, forwarders, and services that should be forwarding to this aggregator. Use IP ranges, not just hostnames, because that is what <client> sections and firewall rules match on.

2. Diff live connections against the inventory. Use checks 1 and 2 above. Unknown source IPs mean either an unauthorized sender or a stale inventory. Resolve that ambiguity first; do not assume benign.

3. Audit the effective config. @include directives mean the <security> block may live in a different file than the <source>. Verify against the running process: query http://localhost:24220/api/config.json on each worker’s monitor_agent port (24220, 24221, …) and inspect the forward source’s configuration as Fluentd actually loaded it.

4. Prove the injection path, safely. From a host that is not an authorized sender, push a tagged test event:

# From an unauthorized host: this should FAIL on a locked-down aggregator
echo '{"msg":"unauthorized injection test"}' | \
  fluent-cat --host <aggregator-ip> --port 24224 security.test

Then check whether security.test events arrive at your outputs. If they do, the pipeline accepted forged events and you have confirmed the exposure end to end. If the connection is refused or the handshake fails, authentication is working.

5. Check for signs it already happened. Look for tags in your outputs that no configured input produces, sudden unexplained input rate spikes (input emit_records rate far above baseline), or buffer pressure events with no corresponding application log volume. The emit records rate gap and unexplained queue growth are your historical breadcrumbs.

6. Check version exposure. If the running version predates the decompression-limit and ${tag} traversal fixes, those mitigations are absent. On unauthenticated inputs they are directly exploitable. The fix landed in v1.19.3; decompression_size_limit (default 256MB) was introduced in that release, so prior versions have no decompressed-size limit at all.

Signals to monitor

How to lock it down

Enable shared-key authentication

Add a <security> section to the forward source on every aggregator. The handshake is a SHA512-based challenge-response (HELO/PING with a nonce); the key itself never crosses the wire. Built in since v0.14.5.

<source>
  @type forward
  port 24224
  <security>
    self_hostname aggregator.internal
    shared_key YOUR_LONG_RANDOM_KEY
    allow_anonymous_source false
    <client>
      network 10.1.0.0/16
      shared_key YOUR_LONG_RANDOM_KEY
    </client>
  </security>
</source>

Two details that bite operators:

• allow_anonymous_source defaults to true even when <security> is present. Setting a shared_key alone does not lock the port down. You must explicitly set it to false and define <client> sections for your sender networks.
• Senders must configure the matching key in out_forward (shared_key in their <server> section). Roll this out in coordination: senders first with the key, then the aggregator, or you will break legitimate traffic.

With authentication enabled, only MessagePack payloads are supported; JSON-format single events over forward are unavailable per the protocol spec. Standard out_forward uses MessagePack, so this mainly matters for custom senders.

Add TLS, preferably mutual TLS

Encryption without authentication only hides the injected traffic. Combine both. Built-in TLS transport exists since v0.14.12; before that, the third-party fluent-plugin-secure-forward gem was required and is now superseded. client_cert_auth true with a ca_path gives mutual TLS, where the aggregator verifies client certificates (mutual TLS support was added in v1.1.1).

<source>
  @type forward
  port 24224
  <transport tls>
    cert_path /etc/fluent/certs/aggregator.pem
    private_key_path /etc/fluent/certs/aggregator.key
    client_cert_auth true
    ca_path /etc/fluent/certs/ca.pem
  </transport>
  <security>
    self_hostname aggregator.internal
    shared_key YOUR_LONG_RANDOM_KEY
    allow_anonymous_source false
    <client>
      network 10.1.0.0/16
      shared_key YOUR_LONG_RANDOM_KEY
    </client>
  </security>
</source>

mTLS is the stronger control because it does not depend on a symmetric key copied to every node. The tradeoff is certificate lifecycle management: expiry causes cliff-edge simultaneous failures across all senders, so monitor certificate end dates.

Restrict the network path

Authentication is defense in depth, not a substitute for reachability control:

• Bind in_forward to an internal interface only (bind 10.1.0.5), not 0.0.0.0, unless you genuinely accept events from multiple networks.
• Firewall or security-group port 24224 to the sender CIDR ranges.
• In Kubernetes, use NetworkPolicy to restrict which pods can reach the aggregator’s port.
• For in_http, which has no auth option, network restriction is the only control. If you cannot restrict it, do not expose it.

Upgrade and harden against known CVEs

• Upgrade to the current Fluentd release that carries the ${tag} path traversal and gzip decompression fixes. (A fix is in Fluentd v1.19.3, bundled for the fluent-package v6.0.4 LTS distribution; CVE-2026-44024 is CVSS 9.8 critical and CVE-2026-44160 is CVSS 7.5 high, and the new decompression_size_limit parameter defaults to 256MB — official Fluentd security advisory and v1.19.3 release notes).
• Independently of version: do not use ${tag} in output file paths when any input accepts events from the network, and run Fluentd as a non-root user so a traversal write cannot touch system files.
• In mixed fleets, check whether Fluent Bit instances face the same exposure; Fluent Bit is a separate codebase with its own in_forward authentication history. (CVE-2025-12969 is a real advisory: Fluent Bit’s in_forward does not enforce the security.users authentication under certain configuration conditions).

Prevention

• Baseline config template. Every aggregator-tier forward source ships with <security>, allow_anonymous_source false, and <client> network sections from day one. Make “no security section” a code review failure.
• Sender allowlist as data. Keep the expected sender CIDR list in one place and generate both the <client> sections and the firewall rules from it. Drift between the two is how gaps appear.
• Continuous peer auditing. Alert on any source IP on 24224 outside the allowlist. It is a binary signal and cheap to evaluate.
• Treat the shared key as a credential. Store it in your secrets system, rotate it on the same schedule as other credentials, and never commit it to a repo or bake it into an image.
• Config file integrity monitoring. A tampered Fluentd config can disable the security section or add an unauthorized output. Alert on config file modification outside the deployment pipeline.
• Version tracking. Track Fluentd versions fleet-wide. Security fixes only help if they are actually deployed.

How Netdata helps

• Netdata collects Fluentd monitor_agent metrics per worker, so input emit_records rate spikes from a flooding sender are visible per aggregator instance rather than averaged away.
• Correlating input rate with buffer_queue_length and buffer_available_buffer_space_ratios on one dashboard separates a flood-driven buffer fill from a destination-driven one: in a flood, input rate leads the queue growth.
• Host-level network visibility shows which peers hold connections to the Fluentd process, making an unknown sender on 24224 visible without a manual ss session.
• Process RSS trends catch the memory-exhaustion variant of an attack (decompression bombs, oversized events) before the OOM killer does.
• Fluentd log collection lets you alarm on repeated TLS and auth handshake failures, which is what probing looks like before a successful injection.

Related guides

• Fluentd broken pipe / connection reset: dropped output connections and LB timeouts
• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd file buffer filling the disk: when the buffer partition runs out
• Fluentd buffer_oldest_timekey lag: how far behind the oldest buffered data is
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd CPU bottleneck: the Ruby GVL caps a single worker at one core
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss
• Fluentd duplicate events: why the same log shows up twice downstream
• Fluentd emit_error_count: the number-one under-monitored data-loss signal