$ guides / fluentd / fluentd-unauthorized-forward ▌

Operations Guides

Fluentd unauthorized in_forward connections: log injection on the aggregator tier

You ran ss -tn on your Fluentd aggregator and saw connections to port 24224 from IP addresses you do not recognize. Or a security review turned up the fact that your aggregator-tier Fluentd accepts forwarded events from anything that can reach it. Either way: your in_forward input is unauthenticated, and any network-reachable host can inject events directly into your log pipeline.

An in_forward source without a <security> section performs zero authentication. Any host that can open a TCP connection to port 24224 can submit events with arbitrary tags and arbitrary record contents. The Fluentd project stated this plainly when authentication was introduced in v0.14.5: anyone who can connect to the TCP port of in_forward can inject events into the Fluentd process.

The aggregator tier is where this hurts most. A node agent tailing local files only trusts the host it runs on. An aggregator exists to accept events from the network, so its exposure surface is the network itself.

What this means

The forward protocol is how Fluentd instances ship events to each other. A node-level agent runs out_forward, the aggregator runs in_forward (default port 24224), and events flow over MessagePack. Without a <security> block, the aggregator does not verify who the sender is, what tags it uses, or what the records contain. The routing engine matches injected tags against your <match> directives and pushes forged events through filters, buffers, and outputs as legitimate traffic.

That single misconfiguration opens three attack modes:

flowchart LR
  A[Any reachable host] -->|TCP 24224, no auth| B[in_forward on aggregator]
  B --> C[Log injection: forged events, false audit trails]
  B --> D[Flooding: buffer fills, overflow fires]
  B --> E[Downstream poisoning: crafted payloads hit ES, SIEM]
  E --> F[tag-based path traversal in file outputs]

Log injection. The attacker picks tags that match your routing rules and writes forged events. Audit trails, access logs, and security events can be fabricated wholesale. If you use logs for forensics or compliance, their evidentiary value is gone the moment an unauthenticated sender can write to them.
Flooding and buffer-overflow DoS. A hostile sender pushes events faster than your outputs can drain. The buffer queue grows, buffer_available_buffer_space_ratios drops toward zero, and the overflow_action fires. With the default throw_exception, legitimate events are discarded. This is the backpressure cascade from buffer queue length growing, triggered deliberately.
Downstream poisoning. Crafted record contents are delivered to systems that trust log data: Elasticsearch, SIEM correlation rules, alerting pipelines. A specific and severe variant: if any output plugin builds file paths with ${tag} placeholders, injected tags containing ../ sequences can write outside the intended directory. A related decompression issue reportedly lets a small gzip-compressed payload expand and exhaust memory because older versions limited compressed payload size but not decompressed size.

There is also a quieter exposure: in_http (default port 9880) has no built-in authentication mechanism. If it is enabled and reachable, it accepts events from anyone, with no <security> option available. Network-level controls are the only defense.

Common causes

Cause	What it looks like	First thing to check
`in_forward` has no `<security>` section	Any host can connect and emit events; unknown peer IPs in `ss` output	`grep -A5 '@type forward' /etc/fluent/fluentd.conf` and look for a `<security>` block
`<security>` exists but `allow_anonymous_source` left at default	Shared key configured, yet unauthenticated senders still accepted	Check for `allow_anonymous_source false` and `<client>` sections; the default is `true` even with `<security>` present
Port 24224 exposed beyond the expected network	Forward port reachable from the internet, other VPCs, or the whole pod network	Test from a host that should NOT have access: `nc -zv <aggregator> 24224`
`in_http` enabled for convenience and forgotten	Events accepted on 9880 from anywhere; no auth option exists	Check config for `@type http` sources and firewall rules for the port
Shared key widely distributed or leaked	Every node agent has the same key; key appears in repos or images	Audit where the key is stored; treat it as a credential
Fluentd version missing current security fixes	Old version without the decompression limit or `${tag}` traversal patch	`fluentd --version`

Quick checks

All read-only and safe to run on a production aggregator.

# 1. Who is connected to the forward port right now? (peer address is column 5)
ss -tn src :24224 | awk 'NR>1 {print $5}' | sort | uniq -c | sort -rn

# 2. Every unique peer seen on the port
ss -tnp | grep ":24224" | awk '{print $5}' | sort -u

# 3. Does the forward source have a security section?
grep -A20 '@type forward' /etc/fluent/fluentd.conf | grep -iE 'security|shared_key|allow_anonymous|transport'
# (Use /etc/td-agent/td-agent.conf for td-agent installs.)

# 4. Is in_http also listening?
grep -B2 -A5 '@type http' /etc/fluent/fluentd.conf
ss -tlnp | grep -E '9880|24224'

# 5. Authentication and TLS errors in the Fluentd log
grep -iE "(tls|ssl|auth|unauthorized|forbidden|certificate)" \
  /var/log/fluent/fluentd.log | tail -20

# 6. Fluentd version (CVE exposure depends on it)
fluentd --version

# 7. Is the port reachable from outside the expected sender network?
# Run this FROM a host that should not be allowed:
nc -zv <aggregator-ip> 24224

Interpretation notes:

Checks 1 and 2 compare reality against your sender inventory. Every peer IP should map to a known node agent, forwarder, or load balancer. Anything else is an incident.
Check 3 returning nothing means no authentication at all. Returning shared_key without allow_anonymous_source false means partial protection only.
Check 7 is the one teams skip. Internal firewalls, security groups, and Kubernetes NetworkPolicies drift. Assume the config file lies and test the path.

How to diagnose it

Inventory your legitimate senders. List the nodes, forwarders, and services that should be forwarding to this aggregator. Use IP ranges, not just hostnames, because that is what <client> sections and firewall rules match on.
Diff live connections against the inventory. Use checks 1 and 2 above. Unknown source IPs mean either an unauthorized sender or a stale inventory. Resolve that ambiguity first; do not assume benign.
Audit the effective config. @include directives mean the <security> block may live in a different file than the <source>. Verify against the running process: query http://localhost:24220/api/config.json on each worker’s monitor_agent port (24220, 24221, …) and inspect the forward source’s configuration as Fluentd actually loaded it.
Prove the injection path, safely. From a host that is not an authorized sender, push a tagged test event:

# From an unauthorized host: this should FAIL on a locked-down aggregator
echo '{"msg":"unauthorized injection test"}' | \
  fluent-cat --host <aggregator-ip> --port 24224 security.test

Then check whether security.test events arrive at your outputs. If they do, the pipeline accepted forged events and you have confirmed the exposure end to end. If the connection is refused or the handshake fails, authentication is working.

Check for signs it already happened. Look for tags in your outputs that no configured input produces, sudden unexplained input rate spikes (input emit_records rate far above baseline), or buffer pressure events with no corresponding application log volume. The emit records rate gap and unexplained queue growth are your historical breadcrumbs.
Check version exposure. If the running version predates the decompression-limit and ${tag} traversal fixes, those mitigations are absent. On unauthenticated inputs they are directly exploitable.

Signals to monitor

Signal	Why it matters	Warning sign
Source IPs on port 24224	The direct unauthorized-sender signal	Any peer outside the sender allowlist
Input `emit_records` rate per source	Flooding shows as a rate spike from one sender before buffers feel it	Sustained deviation >50% from rolling baseline
`buffer_queue_length` and `buffer_available_buffer_space_ratios`	A flood attack manifests here as it becomes a DoS	Queue growing with available ratio under 20%
Tags observed at outputs vs tags produced by configured inputs	Injected events arrive with attacker-chosen tags	Tags present downstream that no input source emits
TLS/auth errors in Fluentd log	Failed handshakes indicate probing or misconfigured senders	Repeated handshake failures from one IP
Fluentd version across the fleet	Determines exposure to the path-traversal and decompression CVEs	Any aggregator running an unpatched version with a network-reachable forward port

How to lock it down

Enable shared-key authentication

Add a <security> section to the forward source on every aggregator. The handshake is a SHA512-based challenge-response (HELO/PING with a nonce); the key itself never crosses the wire. Built in since v0.14.5.

<source>
  @type forward
  port 24224
  <security>
    self_hostname aggregator.internal
    shared_key YOUR_LONG_RANDOM_KEY
    allow_anonymous_source false
    <client>
      network 10.1.0.0/16
      shared_key YOUR_LONG_RANDOM_KEY
    </client>
  </security>
</source>

Two details that bite operators:

allow_anonymous_source defaults to true even when <security> is present. Setting a shared_key alone does not lock the port down. You must explicitly set it to false and define <client> sections for your sender networks.
Senders must configure the matching key in out_forward (shared_key in their <server> section). Roll this out in coordination: senders first with the key, then the aggregator, or you will break legitimate traffic.

With authentication enabled, only MessagePack payloads are supported; JSON-format single events over forward are unavailable per the protocol spec. Standard out_forward uses MessagePack, so this mainly matters for custom senders.

Add TLS, preferably mutual TLS

Encryption without authentication only hides the injected traffic. Combine both. Built-in TLS transport exists since v0.14.12; before that, the third-party fluent-plugin-secure-forward gem was required and is now superseded. client_cert_auth true with a ca_path gives mutual TLS, where the aggregator verifies client certificates.

<source>
  @type forward
  port 24224
  <transport tls>
    cert_path /etc/fluent/certs/aggregator.pem
    private_key_path /etc/fluent/certs/aggregator.key
    client_cert_auth true
    ca_path /etc/fluent/certs/ca.pem
  </transport>
  <security>
    self_hostname aggregator.internal
    shared_key YOUR_LONG_RANDOM_KEY
    allow_anonymous_source false
    <client>
      network 10.1.0.0/16
      shared_key YOUR_LONG_RANDOM_KEY
    </client>
  </security>
</source>

mTLS is the stronger control because it does not depend on a symmetric key copied to every node. The tradeoff is certificate lifecycle management: expiry causes cliff-edge simultaneous failures across all senders, so monitor certificate end dates.

Restrict the network path

Authentication is defense in depth, not a substitute for reachability control:

Bind in_forward to an internal interface only (bind 10.1.0.5), not 0.0.0.0, unless you genuinely accept events from multiple networks.
Firewall or security-group port 24224 to the sender CIDR ranges.
In Kubernetes, use NetworkPolicy to restrict which pods can reach the aggregator’s port.
For in_http, which has no auth option, network restriction is the only control. If you cannot restrict it, do not expose it.

Upgrade and harden against known CVEs

Upgrade to the current Fluentd release that carries the ${tag} path traversal and gzip decompression fixes.
Independently of version: do not use ${tag} in output file paths when any input accepts events from the network, and run Fluentd as a non-root user so a traversal write cannot touch system files.
In mixed fleets, check whether Fluent Bit instances face the same exposure; Fluent Bit is a separate codebase with its own in_forward authentication history.

Prevention

Baseline config template. Every aggregator-tier forward source ships with <security>, allow_anonymous_source false, and <client> network sections from day one. Make “no security section” a code review failure.
Sender allowlist as data. Keep the expected sender CIDR list in one place and generate both the <client> sections and the firewall rules from it. Drift between the two is how gaps appear.
Continuous peer auditing. Alert on any source IP on 24224 outside the allowlist. It is a binary signal and cheap to evaluate.
Treat the shared key as a credential. Store it in your secrets system, rotate it on the same schedule as other credentials, and never commit it to a repo or bake it into an image.
Config file integrity monitoring. A tampered Fluentd config can disable the security section or add an unauthorized output. Alert on config file modification outside the deployment pipeline.
Version tracking. Track Fluentd versions fleet-wide. Security fixes only help if they are actually deployed.

How Netdata helps

Netdata collects Fluentd monitor_agent metrics per worker, so input emit_records rate spikes from a flooding sender are visible per aggregator instance rather than averaged away.
Correlating input rate with buffer_queue_length and buffer_available_buffer_space_ratios on one dashboard separates a flood-driven buffer fill from a destination-driven one: in a flood, input rate leads the queue growth.
Host-level network visibility shows which peers hold connections to the Fluentd process, making an unknown sender on 24224 visible without a manual ss session.
Process RSS trends catch the memory-exhaustion variant of an attack (decompression bombs, oversized events) before the OOM killer does.
Fluentd log collection lets you alarm on repeated TLS and auth handshake failures, which is what probing looks like before a successful injection.