Fluentd broken pipe / connection reset: dropped output connections and LB timeouts

Your Fluentd logs show broken pipe or Connection reset by peer during flush, retries start climbing, and chunks roll back into the buffer queue. The destination is not down. It answers health checks, other clients reach it fine, and the errors come and go in a pattern that looks almost random.

The usual explanation: Fluentd’s output plugin is holding a long-lived TCP connection to the destination, something in the middle (a load balancer, NAT gateway, firewall, or the destination itself) has an idle timeout, and it silently drops the connection after a period of inactivity. Fluentd does not find out until the next flush writes into the dead socket. The kernel returns EPIPE (“broken pipe”) or the peer returns RST (“connection reset by peer”), and the chunk goes back for retry.

The fix is to make Fluentd cycle or probe its connections faster than the middlebox times them out. This guide covers how to confirm the mechanism, how to tell it apart from real destination failures, and how to tune keepalive so it stops happening.

What this means

Fluentd output plugins that speak TCP (out_forward, and most TCP-based outputs) can reuse connections across flushes rather than opening a new one per chunk. Reuse avoids repeated handshake and TLS negotiation cost. The tradeoff: the connection can sit idle between flushes. With flush_interval 60s and bursty traffic, a connection can easily sit idle for minutes.

Any stateful device on the path tracks that connection in a table. When its idle timer expires, it evicts the entry. Many load balancers do this silently, sending no FIN or RST to either side. Fluentd’s socket still looks ESTABLISHED locally, and the next flush writes into the void.

flowchart LR
  F[Fluentd out_forward] -- "keepalive connection, idle between flushes" --> LB[Load balancer]
  LB -- "forwards" --> D[Destination]
  LB -. "idle timer expires, silently evicts flow" .-> X[Connection state dropped]
  F -- "next flush writes to dead socket" --> E[broken pipe / connection reset]
  E -- "chunk rolled back" --> R[retry_count climbs, queue grows]

Two socket-state signatures help you confirm who closed what:

• CLOSE_WAIT on the Fluentd host toward the destination: the remote side sent FIN and closed its end, but Fluentd has not closed its end yet. A growing CLOSE_WAIT count to the destination IP means the destination (or LB) is actively closing connections Fluentd thinks are still usable.
• Silent drop with no FIN: the connection looks ESTABLISHED on both ends but packets go nowhere. This does not show up in ss output; it surfaces as EPIPE or a write timeout on the next flush.

Common causes

Quick checks

All read-only and safe to run during an incident.

# 1. Find the actual error lines and their cadence
grep -E "broken pipe|Connection reset|failed to flush" \
  /var/log/td-agent/td-agent.log | tail -30
# fluent-package installs log to /var/log/fluent/fluentd.log instead.
# Regular spacing between errors points at an idle timeout.

# 2. Count socket states toward the destination
ss -tn dst <destination_ip> | awk '{print $1}' | sort | uniq -c
# A high CLOSE_WAIT count means the remote side is closing connections
# Fluentd has not acknowledged. That is the smoking gun for
# destination/LB-initiated closes.

# 3. Check retry and rollback state on the output
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") |
      {id: .plugin_id, retries: .retry_count, rollbacks: .rollback_count,
       writes: .write_count, retry_state: .retry}'
# retry_count rising in step with the log errors, write_count still
# incrementing between errors = transient connection drops, not a dead
# destination.

# 4. Check whether a flush thread is spinning (keepalive reuse bug)
ps -T -p $(pgrep -f fluentd | head -1) -o spid,%cpu,comm
# One thread pinned at ~100% CPU alongside broken pipe errors is the
# out_forward keepalive reuse bug, not an LB timeout. See fixes below.

# 5. Confirm the destination is independently healthy
nc -vz -w 3 <destination_host> <port>
# Fast connect = destination reachable. The problem is connection
# lifecycle, not reachability.

# 6. Check flush activity per output
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") |
      {id: .plugin_id, flush_time_count: .flush_time_count,
       writes: .write_count}'
# flush_time_count is cumulative; delta(flush_time_count)/delta(write_count)
# gives average flush time. Compare write cadence with flush_interval:
# long gaps between flushes on low-volume tags are when idle timeouts bite.

How to diagnose it

1. Establish the cadence. Pull the timestamps of the broken pipe / reset errors. If they cluster at a fixed idle interval after the last successful flush (for example, consistently 300-360 seconds of idle time before a failure), you are looking at an idle timeout, not random network trouble. Random scatter suggests overload or conntrack pressure instead.

2. Identify the idle timeout on the path. Find every stateful device between Fluentd and the destination: cloud load balancers, NAT gateways, service mesh sidecars, firewalls. Each has an idle timeout and the smallest one wins. A well-known example: AWS NLB has a fixed 350-second idle timeout for TCP flows. If your errors arrive right around that mark after idle periods, that is your culprit. For other LBs, check the vendor config.

3. Confirm direction of close with socket states. Run ss -tn dst <destination_ip> repeatedly over a few minutes. If CLOSE_WAIT sockets accumulate on the Fluentd side, the peer is sending FIN and Fluentd is not cleaning up its half. If sockets simply vanish and errors appear on the next flush with no FIN observed, the middlebox dropped the flow silently.

4. Rule out destination overload. Check slow_flush_count and average flush time (delta(flush_time_count) / delta(write_count)). Idle-timeout drops produce errors with normal flush latency in between. Overload produces rising flush latency first, errors second. Average flush time should stay under half of flush_interval.

5. Rule out the keepalive reuse bug. If you already have keepalive true on out_forward and you see broken pipes plus one flush thread at 100% CPU plus CLOSE_WAIT growth, this is a known Fluentd bug: out_forward could pick a remotely-closed socket out of its keepalive cache and busy-loop writing to it. It was fixed upstream (PR #5309, backported to the v1.19 line) and the fix is present in v1.19.3. If you are on an affected version, upgrading is the fix; tuning will not help.

6. Decide which fix applies based on whether the timeout is on a middlebox you cannot change, the destination, or Fluentd’s own defaults.

Metrics and signals to monitor

Fixes

Set keepalive on the output, shorter than the LB idle timeout

For out_forward, enable connection keepalive and bound the connection lifetime so Fluentd cycles connections before the middlebox evicts them:

<match **>
  @type forward
  <server>
    host aggregator.example.com
    port 24224
  </server>
  keepalive true
  keepalive_timeout 300s   # must be comfortably below the LB idle timeout
  <buffer>
    @type file
    path /var/log/td-agent/buffer/forward
  </buffer>
</match>

keepalive_timeout defaults to nil, meaning connections are reused indefinitely until the remote closes them. Behind an LB, always set an explicit value below the smallest idle timeout on the path. For AWS NLB’s 350s timeout, 300s leaves margin. This trades a small amount of extra connection setup for never writing into a dead socket.

Enable TCP keep-alive probes for silently dropped flows

If the middlebox drops flows without FIN (socket looks ESTABLISHED but is dead), application-level connection cycling may not be enough on its own. The socket helper option send_keepalive_packet true turns on TCP keep-alive (SO_KEEPALIVE), so the kernel both probes dead peers and generates traffic that keeps middlebox flow entries warm.

Two caveats:

• This alone is not sufficient. TCP keep-alive timing is governed by kernel parameters: net.ipv4.tcp_keepalive_time (default 7200 seconds), net.ipv4.tcp_keepalive_intvl, and net.ipv4.tcp_keepalive_probes. The two-hour default is far longer than any LB idle timeout, so you must lower tcp_keepalive_time below the LB timeout (via sysctl on the Fluentd host or pod sysctls in Kubernetes) for the probes to fire in time.
• Kernel tcp_keepalive settings are host-wide or netns-wide. Lowering them affects every socket in that namespace. Usually harmless, but know what you are changing.

Fix the receiving side when Fluentd is also the destination

If your topology is Fluentd-to-Fluentd over the forward protocol, the receiving in_forward on non-Windows closes connections with RST rather than a clean FIN by default (linger behavior). Senders then see connection reset by peer even on intentional closes. Setting a non-zero linger_timeout in the <transport tcp> section of in_forward changes this to a FIN-based close, which senders handle more gracefully.

Upgrade if you hit the keepalive reuse bug

If your symptoms match the reuse bug (keepalive enabled, one flush thread at 100% CPU, CLOSE_WAIT growth), upgrade to a release containing the fix (v1.19.3 or later on the v1.19 line). No configuration change works around it reliably; the only other lever is disabling keepalive, which takes you back to per-flush connections and removes the cached-socket path entirely at the cost of handshake overhead.

If the destination is actually overloaded

Do not confuse the two. If average flush time is rising and slow_flush_count increments, the resets are backpressure from the destination, not idle timeouts. Keepalive tuning will not fix that. See Fluentd failed to flush the buffer and Fluentd buffer queue length growing.

Prevention

• Always set an explicit keepalive_timeout on long-lived outputs deployed behind any load balancer or NAT. Never rely on the nil default in that topology.
• Inventory idle timeouts on the data path and document the smallest one. Treat it as a hard constraint on both keepalive_timeout and kernel tcp_keepalive_time.
• Alert on retry_count and rollback_count rates, not just buffer fill. Connection-drop errors are visible there minutes to hours before the buffer is at risk.
• Track CLOSE_WAIT toward destinations as a host-level signal. It is the earliest direct evidence of peer-initiated closes and of the reuse bug.
• Keep Fluentd current on the v1.19 line; connection-lifecycle bugs in out_forward keepalive are actively being fixed.
• Load-test idle behavior, not just throughput. A pipeline benchmarked under constant load never exercises the idle path where these failures live.

How Netdata helps

• Netdata collects the Fluentd monitor_agent counters per output plugin, so retry_count, rollback_count, write_count, and flush_time_count are graphed together. Broken-pipe incidents show a characteristic fingerprint: retries stepping up while write rate stays mostly intact and average flush time stays flat.
• Per-second host-level socket metrics let you watch CLOSE_WAIT and ESTABLISHED counts toward the destination in the same dashboard as Fluentd’s internal counters, which is the correlation that distinguishes idle-timeout drops from destination overload.
• Per-thread and per-process CPU visibility catches the keepalive reuse busy-loop immediately, instead of being discovered after hours of 100% CPU.
• Alerts on sustained retry_count and on buffer queue growth give you warning while the pipeline is still recovering between drops, well before overflow_action becomes relevant.
• Because buffer and retry metrics are per plugin, you can tell which output (and therefore which network path) is affected when only one destination sits behind an LB.

Related guides

• Fluentd failed to flush the buffer: the output cannot deliver and retries begin
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd emit_error_count: the number-one under-monitored data-loss signal
• Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss
• Fluentd monitoring checklist: the signals every production log pipeline needs
• How Fluentd actually works in production: a mental model for operators