Fluentd failed to flush the buffer: the output cannot deliver and retries begin

Your Fluentd log starts emitting lines like this, over and over:

[warn]: #0 failed to flush the buffer. retry_time=3 next_retry_seconds=2026-07-21 22:50:11 +0000 chunk="5e1a2b..." error_class=Net::OpenTimeout error="execution expired"

An output plugin tried to flush a buffer chunk to its destination and failed. Fluentd did not lose the chunk; it rolled the chunk back into the queue and scheduled a retry with exponential backoff. The warning repeats once per failed attempt, with retry_time climbing and next_retry_seconds drifting further into the future.

The log line almost always carries the root cause in error_class and error. But while retries cycle, new events keep arriving and the buffer queue grows. If the buffer fills before the destination recovers, your overflow_action decides whether you block inputs or lose data.

What this means

Every output plugin sits behind a buffer. Events accumulate in chunks, chunks move from staged to queued, and flush threads deliver queued chunks to the destination. When a flush raises an exception (connection refused, auth rejection, timeout, TLS failure), the chunk goes back to the queue and the retry engine takes over.

The retry engine works per chunk:

• Each failure increments the retry step and logs the failed to flush the buffer warning.
• The wait between attempts grows exponentially (retry_type exponential_backoff is the default, with jitter from retry_randomize true).
• Retries continue until the chunk succeeds, retry_max_times is hit, or retry_timeout (default 72 hours) expires.
• On exhaustion, the chunk is discarded, written to the <secondary> output if one is configured, or (on Fluentd v1.19.0 and later with file buffers) evacuated to ${root_dir}/buffer/${plugin_id}/ for later recovery (when no <system> root_dir is configured the fallback is /tmp/fluent, overridable via FLUENT_BACKUP_DIR). On older versions without a secondary, exhaustion means silent data loss with only a log message.

flowchart LR
  A[Chunk queued] --> B[Flush attempt]
  B -->|success| C[Purged - write_count +1]
  B -->|failure| D[Rollback - rollback_count +1]
  D --> E[Retry scheduled with backoff]
  E -->|next attempt| B
  E -->|retry_max_times or retry_timeout hit| F[Discard / secondary / evacuate]
  F --> G[Data loss unless secondary or v1.19.0+ evacuation]

One stalled chunk is noise. The operational problem is the pattern: retry_count climbing, write_count flat, and buffer_queue_length growing because every new chunk joins the backlog behind the failing one.

Common causes

The error_class and error fields are your primary evidence. Everything else confirms what they already told you.

Quick checks

All of these are read-only.

# 1. Find the failing output and its error
grep "failed to flush the buffer" /var/log/fluent/fluentd.log | tail -20
# td-agent package: /var/log/td-agent/td-agent.log

# 2. See the full error context around the warning
grep -E "failed to flush|retry|temporarily failed|broken pipe" \
  /var/log/fluent/fluentd.log | tail -30

# 3. Per-output retry state and counters
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") |
      {id: .plugin_id, retries: .retry_count, rollbacks: .rollback_count,
       writes: .write_count, queue: .buffer_queue_length,
       avail_pct: .buffer_available_buffer_space_ratios}'

# 4. Current retry schedule (steps and next attempt time)
# Requires include_retry true in the monitor_agent source on v1.19.3+.
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output" and .retry != null) |
      {id: .plugin_id, retry: .retry}'

# 5. Test destination reachability from the Fluentd host
curl -sv --max-time 5 https://<destination-host>:<port>/ -o /dev/null

# 6. Check buffer disk consumption (file-backed buffers)
du -sh /var/log/fluent/buffer/
df -h /var/log/fluent/

Notes on these checks:

• The monitor_agent endpoint requires <source> @type monitor_agent </source> in the config. In multi-worker mode, each worker gets its own port (24220, 24221, …).
• On Fluentd v1.19.3 and later, retry details in the API response require include_retry true in the monitor_agent source configuration, and URL query parameters such as ?with_retry= no longer work; versions before v1.19.3 included the retry object by default.
• Check 4 tells you something retry_count alone cannot: if retry.next_time is 20 or 30 minutes out, the pipeline is effectively stalled even though it is technically “retrying.”

How to diagnose it

1. Identify which output is failing. The warning line includes the plugin id (#0, or a named @id if you set one). Match it to the <match> block in your config to learn which destination is involved.

2. Read the error class, not just the warning. Connection refused points at a down service or firewall. 401/403 points at credentials. 429 points at destination overload. Broken pipe on an otherwise healthy destination points at idle connections killed by an intermediate load balancer. The error string is the shortest path to the cause.

3. Confirm the stall in metrics. Pull monitor_agent output twice, 60 seconds apart. If retry_count and rollback_count increment while write_count does not, the output is fully stalled. If write_count still increments occasionally, the destination is intermittently reachable and you are looking at overload or flapping rather than a hard outage.

4. Check the retry schedule. Look at the retry object: start, steps, next_time. A high step count with next_time far in the future means exponential backoff has pushed recovery far out. Even if you fix the destination now, delivery will not resume until the next scheduled attempt.

5. Check destination health independently. Do not trust Fluentd’s view alone. Query the destination’s own status and test the network path from the Fluentd host. If the destination looks healthy from elsewhere but not from this host, suspect DNS, firewall, or TLS on the path.

6. Measure your runway. Watch buffer_available_buffer_space_ratios and buffer_queue_length. Time to overflow is remaining buffer space divided by the current growth rate. That number decides whether you have hours to fix the destination calmly or minutes before overflow_action fires.

Metrics and signals to monitor

One behavioral caveat: retry_count is cumulative. On current Fluentd versions it clears on process restart, not on recovery, so it does not fall back to zero when the destination comes back. Alert on the delta and on the retry object, not on the absolute value.

Fixes

Fixes depend on the cause in the error string. Restarting Fluentd is not a fix; it only resets retry state.

Destination down or unreachable

Restore the destination, or the network path to it. Fluentd recovers on its own at the next scheduled retry. If backoff has pushed retry.next_time far out and you need delivery to resume immediately, a Fluentd restart resets retry state. That is disruptive: file-backed buffers replay on restart, which produces a burst of duplicate delivery downstream. Weigh that against waiting for the next scheduled attempt.

Authentication failure

Rotate the credential in the Fluentd config (API key, password, IAM role, service account) and reload. Auth errors often appear once with a clear message and then get masked by generic retry warnings, so grep back further than the last few minutes if the current lines are vague.

TLS or certificate expiry

Renew the expired certificate on whichever end failed verification. Certificate expiry is a cliff-edge failure: everything works until one exact moment, then all flushes fail at once. Check expiry dates proactively and alert at 30, 7, and 1 days out.

Destination overload (429, rejected execution)

Reduce pressure or increase capacity. On the Fluentd side, lower flush_thread_count or increase retry_wait and retry_max_interval to back off more gently. On the destination side, scale the cluster or raise its limits. Also check payload size: a chunk serialized from MessagePack to JSON can grow substantially, and an oversized bulk request gets rejected even when chunk sizing looks reasonable.

Broken pipe behind a load balancer

If a load balancer or firewall between Fluentd and the destination closes idle connections, long-lived output connections die silently and the next flush fails with Broken pipe. Enable keepalive on the output plugin and set the keepalive interval shorter than the intermediary’s idle timeout.

Retry exhaustion and data safety

If chunks are approaching retry_timeout (default 72 hours) or retry_max_times:

• Configure a <secondary> output so exhausted chunks fall through to a backup destination instead of being discarded. Watch write_secondary_count so you know when this happens.
• On Fluentd v1.19.0 and later with file buffers, exhausted chunks are evacuated to ${root_dir}/buffer/${plugin_id}/ rather than discarded, so you can recover them after the destination is fixed. This is disabled when a secondary is configured or retry_forever is set. On older versions, exhausted chunks are simply dropped.

Prevention

• Alert on the stall, not just the retry. The firing condition that matters is delta(retry_count) > 0 combined with delta(write_count) == 0 and growing buffer_queue_length. Retry count alone produces noise from transient blips that resolve in one or two attempts.
• Monitor the retry object. Track retry.next_time. A pipeline whose next attempt is 30 minutes away is effectively down, and process-level checks will not tell you.
• Track buffer runway. Alert when buffer_available_buffer_space_ratios drops below 20% while still filling, and treat sub-5% as imminent overflow. Compute time-to-overflow from the growth rate so you know how much time you actually have.
• Set overflow_action deliberately. The default throw_exception drops new events silently when the buffer fills, with no reliable counter. Choose block (backpressure to inputs) or drop_oldest_chunk (visible loss via drop_oldest_chunk_count) consciously per output.
• Configure a secondary for outputs you cannot afford to lose. It converts retry exhaustion from silent data loss into a rerouting event you can measure.
• Watch certificate expiry and credential rotation calendars. A large share of “failed to flush” incidents that are not destination outages are expired certs or rotated keys that nobody propagated to the Fluentd config.

How Netdata helps

• Netdata’s Fluentd collector scrapes monitor_agent and charts retry_count, write_count, rollback_count, and buffer metrics per output plugin, so the stall pattern (retries rising, writes flat) is visible on one dashboard without manual curl | jq loops.
• Per-second collection catches short retry cycles that a slow scrape interval would miss entirely, which matters when a destination is flapping rather than hard down.
• Buffer queue length and available space ratio are charted over time, so you can see the fill rate and extrapolate time-to-overflow directly from the slope.
• Anomaly detection on write_count flags a delivery stall even when retry counters look unremarkable, which helps catch intermittent failures early.
• Correlating Fluentd output metrics with the destination’s own metrics (Elasticsearch indexing rate, broker health) on the same screen shortens the “is it Fluentd or the destination?” question to a glance.

Related guides

• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd overflow_action: throw_exception, block, and drop_oldest_chunk
• Fluentd memory vs file buffer: why the default buffer loses data on restart
• Fluentd monitor_agent not responding: a process that is up but hung
• How Fluentd actually works in production: a mental model for operators
• Fluentd monitoring checklist: the signals every production log pipeline needs