Fluentd retry backoff: a pipeline that is 'retrying' but effectively dead

The Fluentd process is up. The monitor agent responds. retry_count is non-zero, which you already knew, because the destination had a bad night. What retry_count does not tell you is that the next retry attempt is scheduled 4 hours from now. No data is flowing, none will flow for hours, and every dashboard that only tracks the retry counter shows the same flat number it showed an hour ago. The pipeline is technically retrying and operationally dead.

This is the default behavior of Fluentd’s exponential backoff, not a bug. With retry_wait 1s, retry_exponential_backoff_base 2, no retry_max_interval, and the default retry_timeout 72h, the wait between attempts roughly doubles every step. By step 10 the interval is about 8.5 minutes. By step 15 it is about 4.5 hours. The pipeline sits in this state until a retry succeeds, the timeout discards the chunk, or someone intervenes.

This article covers how to see the actual retry state (not the counter), how to tell a stalled pipeline from a recovering one, and how to recover without making things worse.

What this means

Fluentd’s retry engine is a per-output state machine, not a simple counter. The monitor agent exposes two different things:

• retry_count: a cumulative counter of retry error events since process start. It tells you failures happened. It tells you nothing about the current state.
• The retry object: the live state of the active retry cycle, with retry.start (when the cycle began), retry.steps (which backoff step you are on), and retry.next_time (the absolute timestamp of the next attempt).

The failure mode in this article is visible in the second and invisible in the first. When retry.steps is high and retry.next_time is far in the future, three things are true at once:

1. No flush attempts are happening, so write_count is flat.
2. New events keep arriving and accumulating in the buffer.
3. Even if the destination recovers right now, Fluentd will not notice until retry.next_time arrives. Recovery is delayed by the backoff, not by the destination.

A restart resets all retry state (retry.start is set to the current time on initialization), which is why a restart is the standard recovery lever. It is also why long retry stalls are often misdiagnosed: someone restarts Fluentd, the pipeline recovers, and nobody looks at why the backoff was allowed to grow to hours.

One more consequence: when the retry limit is finally reached (retry_timeout exceeded, or retry_max_times hit if configured), the queued chunks are discarded, or written to the <secondary> output if one is configured. A pipeline that was “retrying” for 60 hours can end in silent data loss.

flowchart LR
  A[Flush fails] --> B[Retry step 1: wait ~1s]
  B --> C[Step 10: wait ~8 min]
  C --> D[Step 15: wait ~4.5 h]
  D --> E{Destination fixed?}
  E -->|Yes, but waits anyway| F[Still stalled until next_time]
  E -->|Timeout or max times hit| G[Chunks discarded or sent to secondary]
  F --> H[Restart resets retry state]
  H --> B

Common causes

Version note: the exponential backoff interval and total-timeout calculation were wrong before Fluentd v1.14.6 (the first interval used the wrong exponent, and the timeout math was off). If you are on an older version and the observed intervals do not match retry_wait * base^(steps - 1), that is a known bug. Also note v0.12 used different parameter names (retry_limit default 17, disable_retry_limit) than v1.x (retry_max_times default nil, retry_forever default false).

Quick checks

All read-only. Paths shown for td-agent; adjust for fluent-package (/var/log/fluent/fluentd.log, /etc/fluent/fluentd.conf) or your ConfigMap.

# 1. Get the live retry state for every output, not just the counter
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") |
      {id: .plugin_id, retry_count, retry: .retry}'

# 2. Which backoff step is each output on, and when is the next attempt?
# Compare retry.next_time against "date" on the host.
curl -s http://localhost:24220/api/plugins.json | \
  jq -r '.plugins[] | select(.plugin_category=="output" and .retry != null) |
      "\(.plugin_id) steps=\(.retry.steps) next_time=\(.retry.next_time)"'

# 3. Confirm the stall: write_count flat while queue grows (run twice, 60s apart)
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") |
      {id: .plugin_id, write_count, queue: .buffer_queue_length,
       avail_pct: .buffer_available_buffer_space_ratios}'

# 4. What is actually failing? Look for the first, specific error
grep -E "failed to flush|retry|temporarily failed" /var/log/td-agent/td-agent.log | tail -30
grep -iE "(tls|ssl|auth|401|403|unauthorized|certificate)" /var/log/td-agent/td-agent.log | tail -10

# 5. Check the retry configuration actually in force
curl -s "http://localhost:24220/api/plugins.json?with_config=true" | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, config: .config}'

# 6. How much runway before the buffer overflows?
# Track buffer_total_queued_size growth over two samples to get the fill rate
# (see the time-to-overflow guide linked below for the full calculation)
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") |
      {id: .plugin_id, total_bytes: .buffer_total_queued_size, avail_pct: .buffer_available_buffer_space_ratios}'

If .retry is null for an output, that output is not currently in a retry cycle, which is itself useful information.

How to diagnose it

1. Confirm the stall, not just the failures. Pull the retry object (check 1). If retry.next_time is more than a few minutes in the future and write_count has not moved between two samples, the pipeline is in the dead-but-retrying state. A retry.next_time more than 10 minutes out means recovery will be delayed even after the destination is fixed.

2. Find the root error. retry_count tells you that it is failing; the logs tell you why. Look for the first occurrence of the error, not the thousandth retry warning. Auth and TLS failures often appear once, early, and are then masked by generic retry messages.

3. Check destination health independently. Do not trust Fluentd’s view. Query the destination’s own monitoring, or curl the endpoint from the Fluentd host. If the destination has been healthy for an hour but retry.next_time is still 3 hours out, you are waiting on backoff, not on the destination.

4. Estimate buffer runway. With the output stalled, the buffer is filling at the input rate. Compute time-to-overflow from buffer_available_buffer_space_ratios and the growth rate of buffer_total_queued_size. This tells you whether you have minutes or days, and therefore whether you fix the destination first or shed load first.

5. Check what happens at the end of the retry cycle. Look at the effective config: is retry_forever set? Is retry_max_times set? Is there a <secondary> block? If none of those, the chunks will be discarded when retry_timeout (default 72h) is reached. Know which outcome you are headed toward before deciding whether to wait or restart.

Metrics and signals to monitor

The single most important habit: alert on retry.next_time - now, not on retry_count. A threshold like “next retry more than 10 minutes out for more than 5 minutes” catches the dead-retrying state hours before a buffer-overflow alert would.

Fixes

Fix the destination first

Restarting Fluentd while the destination is still broken just restarts the backoff clock from step 0. Confirm the destination accepts writes before touching Fluentd.

Restart to reset retry state

Once the destination is healthy, a Fluentd restart resets all retry state: retry.steps goes to zero and the first retry happens after ~retry_wait instead of hours. This is the fastest path to recovery from a deep backoff.

Two cautions. First, if any output uses a memory-backed buffer, all unflushed data in it is lost on restart. File-backed buffers survive and replay. Verify your buffer type (/api/plugins.json?with_config=true) before restarting. Second, if the root cause is a permanent error (bad credentials, wrong index name), the restart only buys you one fast failure cycle; the pipeline will re-enter backoff immediately.

Cap the backoff with retry_max_interval

This is the permanent fix, and the one the official troubleshooting guide recommends. Without it, the interval doubles without bound until retry_timeout:

<match **>
  @type elasticsearch
  # ... destination config ...
  <buffer>
    @type file
    path /var/log/td-agent/buffer/es
    retry_max_interval 5m
  </buffer>
</match>

With retry_max_interval 5m, the worst-case delay between attempts is 5 minutes, so a recovered destination is picked up within minutes instead of hours. The tradeoff: more frequent attempts against a struggling destination. retry_randomize (default true) adds a small random jitter to each wait interval, which keeps many Fluentd instances from retrying in lockstep; leave it on, especially in DaemonSet deployments.

Bound the retry cycle deliberately

Decide what should happen when the destination stays down:

• retry_timeout (default 72h): how long a chunk is retried before being discarded. 72 hours of “retrying” with an unbounded interval is exactly the dead-pipeline scenario. Lower it to match your actual recovery SLA.
• retry_max_times (default nil): with the default timeout and exponential backoff, the effective maximum is roughly 18 attempts. Set it explicitly if you want a predictable attempt budget.
• <secondary>: a fallback destination for chunks that exhaust retries. Without it, exhausted chunks are discarded with only a log message. retry_forever true retries recoverable errors indefinitely but still uses the secondary for unrecoverable ones, so the two are not mutually exclusive.

Do not widen the buffer as a substitute

Increasing total_limit_size buys time but does not fix the stall; it just delays overflow. Use the runway calculation to size the buffer for your real destination-outage tolerance, and treat anything beyond that as a destination problem, not a Fluentd problem.

Prevention

• Set retry_max_interval on every production output. This single directive eliminates the dead-retrying failure mode. There is no good reason to let the interval grow past your on-call response time.
• Alert on retry state, not the retry counter. Track retry.next_time - now and retry.steps per output. Keep retry_count as a delta-based corroborating signal.
• Configure a <secondary> output for any destination where discarding chunks is unacceptable, and monitor write_secondary_count so you know when it fires.
• Choose retry limits explicitly. Do not inherit the default 72h retry_timeout and nil retry_max_times by accident. Write down what you want to happen at hour 1, hour 6, and hour 24 of a destination outage, and encode it.
• Use file-backed buffers so that the recovery restart does not itself become a data-loss event.
• Keep Fluentd current. The backoff calculation bug fixed in v1.14.6 means older versions do not even wait the intervals they log.

How Netdata helps

• Netdata collects the Fluentd monitor agent per output plugin, so retry_count, write_count, rollback_count, and buffer gauges are time-series, not point-in-time curl samples. You can see the exact moment write_count went flat.
• Correlating flat write_count with rising buffer_queue_length and falling buffer_available_buffer_space_ratios on one dashboard turns “is it stalled?” into a glance, and gives you the growth rate you need for time-to-overflow math.
• Per-second collection catches short retry cycles that infrequent polling misses between samples, distinguishing flapping destinations from hard-down ones.
• write_secondary_count and drop_oldest_chunk_count alerts tell you when a stalled retry cycle has crossed from “delayed” into “data loss”.
• Historical retention lets you verify post-incident whether the backoff intervals matched your configured retry policy, which is how you catch missing retry_max_interval and version bugs.

Related guides

• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss
• Fluentd emit_error_count: the number-one under-monitored data-loss signal
• How Fluentd actually works in production: a mental model for operators
• Fluentd memory vs file buffer: why the default buffer loses data on restart
• Fluentd monitor_agent not responding: a process that is up but hung
• Fluentd monitoring checklist: the signals every production log pipeline needs
• Fluentd monitoring maturity model: from survival to expert