Fluentd retry_count climbing: the destination is rejecting or unreachable

You pulled /api/plugins.json from the monitor_agent and one of your output plugins shows a nonzero retry_count, and it keeps going up. At the same time, write_count has stopped incrementing and buffer_queue_length is growing. Fluentd itself is alive and inputs are still collecting.

This is the destination-unavailable failure pattern: the output plugin cannot deliver chunks, the retry engine has taken over, and Fluentd is now in exponential backoff against a destination that is rejecting connections, rejecting data, or simply gone. Every event that arrives from now on accumulates in the buffer. The clock you are racing is buffer capacity, not the retry count itself.

The dangerous part is that this state can persist for a long time before anything looks “down.” The process is up. Inputs are up. Retries are nominally a recovery mechanism. But with exponential backoff, each failure pushes the next attempt further into the future, and a pipeline that is “retrying” can be effectively stalled for tens of minutes at a time.

What this means

When a chunk flush fails, the output plugin does not drop the chunk. It rolls the chunk back into the queue and the retry engine schedules another attempt. With the default retry_type exponential_backoff, the wait between attempts grows with each failure. If retry_max_times is not set and retry_forever is false, retries are bounded by retry_timeout (default 72 hours), after which the chunk is discarded: data loss, with only a log line to mark it.

Two things in the monitor_agent response describe this state, and they are not interchangeable:

• retry_count is a cumulative counter of retry error occurrences on that output plugin. It tells you failures have happened, and whether the rate of failures is ongoing. In current Fluentd it does not reset to zero on success; it clears on process restart. (Some older documentation and builds describe it as resetting on a successful flush, so verify against your version before you alert on the raw value.)
• The retry object (retry.start, retry.steps, retry.next_time) describes the live retry cycle: when it started, how many attempts have been made in this cycle, and when the next attempt fires. This is the field that tells you how bad things are right now.

That distinction matters operationally. A retry_count of 40 with no retry object present means past failures that have since recovered. A retry_count of 6 with retry.next_time 25 minutes in the future means the pipeline is stalled right now and getting worse.

flowchart LR
  A[Destination rejects or unreachable] --> B[Chunk flush fails]
  B --> C[Chunk rolled back to queue]
  C --> D[Retry scheduled with growing backoff]
  D -->|fails again| B
  D -->|succeeds| E[Queue drains]
  C --> F[buffer_queue_length grows]
  F --> G[Buffer approaches total_limit_size]
  G --> H[overflow_action: drop, block, or exception]

Common causes

Quick checks

All of these are read-only. Paths shown use the td-agent package layout; fluent-package uses /var/log/fluent/fluentd.log and /etc/fluent/fluentd.conf.

# 1. Per-output retry state and queue depth
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, retries: .retry_count, queue: .buffer_queue_length, writes: .write_count, avail_pct: .buffer_available_buffer_space_ratios}'

# 2. The live retry object (start, steps, next_time)
# On v1.19.3+ add 'include_retry true' to the monitor_agent source config first:
# the ?with_retry=true query parameter stopped working in that release.
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, retry: .retry}'

# 3. Recent output errors from Fluentd's own log
grep -E "failed to flush|retry|temporarily failed|could not connect|broken pipe" \
  /var/log/td-agent/td-agent.log | tail -20

# 4. Auth and TLS rejections
grep -iE "(tls|ssl|auth|401|403|unauthorized|forbidden|certificate)" \
  /var/log/td-agent/td-agent.log | tail -20

# 5. Is the destination reachable at all from this host?
curl -s -o /dev/null -w "%{http_code}\n" --max-time 5 https://your-destination-endpoint/

Two notes on the checks above. First, the retry object is not guaranteed to be in the response: from v1.19.3 onward, monitor_agent’s include_retry defaults to false, and the ?with_retry=true query parameter no longer overrides it (fluent/fluentd PR #5392). If the retry field is missing, set include_retry true in your monitor_agent source configuration rather than assuming there is no active retry. Second, in multi-worker mode each worker has its own monitor_agent port (24220, 24221, and so on): query every worker before concluding a retry is or is not happening.

How to diagnose it

1. Confirm the pattern, not just the counter. Take two samples of retry_count, write_count, and buffer_queue_length a minute apart. The active-failure signature is: retry_count incrementing, write_count flat, buffer_queue_length growing. If write_count is incrementing and the queue is draining, you are looking at a recovered failure and a stale cumulative counter, not an incident.

2. Read the retry object. retry.steps tells you how deep into the backoff curve this chunk is. retry.next_time tells you when Fluentd will try again. If next_time is more than a few minutes out, delivery latency is already severe and will keep growing even if you fix the destination this second, because the scheduled attempt is still far away.

3. Get the actual error from the logs. The counters tell you that flushes fail; the log tells you why. Match the error to the cause table above: connection refused points at the destination or network, 401/403 at credentials, 429 at throttling, TLS errors at certificates. Auth errors often appear once at connection setup and are then masked by generic retry messages, so grep wider than the last few lines.

4. Verify the destination independently. From the Fluentd host, probe the destination endpoint and check the destination’s own health signals (Elasticsearch cluster state, Kafka broker status, S3 error rates). Distinguish “destination is down” from “destination is up but rejecting this data,” because the fixes are completely different.

5. Compute your runway. With the output stalled, the buffer fills at roughly the input rate. Check buffer_available_buffer_space_ratios and the growth of buffer_total_queued_size to estimate time to overflow. When the buffer hits total_limit_size, overflow_action fires: throw_exception (the default) raises BufferOverflowError, which for most inputs means the new event is dropped; drop_oldest_chunk discards old data; block stalls your inputs. None of these are good; all of them are worse than the retry state you are in now.

Metrics and signals to monitor

Fixes

Destination down or unreachable

Fix the destination; there is no Fluentd-side fix for a dead endpoint. While it is down, your job is to buy buffer time: verify buffer_available_buffer_space_ratios is draining slowly enough to outlast the outage, and confirm your overflow_action is the failure mode you actually want when the buffer fills. If you have a <secondary> output configured, chunks that exhaust retries fall through to it instead of being discarded; watch write_secondary_count.

Authentication or TLS failure

Rotate the credential or renew the certificate in the Fluentd configuration and reload. Auth failures never self-heal: retries against a bad credential just generate backoff. After the fix, see the note below about resetting retry state, because the backoff schedule does not care that you fixed the cause.

Throttling or overload (HTTP 429, quota exceeded)

Reduce pressure on the destination or slow Fluentd’s delivery. Check slow_flush_count and average flush time: if flushes were degrading before retries started, the destination was already at capacity. Longer term, the destination needs more capacity or the pipeline needs less volume; retry tuning alone does not fix a throughput deficit.

Recovering from deep backoff

This is the counterintuitive part. Once you fix the destination, chunks sitting deep in exponential backoff may have retry.next_time far in the future, so recovery is slow even though the cause is gone. The standard reset is to restart Fluentd: retry counters and backoff state reset on restart, and file-backed buffers replay their chunks from disk. Expect a flush burst and a temporarily high queue while the backlog drains. That burst is normal; do not mistake it for a new problem. Do not restart before fixing the underlying cause, or you will re-enter backoff from step one.

If you run many Fluentd instances (a Kubernetes DaemonSet, for example), make sure retry_randomize true is in effect. Without jitter, every instance retries on the same backoff schedule, and the synchronized retry burst can knock over a destination that just recovered, starting the cycle again.

Retry tuning

Adjust retry_max_interval to cap how long the backoff can grow, and be deliberate about retry_timeout (default 72 hours) versus retry_forever. A shorter timeout bounds how long stale chunks occupy the buffer but guarantees discard when it expires. retry_forever never discards, but then the buffer is your only bound, and overflow becomes the data-loss path instead. Neither is universally right; pick based on whether losing old data or blocking new data is worse for your pipeline.

Prevention

• Alert on the delta, not the value. retry_count is cumulative and does not reliably return to zero. Alert on delta(retry_count) > 0 sustained, or better, on the presence of a retry object with retry.next_time beyond a threshold.
• Pair retry alerts with buffer state. A retry alert without buffer_queue_length and buffer_available_buffer_space_ratios context cannot tell you how urgent it is.
• Watch flush latency as the leading indicator. Rising flush_time_count / write_count and slow_flush_count show destination degradation before the first retry fires.
• Choose overflow_action explicitly per output. The default throw_exception drops new events at the buffer limit, and no counter reliably tracks those drops.
• Use file-backed buffers in production. They survive restarts, which matters both for durability and for making the retry-state reset procedure safe.
• Configure a <secondary> output for destinations where a multi-hour outage is plausible, so retry exhaustion routes to a fallback instead of the discard path.

How Netdata helps

• Netdata collects the Fluentd monitor_agent fields as time series, so retry_count, write_count, rollback_count, and buffer_queue_length appear on one timeline instead of in manual curl samples. The stall signature (retries up, writes flat, queue growing) is visible at a glance.
• Because Netdata computes rates from cumulative counters, you alert on delta(retry_count) rather than the raw value, which sidesteps the “counter never resets” trap.
• Buffer saturation metrics (buffer_available_buffer_space_ratios, buffer_total_queued_size) next to retry metrics let you estimate time-to-overflow while the destination is down.
• Flush latency derived from flush_time_count / write_count gives you the pre-retry degradation signal, so you see destination slowdowns before the first failure.
• Per-plugin and per-worker breakdowns keep a single failing output, or a single struggling worker in multi-worker mode, from hiding inside aggregates.

Related guides

• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss
• Fluentd emit_error_count: the number-one under-monitored data-loss signal
• Fluentd memory vs file buffer: why the default buffer loses data on restart
• Fluentd monitor_agent not responding: a process that is up but hung
• Fluentd monitoring checklist: the signals every production log pipeline needs
• How Fluentd actually works in production: a mental model for operators