$ guides / fluentd / fluentd-retry-storm ▌

Operations Guides

Fluentd retry storm: thundering-herd resonance that keeps a destination down

Your log destination had a hiccup. It recovered. Then it fell over again. And again. Each time it comes back, it survives for a minute or two, gets hit by a wall of buffered log traffic, and falls over. Fluentd’s own metrics look confusing: retries intermittently succeed, the buffer queue drains a little, then grows again. The destination team insists their service “works fine when we test it.”

This is a retry storm with resonance. After a destination failure, many Fluentd output plugins, or many Fluentd pods in a DaemonSet, enter retry at roughly the same time. Their backoff timers are aligned because they all started failing at the same moment. When the backoff expires, they all retry at once. The just-recovering destination, which has no capacity headroom yet, gets hammered by the synchronized burst and fails again. Retries reset, timers realign, and the cycle repeats. The destination is not the problem anymore. The retry behavior is.

This article covers how to recognize the oscillation pattern, how to confirm it with Fluentd’s monitor_agent API, and how to break the loop without losing buffered data.

What this means

Fluentd’s output plugins retry failed chunk flushes with exponential backoff. That is correct behavior for a single output talking to a struggling destination: back off, give the destination room, try again later. The failure mode appears when the retrying population is large and synchronized:

Many output plugins in one Fluentd instance fail at the same time because they share a destination.
Many Fluentd instances in a Kubernetes DaemonSet fail at the same time because they all talk to the same destination.
All of them started their backoff clocks at the same moment: the moment the destination went down.

Fluentd v1.x randomizes retry timing by default (retry_randomize true), but the jitter window is narrow: the retry interval is multiplied by a random factor between 0.875 and 1.125, a spread of only +/-12.5%. With 50 DaemonSet pods all retrying on a 60-second backoff, that jitter spreads the burst across roughly 15 seconds. To a destination that just came back up cold, that is still one burst.

The result is an oscillator, not a recovery:

flowchart LR
  A[Destination hiccup] --> B[All outputs enter retry at once]
  B --> C[Backoff timers aligned]
  C --> D[Destination recovers]
  D --> E[Synchronized retry burst]
  E --> F{Survives?}
  F -- "no: overloaded" --> G[Destination fails again]
  G --> B
  F -- "yes: drains backlog" --> H[Normal operation]

The distinguishing feature, and the thing that separates this from a plain destination outage, is that retries intermittently succeed. Some chunks flush during the brief up window. write_count ticks up, the queue drains slightly, then the destination falls over and everything rolls back again. A clean recovery shows steady drain. Resonance shows spiky, sawtooth drain.

Common causes

Cause	What it looks like	First thing to check
Synchronized DaemonSet retries	All pods show retry activity in the same time windows; destination load spikes periodically	Compare `retry_count` and `retry.next_time` across several pods
`retry_randomize` overridden to false	Retry attempts across outputs cluster at exact backoff boundaries	`GET /api/plugins.json?with_config=true` and check each output’s retry config
Default backoff too aggressive for the destination	Destination recovers but is hit within seconds by the first retry wave	Check `retry_wait` and whether `retry_max_interval` is set
Destination undersized for backlog drain	Destination is healthy at steady state but cannot absorb buffered backlog plus live traffic	Compare destination capacity to queued bytes across all Fluentd instances
No rate limiting at the destination	Destination accepts connections unboundedly, then collapses under load	Destination-side metrics: connection count, queue depth during the burst

A note on defaults: in Fluentd v1.x, retry_type defaults to exponential_backoff with retry_wait 1s and a backoff base of 2, so retry intervals grow 1s, 2s, 4s, and so on. retry_max_interval defaults to nil, meaning uncapped, so late-stage retries can be hours apart. retry_timeout defaults to 72 hours, after which chunks are discarded. The early retries are the storm risk; the late retries are a slow-recovery risk. You often have to fix both.

Quick checks

All checks assume the monitor_agent is enabled (<source> @type monitor_agent </source>, default port 24220). All are read-only.

# 1. Which outputs are retrying right now, and how often
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, retries: .retry_count, rollbacks: .rollback_count}'

# 2. Current retry state: how deep in backoff, and when is the next attempt
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | select(.retry != null) | {id: .plugin_id, steps: .retry.steps, next_time: .retry.next_time}'

# 3. Is the queue draining or oscillating? Sample twice, 60s apart
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, queue: .buffer_queue_length, bytes: .buffer_total_queued_size}'

# 4. Are writes succeeding intermittently (the resonance signature)?
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, writes: .write_count}'

# 5. Verify retry configuration actually applied to each output
curl -s "http://localhost:24220/api/plugins.json?with_config=true" | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, config: .config}'

Check 2 is the most diagnostic. In a resonance loop, retry.steps stays low (retries keep resetting after brief successes) while retry.next_time values across pods or outputs cluster tightly together. In backoff exhaustion after a long outage, retry.steps is large and retry.next_time is far in the future. Those are different problems with different fixes; see Fluentd BufferOverflowError for the exhaustion side.

For a DaemonSet, run checks 1 and 2 against several pods at the same moment. If their next_time values fall within a few seconds of each other, you have confirmed synchronization.

How to diagnose it

Confirm the destination is flapping, not down. Check the destination’s own health independently of Fluentd: its error rate, connection count, and CPU during the incident window. Resonance requires the destination to be intermittently up. If it has been hard down the whole time, this is a plain backpressure cascade, not a retry storm.
Sample write_count and buffer_queue_length over several minutes. You are looking for a sawtooth: queue drains partially, then grows again; writes increment in bursts, then flatline. Steady drain means normal recovery. Flat queue with zero writes means sustained outage.
Check retry alignment across the fleet. Poll retry.next_time on multiple DaemonSet pods within the same minute. Tight clustering (within the jitter window) confirms synchronized backoff.
Verify retry_randomize is actually in effect. The v1.x default is true, but older config templates carried forward sometimes explicitly set retry_randomize false. Check the applied config with ?with_config=true, not the file you think is deployed.
Estimate the burst size the destination must absorb. Sum buffer_total_queued_size across all instances pointing at this destination, plus the live input rate. If that total exceeds what the destination can ingest in the first minute after recovery, it will fall over again no matter how the retries are timed. The fix must then include destination-side capacity or rate limiting, not just retry tuning.
Rule out a permanent error masquerading as a storm. Authentication failures and schema rejections also produce retries, but they never intermittently succeed. If every retry fails with the same 401, 403, or mapping error, fix the credential or schema problem; no amount of backoff tuning will help. Check Fluentd’s logs for the specific error: grep -iE "(401|403|unauthorized|forbidden)" /var/log/td-agent/td-agent.log | tail -20 (adjust the path for fluent-package or your container log location).

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`retry_count` per output	Counts retry error events; the primary “destination is failing” signal	Oscillating increments across many outputs or pods in the same windows
`retry.steps` and `retry.next_time`	Shows current backoff depth and next attempt time	`next_time` values clustered tightly across the fleet
`write_count` rate	Tells you whether chunks are actually being delivered	Bursty increments alternating with flat periods (sawtooth)
`rollback_count`	Chunks dequeued, failed, and returned to the queue	Rising in the same windows as destination load spikes
`buffer_queue_length`	Backlog depth	Partial drain followed by regrowth, repeating periodically
`buffer_available_buffer_space_ratios`	Proximity to overflow during each failed cycle	Ratcheting downward cycle over cycle; each failed recovery loses headroom
`flush_time_count / write_count`	Average flush latency	Spikes during the burst windows, indicating the destination is struggling before it falls

The most dangerous property of a retry storm is that it compounds. Each failed cycle leaves the buffer fuller than the last, so each subsequent burst is bigger. Watch buffer_available_buffer_space_ratios across cycles: if each trough is lower than the previous one, you are on a countdown to overflow and data loss, not a stable oscillation.

Fixes

Break the synchronization

Verify retry_randomize true. This is the v1.x default, but confirm it in the applied config and remove any explicit retry_randomize false overrides from templates. Be aware of the limit: the default jitter is only +/-12.5%, which helps a handful of outputs but does not spread a 50-pod DaemonSet burst very much. Jitter alone is usually not sufficient.

Increase retry_wait. The default base interval is 1 second, which means the first retry wave hits the destination almost immediately after it recovers. Raising retry_wait (for example to 5s or 10s) gives the destination real recovery time before the first wave arrives, and scales the entire backoff ladder up.

Set retry_max_interval. Without a cap, exponential backoff grows unboundedly, which creates a second problem: after the storm is over, some chunks sit hours away from their next attempt. A cap such as 60s or 300s bounds worst-case delivery delay and also makes the retry cadence more uniform, which is easier to reason about at the destination.

Example buffer section for an output pointing at a fragile destination:

<buffer>
  retry_randomize true
  retry_wait 10s
  retry_max_interval 300s
</buffer>

Apply this with a rolling restart of the DaemonSet, not a simultaneous restart of all pods. A fleet-wide restart re-synchronizes everything: all pods replay their file buffers and start flushing at once, which is itself a thundering herd.

Reduce what the destination must absorb

Rate-limit at the destination. If the destination supports ingress rate limiting or connection caps, use them. A destination that degrades gracefully under overload (rejecting excess load quickly) recovers far better than one that accepts everything and collapses. Fluentd will retry the rejected chunks, which is the correct behavior.

Scale the destination for the drain, not the steady state. The load during recovery is live traffic plus the buffered backlog across every Fluentd instance. If the destination is sized only for steady state, it will fall over every time there is a backlog to drain. Either provision drain headroom or accept slower drain via tighter rate limits.

Stagger the retrying population. If you control the DaemonSet rollout, restarting pods in small batches during the incident desynchronizes their retry timers. This is an operational lever, not a config change, and it works immediately.

What not to do

Do not set retry_forever true or raise retry_timeout as a first response. That keeps chunks alive longer but does nothing about the storm; it just means the oscillation runs longer before data loss. Also note the failure at the other end: when retry limits are hit, chunks are discarded (or, on Fluentd v1.19.0+ with file buffers, evacuated to a backup directory under the buffer root for manual recovery). If you are approaching that point, treat it as an overflow incident and follow Fluentd BufferOverflowError.

Prevention

Set explicit retry parameters on every production output. Do not rely on defaults you have not reviewed: retry_randomize true, a retry_wait that reflects the destination’s real recovery time, and a retry_max_interval cap. Put these in your base config template so new outputs inherit them.
Use file-backed buffers. Memory buffers turn every storm cycle into an OOM risk as the backlog grows. See Fluentd memory vs file buffer.
Size buffers for the storm case. The backlog during an oscillating incident is larger than during a single outage because recovery keeps failing. Compute time-to-overflow under sustained failure and keep headroom; see Fluentd buffer available space low.
Alert on the resonance signature, not just on retries. A retry_count alert fires on any destination hiccup. Add detection for the oscillation: write_count rate alternating between zero and bursty, or buffer_queue_length failing to make net progress over 15 to 30 minutes while retries are active.
Track retry state, not just the counter. retry.steps and retry.next_time tell you whether the pipeline is cycling fast (storm) or parked in deep backoff (exhaustion). Both look like “retrying” if you only watch the counter.
Load-test destination recovery. Periodically verify the destination can absorb a full backlog drain. Most teams test steady-state throughput and never test the recovery burst, which is the actual failure condition.

How Netdata helps

Netdata collects Fluentd’s monitor_agent metrics per output plugin, so retry_count, rollback_count, write_count, and buffer_queue_length are visible on the same timeline, which is the correlation you need to see the sawtooth pattern.
Per-second collection granularity makes the burst structure visible. Coarser sampling smooths the oscillation into a flat line and hides the resonance.
Buffer available space ratio is tracked as a gauge over time, so the cycle-over-cycle ratchet toward overflow is visible before overflow_action fires.
In Kubernetes, per-pod views let you compare retry timing across DaemonSet instances to confirm fleet-wide synchronization rather than a single misbehaving pod.
Anomaly detection on write_count and queue depth flags the oscillation regime (repeating partial drains) even when no static threshold is breached.