$ guides / fluentd / fluentd-log-storm-spike ▌

Operations Guides

Fluentd input spike: a log storm that overwhelms buffers and outputs

Your dashboards show input emit_records suddenly at 5x or 50x baseline. Within minutes, buffer_queue_length starts climbing, average flush time rises, and you are watching the buffer fill in real time. This is a log storm: an input spike large enough that the output side of the pipeline cannot drain it.

The danger is not the spike itself; Fluentd is built to absorb bursts, that is what the buffer is for. The danger is the cascade: the buffer fills to total_limit_size, overflow_action fires, and depending on configuration you either drop new events (throw_exception, the default), stall all input threads (block), or discard your oldest buffered data (drop_oldest_chunk). None of these outcomes is obvious unless you know which counters to watch.

This article covers how to confirm a storm is in progress, how to tell a legitimate batch spike from a runaway source, and how to keep the pipeline alive while you find the root cause.

What this means

Every event in Fluentd flows through one path: input, parser, filter chain, buffer, output. Inputs are the one stage you do not control. When an upstream application enters an error loop, someone leaves debug logging on, or a batch job dumps its run log at once, the input rate can jump by orders of magnitude with no warning.

The buffer absorbs the difference between input rate and output rate. It has finite capacity (total_limit_size: 512MB by default for memory buffers, 64GB for file buffers). If the spike outlasts the buffer’s ability to absorb it, the pipeline tips from “buffering a burst” into backpressure, and from there into one of three loss modes depending on overflow_action.

One amplifier worth knowing about: if Fluentd collects its own log file and the storm causes output errors, Fluentd logs those errors, which Fluentd then ingests, adding volume to the storm. A self-referential collection loop turns a bad situation into a feedback cycle.

flowchart LR
  A[Runaway source: error loop, debug logging, batch dump] --> B[Input emit_records spike]
  B --> C[Buffer stage fills fast]
  C --> D[Queue grows, flush latency rises]
  D --> E{Overflow action}
  E -->|throw_exception default| F[New events silently dropped]
  E -->|block| G[Input threads stall, upstream loss]
  E -->|drop_oldest_chunk| H[Oldest chunks discarded]

Common causes

Cause	What it looks like	First thing to check
Application error loop	Input rate spikes sharply, log lines repeat the same error stack	Sample the source log file for repeated identical messages
Debug/verbose logging left on	Spike starts right after a deploy, volume elevated across one service	Correlate spike start time with deployment timeline
Batch job or scheduled task	Spike is bounded, starts and stops on a schedule, queue drains afterward	Compare spike timing to cron or job scheduler
Retry storm amplifying volume	Destination flaps, retries hammer it, error volume grows on both sides	`retry_count` oscillating, destination intermittently reachable
Self-referential log loop	Fluentd’s own error output is being collected, volume feeds itself	Check whether Fluentd’s log path matches an `in_tail` glob
Legitimate traffic growth	Gradual rise over days or weeks, not a step change	Trend input rate over weeks, not minutes

The operational question is always: is this spike bounded (a batch job that will end) or unbounded (an error loop that will not stop until someone fixes it)? Bounded spikes are a capacity question. Unbounded spikes are a stop-the-source question.

Quick checks

These are read-only and safe to run during an incident. Paths shown are for td-agent; adjust for fluent-package (/var/log/fluent/, /etc/fluent/) or your deployment.

# 1. Total input rate: sum emit_records across all input plugins
curl -s http://localhost:24220/api/plugins.json | \
  jq '[.plugins[] | select(.plugin_category=="input") | .emit_records // 0] | add'

# 2. Per-input breakdown: find WHICH source is spiking
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="input") | {id: .plugin_id, type: .type, records: .emit_records}'

# 3. Output rate for comparison: is output keeping up?
curl -s http://localhost:24220/api/plugins.json | \
  jq '[.plugins[] | select(.plugin_category=="output") | .emit_records // 0] | add'

# 4. Buffer pressure: queue depth and remaining capacity per output
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, queue: .buffer_queue_length, stage: .buffer_stage_length, avail_pct: .buffer_available_buffer_space_ratios}'

# 5. Flush health: cumulative flush time, writes, slow flushes, retries
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, flush_time: .flush_time_count, writes: .write_count, slow: .slow_flush_count, retries: .retry_count}'

# 6. Sample the suspected source file for a repeated error pattern
tail -1000 /var/log/app/suspect.log | sort | uniq -c | sort -rn | head

# 7. Check whether Fluentd is throttling input at the source (if in_tail group limits configured)
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.type=="tail") | {id: .plugin_id, throttled: .throttled_log_count}'

Two caveats on the counters. First, emit_records is cumulative since process start; you need deltas over time, not absolute values, to see the spike. Second, on Fluentd older than v1.19.0, input emit_records requires enable_input_metrics true in <system>. Without it, input counters read zero and you cannot compare input against output. If you get zeros from check 1 on an older version, that is the cause.

How to diagnose it

Confirm the spike and find the source. Run checks 1 and 2 a few minutes apart and compute the per-input rate. A storm usually has one dominant input plugin. If the spike is spread evenly across all inputs, suspect a shared cause: a node-level issue, or a forwarder upstream of you.
Check whether the output is keeping pace. Compare output emit_records rate against input rate over a window of at least max(2 * flush_interval, 10 minutes). During a storm, output rate saturates at its ceiling while input rate climbs. The gap is what the buffer is absorbing.
Measure buffer burn rate. From check 4, watch buffer_available_buffer_space_ratios over two or three samples. Compute time-to-overflow: available_space / growth_rate. If available space is dropping linearly and you have 20 minutes of runway, you know exactly how long you have to act. See Fluentd buffer available space low for the full calculation.
Check flush latency, not just failures. Rising average flush time (delta(flush_time_count) / delta(write_count)) with zero retries means the destination is degrading under the storm load but not yet failing. This is the earliest warning that the output side is saturating. If slow_flush_count starts climbing, flushes are exceeding the slow-flush threshold (20s by default) and the queue will build.
Classify the spike: bounded or unbounded. Sample the source logs (check 6). A wall of identical stack traces or one repeated message means an error loop, and it will not stop on its own. A bounded spike (batch output, deploy-time burst, file replay after rotation) ends by itself, and the right response is usually to let the buffer absorb it and watch the drain.
Rule out the self-referential loop. Check whether Fluentd’s own log file (/var/log/td-agent/td-agent.log) is matched by any in_tail path or glob in your config. If it is, and the storm is causing Fluentd to log errors, you have a feedback loop. Volume will grow even if the original source calms down.
Check for source-side loss already happening. If in_tail group rate limiting is configured, a non-zero and climbing throttled_log_count means Fluentd is already dropping or deferring lines at the input. That is data loss at the source, before the buffer is even involved.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Input `emit_records` rate per plugin	Locates the storm source	Sustained deviation more than 2x rolling baseline
Output `emit_records` rate vs input rate	The gap is what the buffer absorbs	Output rate flat at ceiling while input climbs
`buffer_queue_length`	Backlog depth; distinguishes storm from stall	Sustained growth, especially with `write_count` still incrementing slowly
`buffer_available_buffer_space_ratios`	Direct time-to-overflow input	Below 20% and falling; below 5% is imminent overflow
`flush_time_count / write_count`	Average flush time; earliest saturation signal	Rising trend even with zero retries
`slow_flush_count`	Flushes over the slow-flush threshold	Any sustained increment during the spike
`retry_count` and `retry.next_time`	Destination failing under load	Non-zero retries; `next_time` far in the future means effective stall
`drop_oldest_chunk_count`	Confirmed data loss	Any increment, ever
`throttled_log_count` (in_tail)	Source-side drops from rate limiting	Any non-zero rate during the spike
Buffer stage vs queue lengths	High stage + low queue is healthy batching; low stage + high queue is backpressure	Queue exceeding stage by a wide margin

Fixes

Stop or slow the source

For an error loop, the only real fix is upstream: kill the looping application instance, roll back the deploy that enabled debug logging, or fix the crashing dependency. Fluentd-side tuning buys time but does not solve an unbounded source.

If the source cannot be stopped immediately, dropping the offending events at the filter layer is the fastest relief. A @type grep filter that excludes the repeated error pattern stops the flood before it reaches the buffer. This is deliberate, visible data loss, which is far better than invisible overflow loss, but treat it as a temporary incident measure and remove it when the source is fixed. Test the filter regex against a sample of the source logs before applying it; a wrong pattern under storm conditions can either do nothing or drop legitimate traffic.

Buy buffer runway

If the destination is healthy and just slower than the spike, increasing total_limit_size on a file-backed buffer converts the incident from “minutes to overflow” to “hours of absorption.” This requires free disk on the buffer partition; check df before raising the limit, and remember the change needs a config reload to take effect. For memory-backed buffers, raising the limit trades directly against RSS, and in a container that means OOM risk. During a storm is a bad time to discover you are on memory buffers. If you are, see Fluentd memory vs file buffer.

Choose the right failure mode

If overflow is genuinely unavoidable, pick your loss mode deliberately rather than accepting the default. throw_exception drops new events, and no counter reliably tracks the drops. drop_oldest_chunk sacrifices the oldest buffered data but increments drop_oldest_chunk_count, so the loss is at least measurable. block preserves data by stalling input threads, which pushes the loss upstream (UDP syslog packets dropped by the kernel, forward clients timing out). Note that a blocked input affects the whole input thread, not just the storm source. There is no free option; there is only an explicit one.

Increase output throughput, carefully

Raising flush_thread_count helps only if the destination has headroom. If the destination is the bottleneck, more flush threads just add load and can push a degrading destination into failure, starting retries on top of the storm. Check flush latency first: if average flush time is rising with zero retries, the destination is already near its limit and more threads will not help.

Break the feedback loop

If Fluentd’s own log is in the collection path, exclude it with a filter or remove its path from in_tail globs. This should be a permanent config change, not an incident workaround.

Prevention

Enable input metrics. On Fluentd older than v1.19.0, set enable_input_metrics true in <system>. Without per-input emit_records, you cannot find the storm source quickly, which is the single most valuable piece of information during the incident.
Set per-input baselines. Input rate alerting is relationship-based: deviation from the host’s own rolling baseline, not a static threshold. A host at 100 events/sec and one at 100k events/sec need different alarms.
Use file-backed buffers with honest sizing. A 512MB buffer is trivially exhausted by a real storm. File buffers with tens of gigabytes of headroom turn most storms into non-events.
Set overflow_action explicitly on every output. Never run the default silently. Document which loss mode each pipeline chooses and why.
Monitor the loss counters, not just the pressure gauges. drop_oldest_chunk_count and emit errors are the difference between “the buffer was busy” and “we lost data.” See Fluentd emit_error_count.
Keep Fluentd’s own logs out of its collection path so an incident cannot amplify itself.
Load-test the pipeline against storms, not steady state. The gap between benchmark throughput and production throughput (regex parsers, filters, GVL limits) only shows up when volume spikes. Know your actual ceiling before a storm finds it for you.

How Netdata helps

Netdata charts input and output emit_records rates per plugin side by side, so the divergence that defines a storm is visible in one view instead of two curl commands.
Buffer queue length, stage length, and available space ratio are tracked per output plugin, letting you watch burn rate and estimate time-to-overflow directly from the graph slope.
Flush latency derived from flush_time_count and write_count surfaces destination saturation before the first retry fires.
drop_oldest_chunk_count, retry_count, and slow_flush_count are correlated on the same timeline as the input spike, which makes it obvious whether the storm caused the output failure or merely exposed it.
Per-second collection catches short-lived storms that a 60-second scrape interval would average away entirely.
In multi-worker deployments, per-worker views keep a storm pinned to one worker from being hidden inside healthy aggregate numbers.