Fluentd throttled_log_count: in_tail rate limiting is dropping lines at the source

throttled_log_count is a cumulative counter exposed by the in_tail input plugin. Every increment means Fluentd deferred log lines at the source because a configured rate limit was exceeded. Unlike buffer-side loss, this happens before the event enters the pipeline: no filter sees it, no buffer holds it, and no output will ever deliver it.

The counter only moves when you have configured throttling in in_tail, either via the <group> section with rate rules or via byte-rate limiting on reads. If you never configured throttling, this counter stays at zero forever, and a zero value tells you nothing. If you did configure it, any non-zero increment rate is a decision point: either the throttle is doing what you designed it to do, or it is silently eating log volume you expected to keep.

What this means

When throttling is configured, in_tail enforces a ceiling on how many lines (or bytes) it will read from a file or group of files within a time window. When a source exceeds that ceiling, Fluentd stops reading from that source until the window resets. The lines are not lost in the strict sense: they remain on disk and will be picked up when reading resumes, because in_tail tracks its position in the pos_file. But if the file rotates away before the backlog drains, the unread lines are gone for good.

That last point is what makes this a data-loss signal in practice. A throttle that holds back 30 seconds of logs on a file that rotates every minute is a drop mechanism, not a delay mechanism.

flowchart LR
  A[Application writes log lines] --> B[Log file on disk]
  B --> C{in_tail throttle check}
  C -->|under limit| D[Emit into pipeline]
  C -->|over limit| E[Reading paused
throttled_log_count += 1]
  D --> F[Buffer and output]
  E --> G[Lines wait on disk]
  G -->|file rotates first| H[Unread lines lost]
  G -->|window resets| C

The counter is available from Fluentd v1.17.1 onward via the monitor_agent API, alongside the other in_tail file statistics (opened_file_count, closed_file_count, rotated_file_count). Group-based throttling itself was introduced in v1.15.0, so on versions 1.15.0-1.17.0 the throttling logic runs but the counter is not exposed yet.

Common causes

Quick checks

All of these are read-only and safe to run during an incident.

# 1. Read the counter and related in_tail stats from the monitor agent
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.type=="tail") | {id: .plugin_id, throttled: .throttled_log_count, rotated: .rotated_file_count}'

# 2. Check the input emit rate to see if it plateaus at a suspiciously flat ceiling
curl -s http://localhost:24220/api/plugins.json | \
  jq '[.plugins[] | select(.plugin_category=="input") | .emit_records // 0] | add'

Note: on Fluentd older than v1.19.0, input emit_records requires enable_input_metrics true in <system>. Without it this returns 0 regardless of actual traffic. See Fluentd input emit_records stuck at zero.

# 3. Find throttling configuration in the active config
grep -nE "<group>|rate_period|limit|read_bytes_limit_per_second" /etc/td-agent/td-agent.conf
# fluent-package installs use /etc/fluent/fluentd.conf

# 4. Sample the counter twice, 60 seconds apart, to get the increment rate
curl -s http://localhost:24220/api/plugins.json | \
  jq '[.plugins[] | select(.type=="tail") | .throttled_log_count // 0] | add'
sleep 60
curl -s http://localhost:24220/api/plugins.json | \
  jq '[.plugins[] | select(.type=="tail") | .throttled_log_count // 0] | add'

# 5. Check the pos_file lag: how far behind is in_tail on the throttled file?
grep pos_file /etc/td-agent/td-agent.conf
cat /var/log/td-agent/*.pos
# Compare the recorded byte offset against the actual file size:
ls -l /path/to/tailed/file.log

A large and growing gap between the pos_file offset and the file size, while throttled_log_count increments, confirms the throttle is actively holding the reader back.

# 6. Check Fluentd's own log for throttling-related messages
grep -iE "throttl|rate.?limit" /var/log/td-agent/td-agent.log | tail -20

How to diagnose it

1. Confirm the counter is actually incrementing. Take two samples a minute apart (check 4 above). A static non-zero value is history, not a live problem. Counters reset on restart, so a freshly restarted Fluentd with a large value means throttling was heavy before the restart.

2. Locate the throttle configuration. If grep for <group> finds nothing, also check included config fragments (@include directives) and, in Kubernetes, the ConfigMap backing the Fluentd config. If truly nothing is configured, the counter should not be moving; if it is, verify you are reading the right field and not a different plugin’s output.

3. Compute the effective ceiling. For group throttling, the relevant parameters are the limit on each <rule> and the rate_period (default 5 seconds). The effective ceiling is limit lines per rate_period for the files matching the rule’s match pattern.

4. Compare the ceiling to actual source volume. Measure how fast the log file grows in lines per second during the window when the counter increments: wc -l on the file twice with a known interval, or compute from pos_file offsets. If source volume exceeds the ceiling, the throttle is working as configured and the question becomes whether the configuration is correct.

5. Check whether the plateau is exactly the limit. If input emit_records flattens at precisely the configured line rate during peaks, that is the signature of an active throttle, not a coincidental slowdown. A CPU-bound or GVL-starved pipeline plateaus too, but not at a clean configured number, and it comes with high process CPU rather than a rising throttled_log_count.

6. Assess the loss window. Compare the pos_file offset to the file size, and check when the file next rotates (rotated_file_count cadence, logrotate schedule). If the backlog cannot drain before rotation, the deferred lines will be lost.

7. Decide: intended or unintended. If the throttle was deliberately set to protect a downstream destination from a noisy source, increments during a spike may be acceptable policy. If nobody can explain the limit, or the limit predates current traffic levels, it is a misconfiguration.

One semantics gotcha worth knowing: there is an open upstream report (fluent/fluentd issue #4184, still open) that a group’s limit is applied as an average per member file rather than as a total across the group, so the effective per-file ceiling can be limit / number_of_files, far tighter than the configuration reads.

Metrics and signals to monitor

Fixes

Raise the throttle limit

If the limit is simply too low for current traffic, increase limit on the affected <rule>. This is the right fix when the throttle predates a traffic increase and no one re-validated it.

Tradeoff: the throttle presumably existed to protect something downstream. Before raising it, check buffer headroom (buffer_available_buffer_space_ratios) and output flush health. If the destination was the reason for the limit, raising the input ceiling just moves the pressure to the buffer. See Fluentd buffer queue length growing for what that looks like when it goes wrong.

Remove the throttle entirely

If the configuration is a leftover and no one can name what it protects, delete the <group> section or the byte-rate limit and reload. The counter will stop incrementing and the backlog will drain.

Confirm by watching input rate and buffer metrics for the next peak window.

Scale out instead of raising the limit

If the limit exists because a single Fluentd instance cannot process the full volume, raising the limit without adding capacity just shifts the bottleneck to CPU or buffer. in_tail does not support multi-worker; it must be pinned to a specific worker. Scaling here means distributing files across multiple Fluentd instances (for example, sharding by path or by node in a DaemonSet) rather than flipping a worker count.

Protect against rotation loss while throttled

If you keep the throttle, make sure the read backlog can drain before files rotate. Options include increasing the rotation size threshold, lengthening rotation intervals, or ensuring rotate_wait gives Fluentd time to finish reading the old file. Enabling throttling also changes in_tail’s rotation-wait behavior: when throttling is enabled, the watcher waits until the file reaches EOF and then rotate_wait more seconds before closing, which can hold file handles longer than an unthrottled setup.

Do not restart Fluentd as a fix. A restart resets the counter but does nothing about the limit, and if the pos_file survived, the backlog simply resumes throttling after boot.

Prevention

• Alert on the increment rate, not the value. delta(throttled_log_count) > 0 over a 5-minute window, but only on hosts where throttling is intentionally configured. Everywhere else, the counter should be structurally zero.
• Annotate the limit with its reason. A comment in the config naming the protected downstream and the expected ceiling saves the next operator an hour of archaeology.
• Re-validate limits after traffic changes. Any deploy that increases log verbosity invalidates a line-rate limit set against old volume.
• Watch the plateau. Alerting on emit_records pinned at a constant rate during variable-traffic windows catches an active throttle even if nobody is watching the throttle counter.
• Test rotation under throttle. Force a rotation while the throttle is engaged and confirm no gap appears downstream. This is the failure mode that converts “delayed” into “lost”.

How Netdata helps

• Netdata collects Fluentd monitor_agent metrics continuously, so throttled_log_count becomes a time series rather than something you have to remember to poll twice during an incident.
• Correlating the throttle counter against input emit_records on the same dashboard makes the plateau-at-the-limit signature visible in seconds.
• rotated_file_count alongside throttled_log_count shows the dangerous overlap: throttling active at rotation time, which is when deferred lines become lost lines.
• Buffer signals (buffer_queue_length, available space ratio) on the same view answer the follow-up question before you raise a limit: can the downstream actually absorb the extra volume?
• Anomaly detection on input rates flags the flat-ceiling pattern even when no one has set an explicit alert on the throttle counter.

Related guides

• Fluentd broken pipe / connection reset: dropped output connections and LB timeouts
• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd buffer_oldest_timekey lag: how far behind the oldest buffered data is
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss
• Fluentd emit_error_count: the number-one under-monitored data-loss signal
• Fluentd input emit_records stuck at zero: enable_input_metrics on older versions
• Fluentd end-to-end pipeline latency: stale logs during an incident
• Fluentd failed to flush the buffer: the output cannot deliver and retries begin