Fluentd emit_error_count: the number-one under-monitored data-loss signal

Most Fluentd monitoring setups watch retry_count, buffer_queue_length, and process liveness. Almost none watch emit_error_count. The usual discovery is postmortem: hours of logs are missing, and the missing window is the exact window the incident needed.

Failed emit transactions are events that could not be handed off inside the pipeline and will never be delivered. No Fluentd version exposes a counter literally named emit_error_count in the monitor_agent API; the article uses the name for the failure signal. Any nonzero rate of these failures means data loss is happening right now. This is not a degradation signal or a leading indicator. It is confirmation that events are gone.

This article covers what causes emit failures, how to find them on your installation (there is no dedicated API field; the signal is assembled from Fluentd’s log lines and the counters it does expose), how to tell a buffer overflow from a plugin exception, and which companion signals you must watch because one common overflow path leaves no counter at all.

What this means

Every event in Fluentd travels Input to Parser to Filter chain to Buffer to Output. An emit error is raised when an event cannot complete a handoff along that path. The two dominant causes:

1. The buffer rejected the event. The buffer is full (total_limit_size reached) and overflow_action is throwing an exception for incoming events.
2. A plugin raised an exception while processing the event. A parser, filter, or output plugin crashed mid-transaction, or a malformed event was rejected.

The same failures surface in Fluentd’s own log as emit transaction failed and failed to write data into buffer by buffer overflow messages, which is where the loss is directly visible.

The per-plugin counters Fluentd does expose are cumulative since process start. The absolute value is nearly meaningless on its own (a long-running process may carry a stale nonzero value from an old incident). What matters is the rate: delta(retry_count) (incremented once per failed emit transaction and per failed flush) and delta(drop_oldest_chunk_count) over your sampling interval, plus new emit transaction failed log lines. Zero tolerance in production. Each increment is permanently lost events.

flowchart LR
  A[Input plugins] --> B[Filter chain]
  B --> C{Buffer has space?}
  C -->|yes| D[Chunk staged and queued]
  D --> E[Output flush]
  C -->|no| F[overflow_action fires]
  F -->|throw_exception| G[Event lost at input - often no counter]
  F -->|drop_oldest_chunk| H[Oldest chunk discarded - drop_oldest_chunk_count]
  B -->|plugin exception| I[emit transaction failed - log line]
  F -->|exception path| I

Common causes

Quick checks

All read-only. Paths shown are for the td-agent package; fluent-package uses /var/log/fluent/fluentd.log and /etc/fluent/fluentd.conf, and Kubernetes deployments usually read logs via kubectl logs.

# 1. No Fluentd build exposes an emit_error_count field. Check the per-plugin
#    counters that do move on emit failures:
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, retries: .retry_count, rollbacks: .rollback_count, dropped: .drop_oldest_chunk_count}'

# 2. Count overflow and emit-failure lines in Fluentd's own log
grep -c "buffer overflow\|emit transaction failed" /var/log/td-agent/td-agent.log

# 3. See the most recent occurrences with context
grep -E "buffer overflow|emit transaction failed" /var/log/td-agent/td-agent.log | tail -20

# 4. Check how close each output's buffer is to its limit
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, avail_pct: .buffer_available_buffer_space_ratios, queue: .buffer_queue_length}'

# 5. Check whether chunks are being discarded by overflow policy
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, dropped: .drop_oldest_chunk_count}'

# 6. Compare input vs output volume (sustained deficit = loss somewhere)
curl -s http://localhost:24220/api/plugins.json | \
  jq '[.plugins[] | select(.plugin_category=="input") | .emit_records // 0] | add'
curl -s http://localhost:24220/api/plugins.json | \
  jq '[.plugins[] | select(.plugin_category=="output") | .emit_records // 0] | add'

# 7. Confirm the retry and write state of each output
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, retries: .retry_count, writes: .write_count}'

# 8. Check the configured overflow_action on every output buffer
grep -B2 -A8 "overflow_action\|<buffer" /etc/td-agent/td-agent.conf

Two caveats. First, no Fluentd version exposes an emit_error_count field: retry_count is the counter that increments once per failed emit transaction, drop_oldest_chunk_count tracks discards under overflow_action: drop_oldest_chunk, and check 2 (the log grep) is the direct evidence. Second, depending on how you scrape metrics, the same underlying counters may be exported under different names by your metrics pipeline. Verify what your collector actually ingests rather than assuming field names.

How to diagnose it

1. Confirm loss is active, not historical. The relevant counters are cumulative. Take two readings 60 seconds apart. If the values did not move and no new emit transaction failed lines appeared, you are looking at residue from an old incident; find the incident window in the log instead of chasing a live problem. If it moved, continue.

2. Identify which plugin carries the increment. The counter is per plugin. The plugin_id tells you which output (or which route) is failing. In multi-worker mode, query each worker’s monitor_agent port (24220, 24221, and so on); an aggregate view can hide a single worker that is losing data.

3. Separate the two failure families. Look at the buffer gauges for that output:

   • buffer_available_buffer_space_ratios at or near 0% and buffer_queue_length at maximum: this is buffer overflow. Go to step 4.
   • Buffer healthy, plenty of free space, counter still rising: this is a plugin exception or event rejection. Go to step 5.

4. For overflow: find out why the buffer filled. Check retry_count (destination rejecting), write_count (flat means fully stalled), and slow_flush_count plus flush_time_count / write_count (destination slow rather than dead). Then check the destination directly: cluster health for Elasticsearch, broker state for Kafka, endpoint reachability for S3 or HTTP outputs. The Fluentd error log will usually name the failure: connection refused, 401/403, 429, TLS errors.

5. For exceptions: read the log. emit transaction failed entries are accompanied by an exception class and usually a backtrace. Common patterns are a parser choking on a format change from an upstream application, a filter raising on an unexpected field type, or one poison event hitting a route repeatedly. Correlate the timestamp with any recent deploys, config reloads, or upstream application changes.

6. Check the overflow_action configuration regardless. This determines your exposure during the next destination outage. throw_exception is the default and drops new events at the input when the buffer is full; drop_oldest_chunk discards old buffered data; block stops the input threads and pushes backpressure upstream. None of these is “safe” by default. They just move the loss to a different place.

7. Quantify the loss for the incident record. Compare input emit_records against output emit_records over the incident window. The gap is your lower bound on lost events. Input emit_records for input plugins is only collected when enable_input_metrics true is set in <system>, and that parameter defaults to true only since v1.19.0. Without input metrics, the input counter reads zero and this comparison is impossible.

Metrics and signals to monitor

The last row of that table deserves emphasis. With the default overflow_action: throw_exception, events rejected at a full buffer are dropped silently and no error counter reliably increments. Your only reliable detection for that path is buffer_available_buffer_space_ratios at 0% combined with an input/output rate gap. Monitoring emit-failure signals alone does not close the hole; it closes most of it.

Fixes

Destination failure causing buffer overflow

Restore the destination first; everything else is buying time. While it is down:

• Raise total_limit_size if you have disk. For file-backed buffers this extends time-to-overflow. Compute runway as (total_limit_size - buffer_total_queued_size) / buffer growth rate before and after the change so you know what you bought. Make sure the filesystem holding the buffer has the space, and that the buffer is not sharing a partition with system logs.
• Do not restart Fluentd to “clear” retries as a first move. File-backed buffers survive restart, but memory-backed buffers lose everything unflushed. A restart with memory buffers converts a delayed-delivery problem into confirmed data loss.

Wrong overflow_action for your tolerance

Pick the policy per output based on which loss you can survive:

• block: no Fluentd-side loss, but input threads stall. For UDP syslog inputs the kernel drops packets instead, so the loss moves upstream and becomes invisible to Fluentd entirely. Acceptable when senders buffer (for example, forward protocol clients with their own disk buffers).
• drop_oldest_chunk: pipeline keeps flowing, newest data wins, and the loss is at least countable via drop_oldest_chunk_count. Reasonable for metrics-like logs, dangerous for audit or security logs where the old events are exactly the ones you need.
• throw_exception (default): silent loss at the input with no reliable counter. Rarely what you want in production. If you keep it, monitoring buffer space ratios becomes non-optional.

Changing the buffer section requires a config reload or restart. A SIGHUP reload that partially applies is its own failure mode; verify the plugin list in monitor_agent after reloading.

Plugin exceptions

Fix the root cause in the parser or filter, not the symptom. If an upstream application changed its log format, update the parser to match or route that tag to a permissive fallback while you fix it. If a single malformed event is crashing the pipeline repeatedly, isolate the offending source and quarantine the pattern with a grep filter. Keep in mind that one bad line can also kill the whole process on every restart; see the poison pill pattern in the related guides.

Retry exhaustion

If chunks are being discarded after retry_timeout (default 72 hours) with no secondary configured, add a <secondary> output as a dead-letter destination (a local file or cheap object store works). Then write_secondary_count becomes your tripwire: any increment means the primary failed exhaustively for at least one chunk, and the data is recoverable from the fallback.

Prevention

• Alert on the rate, with zero tolerance. Any sustained nonzero emit-failure rate (new emit transaction failed lines or climbing retry_count) pages. Do not batch this into a weekly review metric; small increments accumulate into significant loss over hours, and they are easy to miss on dashboards.
• Alert on the log pattern too. On builds where the counter is not exposed, buffer overflow and emit transaction failed log lines are the only emission-path loss signal. Ship Fluentd’s own logs somewhere that is not Fluentd.
• Monitor the companions. Buffer space ratio, queue length, retry count, drop_oldest_chunk_count, and the input/output rate balance. The throw_exception path has no counter, so the balance check is your net.
• Choose overflow_action explicitly on every output. Write it into the config with a comment explaining the choice. The default is a data-loss decision made for you.
• Prefer file-backed buffers in production. Memory buffers are fast and volatile; an OOM kill during a destination outage destroys everything buffered. File buffers trade I/O for durability.
• Enable input metrics. Without input-side emit_records, the input/output balance check is impossible. See the version caveat in the diagnosis section before assuming it works on your build.
• Size the buffer for the outage you expect, not the one you hope for. Use the runway formula with your real growth rate and your real mean-time-to-repair for the destination.

How Netdata helps

• Netdata’s Fluentd collector scrapes the monitor_agent endpoint and charts per-plugin buffer and retry stats, so the “buffer filling, retries climbing” part of the overflow cascade is visible without manual polling.
• Cumulative counters are charted as rates, which is the value you actually alert on rather than an ever-growing total.
• Alerts can be set on buffer queue length and retry count, which covers the overflow paths that have no error counter of their own.
• Correlating Fluentd metrics with host-level signals (disk free on the buffer partition, process RSS, OOM events) shortens the jump from “data is being lost” to “here is why.”

Related guides

• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• How Fluentd actually works in production: a mental model for operators
• Fluentd monitor_agent not responding: a process that is up but hung
• Fluentd monitoring checklist: the signals every production log pipeline needs
• Fluentd monitoring maturity model: from survival to expert
• Fluentd plugin load error at startup: LoadError and missing gems
• Fluentd poison pill crash loop: one bad log line that kills the process on every restart
• Fluentd process not running: the log pipeline is dead and the host has gone dark