Most Fluentd monitoring setups watch retry_count, buffer_queue_length, and process liveness. Almost none watch emit_error_count. The usual discovery is postmortem: hours of logs are missing, and the missing window is the exact window the incident needed.

emit_error_count counts emit transactions that failed inside the pipeline: events that could not be handed off and will never be delivered. Any nonzero rate means data loss is happening right now. This is not a degradation signal or a leading indicator. It is confirmation that events are gone.

This article covers what increments the counter, how to find it on your installation (the field is not exposed identically everywhere), how to tell a buffer overflow from a plugin exception, and which companion signals you must watch because one common overflow path never increments this counter at all.

What this means

Every event in Fluentd travels Input to Parser to Filter chain to Buffer to Output. An emit error is raised when an event cannot complete a handoff along that path. The two dominant causes:

  1. The buffer rejected the event. The buffer is full (total_limit_size reached) and overflow_action is throwing an exception for incoming events.
  2. A plugin raised an exception while processing the event. A parser, filter, or output plugin crashed mid-transaction, or a malformed event was rejected.

The same failures also surface in Fluentd’s own log as buffer overflow and emit transaction failed messages, so even on builds where the counter field is not exposed through the monitor_agent API, the loss is visible in the log stream.

The counter is cumulative since process start. The absolute value is nearly meaningless on its own (a long-running process may carry a stale nonzero value from an old incident). What matters is the rate: delta(emit_error_count) over your sampling interval. Zero tolerance in production. Each increment is a permanently lost event.

flowchart LR
  A[Input plugins] --> B[Filter chain]
  B --> C{Buffer has space?}
  C -->|yes| D[Chunk staged and queued]
  D --> E[Output flush]
  C -->|no| F[overflow_action fires]
  F -->|throw_exception| G[Event lost at input - often no counter]
  F -->|drop_oldest_chunk| H[Oldest chunk discarded - drop_oldest_chunk_count]
  B -->|plugin exception| I[emit transaction failed - emit_error_count]
  F -->|exception path| I

Common causes

CauseWhat it looks likeFirst thing to check
Buffer overflow, destination down or slowbuffer_available_buffer_space_ratios near 0%, retry_count climbing, write_count flatDestination health, Fluentd error log for connection or HTTP errors
overflow_action: drop_oldest_chunk doing its jobBuffer never looks full, queue stays low, data silently discarded, drop_oldest_chunk_count risingThe overflow_action setting on each output’s buffer section
Plugin exception during processingemit transaction failed in logs with an exception class and backtrace, throughput normal otherwiseFluentd log around the first failure timestamp
Malformed events rejected by a filter or parserSmall steady trickle of emit errors correlated with one application or tagWhich plugin_id carries the counter; sample the offending source
Retry exhaustion without a secondaryChunks discarded after retry_timeout, buffer drains without deliveryRetry state in monitor_agent, whether <secondary> is configured

Quick checks

All read-only. Paths shown are for the td-agent package; fluent-package uses /var/log/fluent/fluentd.log and /etc/fluent/fluentd.conf, and Kubernetes deployments usually read logs via kubectl logs.

# 1. Check whether emit_error_count is exposed for any plugin
curl -s http://localhost:24220/api/plugins.json | python3 -c "
import sys,json
for p in json.load(sys.stdin)['plugins']:
    if 'emit_error_count' in p:
        print(p['plugin_id'], 'emit_errors=', p['emit_error_count'])"

# 2. Count overflow and emit-failure lines in Fluentd's own log
grep -c "buffer overflow\|emit transaction failed" /var/log/td-agent/td-agent.log

# 3. See the most recent occurrences with context
grep -E "buffer overflow|emit transaction failed" /var/log/td-agent/td-agent.log | tail -20

# 4. Check how close each output's buffer is to its limit
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, avail_pct: .buffer_available_buffer_space_ratios, queue: .buffer_queue_length}'

# 5. Check whether chunks are being discarded by overflow policy
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, dropped: .drop_oldest_chunk_count}'

# 6. Compare input vs output volume (sustained deficit = loss somewhere)
curl -s http://localhost:24220/api/plugins.json | \
  jq '[.plugins[] | select(.plugin_category=="input") | .emit_records // 0] | add'
curl -s http://localhost:24220/api/plugins.json | \
  jq '[.plugins[] | select(.plugin_category=="output") | .emit_records // 0] | add'

# 7. Confirm the retry and write state of each output
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, retries: .retry_count, writes: .write_count}'

# 8. Check the configured overflow_action on every output buffer
grep -B2 -A8 "overflow_action\|<buffer" /etc/td-agent/td-agent.conf

Two caveats on check 1. Field availability varies by Fluentd version and build; on some installations the counter is not present in the monitor_agent response at all, which is exactly why check 2 (the log grep) matters. And depending on how you scrape metrics, the same underlying counter may be exported under a different name by your metrics pipeline. Verify what your collector actually ingests rather than assuming the field name.

How to diagnose it

  1. Confirm loss is active, not historical. The counter is cumulative. Take two readings 60 seconds apart. If the value did not move, you are looking at residue from an old incident; find the incident window in the log instead of chasing a live problem. If it moved, continue.

  2. Identify which plugin carries the increment. The counter is per plugin. The plugin_id tells you which output (or which route) is failing. In multi-worker mode, query each worker’s monitor_agent port (24220, 24221, and so on); an aggregate view can hide a single worker that is losing data.

  3. Separate the two failure families. Look at the buffer gauges for that output:

    • buffer_available_buffer_space_ratios at or near 0% and buffer_queue_length at maximum: this is buffer overflow. Go to step 4.
    • Buffer healthy, plenty of free space, counter still rising: this is a plugin exception or event rejection. Go to step 5.

  4. For overflow: find out why the buffer filled. Check retry_count (destination rejecting), write_count (flat means fully stalled), and slow_flush_count plus flush_time_count / write_count (destination slow rather than dead). Then check the destination directly: cluster health for Elasticsearch, broker state for Kafka, endpoint reachability for S3 or HTTP outputs. The Fluentd error log will usually name the failure: connection refused, 401/403, 429, TLS errors.

  5. For exceptions: read the log. emit transaction failed entries are accompanied by an exception class and usually a backtrace. Common patterns are a parser choking on a format change from an upstream application, a filter raising on an unexpected field type, or one poison event hitting a route repeatedly. Correlate the timestamp with any recent deploys, config reloads, or upstream application changes.

  6. Check the overflow_action configuration regardless. This determines your exposure during the next destination outage. throw_exception is the default and drops new events at the input when the buffer is full; drop_oldest_chunk discards old buffered data; block stops the input threads and pushes backpressure upstream. None of these is “safe” by default. They just move the loss to a different place.

  7. Quantify the loss for the incident record. Compare input emit_records against output emit_records over the incident window. The gap is your lower bound on lost events. On older Fluentd versions, input-side metrics may need to be enabled explicitly; the draft this was edited from cited enable_input_metrics true in <system> for versions before v1.19.0. Without input metrics, the input counter reads zero and this comparison is impossible.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
emit_error_count rateThe direct data-loss signalAny nonzero rate, however small
buffer overflow / emit transaction failed log linesCatches loss on builds where the counter is not exposedAny occurrence in production
drop_oldest_chunk_countConfirmed chunk discard when overflow_action is drop_oldest_chunkAny increment
buffer_available_buffer_space_ratiosLeading indicator before overflow firesBelow 20% and falling
buffer_queue_lengthBacklog depth ahead of the outputSustained growth, or pinned at limit
retry_count and retry stateDestination failing; backoff stretching recoveryNonzero sustained, retry.next_time far in the future
Input vs output emit_records balanceThe net check that catches loss paths with no counterSustained deficit over a 15-minute window
write_count rateProof chunks are actually leavingFlat while input is active

The last row of that table deserves emphasis. With the default overflow_action: throw_exception, events rejected at a full buffer are dropped silently and no error counter reliably increments. Your only reliable detection for that path is buffer_available_buffer_space_ratios at 0% combined with an input/output rate gap. Monitoring emit_error_count alone does not close the hole; it closes most of it.

Fixes

Destination failure causing buffer overflow

Restore the destination first; everything else is buying time. While it is down:

  • Raise total_limit_size if you have disk. For file-backed buffers this extends time-to-overflow. Compute runway as (total_limit_size - buffer_total_queued_size) / buffer growth rate before and after the change so you know what you bought. Make sure the filesystem holding the buffer has the space, and that the buffer is not sharing a partition with system logs.
  • Do not restart Fluentd to “clear” retries as a first move. File-backed buffers survive restart, but memory-backed buffers lose everything unflushed. A restart with memory buffers converts a delayed-delivery problem into confirmed data loss.

Wrong overflow_action for your tolerance

Pick the policy per output based on which loss you can survive:

  • block: no Fluentd-side loss, but input threads stall. For UDP syslog inputs the kernel drops packets instead, so the loss moves upstream and becomes invisible to Fluentd entirely. Acceptable when senders buffer (for example, forward protocol clients with their own disk buffers).
  • drop_oldest_chunk: pipeline keeps flowing, newest data wins, and the loss is at least countable via drop_oldest_chunk_count. Reasonable for metrics-like logs, dangerous for audit or security logs where the old events are exactly the ones you need.
  • throw_exception (default): silent loss at the input with no reliable counter. Rarely what you want in production. If you keep it, monitoring buffer space ratios becomes non-optional.

Changing the buffer section requires a config reload or restart. A SIGHUP reload that partially applies is its own failure mode; verify the plugin list in monitor_agent after reloading.

Plugin exceptions

Fix the root cause in the parser or filter, not the symptom. If an upstream application changed its log format, update the parser to match or route that tag to a permissive fallback while you fix it. If a single malformed event is crashing the pipeline repeatedly, isolate the offending source and quarantine the pattern with a grep filter. Keep in mind that one bad line can also kill the whole process on every restart; see the poison pill pattern in the related guides.

Retry exhaustion

If chunks are being discarded after retry_timeout (default 72 hours) with no secondary configured, add a <secondary> output as a dead-letter destination (a local file or cheap object store works). Then write_secondary_count becomes your tripwire: any increment means the primary failed exhaustively for at least one chunk, and the data is recoverable from the fallback.

Prevention

  • Alert on the rate, with zero tolerance. Any sustained nonzero emit_error_count rate pages. Do not batch this into a weekly review metric; small increments accumulate into significant loss over hours, and they are easy to miss on dashboards.
  • Alert on the log pattern too. On builds where the counter is not exposed, buffer overflow and emit transaction failed log lines are the only emission-path loss signal. Ship Fluentd’s own logs somewhere that is not Fluentd.
  • Monitor the companions. Buffer space ratio, queue length, retry count, drop_oldest_chunk_count, and the input/output rate balance. The throw_exception path has no counter, so the balance check is your net.
  • Choose overflow_action explicitly on every output. Write it into the config with a comment explaining the choice. The default is a data-loss decision made for you.
  • Prefer file-backed buffers in production. Memory buffers are fast and volatile; an OOM kill during a destination outage destroys everything buffered. File buffers trade I/O for durability.
  • Enable input metrics. Without input-side emit_records, the input/output balance check is impossible. See the version caveat in the diagnosis section before assuming it works on your build.
  • Size the buffer for the outage you expect, not the one you hope for. Use the runway formula with your real growth rate and your real mean-time-to-repair for the destination.

How Netdata helps

  • Netdata’s Fluentd collector scrapes the monitor_agent endpoint and charts per-plugin buffer and retry stats, so the “buffer filling, retries climbing” part of the overflow cascade is visible without manual polling.
  • Cumulative counters are charted as rates, which is the value you actually alert on rather than an ever-growing total.
  • Alerts can be set on buffer queue length and retry count, which covers the overflow paths that have no error counter of their own.
  • Correlating Fluentd metrics with host-level signals (disk free on the buffer partition, process RSS, OOM events) shortens the jump from “data is being lost” to “here is why.”