Fluentd input emit_records dropped to zero: ingestion has stopped

The emit_records counter for an input plugin is the number of records that plugin has handed to the Fluentd router since process start. It is cumulative and monotonically increasing. If its computed rate drops to zero and holds there while the upstream source is still writing logs, ingestion is broken: events are being generated somewhere and silently going nowhere.

This is one of the few Fluentd conditions that justifies paging immediately. A dead process is obvious. A live process that has stopped ingesting is a silent observability blackout, and it will not announce itself anywhere except this counter.

The usual suspects: in_tail lost the file after a rotation, a network input port stopped receiving, the parser is rejecting every record, the pos_file is corrupted, or the process ran out of file descriptors. Rule out one false alarm first: on Fluentd older than v1.19.0, input emit_records is always zero unless you set enable_input_metrics true in <system>. If you never enabled it, the counter did not drop to zero. It was never on.

What this means

Every event in Fluentd flows through input, parser, filter chain, buffer, output. emit_records on an input plugin counts the first hop: raw data parsed into a timestamped, tagged record and emitted to the router. When that count stops moving, nothing downstream matters. The buffer, the outputs, the destination can all be healthy, and it is irrelevant, because there is nothing to route.

Because the counter is cumulative, you never alert on its value. You alert on its rate: the delta between two samples divided by the interval. In multi-worker mode, each worker is an independent process with its own counters and its own monitor_agent port (worker 0 on 24220, worker 1 on 24221, and so on), so the rate that matters is the sum across workers. Note that in_tail does not support multi-worker and must be pinned to a specific worker with <worker N>, so a tail input’s counter only ever appears on one worker’s port.

A drop to zero is different from a drop of 50 percent. A partial drop usually means one source among several went quiet. A hard zero across all inputs, or on the only input, means the ingestion path itself has failed.

flowchart TD
  A[Input emit_records rate = 0] --> B{Counter ever nonzero?}
  B -->|No, older Fluentd| C[enable_input_metrics not set - monitoring gap, not an incident]
  B -->|Yes| D{Sources still producing?}
  D -->|No| E[Upstream stopped - not a Fluentd problem]
  D -->|Yes| F{Which input type?}
  F -->|in_tail| G[Check tracked files, pos_file, rotation, FD count]
  F -->|Network input| H[Check listeners and connections on input ports]
  F -->|Any| I[Check Fluentd log for parse failures on every record]

Common causes

Quick checks

All read-only. Run them on the host where the counter flatlined. Paths shown are for the td-agent package; substitute fluent-package paths (/var/log/fluent/fluentd.log, /etc/fluent/fluentd.conf) if that is what you run.

# 1. Confirm the process is actually alive and which workers exist
pgrep -af fluentd

# 2. Sum input emit_records across all inputs on worker 0
curl -s http://localhost:24220/api/plugins.json | \
  jq '[.plugins[] | select(.plugin_category=="input") | .emit_records // 0] | add'

# 3. In multi-worker mode, repeat per worker port (24221, 24222, ...) and sum

# 4. Per-input breakdown to see which input is dead
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="input") | {id: .plugin_id, type, emit_records}'

# 5. Take two samples 60s apart and compute the delta - the rate is what matters

# 6. For in_tail: how many files is it tracking right now (v1.19.0+)
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.type=="tail") | {id: .plugin_id, tracked: .tracked_file_count, rotated: .rotated_file_count}'

# 7. Is the source actually still writing?
ls -la /var/log/app/current.log   # is mtime moving and size growing?

# 8. Fluentd's own log: parse errors, rotation events, FD errors
grep -iE "parse|pattern not match|too many open files|unreadable|rotation" \
  /var/log/td-agent/td-agent.log | tail -30

# 9. FD usage versus limit
FLUENTD_PID=$(pgrep -f fluentd | head -1)
ls /proc/$FLUENTD_PID/fd | wc -l
grep "Max open files" /proc/$FLUENTD_PID/limits

# 10. For network inputs: is the listener up and are senders connected?
ss -tlnp | grep fluentd          # listening sockets
ss -tn | grep ':24224'           # forward-protocol connections

How to diagnose it

Work through these in order. Each step eliminates a layer.

1. Rule out the monitoring gap first. Check the Fluentd version and the <system> section of the config. On versions before v1.19.0, input emit_records requires enable_input_metrics true; from v1.19.0 it defaults to true. If the counter has been zero since the day you deployed and output throughput is normal, you have no incident, you have a blind spot. Fix the config and move on.

2. Verify the source is producing. Check that the tailed file’s size and mtime are still advancing, or that the sending application is still running and connected. If the source went quiet, this is an upstream outage, not a Fluentd one.

3. Sample the counter twice. emit_records is cumulative, so one reading tells you nothing. Two readings 60 seconds apart give you the rate. Do this per worker port and per plugin_id so you know exactly which input on which worker is dead.

4. For in_tail, compare the pos_file to reality. Find the pos_file path from the config (grep pos_file /etc/td-agent/td-agent.conf). It records each tailed file’s path, inode, and byte offset. Compare the recorded inode against stat -c %i on the live file, and the recorded offset against the live file size. An inode that no longer matches any live file means Fluentd lost the file across a rotation. An offset far behind a growing file means it is not keeping up or has stalled.

5. Check the rotation correlation. Look at when rotated_file_count last incremented (v1.14.1+) and when the emit rate flatlined. If they line up, you are looking at the rotation-handling failure mode: Fluentd logged “detected rotation” but never logged “following tail of” for the new file. A known in_tail bug can leave a file permanently unfollowed after it is briefly unreadable during rotation; reloading or restarting Fluentd temporarily restores it, and setting enable_stat_watcher false on the input is a reported workaround. The stall bug (issue #3614) was fixed in v1.16.3 (PR #4327); v1.16.2 had carried the corresponding fixes for the follow_inodes case. If you are on an older release, upgrading is the fix.

6. Check for parse rejection. If the file is being followed (pos_file offset advancing) but nothing is emitted, the parser is dropping everything. Grep the Fluentd log for parse warnings. This typically happens right after an application deploy changes the log format. A record that fails parsing never reaches the router, so a parser rejecting 100 percent of input looks exactly like dead ingestion from this counter’s perspective.

7. Check FD exhaustion. Compare the FD count from /proc/<pid>/fd against the soft limit. Near the limit, new file opens and new connections fail, and in_tail can silently stop picking up files. Look for “too many open files” in the Fluentd log, though the failure is not always logged clearly.

8. For network inputs, check the listener and the senders. Confirm Fluentd still holds the listening socket, then check whether any sender has an established connection. If the listener is up but nobody is connected, the problem is upstream: senders failing to deliver, a firewall change, or a DNS issue on the sender side.

Metrics and signals to monitor

Fixes

in_tail lost the file after rotation

Short term, a Fluentd restart or config reload forces rediscovery of the file. That is a mitigation, not a fix, and it can re-read or skip data depending on pos_file state. The durable fixes are in the tail configuration and the rotation method:

• Prefer rename/create rotation over copytruncate. Copytruncate has an inherent race window where lines written between the copy and the truncate are missed.
• Set follow_inodes true when tailing wildcard paths.
• Increase rotate_wait from its 5s default if the new file appears slowly after rotation, and shorten refresh_interval so new files are discovered faster.
• If you are hitting the known “unreadable file stops being followed” bug, enable_stat_watcher false on the input is a widely reported workaround, and upgrading past the affected versions reduces exposure.

Parser rejecting everything

Fix the parser to match the current log format, or coordinate with the application team on what changed. Records dropped by the parser are already gone; you can only recover from the source files if they still exist and you re-read them (which means manipulating the pos_file, covered below). Validate parser changes against a sample of production log lines before rolling them out.

pos_file corruption or staleness

The pos_file is plain text with no checksum, and corruption (partial write, disk error, entries disappearing) sends Fluentd to wrong offsets, causing gaps or duplicates. Two rules prevent most of this: never share one pos_file between multiple in_tail configurations, and keep the pos_file on a reliable, persistent filesystem, not tmpfs. To recover a badly desynchronized file, stop Fluentd, remove or correct the pos_file, and restart. This is destructive: Fluentd will re-read from the head or skip to the tail of each file depending on read_from_head, so choose deliberately. Do not hand-edit the pos_file of a running Fluentd; offsets are only consistent when the process is stopped.

FD exhaustion

Raise the soft and hard nofile limits in the systemd unit or the container security context. The default 1024 is inadequate for any host tailing more than a handful of files plus buffer chunks and output connections. Then find out why you got there: too many files matching a glob, accumulating buffer chunk files from a stalled output, or an FD leak. Raising the limit without understanding the growth just postpones the next wall.

Network input not receiving

If the listener is up but idle, the fix is upstream: sender configuration, firewall rules, or DNS. Verify end to end by sending a synthetic event from a known host and watching the input counter move.

Input metrics disabled on older Fluentd

Add enable_input_metrics true to the <system> section and reload. This has a CPU cost, which is why it was opt-in before v1.19.0, but operating without input-side visibility is worse. Plan an upgrade; on v1.19.0 and later this is on by default.

Prevention

• Enable and monitor the monitor_agent. Without port 24220 you have no input counters at all and are back to guessing from process liveness.
• Alert on rate, not value. Page when the input emit_records rate is zero for a sustained window while the host’s sources are expected to produce; ticket on deviations >50% from the host’s rolling baseline.
• Sum across workers. Per-worker counters on per-worker ports; an alert built on worker 0 only will miss deaths on other workers.
• Track files, not just records. On v1.19.0+, alert on unexpected changes in tracked_file_count. On older versions, use rotated_file_count to at least confirm rotation detection still fires.
• Test rotation before it tests you. Trigger a real logrotate run against a staging in_tail setup and watch the emit rate across the rotation. Most ingestion stops are discovered at 00:00 when cron runs logrotate.
• Set explicit FD limits sized for tailed files plus buffer chunks plus output connections, with headroom, and monitor usage against the soft limit.
• One pos_file per in_tail, on persistent disk. Sharing corrupts; tmpfs loses everything on reboot.

How Netdata helps

• Netdata collects Fluentd plugin statistics via monitor_agent, so input emit_records becomes a per-second rate you can alert on directly instead of hand-rolling curl-and-jq polling.
• Zero-rate alerting on the input counter, with duration gating, catches the “process alive but ingestion dead” case that process checks miss entirely.
• Correlating the input emit rate with output emit rate on the same dashboard separates ingestion failure (input flat, output drains) from delivery failure (input fine, buffer growing), which are the two most commonly confused Fluentd incidents.
• On v1.19.0+ deployments, tracked_file_count alongside the emit rate shows whether the drop is “files disappeared” versus “files present but not being read”.
• Host-level signals from the same agent, FD usage against limits and Fluentd’s own log errors, close the gap on the two causes the Fluentd API cannot see: FD exhaustion and parser rejection.

Related guides

• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss
• Fluentd emit_error_count: the number-one under-monitored data-loss signal
• Fluentd failed to flush the buffer: the output cannot deliver and retries begin
• How Fluentd actually works in production: a mental model for operators
• Fluentd memory vs file buffer: why the default buffer loses data on restart
• Fluentd monitor_agent not responding: a process that is up but hung
• Fluentd monitoring checklist: the signals every production log pipeline needs