Fluentd in_tail not reading: the file is growing but no events are emitted

The application is writing logs. ls -la shows the file growing. But Fluentd’s input emit_records for that in_tail source has been flat for minutes or hours, and nothing is arriving downstream. The daemon runs, the buffers stay empty, and the only evidence is missing data. This is an input-side stall, and it is one of the quieter Fluentd failures because nothing page-worthy breaks.

A restart forces file re-discovery and usually restores flow temporarily, which is exactly why this failure gets misdiagnosed: the restart masks the cause, and the stall returns at the next rotation or the next burst of new files.

Severity: a pos_file position that is not advancing at all is PAGE-level, because Fluentd is effectively dead for that source while appearing alive. A position that trails the file size but keeps advancing is a TICKET-level lag problem.

What this means

in_tail keeps one open file descriptor and one position entry per tailed file. The position file (pos_file) records where Fluentd stopped reading, so it can resume without re-reading or skipping after a restart. For the pipeline to advance, three things must all be true:

1. The file must match the path glob and be discovered by the watcher (inotify or stat-based, depending on configuration).
2. Fluentd must be able to open and hold an FD for it.
3. Reads must succeed and the pos entry must advance.

A stall means one of those broke. The file growing on disk only proves the application side is fine. Common breakpoints: the watcher lost the file at rotation, the glob stopped matching after a path change, the pos entry desynced from the real file, FD exhaustion blocked new opens, or the input thread is paused by buffer backpressure. The official in_tail FAQ notes that when in_tail receives BufferOverflowError, it stops reading new lines and updating the pos_file until the buffer error clears; with overflow_action block, the input thread blocks outright.

One deployment constraint matters before anything else: in_tail does not support multi-worker mode. It must be pinned to a single worker with <worker N>, and its metrics only appear on that worker’s monitor_agent port (24220 + worker_id). If you are checking the wrong port, emit_records will look flat because you are looking at a worker that never ran the input.

flowchart TD
  A["Log file growing, emit_records flat"] --> B{"Pos file advancing?"}
  B -->|"Yes, but lagging"| C["Read throughput problem: CPU/GVL-bound parser or throttle"]
  B -->|"No, stuck"| D{"tracked_file_count matches expectation?"}
  D -->|"Count dropped"| E["Glob/path mismatch, file deleted, or FD exhaustion"]
  D -->|"Count OK"| F{"Rotation involved?"}
  F -->|"Yes"| G["Rotation handling failure: copytruncate race, rotate_wait, watcher stall bug"]
  F -->|"No"| H["Buffer backpressure blocking input, or hung watcher thread"]

Common causes

That last row is the cheap check everyone skips: on Fluentd before v1.19.0, input emit_records requires enable_input_metrics true in <system>. Without it, the counter is always 0 and the pipeline looks stalled when it is not. Before declaring an incident, confirm the destination is actually missing data.

Quick checks

All read-only. Paths shown for td-agent; fluent-package uses /var/log/fluent/fluentd.log and /etc/fluent/fluentd.conf.

# 1. Confirm the file is really growing
ls -la /var/log/app/app.log
stat -c '%s %i %Y' /var/log/app/app.log   # size, inode, mtime

# 2. Check input emit_records on the CORRECT worker port
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.type=="tail") | {id: .plugin_id, emit: .emit_records}'

# 3. Check tracked and rotated file counts (tracked_file_count needs v1.19.0+)
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.type=="tail") | {id: .plugin_id, tracked: .tracked_file_count, rotated: .rotated_file_count}'

# 4. Read the pos_file and compare position against file size
grep pos_file /etc/td-agent/td-agent.conf
cat /var/log/td-agent/td-agent.pos
# Each line records the watched path, the read position (hex), and the inode (hex).
ls -la /var/log/td-agent/td-agent.pos   # mtime should be recent while tailing is live

# 5. Check file descriptor usage against the limit
ls /proc/$(pgrep -f fluentd | head -1)/fd | wc -l
grep "Max open files" /proc/$(pgrep -f fluentd | head -1)/limits

# 6. Look for watcher and rotation messages in Fluentd's own log
grep -iE "unreadable|Skip update_watcher|detected rotation|too many open files|BufferOverflowError" \
  /var/log/td-agent/td-agent.log | tail -30

# 7. Check whether outputs are full and pushing back on the input
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, avail_pct: .buffer_available_buffer_space_ratios, queue: .buffer_queue_length}'

# 8. Confirm Fluentd version (several stall causes are version-specific)
fluentd --version

How to diagnose it

1. Rule out instrumentation blindness first. If Fluentd is older than v1.19.0 and <system> lacks enable_input_metrics true, input emit_records is always 0. Check the destination for recent events. If data is flowing, your problem is monitoring, not tailing. See enable_input_metrics on older versions.

2. Verify you are querying the right worker. In multi-worker setups in_tail is pinned via <worker N> and its metrics only exist on that worker’s port. Query 24220 + worker_id.

3. Check whether the pos entry is advancing at all. Note the position for the growing file, wait 60 seconds, check again. Advancing but behind: read-throughput problem (CPU-bound parser, throttling, or GVL contention). Not advancing: continue.

4. Check tracked_file_count and the glob. On v1.19.0+, a dropped tracked_file_count means files vanished from the watch set: the path changed, the glob no longer matches, or FDs are exhausted. On older versions, compare opened_file_count and closed_file_count trends. In Kubernetes, remember /var/log/containers/*.log entries are symlinks; if the runtime changed its log path layout (for example Docker to containerd), the pattern breaks silently.

5. Check FD headroom. If FD count is near the soft limit, new files cannot be opened. in_tail does not always log an explicit EMFILE error; it can just stop picking up new files. If this is the cause, raise the limit and reduce the watch set.

6. Correlate the stall start time with rotation. If the flatline began at the logrotate window, compare the inode recorded in the pos_file with stat -c %i on the live file. A persistent mismatch past rotate_wait (default 5s) means the watcher lost the file. Also check Fluentd’s log: the line “unreadable. It is excluded and would be examined next time.” followed by no recovery is the canonical signature of the watcher stall (GitHub issue #3614 ), and v1.15.1+ emits “Skip update_watcher because watcher has been already updated by other inotify event” when it happens. If the “detected rotation of …” line never appears for a file you know rotated, detection itself failed.

7. Check version against known stall bugs. The stop-tailing failure existed for years and was fixed in v1.16.2/v1.16.3 for both follow_inodes true and false cases. The v1.14.3 release fixed a full read stop with enable_watch_timer false plus enable_stat_watcher true (the default), and the v1.14.3 notes recommend at least v1.12.4 for anyone on the v1.12 line because of serious in_tail bugs. On affected versions, upgrading is the fix, not config tuning.

8. Check for backpressure. If buffer_available_buffer_space_ratios is near 0 on any output, the input may be blocked by overflow_action block or paused by BufferOverflowError. The stall is then a symptom of an output problem; diagnose the output side (see related guides below).

Metrics and signals to monitor

Fixes

Rotation and watcher stalls

• Upgrade to v1.16.3 or later. This is the durable fix for the stop-tailing bug that survives restarts and recurs at every rotation. If you are pinned to an older version, keep rotate_wait at or below refresh_interval; the release notes state that a larger rotate_wait reliably triggers the stall.
• Prefer rename/create rotation over copytruncate. Copytruncate has an inherent race: events written between the copy and the truncate can be missed. If copytruncate is unavoidable, treat some loss at the truncation window as a known limitation.
• Size rotate_wait and refresh_interval correctly. rotate_wait (default 5s) must cover the gap between the rename and the new file appearing. A too-long refresh_interval delays discovery of the new file after rotation.
• Use follow_inodes true with wildcard paths under rotation (v1.12.0+) to avoid duplicate reads when * matches both old and new files.
• If you tail many files and hit the “stuck” behavior, the official in_tail FAQ recommends enable_stat_watcher false as a workaround. Tradeoff: rotation detection then relies on the watch timer and refresh cycle rather than stat watching; validate rotation behavior after changing it.
• Do not set enable_watch_timer false casually. Combined with the default stat watcher it caused full read stops on v1.14.2 and earlier, and with multiline parsers it prevents multiline_flush_interval from working, so the last record in a file is never emitted.

Glob and path problems

• Fix the path glob to match reality, including symlink targets in Kubernetes. If you need extended glob syntax ([], ?, {}), glob_policy extended is available in v1.17.0 for both path and exclude_path.
• After fixing the pattern, restart to force re-discovery, and set read_from_head true (temporarily, if needed) to backfill files created during the blind window. Since v1.14.3, newly discovered files are read from head automatically even with read_from_head false, but on older versions initial lines of new files could be skipped with wildcard paths.

FD exhaustion

• Raise the soft limit in the systemd unit or container security context; production deployments commonly need 65536 or more. Budget: tailed files + buffer chunk files + output connections + baseline.
• Reduce the watch set: narrow globs, exclude rotated archives with exclude_path, and split sources across workers or agents if the file count is genuinely large.

pos_file problems

• Keep the pos_file on a persistent, local filesystem. On tmpfs it is lost at reboot and every file is re-read (with read_from_head true) or skipped. The maintainers have stated pos_file is not designed for NFS; aggressive pos_file_compaction_interval values on NFS have caused entries to be deleted and files re-read from the beginning with “Unparsable line in pos_file” warnings.
• Use pos_file_compaction_interval when tailing many files with dynamic paths, so the pos_file does not grow unbounded.
• If positions are corrupted, the controlled recovery is: stop Fluentd, fix or remove the stale entries, restart. Editing a live pos_file while Fluentd runs is not supported.

Backpressure-induced stalls

• The tail stall is downstream of a buffer problem. Fix the destination or resize the buffer; see buffer queue length growing and failed to flush the buffer. Choose overflow_action deliberately: block protects data but stalls inputs, throw_exception (the default) raises BufferOverflowError, which pauses in_tail reads and drops events from push-style inputs with a logged error, and drop_oldest_chunk discards old data.

Prevention

• Alert on the input side, not just outputs. A flat input emit_records rate on an active source is PAGE-worthy. Do not rely on process liveness; a running Fluentd with a stalled tail is a blind spot.
• Enable enable_input_metrics true in <system> on anything older than v1.19.0, or you will not be able to see input stalls at all.
• Track tracked_file_count and rotated_file_count (version permitting) against expectations per host. A rotation counter that stops incrementing on a host with daily logrotate is an early warning.
• Monitor FD usage against the soft limit and alert at 75%.
• Test rotation before production. Run logrotate against a tailed file and verify the pos entry follows to the new inode with no gap and no duplicates. Most in_tail stalls only appear at rotation.
• Pin Fluentd to a version with the in_tail fixes (v1.16.3+) and treat enable_watch_timer false and rotate_wait > refresh_interval as review flags in config changes.
• Keep pos_file on persistent local disk, and include it in any host rebuild or container volume planning.

How Netdata helps

• Netdata charts Fluentd’s monitor_agent metrics per plugin, so a flat input emit_records line next to a growing output-side gap is visible in one dashboard instead of two manual jq queries.
• Per-plugin breakdowns show which in_tail source stalled when you tail dozens of files, rather than forcing you to reason from an aggregate.
• Correlating input rate with buffer_available_buffer_space_ratios and buffer_queue_length distinguishes a true tail stall from backpressure pausing the input, which changes the fix entirely.
• rotated_file_count and tracked_file_count trends make rotation-related stalls obvious: the flatline lines up exactly with the rotation window.
• Host-level FD and RSS metrics sit on the same timeline as Fluentd’s internal counters, so FD exhaustion shows up as a rising resource line converging with the input stall.

Related guides

• Fluentd input emit_records stuck at zero: enable_input_metrics on older versions
• Fluentd failed to flush the buffer: the output cannot deliver and retries begin
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd emit_error_count: the number-one under-monitored data-loss signal
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• How Fluentd actually works in production: a mental model for operators