Fluentd too many open files: file descriptor exhaustion stalls the pipeline

Fluentd logs Errno::EMFILE: Too many open files and the pipeline stops making progress. Log files keep growing on disk, the destination sees nothing new, and the Fluentd process is still alive. It is not crashed; it is wedged against its file descriptor limit, and almost every operation it needs to do next requires opening something.

What makes this failure nasty is the cliff edge. Everything works until the limit is reached, and then several things fail at once: in_tail cannot open newly created log files (and may stop watching them without a loud error), the buffer cannot create new chunk files, and outputs cannot establish new connections. A single resource limit takes out input, buffer, and output simultaneously.

The default OS soft limit of 1024 file descriptors is far too low for any non-trivial Fluentd deployment. Production nodes commonly need 65536 or more. If you have never explicitly raised the limit, this is almost certainly your problem.

What this means

Every Fluentd process holds a budget of open file descriptors set by ulimit -n (the soft limit is what applies in practice). That budget is consumed by four categories:

• Tailed files: in_tail holds one descriptor per actively watched file. Glob paths like /var/log/containers/*.log can match hundreds of files on a busy node, and rotated-but-not-yet-closed files still hold descriptors until Fluentd finishes draining them.
• Buffer chunk files: file-backed buffers create one file per chunk. When the output stalls, chunks accumulate, and so do descriptors.
• Output connections: each destination connection consumes a descriptor, multiplied by flush_thread_count.
• Baseline: the Ruby runtime, listen sockets, pipes, and monitor_agent endpoint, roughly 50 descriptors before any workload.

A rough sizing formula:

expected FDs = tailed files + (output connections x flush_thread_count)
             + active buffer chunk files + ~50 baseline

The failure cascade looks like this:

flowchart TD
  A[FD usage climbs] --> B{At soft limit?}
  B -->|no| A
  B -->|yes| C[in_tail cannot open new files]
  B -->|yes| D[Buffer cannot create chunk files]
  B -->|yes| E[Output cannot open connections]
  C --> F[New logs silently not collected]
  D --> G[Events rejected or lost at buffer]
  E --> H[Retries accumulate, queue grows]
  G --> H
  H --> I[Pipeline stalled, process still alive]

Note the feedback loop: once outputs cannot open connections, buffer chunks accumulate, which requires more descriptors for chunk files, which makes the exhaustion worse. The degradation curve is not gradual; it is a hard wall with a self-reinforcing stall behind it.

Common causes

Quick checks

These are all read-only and safe to run during an incident.

# Find the Fluentd PID (supervisor; check workers too in multi-worker mode)
pgrep -af fluentd

# Count open descriptors for the process
ls /proc/$(pgrep -f fluentd | head -1)/fd | wc -l

# Check the actual limit applied to the running process
cat /proc/$(pgrep -f fluentd | head -1)/limits | grep "Max open files"

# See what the descriptors actually are: files, sockets, pipes, deleted files
ls -la /proc/$(pgrep -f fluentd | head -1)/fd | awk '{print $NF}' | sort | uniq -c | sort -rn | head -20

# Specifically look for rotated-away files still held open
ls -la /proc/$(pgrep -f fluentd | head -1)/fd | grep deleted

# Confirm the error in Fluentd's own log (path varies by package)
grep -i "too many open files" /var/log/td-agent/td-agent.log | tail -20

# Count buffer chunk files (adjust path to your buffer_path)
find /var/log/fluent/buffer/ -type f | wc -l

# Count files matching your in_tail glob
ls /var/log/containers/*.log 2>/dev/null | wc -l

# Check buffer backpressure via monitor_agent
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, queue: .buffer_queue_length, retries: .retry_count}'

Two things to note on that last check. First, the monitor_agent endpoint itself may fail to respond when the process is FD-starved, because accepting your HTTP connection requires a descriptor. A timeout on port 24220 during this incident is itself a signal. Second, in multi-worker mode each worker has its own FD budget and its own monitor_agent port (24220 + worker id), so check each worker.

How to diagnose it

1. Confirm the symptom. Find the EMFILE error in the Fluentd log. It surfaces as Errno::EMFILE: Too many open files - socket(2) for network operations or Too many open files @ rb_sysopen for file operations. If neither appears but collection silently stopped, FD exhaustion is still plausible: in_tail failing to open a new file does not always produce a loud error.

2. Measure current usage against the limit. Compare ls /proc/<pid>/fd | wc -l against the Max open files soft limit from /proc/<pid>/limits. If you are at or near the soft limit, the diagnosis is confirmed. The limit shown in a login shell (ulimit -Sn) is irrelevant; only the running process’s limit matters.

3. Attribute the descriptors. The breakdown tells you the cause. Mostly regular files under your log paths: too many tailed files or rotated files held open. Mostly files under the buffer directory: chunk accumulation from a stalled output. Mostly sockets: output connection churn or leak. A large count of (deleted) entries: rotation handling is holding old files.

4. Check the buffer side. If chunk files dominate, look at buffer_queue_length and retry_count per output. A destination outage that ran for hours can accumulate tens of thousands of chunk files. There are operator reports of buffer directories reaching hundreds of thousands of chunk files after prolonged destination outages (community reports point at td-agent 3.7.x / Fluentd 1.7.x), which can leave the process unable to flush even after the destination recovers, because it cannot open the backlog within its limit.

5. Check the input side. On Fluentd v1.19.0+, tracked_file_count from the monitor_agent shows how many files in_tail is currently watching; compare it to the actual number of files matching the glob. A gap means some files are not being watched. On v1.14.1+, opened_file_count minus closed_file_count gives a cumulative approximation.

6. Decide: raise, reduce, or both. If usage is legitimately high (many pods, many outputs), raise the limit. If usage is inflated (deleted files held open, connection leak, runaway buffer), fix the underlying behavior, because raising the limit only postpones the next cliff.

Metrics and signals to monitor

Fixes

Raise the limit (almost always required)

Set the limit where the process actually gets it. On systemd-managed services, /etc/security/limits.conf does not apply; systemd ignores it for units it starts. Use a drop-in override:

# Create a systemd drop-in (adjust unit name for your package)
systemctl edit td-agent

[Service]
LimitNOFILE=65536

Then reload and restart. Raising LimitNOFILE requires a service restart to take effect, since limits are set at process start. A restart with file-backed buffers will replay queued chunks, so expect a brief flush burst rather than data loss. With memory-backed buffers, unflushed data is lost on restart.

For a running process you cannot restart yet, prlimit can raise the limit live:

# Emergency live raise (needs root or CAP_SYS_RESOURCE on the target; verify it took effect)
prlimit --pid $(pgrep -f fluentd | head -1) --nofile=65536:65536
cat /proc/$(pgrep -f fluentd | head -1)/limits | grep "Max open files"

This buys time but is not persistent; still fix the unit file. In Kubernetes, the FD limit typically comes from the container runtime defaults rather than the pod spec — the Kubernetes API has no per-pod nofile setting. containerd raises the in-container limit to 1048576 by default (and since Go 1.19 the soft limit follows the hard limit); older Docker contexts may apply 1048576 as well, while other runtimes can leave the 1024 inherited default. Verify with /proc/1/limits inside the container before assuming your host settings apply. To change it per runtime, set Rlimits for the runc/containerd runtime options in /etc/containerd/config.toml, or patch the container entrypoint with prlimit/ulimit before starting Fluentd.

Reduce tailed-file count

• Narrow in_tail glob patterns so they do not match files you do not need.
• On dense Kubernetes nodes, confirm you are only tailing /var/log/containers/*.log and not also the underlying pod paths, which would double the watches.
• If logrotate uses nocreate, in_tail can be left holding stale watchers; align rotation config with what in_tail expects, and prefer rename/create over copytruncate for cleaner descriptor handling.

Drain the buffer backlog

If chunk files dominate, the real fix is restoring the output. Check the destination independently, read the Fluentd error log for the specific failure, and let the queue drain. If the backlog is so large that the process cannot open its own chunk files within the limit even after raising it, the documented operator workaround is to stop Fluentd, move or delete the buffer directory, and restart. That discards all buffered data. Treat it as a last resort and say so in the incident review.

Fix connection churn

If sockets dominate the FD list, check output flush_thread_count against the actual connection count, and look for destinations behind load balancers closing idle connections (which forces reconnect storms). See Fluentd broken pipe / connection reset for that specific pattern.

Prevention

• Set the limit deliberately: LimitNOFILE=65536 in the systemd unit (the fluent-package unit already ships this), not in limits.conf. Verify on the running process after every package upgrade.
• Alert at 75% of the soft limit, with a page above 90%. The cliff edge means a threshold alert at 99% is an incident notification, not a warning.
• Trend the FD count. Runway math, (soft limit - current FDs) / growth rate, turns a cliff edge into a scheduled change.
• Size before onboarding new sources: expected FDs = tailed files + (output connections x flush threads) + buffer chunks + ~50. Recompute when pod counts or output destinations change.
• Watch buffer growth as an FD leading indicator: every stalled-output hour is chunk files and descriptors. Time-to-overflow calculations for the buffer double as FD runway estimates; see the buffer guides below.
• In multi-worker mode, monitor per worker. Each worker has its own limit and its own FD count; the aggregate can look fine while one worker (usually the one in_tail is pinned to) is at the wall.

How Netdata helps

• Netdata tracks per-process open file descriptor counts against the configured limit, so the 75% threshold alert fires before the cliff, not after.
• Correlating FD count with Fluentd’s buffer_queue_length and retry_count from the monitor_agent shows whether descriptor growth is input-side (tailed files) or output-side (chunk accumulation), which is the key diagnostic fork.
• Input emit_records rate next to FD count exposes the silent failure: FDs at the limit plus input rate dropping means in_tail has stopped picking up new files.
• Per-second sampling makes the growth ramp into exhaustion visible as a trend, so the runway calculation is actionable during capacity reviews rather than only during incidents.
• Process-level RSS, CPU, and restart counts alongside FD usage help distinguish exhaustion from the adjacent failure modes (OOM, crash loops) that produce similar “pipeline stalled” symptoms.

Related guides

• Fluentd broken pipe / connection reset: dropped output connections and LB timeouts
• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd buffer_oldest_timekey lag: how far behind the oldest buffered data is
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss
• Fluentd emit_error_count: the number-one under-monitored data-loss signal
• Fluentd input emit_records stuck at zero: enable_input_metrics on older versions
• Fluentd end-to-end pipeline latency: stale logs during an incident
• Fluentd failed to flush the buffer: the output cannot deliver and retries begin