Fluentd in_tail with multi-worker: pinning to a single worker

You added workers 4 to <system> to get past the single-core ceiling the Ruby GVL imposes on one Fluentd process, and now Fluentd refuses to start. The log says: Plugin 'tail' does not support multi workers configuration (Fluent::Plugin::TailInput). Or worse: it starts, and you discover weeks later that some log files were read twice by different workers while others were never read at all.

This is a design constraint, not a bug. in_tail is explicitly not multi-worker-ready: it maintains per-file position state, open file descriptors, and rotation tracking that cannot be shared or split across independent worker processes. The fix is to pin every in_tail source to exactly one worker with the <worker N> directive, and to adjust monitoring to match, because only that worker’s monitor_agent port exposes the tail metrics.

Why in_tail cannot run across workers

Multi-worker mode (available since v0.14.12) spawns N independent Ruby worker processes, each with its own event loop, router, buffers, and flush threads. Workers share no state. For inputs like in_forward this works because the supervisor opens a shared socket that all workers serve, so any worker can pick up a connection.

in_tail is different. A tail source is bound to specific files on local disk. It tracks each file by path, inode, and byte offset in its pos_file, holds an open descriptor per watched file, and detects rotation by comparing inodes and file sizes over time. If two workers instantiate the same tail source, both read the same files independently: duplicate events downstream and competing writes to any shared pos_file. There is no built-in mechanism to partition a wildcard’s matched files across workers, so a split-by-hand glob either overlaps or leaves gaps at rotation time. The plugin’s multi_workers_ready? returns false precisely to fail fast at startup instead of corrupting your stream.

The <worker N> directive (introduced in v0.14.15) scopes a block of configuration to one zero-indexed worker process so non-multi-worker plugins can coexist with a multi-worker pipeline.

flowchart TD
  S[Supervisor process] --> W0[Worker 0]
  S --> W1[Worker 1]
  S --> W2[Worker N]
  W0 --> T[in_tail pinned here]
  W0 --> M0[monitor_agent :24220]
  W1 --> F[in_forward / filters / outputs]
  W1 --> M1[monitor_agent :24221]
  W2 --> F2[in_forward / filters / outputs]
  W2 --> M2[monitor_agent :24222]
  T --> P[pos_file owned by worker 0 only]

Prerequisites

• Fluentd v0.14.15 or later (any v1.x release qualifies). <worker N> does not exist on older versions.
• Multi-worker mode declared in <system>: workers N. Without it, <worker N> blocks are unnecessary.
• A dedicated pos_file path per tail source, on persistent local disk. Not tmpfs, not a shared filesystem, and never shared between two tail configurations.
• monitor_agent enabled if you want per-worker metrics. Examples below use td-agent paths; adjust for fluent-package (/etc/fluent/fluentd.conf, /var/log/fluent/fluentd.log) or your container layout.

Procedure

1. Confirm the failure or plan the migration. If Fluentd is failing to start, confirm the cause:

# Check for the multi-worker rejection at startup
journalctl -u td-agent --since "10 minutes ago" | grep -i "does not support multi workers"

2. Decide which worker owns the tails. Convention is worker 0. Tail work is I/O-bound (inotify plus reads), so it coexists fine with the routing and flush load that worker also carries, unless you have hundreds of watched files.

3. Wrap each tail source in a <worker N> block. Worker indexes are zero-based:

<system>
  workers 4
</system>

<worker 0>
  <source>
    @type tail
    path /var/log/app/*.log
    pos_file /var/log/td-agent/app.pos
    tag app.logs
    <parse>
      @type json
    </parse>
  </source>
</worker>

Several distinct tail sources (different paths, different tags) can all live inside the same <worker 0> block, or you can split them across workers explicitly, one source per <worker N> block, with a unique pos_file for each.

4. Keep network inputs and outputs unpinned. Sources like in_forward and in_http, plus filters and outputs, stay at top level so all workers run them. That is where the parallelism you enabled workers for comes from.

5. Make buffer paths worker-safe. Output buffers are per worker. Include #{worker_id} in file buffer paths so workers do not collide on chunk files, for example path /var/log/td-agent/buffer/#{worker_id}/es. Omitting this risks chunk file corruption and loss that looks like an output problem but is really two workers writing the same directory.

6. Validate and restart. Check config syntax, then restart. Use a full restart, not SIGHUP, when changing worker topology:

# Validate config before restarting
td-agent --dry-run -c /etc/td-agent/td-agent.conf
systemctl restart td-agent

Restarting interrupts log collection briefly. File-backed pos_files and buffers mean in-flight data survives, but schedule it accordingly.

Verifying it works

The defining detail: with workers N and monitor_agent on port 24220, worker 0 serves on 24220, worker 1 on 24221, and so on. Tail metrics exist only on the worker that owns the tail source.

# Confirm all workers are up (supervisor plus N children)
pgrep -af fluentd

# Tail metrics only appear on the pinned worker's port
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.type=="tail") | {id: .plugin_id, tracked: .tracked_file_count, rotated: .rotated_file_count}'

# The same query on another worker's port returns nothing for type tail
curl -s http://localhost:24221/api/plugins.json | \
  jq '[.plugins[] | select(.type=="tail")] | length'

If the first query returns your tail plugin and the second returns 0, the pinning is correct. Then sanity-check the data path: tracked_file_count should match the number of files your glob actually matches (ls /var/log/app/*.log | wc -l), and input emit_records on worker 0 should be non-zero and growing. On Fluentd older than v1.19.0, input emit_records requires enable_input_metrics true in <system>; without it the counter reads 0 and looks like the tail is broken when it is not.

Finally, verify no duplicates downstream. Query your destination for a known host and time window and check that each log line appears exactly once. Duplicates after this change almost always mean two workers are still reading the same path (see pitfalls).

Common pitfalls

• Tail source left at top level. Fluentd fails to start with Plugin 'tail' does not support multi workers configuration. This is the safe failure. Fix by wrapping in <worker N>.

• The same path pinned to two workers. Wrapping the same path in both <worker 0> and <worker 1> blocks starts cleanly and reads every file twice. Duplicates, not an error. Each worker must own a disjoint set of files.

• Wildcard paths split by hand. There is no built-in way to hash a glob across workers. <worker 0> with path /var/log/a/*.log and <worker 1> with path /var/log/b/*.log works because the sets are disjoint. Two blocks with the same glob do not.

• Shared pos_file. Two tail configurations pointing at one pos_file corrupt each other’s offsets: competing writes produce wrong positions, surfacing as re-reads (duplicates) or skipped ranges (gaps). One pos_file per tail source, always.

• pos_file on volatile storage. On tmpfs or a container layer, restart wipes positions. With read_from_head true you re-ingest everything; with false you skip to EOF and lose the gap. Persist it on disk, and in Kubernetes mount a hostPath or volume for it.

• Monitoring only port 24220. If you pinned tails to worker 1 (port 24221) and your scraper only hits 24220, tracked_file_count, rotated_file_count, and throttled_log_count silently vanish from your dashboards. Alert rules then either go dark or fire on absence. Scrape every worker port and know which one owns the tails.

• Assuming worker 0 health equals pipeline health. In multi-worker mode, individual workers can die while the supervisor stays “active” per systemd. A dead worker 0 with the tails on it is a full collection outage that process-level checks miss.

Signals to monitor

Duplicate detection is downstream, not in Fluentd: no Fluentd metric counts “events read by two workers.” If duplicates are a real risk in your setup, spot-check the destination periodically.

How Netdata helps

• Netdata collects Fluentd’s monitor_agent output, so per-worker visibility comes down to pointing it at each worker port; once scraped, tail metrics from the pinned worker chart alongside buffer and retry metrics from the others.
• Correlating tracked_file_count with input emit_records on the same worker separates “files disappeared” from “files exist but are not being read” in one view.
• Comparing input and output emit rates per worker surfaces the imbalance pattern (a two-worker duplicate read shows up as output volume roughly doubling input volume), which is otherwise invisible in Fluentd’s own metrics.
• Per-process CPU and RSS charts for each worker catch the “supervisor healthy, one worker dead or leaking” case that systemd status hides.
• Buffer queue length and available space ratio per worker make it obvious when the tail-owning worker’s pipeline is backing up while the forwarding workers are fine.

Related guides

• Fluentd broken pipe / connection reset: dropped output connections and LB timeouts
• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd file buffer filling the disk: when the buffer partition runs out
• Fluentd buffer_oldest_timekey lag: how far behind the oldest buffered data is
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd CPU bottleneck: the Ruby GVL caps a single worker at one core
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss
• Fluentd duplicate events: why the same log shows up twice downstream
• Fluentd emit_error_count: the number-one under-monitored data-loss signal