Fluentd per-worker imbalance: one struggling worker hidden by aggregate metrics

Your Fluentd dashboards look fine. Input rate is steady, summed output rate roughly matches, and average buffer usage across the instance is comfortably below the limit. Then someone notices a three-hour gap in the destination for a subset of logs, or the node starts flapping on disk pressure, and the aggregate metrics still insist nothing is wrong.

This is the defining trap of multi-worker Fluentd. When you set workers N in <system>, Fluentd spawns N independent Ruby processes. Each worker has its own event router, its own buffers, its own flush threads, its own retry state, and its own memory footprint. Workers do not share buffer state. If worker 1’s output is stuck in a retry storm while workers 0, 2, and 3 are draining normally, any metric you sum or average across workers will report “mostly healthy” right up until worker 1 starts losing data.

What this means

Multi-worker mode exists because a single Fluentd worker is constrained by the Ruby GVL: CPU-bound work like parsing and serialization does not parallelize within one process. Workers are the only way to use multiple cores. The cost is operational: you no longer have one pipeline, you have N pipelines that happen to share a supervisor and a config file.

Three consequences matter for monitoring:

• Independent buffers. Each worker accumulates its own chunks against its own total_limit_size. One worker can hit overflow while the others have empty queues.
• Independent retry state. retry_count, retry.steps, and retry.next_time are per worker per output. Worker 1 can be 30 minutes from its next retry attempt while worker 0 flushes every few seconds.
• Independent metrics endpoints. The monitor_agent HTTP server auto-increments its port per worker: worker 0 answers on 24220, worker 1 on 24221, and so on. If you only scrape 24220, you are only watching worker 0.

Summing hides imbalance in both directions. A full buffer on one worker averaged with three empty ones reads as 25% used. A retry counter that is flat on three workers and climbing on one reads as a slow, unalarming creep when summed.

flowchart LR
  subgraph inputs[Inputs]
    F[in_forward / in_http]
    T[in_tail pinned to worker 0]
  end
  subgraph sup[Fluentd supervisor]
    W0[worker 0\nbuffer A\nport 24220]
    W1[worker 1\nbuffer B FULL, retrying\nport 24221]
    W2[worker 2\nbuffer C\nport 24222]
  end
  F -->|SO_REUSEPORT, kernel picks worker| W0
  F --> W1
  F --> W2
  T --> W0
  W0 --> OUT[(destination)]
  W1 -.->|write stalled| OUT
  W2 --> OUT
  AGG[aggregate scrape or summed PromQL] -.->|masks worker 1| W0
  AGG -.-> W1
  AGG -.-> W2

Common causes

Quick checks

All read-only. The core move in every check is the same: query each worker’s port, not just 24220.

# Confirm worker count and ports are listening
ss -tlnp | grep -E '2422[0-9]'

# Buffer health per worker: queue length and available space
for p in 24220 24221 24222 24223; do
  echo "== port $p =="
  curl -s http://localhost:$p/api/plugins.json | \
    jq '.plugins[] | select(.plugin_category=="output") |
        {id: .plugin_id,
         queue: .buffer_queue_length,
         avail_pct: .buffer_available_buffer_space_ratios,
         retries: .retry_count,
         writes: .write_count}'
done

# Input vs output record rates per worker (counters; sample twice and diff)
# Input emit_records requires enable_input_metrics on older Fluentd versions
for p in 24220 24221 24222 24223; do
  echo "== port $p =="
  curl -s http://localhost:$p/api/plugins.json | \
    jq '{in: [.plugins[] | select(.plugin_category=="input") | .emit_records // 0] | add,
         out: [.plugins[] | select(.plugin_category=="output") | .emit_records // 0] | add}'
done

# Retry state detail: how far out is the next attempt per worker
curl -s "http://localhost:24221/api/plugins.json?with_retry=true" | \
  jq '.plugins[] | select(.plugin_category=="output") |
      {id: .plugin_id, retry_count: .retry_count, retry: .retry}'

# Per-worker RSS and CPU (each worker is its own PID)
ps -o pid,rss,%cpu,etime,cmd -C ruby | grep -i fluent
# or
pgrep -af fluentd

# Per-worker file descriptor usage (needs read access to /proc/<pid>/fd)
for pid in $(pgrep -f fluentd); do
  echo "$pid: $(ls /proc/$pid/fd 2>/dev/null | wc -l) fds"
done

# Which worker is running in_tail (it must be pinned to one worker)
for p in 24220 24221 24222 24223; do
  echo "== port $p =="
  curl -s http://localhost:$p/api/plugins.json | \
    jq '.plugins[] | select(.type=="tail") | {id: .plugin_id, tracked: .tracked_file_count}'
done

How to diagnose it

1. Enumerate the workers. Get the configured worker count from <system> workers N</system> and verify N monitor ports are listening (24220 through 24220+N-1). If a port is missing, that worker died or never started; check the Fluentd log and dmesg | grep -i oom for an OOM kill on one worker while the supervisor stayed up.

2. Snapshot buffer state on every port. Pull buffer_queue_length, buffer_available_buffer_space_ratios, retry_count, and write_count from each worker’s /api/plugins.json. Imbalance shows up immediately as one worker diverging from the pack. Note which output plugin on the struggling worker is affected.

3. Characterize the struggling worker. Three shapes cover most cases:

   • Retrying and backing up: retry_count rising, write_count flat, queue growing. The worker’s output path is failing. Read its log lines for the specific error (connection refused, reset, broken pipe, 401/403, 429).
   • Busy but healthy output: write_count incrementing, queue stable or growing slowly, but input rate far above the other workers. The worker is overloaded at the input or parse stage, not the output stage.
   • Input starved: low input emit_records on the other workers while this one is hot. Distribution problem upstream.

4. Check input distribution. Compare input emit_records deltas across workers over a fixed window (60 seconds is enough). Network inputs use SO_REUSEPORT, so the kernel hashes connections across workers; long-lived forward connections from a small number of senders can pin most of the volume onto one worker. If in_tail is in use, it does not support multi-worker and must be pinned with <worker N>, so its entire load is on one worker by design. The question becomes whether that worker is sized for it.

5. Check the worker’s resources. If one worker’s CPU is pinned at ~100% of a core while others sit lower, that worker is GVL-bound: parsing or filtering is its ceiling, and events queue behind it regardless of destination health. If one worker’s RSS or FD count is climbing ahead of the pack, it will hit OOM or EMFILE alone, and the supervisor process staying alive will make the partial outage easy to miss.

6. Verify against the destination. Per-worker imbalance can also originate downstream: a destination behind a load balancer can drop or throttle one worker’s connections (idle-timeout broken pipe on long-lived connections) while others keep working. Compare flush latency per worker via flush_time_count / write_count deltas.

Metrics and signals to monitor

Every signal below must be collected per worker port. The warning sign is divergence between workers, not the absolute value.

Fixes

Rebalance or pin the input deliberately

If network inputs distribute unevenly because a few long-lived senders dominate, add more sender-side connections or place a balancing layer in front of in_forward. If in_tail is the hot path, accept that it lives on one worker and treat that worker as a dedicated ingestion pipeline: watch its CPU, position lag, and buffer separately, and consider whether its outputs need their own tuning. Do not try to run in_tail on multiple workers; it does not support multi-worker and must be pinned with <worker N>.

Fix the worker-specific output failure

When only one worker retries, the cause is usually in that worker’s connection path, not the destination as a whole. Check for destination-side connection timeouts cutting long-lived connections (the classic broken pipe after idle), and for credentials or network policy that differ per source. A restart of Fluentd resets retry state, but treat that as recovery after fixing the underlying error, not as the fix.

Relieve a CPU-bound worker

If one worker is GVL-saturated, the durable fix is reducing per-worker CPU work: simplify complex regex parsers, prefer structured formats, and move expensive filter logic off the hot path. Adding more workers only helps if the input can actually spread across them.

Right-size per-worker buffers

total_limit_size applies per output plugin per worker. If one worker legitimately carries more load (for example, the one running in_tail), a uniform buffer config is effectively a smaller buffer for that worker relative to its traffic. Size buffers against the busiest worker’s observed throughput, and keep filesystem headroom for the sum across workers when using file-backed buffers on a shared partition.

Prevention

• Scrape every worker port. Enumerate 24220 through 24220+N-1 as separate targets. A scrape of only 24220 is monitoring one worker out of N.
• Alert on per-worker values, not just sums. Keep aggregate dashboards for capacity, but page and ticket on per-worker thresholds: available buffer space, retry state, and queue growth per port.
• Track distribution as a metric. Compute each worker’s share of input emit_records rate and alert when one worker’s share drifts far from 1/N for a sustained window.
• Set @id on every plugin. Without explicit @id, correlating a plugin’s metrics across workers and across restarts is guesswork. Stable IDs are what make per-worker comparison possible.
• Watch per-worker process health. A dead worker does not necessarily show up as a dead service; the supervisor stays up. Alert when the number of responding monitor ports drops below N.

How Netdata helps

• Netdata collects Fluentd monitor_agent metrics per plugin and per worker endpoint, so buffer_queue_length, buffer_available_buffer_space_ratios, retry_count, and write_count stay disaggregated instead of being collapsed into an average that hides the struggling worker.
• Per-second collection catches short retry and queue cycles on a single worker that minute-level sampling smooths away.
• Correlating a worker’s buffer growth with its flat write_count and rising retry_count on one dashboard separates “destination failing for this worker” from “worker overloaded at input” without manual curl loops.
• Host-level per-PID metrics (RSS, CPU, open FDs) alongside the per-worker API metrics make it visible when one worker is heading for OOM or FD exhaustion while the supervisor and siblings look fine.
• Anomaly detection on per-worker series flags the divergence itself (one worker deviating from its peers and its own baseline), which is the actual failure signature of this pattern.

Related guides

• Fluentd broken pipe / connection reset: dropped output connections and LB timeouts
• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd file buffer filling the disk: when the buffer partition runs out
• Fluentd buffer_oldest_timekey lag: how far behind the oldest buffered data is
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss
• Fluentd duplicate events: why the same log shows up twice downstream
• Fluentd emit_error_count: the number-one under-monitored data-loss signal
• Fluentd input emit_records stuck at zero: enable_input_metrics on older versions