Fluentd monitoring checklist: the signals every production log pipeline needs

Most Fluentd monitoring setups answer one question: is the process running? That is necessary and nowhere near sufficient. A Fluentd process can be alive, responsive, and completely idle because the buffer is full and the default overflow_action (throw_exception) is dropping every new event at the input. No error counter reliably increments for that path. The process looks healthy while the pipeline loses data.

This checklist covers what to actually watch, in two parts: the core signal set every production deployment needs regardless of size, and the maturity levels that tell you what to add as the pipeline becomes more critical. Everything here comes from the monitor_agent API (default port 24220), Fluentd’s own logs, and a small number of OS-level signals.

Use it three ways: as an audit of your current monitoring (walk the core list and check coverage), as an alert design spec (each signal includes its warning condition), and as a maturity assessment (find the highest level you fully cover).

How to use this checklist

Each signal lists its source and its warning condition. Two rules apply to almost everything below:

• Counters vs gauges. emit_records, retry_count, rollback_count, write_count, flush_time_count, drop_oldest_chunk_count are cumulative counters. They only reset on Fluentd restart. Never alert on the raw value; alert on the rate (delta over time). buffer_queue_length, buffer_total_queued_size, buffer_available_buffer_space_ratios are point-in-time gauges; alert on level plus trend.
• Per-worker in multi-worker mode. Each worker is an independent Ruby process with its own buffers and its own monitor_agent port (worker 0 is 24220, worker 1 is 24221, and so on). Aggregate dashboards mask a single struggling worker. Collect from every worker’s port.

Prerequisite: the monitor_agent input plugin must be configured:

<source>
  @type monitor_agent
  bind 127.0.0.1
  port 24220
</source>

Since v1.19.3, include_config, include_retry, and include_debug_info default to false (the fix for CVE-2026-44025, which previously exposed internal plugin state including credentials); the default bind is still 0.0.0.0, so bind monitor_agent to localhost or firewall the port. If you are on v1.19.2 or older, either upgrade or bind monitor_agent to localhost. If you need the retry object (retry state per output), set include_retry true explicitly on v1.19.3+; the with_retry and with_config query parameters are no longer available.

The core signal set

If you monitor nothing else, monitor these.

A note on the data-loss row: Fluentd core does not expose an emit_error_count field in the monitor_agent output. The dependable, documented proxies are drop_oldest_chunk_count (exposed since v1.19.0; increments only when overflow_action drop_oldest_chunk discards a chunk), rollback_count (chunks that failed and went back to the queue), and write_secondary_count (exposed since v1.19.0; chunks that fell through to the secondary output after primary retry exhaustion). Alert on any increment of any of these. And remember the blind spot: with the default throw_exception overflow action, drops happen in the input path and none of these counters fire. Your only signals are buffer_available_buffer_space_ratios pinned near 0%, BufferOverflowError warnings in the Fluentd log, and the input/output rate gap.

Quick collection commands

# Process alive (adjust unit name for your package: td-agent or fluentd)
systemctl is-active td-agent

# Monitor agent responsiveness
curl -s -o /dev/null -w "%{http_code}" --max-time 5 \
  http://localhost:24220/api/plugins.json

# Input vs output emit_records totals
curl -s http://localhost:24220/api/plugins.json | \
  jq '[.plugins[] | select(.plugin_category=="input") | .emit_records // 0] | add'
curl -s http://localhost:24220/api/plugins.json | \
  jq '[.plugins[] | select(.plugin_category=="output") | .emit_records // 0] | add'

# Buffer state per output
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") |
      {id: .plugin_id, queue: .buffer_queue_length,
       avail_pct: .buffer_available_buffer_space_ratios,
       retries: .retry_count, dropped: .drop_oldest_chunk_count}'

# RSS and open FDs (host-level, not in the API)
ps -o rss= -p $(pgrep -f fluentd | head -1) | awk '{print $1/1024 " MB"}'
ls /proc/$(pgrep -f fluentd | head -1)/fd 2>/dev/null | wc -l

Maturity levels

flowchart TD
  L1["Level 1 - Survival
process alive, log errors, destination reachable"]
  L2["Level 2 - Operational
monitor_agent, buffer and retry metrics, RSS, rate balance"]
  L3["Level 3 - Mature
per-output breakdown, flush latency, drop counters, FDs, in_tail"]
  L4["Level 4 - Expert
end-to-end latency, duplicates, per-worker, routing completeness"]
  L1 --> L2 --> L3 --> L4

Level 1: Survival

The floor. You know the pipeline exists at all.

• Process alive, gated on sustained absence (more than 2 minutes) so restarts and rolling updates do not page. In Kubernetes, CrashLoopBackOff shows up as repeated brief absences rather than one long one; count restarts too.
• Error scan of Fluentd’s own log (/var/log/td-agent/td-agent.log or /var/log/fluent/fluentd.log depending on package). The log pipeline’s own logs are the last place teams look.
• Destination reachable: an independent check that the output target responds.

This catches total failure. It misses every form of silent degradation.

Level 2: Operational

You know when the pipeline is backing up or retrying. This is where the core signal set lives.

• monitor_agent enabled and scraped on every worker port.
• buffer_queue_length per output. Distinguish stage from queue: buffer_stage_length high with buffer_queue_length low is normal batching. Queue growth is backpressure.
• buffer_available_buffer_space_ratios per output, alerted below 20% while buffer_total_queued_size is still growing. Compute time-to-overflow as available space divided by growth rate, not a static threshold. Defaults are 512MB total for memory buffers and 64GB for file buffers, so “20% free” means very different things.
• retry_count rate per output. Any sustained non-zero rate is abnormal. It is cumulative and clears only on restart: alert on the delta, never the value.
• Input vs output emit_records balance. Over a window of at least max(2 * flush_interval, 10 minutes), output rate should track input rate. Require input rate above zero to avoid dividing by zero on idle hosts, and confirm with growing buffer_queue_length. On Fluentd older than v1.19.0, input emit_records is always 0 unless you set enable_input_metrics true in <system>; on v1.19.0+ input metrics are on by default.
• RSS vs memory limit. Ruby RSS grows to a fragmentation plateau and stays there; a high but stable plateau is normal. Alert on monotonic growth, and in containers alert above 80% of the limit because OOM kills are immediate and unbuffered memory-buffer data dies with the process.

Level 3: Mature

You can say where and why the pipeline is degrading, not just that it is.

• Average flush time: delta(flush_time_count) / delta(write_count) per output. Rising flush time is the earliest destination-degradation signal, appearing before retries start. Pair with slow_flush_count (flushes over slow_flush_log_threshold, default 20s) as a ratio of slow flushes to total writes.
• The drop counters: drop_oldest_chunk_count, write_secondary_count, rollback_count. Any increment of drop_oldest_chunk_count is confirmed data loss.
• buffer_oldest_timekey for data freshness: now - buffer_oldest_timekey beyond roughly 2 * flush_interval (or timekey + timekey_wait for time-sliced outputs) means old data is stuck. Safer than rate comparisons for bursty or time-sliced workloads. Only present with time-based chunking.
• FD count vs ulimit -Sn. FDs are consumed by tailed files, buffer chunk files, and output connections. Exhaustion produces “too many open files” and can silently stop in_tail from following new files. Set the limit generously (65536 is the common recommendation) and alert at 75% of the soft limit.
• in_tail file tracking (v1.14.1+ for rotated_file_count; v1.17.1+ for throttled_log_count; v1.19.0+ for the tracked_file_count gauge). A drop in tracked files means a glob stopped matching or rotation ran away. throttled_log_count incrementing means input-side data loss under group rate limiting.
• The retry object, not just the counter. With include_retry true, retry.steps and retry.next_time tell you how stalled the pipeline really is. If retry.next_time is 30 minutes out, exponential backoff has effectively parked the chunk; a restart resets retry state once the destination is fixed.
• Stage vs queue divergence as a diagnostic: queue length sustained above roughly 5x stage length means output is struggling.

Level 4: Expert

You catch silent failures, not just loud ones.

• End-to-end latency: event timestamp vs arrival time at the destination, or synthetic events injected with fluent-cat. Fluentd exposes no single metric for this.
• Duplicate detection at the destination after restarts. File buffers replay unflushed chunks by design; exactly-once is not guaranteed.
• Per-worker decomposition of every buffer and throughput metric.
• Tag routing completeness: expected tags vs tags actually observed, to catch events flowing to a null output while all rates look healthy.
• pos_file integrity: the position file has no checksums, and staleness or corruption causes silent re-reads or gaps. Watch input emit_records for unexplained spikes or drops around rotation windows.
• Security signals: unauthorized inbound connections to forward/HTTP input ports, outbound connections outside the destination allowlist, config file changes outside the deployment pipeline.

Gotchas that quietly break this checklist

• Alerting on raw retry_count. It never decreases between restarts. A destination that failed once last month looks identical to one failing now. Use rate.
• buffer_available_buffer_space_ratios on old versions. A rounding bug made this metric report only 0 or 100 on versions before v1.10.0 (fixed in v1.10.0). Treat it as unreliable on anything older.
• Trusting emit_records as an exact record counter for outputs. emit_records counts records written into the buffer; a chunk that is written, rolled back, and retried can be counted again, so the counter can overstate unique records, and every restart resets all counters. For monotonic accounting, prefer chunk-level counters like write_count and rollback_count, and handle counter resets in rate()/increase().
• File buffer on a shared partition. total_limit_size is not the real ceiling if the buffer directory shares a filesystem with system logs. Watch filesystem free space on the buffer mount alongside the ratio.
• Time-sliced outputs look like lag. Outputs that flush on time boundaries (daily S3 files) hold chunks until the slice expires. Input/output rate divergence and old buffer_oldest_timekey are expected there; use the timekey + timekey_wait bound instead.
• Zombie liveness. A process can exist while the event loop is hung (GC storm, deadlock, blocked I/O). That is why the checklist pairs process liveness with monitor_agent responsiveness; a live PID with a dead port is a hung pipeline.

How Netdata helps

• Netdata collects per-process RSS, CPU, and open FD counts from /proc at per-second resolution, which covers the two host-level core signals (RSS vs limit, FDs vs ulimit) without custom scripting.
• Process liveness and restart counting are visible as process presence over time, which makes CrashLoopBackOff oscillation and sustained-absence gating straightforward to evaluate.
• Because Netdata already watches the destination side of the pipeline on the same hosts (Elasticsearch, Kafka, disk, network), you can correlate a rising Fluentd flush time or retry rate with destination saturation in one view instead of two dashboards.
• Filesystem free space on the buffer mount is collected alongside the process metrics, so the file-buffer cliff edge (partition full before total_limit_size) shows up next to the buffer symptoms it causes.
• Anomaly detection on throughput metrics flags the input/output rate divergence pattern that indicates silent loss, without requiring a hand-tuned static threshold per host.

Related guides

• How Fluentd actually works in production: a mental model for operators
• Fluentd monitoring maturity model: from survival to expert
• Fluentd process not running: the log pipeline is dead and the host has gone dark
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• Fluentd poison pill crash loop: one bad log line that kills the process on every restart
• Fluentd monitor_agent not responding: a process that is up but hung
• Fluentd plugin load error at startup: LoadError and missing gems
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer stage vs queue: telling healthy batching from backpressure
• Fluentd buffer available space low: computing time-to-overflow before it fires