Most Fluentd monitoring setups answer one question: is the process running? That is necessary and nowhere near sufficient. A Fluentd process can be alive, responsive, and completely idle because the buffer is full and the default overflow_action (throw_exception) is dropping every new event at the input. No error counter reliably increments for that path. The process looks healthy while the pipeline loses data.

This checklist covers what to actually watch, in two parts: the core signal set every production deployment needs regardless of size, and the maturity levels that tell you what to add as the pipeline becomes more critical. Everything here comes from the monitor_agent API (default port 24220), Fluentd’s own logs, and a small number of OS-level signals.

Use it three ways: as an audit of your current monitoring (walk the core list and check coverage), as an alert design spec (each signal includes its warning condition), and as a maturity assessment (find the highest level you fully cover).

How to use this checklist

Each signal lists its source and its warning condition. Two rules apply to almost everything below:

  • Counters vs gauges. emit_records, retry_count, rollback_count, write_count, flush_time_count, drop_oldest_chunk_count are cumulative counters. They only reset on Fluentd restart. Never alert on the raw value; alert on the rate (delta over time). buffer_queue_length, buffer_total_queued_size, buffer_available_buffer_space_ratios are point-in-time gauges; alert on level plus trend.
  • Per-worker in multi-worker mode. Each worker is an independent Ruby process with its own buffers and its own monitor_agent port (worker 0 is 24220, worker 1 is 24221, and so on). Aggregate dashboards mask a single struggling worker. Collect from every worker’s port.

Prerequisite: the monitor_agent input plugin must be configured:

<source>
  @type monitor_agent
  bind 127.0.0.1
  port 24220
</source>

Since v1.19.3, the default bind is 127.0.0.1, and include_config, include_retry, and include_debug_info default to false after CVE-2026-44025 (the API previously exposed internal state including credentials). If you are on v1.19.2 or older, either upgrade or bind monitor_agent to localhost. If you need the retry object (retry state per output), set include_retry true explicitly on v1.19.3+; the old with_retry and with_config query parameters were removed.

The core signal set

If you monitor nothing else, monitor these.

SignalSourceWarning sign
Process alivesystemd, pidfile, pod statusAbsent or unresponsive for more than 2 minutes sustained
monitor_agent responsiveGET /api/plugins.json on port 24220Non-200 or timeout over 5s while process exists
Input vs output emit_records balancemonitor_agent, input and output pluginsOutput rate sustained below input rate
buffer_queue_lengthmonitor_agent, per output pluginSustained upward trend
buffer_available_buffer_space_ratiosmonitor_agent, per output pluginBelow 20% and still filling
retry_count ratemonitor_agent, per output pluginAny sustained non-zero increment rate
emit_error_count / drop countersmonitor_agent, per output pluginAny increment at all
Process RSS vs memory limitOS: /proc/<pid>/status VmRSSMonotonic growth, or above 80% of container limit
Open FDs vs ulimit -SnOS: /proc/<pid>/fd countAbove 75% of soft limit

A note on the data-loss row: the playbook-level name is emit_error_count, and some builds expose it, but field availability varies by version. The dependable, documented proxies are drop_oldest_chunk_count (increments only when overflow_action drop_oldest_chunk discards a chunk), rollback_count (chunks that failed and went back to the queue), and write_secondary_count (chunks that fell through to the secondary output after primary retry exhaustion). Alert on any increment of any of these. And remember the blind spot: with the default throw_exception overflow action, drops happen in the input path and none of these counters fire. Your only signals are buffer_available_buffer_space_ratios pinned near 0%, BufferOverflowError warnings in the Fluentd log, and the input/output rate gap.

Quick collection commands

# Process alive (adjust unit name for your package: td-agent or fluentd)
systemctl is-active td-agent

# Monitor agent responsiveness
curl -s -o /dev/null -w "%{http_code}" --max-time 5 \
  http://localhost:24220/api/plugins.json

# Input vs output emit_records totals
curl -s http://localhost:24220/api/plugins.json | \
  jq '[.plugins[] | select(.plugin_category=="input") | .emit_records // 0] | add'
curl -s http://localhost:24220/api/plugins.json | \
  jq '[.plugins[] | select(.plugin_category=="output") | .emit_records // 0] | add'

# Buffer state per output
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") |
      {id: .plugin_id, queue: .buffer_queue_length,
       avail_pct: .buffer_available_buffer_space_ratios,
       retries: .retry_count, dropped: .drop_oldest_chunk_count}'

# RSS and open FDs (host-level, not in the API)
ps -o rss= -p $(pgrep -f fluentd | head -1) | awk '{print $1/1024 " MB"}'
ls /proc/$(pgrep -f fluentd | head -1)/fd 2>/dev/null | wc -l

Maturity levels

flowchart TD
  L1["Level 1 - Survival
process alive, log errors, destination reachable"]
  L2["Level 2 - Operational
monitor_agent, buffer and retry metrics, RSS, rate balance"]
  L3["Level 3 - Mature
per-output breakdown, flush latency, drop counters, FDs, in_tail"]
  L4["Level 4 - Expert
end-to-end latency, duplicates, per-worker, routing completeness"]
  L1 --> L2 --> L3 --> L4

Level 1: Survival

The floor. You know the pipeline exists at all.

  • Process alive, gated on sustained absence (more than 2 minutes) so restarts and rolling updates do not page. In Kubernetes, CrashLoopBackOff shows up as repeated brief absences rather than one long one; count restarts too.
  • Error scan of Fluentd’s own log (/var/log/td-agent/td-agent.log or /var/log/fluent/fluentd.log depending on package). The log pipeline’s own logs are the last place teams look.
  • Destination reachable: an independent check that the output target responds.

This catches total failure. It misses every form of silent degradation.

Level 2: Operational

You know when the pipeline is backing up or retrying. This is where the core signal set lives.

  • monitor_agent enabled and scraped on every worker port.
  • buffer_queue_length per output. Distinguish stage from queue: buffer_stage_length high with buffer_queue_length low is normal batching. Queue growth is backpressure.
  • buffer_available_buffer_space_ratios per output, alerted below 20% while buffer_total_queued_size is still growing. Compute time-to-overflow as available space divided by growth rate, not a static threshold. Defaults are 512MB total for memory buffers and 64GB for file buffers, so “20% free” means very different things.
  • retry_count rate per output. Any sustained non-zero rate is abnormal. It is cumulative and clears only on restart: alert on the delta, never the value.
  • Input vs output emit_records balance. Over a window of at least max(2 * flush_interval, 10 minutes), output rate should track input rate. Require input rate above zero to avoid dividing by zero on idle hosts, and confirm with growing buffer_queue_length. On Fluentd older than v1.19.0, input emit_records is always 0 unless you set enable_input_metrics true in <system>; on v1.19.0+ input metrics are on by default.
  • RSS vs memory limit. Ruby RSS grows to a fragmentation plateau and stays there; a high but stable plateau is normal. Alert on monotonic growth, and in containers alert above 80% of the limit because OOM kills are immediate and unbuffered memory-buffer data dies with the process.

Level 3: Mature

You can say where and why the pipeline is degrading, not just that it is.

  • Average flush time: delta(flush_time_count) / delta(write_count) per output. Rising flush time is the earliest destination-degradation signal, appearing before retries start. Pair with slow_flush_count (flushes over slow_flush_log_threshold, default 20s) as a ratio of slow flushes to total writes.
  • The drop counters: drop_oldest_chunk_count, write_secondary_count, rollback_count. Any increment of drop_oldest_chunk_count is confirmed data loss.
  • buffer_oldest_timekey for data freshness: now - buffer_oldest_timekey beyond roughly 2 * flush_interval (or timekey + timekey_wait for time-sliced outputs) means old data is stuck. Safer than rate comparisons for bursty or time-sliced workloads. Only present with time-based chunking.
  • FD count vs ulimit -Sn. FDs are consumed by tailed files, buffer chunk files, and output connections. Exhaustion produces “too many open files” and can silently stop in_tail from following new files. Set the limit generously (65536 is the common recommendation) and alert at 75% of the soft limit.
  • in_tail file tracking (v1.14.1+ for rotated_file_count, throttled_log_count; v1.19.0+ for the tracked_file_count gauge). A drop in tracked files means a glob stopped matching or rotation ran away. throttled_log_count incrementing means input-side data loss under group rate limiting.
  • The retry object, not just the counter. With include_retry true, retry.steps and retry.next_time tell you how stalled the pipeline really is. If retry.next_time is 30 minutes out, exponential backoff has effectively parked the chunk; a restart resets retry state once the destination is fixed.
  • Stage vs queue divergence as a diagnostic: queue length sustained above roughly 5x stage length means output is struggling.

Level 4: Expert

You catch silent failures, not just loud ones.

  • End-to-end latency: event timestamp vs arrival time at the destination, or synthetic events injected with fluent-cat. Fluentd exposes no single metric for this.
  • Duplicate detection at the destination after restarts. File buffers replay unflushed chunks by design; exactly-once is not guaranteed.
  • Per-worker decomposition of every buffer and throughput metric.
  • Tag routing completeness: expected tags vs tags actually observed, to catch events flowing to a null output while all rates look healthy.
  • pos_file integrity: the position file has no checksums, and staleness or corruption causes silent re-reads or gaps. Watch input emit_records for unexplained spikes or drops around rotation windows.
  • Security signals: unauthorized inbound connections to forward/HTTP input ports, outbound connections outside the destination allowlist, config file changes outside the deployment pipeline.

Gotchas that quietly break this checklist

  • Alerting on raw retry_count. It never decreases between restarts. A destination that failed once last month looks identical to one failing now. Use rate.
  • buffer_available_buffer_space_ratios on old versions. A rounding bug made this metric report only 0 or 100 on versions before v1.10.0 (fixed by v1.16.0 at the latest). Treat it as unreliable on anything older.
  • Trusting emit_records as a strict counter for outputs. Rollbacks can make output record counters non-monotonic in some versions, which poisons naive rate() calculations. For monotonic accounting, chunk-level counters like write_count are safer.
  • File buffer on a shared partition. total_limit_size is not the real ceiling if the buffer directory shares a filesystem with system logs. Watch filesystem free space on the buffer mount alongside the ratio.
  • Time-sliced outputs look like lag. Outputs that flush on time boundaries (daily S3 files) hold chunks until the slice expires. Input/output rate divergence and old buffer_oldest_timekey are expected there; use the timekey + timekey_wait bound instead.
  • Zombie liveness. A process can exist while the event loop is hung (GC storm, deadlock, blocked I/O). That is why the checklist pairs process liveness with monitor_agent responsiveness; a live PID with a dead port is a hung pipeline.

How Netdata helps

  • Netdata collects per-process RSS, CPU, and open FD counts from /proc at per-second resolution, which covers the two host-level core signals (RSS vs limit, FDs vs ulimit) without custom scripting.
  • Process liveness and restart counting are visible as process presence over time, which makes CrashLoopBackOff oscillation and sustained-absence gating straightforward to evaluate.
  • Because Netdata already watches the destination side of the pipeline on the same hosts (Elasticsearch, Kafka, disk, network), you can correlate a rising Fluentd flush time or retry rate with destination saturation in one view instead of two dashboards.
  • Filesystem free space on the buffer mount is collected alongside the process metrics, so the file-buffer cliff edge (partition full before total_limit_size) shows up next to the buffer symptoms it causes.
  • Anomaly detection on throughput metrics flags the input/output rate divergence pattern that indicates silent loss, without requiring a hand-tuned static threshold per host.