Fluentd monitoring maturity model: from survival to expert

Most teams monitoring Fluentd stop at “is the process running?” and discover the gap during an incident, when the SIEM is missing the exact logs they need for a postmortem. Fluentd can be alive, responsive, and green on every dashboard while silently dropping events, accumulating a buffer that will overflow in forty minutes, or retrying into a backoff so deep the pipeline is effectively dead.

This is a maturity model for Fluentd monitoring: four levels, each with the specific signals, collection commands, and failure modes it catches. Use it as a self-assessment. Find the highest level where you have every signal covered, with alerting, in production. Everything above that is your roadmap.

The model is cumulative. Level 3 without Level 1 coverage is a monitoring illusion: sophisticated dashboards on top of a pipeline that can still die unobserved.

flowchart TD
  L1["Level 1: Survival
Process alive, emit errors,
retries, RSS, error logs"]
  L2["Level 2: Operational
monitor_agent, per-output buffers,
in/out rate balance, FD, CPU, disk"]
  L3["Level 3: Mature
pos lag, flush latency, oldest timekey,
stage vs queue, retry backoff, TLS expiry"]
  L4["Level 4: Expert
Ruby GC, per-thread CPU, UDP drops,
inotify, per-worker, config integrity"]
  L1 --> L2 --> L3 --> L4

Level 1: Survival

Goal: you know within minutes if Fluentd is dead, hung, or actively losing data. Five signals. Nothing here requires the monitor_agent API.

The signals

Process liveness. Not just “a PID exists”: in multi-worker mode the supervisor can stay alive while workers die, and systemctl status showing active only confirms the supervisor. Gate the page on sustained absence (more than 2 minutes) so transient restarts, rolling updates, and container rescheduling do not page you. In Kubernetes, CrashLoopBackOff shows up as repeated brief absences rather than one long one.

# Process liveness
systemctl is-active td-agent    # td-agent package
systemctl is-active fluentd     # fluent-package
pgrep -af fluentd               # generic; count workers vs configured N

Emit failures. The data-loss signal. Any failed emit means events were dropped and will never be delivered, most commonly because the buffer hit total_limit_size and the default overflow_action: throw_exception rejected them. Fluentd has no emit_error_count field in the monitor_agent API; watch the emit transaction failed and buffer overflow lines in the Fluentd log and the per-plugin counters (retry_count, drop_oldest_chunk_count).

retry_count per output. Any sustained non-zero value means a destination is failing and events are accumulating in buffers. This is a cumulative counter: treat any increment in a healthy system as a ticket.

Process RSS trend. Ruby memory grows and plateaus; a high-but-stable RSS is normal. A monotonically rising trend over hours is a leak, unbounded memory buffers, or tag explosion, and it ends in an OOM kill. In containers, alert at 80% of the cgroup limit. After an OOM, confirm with dmesg | grep -i oom.

# RSS trend (collect over time, not as a one-shot)
ps -o rss= -p $(pgrep -f fluentd | head -1) | awk '{print $1/1024 " MB"}'

Fluentd error-log check. Fluentd’s own logs are the last place teams look. Tail them for connection failures, TLS errors, and plugin exceptions:

# Error-log check (paths vary: /var/log/td-agent/ or /var/log/fluent/)
grep -iE "error|failed|retry|overflow" /var/log/td-agent/td-agent.log | tail -20

What Level 1 misses

A running process with a full buffer and a stalled output looks identical to a healthy one. Level 1 tells you the pipeline is broken; it does not tell you where, and it does not warn you before data loss starts.

Level 2: Operational

Goal: you know when the pipeline is backing up or retrying, per output, before the buffer overflows. This level requires the monitor_agent API.

Enable the API

<source>
  @type monitor_agent
  bind 127.0.0.1
  port 24220
</source>

Notes: the port auto-increments per worker (worker 0 = 24220, worker 1 = 24221). Bind to localhost or firewall it; the endpoint exposes internal state. Since v1.19.3 the monitor_agent security defaults changed: include_config and include_retry now default to false (and include_debug_info, new in v1.19.3, also defaults to false). If your dashboards or scripts relied on config or retry fields from older versions, verify they still return data after upgrading.

The signals

Per-output buffer_queue_length and buffer_total_queued_size. Queue length tells you the backlog in chunks; total queued bytes tells you the real footprint, since chunk sizes vary. Alert on sustained growth, not a static threshold: legitimate batch work creates temporary queues. Compare total bytes against total_limit_size and, for file buffers, against filesystem free space, whichever binds first.

# Per-output buffer state
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") |
      {id: .plugin_id, queue: .buffer_queue_length,
       bytes: .buffer_total_queued_size,
       avail_pct: .buffer_available_buffer_space_ratios}'

buffer_available_buffer_space_ratios. Percentage of configured buffer space remaining. Below 20% and still filling is a ticket; below 5% and filling, overflow is imminent. Estimate time-to-overflow as available_space / growth_rate.

Input vs output emit_records balance. The single most telling health metric of the pipeline. Compute both rates as deltas over time:

# Input and output record totals (derive rates from deltas)
curl -s http://localhost:24220/api/plugins.json | \
  jq '{in: [.plugins[] | select(.plugin_category=="input") | .emit_records // 0] | add,
       out: [.plugins[] | select(.plugin_category=="output") | .emit_records // 0] | add}'

The output/input ratio should approach 1.0 over a window of at least max(2 * flush_interval, 10 minutes). A sustained deficit means data loss or unbounded buffer growth. Caveats: on Fluentd versions before v1.19.0, input emit_records requires enable_input_metrics true in <system> or the counter is always 0 (the parameter was introduced in v1.14.0 with default false, and the default flipped to true in v1.19.0); verify on your version before trusting the ratio. Time-sliced outputs (daily S3 files) legitimately diverge; use buffer_oldest_timekey (Level 3) for those.

File descriptors. Fluentd holds one FD per tailed file, plus buffer chunk files, plus output connections. Alert above 75% of the soft limit:

# FD usage vs limit
ls /proc/$(pgrep -f fluentd | head -1)/fd | wc -l
grep "Max open files" /proc/$(pgrep -f fluentd | head -1)/limits

The default ulimit -n of 1024 is inadequate for production. FD exhaustion shows up as in_tail silently losing files or outputs failing with cryptic errors, not a clear message.

CPU per worker process. Due to the GVL, a single worker saturates one core: “100% CPU” on the process while the host looks idle means the pipeline is CPU-bound. Sustained usage above roughly 70% of one core caps throughput; plan for multi-worker or simpler parsers.

Buffer disk usage. For file-backed buffers, watch both the buffer directory size and the partition it lives on. Buffer files sharing a partition with system logs is a cascade waiting to happen.

What Level 2 misses

You see that the pipeline is degrading, but not why, and not how stale the data is. Level 2 also cannot see retry state depth, input-side lag, or slow-but-successful flushes.

Level 3: Mature

Goal: you understand where and why the pipeline is degrading, with leading indicators instead of cliff-edge alerts.

The signals

in_tail position lag. Compare the recorded position in the pos_file against actual file size. A gap that grows over time means ingestion is falling behind; an inode mismatch after rotation means Fluentd lost the file:

# pos_file vs reality (format: path<TAB>position<TAB>inode)
while IFS=$'\t' read -r filepath pos inode; do
  printf '%s pos=%s pos_inode=%s actual_size=%s actual_inode=%s\n' \
    "$filepath" "$pos" "$inode" \
    "$(stat -c%s "$filepath" 2>/dev/null || echo 0)" \
    "$(stat -c%i "$filepath" 2>/dev/null || echo MISSING)"
done < /var/log/td-agent/td-agent.pos

Also track tracked_file_count (v1.19.0+, a gauge of files currently tailed) and rotated_file_count (v1.14.1+) to confirm rotation is being detected on schedule.

Flush latency. flush_time_count / write_count gives average flush time in milliseconds. Rising average flush time is the earliest indicator of destination degradation, appearing before retries start. Track slow_flush_count (flushes over slow_flush_log_threshold, default 20s) as the outlier counter. Healthy: average flush time under 50% of flush_interval.

buffer_oldest_timekey. The age of the oldest buffered data. now - buffer_oldest_timekey beyond roughly 2 * flush_interval for non-time-sliced outputs means a severe delivery backlog. This is safer than rate comparisons for time-sliced outputs and bursty workloads.

Stage vs queue distinction. High buffer_stage_length with low buffer_queue_length is healthy batching. Low stage with high queue is backpressure. Teams that watch one “buffer usage” number miss this entirely. Rule of thumb: queue sustained above 5x stage means the output is struggling.

Retry backoff state. retry_count tells you errors happened; the retry object tells you how bad it is. retry.steps climbing with retry.next_time far in the future means the pipeline is effectively stalled even while “retrying”. If next_time is 30 minutes out, recovery will be slow even after the destination returns. Note the default change above: retry fields require include_retry true in the monitor_agent config on current versions.

TLS certificate expiry. Certificates expire at an exact moment and fail every output connection at once, as a cliff. Fluentd does not expose expiry through the API; check externally:

# Certificate expiry (paths from your config)
openssl x509 -enddate -noout -in /path/to/cert.pem

Alert at 30 days (plan), 7 days (act), 24 hours (page).

Confirmed-loss counters. drop_oldest_chunk_count incrementing means data was permanently discarded; write_secondary_count non-zero means the primary output failed past its retry limits and chunks went to the fallback. Both are ticket-worthy on any increment; sustained drops with a near-full buffer and a stalled output are a page.

Level 4: Expert

Goal: you catch silent failures, runtime-level contention, and configuration drift that standard metrics never surface. These signals typically get added after painful incidents.

Ruby GC behavior. GC pauses block all event processing. GC statistics are not cleanly exposed externally; infer pressure from RSS churn, context switches, and periodic throughput dips correlating with GC cycles. Major GC storms create a feedback loop: pauses cause flush timeouts, which cause retries, which allocate more objects. Tuning is environment-dependent; the official performance guide suggests RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR=1.2 for memory-constrained environments (default 2.0).

Per-thread CPU for GVL contention. One thread pegged while others idle confirms parsing or serialization is starving flush threads:

# Per-thread CPU
ps -T -p $(pgrep -f fluentd | head -1) -o spid,%cpu,comm

The fix is multi-worker mode or simpler parsers, not more flush threads: CPU-bound work does not parallelize across Ruby threads.

Kernel UDP drops. For UDP inputs (syslog, forward over UDP), the kernel drops packets before Fluentd ever sees them when socket buffers overflow. Check the drops column in /proc/net/udp. This is upstream data loss invisible to every Fluentd metric.

Inotify watch exhaustion. in_tail relies on inotify on Linux. If watches run out, it falls back to polling, which is far less responsive:

# Inotify limit
cat /proc/sys/fs/inotify/max_user_watches

Per-worker decomposition. Workers have independent buffers, queues, and event loops. Aggregate metrics mask a struggling worker. Query each worker’s monitor_agent port (24220 + worker_id) separately. Remember that in_tail does not support multi-worker and must be pinned with <worker N>, so only that worker’s port shows tail metrics.

Configuration integrity. A tampered config can silently redirect logs or disable collection. Track modification times and hashes against your deployment pipeline:

# Config integrity
stat -c '%Y' /etc/td-agent/td-agent.conf
sha256sum /etc/td-agent/td-agent.conf

Complement this with network-level checks: outbound connections from the Fluentd process to unexpected destinations (ss -tnp | grep fluentd), and inbound connections to forward ports from outside the sender allowlist.

Assessing yourself: quick checklist

• Liveness with sustained-failure gating, not a raw PID check. Worker count verified against configuration.
• Data-loss counters (emit_error_count, drop_oldest_chunk_count) alerting on any increment, not on thresholds.
• Input/output rate balance computed from deltas, with input metrics confirmed non-zero (the enable_input_metrics trap on older versions).
• Stage and queue tracked separately. One combined buffer number is a blind spot.
• Retry state, not just retry count. next_time in the future is a stalled pipeline.
• Host-level signals covered: RSS trend, FD vs ulimit, buffer partition free space. None of these come from the API.
• Version assumptions verified. monitor_agent defaults and available fields changed across v1.14.x, v1.19.0, and later releases; confirm what your version actually returns.

How Netdata helps

Fluentd’s signals split across two planes: the monitor_agent API (buffer, retry, flush, and emit metrics) and the host (RSS, FD count, disk, per-thread CPU, kernel UDP drops, inotify). Correlating across that split is where most manual setups fall apart. Specifically:

• Netdata collects Fluentd plugin metrics from the monitor_agent endpoint and keeps them as per-second time series, so rate derivation (emit_records, flush_time_count, retry_count) and spike detection between infrequent samples are handled for you.
• Buffer gauges (buffer_queue_length, buffer_total_queued_size, buffer_available_buffer_space_ratios) charted next to process RSS and FD usage make the backpressure-cascade pattern visible in one view instead of three terminals.
• Per-worker decomposition maps naturally to per-instance and per-dimension views, so a single struggling worker is not averaged away.
• Anomaly detection on input/output rate balance catches slow divergence, the 5% sustained deficit that accumulates into millions of lost events, without a hand-tuned threshold.
• Host-level signals the API never exposes (per-thread CPU, disk usage of the buffer partition, /proc/net/udp drops) are collected by the same agent, closing the Level 4 gap.

Related guides

• Fluentd monitoring checklist: the signals every production log pipeline needs
• How Fluentd actually works in production: a mental model for operators
• Fluentd process not running: the log pipeline is dead and the host has gone dark
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• Fluentd poison pill crash loop: one bad log line that kills the process on every restart
• Fluentd monitor_agent not responding: a process that is up but hung
• Fluentd plugin load error at startup: LoadError and missing gems
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer stage vs queue: telling healthy batching from backpressure
• Fluentd buffer available space low: computing time-to-overflow before it fires