Fluentd monitor_agent not responding: a process that is up but hung

Your liveness check says Fluentd is fine. The PID exists, systemctl status shows active, but curl http://localhost:24220/api/plugins.json hangs until it times out, or returns something other than 200, and no logs are moving.

This is the zombie state: a process that passes every cheap liveness check but is functionally dead. The monitor_agent endpoint is the cheapest honest probe you have for this. If it does not answer, the internal runtime is not making progress, regardless of what the process table says.

This guide covers how to confirm the hang, tell the underlying causes apart, and make sure monitoring catches it next time instead of relying on a bare PID check.

What this means

Fluentd’s monitor_agent plugin (in_monitor_agent) runs an HTTP server inside the Fluentd process. When you add this to the config, each worker exposes its own endpoint:

<source>
  @type monitor_agent
  bind 0.0.0.0
  port 24220
</source>

Default port is 24220. In multi-worker mode, each worker gets its own port: worker 0 answers on 24220, worker 1 on 24221, and so on (24220 + worker_id). A healthy response from /api/plugins.json means two things at once: the process exists, and the Ruby runtime can still schedule threads and execute code. That second condition is exactly what a PID check cannot tell you.

When the endpoint stops answering while the process still exists, the usual suspects are:

• The event loop is blocked on a synchronous operation, typically a network call to an output destination without a timeout.
• The GVL (Global VM Lock) is held by one thread doing CPU-bound work, so no other thread, including the monitor_agent HTTP handler, ever runs. CRuby threads share one GVL per worker process, and CPU-bound work like regex parsing or serialization does not release it.
• A genuine deadlock between threads.
• A GC storm: Ruby garbage collection consuming all available CPU because the heap is fragmented or under pressure.

The diagram shows why a single stuck thread can take down the whole probe surface.

flowchart LR
  T[tail input thread] --> GVL[GVL per worker]
  P[parser thread - stuck regex] --> GVL
  F[flush threads] --> GVL
  M[monitor_agent HTTP] --> GVL
  GVL -->|held by stuck thread| S[all other threads starve]
  S --> R[/api/plugins.json times out/]

Common causes

Do not confuse the hang with a different symptom. “Connection refused” on the first-ever attempt usually means monitor_agent is not in the config, or (in containers) the port is not published. That is a configuration gap, not a hang. A port that previously answered and now times out on a live process is the zombie state this guide is about.

Quick checks

All read-only. Run these before touching the process.

# 1. Confirm the process exists and note the PID
pgrep -af fluentd

# 2. Probe the endpoint with an explicit timeout (5s is the alerting threshold)
curl -s -o /dev/null -w "%{http_code}\n" --max-time 5 http://localhost:24220/api/plugins.json

# 3. Confirm nothing else stole the port
ss -tlnp | grep 24220

# 4. Check per-thread CPU to distinguish GVL starvation from deadlock
ps -T -p <pid> -o spid,%cpu,comm

# 5. Check RSS and thread count (GC storm candidate)
ps -o pid,rss,%mem,%cpu,comm -p <pid>
grep -E "VmRSS|Threads" /proc/<pid>/status

# 6. In multi-worker mode, probe every worker port
for p in 24220 24221 24222; do
  printf "%s: " "$p"
  curl -s -o /dev/null -w "%{http_code}\n" --max-time 5 "http://localhost:$p/api/plugins.json"
done

Interpretation shortcuts:

• Step 2 returns 000 (timeout) and step 4 shows one thread at ~100% CPU: GVL starvation, likely a parser or serialization loop.
• Step 2 returns 000 and the process is near 0% CPU: blocked on I/O or deadlock.
• Step 2 returns 000 with high CPU plus step 5 showing RSS near the container limit: GC storm under memory pressure.
• Step 3 shows a different process owning 24220: port conflict, not a hang.

How to diagnose it

1. Verify the zombie state, not just the symptom. Confirm the PID exists, the port is owned by Fluentd, and the endpoint times out. This separates “hung” from “never configured” and “port conflict” in one pass.

2. Classify by CPU pattern. ps -T -p <pid> -o spid,%cpu,comm splits the causes immediately. One hot thread points at CPU-bound work holding the GVL. A fully idle process points at blocked I/O or deadlock.

3. Capture thread backtraces. Standard Fluentd builds (via the serverengine and sigdump gems) dump all thread backtraces, GC statistics, and object counts to a file when the process receives SIGCONT. The dump lands in /tmp/sigdump-<pid>.log — not in the Fluentd log — unless SIGDUMP_PATH overrides the location:

   # SIGCONT is harmless on a running process, but confirm your build's
   # behavior before sending any signal to a production process.
   kill -CONT <pid>
   cat /tmp/sigdump-<pid>.log   # dump path: /tmp/sigdump-<pid>.log, or $SIGDUMP_PATH
   

   Historical hang reports show backtraces with threads stuck in HTTP client connect calls (synchronous I/O without timeouts) or stuck inside regex matching in the parser. Either way, the backtrace tells you which plugin owns the stuck thread. If SIGCONT produces nothing, an intrusive alternative is attaching gdb, but that pauses the process; treat it as a last resort on a node you have already drained.

4. Correlate with the last good metrics. If your monitoring scraped the API before the hang, look at the last samples: was buffer_queue_length climbing? Was RSS trending up for hours (GC pressure building)? Was single-core CPU creeping toward 100% over weeks (GVL saturation)? The trajectory before the hang usually names the cause.

5. Check the Fluentd log one more time. Even a hung process often emitted warnings before freezing: slow flush warnings, retry messages, allocation failures. grep -iE "error|warn|slow" /var/log/td-agent/td-agent.log | tail -50.

6. In multi-worker mode, check each worker independently. One hung worker out of four leaves 75% of capacity running and is easy to miss in aggregate metrics. Probe every worker port before concluding anything.

Metrics and signals to monitor

These are the signals that distinguish “process exists” from “process works.” The first one is the detection signal for this exact failure; the rest explain the cause.

On severity: a dead process is a page. An unresponsive monitor_agent on a live process is a ticket: investigate immediately, but you still have a process to interrogate, which is an opportunity, not just an alert.

Fixes

Restarting clears the symptom in every case and is sometimes the right call after you have captured evidence, but it destroys the diagnostic state and, with memory-backed buffers, destroys any unflushed data. Capture backtraces and last-known metrics first.

Blocked on synchronous output I/O

The pattern is an output plugin making a network call without a timeout and waiting forever on a dead-but-not-closed connection.

• Short term: restart the process after capturing backtraces. File-backed buffers replay on restart; memory-backed buffer contents are lost.
• Long term: set explicit open and read timeouts on HTTP-based output plugins. out_http exposes open_timeout, read_timeout, and ssl_timeout (the latter two since v1.7.0), and out_forward exposes connect_timeout (since v1.6.0). Historical reports of silent multi-hour hangs were resolved exactly this way: threads stuck in net/http connect calls because no open_timeout was set.
• If you are on an older Fluentd version, check release notes for your output plugin. There are known fixed bugs where a dropped connection after TLS handshake caused an infinite loop in connection establishment in out_forward (issue #4396); the fix, PR #5138 (backport of PR #5104), shipped in v1.19.2.

GVL starvation from CPU-bound work

One thread doing regex parsing, JSON serialization, or heavy filtering holds the GVL and starves everything, including the monitor HTTP handler.

• Simplify parsers. Replace complex regex with structured formats (JSON, LTSV) where the log source allows it. Catastrophic regex backtracking on malformed input can pin a thread indefinitely; there are reports of the event loop stuck inside the regex engine on pathological lines.
• Reduce expensive filters, especially record_transformer blocks doing Ruby work per event.
• Enable multi-worker mode (workers N in <system>) so CPU work spreads across processes, each with its own GVL. Remember that in_tail does not support multi-worker and must be pinned to a specific worker with <worker N>.

GC storm / memory pressure

• Switch buffers from memory-backed to file-backed (@type file) to move buffered data out of the Ruby heap.
• Increase chunk_limit_size. Larger chunks mean fewer Ruby objects and less GC overhead. Millions of small objects from tiny chunks are a classic GC-pressure source.
• Give Ruby headroom: keep container memory limits at least 20-30% above normal RSS, and watch the trend rather than the absolute value. A stable high plateau is normal Ruby fragmentation behavior; a rising line is not.

Deadlock

A true deadlock with zero CPU is the rarest case. Capture backtraces, restart, and file the evidence. If the backtrace implicates a specific third-party plugin, that plugin is the fix target.

Port conflict

If another process owns 24220, move monitor_agent to a free port in the config and update your probes. This is a config fix, not a Fluentd fix.

Prevention

• Alert on the API, not the PID. A bare PID check passes on a zombie. The detection rule is: any non-200 response or timeout beyond 5 seconds on the monitor_agent endpoint, sustained, when monitor_agent is configured. Pair it with the process-alive check so you can tell “dead” from “hung.”
• Probe every worker port. In multi-worker mode, per-worker monitoring on 24220 + worker_id is the only way to see a single hung worker.
• Set timeouts on every output that makes network calls. Unbounded blocking I/O is the most repeatable cause of this failure class.
• Track single-core CPU and RSS trends. Both GVL saturation and GC pressure announce themselves days before the hang if you graph them.
• Watch buffer_queue_length and write_count rates. A stalling output is the most common precursor to a blocked event loop.

How Netdata helps

• Netdata’s Fluentd collector polls the monitor_agent API itself, so a hung endpoint shows up as a stopped data stream for the instance: the failure surfaces in your dashboards instead of hiding behind a green process check.
• Per-plugin charts for buffer_queue_length, retry_count, and write_count show the output stall that typically precedes a blocked event loop, visible in the minutes before the API dies.
• Process-level CPU and RSS charts make the GVL-starvation pattern (one pinned core) and the GC-pressure pattern (monotonic RSS growth) distinguishable at a glance.
• Per-worker collection from each monitor_agent port keeps a single hung worker visible instead of averaging it away.
• Alerting on collector unreachability plus process existence gives you the exact “alive but hung” composite this article is about, instead of a binary up/down check.

Related guides

• How Fluentd actually works in production: a mental model for operators
• Fluentd monitoring checklist: the signals every production log pipeline needs
• Fluentd monitoring maturity model: from survival to expert
• Fluentd process not running: the log pipeline is dead and the host has gone dark