Fluentd plugin load error at startup: LoadError and missing gems

Fluentd fails to start, or starts with part of the pipeline missing, and the log shows LoadError, cannot load such file, or a load_plugin failure. If the failed plugin is an output, that destination silently receives no data while everything else looks healthy. If the plugin is critical to the config, the process refuses to start entirely and you find out via your process-alive check, or via a gap in downstream logs.

This failure almost always traces back to the Ruby gem layer: a plugin gem that is missing, installed into the wrong Ruby, built against the wrong Ruby version, or pinned to a version incompatible with the rest of the dependency tree. The classic trigger is an upgrade, of Fluentd itself, of the package (td-agent to fluent-package), or of the embedded Ruby, after which previously working plugins no longer load.

What this means

Fluentd plugins are Ruby gems. At startup, Fluentd reads the config, resolves each @type to a plugin, and requires the corresponding gem. That require can fail at several distinct layers, and the error text tells you which one:

1. The gem is not installed at all in the Ruby environment Fluentd actually uses.
2. The gem is installed, but for a different Ruby. The single most common cause: the operator ran gem install with the system Ruby instead of fluent-gem or td-agent-gem, so the gem landed in a gem path Fluentd never searches.
3. The gem is installed but a dependency fails to require. You get a LoadError naming some other file: a transitive dependency, a native extension, or a default gem that is not bundled.
4. A native (C) extension was compiled against a different Ruby. Ruby does not guarantee C extension compatibility across major versions. After the packaged Ruby is upgraded, every plugin with native extensions must be rebuilt.
5. The plugin version is incompatible with the Fluentd core version. Plugins written against the v0.12 API do not work with Fluentd v1, and certain dependency bumps (such as the console gem) have broken plugin loading in specific Fluentd releases.

The blast radius depends on where the plugin sits. A failed input means that source is not collected. A failed output means that destination silently gets nothing. A failed parser or filter can break an entire match block. If Fluentd cannot construct a plugin the config requires, it aborts startup, which is when your process-dead alert fires.

flowchart TD
  A[Startup log shows LoadError or load_plugin failure] --> B{Does the process stay up?}
  B -->|No| C[Critical plugin failed - process-alive alert path]
  B -->|Yes| D[Pipeline running degraded - one input or output dead]
  C --> E[Identify plugin and error layer in logs]
  D --> E
  E --> F{Gem installed in Fluentd's Ruby?}
  F -->|No| G[Reinstall with fluent-gem or td-agent-gem]
  F -->|Yes| H{Dependency or native extension failing?}
  H -->|Dependency| I[Pin or install the missing dependency gem]
  H -->|Native ext| J[Rebuild against current Ruby - needs gcc, make, headers]
  H -->|Version conflict| K[Pin compatible plugin and dependency versions]

Common causes

Quick checks

All read-only. Paths shown are for td-agent; for fluent-package, substitute fluentd for td-agent in unit names, /var/log/fluent/fluentd.log for the log path, and /etc/fluent/ for the config path.

# 1. Find the actual load error in startup logs
journalctl -u td-agent --since "30 minutes ago" | \
  grep -iE "(load_plugin|LoadError|uncaught|cannot load)"

# 2. Confirm whether the process survived
systemctl status td-agent
ps aux | grep '[f]luentd'

# 3. List plugins visible to Fluentd's own Ruby (td-agent)
td-agent-gem list | grep fluent-plugin

# 3b. fluent-package equivalent
fluent-gem list | grep fluent-plugin

# 4. Compare with what the system Ruby sees (common mismatch)
gem list | grep fluent-plugin

# 5. Which plugins does the config actually reference?
grep -E "@type" /etc/td-agent/td-agent.conf | sort -u

# 6. Check what plugins the running process loaded (if monitor_agent is up)
curl -s http://localhost:24220/api/plugins.json | \
  jq -r '.plugins[] | "\(.plugin_category)\t\(.type)\t\(.plugin_id)"'

Check 4 catches the most common cause: if gem list shows the plugin but td-agent-gem list (or fluent-gem list) does not, you installed into the wrong Ruby. Check 6 tells you what actually loaded: if the process is up but a <match> plugin is absent from the list, that output is dead and its destination is receiving nothing.

How to diagnose it

1. Capture the exact error. Run the journalctl grep from the quick checks, or grep the log file directly: grep -iE "(load_plugin|LoadError|cannot load)" /var/log/td-agent/td-agent.log. Note three things: the plugin name, the file Ruby failed to require, and the error class. cannot load such file -- fluent/plugin/out_foo points at the plugin gem itself. cannot load such file -- some_dependency points at a transitive dependency inside an installed gem.

2. Determine the blast radius. Is the process running (systemctl status)? If it is up, the failure was non-fatal and one branch of the pipeline is dark. Cross-reference the failed plugin against your config: is it an input (source lost), a filter (records flow unfiltered or the match fails), or an output (destination starved)? If the process is down, this is the process-alive incident path; see Fluentd process not running.

3. Verify installation against the correct Ruby. Fluentd packages ship an embedded Ruby with its own gem path. For td-agent that Ruby lives under /opt/td-agent/embedded/ and gems are managed with td-agent-gem; for fluent-package, use fluent-gem. Run the package’s gem wrapper and confirm the plugin and its version are present. If you historically installed with bare gem, assume the gem went to the system Ruby and reinstall with the wrapper.

4. If the gem is present, check the dependency layer. A LoadError on a file that is not the plugin itself means an installed gem has a broken or missing dependency. Two documented cases: console gem v1.25 and later caused a LoadError that broke plugins such as fluent-plugin-prometheus, fixed in Fluentd v1.16.6 and v1.17.1; and Elasticsearch 8 removed an internal transport module, breaking fluent-plugin-elasticsearch unless the elasticsearch gem is pinned to v7.17.0 with fluent-plugin-elasticsearch v5.0.3. Check your Fluentd version against its release notes before pinning blindly.

5. If the error names a native extension, suspect a Ruby upgrade. C extensions are compiled against a specific Ruby ABI. When the package upgrades its embedded Ruby, previously compiled extensions fail to load. Reinstalling the plugin with the package’s gem wrapper forces a rebuild. If the build itself fails with Failed to build gem native extension, the host is missing gcc, make, or the Ruby development headers.

6. Check file permissions on the plugin files. A gem installed as root with a restrictive umask can leave plugin files unreadable by the Fluentd service user, producing Permission denied @ rb_sysopen. Verify the service user can read the plugin gem directory and files (755 directories, 644 files are the norm for gem installations).

7. Confirm the fix end to end. After reinstalling or pinning, restart the service and re-run the startup grep plus the monitor_agent plugin listing. Then verify data flow, not just startup: an output that was starved during the outage will drain its buffer backlog, which looks like a flush spike. That is normal replay behavior.

Metrics and signals to monitor

The subtle case is the non-fatal one: process up, no page, one destination dark. The only reliable detectors are comparing the loaded-plugin inventory against the config, and watching per-plugin throughput for a branch that went to zero. Once a starved output’s buffer hits its limit, the outcome depends on overflow_action and Fluentd version, and none of the outcomes is a clean, obvious error counter. Detecting the load failure itself is the real defense.

Fixes

Reinstall the plugin into the correct Ruby

# td-agent
sudo td-agent-gem install fluent-plugin-<name>

# fluent-package
sudo fluent-gem install fluent-plugin-<name>

Never use bare gem install for Fluentd plugins on packaged installs; it targets the system Ruby and the gem will be invisible to Fluentd. After installing, restart the service and verify with the package gem wrapper’s list command and the monitor_agent inventory. A restart briefly interrupts ingestion, so plan for it on busy pipelines.

Reinstall plugins after a package upgrade

Treat plugin reinstallation as a mandatory step of any td-agent or fluent-package upgrade, not an afterthought. Upgrades between package families (td-agent v4 to fluent-package v5 or v6) are known to drop previously installed plugins, and upgrades that change the embedded Ruby invalidate native extensions. Before upgrading, inventory what you have:

# Snapshot installed plugins before upgrade
td-agent-gem list | grep fluent-plugin > /root/fluent-plugins-before.txt

After the upgrade, reinstall everything in the snapshot with the new package’s gem wrapper, then restart and verify. fluent-package v5.0.2 and later ships fluent-diagtool, which can help enumerate manually installed plugins.

Pin compatible dependency versions

When the LoadError names a dependency rather than the plugin, pin the working versions instead of taking the latest:

• For the console gem breakage, upgrade Fluentd to a release that contains the fix.
• For Elasticsearch 8 destinations, pin the elasticsearch gem to v7.17.0 and fluent-plugin-elasticsearch to v5.0.3.

The maintainable way to pin versions in production is a Bundler Gemfile, so every install and upgrade resolves the same locked dependency set.

Fix native extension build failures

If installation fails at build time, install the toolchain first, then retry the gem install with the package wrapper. On Debian-family systems that means gcc, make, and the Ruby development headers; on minimal container images you may need a full build base. In Docker images, install gems as root and switch back to the fluent user afterward.

Fix unreadable plugin files

If the error is Permission denied @ rb_sysopen, correct permissions on the affected gem files so the Fluentd service user can read them, then restart. Check for this after any manual gem install done as root with a restrictive umask.

Missing default gems on minimal images

On Alpine-based images, the minimal Ruby package may not include default gems such as json, producing cannot load such file -- json (LoadError). Install the missing gem explicitly in the image build.

Prevention

• Always use the package gem wrapper. Standardize on fluent-gem or td-agent-gem in runbooks, configuration management, and image builds. The wrong-Ruby install is the most common cause and the easiest to prevent.
• Manage plugin versions as code. Use a locked Gemfile so plugin and dependency versions are pinned, reviewable, and reproducible across hosts and rebuilds.
• Make plugin reinstallation part of the upgrade runbook. Snapshot gem list before, reinstall after, and verify the loaded-plugin inventory against the config before declaring the upgrade done.
• Test startup in staging with the production config. A config that references a plugin missing from the image fails exactly the same way in staging as in production. Boot the service, grep for load errors, and diff the monitor_agent plugin list.
• Monitor startup logs, not just process state. A process that starts degraded (one plugin failed) passes a process-alive check. Alert on any LoadError or load_plugin match in startup logs.
• Keep Fluentd current. Dependency-related LoadErrors (console gem, json parser) were fixed in specific releases; running an affected version means re-discovering known bugs. fluent-package v5 LTS and td-agent v4 are past or near end of life, so plan migrations to fluent-package v6 LTS.

How Netdata helps

• Netdata’s process and systemd unit monitoring catches the fatal case immediately: the Fluentd process missing or crash-looping after a plugin failure aborts startup, correlated with restart counts so a flapping service is visible instead of masked by auto-restart.
• Per-second host metrics let you correlate the plugin failure with its trigger: the restart, deploy, or package upgrade that preceded the LoadError shows up on the same timeline as CPU, memory, and disk activity.
• Netdata can tail and alert on Fluentd’s own log file, so a LoadError or cannot load pattern at startup fires as an alert rather than sitting unread in journald.
• Combined with Fluentd’s monitor_agent endpoint, Netdata charts per-plugin buffer queue length, retry count, and emit rates, which surfaces the degraded case: process alive but one output’s throughput flatlined after a restart.
• Long retention on these signals makes post-upgrade verification practical: you can compare per-plugin throughput before and after the upgrade window and confirm every branch recovered.

Related guides

• How Fluentd actually works in production: a mental model for operators
• Fluentd monitoring checklist: the signals every production log pipeline needs
• Fluentd monitoring maturity model: from survival to expert
• Fluentd process not running: the log pipeline is dead and the host has gone dark