Fluentd output authentication errors: 401/403 and rejected credentials

A 401, 403, unauthorized, forbidden, or TLS certificate error in the Fluentd log means the destination is actively rejecting the output plugin’s connection. The pipeline does not crash. Fluentd keeps accepting input, buffering events, and attempting flushes that keep failing, so from the outside the agent looks alive while no data reaches that destination.

The operational risk is the backlog that builds behind the failure. Every rejected flush pushes chunks back into the buffer, and if credentials stay broken long enough the buffer fills and the configured overflow_action decides what data you lose. With the default throw_exception, new events are silently discarded. With drop_oldest_chunk, the oldest buffered chunks are destroyed. Neither emits an obvious alert unless you are watching the right counters.

The most common trigger is not a misconfiguration you wrote. It is credential lifecycle: an API key rotated by a secrets manager, an IAM role whose permissions changed, a service account token that expired, or a TLS certificate that rolled past its validity date. Fluentd output plugins generally hold static credentials loaded at startup, so a destination that rotated credentials underneath a running agent produces exactly this symptom with no config change on your side.

What this means

An auth failure sits in the “destination rejecting data” family of the backpressure cascade. The signature is consistent across plugins:

• The Fluentd log shows a one-time or repeating auth error (401, 403, unauthorized, forbidden, or a certificate validation failure). The exact message format varies by output plugin, and some plugins log the real auth error only once, then mask it behind generic retry messages.
• retry_count for that output plugin climbs steadily.
• write_count for that output plugin stops incrementing.
• buffer_queue_length and buffer_total_queued_size grow as new input keeps arriving.
• Input-side emit_records stays normal, because inputs do not know anything is wrong.

flowchart TD
  A[Destination rejects credentials: 401/403/cert error] --> B[Flush fails, chunk rolls back]
  B --> C[retry_count climbs, write_count flat]
  C --> D[Buffer queue grows as input continues]
  D --> E{Buffer reaches total_limit_size}
  E -->|throw_exception - default| F[New events silently dropped]
  E -->|drop_oldest_chunk| G[Oldest chunks discarded]
  E -->|block| H[Inputs stall, upstream pressure]

Two nuances matter for diagnosis. First, a single auth error line during a deploy or certificate rollover is normal and self-heals; do not page on it. The actionable condition is the correlation: auth errors plus rising retry_count plus flat write_count. Second, some plugins treat auth failures as unrecoverable rather than retryable, which changes what you see: instead of a growing retry loop, chunks get moved aside immediately. Both paths are covered below.

Common causes

Quick checks

All read-only. Paths shown are for the td-agent package; for fluent-package use /var/log/fluent/fluentd.log and /etc/fluent/fluentd.conf.

# 1. Find the auth error in Fluentd's own log
grep -iE "(tls|ssl|auth|401|403|unauthorized|forbidden|certificate)" \
  /var/log/td-agent/td-agent.log | tail -20

# 2. Confirm the retry/stall correlation per output plugin
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") |
      {id: .plugin_id, retries: .retry_count, writes: .write_count,
       queue: .buffer_queue_length, rollbacks: .rollback_count}'

# 3. Check current retry state (how long until the next attempt)
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") |
      {id: .plugin_id, retry: .retry}'

# 4. Verify the destination is reachable and what it says about the credentials
#    (adjust host, port, and auth to your output plugin config)
curl -sv -u "$USER:$PASS" https://your-destination:9200/ 2>&1 | tail -20

# 5. Check TLS certificate expiry if the output uses TLS
grep -E "tls_cert|ca_cert|ssl_cert|cert_path" /etc/td-agent/td-agent.conf
openssl x509 -enddate -noout -in /path/to/cert.pem

# 6. If credentials come from environment variables (Kubernetes secrets),
#    check for trailing whitespace
printenv FLUENT_ELASTICSEARCH_PASSWORD | od -c | tail -3

In multi-worker mode, repeat the monitor_agent queries against each worker’s port (24220, 24221, and so on). Auth failures can be worker-specific if only some workers load the affected plugin.

How to diagnose it

1. Identify which output is failing. The log grep tells you the plugin and usually the destination host. Match it to the <match> block in the config. If multiple outputs share a worker, note that one misbehaving output can starve the others of flush threads.

2. Confirm the failure is auth, not reachability. Auth failures return fast with a specific status or handshake error. Reachability failures look like connection refused, timeouts, or broken pipe (see the broken pipe / connection reset guide). The distinction matters: for auth failures, no amount of retry tuning will help until the credential is fixed.

3. Check whether the plugin retries or gives up. Query retry_count twice, a minute apart. Rising count means the plugin treats the failure as retryable and you are in a backoff loop. Flat retry_count with a drained queue and no delivery means the plugin classified the error as unrecoverable and parked or diverted the chunks. For out_http on Fluentd v1.7+, the default is error_response_as_unrecoverable true with retryable_response_codes [503] (both parameters were introduced in v1.7.0), so a 401 or 403 moves the chunk aside immediately with no retry. Check write_secondary_count and any backup directory for where those chunks went.

4. Check retry depth. If it is retrying, look at the retry object: retry.steps tells you how many consecutive failures have occurred, and retry.next_time tells you when the next attempt fires. Under exponential backoff, a long-broken credential can push next_time tens of minutes out, so even after you fix the credential, recovery is slow until a flush succeeds or the agent restarts.

5. Validate the credential independently of Fluentd. Replay the destination request with curl (check 4 above) using exactly the credential material Fluentd has: same header format, same source IP (IAM policies and IP allowlists are source-dependent), same CA bundle. If curl succeeds from the same host with the same material, the problem is inside the plugin config: stale value loaded at startup, a newline in an injected env var, or a cached connection.

6. Find what changed. Auth failures that start suddenly on an untouched agent are almost always a destination-side change: key rotation, IAM policy update, cert rollover, service account deletion. Check the destination’s audit log and your secrets manager’s rotation history against the timestamp of the first error in the Fluentd log.

7. Assess the backlog. While credentials are broken, watch buffer_available_buffer_space_ratios and estimate time-to-overflow: available space / buffer growth rate. That number is your deadline for fixing the credential before overflow_action starts costing you data.

Metrics and signals to monitor

One important alerting note: retry_count is a cumulative counter of all flush failures since process start. It does not reliably return to zero after a successful retry, and it resets only on restart. Do not alert on retry_count > 0 in Prometheus-style tooling; alert on its rate of increase, or on retry.steps for currently active retry state.

Fixes

Rotated or expired credentials

Update the credential in the Fluentd config (or the secret/env var it reads) and reload. Fluentd output plugins load credentials at startup; there is no dynamic credential refresh, so a running agent never picks up a rotated key on its own. Send SIGHUP or restart the service after the update. A SIGHUP reload can partially apply if the new config has errors, so validate first with a dry run (fluentd --dry-run -c /etc/td-agent/td-agent.conf) and confirm the plugin list after reload (see the config reload failed guide).

Tradeoff: a full restart resets retry state and backoff timers, which speeds recovery after the fix but replays file-buffer chunks and briefly duplicates delivery. A SIGHUP preserves buffer state but keeps existing backoff timers until the next scheduled attempt.

IAM role or service account problems

Fix the policy or binding on the destination side, not in Fluentd. Then verify from the Fluentd host’s network identity. If nodes have different instance profiles, expect per-node divergence and check each affected host. No Fluentd restart is needed if the plugin retries; if backoff has grown large, a restart shortens recovery.

TLS certificate issues

Replace the expired certificate or fix the CA chain referenced in the output config, then reload. For destinations behind certificates you do not control, the fix is on the destination; your action is confirming the new chain is trusted by the CA bundle the plugin uses. Because cert expiry fails every connection simultaneously at an exact moment, monitor expiry ahead of time rather than reacting to the outage.

Secret injection artifacts (Kubernetes)

If the credential comes from a mounted secret or env var and fails despite looking correct, check for trailing newlines and encoding issues in the decoded value. Fix the secret at the source; patching it inside the running container does not survive restarts.

Plugin treats auth errors as unrecoverable

For out_http on v1.7+, if you want 401/403 to be retried (for example, because the destination rotates tokens and a brief rejection window is expected), set error_response_as_unrecoverable false and add 401, 403 to retryable_response_codes. Tradeoff: while credentials are genuinely broken, chunks now churn through retry instead of moving aside, so buffer pressure grows. Pair this with a <secondary> output so exhausted chunks go somewhere recoverable instead of being discarded.

If chunks were already diverted during the incident, recover them from the secondary destination or the plugin’s backup directory after credentials are fixed. Without a configured secondary, exhausted or unrecoverable chunks are discarded with only a log line.

Stale connections caching bad credentials

Some output plugins keep long-lived connections and do not re-authenticate after an auth error, so fixing the config has no effect until the connection is rebuilt. The Elasticsearch output plugin, for example, does not reconnect on 401/403 by default; set reconnect_on_error true to force a fresh connection on auth failures (the parameter has been documented since fluent-plugin-elasticsearch v1.9.0: without it, the plugin reconnects only on host-unreachable exceptions, not on 401/403). Where no such option exists, restart the agent after updating credentials.

Worker crash loops from bad credentials

Some plugins fail hard at configure or first-flush time on invalid credentials rather than retrying. The S3 output plugin has a known issue where invalid credentials raise an exception that crashes the worker, and the supervisor restarts it into the same failure (fluent-plugin-s3 issue #417, closed as expected behavior rather than fixed). If you see rapid restart cycling correlated with credential errors, treat it as a CrashLoopBackOff scenario and fix the credential first; the crash is a symptom.

Prevention

• Track credential expiry as a metric. TLS certs, API keys with TTLs, and token lifetimes should have alerts at 30/7/1 days out. Auth outages from expiry are fully predictable.
• Gate auth-error alerts on correlation. Alert on auth errors in the log only when retry_count is rising and write_count is flat. A one-off during a deploy or cert rollover is noise; the combination is the incident.
• Alert on retry rate, not retry_count value. The counter is cumulative and only resets on restart. Use delta(retry_count) or retry.steps.
• Configure a <secondary> output for any destination whose credentials can break, so exhausted chunks land somewhere recoverable.
• Decide overflow_action deliberately. The default throw_exception silently drops new events when the buffer fills during a long auth outage. Choose block or drop_oldest_chunk consciously, and monitor drop_oldest_chunk_count (see the drop_oldest_chunk guide).
• Integrate credential rotation with Fluentd reloads. If your secrets manager rotates keys on a schedule, the rotation pipeline must also push the new value to Fluentd’s config and trigger a validated reload. Static credentials plus automated rotation guarantees this incident repeats.
• Watch the buffer during incidents. buffer_available_buffer_space_ratios plus the growth rate gives you a time-to-overflow deadline; see the buffer available space guide for the calculation.

How Netdata helps

• Per-output-plugin retry_count and write_count side by side, so the “retrying but never delivering” signature of an auth failure is visible without log spelunking.
• buffer_queue_length and buffer_available_buffer_space_ratios trends, so you can see the backlog building while credentials are broken and estimate time-to-overflow.
• Retry state correlation: distinguishing an active, deepening retry loop from a recovered one, rather than alerting on a cumulative counter that never resets.
• write_secondary_count and drop_oldest_chunk_count visibility, so diverted or discarded chunks during an auth outage are detected instead of discovered later.
• Multi-worker breakdown, which catches cases where only some workers loaded the output with the bad credential.

Related guides

• Fluentd broken pipe / connection reset: dropped output connections and LB timeouts
• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd file buffer filling the disk: when the buffer partition runs out
• Fluentd buffer_oldest_timekey lag: how far behind the oldest buffered data is
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd CPU bottleneck: the Ruby GVL caps a single worker at one core
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss
• Fluentd duplicate events: why the same log shows up twice downstream
• Fluentd emit_error_count: the number-one under-monitored data-loss signal