Fluentd TLS certificate expiry: a cliff-edge that stops every output at once

Every TLS output on a Fluentd node was working. Then, at one exact second, all of them started failing with the same handshake error. retry_count is climbing on every output that uses TLS, write_count has flatlined, and the buffer is filling. Nothing was deployed. Nothing changed on the network. The only thing that changed is the wall clock: the certificate crossed its notAfter timestamp.

Certificate expiry is one of the few failures that is both perfectly predictable and a total outage. There is no gradual degradation, no partial failure, no per-connection luck. TLS connections to a destination succeed right up to the expiry instant and then every new handshake fails identically, for every output plugin, on every Fluentd node presenting or validating that certificate.

The trap is that Fluentd has no metric, log line, or API field that tells you a certificate is approaching expiry. The only in-band signals are the per-flush SSL errors after the fact and the buffer backpressure that follows. If you are not checking certificate validity externally, the first notification you get is the outage itself.

What this means

Fluentd loads TLS certificates from disk when the process starts or the configuration is reloaded. The certificate and key are read into memory and the OpenSSL context is built once. Fluentd does not watch the certificate files for changes and does not support automatic certificate reload. If a certificate management tool renews the files on disk, the running Fluentd process keeps using the old in-memory certificate until a restart or reload forces it to re-read the files.

This applies in two directions, and both bite:

• Output side. Outputs with <transport tls> (Elasticsearch, HTTP, and others) or out_forward with its tls_* parameters present a client certificate and/or validate the destination’s server certificate. When the relevant certificate expires, every flush to that destination fails at the handshake.
• Forward protocol on aggregators. An aggregator running in_forward with TLS terminates connections from many senders. When the aggregator’s server certificate expires, every sender’s out_forward starts failing at once. One expired certificate on one aggregator can stall an entire log tier.

After expiry, Fluentd’s normal retry machinery kicks in, which makes the failure look like an ordinary destination outage: retry_count increments, exponential backoff stretches out, buffer_queue_length grows. The distinguishing evidence is in the log: OpenSSL handshake errors on every flush attempt, appearing on all TLS outputs simultaneously.

flowchart TD
  A["Certificate notAfter passes"] --> B["Every new TLS handshake fails"]
  B --> C["Flush attempts raise SSL errors"]
  C --> D["retry_count climbs, write_count flatlines"]
  D --> E["buffer_queue_length grows"]
  E --> F["Buffer approaches total_limit_size"]
  F --> G["overflow_action fires: drops, block, or exception"]

The end state is the standard backpressure cascade, but the trigger is a timestamp, not a capacity problem. Adding flush threads or buffer space will not fix it. Only a valid certificate will.

Common causes

The last row is the common trap. If your automation renewed the certificate but nothing restarted Fluentd, the fix is a reload, not another renewal.

Quick checks

All of these are read-only and safe to run during an incident.

# 1. Find the SSL errors in Fluentd's log (adjust path for your package)
grep -iE "(ssl|tls|certificate)" /var/log/td-agent/td-agent.log | tail -20

Look for lines resembling SSL_connect returned=1 errno=0 state=error: certificate verify failed. The exact text varies with the Ruby OpenSSL version and the plugin, but “certificate verify failed” near the failure window is the fingerprint. A one-off error during a rollover is noise; a continuous stream correlated with rising retry_count is the incident.

# 2. Find which certificate files the config references
grep -iE "(tls_|ssl_|cert|ca_file|ca_cert|client_key)" /etc/td-agent/td-agent.conf

Different plugins name these parameters differently (tls_cert_path and ca_cert_path on out_forward, ca_file and client_cert on Elasticsearch-style outputs), so grep broadly.

# 3. Check expiry of the local (client or server) certificate file
openssl x509 -enddate -noout -in /path/to/cert.pem

# 4. Check the certificate the destination is actually serving
echo | openssl s_client -connect destination-host:9200 \
  -servername destination-host 2>/dev/null | \
  openssl x509 -noout -enddate

The -servername flag sends SNI; without it, multi-tenant destinations may present a default certificate that is not the one Fluentd validates. For the forward protocol on an aggregator, point s_client at the aggregator’s forward port (default 24224) from a sender host. Comparing check 3 (what Fluentd has on disk) against check 4 (what the peer is serving) tells you which side of the handshake is expired.

# 5. Confirm the retry/backpressure signature per output
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, retries: .retry_count, writes: .write_count, queue: .buffer_queue_length}'

The TLS-expiry signature: retry_count incrementing, write_count flat, buffer_queue_length growing, on every TLS-using output at once, while non-TLS outputs (if any) keep flushing normally.

# 6. Check whether Fluentd started before or after the cert file changed
ps -o lstart= -p $(pgrep -f fluentd | head -1)
stat -c '%y %n' /path/to/cert.pem

If the cert file is newer than the process, Fluentd is running the old certificate from memory.

How to diagnose it

1. Establish the shape of the failure. From the monitor agent, confirm that multiple outputs entered retry simultaneously and write_count stopped on all of them at the same time. A single output failing points at that destination; all TLS outputs failing at once points at a certificate or CA issue.
2. Pull the actual error from the log. Do not diagnose from metrics alone. Get the OpenSSL error string (check 1). “certificate verify failed” plus simultaneous onset is expiry or chain; “broken pipe” or “connection reset” is a different failure entirely (see the related guide on connection resets).
3. Identify which certificate is expired. Run checks 3 and 4 for every TLS hop: local client cert files, the destination’s served cert, and the aggregator’s forward listener if you forward through one. In a forward chain, the expired cert is often on the aggregator, not on the node where you are looking.
4. Check for the renewed-but-not-loaded case. If the cert file on disk is valid and new, but errors persist, compare file mtime against process start time (check 6). Fluentd does not watch cert files; it must be reloaded to pick up new material.
5. Estimate your runway while you fix it. From the monitor agent, watch buffer_available_buffer_space_ratios and the growth rate of buffer_total_queued_size. Time to overflow is roughly available_space / growth_rate. This tells you whether you have minutes or hours, and whether you need to think about overflow_action consequences before the cert is fixed.
6. Rule out lookalikes. A destination that is down produces retries without SSL errors. A hostname mismatch or missing CA produces certificate verify failed but starts at a config change or deploy, not at a timestamp. Anchor the onset time: if failures began at one exact second with no deploy, expiry is the likely cause, and the notAfter value will confirm it.

Metrics and signals to monitor

Treat the first row as mandatory. Every other row tells you the expiry already happened.

Fixes

Expired certificate: renew and reload

Issue or install the renewed certificate and key at the paths the Fluentd config references, then force Fluentd to re-read them. Sending SIGHUP to the Fluentd supervisor gracefully restarts the worker process, which re-reads configuration and certificate files from disk; the supervisor keeps listening sockets open (for sockets provided as shared sockets by server_helper), so inbound connections are not dropped during the restart. Verify after the reload that write_count resumes incrementing and the buffer starts draining.

Expect a burst of flushes after recovery as the buffer drains, and expect possible duplicates from chunk retries. Fluentd’s delivery is at-least-once; chunks that were mid-retry when the cert expired will be re-sent.

Renewed on disk but still failing

If the cert files are valid and new but errors continue, the running process is still holding the old certificate in memory. Reload or restart. Do not troubleshoot the destination; it has been seeing the old cert the whole time.

Aggregator forward certificate expired

Renew the certificate on the aggregator and reload it there. Every sender’s out_forward will recover on its own retry schedule. If senders have been retrying for a long time, their exponential backoff may push retry.next_time far into the future; check the retry object in the monitor agent API, and be prepared for a slow, staggered drain rather than an instant recovery. Watch the aggregator for a retry-storm spike as many senders reconnect at once.

Buying time when renewal is blocked

If you cannot renew immediately, your only lever is buffer capacity: the buffer is absorbing everything until the cert is fixed. Know your overflow_action before the buffer fills, because the default throw_exception drops new events, block pushes backpressure onto inputs, and drop_oldest_chunk discards the oldest buffered data. None of these are good; they are choices about which data you lose. A <secondary> output to a non-TLS or differently-authenticated destination can act as a safety net if one is configured. Do not disable TLS verification as a workaround; tls_insecure_mode-style options turn an availability incident into a silent integrity exposure.

Prevention

• Monitor expiry externally, on a schedule. Certificate expiry is not a Fluentd metric. Run openssl x509 -enddate -noout -in <cert> (and openssl s_client ... | openssl x509 -noout -enddate for served certs) from a cron job, host agent, or your monitoring system against every cert in the log path: output client certs, destination server certs, and aggregator forward listeners.
• Alert in tiers. 30 days out: planning-level alert so renewal can be scheduled. 7 days: ticket, someone owns it. 24 hours: page. An expired certificate is a fully avoidable page.
• Automate the reload, not just the renewal. Renewal without reload is the classic half-fix, because Fluentd caches certificates in memory. Your certificate automation must end with a Fluentd reload or restart, and ideally with a verification step that compares the cert on disk against the cert the process is serving or presenting.
• Inventory every TLS hop. Expiry on an aggregator’s forward listener stalls every sender. Keep a list of which nodes terminate TLS and which present client certificates, so the external expiry check covers all of them.
• Know your overflow behavior before you need it. During a cert outage the buffer is the only thing between you and data loss. Size it for realistic renewal-plus-notice time, and choose overflow_action deliberately rather than inheriting the default.

How Netdata helps

• Netdata’s Fluentd monitoring surfaces the per-output counters that show the expiry signature in one view: retry_count climbing, write_count flat, and buffer_queue_length rising across all TLS outputs simultaneously, which is what separates a cert cliff-edge from a single slow destination.
• Buffer gauges such as buffer_total_queued_size and buffer_available_buffer_space_ratios give you a live time-to-overflow estimate while you work on the certificate, so you know whether the overflow_action decision is urgent.
• buffer_oldest_timekey tracks how stale the oldest undelivered data is getting during the outage, which maps directly to your delivery SLA and post-incident gap analysis.
• Netdata’s x509 certificate expiry check can run on the same nodes and be alerted on with tiered thresholds (30 days, 7 days, 24 hours), closing the gap that Fluentd’s own instrumentation leaves open.
• Correlating log-based SSL error patterns with the retry and buffer metrics on one timeline shortens the “is it the destination or is it us?” loop to a single glance.

Related guides

• Fluentd broken pipe / connection reset: dropped output connections and LB timeouts
• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd file buffer filling the disk: when the buffer partition runs out
• Fluentd buffer_oldest_timekey lag: how far behind the oldest buffered data is
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd CPU bottleneck: the Ruby GVL caps a single worker at one core
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss
• Fluentd duplicate events: why the same log shows up twice downstream
• Fluentd emit_error_count: the number-one under-monitored data-loss signal