Logstash certificate expiry: the silent, total outage no built-in metric shows

Every TLS connection into or out of Logstash depends on a certificate with a hard expiry date. When that date passes, the failure is not gradual: every new TLS handshake is rejected, every client disconnects, and every output stalls at the exact second the certificate lapses. Filebeat agents stop shipping. Elasticsearch outputs throw handshake errors. Nothing is delivered.

The dangerous part is what Logstash does not tell you. The Node Stats API exposes JVM, pipeline, event, queue, and plugin metrics, but there is no certificate expiry date, no days-until-expiry gauge, and no TLS handshake failure counter anywhere in it. Your dashboards can be completely green at T-minus one hour and show a total ingestion outage at T-plus zero. Certificate expiry is a property of files on disk, so it needs an external check, not a Logstash metric.

This guide covers how the failure presents, how to inventory every certificate Logstash depends on, and how to build the expiry monitoring Logstash will never give you.

What this means

A Logstash deployment typically terminates TLS in at least three places, each with its own certificate and its own expiry date:

Beats input (server side): Logstash presents a server certificate to Filebeat, Metricbeat, and other Beats clients. When it expires, every client handshake fails and agents stop sending.
Outputs (client side): The Elasticsearch and Kafka outputs may present client certificates for mutual TLS, and always validate the destination’s server certificate against a CA bundle. An expired local client cert fails the handshake from Logstash’s side. An expired CA in the trust bundle makes every destination certificate untrusted.
Monitoring and management traffic: The monitoring pipeline that ships Logstash metrics to a monitoring cluster carries its own TLS settings. If that certificate expires, monitoring data silently stops flowing, so you lose visibility at the same time you lose the pipeline.

flowchart LR
  A[Certificate passes expiry date] --> B[Every new TLS handshake rejected]
  B --> C[Beats clients disconnect and retry forever]
  B --> D[Output connections fail to ES or Kafka]
  C --> E[events.in drops to zero]
  D --> F[queue fills, backpressure builds]
  E --> G[Total outage, no cert metric fired]
  F --> G

The per-connection error shows up only in the logs. A typical input-side failure looks like javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired, logged once per connection attempt. On the Elasticsearch output side, an expired or untrusted chain surfaces as a path validation error such as PKIX path validation failed ... validity check failed. With thousands of Beats agents reconnecting, these errors flood the log, which paradoxically makes the signal easy to miss in the noise.

One version caveat worth knowing: current versions handle the failure per-connection: the process stays up, the API on port 9600 keeps answering, and only the affected socket dies. That is precisely what makes the modern failure mode silent. Process liveness checks keep passing while nothing flows.

Common causes

Cause	What it looks like	First thing to check
Beats input server certificate expired	All Beats agents disconnected at the same moment; `certificate_expired` alerts in the Logstash log; `events.in` at zero	`openssl x509 -enddate -noout` on the input’s `ssl_certificate` file
Output client certificate expired	Elasticsearch or Kafka output blocked; handshake or path validation errors in the log; queue growing	`openssl x509 -enddate -noout` on the output’s client cert
CA bundle or intermediate expired	`unknown_ca` or path validation failures against a destination that renewed its cert recently; looks like the destination’s fault	Check every cert in the CA bundle file, not just the leaf
Destination’s certificate expired (not yours)	Output failures with `certificate_expired` received as a fatal alert from the far end	Verify the destination’s presented cert with `openssl s_client`
Partial expiry in a chain	Some clients connect, some fail, depending on which chain path their TLS library builds	Inspect the full chain, including cross-signed intermediates

A related trap: setting ssl_verify => false on an input means Logstash does not verify client certificates. It does not exempt Logstash’s own server certificate from expiry. If the server cert has lapsed, the handshake still fails regardless of that setting.

Quick checks

These are read-only and safe to run any time.

# 1. Check expiry of a specific certificate file
openssl x509 -in /etc/logstash/certs/logstash-server.pem -enddate -noout

# 2. Check whether a cert expires within N seconds (exit 1 = expiring soon or expired)
#    This example warns at 7 days (604800 seconds)
openssl x509 -checkend 604800 -noout -in /etc/logstash/certs/logstash-server.pem && echo "OK" || echo "EXPIRING OR EXPIRED"

# 3. Check every certificate in a CA bundle, not just the first
awk 'BEGIN {c=0} /BEGIN CERTIFICATE/ {c++} {print > "/tmp/bundle-" c ".pem"}' /etc/logstash/certs/ca-bundle.pem
for f in /tmp/bundle-*.pem; do echo "== $f"; openssl x509 -in "$f" -noout -subject -enddate; done
rm -f /tmp/bundle-*.pem

# 4. Check what a remote destination actually presents (chain and dates)
openssl s_client -connect elasticsearch.example.com:9200 -servername elasticsearch.example.com </dev/null 2>/dev/null | openssl x509 -noout -subject -issuer -enddate

# 5. Grep for TLS failures in the Logstash log
grep -Ei '(SSL|TLS|certificate|handshake)' /var/log/logstash/logstash-plain.log | tail -n 200

# 6. Confirm whether events are actually flowing right now
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty

Command 2 is the one to automate: openssl x509 -checkend <seconds> exits non-zero when the certificate expires within the window, which makes it a drop-in check for any scheduler or monitoring agent.

How to diagnose it

When ingestion has stopped and you suspect a certificate:

Confirm the blast radius. Query /_node/stats/pipelines and compare events.in and events.out per pipeline. All pipelines dead at once points at a shared dependency: the Beats input cert, a shared CA bundle, or the destination. One pipeline dead points at that pipeline’s own cert configuration.
Check the log for handshake errors. The grep in step 5 above. certificate_expired means the local or peer cert lapsed. unknown_ca means a trust chain problem: a CA or intermediate expired, or the destination renewed against a CA your bundle does not contain.
Date-check every certificate file in the config. Read the pipeline configs under /etc/logstash/conf.d/ and collect every ssl_certificate, ssl_key, ssl_certificate_authorities, cacert, and truststore reference. Run openssl x509 -enddate against each PEM file.
Date-check the peer. If your local certs are valid, verify what the destination presents with openssl s_client. Destinations renew too, and a lapsed cert on the Elasticsearch side produces the same outage from Logstash’s perspective.
Check the clients. Filebeat-side errors like remote error: tls: bad certificate confirm clients are rejecting the Logstash server cert.
Do not restart first. A restart does not extend a certificate. Diagnose the dates, renew what is expired, then reload or restart to pick up the new files. Pipeline stats reset on config reload, so expect artificial zero-dips in rate graphs afterward.

Signals to monitor

Signal	Why it matters	Warning sign
Days until cert expiry (external check per cert file)	The only early warning that exists; Logstash exposes nothing	Any cert under 30 days
TLS/handshake errors in the log	The only in-band symptom once failure begins	Any sustained `certificate_expired` or `unknown_ca` pattern
`events.in` per pipeline	Input-side total failure shows here first	Drop to zero while sources are known active
Output errors and retries	Output-side TLS failure appears as connection errors before throughput collapses	Sustained non-zero error pattern correlating with queue growth
PQ capacity (`queue.capacity.queue_size_in_bytes`)	After an output TLS failure, the persistent queue absorbs the gap and gives you runway	Sustained growth with output rate below input rate

Treat the log signal as ticket-level and let escalation happen through the composite backpressure and queue-runway signals, not from log lines alone.

Fixes

Renew the expired certificate. Issue the replacement from your CA or internal PKI, place the new cert and key at the configured paths, and reload or restart Logstash. Verify file ownership and permissions match what the Logstash service user can read; a renewed cert with wrong permissions fails as a different error and prolongs the incident.

Update the CA bundle when a CA or intermediate lapsed. Replace the bundle file and make sure every CA Logstash needs to trust is present and unexpired. The September 2021 DST Root CA X3 expiry is the canonical example of a root expiry cascading into Logstash failures across unrelated plugins (geoip downloader, S3 output and input, RSS) via its TLS library; Elastic’s fix (Logstash 7.15.1/7.16.0) replaced Faraday with Manticore to sidestep jruby-openssl’s chain-validation bug. Keep Logstash reasonably current so the TLS stack itself does not become the weak link.

If the destination’s cert expired, that is the destination team’s incident, but your queue is the buffer that buys them time. Estimate PQ runway as (max_queue_size_in_bytes - queue_size_in_bytes) / fill rate and shed non-critical traffic if runway is short.

Tradeoff to know: restarting Logstash to pick up new certificates drops the in-flight contents of a memory queue. If you run a persistent queue, events survive the restart; if not, prefer a config reload where the plugin supports picking up new cert files, and accept that some input plugins still need a restart.

Prevention

This outage is 100% preventable because the trigger date is printed inside the certificate years in advance.

Build a certificate inventory. Enumerate every cert Logstash touches: Beats input server certs, Elasticsearch and Kafka output client certs, CA bundles, monitoring pipeline certs, and any keystore or truststore files. Store the list with owners and renewal procedures, not just paths.
Check expiry externally, on a schedule. Run openssl x509 -checkend per certificate from cron, your config management, or your monitoring agent. Do not wait for a Logstash metric; there is none.
Alert at 30 days, page inside 7. Thirty days gives time for normal change management and CA turnaround. Inside seven days, renewal is an operational emergency because a weekend, a holiday, or a stuck approval now sits between you and a total outage.
Check the whole chain, not just the leaf. Intermediates and cross-signed roots expire too, and produce unknown_ca failures that look like someone else’s problem.
Prefer short-lived certs with automated renewal where your PKI supports it. Automation removes the calendar dependency, but monitor the renewal job itself; a silently broken renewal pipeline is the same incident with extra steps.
Load balancer caveat: when Logstash sits behind an LB, handshake error logs show the LB’s address rather than the real client, which slows client-side identification during an incident. Note it in your runbook.

How Netdata helps

Netdata monitors the Logstash Node Stats API per pipeline, so the moment handshakes start failing you see events.in and events.out collapse in per-second resolution rather than at the next dashboard refresh.
Correlating input throughput against queue occupancy and output errors on one dashboard distinguishes an input-side TLS failure (input rate zero, queue empty) from an output-side one (input still arriving, queue filling).
ML anomaly detection flags the deviation from the pipeline’s throughput baseline even when absolute thresholds would miss it, which matters for low-traffic pipelines where “zero” looks normal at night.
Alerting on queue occupancy and growth rate gives you the runway signal while an output-side cert problem is being fixed downstream.
Because cert expiry itself has no Logstash metric, pair Netdata’s pipeline monitoring with an external openssl x509 -checkend job for each cert in your inventory: the external check warns 30 days out, Netdata catches the impact if the warning is ever missed.