$ guides / logstash / logstash-ssl-tls-handshake-failures ▌

Operations Guides

Logstash SSL/TLS handshake failures: expired certs and broken trust chains

Your Logstash logs are filling with PKIX path building failed or handshake_failure errors. Depending on which side is failing, you either have a delivery outage in progress (an output that can no longer connect to its destination) or a security signal (clients failing to authenticate against an input). The triage path starts the same either way: find the failing plugin, read the actual certificate error, and determine whether this is an availability problem, a security problem, or a planned rotation you forgot about.

The two most common root causes are an expired certificate somewhere in the chain and a trust chain mismatch, usually after a CA rotation or a credential rollout that only partially landed. Both are cheap to confirm once you know where to look. The expensive mistake is misreading the blast radius: an output-side TLS failure is the first stage of the classic backpressure cascade where workers block, the queue fills, and inputs stall.

What this means

Logstash plugins establish TLS connections in two directions. Inputs (beats, tcp, http) terminate inbound TLS from clients. Outputs (elasticsearch, http, tcp, kafka) initiate outbound TLS to destinations. A handshake failure on either side produces stack traces in /var/log/logstash/logstash-plain.log, but the operational consequences differ sharply.

On the output side, a plugin that cannot complete a handshake cannot deliver. Workers block on the failing output, retry, block again, and some plugins retry aggressively enough to flood the log. If the failure is sustained, the standard cascade follows: queue grows, backpressure propagates to inputs, upstream senders buffer or drop. The TLS error is the root cause; the queue metrics are how you see the damage.

On the input side, a handshake failure means a client could not authenticate or negotiate. If your expected senders are healthy and still delivering, repeated input-side failures usually mean unauthorized or misconfigured connection attempts, or one sender with a stale certificate. If all senders fail at once, suspect the server-side certificate on the input itself.

One benign case is common: planned certificate rotation. During a rotation window, some clients or servers present the new chain while others still present the old one, producing a burst of handshake failures that self-resolves. Confirm whether a rotation was scheduled before treating the pattern as an incident.

Common causes

Cause	What it looks like	First thing to check
Expired certificate	`certificate_expired` / `CertificateExpiredException` in the log; failures start abruptly at a specific time	`openssl x509 -enddate -noout -in` on every cert in the chain
Broken trust chain after CA rotation	`PKIX path building failed: unable to find valid certification path to requested target`; starts right after a CA or intermediate change	Does the configured CA bundle contain the full new chain, including intermediates?
Credential rotation partially deployed	Some senders connect, others fail; errors correlate with a recent deployment	Which clients are failing? Are they on the old cert?
Hostname/SAN mismatch	Handshake or verification failure even though the cert is valid and unexpired; often when connecting by IP or an alias	Compare the hostname in the `hosts` URL against the certificate SANs
Unauthorized client attempts on an input	Repeated handshake failures from unknown source IPs; pipeline throughput unaffected	Source IPs of the failing connections
Deprecated TLS protocol or cipher	`handshake_failure` with no cert error, often after a JDK or Logstash upgrade	Configured protocol versions versus what the peer supports
Upgrade removed old SSL setting names	Pipeline fails to start or reload after a major version upgrade, with config validation errors	Whether the config uses pre-9.0 setting names

Quick checks

All of these are read-only.

# 1. Find TLS and auth errors in the Logstash log
grep -Ei '(SSL|TLS|certificate|handshake|PKIX|unable to find valid certification path|authentication|unauthorized)' \
  /var/log/logstash/logstash-plain.log | tail -n 200

The logger name and pipeline ID in each line tell you which plugin and direction is failing. An [elasticsearch] output logger means delivery is at risk. A [beats] input logger means a client failed to connect.

# 2. When did the failures start, and how many are there?
grep -Ei '(SSL|TLS|certificate|handshake|PKIX)' /var/log/logstash/logstash-plain.log \
  | awk '{print $1}' | head -n 3
grep -cEi '(certificate|handshake|PKIX)' /var/log/logstash/logstash-plain.log

An abrupt start at a specific timestamp points at expiry or a change event. A trickle over weeks points at drifting clients.

# 3. Check the certificate Logstash presents or trusts, without restarting anything
openssl x509 -enddate -noout -in /path/to/logstash-cert.pem
openssl x509 -issuer -subject -noout -in /path/to/logstash-cert.pem

Repeat for every file referenced by the failing plugin’s SSL settings, and for the CA bundle. An expired intermediate is as fatal as an expired leaf.

# 4. Inspect what the peer actually presents (output-side failures)
openssl s_client -connect elasticsearch-host:9200 -servername elasticsearch-host </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -enddate

Compare the issuer chain against the CA bundle configured in the output plugin. If the peer presents a chain your bundle cannot build to a trusted root, that is your answer.

# 5. Is the failure already hurting the pipeline?
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty

Look at flow.output_throughput.current, queue.events_count, flow.queue_backpressure, and for persistent queues queue.queue_size_in_bytes versus queue.max_queue_size_in_bytes. This tells you whether you have a log noise problem or a delivery outage.

# 6. Was anything deployed or rotated recently?
find /etc/logstash -maxdepth 2 -type f -printf '%TY-%Tm-%Td %TT %p\n' | sort | tail

Correlate certificate file mtimes and config mtimes with the first failure timestamp.

How to diagnose it

Work through these in order. Stop as soon as the failure class is confirmed.

Localize the failing plugin. From check 1, identify whether errors come from an input or an output, which pipeline, and which destination or listener. If errors span multiple unrelated plugins at the same moment, suspect a shared dependency: a common CA bundle, a keystore, or a JDK-level change.
Classify the error string. CertificateExpiredException or certificate_expired is expiry, full stop. PKIX path building failed or unable to find valid certification path is a trust chain problem: the peer’s chain does not terminate in anything your trust material contains, or an intermediate is missing. A bare handshake_failure with no certificate detail is usually protocol or cipher negotiation, not trust. unauthorized or authentication without TLS detail is a credential problem, not a certificate problem.
Check expiry across the whole chain. Run check 3 against the leaf, every intermediate, and the CA files on both sides of the failing connection. Check the peer’s presented cert as well (check 4). A common trap: the leaf was renewed but an intermediate in the served chain expired, or the renewed cert was issued by a new intermediate that the peer’s CA bundle does not include.
Verify chain completeness. The configured CA bundle must let the verifier build a path from the presented leaf to a trusted root. Providing only the leaf, or only the root without intermediates, produces exactly the PKIX path building failed error. If a CA rotation happened recently, confirm the bundle on the verifying side contains the new chain, and check whether old-chain clients are still expected during the overlap window.
Check hostname verification. If verification mode is full (the default on the elasticsearch output), the hostname in the plugin’s hosts URL must match a SAN on the certificate. Connecting by IP address or by a DNS alias not on the cert fails verification even when the cert is otherwise valid and trusted. This failure can surface as a certification path or verification error, so rule it out before rebuilding trust material.
Assess blast radius. From check 5, decide what you are dealing with:
- Output failing, queue flat, retries absorbing: log noise plus latency risk. Fix during business hours.
- Output failing, queue growing, PQ occupancy climbing: you are inside the backpressure cascade. Estimate runway: (max_queue_size_in_bytes - queue_size_in_bytes) / current growth rate. Treat as an incident.
- Input failing, expected senders healthy, throughput normal: security or hygiene issue. Identify the source IPs of the failing connections.
- Input failing, all senders failing, input throughput zero: the input’s own certificate or listener config is broken. This is an ingestion outage.

flowchart TD
  A[TLS errors in logstash-plain.log] --> B{Which side?}
  B -->|Output plugin| C[Delivery blocked]
  B -->|Input plugin| D{Expected senders still healthy?}
  C --> E{Error class}
  E -->|certificate_expired| F[Find expired cert in chain]
  E -->|PKIX path building failed| G[Trust chain mismatch or missing intermediate]
  E -->|handshake_failure only| H[Protocol or cipher mismatch]
  C --> I{Queue growing?}
  I -->|Yes| J[Backpressure cascade - estimate PQ runway]
  I -->|No| K[Retries absorbing - fix in business hours]
  D -->|Yes| L[Unauthorized or stale client - check source IPs]
  D -->|No, all failing| M[Input cert or listener broken - ingestion outage]

Metrics and signals to monitor

Signal	Why it matters	Warning sign
TLS/auth error lines in `logstash-plain.log`	The only direct signal for these failures; there is no API metric for handshake errors	Any sustained pattern; volume spikes from aggressive plugin retries
`flow.output_throughput.current` per pipeline	Tells you whether an output-side TLS failure has become a delivery outage	Drops toward zero while `flow.input_throughput.current` is positive
`queue.events_count` / PQ occupancy	Measures how much failure the queue is absorbing while the output is down	Monotonic growth; occupancy above 80% of `max_queue_size_in_bytes`
`flow.queue_backpressure`	Shows input threads being throttled by the full queue	Rising significantly above the pipeline baseline for more than 10 minutes
`flow.queue_persisted_growth_bytes`	Direct fill-rate measurement for PQ runway estimation	Sustained positive values during the outage
Retry/error patterns in the log alongside TLS errors	Distinguishes a hard auth failure from a transient connectivity problem	Retries rising in step with queue growth

A note on severity: a standalone TLS error pattern is a ticket-level signal. It escalates to page-level only through the composite: active input, sustained output impairment, corroborating errors, and either a blocking memory queue or PQ runway under 30 minutes. Do not page on the log pattern alone, because planned rotation and unauthorized-client noise will burn you.

Fixes

Group the fix by the failure class you confirmed.

Expired certificate

Renew the expired cert, deploy it to the side that presents it, and reload or restart the affected pipeline. If the expired cert is the peer’s (for example, Elasticsearch renewed on its side), you may only need to update the CA bundle Logstash trusts. If an intermediate expired, get a corrected chain from your CA rather than editing intermediates by hand. Tradeoff: a full Logstash restart drops in-flight memory-queue events; a config reload avoids that, but only if the plugin picks up the cert change on reload. Verify the pipeline actually reloaded by checking reloads.successes and reloads.failures in the pipeline stats, since a failed reload leaves the old (still broken) config running invisibly.

Broken trust chain after CA rotation

Rebuild the CA bundle the verifying side uses so it contains the complete new chain: root plus all intermediates. During a rotation overlap, the bundle may need to trust both old and new roots until every peer has migrated. Do not “fix” this by disabling verification. The elasticsearch and http outputs both support ssl_verification_mode values full and none (default full); setting it to none is a temporary diagnostic step only and should never survive to the end of the incident.

Hostname mismatch

Either connect using a hostname that appears in the certificate’s SANs, or reissue the certificate with the names and addresses Logstash actually uses. For environments where Logstash must connect by IP, the IP must be present as a SAN entry.

Unauthorized clients on an input

If the failing clients should not be connecting at all, this is working as intended, and the fix is network-level access control and log noise management, not a Logstash change. If they should be connecting, they are presenting stale or wrong certificates: coordinate their rotation. Note for the beats input: in current plugin versions, configuring ssl_certificate_authorities alone does not enable client certificate verification — you must also set ssl_client_authentication to optional or required (default none). Unexpected failures after an upgrade are more likely a Filebeat config still using the removed pre-7.0.0 setting name ssl_verify_mode, which the current plugin rejects at startup.

Retry storms flooding the log

While the outage is being fixed, aggressive plugin retries can flood logstash-plain.log and, at high volume, add disk I/O pressure on the log volume. Watch disk space on /var/log/logstash during a prolonged output outage. The durable fix is resolving the TLS failure; log volume management is mitigation.

Logstash 9.0 removed the deprecated SSL setting names across many plugins. If TLS broke exactly at an 8.x to 9.x upgrade, your config is likely using removed names such as cacert (now ssl_certificate_authorities), ssl (now ssl_enabled), or ssl_certificate_verification (now ssl_verification_mode). In 8.x these produce deprecation warnings and still work; in 9.x they fail. The per-plugin rename tables are in the Logstash 9.0 breaking changes documentation; check the table for your plugin before applying.

Prevention

Track certificate expiry as a metric, not a calendar invite. Scrape notAfter for every cert Logstash presents or trusts and alert with enough lead time for your renewal process. Expiry is the most predictable failure in this entire guide.
Treat CA rotation as a deployment with an overlap window. Trust both chains during migration, and monitor handshake failure rates as the migration progress signal.
Monitor the escalation path, not just the error. TLS failures have no API metric. What the API does give you is the consequence: output throughput, queue occupancy, flow.queue_backpressure, and PQ fill rate. Alert on the composite so a blocked output pages before the queue fills.
Watch reload state after every cert or config change. reloads.failures greater than zero means your fix did not land even though the deploy “succeeded”.
Baseline input-side failure rates. A low background rate of failed handshakes on internet-reachable or shared inputs is normal. A deviation from baseline is what matters, both for attack detection and for catching one sender stuck on an old cert.
Plan rotation windows into your alert tuning. Suppress or downgrade TLS-pattern alerts during announced rotation windows so the benign case does not train the team to ignore the signal.

How Netdata helps

Log pattern surveillance: Netdata’s log monitoring can count TLS, certificate, and handshake error lines in logstash-plain.log continuously, turning a grep-you-run-during-incidents into a baseline with deviation alerts.
Pipeline consequence correlation: Netdata collects the Logstash monitoring API on port 9600, so you can see output throughput, queue depth, and backpressure next to the error pattern on one dashboard and judge blast radius in seconds.
PQ runway visibility: Persistent queue occupancy and growth trends are charted over time, which is exactly what you need to estimate time-to-full while an output is down.
Retry storm detection: Sustained error-rate charts distinguish a hard TLS failure from transient noise and show whether log flooding is adding disk pressure.
Escalation-ready composites: Combining the log error pattern with queue growth and throughput drop lets you alert on the cascade instead of the symptom, so rotation noise stays a ticket and real delivery outages page.