$ guides / envoy / envoy-tls-fail-verify-error ▌

Operations Guides

Envoy ssl.fail_verify_error: certificate verification failures on the TLS path

When ssl.fail_verify_error starts climbing on an Envoy proxy, TLS handshakes are failing peer certificate verification. A small burst during a planned rotation is operational noise. A sustained climb past roughly 1% of handshakes means the trust relationship between Envoy and its peers has broken.

This counter is one of several SSL stats Envoy tracks, and the others help narrow the failure mode. ssl.connection_error covers protocol-level issues such as TLS version or cipher mismatch. ssl.no_certificate fires when a client presented no certificate where one was required. ssl.fail_verify_error means a certificate was presented and parsed, but Envoy could not validate it against its configured trust context, SAN matchers, or pin hashes.

The same stat lives on both sides of the proxy. On the listener side it appears as listener.<address>.ssl.fail_verify_error and reflects client certificates Envoy rejected. On the cluster side it appears as cluster.<name>.ssl.fail_verify_error and reflects server certificates Envoy rejected when dialing upstream. The diagnostic path differs for each, so the first task is always to localize.

What this means

Envoy increments ssl.fail_verify_error when the configured CertificateValidationContext rejects a peer certificate that was received and parsed. The common reasons, in roughly the order you meet them in production:

The peer certificate is signed by a CA Envoy does not trust. Missing or stale trusted_ca, or a CA rotation that has not propagated to Envoy’s validation context.
The certificate is expired, or the chain crosses an expired intermediate or root.
A SAN matcher (match_typed_subject_alt_names) does not match what the certificate actually contains.
A certificate hash or SPKI pin does not match.
In strict mTLS, the peer is presenting a certificate but it does not satisfy the validation policy. A non-mesh host added to a STRICT mTLS cluster fails every connection to it.

A baseline rate near zero is normal. The abnormal threshold is fail_verify_error / handshake > 0.01 sustained. A burst during a rotation is expected; a sustained climb is not. Certificate verification is not enabled by default: Envoy only enforces it when a validation context specifies one or more trusted authorities, so a non-zero counter implies verification is configured and active.

Common causes

Cause	What it looks like	First thing to check
Certificate or CA expired	Climbing failures across many clusters or listeners at once, often a hard step at a specific wall-clock time	`GET /certs` for `days_until_expiration`; `openssl s_client` against the peer
CA rotation not propagated	Failures start after a CA change; some Envoys affected, others not	`config_dump` validation context and `trusted_ca` hash on affected vs unaffected instances
SAN mismatch	Failures on a specific upstream regardless of cert validity	Compare SANs in the peer cert against `match_typed_subject_alt_names`
Strict mTLS mode mismatch	Downstream failures after PERMISSIVE to STRICT switch, or after adding a non-mesh host	mTLS mode policy and sidecar presence on the calling pod
SDS load failure	Failures correlate with `control_plane.connected_state = 0` or stale SDS version; transport reason may say “Secret is not supplied by SDS”	Secret version in `config_dump`; SDS/xDS connection state

Quick checks

Read-only commands to localize the failure. The default admin port is 9901; Istio sidecars typically expose the admin endpoint on 15000.

# All ssl.* counters across listener and cluster sides
curl -s http://localhost:9901/stats | grep 'ssl\.'

# Isolate listener (downstream) verification failures
curl -s http://localhost:9901/stats | grep 'listener.*ssl\.fail_verify'

# Isolate cluster (upstream) verification failures
curl -s http://localhost:9901/stats | grep 'cluster.*ssl\.fail_verify'

# Compute failure ratio against successful handshakes
curl -s http://localhost:9901/stats | grep -E 'ssl\.(handshake|fail_verify_error)'

# Inspect loaded certs and days until expiry
curl -s http://localhost:9901/certs | jq '.certificates[] | {subject: .cert_chain[].subject, days_until_expiration: .days_until_expiration}'

# Validation context and SAN matchers Envoy is actually applying
curl -s http://localhost:9901/config_dump | jq '.configs[] | .. | .validation_context? // empty'

# Control plane and SDS delivery health
curl -s http://localhost:9901/stats | grep -E 'control_plane\.connected_state|update_(failure|rejected)'

# Verify the peer cert chain directly (replace host:port)
echo | openssl s_client -connect upstream.host:443 -showcerts 2>/dev/null | openssl x509 -noout -dates -subject -issuer

How to diagnose it

The diagnostic flow branches early on which side of the proxy is failing and then on the transport failure reason string in the access log.

flowchart TD
    A[fail_verify_error climbing] --> B{Which side?}
    B -->|listener.*| C[Downstream: client cert rejected]
    B -->|cluster.*| D[Upstream: server cert rejected]
    C --> E[Check mTLS mode and client identity]
    D --> F[Check SAN, CA trust, cert expiry]
    E --> G[Read transport_failure_reason]
    F --> G
    G --> H{Error category}
    H -->|EXPIRED| I[Cert or CA expired]
    H -->|UNKNOWN_CA| J[Trust gap or CA rotation]
    H -->|SAN or hash mismatch| K[Matcher or pin mismatch]
    H -->|Secret not supplied by SDS| L[SDS delivery failure]

Localize the side. Failures on listener.* mean Envoy is rejecting client certificates; failures on cluster.* mean Envoy is rejecting upstream server certificates. The fix path is different.
Quantify severity. Sample two points roughly 60 seconds apart and compute fail_verify_error / handshake. Above 1% sustained is abnormal.
Scope the blast radius. A single cluster points to a backend cert or its CA. Many clusters or listeners at once points to a root CA, an SDS outage, or a mesh-wide policy change.
Pull the transport failure reason from access logs. The UPSTREAM_TRANSPORT_FAILURE_REASON field (or DOWNSTREAM_TRANSPORT_FAILURE_REASON for listener-side failures) carries the BoringSSL alert string. The mapping to cause is direct.
Check certificate validity. Both Envoy’s loaded certs (/certs) and the peer cert (openssl s_client). Look at the full chain, not just the leaf.
Verify the validation context. Confirm trusted_ca matches the issuer, and confirm match_typed_subject_alt_names matches the SAN types the cert actually carries.
Check SDS and xDS state. control_plane.connected_state = 0 or a stale secret version in config_dump explains a missing rotation.
Cross-check the mesh policy. In Istio, confirm the PeerAuthentication mode for the workload and confirm callers have a sidecar.

Transport failure reason string	Likely cause
`SSLV3_ALERT_CERTIFICATE_EXPIRED`	Peer certificate, intermediate, or root is expired
`TLSV1_ALERT_UNKNOWN_CA`	CA is not in Envoy’s `trusted_ca` set
`SSLV3_ALERT_CERTIFICATE_UNKNOWN`	Certificate does not match a configured SPKI pin
`SSLV3_ALERT_HANDSHAKE_FAILURE`	Often: peer requires a client cert and none was presented
`TLSV1_ALERT_PROTOCOL_VERSION`	TLS version negotiation mismatch
`Secret is not supplied by SDS`	SDS has not delivered the referenced secret

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`listener.<addr>.ssl.fail_verify_error`	Downstream client cert verification failures	Sustained ratio >1% of `ssl.handshake`
`cluster.<name>.ssl.fail_verify_error`	Upstream server cert verification failures	Same threshold
`listener.<addr>.ssl.connection_error`	Protocol, version, or cipher mismatch, distinct from verification	Spike alongside handshake changes
`listener.<addr>.ssl.no_certificate`	Client presented no cert when one was required	Nonzero in strict mTLS environments
`control_plane.connected_state`	SDS and xDS delivery stops when disconnected	Drops to 0 and stays there
`ssl.certificate.<name>.days_until_expiration`	Runway before automated rotation must succeed	Below 14 days, or trending down without rotation
`server.watchdog_miss`	TLS handshake bursts saturate workers and amplify failures	Any nonzero value during the incident

Fixes

Expired certificate or CA

Replace the certificate. If SDS-managed, push a forced rotation rather than waiting for the natural schedule. If an expired intermediate or root is present in the trust store alongside a valid one, Envoy may build the chain through the expired entry; remove the expired material from the validation context’s trusted_ca bundle rather than relying on Envoy to pick the valid one. The deterministic fix is to prune the trust store.

CA rotation not propagated

Pull config_dump on an affected Envoy and compare the trusted_ca hash against the new CA bundle. If Envoy is still on the old context, the rotation did not reach it. Common causes: SDS connectivity loss, a NACKed config push (check update_rejected), or a stale node identity. Restarting Envoy should be a last resort. First force an SDS re-request by retriggering a config push from the control plane.

SAN mismatch

The certificate’s Subject Alternative Names must satisfy at least one matcher in match_typed_subject_alt_names. Common failure: matching on DNS SAN when the cert only carries URI SANs (typical in SPIFFE-based meshes), or the reverse. Pull the SAN list with openssl x509 -text and compare against the configured matchers. Note that match_subject_alt_names is deprecated in favor of match_typed_subject_alt_names; if both are specified, the deprecated field is ignored.

Specifying only trusted_ca without any SAN matcher verifies the chain but not the subject. That is a security gap. If you intended identity binding, the matcher is missing.

Strict mTLS mode mismatch

In Istio, switching a workload from PERMISSIVE to STRICT without every caller having a sidecar produces downstream fail_verify_error on the strict-mode listener. The fix is either to roll sidecars out to all callers first, or to relax the PeerAuthentication for the affected workloads. Symmetrically, adding a non-mesh host to a cluster whose TLS settings assume mTLS causes every connection to that host to fail. The cluster needs an exception (for example trafficPolicy.tls.mode: DISABLE on the destination rule) or the host needs to be brought into the mesh.

SDS load failure

If SDS is not delivering the required secret, Envoy may keep the listener bound to the port but reset incoming connections, or continue serving on the last known certificate. The signal is Secret is not supplied by SDS in the transport failure reason, paired with control_plane.connected_state = 0 or an SDS update failure. The fix is to restore SDS connectivity and confirm the referenced secret resource exists. Envoy applies the new secret on the next handshake once SDS pushes it; a hot restart is not required.

Prevention

Monitor cert runway. Track ssl.certificate.<name>.days_until_expiration for every loaded cert. In SDS environments this is the canary that tells you rotation is broken before the cert expires.
Alert on the failure ratio, not the raw counter. Alert on ssl.fail_verify_error / ssl.handshake > 0.01 sustained for more than 5 minutes on any listener or cluster.
Treat control plane disconnection as a ticket. SDS rotation stops when control_plane.connected_state drops to 0. Do not wait for an expired cert to find out.
Keep SAN matchers in lockstep with the identity format. Match match_typed_subject_alt_names to what the CA actually issues (DNS, URI/SPIFFE, etc).
Prune the trust store on every rotation. Do not leave expired CAs or intermediates in trusted_ca and rely on Envoy to prefer the valid duplicate.
Roll strict mTLS per workload. Watch downstream fail_verify_error on each PERMISSIVE to STRICT transition before moving to the next.

How Netdata helps

Per-second ssl.fail_verify_error, ssl.connection_error, ssl.handshake, and ssl.no_certificate counters on both listener and cluster sides let you see the failure start and stop at second resolution, which is what you need to correlate with cert rotation events or config pushes.
ML anomaly detection on the fail_verify_error / handshake ratio catches slow climbs before they cross the 1% threshold.
Envoy TLS metrics sit alongside control_plane.connected_state, xDS update counters, and worker CPU in the same view, which is how you tell a cert problem from an SDS delivery failure from a TLS handshake CPU storm.
Cert expiry dashboards surface days_until_expiration as a countdown rather than requiring a manual /certs poll during an incident.
Anomaly advisors flag correlated signals, such as fail_verify_error rising at the same moment update_rejected ticks up, which points the investigation at a bad config push rather than a CA problem.