Envoy TLS certificate expiry: expired certs, broken SDS rotation, and total outage

TLS certificate expiry in Envoy is one of the few failure modes that can take down an entire service mesh at once. A single expired server certificate breaks new handshakes on one listener. An expired CA root breaks every mTLS connection in the data plane, every Envoy-to-control-plane link, and every health check that uses TLS.

Expiry is silent until handshakes start failing. Envoy does not expose certificate expiration as a standard metric. The only authoritative source is the /certs admin endpoint, which must be polled externally. Teams running automated rotation via SDS or cert-manager often assume rotation is working and discover otherwise when TLS breaks across the fleet.

What this means

Failure manifests differently depending on what expired and where it is used.

Leaf (server) certificate expiry causes new TLS handshakes to fail. The peer receives a certificate-level error. On Envoy’s side, ssl.fail_verify_error spikes on the affected listener or cluster, and ssl.connection_error may also climb. Existing connections may persist briefly, but every new handshake fails.

CA root certificate expiry is far worse. Every certificate issued by that CA becomes unverifiable at the same instant. In a service mesh with mTLS, this breaks all service-to-service communication, Envoy-to-control-plane xDS connections, and health checks simultaneously. control_plane.connected_state drops to 0 because the xDS connection uses the same PKI. ssl.fail_verify_error spikes across every listener and cluster that uses the affected trust chain. upstream_cx_connect_fail increases across many clusters at once.

Broken SDS rotation is the upstream cause in most cases. When SDS silently fails, Envoy continues serving the old certificate and no error surfaces immediately. The certificate’s days_until_expiration countdown keeps ticking toward zero. When it reaches zero, TLS breaks. The gap between SDS failure and user-visible impact can be days or weeks, making root cause correlation difficult during the incident.

flowchart TD
    A[SDS rotation broken] --> B[days_until_expiration counts toward 0]
    B --> C{What expires?}
    C -->|Leaf cert| D[New handshakes fail
on one listener or cluster]
    C -->|CA root cert| E[All mTLS breaks simultaneously]
    D --> F[ssl.fail_verify_error spikes
on one surface]
    E --> G[ssl.fail_verify_error spikes everywhere
control_plane.connected_state drops to 0]

Common causes

Quick checks

These are safe, read-only commands. Adjust the admin port if you run Istio sidecars, which typically use 15000 instead of 9901. The Envoy admin interface binds to 127.0.0.1 by default; do not expose it externally.

# Check certificate expiry countdown via /certs admin endpoint
# days_until_expiration is the minimum across the cert chain
curl -s http://localhost:9901/certs | jq '.certificates[] | {subject: .cert_chain[].subject, days_until_expiration: .days_until_expiration}'

# Check TLS handshake verification failures (downstream and upstream)
curl -s http://localhost:9901/stats | grep 'ssl.fail_verify_error'

# Check TLS connection errors
curl -s http://localhost:9901/stats | grep 'ssl.connection_error'

# Check SDS/xDS connection state (CA expiry drops this to 0)
curl -s http://localhost:9901/stats | grep 'control_plane.connected_state'

# Check system clock for skew
timedatectl status

# Check cert file dates directly (for file-based certs)
openssl x509 -in /etc/envoy/tls/tls.crt -noout -dates

# Check live cert presented by a TLS endpoint
echo | openssl s_client -connect localhost:8443 -servername example.com 2>/dev/null | openssl x509 -noout -dates

# Check upstream connection failures across all clusters (CA expiry hits many at once)
curl -s http://localhost:9901/stats | grep 'upstream_cx_connect_fail'

uint64 wrap gotcha. The days_until_expiration field in /certs is defined as a uint64, not a signed integer. On some Envoy versions, an already-expired certificate shows days_until_expiration as a very large positive number (close to 18446744073709551615) instead of a negative value. If you are polling /certs and checking only for values near zero, you will miss expired certs entirely. Treat any value above 1e15 as expired.

How to diagnose it

1. Poll /certs on the affected Envoy. Look at days_until_expiration for every certificate in every chain. The field represents the minimum across the chain, so if any cert in the chain is expired, the value reflects that. Remember the uint64 wrap: values near UINT64_MAX mean the cert is already expired.

2. Check ssl.fail_verify_error and ssl.connection_error. If these are spiking on a single listener or cluster, the problem is localized to one certificate. If they are spiking across every listener and cluster simultaneously, suspect CA root expiry or a widespread SDS failure.

3. Check control_plane.connected_state. If this is 0, the xDS connection has dropped. In mTLS environments, the xDS connection uses the same PKI as the data plane. A CA root expiry will drop both at the same time.

4. Verify the system clock. Run timedatectl status or chronyc tracking. If the system clock is ahead of real time, valid certificates will appear expired to Envoy. The days_until_expiration value from /certs reflects the system clock, not the real wall clock.

5. Check the SDS server or cert-manager. If days_until_expiration is counting down with no rotation, the rotation pipeline is broken. Look at SDS server logs (Istiod, custom xDS server) or cert-manager controller logs for errors. On Envoy’s side, check whether SDS requests are being made at all via the xDS connection stats.

6. Cross-reference with /config_dump. If you suspect a CA root issue, inspect the trust chain in the active configuration to confirm which root certificate Envoy is using and when it expires.

Metrics and signals to monitor

Fixes

Broken SDS rotation

If days_until_expiration is dropping toward zero and no rotation has occurred, the SDS pipeline is broken somewhere between the SDS server and Envoy.

1. Verify Envoy is still connected to the control plane (control_plane.connected_state = 1). If it is 0, the SDS/xDS connection itself failed, possibly due to an earlier cert expiry in the trust chain.
2. Check the SDS server (Istiod, custom xDS server) for errors. Look for push failures, authentication errors, or rate limiting.
3. If the SDS server is healthy but Envoy is not receiving updates, check the network path and the SDS-specific xDS stream.
4. As an immediate mitigation, push a new cert via a redeploy or config update. Do not wait for the automated pipeline to self-heal if you are inside the 7-day window.

CA root expiry

This is a total mesh outage. Every mTLS connection fails simultaneously. Recovery requires restoring the CA trust chain.

1. Identify the expired CA certificate. Check /certs and /config_dump on any still-reachable Envoy instance.
2. Restore or renew the CA root certificate. This is a PKI infrastructure operation, not an Envoy operation.
3. Rotate all leaf certificates issued by the renewed CA. In SDS environments, push the new root and leaf certs through the control plane.
4. Verify recovery by checking that ssl.fail_verify_error drops to zero and control_plane.connected_state returns to 1 across the fleet.

Clock skew

If the system clock is ahead of real time, valid certificates appear expired.

1. Check the clock: timedatectl status or chronyc tracking.
2. Correct the clock via NTP or chrony.
3. Verify that ssl.fail_verify_error drops after correction.
4. Investigate the root cause of the skew (VM migration, NTP server unreachable, container clock drift).

Static file cert expiry

File-based certificates do not rotate dynamically. Per Envoy’s SDS documentation, when file-based certificates expire, the secrets must be updated and the proxy containers must be redeployed.

1. Update the certificate file on disk or in the mounted secret.
2. Redeploy or restart the Envoy container to pick up the new cert.
3. Migrate to SDS-based rotation to avoid this in the future.

Prevention

• Poll /certs externally. Certificate expiry is not a standard Envoy stat. Set up an external check that polls /certs and extracts days_until_expiration for every certificate in every chain.
• Alert at 14 days, page at 7. Give yourself enough runway to fix a broken rotation pipeline before the cert expires.
• Watch for the uint64 wrap. If your polling code only checks for small values, expired certs will be invisible. Treat values above 1e15 as expired.
• Monitor SDS connection state. control_plane.connected_state = 0 means cert rotations have stopped. Do not let this persist.
• Track CA root cert expiry separately. CA certs have multi-year lifetimes and are easy to forget. They are also the highest-impact expiry event.
• Test rotation before you need it. Force a cert rotation in a staging environment and verify that days_until_expiration resets. If it does not, your rotation pipeline is broken and you will find out at the worst possible time.
• Monitor system clock drift. Set up alerts on clock skew. A clock that drifts ahead can make valid certs appear expired.
• Distinguish leaf expiry from CA expiry in your runbook. Leaf expiry is a targeted fix. CA expiry is a fleet-wide PKI operation.

How Netdata helps

Netdata can correlate the signals that surround a certificate expiry event, giving you early warning before TLS breaks.

• Per-second TLS handshake metrics. ssl.fail_verify_error and ssl.connection_error are collected at per-second resolution, so spikes surface immediately rather than at the next scrape interval.
• ML-based anomaly detection on ssl stats. A sudden increase in TLS verification failures triggers an anomaly alert even if you have not set a fixed threshold.
• Correlation with control plane state. When control_plane.connected_state drops alongside an ssl.fail_verify_error spike, the composite pattern points directly at CA root expiry rather than a localized cert issue.
• Correlation with upstream connection failures. Widespread upstream_cx_connect_fail across multiple clusters, combined with SSL errors, distinguishes a PKI failure from a network partition.
• External cert expiry polling.

Related guides

• Envoy 502 and upstream resets: rx_reset, tx_reset, and mid-response failures
• Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend
• Envoy 504 upstream timeout: upstream_rq_timeout, per-try timeouts, and the UT flag
• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy clusters stuck warming: warming_clusters non-zero and routes returning 503
• Envoy connection pool exhaustion: a slow upstream that fills the pool
• Envoy control_plane.connected_state = 0: running on stale xDS config
• Envoy downstream 4xx spike: 401s, 403s, and 404s from the client side
• Envoy downstream_cx_active growing: connection leaks and idle-timeout gaps
• Envoy downstream_cx_overflow and overload_reject: connections turned away at the door
• Envoy downstream_rq_time high: client-observed latency and proxy overhead
• Envoy file descriptor exhaustion: the FD cliff that refuses every new connection