$ guides / consul / consul-connect-certificate-expiry ▌

Operations Guides

Consul Connect certificate expired: mTLS handshakes failing across the mesh

Consul Connect relies on short-lived leaf certificates for service-to-service mTLS. When those certificates expire without rotating, the failure is cliff-edged: handshakes that worked seconds ago start failing, Envoy logs fill with TLS errors, and dependent services stop communicating. The blast radius grows over minutes to hours as each leaf certificate reaches its own expiry independently.

The first symptom is usually intermittent: a specific upstream starts returning connection resets or 5xx errors. As more leaves expire, failures fan out across service pairs until the mesh is broadly broken. Expiry alone is a countdown with no early warning. The signal that matters earlier is renewal success: are leaves actually being re-issued, or is the rotation pipeline silently broken while the clock runs out?

This guide covers how to recognize the failure, narrow it to a root cause (clock skew, Vault backend unavailability, CA root or intermediate expiry, or a rotation bug), and restore mesh trust before the outage spreads.

What this means

A Connect leaf certificate carries an mTLS identity for a service. Envoy sidecars present it on every connection and validate it against the trusted CA roots. When notAfter passes, the peer rejects the handshake. Envoy surfaces this as:

TLS error: 268436501:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_EXPIRED

Three failure shapes exist:

Leaf expiry: the common one, and the one that cascades. Each leaf has its own issuance time, so each expires independently. Failures progress across service pairs over minutes to hours rather than all at once.
CA root expiry: catastrophic and sudden. When the active root expires, no new leaf can be validated and rotations fail cluster-wide. This should be a planned event with months of runway, not an incident.
Intermediate signing CA expiry or rotation failure: leaf issuance depends on a valid signing chain. If the intermediate is gone or broken, leaves cannot be re-issued even though the root is fine.

flowchart TD
    A[CA root valid?] -->|yes| B[Intermediate / signing CA valid?]
    A -->|no| R[root rotation required - planned event]
    B -->|yes| C[Leaves rotating before expiry?]
    B -->|no| I[issuance blocked - new leaves fail]
    C -->|yes| H[mTLS healthy]
    C -->|no| D[Clock skew on server or agent?]
    D -->|yes| S[fix NTP - certs seen as expired or not-yet-valid]
    D -->|no| E[Vault / CA backend reachable?]
    E -->|no| V[restore CA backend]
    E -->|yes| F[rotation bug - check Consul and Vault versions]

The diagram is the decision spine for the rest of this article.

Common causes

Cause	What it looks like	First thing to check
Clock skew on a server or agent	`x509: certificate has expired or is not yet valid` with dates that look valid; failures are node-specific, not mesh-wide	`timedatectl status` and `chronyc tracking` on the affected node vs the leader
Vault CA backend unavailable	New leaves fail to issue; existing leaves keep working until TTL runs out, then fail progressively	Vault health endpoint and Consul server logs for CA errors
CA root or intermediate expired	All rotations fail at once; `/v1/connect/ca/roots` shows an expired `NotAfter`	Root and intermediate expiry timestamps
Rotation bug (Vault 1.11+ multi-issuer, old Consul)	Leaf cert size grows, or an expired intermediate is still signing; errors mention oversized handshakes	Consul and Vault versions against known-bad combinations
Leaves not rotating at all	More than 50% of TTL elapsed with no renewal seen; a countdown to the cliff	Renewal success rate, not just expiry time

Quick checks

Run these read-only. None mutate state.

# Check trusted CA roots and which is active
curl -s http://127.0.0.1:8500/v1/connect/ca/roots | jq '.[] | {ID, Active, NotAfter}'

# CA configuration (provider, leaf_cert_ttl, etc.)
curl -s http://127.0.0.1:8500/v1/connect/ca/configuration

# Leaf cert as served to an Envoy sidecar (default admin port 19000)
curl -s http://127.0.0.1:19000/certs | jq '.certificates[].cert_chain[].days_until_expiration'

# Server-side CA metrics
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep consul_connect_ca

# Recent certificate-related errors
journalctl -u consul --since '1 hour ago' | grep -iE 'cert|x509|ca leaf|signing|vault'

# Clock state on this node
timedatectl status
chronyc tracking 2>/dev/null || ntpq -p 2>/dev/null

If Vault is the CA backend:

# Vault health
curl -s http://127.0.0.1:8200/v1/sys/health

# Confirm Consul's Vault token is not expired or revoked
journalctl -u consul --since '1 hour ago' | grep -iE 'vault|permission denied|token'

How to diagnose it

Confirm it is actually certificate expiry, not a generic TLS or network issue. Pull the Envoy cert list and look at days_until_expiration. A 0 or negative value on the failing sidecar means leaf expiry. If the values look fine but handshakes still fail, suspect clock skew next.
Check the CA roots. GET /v1/connect/ca/roots lists every trusted root; the active one has "Active": true. Compare each NotAfter against the current time on the leader. If the active root is expired, you are in root expiry territory, which is a different and more severe response.
Check clock skew before anything else. This is the silent killer. If the leader’s clock is ahead, it may issue leaves that agents reject as “not yet valid.” If an agent’s clock is ahead, it rejects valid leaves as expired. Compare timedatectl and chronyc tracking across servers and the affected agent. The error string is the same (x509: certificate has expired or is not yet valid) whether the cert is genuinely expired or just appears so to a skewed clock.
If leaves are not rotating, check the CA backend. When Consul uses Vault as the Connect CA, Vault unavailability blocks all new leaf issuance. Existing leaves keep working until their TTL expires, then fail. The lag between Vault going down and the outage equals the leaf TTL. Verify Vault is reachable, healthy, and that Consul’s token still has permission to sign.
Check versions against known rotation bugs. Two combinations are well-documented:
- Vault 1.11.0+ as Consul’s CA provider with Consul older than 1.12.8 / 1.13.5 / 1.14.3 breaks intermediate CA rotation. Leaf certificates grow in size as intermediates accumulate until TLS handshakes fail with an oversized-message error. These three branches were released simultaneously (Dec 13, 2022) with the Vault CA provider backport
- Consul older than 1.11.0 could continue using an expired intermediate or root to sign leaves, with validity only checked at TLS request time. Upgrading fixes the bug but exposes any already-broken chain.
Distinguish “rotation is failing” from “rotation is slow.” A leaf at 50% of TTL with no renewal is a ticket. A leaf at 80% with no renewal is heading toward a page. Track renewal as a rate, not just the expiry countdown.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Leaf certificate time-to-expiry	Direct countdown to handshake failure	Any leaf under 1h (page); under 25% of TTL
Leaf renewal success rate	The only early-warning signal; expiry alone is too late	Renewals not happening with more than 50% of TTL elapsed
CA root time-to-expiry	Root expiry breaks the entire mesh at once	Root under 24h (page); under 7 days is urgent
Intermediate / signing CA expiry	Leaves cannot be issued without a valid signing chain	Intermediate under 30 days
CA backend health (Vault)	Vault down blocks all new issuance with a delay equal to leaf TTL	Vault unhealthy or Consul token denied
System clock offset	Clock skew makes valid certs appear expired or not-yet-valid	Offset beyond a few hundred ms on any server
Envoy TLS error rate	The user-visible symptom of expiry	Spike in `SSLV3_ALERT_CERTIFICATE_EXPIRED`

The playbook exposes CA-related telemetry under the consul_connect_ca prefix. Confirmed metrics: consul.mesh.active-root-ca.expiry and consul.mesh.active-signing-ca.expiry (gauges, seconds); leaf expiry is not exposed as a dedicated telemetry metric Treat expiry and renewal as two separate alerting surfaces. A green expiry dashboard with a dead renewal pipeline is exactly the failure this article is about.

Fixes

Pick the fix by root cause. Do not restart services as a first move; it rarely helps and can mask the real signal.

Clock skew

Fix NTP on the affected node or nodes. For containers, confirm the host clock is correct and that the container is not drifting. Once the clock is correct, leaf rotation should resume automatically; already-issued leaves that were rejected get replaced as the sidecar fetches fresh xDS. Verify with the Envoy /certs endpoint after a rotation interval.

Vault CA backend unavailable

Restore Vault first. If Vault is genuinely down and the outage is urgent, you can consider switching the Connect CA provider to Consul’s built-in CA, but this is disruptive: it changes the trust root and requires every sidecar to pick up new roots and new leaves. This is not a 3 a.m. decision unless there is no path to restoring Vault within your remaining leaf TTL runway.

If Vault is up but Consul’s token is expired or revoked, rotate the token per your Vault integration’s documented procedure. Existing leaves continue to work until expiry; the recovery clock is the shortest leaf TTL in the fleet.

CA root or intermediate expired

Root and intermediate rotations must be planned, not improvised mid-outage. Consul orchestrates root rotation when the CA configuration is updated in a way that changes the root key, and uses cross-signing to keep both old and new roots trusted during the rollout. If the old root is removed before every service has the new root, mTLS breaks for the services that lag. For intermediates, plan rotation well before expiry (72 hours before is a standard window, with alerting at 30 days out).

If you are already in an expired-root outage, the recovery is a forced rotation, which is disruptive. Test rotation procedures regularly so this is not the first time you run one.

Rotation bug (Vault 1.11+ multi-issuer or old Consul)

Upgrade Consul to a fixed version (1.12.8+, 1.13.5+, 1.14.3+, or 1.15.0+) if you are on Vault 1.11+ and seeing leaf cert bloat or oversized handshake errors. After upgrade, force a CA root rotation to replace the bloated chain. If you are on Consul older than 1.11.0 and an intermediate expired while still signing leaves, upgrading will surface the broken chain immediately, so have a rotation ready.

Leaves not rotating at all

This is usually one of the above causes in disguise: backend down, token expired, or signing CA invalid. If none of those apply, capture server logs filtered to the CA subsystem and look for CSR timeouts or rejections. Restarting a sidecar or the local Consul client can force a renewal fetch in some cases, but treat that as a confirmation step, not a fix. If rotation is broken at the server, the renewed leaf will expire again on the same schedule.

Prevention

Alert on renewal success, not just expiry. Renewal failure with more than 50% of TTL elapsed is the signal that catches this hours before the cliff.
Page at the right thresholds. Any leaf within 1 hour of expiry, or the CA root within 24 hours, is a page. Leaves not rotating past 50% of TTL is a ticket.
Monitor the CA backend independently. When Vault is the CA provider, Vault health is Consul Connect health with a delay equal to the leaf TTL. Alert on Vault health and on the Consul token’s validity as first-class signals.
Track clock offset on every server. A few hundred milliseconds is usually fine; anything approaching seconds is a candidate for the “cert not yet valid” failure.
Watch CA root and intermediate expiry on a long horizon. Root expiry is a business-continuity event with months of runway. Intermediate rotation should be planned 72 hours before expiry, with alerting at 30 days.
Keep Consul and Vault off the known-bad version combinations. Document the version matrix as part of change management, not as an incident postmortem.
Test rotation procedures regularly. Forced rotation is disruptive; doing it for the first time during an outage makes it worse.

How Netdata helps

Per-second metrics on CA subsystem health let you see renewal rate drop before expiry becomes the headline signal. The consul_connect_ca family is where renewal-success alerts belong.
Correlate Envoy TLS errors with certificate expiry in the same time window. A spike in handshake failures that lines up with a leaf crossing its notAfter is the signature of this incident, and per-second resolution makes the alignment obvious.
Track system clock offset alongside Consul metrics. When cert errors appear, the first question is whether the clock moved. Having NTP or chrony offset on the same dashboard as Consul CA metrics shortens the clock-skew diagnosis from hours to minutes.
Monitor Vault as a dependency of Consul Connect. When Vault is the CA backend, Netdata’s Vault collector next to the Consul collector lets you see the issuance path break before leaves start expiring.