Consul Connect CA rotation failure: a root roll that never finished

Consul Connect CA root rotation promotes a new root, cross-signs the new intermediate against the old root, lets both roots coexist while leaf certificates roll over, and retires the old root only after the last leaf signed by it expires. When that rollout never completes, the first thing you notice is intermittent mTLS failures between specific service pairs, hours after the rotation was triggered, with no obvious network or config change to blame.

The pattern is distinctive but easy to misread. Leaf certs have short, independent TTLs (72h by default), so failures do not break all service-to-service traffic at once. Each pair breaks at the moment its cached leaf expires and a renewal either fails or produces a cert the peer no longer trusts. It looks like a rolling outage of the mesh, blamed on the application, Envoy, or a flaky node, when the actual cause is upstream in the CA.

What this means

A “root-roll that never finished” is not a single failure. It is one of several distinct modes that all surface as intermittent, time-spreading mTLS failures:

• Premature root retirement. The new root was promoted to active, but the old root was removed before every agent had observed and distributed the new root’s trust bundle. Leafs signed by the old root continue to validate locally until they expire, then fail at the peer.
• Stuck mid-roll. The new root is active on the leader, but CSRs are being rejected or timing out. Renewals are not happening at all. Leafs age toward expiry with no replacement, then break in the order they were issued.
• Root itself expired. Consul does not automatically rotate the root CA for either the built-in or the Vault provider. Existing leafs keep working until their own TTL elapses, then every new issuance fails.
• Cross-signing bridge missing. Cross-signing between the old and new roots failed (key type mismatch, backend unreachable). The trust bridge that lets old and new roots coexist during the roll was never built, so any leaf signed by the old root is rejected the moment the new root becomes the only one trusted.

The defining behavior is progressive, pair-by-pair failure over hours or days, not simultaneous outage. If every Connect connection fails at the same instant, suspect a network or intention change before CA rotation.

flowchart TD
    A[CA rotation triggered] --> B[New root active, old root coexists]
    B --> C[Service requests leaf renewal]
    C --> D{Backend can sign?}
    D -->|No: Vault down, root expired, CSR rejected| E[Old leaf kept]
    D -->|Yes| F[New leaf issued under new root]
    E --> G[Old TTL counts down]
    G --> H[Pair breaks when peer no longer trusts old root]
    F --> I[Pair survives]
    H --> J[Failures spread pair by pair as each leaf expires]

Common causes

Quick checks

All read-only and safe to run during an active incident. Run against a Consul server unless noted.

# Active and inactive CA roots, with expiry
curl -s http://127.0.0.1:8500/v1/connect/ca/roots | jq '.Roots[] | {ID, Active, NotAfter}'

# Which root the leader considers active
curl -s http://127.0.0.1:8500/v1/connect/ca/roots | jq '.ActiveRootID'

# CA configuration in effect (provider, key type, leaf TTL)
curl -s http://127.0.0.1:8500/v1/connect/ca/configuration | jq

# Connect CA telemetry: issuance errors and rates
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep consul_connect_ca

# Leaf cert validity as seen by a sidecar's Envoy admin port (run on the failing node)
curl -s http://127.0.0.1:19000/certs | jq '.certificates[].cert_chain[].expiration_time'

# Server logs for cert, CA, CSR, and signing errors
journalctl -u consul --since '4 hours ago' | grep -iE 'cert|ca |csr|sign|rotate'

# If Vault is the CA backend, verify reachability from a Consul server's perspective.
# Set VAULT_ADDR to the endpoint Consul is configured to use.
VAULT_ADDR=<consul's vault endpoint> vault status

The first three give you the state of the rotation. The fourth tells you whether the system is even trying to issue. The fifth tells you what each Envoy sidecar believes about its own cert. The sixth is where the actual error message usually lives.

How to diagnose it

1. Identify the active root and its expiry. From a server, pull /v1/connect/ca/roots. Confirm ActiveRootID points to a root whose NotAfter is in the future. If the active root has already expired, you are in the “root expired” mode: every new issuance will fail until the root is rotated.
2. Check whether both roots coexist. During a clean roll, /v1/connect/ca/roots should show the old root with Active: false alongside the new active root. If only one root is listed and the rotation is recent, cross-signing may have failed or the old root was force-removed. Either way, leafs signed by the missing root will fail as they expire.
3. Verify the backend. If the provider is consul-vault, check Vault health from a Consul server’s perspective (vault status with the right VAULT_ADDR). Verify the Vault token used for PKI is still valid and that the PKI mount is not sealed, deleted, or modified outside of Consul. If the PKI mount has been deleted, recovery may require rebuilding the server set; this is a documented failure mode.
4. Look for CSR errors in the leader logs. journalctl -u consul | grep -iE 'csr|sign|ca ' often surfaces the exact signing failure. Common strings include signature algorithm mismatches, timeouts reaching the backend, and permission denied on the PKI role.
5. Confirm the spread is cert-driven. Cross-reference Envoy access logs for TLS handshake failures with the NotAfter of the leafs reported by /certs. If failures cluster around cert expiry timestamps, the diagnosis is confirmed.
6. Rule out clock skew. Check NTP offset on servers and on the agents reporting failures. A server ahead of its peers can issue certs that other agents consider “not yet valid”; an agent ahead can mark a valid cert as expired. Skew of more than a few minutes is dangerous in this context.

Metrics and signals to monitor

Fixes

Vault backend unreachable

The most common cause. Confirm with vault status (with VAULT_ADDR set to the endpoint Consul uses) run from a Consul server. If Vault is sealed or down, restoring Vault availability is the only path that preserves the existing CA. Existing leafs continue to work until their TTL elapses, so you usually have hours of runway once Vault is healthy again.

If the Vault token has been revoked or expired, rotate it through consul connect ca set-config (or the /v1/connect/ca/configuration API) with a valid token. This is a configuration change against the live CA; stage it during a maintenance window if the mesh is already degraded. Do not edit the agent config file: after the CA is initialized, CA changes through the file are ignored.

Switching from Vault to the built-in CA provider mid-incident is technically possible but disruptive. It changes the root of trust for every service. Treat it as a planned migration, not a 3 a.m. fix.

Old root expired before rollout completed

If the active root has already expired, no new leaf will be accepted. You must rotate to a new root. For the built-in CA, generate a new CA pair with consul tls ca create (consul-agent-ca.pem and consul-agent-ca-key.pem), then apply the generated material through /v1/connect/ca/configuration or consul connect ca set-config; generating files alone does not rotate the running CA. For Vault, the operator must provision a new PKI mount before the existing root expires. Consul does not auto-rotate the Vault PKI root.

Server cannot process CSRs

If the CA itself is healthy but the server cannot sign, the cause is usually Raft pressure or resource exhaustion. Reduce catalog churn, check disk I/O latency on the Raft volume, verify FD usage is not near the limit, and confirm leader stability. CSRs go through Raft, so a stalled commit pipeline stalls issuance. Do not attempt a forced rotation while the leader is unstable.

Clock skew

Bring the offending node back into sync via NTP. Existing “expired” errors will clear once the agent’s clock is correct. If the server has been issuing certs with the wrong clock, those certs may need to be reissued after the clock is corrected.

Cross-signing key type mismatch

If logs show x509: requested SignatureAlgorithm does not match private key type, the old and new roots were created with different key algorithms (for example EC vs RSA), and cross-signing cannot bridge them. Set PrivateKeyType and PrivateKeyBits explicitly in the CA config and ensure they match between rotations. Consul 1.11.0 made edits to the built-in CA key type/bits regenerate roots, and required a new Vault mount when the Vault provider key type/bits changed; before that release, key changes could leave the existing root in place. This class of error has been reported on Vault-backed providers as well, where the underlying limitation was on the Vault side.

ForceWithoutCrossSigning forces a rotation without the cross-signing bridge. This is disruptive: use it only when you can guarantee every agent has already observed the new root, otherwise stragglers will lose trust and fail.

Prevention

• Monitor CA root expiry as a first-class signal. Alert at 25% of lifetime remaining, page at 7 days. Treat root CA expiry as a business continuity event with weeks of runway.
• Track leaf renewal success rate, not just expiry time. Expiry time without renewal tracking is a countdown clock with no early warning. Renewal failures precede expiries by hours.
• For Vault CA, alert on Vault independently. Vault outages become Connect outages at leaf TTL. The Consul UI can show green even when mTLS is failing because no new issuance is happening.
• Pin CA key type and bits. Explicit PrivateKeyType and PrivateKeyBits prevent cross-signing mismatches across rotations.
• Rehearse rotations in non-prod. Test the full root rotation including cross-signing before doing it in production. Many rotation bugs only surface on the second or later rotation.
• Plan for multi-DC propagation. Secondary datacenters pick up CA changes asynchronously. Do not retire the old root until secondaries have confirmed the new one.
• Manage CA config via API only. Changes to ca_provider in the agent config file are ignored after initialization. All CA changes must go through the /v1/connect/ca/configuration API or consul connect ca set-config.

How Netdata helps

• Per-second consul_connect_ca telemetry surfaces issuance errors before leaves start expiring, which pure expiry monitoring misses.
• Pairing CA issuance errors with Envoy-side TLS handshake metrics isolates which service pairs are failing first, confirming the cert-driven spread pattern.
• Correlating CA errors with separate Vault backend health checks localizes the cause to the CA infrastructure rather than Consul itself.
• Tracking NTP offset alongside cert expiry timelines distinguishes clock skew from real expiry, the most misdiagnosed variant of this incident.
• Composite dashboards bring CA root status, Raft commit time, and leader stability together, so a rotation stalled by server load is not confused with a Consul Raft problem.
• ML anomaly detection on the leaf renewal rate flags stalls hours before the 72-hour cliff, when there is still time to fix the backend.

Related guides

• Consul gossip encryption key mismatch: a botched keyring rotation splits the pool
• Consul client rpc failed: agents alive but the catalog is going stale
• Consul anti-entropy not syncing: local agent state and the catalog drifting apart
• Consul registration storm: catalog churn overwhelming Raft
• Consul goroutine count climbing: the leak behind slow resource exhaustion