vCenter certificate expired: the STS signing cert outage nobody saw coming

vCenter is down. Not “slow” or “degraded.” Down. The vSphere Client shows a white screen or a 503. PowerCLI sessions fail to connect. API calls return authentication errors. ESXi hosts show as disconnected in bulk. Every integration that depends on vCenter (NSX, vRA, SRM, backup products) has lost connectivity simultaneously. VMs on the hosts are still running, but you cannot manage, migrate, or orchestrate anything.

You check the browser certificate on the vCenter URL. It looks fine. Months left. You check NTP. Synchronized. You check disk space. Plenty. Nothing in your standard monitoring explains why the entire management plane went dark at once.

The culprit is the STS signing certificate. It is not the certificate you see in the browser. It is an internal certificate used by the Security Token Service to sign SAML tokens for all vCenter authentication. When it expires, STS cannot issue valid tokens. Every authentication attempt fails. Every service that depends on inter-service authentication begins failing. There is no graceful degradation: everything works until the moment of expiry, then everything breaks together.

What this means

The STS signing certificate is the most critical certificate in the vCenter ecosystem and has caused more real-world outages than any other certificate type. It is separate from the machine SSL certificate (the one you see in the browser) and has its own independent lifecycle. VMCA-signed machine certificates default to a 2-year validity period. The STS signing certificate has its own default lifetime that varies by vSphere version.

When the STS signing cert expires, the cascade is immediate and total:

flowchart TD
    A[STS signing cert expires] --> B[STS cannot sign valid SAML tokens]
    B --> C[All authentication fails at once]
    C --> D[vSphere Client: white screen or 503]
    C --> E[PowerCLI and API: auth rejected]
    C --> F[ESXi hosts: bulk disconnect]
    C --> G[Integrations drop: NSX, vRA, SRM, backup]
    B --> H[Inter-service auth fails]
    H --> I[Management plane goes dark]

VMs continue running on ESXi hosts. The hypervisor plane is unaffected. But the management plane is completely inaccessible. You cannot log in, cannot migrate, cannot provision, cannot reconfigure. HA continues to operate on the hosts because the FDM agents run independently of vCenter, but you cannot reconfigure HA or respond to host failures through vCenter.

Common causes

CauseWhat it looks likeFirst thing to check
STS signing cert expiredAll auth fails at once. Browser cert looks fine. vmware-stsd fails to start.checksts.py from VMware KB 79248
Machine SSL cert expiredTLS handshake failure on port 443. Browser shows cert warning.openssl s_client -connect vcenter:443
VMCA root cert expiredCascading failure. Multiple cert types fail simultaneously.vecs-cli TRUSTED_ROOTS store
Post-upgrade renewal missedvCenter was upgraded but cert renewal step was skipped or failed silently.Compare cert creation dates to upgrade date
Failed renewal left some certs expiredRenewal appeared to succeed but one or more stores still hold expired certs.Check every store, not just MACHINE_SSL_CERT
Clock skew causing false validation failuresCerts are not actually expired but validation fails. NTP offset is large.chronyc tracking on VCSA and hosts

Quick checks

These are safe, read-only commands. Run them from an SSH session to the VCSA.

# List all certificate stores in VECS
/usr/lib/vmware-vmafd/bin/vecs-cli store list

# Check machine SSL certificate dates
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT \
  --alias __MACHINE_CERT | openssl x509 -noout -dates -subject

# Check STS-related certificate in VECS
# NOTE: STS_INTERNAL_SSL_CERT is the endpoint TLS cert, which may differ
# from the STS *signing* cert that causes the outage. Use checksts.py
# below for the authoritative signing cert check.
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store STS_INTERNAL_SSL_CERT \
  --alias __MACHINE_CERT 2>/dev/null | openssl x509 -noout -dates

<!-- TODO: verify the correct alias for STS_INTERNAL_SSL_CERT across versions. The __MACHINE_CERT alias may not be correct for this store. -->

# Check the externally-facing certificate
echo | openssl s_client -connect localhost:443 2>/dev/null | \
  openssl x509 -noout -dates -subject

# Check STS signing certificate with checksts.py
# This script is NOT installed by default. Download it from VMware KB 79248
# and upload to the VCSA (e.g., /tmp/checksts.py), then run:
python3 /tmp/checksts.py

# Check service health
service-control --status --all

# Look for certificate errors in vpxd log
grep -i "ssl\|certificate.*expir\|cert.*invalid" /var/log/vmware/vpxd/vpxd.log | tail -50

# Count SSO authentication failures
grep -c "Authentication attempt failed\|LOGIN_FAILED\|AuthenticationFailed" \
  /var/log/vmware/sso/vmware-sts-idmd.log

# Verify NTP synchronization (clock skew causes false cert validation failures)
chronyc tracking

How to diagnose it

  1. Confirm the symptom pattern. If all authentication fails simultaneously (not gradually), the browser cert looks fine, and multiple hosts disconnected at the same time, suspect the STS signing cert. A gradual degradation or single-service failure points elsewhere.

  2. Check the STS signing certificate. Download checksts.py from VMware KB 79248, upload it to the VCSA, and run it. This is the most reliable way to check STS signing certificate expiry. The VECS STS_INTERNAL_SSL_CERT store contains the STS endpoint TLS certificate, which is related but not necessarily the same certificate that signs SAML tokens.

  3. Check NTP synchronization. Clock skew between VCSA and ESXi hosts can cause certificate validation failures even when certificates are not expired. Run chronyc tracking on the VCSA. An offset of more than 5 minutes will break Kerberos authentication and likely SAML token validation. NTP synchronization is a prerequisite for certificate health.

  4. Check service health. Run service-control --status --all. If vmware-stsd (the STS service) is STOPPED or in a crash loop, that confirms the STS layer is the problem. If vpxd is also down, it may be because it cannot authenticate to dependent services.

  5. Check host connection states. If multiple hosts show notResponding simultaneously, the problem is vCenter-side, not host-side. This distinguishes a cert expiry cascade from individual host failures.

  6. Check vpxd logs for certificate errors. Grep for SSL, certificate, and authentication errors. The specific error patterns confirm which certificate is causing the failure.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Certificate expiry dates (all types)Cert expiry is the leading cause of preventable vCenter outages. The STS cert is invisible in the browser.STS cert < 30 days to expiry. VMCA root < 90 days.
SSO authentication failure rateSpikes to 100% when STS cannot issue tokens.Failure rate jumps from near-zero to 100% at the moment of expiry.
vpxd log error patternsCertificate and SSL errors in vpxd.log are the earliest log signal.New certificate.*expir or cert.*invalid entries appearing.
ESXi host connection stateBulk disconnection indicates vCenter-side failure, not host failure.Multiple hosts notResponding simultaneously.
vCenter service healthvmware-stsd STOPPED or crash-looping confirms STS layer failure.STS service not in STARTED/GREEN state.
NTP synchronization offsetClock skew causes false certificate validation failures.Offset > 5 seconds is unusual. > 5 minutes breaks auth.

Fixes

Warning: certificate renewal requires service restarts and will interrupt vCenter management. Take snapshots of all affected nodes before proceeding. Schedule a maintenance window.

STS signing certificate expired

This is the most complex renewal. Follow VMware KB 79248 (76719 for older versions) for the documented procedure to regenerate STS certs. The renewal process is complex enough that 30 days of lead time is barely sufficient.

Do not attempt to renew other certificates first. If the STS cert is expired, certificate regeneration for other cert types will fail because dependent services like vmware-stsd and vmware-vapi-endpoint cannot start without a valid token.

Machine SSL certificate expired

Use /usr/lib/vmware-vmca/bin/certificate-manager to renew. This is simpler than STS renewal but still requires service restarts. Plan for a maintenance window.

VMCA root certificate expired

Renewing the VMCA root requires re-issuing all dependent certificates. This is a major operation, not a quick fix. Begin renewal at 120 days to allow for planning and cascading renewal of all dependent certs.

Post-renewal service restarts

After certificate renewal, some services cache old certificates and require explicit restarts. A “certificate renewed but services still failing” scenario is common. Restart services in dependency order using service-control.

Enhanced Linked Mode considerations

In Enhanced Linked Mode (ELM) topologies, all vCenter/PSC nodes in the SSO domain must be snapshotted before STS cert operations. A single-node snapshot is insufficient. In vSphere 6.x with external PSC, certificates exist on both the PSC and the vCenter. Both must be monitored and renewed.

Prevention

  • Alert at 90 days for VMCA root, 30 days for STS, not 7 days. A 7-day alert means you are already in an emergency. The STS renewal process is complex enough that 30 days is barely sufficient. VMCA root renewal cascades to all dependent certificates and needs 60 to 90 days of lead time.
  • Monitor all certificate types, not just machine SSL. The STS signing certificate is not visible in the browser. It requires specific tooling (vecs-cli, checksts.py) to check. Most teams discover its existence only when it expires and everything breaks.
  • Verify NTP synchronization as a prerequisite. Clock skew between VCSA and ESXi hosts can cause certificate validation failures even when certificates are not expired.
  • Snapshot all nodes in ELM before cert operations. In Enhanced Linked Mode, all vCenter/PSC nodes in the SSO domain must be snapshotted. A single-node snapshot is insufficient.
  • Document the renewal procedure per version. VMware has changed the certificate architecture significantly across versions. The tools and stores differ between 6.5, 6.7, 7.0, and 8.0. Verify which tools work on your specific version before you need them in an emergency.

How Netdata helps

  • Correlate certificate expiry with authentication failures. If you feed certificate expiry dates and SSO authentication failure rates into Netdata, the correlation between approaching STS cert expiry and auth failure spikes is immediately visible on a single dashboard.
  • Per-second service health metrics. Netdata monitors VCSA systemd services (vmware-stsd, vpxd, vmware-vapi-endpoint) at per-second resolution. A service entering a crash loop is visible within seconds, not at the next 5-minute polling interval.
  • NTP offset tracking. Netdata’s chrony collector tracks clock synchronization continuously. Clock skew that could cause false certificate validation failures is caught before it breaks authentication.
  • ESXi host connection state visibility. When host connection states are collected (via SNMP or vSphere API integration), bulk disconnection events with multiple hosts going notResponding simultaneously are immediately distinguishable from individual host failures.
  • Log-based early warning. If you forward vpxd.log to Netdata, certificate and SSL error patterns can surface as metric anomalies before the full authentication failure cascade.