NATS TLS certificate expiry: an expired cert that locks out every new connection

Every new client connection to your NATS server is failing at the TLS handshake with a verification error, typically x509: certificate has expired or is not yet valid. Clients that were already connected still work. That is what makes this failure mode deceptive: the server passes every process-level check while refusing all new work.

If you run mutual TLS on cluster routes and gateways, the blast radius is bigger. Route and gateway connections also fail to establish, so servers that restart or reconnect after a network event cannot rejoin the cluster. A certificate that expires during an unrelated incident can turn a recoverable blip into a full partition.

This is a cliff-edge failure with a long, visible runway. The expiry timestamp is knowable months in advance. If you got here without warning, the real bug is in your renewal automation or your monitoring, and both need fixing after the fire is out.

What this means

The NATS server presents a TLS certificate during every handshake on TLS-enabled listeners. Clients validate that certificate against their trust store and the current time. Once the certificate’s notAfter timestamp passes, validation fails and the handshake is aborted. The NATS protocol exchange never happens, so this is not an authentication failure, a permissions problem, or a connection limit. The TLS layer itself refuses to proceed.

Three properties matter for diagnosis:

• Existing connections survive. TLS is validated at handshake time, not continuously. Already-connected clients, routes, and gateways keep working until they disconnect for any other reason.
• Every reconnect becomes a failure. Any client that drops and re-establishes hits the expired cert. Natural connection churn drains your connected population toward zero.
• It looks different depending on where you stand. From the server, the process is healthy and throughput on surviving connections is normal. From a new client, the server is dead.

flowchart TD
  A[Server cert expires] --> B[New client TLS handshakes fail]
  A --> C[Route and gateway handshakes fail with mTLS]
  B --> D[Reconnecting clients locked out]
  C --> E[Restarted or partitioned servers cannot rejoin cluster]
  D --> F[Connected population drains toward zero over time]
  E --> F
  F --> G[Full outage despite healthy server process]

Common causes

Quick checks

# 1. Check the expiry timestamp the server reports, if your version exposes it
curl -s http://localhost:8222/varz | jq .tls_cert_not_after

# 2. Check the cert file on disk directly
openssl x509 -enddate -noout -in /path/to/server-cert.pem

# 3. Check the cert the server actually presents to clients
echo | openssl s_client -connect localhost:4222 2>/dev/null | openssl x509 -noout -enddate

# 4. Compute days remaining from the expiry timestamp
expiry=$(curl -s http://localhost:8222/varz | jq -r .tls_cert_not_after)
echo $(( ($(date -d "$expiry" +%s) - $(date +%s)) / 86400 )) days remaining

# 5. Check server uptime: has it restarted since the cert was renewed?
curl -s http://localhost:8222/varz | jq .uptime

# 6. Watch the connection population drain
curl -s http://localhost:8222/varz | jq '{active: .connections, total: .total_connections}'

# 7. In a cluster, check whether routes are intact
curl -s http://localhost:8222/routez | jq '.routes | length'

Checks 1 and 3 answer different questions, and the difference is the whole game: /varz tells you what the server believes about its configured cert, while s_client tells you what clients actually receive. If they disagree, the file was renewed but the running server never picked it up. Check 6 distinguishes expiry from a connection storm: here connections falls or stays flat while total_connections keeps climbing from failed reconnect attempts. Check 7 matters because a cluster can look fine right up until the first server restarts.

How to diagnose it

1. Confirm the server cert is actually expired. Compare the served cert’s notAfter against the current time using the openssl checks above, or the /varz field if your version exposes it.

2. Determine the scope: one client or everyone. If a single service cannot connect while others connect fine, suspect that client’s certificate, not the server’s. In mutual TLS setups, an expired client cert produces a TLS handshake verification error that is easy to misread as an authentication failure. The giveaway: exactly one identity is affected, and the server-side cert checks all pass.

3. Compare disk state to served state. If the cert file on disk is valid but the listener still presents an expired cert, the renewal automation worked and the reload did not. Check server uptime: if the server has been running since before the renewal, it is still holding the old cert in memory.

4. Check whether a reload was even attempted. Renewing the file on disk does nothing by itself. If you use cert-manager, ACME, or a configuration management system to renew certs, verify that the renewal actually triggered a reload signal to nats-server. A common gap: the secret or file updates, nothing signals the server, and the server keeps serving the in-memory cert until restart.

5. Verify every listener type after any reload. NATS supports reloading TLS configuration via SIGHUP, but cert reload behaviour for route and gateway connections varies across versions. After a reload, test the client port, the route port (from a peer’s perspective), and any gateway connections before declaring the problem fixed.

6. In a cluster, assess route and gateway damage. Check route counts against the expected N-1 mesh. If mTLS is in play and the cert has been expired for a while, any route that dropped during the expiry window could not re-establish. Treat restored routes as part of the recovery checklist, not an afterthought.

7. Confirm recovery after the fix. Once a valid cert is being served, watch connections climb back toward baseline and total_connections churn settle. Verify the served cert now shows the new expiry.

Metrics and signals to monitor

Fixes

Renew the certificate and get the server to serve it

Renew the cert through whatever PKI issued it, then get it loaded. The reliable path is a restart; the graceful path is a config reload (SIGHUP), with a caveat: reload behaviour for route and gateway listeners varies across versions, so a reload that fixes the client port may silently leave peers locked out. In a cluster, roll the restart one server at a time so clients and routes can fail over while you work. A restart is disruptive, but serving a valid cert from a restarted server beats serving an expired cert from an uptime-proud one.

Repair the renewal-to-reload pipeline

If the cert was renewed on disk but never loaded, the fix is not the cert, it is the pipeline. Whatever renews the certificate must also signal nats-server to reload (or restart it), and that signal must be verified end to end. If you run cert-manager or ACME on Kubernetes, do not assume a rotated secret reaches the running process. Some deployment charts ship a reloader sidecar for exactly this gap; verify what your deployment actually does after a renewal.

Fix the one-client case

If the diagnosis narrowed the failure to a single client’s certificate, renew and redistribute that client’s cert. Nothing on the server needs to change. Watch for recurrence: clients provisioned at the same time from the same CA tend to expire together, so one expiring client cert often predicts a wave.

Restore cluster connectivity

After the server certs are valid, verify routes and gateways re-established. Routes reconnect automatically, but “should have reconnected” and “did reconnect” are different observations. Check /routez on every server and /gatewayz on supercluster hubs, and only then close the incident.

Prevention

• Alert on cert expiry at 30 days, page inside 7. The 30-day threshold gives you a full renewal cycle of slack; 7 days means your automation already failed and a human needs to intervene. Where the server exposes expiry via /varz, alert on it directly; otherwise probe the listeners externally.
• Pair server monitoring with external certificate monitoring. Independent probing of the TLS listeners catches the case where the server is misconfigured about its own cert, and covers client certs the server never sees. On versions that expose no expiry field, external checks are your only coverage.
• Test the reload path before you depend on it. On a staging server, renew a cert, send SIGHUP, and verify the client port, route port, and gateway connections all present the new cert. If your version does not reload route/gateway certs cleanly, your runbook should say “restart” instead of “reload.”
• Rehearse CA and intermediate rotations. When you rotate an issuing CA, update trust anchors on every peer and client before the old chain stops validating, and prefer bundles that carry current and future issuers over single-intermediate files.
• Track client cert expiry as a fleet. Client certs in mTLS setups expire independently and each one looks like an isolated auth failure. Inventory them with their expiry dates so they rotate as a batch, not as a series of surprises.

How Netdata helps

• Netdata’s NATS collector polls the HTTP monitoring endpoints, so you see the blast radius of an expiry event as it unfolds: active connections draining while total_connections churn climbs, route counts dropping below the mesh, and uptime resetting during recovery restarts.
• Correlating connection churn against throughput separates cert expiry from a genuine connection storm: in expiry, the reconnect attempts never succeed, so the active population trends down instead of oscillating.
• Per-second visibility into routes and gateways confirms whether cluster links re-established after the cert was fixed, which is the step most often skipped during incident closure.
• If the collector does not expose cert expiry for your nats-server version, treat expiry tracking as a gap to close with external certificate monitoring alongside Netdata rather than expecting it on the NATS dashboard.

Related guides

• NATS Authorization Violation: authentication failures and credential rotation
• NATS connection churn: a stable connection count hiding constant reconnects
• NATS connection storm: reconnect thundering herd after a network event
• NATS route RTT high: inter-server latency that triggers Raft elections
• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS file descriptor exhaustion: too many open files and the ulimit cliff