RabbitMQ TLS certificate expiry: expired certs and handshake failures that lock everyone out

Every client that connects to your RabbitMQ cluster over TLS fails at the same time. No memory alarm, no disk alarm, no queue buildup, no crash. The broker process is running, the management API may even respond, but every AMQP connection attempt dies during the TLS handshake. If inter-node (Erlang distribution) TLS is also enabled, cluster nodes cannot talk to each other and CLI tools may fail to connect as well.

This is what an expired TLS certificate looks like in RabbitMQ. Once every client uses TLS, expiry is a total outage, and it is silent in a specific way: the broker’s metrics stay green because the failure happens before any AMQP connection is established. Nothing ever becomes a connection, so connection counts, churn rates, and message rates simply drop. There is no “TLS handshake failed” counter to alert on.

The other trap: certificate expiry is not exposed as a broker metric. It is not in the Management API node stats and not in the Prometheus endpoint. If your monitoring only scrapes RabbitMQ’s own metrics, you get zero warning. The signals that do exist live in the certificate files on disk, a dedicated health check command, and the RabbitMQ log.

What this means

TLS termination happens before AMQP. A client opens a TCP connection to port 5671, the TLS handshake runs, and only then does AMQP negotiation begin. When the broker’s certificate is expired (or the chain is broken, or peer verification fails), the handshake aborts and the TCP connection closes. From RabbitMQ’s perspective there was never an AMQP connection: no connection process, no channel process, nothing in object_totals, nothing in /api/connections.

The same applies to inter-node traffic when Erlang distribution is secured with TLS on port 25672. An expired inter-node certificate means nodes cannot form or maintain the cluster link, and CLI tools that connect over distribution fail too. The playbook’s guide on Erlang distribution port saturation covers the healthy-state behavior of that link; an expired cert takes it to zero in a different way.

The only places the failure is visible:

  • The client side, as TLS errors in application logs.
  • The RabbitMQ log, as TLS alert entries such as TLS server generated SERVER ALERT: Fatal - Certificate Expired.
  • The certificate files themselves, which you can inspect directly.
flowchart LR
  C[AMQP client] -->|TCP connect 5671| H[TLS handshake]
  H -->|cert valid| A[AMQP negotiation - visible in metrics]
  H -->|cert expired / verify fails| X[handshake aborted - log entry only]
  N2[Cluster peer] -->|TCP connect 25672| H2[Distribution TLS handshake]
  H2 -->|cert expired| X2[cluster link down - log entry only]

Common causes

CauseWhat it looks likeFirst thing to check
Broker server certificate expiredAll TLS clients fail simultaneously; Certificate Expired alerts in the broker logopenssl x509 -enddate on the server cert
Inter-node (Erlang distribution) certificate expiredNodes cannot cluster; CLI tools fail; partition-like symptoms without a network faultExpiry on the distribution cert; distribution handshake errors in the log
Intermediate or CA certificate in the chain expiredHandshakes fail even though the leaf cert looks validInspect every cert in the chain file, not just the first
Peer verification failure after upgradeOutgoing TLS (federation, shovel, LDAP) fails after moving to Erlang 26+Whether the peer’s chain verifies under the new defaults
Unreadable cert or key filesTLS listener hangs or fails after cert replacementOwnership and read permission for the rabbitmq user
Verification depth too smallClient certs signed by an intermediate CA are rejectedverify and depth settings vs. your chain length

Quick checks

All read-only. Run on the broker host unless noted.

# 1. Check expiry of the broker's server certificate
openssl x509 -in /etc/rabbitmq/tls/server_cert.pem -noout -enddate

# 2. Inspect every certificate in the chain file, not just the leaf
openssl crl2pkcs7 -nocrl -certfile /etc/rabbitmq/tls/server_cert.pem | \
  openssl pkcs7 -print_certs -noout -text | grep -E "Subject:|Not After"

# 3. Built-in health check: certs expiring within 6 weeks, per TLS listener
rabbitmq-diagnostics check_certificate_expiration --unit weeks --within 6

# 4. Show the certificates presented by every TLS listener
rabbitmq-diagnostics certificates

# 5. Look for handshake failures in the broker log
grep -i "tls\|ssl\|handshake" /var/log/rabbitmq/rabbit@$(hostname).log | tail -20

# 6. Confirm the cert files are readable by the rabbitmq user
ls -l /etc/rabbitmq/tls/
sudo -u rabbitmq head -c 1 /etc/rabbitmq/tls/server_key.pem

# 7. From a client host, test the handshake and see the presented chain
openssl s_client -connect broker.example.com:5671 -servername broker.example.com </dev/null

# 8. HTTP API equivalent of the expiration check (management plugin)
curl -s -u guest:guest "http://localhost:15672/api/health/checks/certificate-expiration/within/30/days" <!-- TODO: verify exact HTTP API endpoint path and response schema -->

Adjust paths to your deployment; /etc/rabbitmq/tls/ is a common convention, not a RabbitMQ default. Your rabbitmq.conf ssl_options.certfile, ssl_options.keyfile, and ssl_options.cacertfile values are the authoritative locations.

How to diagnose it

  1. Confirm the failure is at the TLS layer, not AMQP. From a client host, run openssl s_client -connect <broker>:5671 (step 7 above). If the handshake fails or shows an expired chain, you have your answer before touching the broker. If the handshake succeeds, the problem is elsewhere: authentication, vhost permissions, or plain connectivity.
  2. Check the broker log for the exact alert. An expired server certificate produces a fatal Certificate Expired alert. Verification failures produce different alerts (unknown CA, bad certificate, hostname check failures), which point to chain or trust problems rather than expiry.
  3. Date the certificate files. Run openssl x509 -enddate -noout -in against the server cert, and inspect every certificate in the chain file. An expired intermediate causes the same client-visible failure as an expired leaf, and it is the one most often missed because operators only check the first certificate in the file.
  4. If inter-node TLS is enabled, check the distribution certificate separately. Symptoms: nodes that are individually healthy cannot cluster, rabbitmqctl/rabbitmq-diagnostics commands that go over distribution fail, and the log shows distribution handshake errors. This looks like a network partition but the network is fine. Confirm by checking expiry on the cert configured for Erlang distribution and by testing raw TCP reachability to port 25672 to rule out an actual network fault.
  5. If certs were recently replaced, check permissions. Certificate and key files must be readable by the rabbitmq user. Files copied in as root with restrictive modes are a common cause of TLS failing right after a renewal that should have fixed everything.
  6. If only some clients fail, suspect verification depth and client trust. When client certificates are signed by an intermediate CA, the default verification depth of 1 is not enough and peer verification fails for those clients only. Clients on a different, valid chain keep working, which makes this look partial and confusing.
  7. If outgoing TLS (federation, shovel, LDAP) broke after an upgrade, check Erlang 26 behavior. On Erlang 26 and later (required by RabbitMQ 4.x), outgoing TLS connections verify the peer by default where earlier versions did not. Previously tolerated bad or expired peer certificates on remote endpoints start failing after the upgrade, with no RabbitMQ config change on your side.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Days to certificate expiry (external check)Not a broker metric; must come from cert inspection or the health check< 30 days: ticket. < 7 days: page
rabbitmq-diagnostics check_certificate_expiration resultBuilt-in per-listener check, scriptable in cron/CIAny non-zero exit outside your expected window
TLS handshake errors in the broker logThe only broker-side runtime signal for handshake failureAny Certificate Expired or rising handshake failure rate
AMQP listener reachability (active probe)Metrics cannot see failed handshakes; a TCP/TLS probe canProbe connects TCP but TLS handshake fails
Connection count vs. baselineA cliff-drop with no alarm and no churn is characteristic of TLS-layer failureSudden drop toward zero while the node reports healthy
Connection churn rateClients retrying failed TLS handshakes look like a reconnect stormCreation rate well above baseline with short connection lifetimes

One gotcha on the health check: check_certificate_expiration (and its HTTP API equivalent) inspects all certificates in the configured CA bundle file, not just the listener’s own certificate. If ssl_options.cacertfile points at a system CA bundle that contains an expired root, the check reports failure even when your server certificate is valid. This behavior is documented as intentional in RabbitMQ 4.0.6/4.1.0; if expired roots in the bundle are normal for you, do not rely on this check alone and use openssl x509 -enddate against the specific listener certs instead.

Fixes

Renew and deploy the expired certificate

Issue the replacement from your CA, install the new cert and key at the paths referenced by ssl_options.certfile and ssl_options.keyfile, and make sure the files are readable by the rabbitmq user. Then reload or restart the broker so the listener picks up the new certificate. A restart drops all connections; with clients that reconnect cleanly this is usually acceptable during an outage that is already total, but plan it rather than improvising it. Verify with openssl s_client from a client host before declaring done.

Tradeoff: if your deployment mounts certs from a secrets store or config management, fix the source, not just the file on one node, or the next agent run will revert your emergency fix.

Fix the chain, not just the leaf

If an intermediate expired, replace the full chain file. A renewed leaf served with a stale intermediate still fails verification on clients that do not already have the new intermediate cached.

Restore inter-node TLS

For an expired Erlang distribution certificate, deploy the new cert to every node and restart nodes as required by your distribution TLS setup. Do this one node at a time in a cluster so quorum queues keep a majority, and confirm cluster membership after each node returns. If CLI tools also use distribution TLS, update their certificate configuration too.

If Erlang 26’s verify-by-default change broke outgoing federation, shovel, or LDAP connections, the correct fix is to make the remote endpoint present a valid, verifiable chain. Explicitly disabling verification restores the old behavior but also removes the protection; treat it as a temporary bridge, not the fix.

Prevention

  • Alert on expiry, not on failure. External check (cert inspection or the health check) on a schedule: ticket at 30 days, page at 7 days. This severity guidance exists because the failure mode itself gives you zero warning time.
  • Monitor every TLS surface separately. Client-facing listeners, the management UI, and Erlang distribution can use different certificates with different expiry dates. One expiring does not imply the others are fine.
  • Probe the AMQP TLS listener actively. A TCP connect is not enough; the probe must complete a TLS handshake. This is the only check that sees what clients see.
  • Log-scan for handshake failures. TLS errors appear only in the RabbitMQ log. Ship or scan it so a rising failure rate (a cert approaching expiry on one node, a client with a stale trust store) is visible before it becomes universal.
  • Track cert deployment as a change event. Most TLS incidents that are not pure expiry are botched renewals: wrong chain, wrong permissions, reverted files. Correlate TLS errors with your config management runs.
  • Rehearse the renewal. Know which config keys point at which files, whether your version reloads certs without a restart, and how clients behave during the cutover.

How Netdata helps

  • Connection count and churn correlation. Netdata charts connections, channels, and churn rates per second. The TLS-expiry signature is distinctive: connections cliff to zero while churn shows a wall of short-lived failed attempts and no resource alarm fires. That combination separates TLS failure from a connection storm or a resource-alarm publisher block at a glance.
  • Alarms as an exclusion filter. Because mem_alarm and disk_free_alarm are charted alongside connection state, you can rule out the usual suspects in one view and get to “check the certs” faster.
  • Cluster link visibility. Per-node cluster link traffic dropping to zero, correlated with nodes reporting healthy, points at distribution-layer failure (expired inter-node cert or network fault) rather than a crash.
  • Log and probe signals alongside broker metrics. Because certificate expiry and handshake failures are not broker metrics, pair Netdata’s RabbitMQ collector with an external expiry check and an active TLS probe so the warning exists before the outage does.