When every component in a Pulsar cluster uses TLS (brokers, bookies, ZooKeeper, proxies, clients), a single expired certificate causes immediate handshake failures. Existing connections may persist briefly, but every new connection attempt fails the TLS handshake. Producers cannot publish. Consumers cannot subscribe. Brokers cannot reach bookies. Brokers cannot reach ZooKeeper.

Pulsar does not expose a metric for certificate expiry. There is no pulsar_cert_days_until_expiry gauge. The first signals you see are indirect: authentication failures spike, connections drop, and SSL handshake exceptions fill the logs. By that point, the outage is already happening. The only reliable defense is external certificate expiry monitoring that alerts you days or weeks before the cert becomes invalid.

This guide covers how to check certificate expiry across all Pulsar components, identify which certificate caused an outage, recover, and build monitoring that prevents the next one.

What this means

Apache Pulsar uses TLS for two distinct purposes: transport encryption (between clients and brokers, and between brokers and bookies) and authentication (mTLS, where the client certificate identity is used as the principal). When a certificate expires, both functions break simultaneously.

The broker’s TLS listener on port 6651 (the Pulsar binary protocol TLS port) rejects any connection whose certificate chain is invalid, including expired certificates on either side of the handshake. The broker’s web service TLS port (typically 8081) does the same for admin API connections. Internally, broker-to-bookie and broker-to-ZooKeeper TLS connections fail the same way.

The broker has a configuration setting, tlsCertRefreshCheckDurationSec (default 300 seconds), that controls how often it checks for updated certificate files on disk. This means the broker can pick up rotated certificate files without a restart, but only for its own server certificate. Components with known certificate caching bugs (detailed below) do not reload even when the file on disk changes.

flowchart TD
    A["TLS cert expires"] --> B["Broker-to-bookie handshake fails"]
    A --> C["Client-to-broker handshake fails"]
    A --> D["Broker-to-ZK handshake fails"]
    B --> E["Writes cannot be acknowledged"]
    C --> F["Producers and consumers disconnected"]
    D --> G["Broker loses session and topic ownership"]
    E --> H["Total cluster outage"]
    F --> H
    G --> H

The cascade is fast. Once the broker cannot establish TLS connections to ZooKeeper, its session expires, it loses topic ownership, and all topics on that broker become unavailable. If the expired certificate is on a bookie, brokers cannot write new entries, and pending writes time out.

Common causes

CauseWhat it looks likeFirst thing to check
Broker server certificate expiredAll TLS clients fail to connect; pulsar_authentication_failures_total spikes with TLS auth method labelopenssl s_client -connect broker-host:6651 and inspect the returned certificate dates
Bookie certificate expiredBroker logs show SSLHandshakeException when writing to bookies; publish latency spikes then writes failCheck the bookie’s certificate file expiry date directly
ZooKeeper certificate expiredBrokers lose ZK sessions; bundle ownership operations fail; broker logs show connection errors to ZKCheck ZK certificate files on disk
Client certificate expiredSpecific client or application fleet fails to authenticate; auth failures from known client IPsCheck the client certificate or its issuing CA chain
Stale cached certificate after rotationComponent continues using old cert even after files were replaced on disk; requires process restartCompare process start time against cert file modification time

Quick checks

All of these commands are read-only. None modify certificates, restart processes, or change configuration.

# Check broker certificate expiration via TLS handshake
openssl s_client -connect <broker-host>:6651 2>/dev/null | openssl x509 -enddate -noout

# Check broker web service TLS port
openssl s_client -connect <broker-host>:8081 2>/dev/null | openssl x509 -enddate -noout

# Check a certificate file directly (broker)
openssl x509 -enddate -noout -in /path/to/broker-cert.pem

# Check a certificate file directly (bookie)
openssl x509 -enddate -noout -in /path/to/bookie-cert.pem

# Check a certificate file directly (ZooKeeper)
openssl x509 -enddate -noout -in /path/to/zk-cert.pem

# Calculate days until expiration from a file (GNU date / Linux)
echo "($(date -d "$(openssl x509 -enddate -noout -in /path/to/cert.pem | cut -d= -f2)" +%s) - $(date +%s)) / 86400" | bc

# Check authentication failure rate for TLS-specific spikes
# Adjust scheme/port: HTTPS on 8081 if web service TLS is enabled, HTTP on 8080 otherwise
curl -sk https://<broker-host>:8081/metrics | grep pulsar_authentication_failures_total

# Search broker logs for TLS handshake errors
grep -i "SSLHandshakeException\|certificate" /var/log/pulsar/broker.log | tail -20

# Search bookie logs for TLS errors
grep -i "SSLHandshakeException\|certificate" /var/log/pulsar/bookkeeper-server.log | tail -20

How to diagnose it

  1. Check if the outage is certificate-related. If you have a sudden cluster-wide TLS failure with no recent configuration changes, check certificate expiry first. Run openssl s_client against the broker’s TLS port. If the handshake fails entirely (no certificate returned), the broker may have a corrupted or missing keystore. If the certificate is returned but shows an expired notAfter date, the server certificate has expired.

  2. Identify which component’s certificate expired. Check all certificate files across the cluster: broker, bookie, ZooKeeper, and proxy. The expired certificate may not be the broker’s. If broker-to-bookie writes are failing but client-to-broker connections work, the bookie certificate is the likely culprit.

  3. Check for known certificate caching bugs. Two well-documented issues cause certificates to not reload after rotation:

    • Bookie client TLS cert caching (GitHub issue #6010): The bookkeeper client caches TLS certificates when it first connects to a bookie. If the client certificate expires and the broker disconnects, the broker cannot reconnect to the bookie until the broker process is restarted. The error is javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem.
    • ZooKeeper certificate reload (GitHub issue #359): When using the Pulsar Helm chart with cert-manager, ZooKeeper continues using the old expired certificate after rotation until the ZooKeeper pod is restarted. A fix (PR #613) switches ZooKeeper to use PEM files directly as keystore and truststore, eliminating the PEM-to-JKS conversion step that prevented reload.

  4. Correlate with authentication failure metrics. Check pulsar_authentication_failures_total for spikes. The auth_method and reason labels help distinguish TLS certificate failures from other authentication problems (expired JWT tokens, wrong credentials). TLS cert expiry typically shows a sustained spike across all connections from affected clients.

  5. Verify the scope of impact. Determine whether the failure affects all components or a subset. If only broker-to-bookie connections fail but client connections work, the scope is narrower and recovery is faster.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Certificate expiry date (external check)The only reliable way to detect expiry before handshakes break. Pulsar has no built-in metric for this.Less than 30 days remaining
pulsar_authentication_failures_totalCorrelates directly with cert expiry. Spikes when certificates expire and handshakes fail.Sudden sustained spike, especially with TLS auth method label
Active connection count dropsMass disconnections occur when TLS handshakes fail on reconnection.Sudden drop in active connection gauge
SSL handshake errors in logsDirect evidence of certificate validation failure.SSLHandshakeException, General OpenSslEngine problem, or certificate expired in broker or bookie logs
Broker publish latency and write errorsSecondary indicator: if broker cannot reach bookies due to TLS, writes fail and latency spikes before timing out.Sustained latency increase followed by write errors

Fixes

Rotate the expired certificate

Replace the expired certificate file on disk with a valid one. For PEM-based configurations, this means replacing the certificate and key files referenced in the broker, bookie, or ZooKeeper configuration.

If you are using cert-manager or a similar automated rotation tool, the file may already be updated on disk. The problem is that some components do not pick up the new file without a restart.

Restart components with certificate caching bugs

WARNING: Both procedures below are disruptive. Coordinate during a maintenance window or follow a controlled rolling restart pattern.

Bookie client caching (broker restart required): If the broker cannot reconnect to bookies after certificate rotation due to the bookkeeper client caching bug, you must restart the affected broker. Restarting a broker triggers bundle unloads and client reconnections. Restart one broker at a time in a multi-broker cluster, verifying that bundles redistribute and clients reconnect before proceeding to the next.

ZooKeeper (pod restart required on Helm deployments): If ZooKeeper is using the old certificate after rotation, restart the ZK pods. This is high-risk: losing ZK quorum destabilizes the entire cluster. Restart ZK nodes one at a time, waiting for each to rejoin the ensemble and confirm a healthy quorum before restarting the next.

Verify recovery

After rotation and any necessary restarts, verify that TLS handshakes succeed:

# Verify broker TLS handshake succeeds and returns a valid (non-expired) cert
openssl s_client -connect <broker-host>:6651 2>/dev/null | openssl x509 -enddate -noout

# Verify authentication failures have stopped
curl -sk https://<broker-host>:8081/metrics | grep pulsar_authentication_failures_total

# Verify client connections are re-established
curl -sk https://<broker-host>:8081/metrics | grep pulsar_connections

Prevention

Certificate expiry is entirely preventable with external monitoring. The key principle: certificate expiry is not a Pulsar metric. You must monitor it outside the Pulsar metrics surface.

Alert thresholds:

  • TICKET at 30 days remaining. Gives you time to plan rotation, coordinate maintenance windows, and handle unexpected issues.
  • TICKET with escalated urgency at 7 days remaining. Rotation must be in progress or scheduled within hours.
  • PAGE only on actual handshake failures caused by an already-expired certificate. Do not PAGE on future expiry risk. A future expiry is a planning concern, not an active operational fault.

Check all certificates, not just the broker:

  • Broker server certificates (TLS port 6651, web service TLS port 8081)
  • Bookie certificates
  • ZooKeeper certificates
  • Proxy certificates (if using Pulsar proxy)
  • Client certificates (for mTLS deployments)
  • CA certificates (intermediate and root)

Automate rotation. Manual rotation is where mistakes happen. Use cert-manager on Kubernetes, or a configuration management tool that replaces certificate files and triggers process reloads. If a component requires a restart after rotation (due to the known caching bugs), the automation must include that restart step.

Test rotation in non-production first. Verify that every component picks up the new certificate. Specifically test: (1) broker server cert rotation, (2) bookie cert rotation, (3) ZooKeeper cert rotation. Document which components require restarts and build that into the rotation runbook.

How Netdata helps

  • Certificate file expiry monitoring. Netdata can monitor certificate files on disk and alert on approaching expiry dates, covering the gap where Pulsar provides no built-in metric. Check broker, bookie, and ZooKeeper certificates independently.
  • Authentication failure correlation. Netdata collects pulsar_authentication_failures_total with label breakdowns. Correlating a spike in TLS auth failures with certificate expiry timestamps confirms the root cause in seconds rather than digging through logs.
  • Connection count tracking. A sudden drop in active connections alongside an auth failure spike is the signature pattern of a TLS certificate expiry event. Netdata’s per-second granularity captures the exact moment of failure.
  • Process restart detection. When you rotate certificates and restart components, Netdata shows process restarts and the subsequent recovery of metrics, confirming the fix took effect.
  • Multi-component visibility. Netdata monitors broker, bookie, and ZooKeeper metrics in a single view, so you can see the full scope of a TLS expiry cascade across all components simultaneously.