Apache Pulsar TLS certificate expiry: the silent, total outage no metric warns you about

When every component in a Pulsar cluster uses TLS (brokers, bookies, ZooKeeper, proxies, clients), a single expired certificate causes immediate handshake failures. Existing connections may persist briefly, but every new connection attempt fails the TLS handshake. Producers cannot publish. Consumers cannot subscribe. Brokers cannot reach bookies. Brokers cannot reach ZooKeeper.

Pulsar does not expose a metric for certificate expiry. There is no pulsar_cert_days_until_expiry gauge. The first signals you see are indirect: authentication failures spike, connections drop, and SSL handshake exceptions fill the logs. By that point, the outage is already happening. The only reliable defense is external certificate expiry monitoring that alerts you days or weeks before the cert becomes invalid.

This guide covers how to check certificate expiry across all Pulsar components, identify which certificate caused an outage, recover, and build monitoring that prevents the next one.

What this means

Apache Pulsar uses TLS for two distinct purposes: transport encryption (between clients and brokers, and between brokers and bookies) and authentication (mTLS, where the client certificate identity is used as the principal). When a certificate expires, both functions break simultaneously.

The broker’s TLS listener on port 6651 (the Pulsar binary protocol TLS port) rejects any connection whose certificate chain is invalid, including expired certificates on either side of the handshake. The broker’s web service TLS port (typically 8081) does the same for admin API connections. Internally, broker-to-bookie and broker-to-ZooKeeper TLS connections fail the same way.

The broker has a configuration setting, tlsCertRefreshCheckDurationSec (default 300 seconds), that controls how often it checks for updated certificate files on disk. This means the broker can pick up rotated certificate files without a restart, but only for its own server certificate. Components with known certificate caching bugs (detailed below) do not reload even when the file on disk changes.

flowchart TD
    A["TLS cert expires"] --> B["Broker-to-bookie handshake fails"]
    A --> C["Client-to-broker handshake fails"]
    A --> D["Broker-to-ZK handshake fails"]
    B --> E["Writes cannot be acknowledged"]
    C --> F["Producers and consumers disconnected"]
    D --> G["Broker loses session and topic ownership"]
    E --> H["Total cluster outage"]
    F --> H
    G --> H

The cascade is fast. Once the broker cannot establish TLS connections to ZooKeeper, its session expires, it loses topic ownership, and all topics on that broker become unavailable. If the expired certificate is on a bookie, brokers cannot write new entries, and pending writes time out.

Common causes

Quick checks

All of these commands are read-only. None modify certificates, restart processes, or change configuration.

# Check broker certificate expiration via TLS handshake
openssl s_client -connect <broker-host>:6651 2>/dev/null | openssl x509 -enddate -noout

# Check broker web service TLS port
openssl s_client -connect <broker-host>:8081 2>/dev/null | openssl x509 -enddate -noout

# Check a certificate file directly (broker)
openssl x509 -enddate -noout -in /path/to/broker-cert.pem

# Check a certificate file directly (bookie)
openssl x509 -enddate -noout -in /path/to/bookie-cert.pem

# Check a certificate file directly (ZooKeeper)
openssl x509 -enddate -noout -in /path/to/zk-cert.pem

# Calculate days until expiration from a file (GNU date / Linux)
echo "($(date -d "$(openssl x509 -enddate -noout -in /path/to/cert.pem | cut -d= -f2)" +%s) - $(date +%s)) / 86400" | bc

# Check authentication failure rate for TLS-specific spikes
# Adjust scheme/port: HTTPS on 8081 if web service TLS is enabled, HTTP on 8080 otherwise
curl -sk https://<broker-host>:8081/metrics | grep pulsar_authentication_failures_total

# Search broker logs for TLS handshake errors
grep -i "SSLHandshakeException\|certificate" /var/log/pulsar/broker.log | tail -20

# Search bookie logs for TLS errors
grep -i "SSLHandshakeException\|certificate" /var/log/pulsar/bookkeeper-server.log | tail -20

How to diagnose it

1. Check if the outage is certificate-related. If you have a sudden cluster-wide TLS failure with no recent configuration changes, check certificate expiry first. Run openssl s_client against the broker’s TLS port. If the handshake fails entirely (no certificate returned), the broker may have a corrupted or missing keystore. If the certificate is returned but shows an expired notAfter date, the server certificate has expired.

2. Identify which component’s certificate expired. Check all certificate files across the cluster: broker, bookie, ZooKeeper, and proxy. The expired certificate may not be the broker’s. If broker-to-bookie writes are failing but client-to-broker connections work, the bookie certificate is the likely culprit.

3. Check for known certificate caching bugs. Two well-documented issues cause certificates to not reload after rotation:

   • Bookie client TLS cert caching (GitHub issue #6010): The bookkeeper client caches TLS certificates when it first connects to a bookie. If the client certificate expires and the broker disconnects, the broker cannot reconnect to the bookie until the broker process is restarted. The error is javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem. This bug was fixed upstream: BookKeeper added client-side certificate auto-refresh (apache/bookkeeper#2235), and modern Pulsar enables it by default via bookkeeperTlsCertFilesRefreshDurationSeconds (default 300). The restart workaround applies only to older versions.
   • ZooKeeper certificate reload (GitHub issue #359): When using the Pulsar Helm chart with cert-manager, ZooKeeper continues using the old expired certificate after rotation until the ZooKeeper pod is restarted. A fix (PR #613) switches ZooKeeper to use PEM files directly as keystore and truststore, eliminating the PEM-to-JKS conversion step that prevented reload. The fix shipped in pulsar-helm-chart 4.2.0 (August 2025), not in the Pulsar broker itself.

4. Correlate with authentication failure metrics. Check pulsar_authentication_failures_total for spikes. The auth_method and reason labels help distinguish TLS certificate failures from other authentication problems (expired JWT tokens, wrong credentials). TLS cert expiry typically shows a sustained spike across all connections from affected clients.

5. Verify the scope of impact. Determine whether the failure affects all components or a subset. If only broker-to-bookie connections fail but client connections work, the scope is narrower and recovery is faster.

Metrics and signals to monitor

Fixes

Rotate the expired certificate

Replace the expired certificate file on disk with a valid one. For PEM-based configurations, this means replacing the certificate and key files referenced in the broker, bookie, or ZooKeeper configuration.

If you are using cert-manager or a similar automated rotation tool, the file may already be updated on disk. The problem is that some components do not pick up the new file without a restart.

Restart components with certificate caching bugs

WARNING: Both procedures below are disruptive. Coordinate during a maintenance window or follow a controlled rolling restart pattern.

Bookie client caching (broker restart required): If the broker cannot reconnect to bookies after certificate rotation due to the bookkeeper client caching bug, you must restart the affected broker. Restarting a broker triggers bundle unloads and client reconnections. Restart one broker at a time in a multi-broker cluster, verifying that bundles redistribute and clients reconnect before proceeding to the next.

ZooKeeper (pod restart required on Helm deployments): If ZooKeeper is using the old certificate after rotation, restart the ZK pods. This is high-risk: losing ZK quorum destabilizes the entire cluster. Restart ZK nodes one at a time, waiting for each to rejoin the ensemble and confirm a healthy quorum before restarting the next.

Verify recovery

After rotation and any necessary restarts, verify that TLS handshakes succeed:

# Verify broker TLS handshake succeeds and returns a valid (non-expired) cert
openssl s_client -connect <broker-host>:6651 2>/dev/null | openssl x509 -enddate -noout

# Verify authentication failures have stopped
curl -sk https://<broker-host>:8081/metrics | grep pulsar_authentication_failures_total

# Verify client connections are re-established
curl -sk https://<broker-host>:8081/metrics | grep pulsar_active_connections

Prevention

Certificate expiry is entirely preventable with external monitoring. The key principle: certificate expiry is not a Pulsar metric. You must monitor it outside the Pulsar metrics surface.

Alert thresholds:

• TICKET at 30 days remaining. Gives you time to plan rotation, coordinate maintenance windows, and handle unexpected issues.
• TICKET with escalated urgency at 7 days remaining. Rotation must be in progress or scheduled within hours.
• PAGE only on actual handshake failures caused by an already-expired certificate. Do not PAGE on future expiry risk. A future expiry is a planning concern, not an active operational fault.

Check all certificates, not just the broker:

• Broker server certificates (TLS port 6651, web service TLS port 8081)
• Bookie certificates
• ZooKeeper certificates
• Proxy certificates (if using Pulsar proxy)
• Client certificates (for mTLS deployments)
• CA certificates (intermediate and root)

Automate rotation. Manual rotation is where mistakes happen. Use cert-manager on Kubernetes, or a configuration management tool that replaces certificate files and triggers process reloads. If a component requires a restart after rotation (due to the known caching bugs), the automation must include that restart step.

Test rotation in non-production first. Verify that every component picks up the new certificate. Specifically test: (1) broker server cert rotation, (2) bookie cert rotation, (3) ZooKeeper cert rotation. Document which components require restarts and build that into the rotation runbook.

How Netdata helps

• Certificate file expiry monitoring. Netdata can monitor certificate files on disk and alert on approaching expiry dates, covering the gap where Pulsar provides no built-in metric. Check broker, bookie, and ZooKeeper certificates independently.
• Authentication failure correlation. Netdata collects pulsar_authentication_failures_total with label breakdowns. Correlating a spike in TLS auth failures with certificate expiry timestamps confirms the root cause in seconds rather than digging through logs.
• Connection count tracking. A sudden drop in active connections alongside an auth failure spike is the signature pattern of a TLS certificate expiry event. Netdata’s per-second granularity captures the exact moment of failure.
• Process restart detection. When you rotate certificates and restart components, Netdata shows process restarts and the subsequent recovery of metrics, confirming the fix took effect.
• Multi-component visibility. Netdata monitors broker, bookie, and ZooKeeper metrics in a single view, so you can see the full scope of a TLS expiry cascade across all components simultaneously.

Related guides

• Apache Pulsar active connections climbing: connection leaks and file descriptor exhaustion
• Apache Pulsar bookie add-entry queue not draining: writes arriving faster than the disk can commit
• Apache Pulsar AutoRecovery stalled: under-replicated ledgers that never heal
• Apache Pulsar backlog age vs size: the latency depth alone cannot show
• Apache Pulsar backlog quota exceeded: producers held or rejected when consumers stall
• Apache Pulsar bookie disk filling: runway to read-only and how to reclaim space
• Apache Pulsar bookie failure cascade: recovery I/O that topples surviving bookies
• Apache Pulsar bookie read latency high: catch-up reads competing with the write path
• Apache Pulsar bookie read-only: disk full and bookie_SERVER_STATUS at zero
• Apache Pulsar broker down: telling a dead broker from a fenced one
• Apache Pulsar broker GC death spiral: heap pressure, stop-the-world pauses, and lost topic ownership
• Apache Pulsar broker hotspot: one broker owning far more topics than the rest