ZooKeeper TLS handshake failures: unsuccessful handshakes and non-mTLS connections

When ZooKeeper is configured for TLS, the TLS layer becomes a new failure surface between clients and the ensemble. A spike in zk_unsuccessful_handshake or zk_tls_handshake_exceeded means clients are attempting TLS connections that never complete. If your environment requires mutual TLS, any non-zero value in zk_non_mtls_remote_conn_count means a client bypassed mTLS entirely.

These failures are noisy in a particular way. The client sees a connection timeout or a refused handshake, while the server logs an “Unsuccessful handshake” entry that does not always explain why. Certificate expiry, cipher mismatch, hostname verification failure, and low host entropy all produce similar surface symptoms. The fix is rarely the server itself; it is usually a client configuration error, an expired credential, or a monitoring tool sending plaintext to a secure port.

What this means

TLS in ZooKeeper is layered on top of the Netty connection factory. When a client connects to the secure client port, ZooKeeper must complete a TLS handshake before it can process any session establishment. If the handshake fails, whether from a bad certificate, a cipher mismatch, or plaintext bytes sent to a TLS port, the connection is dropped before a session is ever created.

The signals to track:

  • zk_unsuccessful_handshake: counter of TLS handshakes that failed to complete. Increments whenever a client connects to the secure port but cannot negotiate TLS successfully.
  • zk_tls_handshake_exceeded: counter of TLS handshakes that exceeded the configured handshake timeout. This is the slow-handshake signal. The client started a handshake but the server could not finish it in time. Low host entropy is a classic cause.
  • zk_non_mtls_remote_conn_count: count of remote connections that completed without mutual TLS. In an mTLS-required environment this should be zero. A non-zero count means a client connected without presenting a client certificate, which indicates either a misconfigured port or a policy bypass.

A critical caveat: certificate expiry is a silent, total outage once every client uses TLS. ZooKeeper does not expose certificate expiry as a metric. You must check it externally with openssl or an equivalent tool. An expired server certificate causes every TLS client to fail the handshake, but zk_unsuccessful_handshake will not tell you “the cert expired.” It will just count the failures.

flowchart TD
    A["Handshake failures
on secure port"] --> B{"Cert valid?"}
    B -->|No| C["Renew and reload cert"]
    B -->|Yes| D{"Plaintext bytes
to TLS port?"}
    D -->|Yes| E["Fix client port
or use TLS client"]
    D -->|No| F{"Handshakes timing out?"}
    F -->|Yes| G["Check host entropy"]
    F -->|No| H["Cipher or SAN mismatch"]

Common causes

CauseWhat it looks likeFirst thing to check
Expired server or client certificateEvery TLS client fails handshake simultaneously; zk_unsuccessful_handshake spikesopenssl s_client -connect host:secureClientPort and inspect notAfter
Plaintext client hitting secure portNotSslRecordException in server log; monitoring tools (ruok/mntr via nc) failClient connection string port number
Cipher or protocol mismatchHandshake fails on specific JDK versions or distributions; works elsewhereTLS protocol version and cipher list on both ends
Hostname verification failureClient connects, cert is valid, but SAN does not match hostname the client usedCertificate SAN entries versus client connect string
Low host entropyzk_tls_handshake_exceeded increments; handshakes stall then complete or time out/proc/sys/kernel/random/entropy_avail
Wrong client TLS propertyClient logs “Unsuccessful handshake with session 0x0” despite correct certClient JVM property: zookeeper.client.secure not zookeeper.ssl.client.enable

Quick checks

These are read-only and safe to run during an incident. Adjust the secure client port and log path to match your deployment. If only the secure port is enabled and the plaintext port (2181) is closed, use a TLS-capable client for mntr instead of nc.

# Check TLS handshake counters via mntr
echo mntr | nc localhost 2181 | grep -E 'zk_unsuccessful_handshake|zk_tls_handshake_exceeded|zk_non_mtls'

# Verify the server certificate is not expired
echo | openssl s_client -connect localhost:2281 -showcerts 2>/dev/null | openssl x509 -noout -enddate -subject -issuer

# Check available host entropy (low values can stall TLS handshakes)
cat /proc/sys/kernel/random/entropy_avail

# Confirm which port is the secure client port versus plaintext
grep -E 'secureClientPort|clientPort|portUnification' /etc/zookeeper/zoo.cfg

# Check whether mTLS is required
grep 'ssl.clientAuth' /etc/zookeeper/zoo.cfg

# Look for TLS-specific errors in the server log
grep -E 'Unsuccessful handshake|NotSslRecordException|SSLHandshakeException' /var/log/zookeeper/zookeeper.log | tail -30

# Test a TLS connection from the server host to itself
# Add -cert/-key flags if mTLS is required
echo ruok | openssl s_client -connect localhost:2281 -quiet 2>/dev/null

How to diagnose it

  1. Confirm the scope. Is zk_unsuccessful_handshake incrementing on all ensemble members or just one? A single-node pattern points to a node-specific cert or config problem. All nodes points to a fleet-wide cert expiry or a widespread client misconfiguration.

  2. Check certificate validity first. Cert expiry is the highest-priority cause because it is a silent total outage. Run openssl s_client against the secure port on each node and inspect notAfter. If the cert expired, no amount of server tuning will fix the handshakes until the cert is renewed and reloaded.

  3. Inspect the server log for the specific failure. “Unsuccessful handshake” is the generic signal. The log line near it usually contains the real cause: NotSslRecordException (plaintext to TLS port), SSLHandshakeException (cert or cipher problem), or a timeout (entropy or network). Filter the log by the failing client IP if known.

  4. Distinguish client misconfiguration from a server problem. If a single client or client type fails while others succeed, the server is fine. The failing client is either using the wrong port, the wrong TLS property, an expired client cert, or a JDK with incompatible cipher defaults.

  5. Verify mTLS enforcement. If zk_non_mtls_remote_conn_count is non-zero in an mTLS-required environment, identify which clients connected without a client certificate. Check whether port unification is enabled (which accepts both TLS and plaintext on one port) and whether ssl.clientAuth is set to need.

  6. Check host entropy. If zk_tls_handshake_exceeded is incrementing but handshakes eventually succeed, the host may be starved for entropy. This is more common on virtualized hosts and containers without a hardware RNG.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
zk_unsuccessful_handshakeCounts failed TLS negotiations on the secure portAny sustained non-zero rate
zk_tls_handshake_exceededCounts handshakes that timed out during negotiationNon-zero rate indicates entropy, CPU, or network pressure
zk_non_mtls_remote_conn_countRemote connections that skipped mTLSNon-zero in mTLS-required environments
zk_non_mtls_local_conn_countLocal connections without mTLSOften acceptable; investigate if policy requires mTLS for localhost
Certificate notAfterZooKeeper does not expose cert expiry as a metricAny cert within 30 days of expiry
/proc/sys/kernel/random/entropy_availLow entropy stalls TLS handshakesSustained values below 200

Fixes

Expired certificate

Renew the certificate in your keystore and restart the affected ZooKeeper node. If the cert is managed by an external CA or cert-manager, trigger the renewal and verify the keystore was updated before restarting. Older ZooKeeper versions do not hot-reload TLS certificates, so a rolling restart is required.

For client certificates the same applies. The client must renew and reload its keystore. An expired client cert produces the same “Unsuccessful handshake” on the server side.

Plaintext client hitting the secure port

The most common offender is a monitoring tool using nc or a raw TCP socket to send ruok or mntr to what is now the secure port. The fix is to point the monitoring tool at the plaintext client port (default 2181) if it is still open, or to use a TLS-capable client for the health check.

If port unification is enabled, both TLS and plaintext are accepted on one port. Verify the client connects to the unified port and is not sending plaintext to a TLS-only port or TLS to a plaintext-only port.

Wrong client TLS property

The correct client-side JVM property to enable TLS is zookeeper.client.secure=true. The property zookeeper.ssl.client.enable=true does not work and produces “Unsuccessful handshake with session 0x0” on the server. This is the most frequent client-side misconfiguration.

Cipher or protocol mismatch

If handshakes fail only on certain JDK distributions, check the cipher list configured on both ends. JDK distributions may use different cipher naming conventions, and hardcoded cipher overrides in some ZooKeeper versions can break negotiation on JDKs that use different prefixes.

If you control both ends, force a specific TLS protocol version (TLSv1.2 or TLSv1.3) on both client and server to eliminate negotiation ambiguity.

Hostname verification failure

This typically appears in Kubernetes or dynamic infrastructure where the certificate SAN entries do not match the hostname or IP the client used to connect. The fix is either to regenerate the certificate with the correct SANs or to disable hostname verification on the client, which weakens security and should only be a temporary stopgap.

For quorum TLS, hostname verification is enabled by default. In Kubernetes, reverse DNS lookups for pod IPs may return names that do not match the certificate SANs, breaking quorum formation.

Low host entropy

Install haveged or ensure the kernel RNG is properly seeded. On virtualized hosts, confirm that the hypervisor provides entropy via virtio-rng or an equivalent mechanism. Low entropy manifests as zk_tls_handshake_exceeded incrementing alongside slow or incomplete handshakes.

Prevention

  • Monitor certificate expiry externally. ZooKeeper does not expose cert expiry as a metric. Use openssl s_client, cert-manager in Kubernetes, or your PKI tooling to alert on certificates approaching expiry. Treat any cert within 30 days of expiry as a ticket.
  • Alert on zk_unsuccessful_handshake rate. Any sustained non-zero rate in production warrants investigation. A single failing client is noisy; a fleet-wide spike is an outage.
  • Alert on zk_non_mtls_remote_conn_count in mTLS-required environments. The count should be zero. Non-zero means a client bypassed mutual TLS.
  • Test TLS after every cert rotation. Automated renewal that updates the keystore but does not trigger a ZooKeeper reload leaves the old cert in memory. Verify the running process presents the new cert.
  • Standardize the client TLS property. Document that zookeeper.client.secure=true is the correct property and that zookeeper.ssl.client.enable does not work. This prevents the most common client-side error.
  • Verify entropy on all ZooKeeper hosts. Add /proc/sys/kernel/random/entropy_avail to host monitoring. Entropy starvation is a known cause of TLS handshake stalls.

How Netdata helps

Netdata collects ZooKeeper TLS counters at per-second resolution alongside the host-level metrics that explain them. The useful correlations during a handshake incident:

  • Handshake failure rate versus cert rotation timing. A spike in zk_unsuccessful_handshake or zk_tls_handshake_exceeded that aligns with a deployment window or cert renewal event narrows the cause immediately.
  • zk_non_mtls_remote_conn_count as a standalone metric. A non-mTLS connection in an mTLS-required environment is visible on the dashboard without parsing server logs.
  • Host entropy from /proc/sys/kernel/random/entropy_avail collected by the Linux plugin. Correlate zk_tls_handshake_exceeded with entropy drops on the same host.
  • zk_unsuccessful_handshake alongside zk_num_alive_connections and zk_connection_drop_count. If connections drop during handshake but never establish a session, the problem is TLS, not session management or ephemeral timeouts.
  • JVM and GC metrics on the same host to rule out process-level stalls as a secondary cause of handshake timeouts.