HAProxy TLS handshake failures: broken clients after a cert or cipher change

A certificate rotation or TLS policy change goes out, HAProxy reloads cleanly, health checks pass, traffic graphs look normal, and then the tickets start: a subset of clients can no longer connect. The load balancer itself is fine. The handshake rate even looks normal. What changed is the failure rate within handshakes, and HAProxy has no dedicated stats counter that shows it to you directly.

The failing clients never send an HTTP request, so they do not appear in request rates, response codes, or backend metrics. They surface only as ereq increments on SSL frontends and as “SSL handshake failure” lines in the logs. If you are not watching those two places, the first signal you get is a human.

What this means

A TLS handshake failure means the client and HAProxy could not agree on the cryptographic parameters of the connection: protocol version, cipher suite, or certificate trust. The TCP connection is established, the ClientHello arrives, and the handshake dies before any HTTP request exists. From HAProxy’s perspective the session is a request error, which is why these failures land in ereq on the frontend rather than in any response-code counter.

The four root causes cover almost every incident:

  1. Wrong TLS version: the client only supports a version your new policy no longer accepts (or vice versa).
  2. No shared cipher: the client’s cipher list and your configured cipher list do not intersect. A subtle variant is an ECDSA certificate on HAProxy with a client that only supports RSA cipher suites, or the reverse. The cipher names appear to overlap, but the certificate key type cannot complete the key exchange.
  3. Untrusted or expired certificate: the client cannot build a trust chain, the cert expired, or the intermediate chain is incomplete. Some clients (especially older mobile and embedded stacks) fail on missing intermediates while browsers succeed.
  4. SNI mismatch: the client requests a hostname for which HAProxy has no matching certificate and either serves the wrong one or fails outright.

The diagnostic trap is the aggregate handshake rate. SslFrontendKeyRate counts handshakes attempted, not handshakes succeeded. If 5% of your clients broke, the rate barely moves while the failure rate within those handshakes went from zero to one in twenty. The failure rate is the real signal, and you can only get it from logs.

flowchart LR
  A[Client connects] --> B{TLS handshake}
  B -->|success| C[HTTP request - normal path]
  B -->|fails| D[ereq increments on frontend]
  B -->|fails| E[SSL handshake failure in logs]
  D --> F[No req_rate, no hrsp counters]
  E --> G[Log parsing required to see cause]

Common causes

CauseWhat it looks likeFirst thing to check
TLS minimum version raisedOld clients (embedded, legacy OS, old Java) fail immediately after the change; modern clients finessl-min-ver on the bind line vs. %sslv values in logs from failing clients
Cipher list narrowed“no shared cipher” failures; possibly only EC or only RSA clients affectedCompare client-offered ciphers in logs against the configured cipher list; check whether the cert is ECDSA or RSA
Expired or wrong certificateAll clients for one SNI name fail with trust errors; other names on the same frontend fineshow ssl cert and openssl x509 -enddate on the served cert
Incomplete certificate chainBrowsers work, older clients and some mobile stacks fail with untrusted certInspect the PEM file for the full chain, leaf plus intermediates
SNI mismatch after rotationOne hostname fails while others on the same bind succeed; default cert served for the wrong nameCert-to-SNI mapping, fallback cert configuration
Version upgrade changed defaultsClients that worked before an HAProxy or OS upgrade now fail, with no explicit config changeDiff the effective TLS defaults: HAProxy has defaulted to TLSv1.2 minimum since 2.2; on OpenSSL 3.x distributions with the default security level, TLS below 1.2 is not negotiable at all
Health check noise“SSL handshake failure” log lines with no real client impact, steady low rateCorrelate log timestamps with health check intervals; SSL-enabled checks tearing down connections can log as failures

Quick checks

These are all read-only. Run them before changing anything.

# 1. Confirm ereq is incrementing on the SSL frontend
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 == "FRONTEND" {print $1": ereq="$13}'

# 2. Baseline handshake rate (attempted, not succeeded)
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | \
  grep -E "^Ssl(FrontendKeyRate|Rate):"

# 3. Session cache behavior (a reload or ticket rotation causes a miss spike)
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | \
  grep -E "^SslCache(Lookups|Misses):"

# 4. Certificate inventory and expiry as HAProxy sees it
echo "show ssl cert" | socat unix-connect:/var/run/haproxy.sock stdio

# 5. Expiry and chain for a specific certificate
echo "show ssl cert /etc/haproxy/certs/example.pem" | \
  socat unix-connect:/var/run/haproxy.sock stdio

# 6. Same check from the file on disk (adjust path to your deployment)
openssl x509 -in /etc/haproxy/certs/example.pem -noout -enddate -issuer -subject

# 7. Find handshake failures in the logs (path depends on your syslog setup)
grep -i "SSL handshake failure" /var/log/haproxy.log | tail -20

# 8. What TLS versions and ciphers are succeeding (requires %sslv %sslc in log-format)
grep -oP 'TLSv[0-9.]+' /var/log/haproxy.log | sort | uniq -c | sort -rn

Check 1 tells you whether the failure is real and which frontend it hits. Checks 4 through 6 catch the most self-inflicted cause: the wrong or expired cert being served. Check 7 is where the actual diagnosis lives. If check 8 returns nothing, your log format does not capture SSL fields, which is itself a finding to fix in the prevention step.

How to diagnose it

  1. Confirm the signal. Take two ereq samples 60 seconds apart on the SSL frontend. A rising delta confirms live request errors. On an internet-facing frontend a low steady rate is scanner noise; what matters is a step change, especially one that lines up with a deploy or reload timestamp.

  2. Line the spike up with a change. Check HAProxy’s reload time (Uptime_sec in show info resets on reload) and your config management history. An ereq spike that starts at the moment of a cert rotation or a TLS policy edit is almost always the change. This correlation is the single fastest path to root cause.

  3. Pull the failing handshakes from logs. Grep for handshake failure lines and look at the source IPs, SNI names if logged, and any SSL fields your log format captures. The goal is to characterize the failing population: one hostname, one TLS version, one client family, or everything.

  4. Classify the failure. Map what you see to one of the four root causes:

    • Failing clients all on TLSv1.0/1.1: version policy. Since HAProxy 2.2 the default minimum is TLSv1.2, and on OpenSSL 3.x distributions the default security level blocks anything below it even if configured. An upgrade can break legacy clients with no config diff at all.
    • “no shared cipher” errors: compare the client-offered cipher list against your configured list, and check certificate key type (ECDSA vs RSA). A cipher name match is not sufficient if the cert key type cannot do the exchange.
    • Trust or expiry errors: verify the served cert and chain with show ssl cert and openssl x509. Check that the PEM contains the intermediates, not just the leaf.
    • One hostname failing on a multi-cert frontend: SNI mapping or fallback cert behavior after rotation.

  5. Reproduce from a client you control. Once you have a hypothesis, replay the failing handshake with openssl s_client pinned to the failing version or cipher, for example openssl s_client -connect your.host:443 -servername your.host -tls1_1 or with -cipher restricted to the failing suite. A clean reproduction confirms the cause before you change anything.

  6. Rule out noise. If the failure lines are steady, low-rate, and timestamp-aligned with health check intervals, you may be looking at SSL health checks tearing down connections, which can log as handshake failures. Benign, but it pollutes the signal and is worth filtering.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
ereq on SSL frontendsThe only stats-level counter that captures handshake failures, folded in with other request errorsStep change aligned with a reload or config change
Handshake failure lines in logsThe only place the failure cause (version, cipher, cert) is visibleAny spike above the steady scanner-noise baseline
SslFrontendKeyRateHandshake attempt rate; the denominator for judging failure proportionNormal rate while failures rise: attempts fine, success rate collapsed
SslCacheLookups / SslCacheMissesCache miss spikes after reload or ticket rotation are expected; sustained misses are notMiss rate stays high long after a rotation
Cert expiry via show ssl certExpiry is a total, immediate failure for every client on that certExpiry inside 7 days
DroppedLogs in show infoHandshake failures are only visible in logs; losing logs during this incident blinds youAny increment during an investigation

Fixes

Version policy broke legacy clients

Decide deliberately which clients you are willing to drop. If legacy clients must be supported, set ssl-min-ver explicitly on the bind line rather than relying on defaults. Note the hard floor: on distributions shipping OpenSSL 3.x with the default security level, TLS versions below 1.2 cannot be negotiated regardless of configuration. The tradeoff is real: supporting TLSv1.0/1.1 for a few clients weakens policy for everyone on that frontend. A separate frontend or listener for legacy clients is usually the better answer than loosening the main one.

Cipher mismatch

Widen the cipher list to include suites the affected clients offer, or accept that those clients are dropped. If the failure is an EC/RSA certificate type mismatch (cipher names overlap but the key exchange cannot complete), the fix is on the certificate side: deploy both an RSA and an ECDSA certificate so each client family gets a compatible chain. Test changes with openssl s_client -cipher before rolling them out.

Certificate problems

Replace or complete the certificate: full chain in the PEM, correct SAN coverage, valid dates. On HAProxy 2.x and later you can update a certificate without a reload using the runtime API: set ssl cert <path> to stage the new PEM, then commit ssl cert <path> to apply it. Verify with show ssl cert <path> afterwards, and confirm the failing clients recover in the logs before declaring done.

SNI mismatch

Fix the certificate-to-name mapping and verify each hostname on the bind with openssl s_client -servername. Ensure the fallback certificate served when no SNI matches is one you are comfortable presenting to unknown clients.

Do not restart as a first move

None of these fixes require a restart. Config changes go through reload; cert changes can go through the runtime API. A restart drops every live connection, including the healthy majority, to fix a problem affecting a minority.

Prevention

  • Log SSL fields. Add %sslv and %sslc to the frontend log format so version and cipher distribution is visible without an incident. Without this, every future TLS change is a blind rollout.
  • Watch ereq after every change. Any certificate rotation, cipher change, or ssl-min-ver edit should be followed by a deliberate check of ereq delta and the handshake failure lines in logs for at least 15 to 30 minutes. Make it a step in the change procedure, not an afterthought.
  • Alert on the step change, not the rate. Internet-facing frontends always have some ereq noise. Alert on a sustained deviation from baseline, and gate alerts with Uptime_sec so counter resets on reload do not produce phantom signals.
  • Monitor certificate expiry with escalating thresholds (30 days, 7 days, 24 hours) via show ssl cert or direct file checks. Expiry is a scheduled outage; there is no excuse for being surprised.
  • Canary the change. Where possible, roll TLS policy changes to one listener or one HAProxy instance first and watch the failure signals before widening.
  • Know your client population. Keep an inventory of client TLS capabilities (from the version and cipher distribution in logs) so a policy change can be evaluated against reality before it ships, not against assumptions.

How Netdata helps

  • Netdata collects the HAProxy stats CSV continuously, so ereq per frontend is charted as a rate over time. The step change after a cert or cipher change is visible in seconds rather than discovered in a ticket queue.
  • Because Netdata keeps per-second history across reloads, you can line an ereq spike up with the exact reload timestamp (Uptime_sec reset) to confirm causation instead of guessing.
  • SSL signals from show info (handshake rate, cache lookups and misses) sit on the same dashboard as ereq, so the “attempt rate normal, failure rate spiking” pattern is visible in one view.
  • DroppedLogs is monitored alongside, warning you when the log pipeline that carries your handshake failure detail is itself failing.
  • Correlating ereq with frontend session rate and request rate separates “clients failing the handshake” from “traffic disappeared upstream,” two failures that look identical from the application’s side.