OCSP stapling failure is a textbook “silently catastrophic” signal: nothing in the error log at default levels, no 5xx, no worker pile-up, and yet every new TLS handshake is slower than it should be. When Apache cannot fetch or attach a stapled OCSP response, each client that wants revocation information makes its own OCSP request to the CA’s responder, adding hundreds of milliseconds before the handshake completes. The server looks healthy from the inside. The slowness only exists on the client side.

None of the usual Apache signals move: request rate is normal, BusyWorkers are normal, the scoreboard is normal, the error log is quiet. The only way to see the failure is to probe the handshake itself or to know which two error codes to grep for.

This guide covers how stapling is supposed to work, how to confirm it is broken, the small set of causes that account for almost all failures, and how to stop it from regressing silently after the next restart or certificate renewal.

What this means

With OCSP stapling, Apache periodically fetches a signed revocation status for its certificate from the CA’s OCSP responder (the URL comes from the certificate’s AIA extension) and attaches (“staples”) it to the TLS handshake. The client gets proof of freshness without contacting the CA.

When stapling is broken, Apache completes handshakes without a staple. Clients that check revocation (browsers under enterprise policy, many API clients, anything enforcing Must-Staple) then make their own round trip to the OCSP responder before trusting the connection. That round trip is on the critical path of connection establishment.

Two properties make this nasty:

  • No startup prefetch and no persistent cache. Apache does not fetch OCSP responses at startup; it fetches on demand during the first handshake that needs one, and the cache lives in shared memory (shmcb), so it does not survive restarts. The first clients after every restart get no staple, and every restart resets the clock.
  • Default logging hides it. Not all failure modes emit messages at default log levels. You either probe the handshake externally or know to look for AH01929 and AH02217.
flowchart LR
  subgraph Stapled["Stapling working"]
    C1[Client] -->|TLS handshake| A1[Apache]
    A1 -->|periodic refresh| R[CA OCSP responder]
    A1 -->|cert + staple| C1
  end
  subgraph Broken["Stapling broken"]
    C2[Client] -->|TLS handshake| A2[Apache]
    A2 -->|cert, no staple| C2
    C2 -->|own OCSP lookup, +100s of ms| R2[CA OCSP responder]
  end

The right-hand path is what “silent” means: every client pays the latency tax individually, and none of that traffic touches your logs or metrics.

Common causes

CauseWhat it looks likeFirst thing to check
Stapling not fully configuredSSLUseStapling on set but no SSLStaplingCache, or stapling never enabledConfig: both directives present; SSLStaplingCache is mandatory and has no default
Missing issuer chainAH02217 in error log; no staple served for that vhostWhether SSLCertificateFile includes the intermediate certificates
Stapling cache too smallAH01929 in error log; responses evicted or never storedCache size vs. number of certificates (responses can be up to ~10 KB each)
Egress blocked to OCSP responderStaples expire and never refresh; no staple served after cache lifetimeOutbound connectivity from the server to the responder URL in the cert’s AIA extension
Responder errors propagated to clientsClients see OCSP errors instead of a plain handshakeSSLStaplingReturnResponderErrors still at its default on
Restart cleared the cacheStaples missing for a window after every restart; first handshakes slowUptime vs. when clients stopped receiving staples
Must-Staple certificateHandshakes fail (not just slow) for enforcing clients when no staple is availableCertificate extensions for the RFC 7633 Must-Staple flag

Quick checks

These are all read-only.

# 1. Does the server actually staple? Look for the OCSP response section.
echo | openssl s_client -connect example.com:443 -servername example.com -status 2>/dev/null | grep -A2 "OCSP response"
# Working:  "OCSP Response Status: successful"
# Broken:   "OCSP response: no response sent"

# 2. Grep the error log for the two stapling codes.
grep -E "AH01929|AH02217" /var/log/apache2/error.log | tail -20
# RHEL path: /var/log/httpd/error_log

# 3. Confirm both directives are configured (paths vary by distro).
grep -rn "SSLUseStapling\|SSLStaplingCache" /etc/apache2/ /etc/httpd/ 2>/dev/null

# 4. Extract the OCSP responder URL the certificate points to.
openssl x509 -in /path/to/cert.pem -noout -ocsp_uri

# 5. Test egress from the server to that responder (must be run on the Apache host).
curl -s -o /dev/null -w "%{http_code}\n" --max-time 5 "$(openssl x509 -in /path/to/cert.pem -noout -ocsp_uri)"

# 6. Check whether the served chain includes the issuer certificate.
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null | grep -E "^ *[0-9]+ s:|^ *i:"

Check 1 is the definitive test and the one to automate. Run it per vhost, not just the default: each vhost can have a different certificate and a different stapling outcome. Check 6 matters because AH02217 (issuer not configured) is one of the two log-visible causes.

How to diagnose it

  1. Establish the symptom externally. Run check 1 from a host outside the load balancer, against each HTTPS vhost. “No response sent” on any of them is your confirmation. If staples are present everywhere, the latency is elsewhere; see Apache backend response time: telling ‘Apache is slow’ from ’the backend is slow’.

  2. Correlate with restarts. Compare ServerUptimeSeconds from mod_status with when stapling broke. If stapling fails only in the first minutes after a restart and then recovers, you are seeing the empty shmcb cache plus on-demand fetch, not a permanent misconfiguration.

  3. Classify the cause from the error log. AH01929 means the stapling cache cannot hold the response (cache too small). AH02217 means the issuer chain is not configured for a certificate stapling is trying to serve. No message at all, with staples missing, points at responder reachability or responder-side errors.

  4. Test responder reachability from the Apache host. Check 5 above. OCSP responder URLs are typically plain HTTP from the AIA extension, so an egress proxy or firewall rule that only allows 443 outbound will silently block refresh. This is one of the most common causes and produces no Apache-side error at default log levels.

  5. Verify the chain on disk. For AH02217: since 2.4.8, SSLCertificateChainFile is deprecated and SSLCertificateFile is expected to load the intermediates directly. If the file contains only the leaf, stapling cannot build the validation path and refuses to staple for that vhost.

  6. Check for Must-Staple. If clients report hard handshake failures rather than slowness, inspect the certificate for the Must-Staple extension. With Must-Staple, a missing staple is a connection failure for enforcing clients, which converts this from a latency bug into an outage.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Staple presence per vhost (external probe with -status)The only direct measure of the failure“No response sent” on any vhost beyond a short post-restart window
AH01929 / AH02217 counts in error logThe only server-side log evidence at reachable log levelsAny occurrence
Client-observed TTFB for TLS connectionsStapling failure shows up as handshake-phase latency, not request latencyBaseline shift in connection setup time with no change in %D
%D from access logRequest-time latency stays flat during stapling failure; that flatness is the discriminatorIf TTFB rises but %D does not, suspect the handshake, not the app
Certificate days-to-expiry per vhostRenewal events are when chains change and stapling silently breaksNew cert deployed without chain verification
TLS handshake CPUClients doing their own OCSP checks reconnect more aggressively in some stacksCPU up with no request-rate change

Note the deliberate inclusion of %D as a negative signal. Stapling failure inflates connection setup, which happens before the request, so %D never sees it. Flat request latency plus slow connections is exactly the fingerprint.

Fixes

Complete the configuration

Both directives are required. SSLUseStapling defaults to off, and SSLStaplingCache has no default at all; stapling without a cache is silently non-functional. A minimal working block:

SSLUseStapling on
SSLStaplingCache shmcb:/var/run/ocsp(131072)

The cache path and size differ by distro; size it comfortably above the number of stapled certificates multiplied by the worst-case response size (responses can approach 10 KB each). Too small is not a soft failure: it is exactly what AH01929 reports. A graceful reload applies the change; the cache will be empty either way until the first fetches complete.

Fix the issuer chain (AH02217)

Append the intermediate certificates to the file referenced by SSLCertificateFile (leaf first, then intermediates). Since 2.4.8 this is the supported mechanism; do not resurrect SSLCertificateChainFile. Verify with check 6 that the server now sends the full chain, then re-run check 1.

Stop propagating responder failures to clients

SSLStaplingReturnResponderErrors defaults to on, which means a responder-side error is handed to the client instead of Apache simply falling back to an unstapled handshake. The production tuning guidance in the official SSL how-to recommends turning it off and tightening the refresh behavior:

SSLStaplingReturnResponderErrors off
SSLStaplingResponderTimeout 4
SSLStaplingStandardCacheTimeout 172800
SSLStaplingErrorCacheTimeout 60

The tradeoff: with responder errors suppressed, clients receive no revocation status during responder outages and must decide for themselves whether to soft-fail. For most deployments that is preferable to injecting errors into every handshake. Be aware that SSLStaplingFakeTryLater (default on) synthesizes a “tryLater” response when the responder is unreachable, and its interaction with SSLStaplingReturnResponderErrors off has been reported to differ from what the documentation states. Test against your actual clients after changing these.

Open egress to the responder

Allow outbound connections from the Apache hosts to the responder URL from check 4. This is infrastructure, not Apache config, which is why it gets missed: the TLS config looks perfect and nothing in Apache logs the refused connection. If egress must go through a proxy, note that mod_ssl’s stapling fetcher has limited proxy support; verify behavior on your version before relying on it.

Handle restarts and renewal deliberately

You cannot make the cache persistent, but you can shrink the impact window: probe each vhost with check 1 immediately after restarts and certificate renewals, and alert on a staple that is still missing after the expected on-demand fetch has had time to complete. If you use Let’s Encrypt or another short-lived cert, fold the chain-completeness check into the renewal hook, because that is when AH02217 regressions happen.

If you run mod_md for certificate management, its own stapling implementation (MDStapling on) is reported to behave better than mod_ssl’s.

Prevention

  • Probe stapling, not just the port. Add an external per-vhost check equivalent to check 1. This is the single highest-value step; every other item is a secondary control.
  • Alert on the two AH codes. AH01929 and AH02217 belong in the same error-log alerting set as AH00484. See Apache error log monitoring: severity levels, AH codes, and what to alert on.
  • Gate renewals on chain completeness. A renewal that swaps in a leaf-only SSLCertificateFile passes configtest and breaks stapling silently.
  • Treat Must-Staple as an availability commitment. Do not request Must-Staple certificates unless stapling is monitored as a first-class signal; with it, every stapling failure is a handshake failure.
  • Track it as a maturity item. Stapling success rate is a Level 4 (“expert”) signal in the Apache monitoring maturity model precisely because most teams only discover it after a latency complaint they cannot reproduce server-side.

How Netdata helps

  • Access-log latency percentiles from %D give you the flat-request-latency baseline that makes the handshake-phase fingerprint visible: connection time degrades while request time does not.
  • Certificate expiry monitoring per endpoint covers the adjacent silent-TLS failure and alerts before renewal-time chain regressions become incidents.
  • Error-log monitoring surfaces AH01929 and AH02217 the moment they appear instead of after a client complains.
  • mod_status collection keeps worker, scoreboard, and uptime context alongside TLS symptoms, so you can confirm in one view that stapling failure is not accompanied by worker or backend problems.
  • Restart correlation via uptime tracking lets you tie a stapling gap directly to the restart or reload that emptied the cache.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.