$ guides / apache-httpd / apache-httpd-ocsp-stapling-failure ▌

Operations Guides

Apache OCSP stapling failure: the silent handshake that adds client latency

OCSP stapling failure is a textbook “silently catastrophic” signal: nothing in the error log at default levels, no 5xx, no worker pile-up, and yet every new TLS handshake is slower than it should be. When Apache cannot fetch or attach a stapled OCSP response, each client that wants revocation information makes its own OCSP request to the CA’s responder, adding hundreds of milliseconds before the handshake completes. The server looks healthy from the inside. The slowness only exists on the client side.

None of the usual Apache signals move: request rate is normal, BusyWorkers are normal, the scoreboard is normal, the error log is quiet. The only way to see the failure is to probe the handshake itself or to know which two error codes to grep for.

This guide covers how stapling is supposed to work, how to confirm it is broken, the small set of causes that account for almost all failures, and how to stop it from regressing silently after the next restart or certificate renewal.

What this means

With OCSP stapling, Apache periodically fetches a signed revocation status for its certificate from the CA’s OCSP responder (the URL comes from the certificate’s AIA extension) and attaches (“staples”) it to the TLS handshake. The client gets proof of freshness without contacting the CA.

When stapling is broken, Apache completes handshakes without a staple. Clients that check revocation (browsers under enterprise policy, many API clients, anything enforcing Must-Staple) then make their own round trip to the OCSP responder before trusting the connection. That round trip is on the critical path of connection establishment.

Two properties make this nasty:

No startup prefetch and no persistent cache. Apache does not fetch OCSP responses at startup; it fetches on demand during the first handshake that needs one, and the cache lives in shared memory (shmcb), so it does not survive restarts. The first clients after every restart get no staple, and every restart resets the clock.
Default logging hides it. Not all failure modes emit messages at default log levels. You either probe the handshake externally or know to look for AH01929 and AH02217.

flowchart LR
  subgraph Stapled["Stapling working"]
    C1[Client] -->|TLS handshake| A1[Apache]
    A1 -->|periodic refresh| R[CA OCSP responder]
    A1 -->|cert + staple| C1
  end
  subgraph Broken["Stapling broken"]
    C2[Client] -->|TLS handshake| A2[Apache]
    A2 -->|cert, no staple| C2
    C2 -->|own OCSP lookup, +100s of ms| R2[CA OCSP responder]
  end

The right-hand path is what “silent” means: every client pays the latency tax individually, and none of that traffic touches your logs or metrics.

Common causes

Cause	What it looks like	First thing to check
Stapling not fully configured	`SSLUseStapling on` set but no `SSLStaplingCache`, or stapling never enabled	Config: both directives present; `SSLStaplingCache` is mandatory and has no default
Missing issuer chain	AH02217 in error log; no staple served for that vhost	Whether `SSLCertificateFile` includes the intermediate certificates
Stapling cache too small	AH01929 in error log; responses evicted or never stored	Cache size vs. number of certificates (responses can be up to ~10 KB each)
Egress blocked to OCSP responder	Staples expire and never refresh; no staple served after cache lifetime	Outbound connectivity from the server to the responder URL in the cert’s AIA extension
Responder errors propagated to clients	Clients see OCSP errors instead of a plain handshake	`SSLStaplingReturnResponderErrors` still at its default `on`
Restart cleared the cache	Staples missing for a window after every restart; first handshakes slow	Uptime vs. when clients stopped receiving staples
Must-Staple certificate	Handshakes fail (not just slow) for enforcing clients when no staple is available	Certificate extensions for the RFC 7633 Must-Staple flag

Quick checks

These are all read-only.

# 1. Does the server actually staple? Look for the OCSP response section.
echo | openssl s_client -connect example.com:443 -servername example.com -status 2>/dev/null | grep -A2 "OCSP response"
# Working:  "OCSP Response Status: successful"
# Broken:   "OCSP response: no response sent"

# 2. Grep the error log for the two stapling codes.
grep -E "AH01929|AH02217" /var/log/apache2/error.log | tail -20
# RHEL path: /var/log/httpd/error_log

# 3. Confirm both directives are configured (paths vary by distro).
grep -rn "SSLUseStapling\|SSLStaplingCache" /etc/apache2/ /etc/httpd/ 2>/dev/null

# 4. Extract the OCSP responder URL the certificate points to.
openssl x509 -in /path/to/cert.pem -noout -ocsp_uri

# 5. Test egress from the server to that responder (must be run on the Apache host).
curl -s -o /dev/null -w "%{http_code}\n" --max-time 5 "$(openssl x509 -in /path/to/cert.pem -noout -ocsp_uri)"

# 6. Check whether the served chain includes the issuer certificate.
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null | grep -E "^ *[0-9]+ s:|^ *i:"

Check 1 is the definitive test and the one to automate. Run it per vhost, not just the default: each vhost can have a different certificate and a different stapling outcome. Check 6 matters because AH02217 (issuer not configured) is one of the two log-visible causes.

How to diagnose it

Establish the symptom externally. Run check 1 from a host outside the load balancer, against each HTTPS vhost. “No response sent” on any of them is your confirmation. If staples are present everywhere, the latency is elsewhere; see Apache backend response time: telling ‘Apache is slow’ from ’the backend is slow’.
Correlate with restarts. Compare ServerUptimeSeconds from mod_status with when stapling broke. If stapling fails only in the first minutes after a restart and then recovers, you are seeing the empty shmcb cache plus on-demand fetch, not a permanent misconfiguration.
Classify the cause from the error log. AH01929 means the stapling cache cannot hold the response (cache too small). AH02217 means the issuer chain is not configured for a certificate stapling is trying to serve. No message at all, with staples missing, points at responder reachability or responder-side errors.
Test responder reachability from the Apache host. Check 5 above. OCSP responder URLs are typically plain HTTP from the AIA extension, so an egress proxy or firewall rule that only allows 443 outbound will silently block refresh. This is one of the most common causes and produces no Apache-side error at default log levels.
Verify the chain on disk. For AH02217: since 2.4.8, SSLCertificateChainFile is deprecated and SSLCertificateFile is expected to load the intermediates directly. If the file contains only the leaf, stapling cannot build the validation path and refuses to staple for that vhost.
Check for Must-Staple. If clients report hard handshake failures rather than slowness, inspect the certificate for the Must-Staple extension. With Must-Staple, a missing staple is a connection failure for enforcing clients, which converts this from a latency bug into an outage.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Staple presence per vhost (external probe with `-status`)	The only direct measure of the failure	“No response sent” on any vhost beyond a short post-restart window
AH01929 / AH02217 counts in error log	The only server-side log evidence at reachable log levels	Any occurrence
Client-observed TTFB for TLS connections	Stapling failure shows up as handshake-phase latency, not request latency	Baseline shift in connection setup time with no change in `%D`
`%D` from access log	Request-time latency stays flat during stapling failure; that flatness is the discriminator	If TTFB rises but `%D` does not, suspect the handshake, not the app
Certificate days-to-expiry per vhost	Renewal events are when chains change and stapling silently breaks	New cert deployed without chain verification
TLS handshake CPU	Clients doing their own OCSP checks reconnect more aggressively in some stacks	CPU up with no request-rate change

Note the deliberate inclusion of %D as a negative signal. Stapling failure inflates connection setup, which happens before the request, so %D never sees it. Flat request latency plus slow connections is exactly the fingerprint.

Fixes

Complete the configuration

Both directives are required. SSLUseStapling defaults to off, and SSLStaplingCache has no default at all; stapling without a cache is silently non-functional. A minimal working block:

SSLUseStapling on
SSLStaplingCache shmcb:/var/run/ocsp(131072)

The cache path and size differ by distro; size it comfortably above the number of stapled certificates multiplied by the worst-case response size (responses can approach 10 KB each). Too small is not a soft failure: it is exactly what AH01929 reports. A graceful reload applies the change; the cache will be empty either way until the first fetches complete.

Fix the issuer chain (AH02217)

Append the intermediate certificates to the file referenced by SSLCertificateFile (leaf first, then intermediates). Since 2.4.8 this is the supported mechanism; do not resurrect SSLCertificateChainFile. Verify with check 6 that the server now sends the full chain, then re-run check 1.

Stop propagating responder failures to clients

SSLStaplingReturnResponderErrors defaults to on, which means a responder-side error is handed to the client instead of Apache simply falling back to an unstapled handshake. The production tuning guidance in the official SSL how-to recommends turning it off and tightening the refresh behavior:

SSLStaplingReturnResponderErrors off
SSLStaplingResponderTimeout 4
SSLStaplingStandardCacheTimeout 172800
SSLStaplingErrorCacheTimeout 60

The tradeoff: with responder errors suppressed, clients receive no revocation status during responder outages and must decide for themselves whether to soft-fail. For most deployments that is preferable to injecting errors into every handshake. Be aware that SSLStaplingFakeTryLater (default on) synthesizes a “tryLater” response when the responder is unreachable. In the shipped 2.4.x source this synthesis only reaches clients when SSLStaplingReturnResponderErrors is also enabled; with it off, mod_ssl answers the handshake without a staple and clients soft-fail, matching the documented interaction. Test against your actual clients after changing these.

Open egress to the responder

Allow outbound connections from the Apache hosts to the responder URL from check 4. This is infrastructure, not Apache config, which is why it gets missed: the TLS config looks perfect and nothing in Apache logs the refused connection. If egress must go through a proxy, note that mod_ssl’s stapling fetcher has limited proxy support; verify behavior on your version before relying on it.

Handle restarts and renewal deliberately

You cannot make the cache persistent, but you can shrink the impact window: probe each vhost with check 1 immediately after restarts and certificate renewals, and alert on a staple that is still missing after the expected on-demand fetch has had time to complete. If you use Let’s Encrypt or another short-lived cert, fold the chain-completeness check into the renewal hook, because that is when AH02217 regressions happen.

If you run mod_md for certificate management, its own stapling implementation (MDStapling on) is reported to behave better than mod_ssl’s. MDStapling requires Apache HTTP Server 2.4.42 or later with mod_md built in; mod_md is part of the 2.4 tree and needs libcurl available at build time, but no extra compile flag.

Prevention

Probe stapling, not just the port. Add an external per-vhost check equivalent to check 1. This is the single highest-value step; every other item is a secondary control.
Alert on the two AH codes. AH01929 and AH02217 belong in the same error-log alerting set as AH00484. See Apache error log monitoring: severity levels, AH codes, and what to alert on.
Gate renewals on chain completeness. A renewal that swaps in a leaf-only SSLCertificateFile passes configtest and breaks stapling silently.
Treat Must-Staple as an availability commitment. Do not request Must-Staple certificates unless stapling is monitored as a first-class signal; with it, every stapling failure is a handshake failure.
Track it as a maturity item. Stapling success rate is a Level 4 (“expert”) signal in the Apache monitoring maturity model precisely because most teams only discover it after a latency complaint they cannot reproduce server-side.

How Netdata helps

Access-log latency percentiles from %D give you the flat-request-latency baseline that makes the handshake-phase fingerprint visible: connection time degrades while request time does not.
Certificate expiry monitoring per endpoint covers the adjacent silent-TLS failure and alerts before renewal-time chain regressions become incidents.
Error-log monitoring surfaces AH01929 and AH02217 the moment they appear instead of after a client complains.
mod_status collection keeps worker, scoreboard, and uptime context alongside TLS symptoms, so you can confirm in one view that stapling failure is not accompanied by worker or backend problems.
Restart correlation via uptime tracking lets you tie a stapling gap directly to the restart or reload that emptied the cache.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

The Netdata solution

Apache HTTP Server monitoring with Netdata

Netdata monitors Apache HTTP Server with per-second metrics from mod_status, pre-built dashboards, and ML-powered anomaly detection. Watch busy versus idle workers and the scoreboard state mix, requests per second, bytes served per second, and request processing duration alongside the rest of your stack, so you catch the worker-exhaustion, slow-backend, and memory incidents in these runbooks before they page anyone.

See web server monitoring → Start monitoring free