$ guides / memcached / memcached-cache-stampede ▌

Operations Guides

Memcached cache stampede: a hot key expires and the backend takes the hit

A cache stampede happens when a popular cached key expires or is evicted and many concurrent requests miss at the same instant. Each miss falls through to the backend independently. The backend, usually sized to handle only the small fraction of traffic that misses the cache, receives the full load for that key at once. Memcached itself is not malfunctioning. It is correctly reporting misses and returning them as fast as it can. The victim is the backend, and the fix lives in the application layer.

The memcached process looks healthy during a stampede. CPU is normal or slightly elevated, memory is not full, evictions are zero. The dashboard that matters is the backend’s: database connections, query latency, and error rates. The memcached dashboard only shows the shape of the miss burst.

The distinguishing signature: cmd_get spikes, get_misses spikes in lockstep, evictions stays low or zero because memory is not the constraint, and backend load spikes simultaneously. If you see high misses paired with high evictions, you have a different problem. If you see high misses with low evictions and no backend spike, you may have a cold start or a key-pattern change rather than a stampede on a hot key.

What this means

When a hot key expires, every request that arrives in the next few milliseconds discovers the miss on its own. There is no coordination between requests in the text or binary protocol. Each miss is independent. If the key handles 5,000 requests per second and the backend query takes 50 ms, roughly 250 concurrent backend queries for the same data fire before the first one completes and repopulates the cache. The backend, typically provisioned for the steady-state miss rate of a few percent of total traffic, is now handling a burst that is orders of magnitude larger.

The same dynamic occurs on a cold start after a restart or flush_all, but a single-key stampede is narrower. Aggregate hit ratio may dip only slightly because the rest of the cache is fine, while the backend sees a targeted spike on the specific queries behind that one key. A cold start hits everything at once and is easier to spot from the memcached side.

Memcached has no server-side request coordination in the text or binary protocols; every miss stands alone. Prevention belongs in the application layer.

flowchart TD
    A[Hot key hits TTL] --> B[N concurrent requests arrive]
    B --> C[All N miss independently]
    C --> D[All N query backend at once]
    D --> E[Backend saturates]
    E --> F[First response repopulates cache]
    F --> G[Later requests hit]

Common causes

Cause	What it looks like	First thing to check
Hot key TTL expiry	Single key’s miss rate spikes on a predictable interval; backend load spikes on the same cadence	Correlate miss bursts with the key’s TTL
Synchronized TTLs	Many keys set with the same TTL expire together; broad miss spike	Check whether the application sets identical TTLs at deploy time
Cold start after restart	`uptime` reset, `curr_items` near zero, hit ratio near 0% everywhere	Check `uptime` and `dmesg` for OOM kills
`flush_all` issued	`cmd_flush` incremented, `get_flushed` spikes, all keys invalidated	Check `cmd_flush` counter and audit access
Hot key evicted	Hot key evicted rather than expired; `evictions` non-zero, `evicted_time` low	Check `stats items` per-slab eviction counts

Quick checks

# Check daemon responsiveness and key counters
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT (uptime|cmd_get|get_misses|evictions|cmd_flush)"

# Check hit ratio components
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT get_(hits|misses)"

# Check for flush_all events and their read-side impact
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT (cmd_flush|get_flushed)"

# Check memory pressure (should be low for a pure stampede)
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT (bytes |limit_maxbytes|evictions)"

# Check per-slab eviction distribution to rule out slab-driven eviction
echo "stats items" | nc -q1 localhost 11211 | grep -E "evicted"

# Check CPU usage (cumulative; compute delta between two samples)
echo "stats" | nc -q1 localhost 11211 | grep "STAT rusage"

These are read-only. Under extreme throughput, stats briefly contends on internal locks; sample at 10-second or longer intervals, not in a tight loop during an active incident.

How to diagnose it

Confirm the shape. Sample cmd_get, get_misses, and evictions twice with a known interval and compute rates. A stampede shows get_misses rate spiking while evictions rate stays near zero. If evictions is also spiking, you are looking at an eviction cascade or slab imbalance, not a pure stampede.
Rule out cold start. Check uptime. If it recently reset, the entire cache is cold and every key is missing, not just one. Check cmd_flush. If it incremented, someone issued flush_all, which lazily invalidates all items.
Correlate with backend load. The definitive signal is backend load rising in lockstep with the memcached miss spike. If backend load is flat while misses spike, the misses may be for keys that do not hit an expensive backend path, or the backend has enough headroom to absorb them silently.
Identify the hot key if possible. Memcached does not expose per-key access statistics. You need application-level telemetry, distributed tracing, or client-side instrumentation to find which key is driving the misses. stats cachedump is limited, expensive, and will not reliably identify a hot key under production load.
Check for synchronized TTLs. If the miss spike recurs on a regular interval, examine how the application sets TTLs. A common antipattern is setting the same TTL for many keys at deploy time, causing them all to expire at the same instant.
Manually warm the key as immediate mitigation. If you can identify the hot key and generate its value cheaply, set it directly to stop the stampede while you work on a structural fix.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`get_misses` rate	Direct measure of cache misses reaching the backend	Sudden spike with low `evictions`
`cmd_get` rate	Total read load on memcached	Sharp increase correlated with miss spike
`evictions` rate	Distinguishes stampede from memory pressure	Stays low or zero in a pure stampede
Hit ratio, from `get_hits` / (`get_hits` + `get_misses`)	Effectiveness metric computed from deltas	Sudden drop, especially narrow or recurring
`rusage_user` + `rusage_system`	CPU load from processing miss responses	Rises during burst
Backend load, external	The actual victim of the stampede	Spikes in lockstep with misses
`cmd_flush`	Rules out `flush_all` as the trigger	Any increment in production
`uptime`	Rules out restart as the trigger	Unexpected reset

Fixes

Request coalescing, single-flight

On a miss, the first request acquires a lock (distributed lock, in-process mutex, or lease) and queries the backend. Concurrent requests for the same key wait on the lock and receive the result from the first request’s backend query. Only one backend call is made.

Tradeoff: adds client complexity. If the lock holder crashes or times out without releasing, other requests need a fallback such as wait-with-timeout then proceed independently. Lock contention itself can become a bottleneck if the key is extremely hot.

Server-assisted locks and leases via Meta Protocol

Memcached’s Meta Protocol provides server-side support for this pattern. The mg command with the N flag instructs memcached to auto-create a placeholder item on a miss. The first client receives a W (win) flag and the right to recache. Subsequent clients receive a Z flag indicating another client is already handling it. This is the upstream equivalent of the lease mechanism described in Facebook’s “Scaling Memcache at Facebook” paper.

The meta protocol and its stampede flags were introduced in memcached 1.6.0 (released March 2020) and are documented as no longer experimental in the official protocol reference.

Tradeoff: requires a client library that speaks the Meta Protocol (mg, ms, md commands). Many older client libraries only support the text or binary protocol. The binary protocol is officially deprecated as of 1.6.0; it still receives fixes but will not gain new features, and the meta protocol is the recommended replacement.

Probabilistic early refresh

Instead of waiting for TTL expiry, each request probabilistically decides to refresh the cache early. The probability increases as the remaining TTL approaches zero. This spreads the refresh load across many requests instead of concentrating it at the expiry instant.

Tradeoff: slightly more backend load than strictly necessary because some early refreshes happen when the key was not about to expire. The behavior is tuned by a parameter that controls how aggressively refreshes are pulled forward. Simpler than locking and requires no coordination between requests.

Stale-while-revalidate

Serve the stale value while one request refreshes the cache in the background. The Meta Protocol supports this via the md (meta delete) command with the I flag, which marks an item as stale. Subsequent mg calls receive W (win, you do the refresh) and X (stale, serve this value) flags.

Tradeoff: requires the application to tolerate serving slightly stale data. For keys where staleness is acceptable (configuration, feature flags, rendered HTML), this is often the best tradeoff. For keys where it is not (session data, real-time counters), use locking instead.

TTL jitter

Add random jitter to TTLs to prevent synchronized expiry. If keys that would all expire at the same instant instead expire over a spread window, the miss burst flattens.

Tradeoff: does not reduce the total number of backend queries, only spreads them in time. Useful as a baseline mitigation, especially for synchronized TTL problems, but does not prevent single-hot-key stampedes on its own. Combine with one of the above patterns for genuinely hot keys.

Prevention

Identify hot keys proactively. Use application telemetry or distributed tracing to find keys with disproportionate traffic. Feature flags, configuration, and user-profile keys are common stampede candidates.
Set TTLs with jitter. Never set identical TTLs on many keys at the same instant. Add random jitter (for example, plus or minus 10% of the TTL) to spread expirations.
Implement single-flight or lease logic for known hot keys. The cost is low and the payoff is large for any key that backs an expensive backend query.
Monitor the backend, not just memcached. A stampede is a backend problem that surfaces in memcached stats. Alert on backend latency and error rates, not just memcached miss rate.
Have a cache-warming procedure. After a restart or flush_all, the entire cache is cold. Pre-warm hot keys before admitting full traffic if your architecture allows it.
Audit flush_all access. flush_all is destructive: it invalidates all keys and does not lock the server. Any increment of cmd_flush in production should alert. Restrict who can issue it.

How Netdata helps

Per-second collection of cmd_get, get_misses, evictions, and hit ratio shows the stampede forming as it happens, not minutes after the backend saturates.
Correlation across signals: miss rate spiking while evictions stay flat and backend latency rises is the stampede signature.
ML anomaly detection on miss rate and hit ratio can flag the burst even when absolute values are within normal ranges for your workload.
Backend metrics alongside memcached metrics (database connections, query latency) confirm the cascade impact in the same time window.
cmd_flush and uptime tracking rules out flush_all and restart as the trigger without manual checks during the incident.