Memcached eviction cascade: when a full cache overloads the backend

The most common memcached-related production incident is not a memcached crash. The process stays up, serves hits at sub-millisecond latency, CPU is flat, and memory looks fine in aggregate. What breaks is the backend.

An eviction cascade starts when the working set outgrows the cache. Memory fills, evictions accelerate, and every evicted item becomes a future miss. Those misses fall through to the database, which was sized for cache-assisted load, not raw traffic. The backend saturates, latency climbs stack-wide, and slow responses drive retries that add still more load. It is a positive feedback loop, and memcached is not where it closes.

The distinguishing signal is the shape of the hit ratio collapse. An eviction cascade degrades hit ratio gradually, over minutes, as evictions compound. An instant drop to near-zero is a flush or a restart, not eviction pressure. A slow slide rather than a cliff points to this pattern.

What this means

Memcached is behaving correctly: when memory is full and a new SET arrives, it evicts something. The problem is that the evicted item is data the application still wants. When misses outpace the backend’s capacity to absorb them, the cascade begins.

Two things make this incident hard to catch early. First, memcached itself looks healthy. The items it does serve come back fast, CPU is normal, and stats returns clean numbers. The victim is downstream, not in the cache process. Second, global memory utilization is a poor early signal. With a slab allocator, a single hot slab class can be 100% full and evicting aggressively while global bytes sits at 60% of limit_maxbytes. If you only watch global memory, you miss the pressure until hit ratio has already slipped.

The cascade is self-reinforcing: evictions produce misses, misses produce backend load, backend load produces latency, latency produces timeouts and retries, and retries produce more SET traffic as applications refresh stale data.

flowchart TD
    A[Working set exceeds cache] --> B[Hot slab class fills]
    B --> C[Evictions accelerate]
    C --> D[Evicted items become future misses]
    D --> E[Misses fall through to backend]
    E --> F[Backend saturates]
    F --> G[Latency climbs stack-wide]
    G --> H[Timeouts drive retries]
    H --> E

Common causes

Quick checks

These are read-only. Run them against the memcached port (default 11211). The -q1 flag is netcat-openbsd syntax; adjust for your variant.

# Confirm process is alive and responding
echo "version" | nc -q1 localhost 11211

# Check uptime - a recent restart means cold cache, not eviction pressure
echo "stats" | nc -q1 localhost 11211 | grep "STAT uptime"

# Confirm cmd_flush is not the cause - any increment means a flush_all happened
echo "stats" | nc -q1 localhost 11211 | grep "STAT cmd_flush"

# Hit and miss counters - compute ratio from deltas, not absolutes
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT get_(hits|misses)"

# Global eviction rate - sustained non-zero means pressure somewhere
echo "stats" | nc -q1 localhost 11211 | grep "STAT evictions"

# Global memory utilization - misleading for slab-allocated caches, but a sanity check
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT (bytes |limit_maxbytes)"

# Per-slab eviction distribution - the single most important check
echo "stats items" | nc -q1 localhost 11211 | grep -E "(evicted|evicted_time|outofmemory)"

# Per-slab memory distribution - find the imbalance
echo "stats slabs" | nc -q1 localhost 11211 | grep -E "(used_chunks|free_chunks|total_pages)"

# Direct reclaims - worker threads doing the LRU maintainer's job
echo "stats" | nc -q1 localhost 11211 | grep "STAT direct_reclaims"

# Is slab automove enabled?
echo "stats settings" | nc -q1 localhost 11211 | grep "slab_automove"

How to diagnose it

1. Confirm the shape. Pull get_hits and get_misses at two points 60-120 seconds apart and compute the ratio. A gradual slide means eviction territory. An instant drop means check uptime and cmd_flush first.

2. Rule out flush and restart. If uptime is low or cmd_flush incremented, the cause is a cold cache, not an eviction cascade. Treat that differently. See Memcached unexpected restart.

3. Find the saturated slab class. Run stats items and look for classes with non-zero evicted counters. Cross-reference with stats slabs to see which classes have free_chunks == 0.

4. Check eviction quality. For each evicting slab class, read evicted_time. This is the number of seconds since the last access of the most recently evicted item. Below 300 seconds means the cache is evicting recently-active data. Below 60 seconds means severe thrash. If evicted items are old, the LRU is doing its job and the hit ratio drop has a different cause.

5. Check whether pressure is global or local. If only one or two slab classes are evicting while others have free chunks, you have slab calcification on top of the cascade. If every active class is evicting, the cache is undersized for the working set.

6. Correlate with backend load. The defining signal of the cascade is that miss rate tracks backend load. Plot miss rate against database queries per second; they should move together.

7. Check for direct reclaims. Non-zero direct_reclaims means the LRU maintainer thread is falling behind and worker threads are evicting inline. This is acute pressure, not background churn.

Metrics and signals to monitor

Fixes

If the cache is globally undersized

If every active slab class is evicting and there is no slab imbalance, the working set has outgrown the allocation. The cleanest fix is more memory.

# Raise memcached memory limit at runtime (1.4.27+)
echo "cache_memlimit 8192" | nc localhost 11211

If this command is unavailable, restart memcached with a higher -m value, accepting a cold cache. Either way, the host needs the RAM. Moving memcached to a larger host, or adding instances to the consistent hashing ring, is the structural fix.

If the problem is slab calcification

Memory is available globally but locked in the wrong slab classes. This is the most under-diagnosed cause of eviction cascades.

# Check if automove is on
echo "stats settings" | nc -q1 localhost 11211 | grep slab_automove

# Enable conservative automove at runtime
echo "slabs automove 1" | nc localhost 11211

slab_automove mode 1 is conservative. It moves pages slowly and only from classes with low recent eviction activity. Mode 2 is more aggressive but can cause jitter and is not recommended for sustained use. If you cannot wait for automove, you can move a page manually:

# Manually move one page from source slab class to destination.
# OK schedules an asynchronous move; it is not completion confirmation.
# WARNING: this is destructive. All items in the source page are evicted.
# Only move from classes with free chunks and no recent evictions.
echo "slabs reassign <src_class> <dst_class>" | nc localhost 11211

If the problem is TTL or warming

If TTLs were recently shortened, expired items free memory but useful items also expire sooner, accelerating effective eviction. Revert the TTL change if you can.

If a warming job stopped, the working set now relies on organic traffic to repopulate. Restart the warmer. Watch cmd_set rate climb and hit ratio recover.

If the backend is already saturating

The cascade has moved downstream. Memcached fixes will not help in the seconds that matter. Buy the backend room first:

• Shed load at the edge if you can (rate limit, circuit break).
• Disable non-critical read paths that hit the backend.
• If you have a cache-warming script, run it.
• Once backend latency stabilizes, work the memcached side.

Prevention

• Per-slab eviction distribution. The single most common miss is treating global memory as the capacity signal.
• Alert on low evicted_time. Below 300 seconds in any active slab class plus sustained evictions is the early warning before the cascade closes.
• Track hit ratio as rate of change. A 15-point slide over 10 minutes is the signal; a stable 85% is just your baseline.
• Keep 15-20% global headroom. With slab_automove active, global headroom is meaningful; without it, per-slab headroom is what matters.
• Confirm slab_automove is mode 1 at boot. Mode 1 is the 1.5.0+ default, but startup flags such as -o no_slab_automove can override it. Do not assume the default if your startup scripts were written years ago.
• Keep a cache-warming path operational. The cascade is much worse when warming is broken and nobody notices until traffic spikes.
• Watch direct_reclaims. It is a late signal, but it confirms the LRU maintainer is no longer keeping up.

How Netdata helps

• Per-second collection of evictions, get_hits, get_misses, and cmd_get shows the hit-ratio slide as it starts, not minutes after the backend is already saturated.
• Per-slab metrics (evicted_time, used_chunks, free_chunks per class) make slab calcification visible without running stats items by hand during the incident.
• Anomaly detection flags the gradual hit-ratio decline and rising eviction rate against the instance’s own baseline, which is the shape of this incident.
• Correlating memcached signals with backend metrics (database queries per second, database latency) confirms whether misses have reached the database yet.
• Alerting on cmd_flush and uptime resets rules out the two other causes of sudden hit-ratio collapse.
• Alerting on low evicted_time with sustained evictions catches the cascade before hit ratio visibly slips.

Related guides

• Memcached evicted_unfetched and expired_unfetched: caching data nobody ever reads
• Memcached cas_badval climbing: check-and-set contention and lost updates
• Memcached connection refused: telling a dead process from a hung or full one
• Memcached evictions climbing: the cache is full and discarding live data
• Memcached high miss rate: separating cold start, new key patterns, and memory pressure
• How Memcached actually works in production: a mental model for operators
• Memcached incr/decr misses: evicted counters that silently break rate limiters and locks
• Memcached hit ratio dropping: reading get_hits, get_misses, and cache effectiveness
• Memcached monitoring checklist: the signals every production cache needs
• Memcached monitoring maturity model: from survival to expert
• Memcached alive but not responding: the silent process hang
• Memcached unexpected restart: uptime reset, wiped cache, and the cold-start backend spike