$ guides / memcached / memcached-evictions-high ▌

Operations Guides

Memcached evictions climbing: the cache is full and discarding live data

The evictions counter is climbing. At least one slab class is saturated, and the daemon is removing valid, non-expired items to make room for new sets. The severity question is whether the cache is discarding cold data the application no longer needs, or live data that will be requested again within seconds.

Adding memory is the right response only when the cache is globally undersized. If the problem is slab calcification (memory concentrated in the wrong size classes), adding RAM does nothing useful and may mask the real issue. First determine whether evictions are healthy LRU turnover or harmful thrash, and whether the pressure is global or per-slab.

What this means

The evictions counter in the stats output is a cumulative 64-bit counter. It increments once per valid, non-expired item that memcached removes to make room for a new SET. It is distinct from reclaimed, which counts expired slots reused by new sets. Only evictions indicates pressure: the daemon is throwing away data that has not yet expired.

The absolute counter is meaningless. What matters is the rate relative to your write volume:

evictions/sec near zero with stable workload: cache is appropriately sized or oversized.
evictions/sec / sets/sec above 0.01: investigate. More than 1% of sets are forcing an eviction.
evictions/sec / sets/sec above 0.5: severely undersized. Half of all sets cause an eviction.

The discriminator between healthy and harmful evictions is evicted_time, reported per slab class in stats items. It is the age, in seconds since last access, of the most recently evicted item. High values mean old, cold items are being turned over: the LRU is doing its job. Low values, under roughly 300 seconds, mean recently-active data is being discarded: the cache is thrashing and providing little value for that size class.

The other critical distinction is global versus per-slab pressure. Global memory utilization (bytes / limit_maxbytes) can be at 50% while one slab class is 100% full and evicting aggressively. This is slab calcification, and it is the most underdiagnosed memcached problem in production.

flowchart TD
    A["evictions rising"] --> B{"hit ratio stable?"}
    B -- Yes --> C["healthy LRU churn"]
    B -- No, declining --> D{"global bytes near limit?"}
    D -- "yes, over 90%" --> E["global undersizing"]
    D -- "no, under 80%" --> F["slab calcification"]
    F --> G["stats slabs: identify evicting class"]
    E --> H{"evicted_time in evicting class?"}
    G --> H
    H -- "high (hours)" --> I["healthy turnover, add capacity"]
    H -- "low (under 300s)" --> J["harmful thrash"]

Common causes

Cause	What it looks like	First thing to check
Cache globally undersized	`bytes` near `limit_maxbytes`, evictions across multiple slab classes, hit ratio declining	`bytes / limit_maxbytes` ratio over time
Slab calcification	Global memory at 50-80%, evictions in one or two classes, others idle with free chunks	`stats slabs` for `free_chunks` per class
Working set grew	Gradual eviction climb over days, hit ratio eroding slowly	`curr_items` trend over time
TTL misconfiguration	Low `reclaimed` relative to `evictions`, items not expiring before they need space	TTL distribution in application code
Write-only waste	High `evicted_unfetched / evictions`, storing data nobody reads	`evicted_unfetched` counter

Quick checks

Run these read-only commands. None modify cache state. Adjust the netcat timeout flag (-q1) if your variant requires -w 1 or similar.

# Check eviction rate and global memory
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT (evictions|bytes |limit_maxbytes|reclaimed)"

# Check per-slab eviction distribution and age of evicted items
echo "stats items" | nc -q1 localhost 11211 | grep -E "(evicted|number)"

# Check slab memory allocation, find classes with zero free chunks
echo "stats slabs" | nc -q1 localhost 11211 | grep -E "(used_chunks|free_chunks|total_pages|chunk_size)"

# Check whether LRU maintainer is falling behind
echo "stats" | nc -q1 localhost 11211 | grep "STAT direct_reclaims"

# Check hit ratio signals to correlate with evictions
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT get_(hits|misses)"

# Check for write-only waste (items evicted that were never read)
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT (evicted|expired)_unfetched"

# Verify slab_automove is enabled <!-- TODO: verify default changed to mode 1 in 1.5.0 -->
echo "stats settings" | nc -q1 localhost 11211 | grep -E "slab_(automove|reassign)"

How to diagnose it

Compute the eviction rate. Sample evictions twice with a known interval and derive per-second. Do the same for cmd_set. Compute the ratio. If it is below 0.01 and hit ratio is stable, evictions are healthy turnover of cold items. Stop here unless hit ratio is also declining.
Check global memory utilization. Look at bytes / limit_maxbytes. If above 90% and evictions are spread across multiple slab classes, the cache is globally undersized or the working set grew. If below 80% but evictions are climbing, suspect slab calcification and continue.
Find the saturated slab class. From stats items, identify slab class IDs where evicted is non-zero. From stats slabs, check free_chunks for those classes. A class with zero free_chunks and active evictions is the one under pressure. Note its chunk_size to understand what item sizes are affected.
Read evicted_time for the saturated class. This is the key signal. From stats items, find evicted_time for the evicting class. If it is hours or days, the LRU is evicting cold items: healthy. If it is under 300 seconds, the class is thrashing and discarding actively-used data. This distinguishes a cache that is full but working from one that is full and failing.
Check direct_reclaims. When the LRU maintainer background thread cannot keep up, worker threads do inline eviction and block SET operations. Any sustained non-zero rate of direct_reclaims means memory pressure is acute enough to affect write latency.
Correlate with hit ratio. Compute hit ratio from deltas of get_hits and get_misses. Evictions with a stable high hit ratio are fine. Evictions with a declining hit ratio mean the cache is losing useful data. This correlation is the final arbiter of whether action is needed.
Check for write-only waste. If evicted_unfetched / evictions is high (above 0.5), the application is caching data that is never read before being evicted. This is an application-level issue, not a memory sizing issue.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`evictions` rate	Primary pressure indicator	Sustained non-zero with declining hit ratio
`evictions/sec / sets/sec`	Pressure relative to write volume	Above 0.5 means severely undersized
`evicted_time` per slab	Age of evicted items, healthy vs harmful	Under 300 seconds in an evicting class
`bytes / limit_maxbytes`	Global memory utilization	Above 90% suggests global undersizing
Per-slab `free_chunks`	Identifies slab calcification	Zero in one class while others have free chunks
`direct_reclaims`	LRU maintainer falling behind	Any sustained non-zero rate
`evicted_unfetched / evictions`	Write-only waste ratio	Above 0.5 means caching data nobody reads
`reclaimed` rate	Expired slots reused before eviction needed	Low reclaim with high eviction suggests TTLs too long
Hit ratio from `get_hits` / `get_misses`	Whether evictions are causing misses	Declining alongside rising evictions

Fixes

Cache globally undersized

If bytes is at or near limit_maxbytes, evictions are spread across multiple slab classes, and hit ratio is declining, the working set exceeds allocated RAM.

Increase memory at runtime using the cache_memlimit <MB> command . This avoids a restart but only works if the host has free RAM available.
Increase the -m flag and restart if the runtime increase is insufficient or the host needs more RAM allocated. A restart wipes the cache and causes a cold-start backend spike, so plan for warming.
Size for 20-30% headroom under peak workload. With slab_automove active, global headroom of 15-20% is workable because pages can be rebalanced across classes.

This is the one case where adding memory is the correct first response.

Slab calcification

If global memory is at 50-80% but one or two slab classes are evicting while others have many free chunks, the problem is per-slab, not global. Adding memory does not help directly because new pages may land in idle classes.

Verify slab_automove is enabled. Check stats settings | grep slab_automove. Mode 1 (default since 1.5.0) moves one page per 10 seconds from idle classes to evicting ones. If it is off, enable it: slabs automove 1.
Manually reassign a page for immediate relief: slabs reassign <source_class> <dest_class>. Warning: this is destructive to the source class. Items in the moved page are evicted. Choose a source class with high free_chunks and zero evictions.
Avoid mode 2 long-term. slabs automove 2 is aggressive and can cause jitter. Use it only temporarily during an acute calcification event.
Fix the root cause on next restart. If the item size distribution shifted (serialization change, new data type), consider adjusting the -f growth factor so slab class boundaries better match the workload. This requires a restart.
Review application serialization. If objects grew unexpectedly, a deploy may have changed serialization format or added fields. Compare current item sizes against expectations.

TTL misconfiguration

If reclaimed is low relative to evictions, items are not expiring before they need to be evicted. TTLs may be too long for data that churns quickly.

Shorten TTLs for data that is rarely re-read. This lets the LRU crawler reclaim expired slots proactively instead of the LRU evicting valid items.
Check for items with no TTL. Items set without expiration live until evicted. If the application sets long-lived items that are rarely accessed, they consume memory indefinitely and crowd out active data.

Write-only waste

If evicted_unfetched / evictions is high, the application is caching data nobody reads. This displaces useful items.

Identify the write-only keys. This requires application-level instrumentation or lru_crawler metadump in a non-production environment. The fix is in the application: stop caching data that is never retrieved.
Review cache-aside logic. Some patterns write to cache after every database update regardless of read demand. If reads do not follow, the writes are pure waste.

Prevention

Monitor per-slab, not just global. Track free_chunks and evicted_time per slab class. Global bytes / limit_maxbytes hides slab calcification.
Alert on evicted_time, not just evictions. A threshold of evicted_time under 300 seconds in any slab class with active evictions catches harmful thrash early. Evictions alone are too noisy.
Alert on the eviction-to-set ratio, not the absolute rate. A cache handling 500K gets/sec with 100 evictions/sec may be healthy. A cache handling 1K gets/sec with the same eviction rate is in crisis. Use relative thresholds.
Keep slab_automove enabled. Mode 1 prevents most calcification automatically. Do not disable it without a specific reason.
Track evicted_unfetched and expired_unfetched. Rising ratios indicate the application is caching data nobody reads, which wastes memory and displaces useful items.
Establish baselines per workload. Absolute thresholds generate false positives. Know what normal looks like for your traffic patterns and alert on deviations from baseline.

How Netdata helps

Netdata’s memcached collector surfaces per-second metrics for the signals that matter during an eviction event:

Per-slab eviction distribution from stats items, so you can see immediately whether pressure is global or concentrated in one class.
evicted_time per slab class, graphed alongside eviction rate, so you can distinguish healthy turnover from harmful thrash without manual nc sessions.
Eviction rate correlated with hit ratio and backend load, so the inverse relationship is visible in one view. Declining hit ratio alongside rising evictions is the pattern that confirms harmful pressure.
direct_reclaims tracking, which catches the moment the LRU maintainer falls behind and worker threads start doing inline eviction.
evicted_unfetched and expired_unfetched ratios, highlighting write-only waste that displaces useful data.
Anomaly detection on eviction rate and per-slab counters, flagging the onset of pressure before static thresholds are crossed.