Memcached evicted_time low: distinguishing healthy turnover from cache thrash

A high global eviction counter is one of the most over-alerted memcached signals. Operators see evictions climbing and page someone at 3 a.m. or reflexively add memory. Both reactions are usually wrong. Raw eviction count says nothing about whether the evicted items mattered. The discriminator that turns “evictions are high” into “evictions are harmful” is evicted_time, reported per slab class in the output of stats items.

evicted_time is the age, in seconds since last access, of the most recently evicted item in a given slab class. Hours or days means the cache is discarding cold data to make room for new entries. Seconds or low minutes means the cache is thrashing: evicting items that were accessed moments ago and would have been hit again. The same eviction rate can be healthy at one value of evicted_time and catastrophic at another.

What it is and why it matters

evicted_time is reported by stats items, not by the plain stats command. It appears once per slab class:

# Read evicted_time for every slab class.
# On systems where nc lacks -q (Alpine, macOS), use -w 1 instead.
echo "stats items" | nc -q1 localhost 11211 | grep evicted_time

The protocol defines it as the number of seconds since the last access of the most recent item evicted from that class. Three properties matter for interpretation:

• Per slab class, not global. There is no single evicted_time for the instance. Each size class reports its own. A cache with a healthy aggregate can have one class thrashing badly while the rest idle.
• Last item, not average. It reflects the single most recent eviction, not a rolling mean. A single very old item evicted among many young ones can briefly inflate the number.
• Only meaningful under active eviction. When a slab class has zero evictions, evicted_time is 0. That zero is silence, not a measurement. Do not alert on evicted_time < 60s without first confirming the class is actually evicting.

Eviction rate alone is not actionable. Treat standalone eviction alerts as informational, not page-worthy, because without evicted_time you cannot tell healthy turnover from harmful thrash. The combination of sustained evictions and low evicted_time is the condition that warrants urgency.

How it works

Memcached evicts items from the tail of the per-slab-class LRU when memory in that class is exhausted and the LRU maintainer cannot reclaim enough expired items to satisfy an incoming SET. The item removed is the one judged least recently used. evicted_time reports how long ago that specific item was last accessed.

Since memcached 1.5.0 the default per-class LRU is the segmented LRU, with three active segments per class: HOT, WARM, and COLD. New inserts land in HOT. Items that age out of HOT without re-access move to COLD. Items re-accessed in COLD promote to WARM. Evictions happen from the tail of the COLD segment. The LRU maintainer background thread moves items between segments; the LRU crawler reclaims expired items from the tails. Before 1.5.0 the flat per-class LRU was the default and the maintainer was opt-in via -o lru_maintainer.

This matters for reading evicted_time. In the segmented model the COLD tail is already the least-recently-used population. A low evicted_time means items are reaching the COLD tail while still comparatively fresh, which is the signature of a working set larger than the memory allocated to that class. A high evicted_time means the COLD tail is full of genuinely stale items, which is the LRU doing exactly what it was designed to do.

flowchart TD
  A["stats items: per slab class"] --> B{"evictions occurring
in this class?"}
  B -- No --> C["evicted_time = 0
ignore, not a measurement"]
  B -- Yes --> D{"evicted_time
seconds since last access"}
  D -- "hours to days" --> E["healthy turnover
evicting cold items"]
  D -- "300s to hours" --> F["borderline
compare to hot key access interval"]
  D -- "under 300s" --> G["thrash
evicting recently-active data"]
  D -- "under 60s" --> H["severe thrash
cache nearly useless for this class"]

Where it shows up in production

evicted_time low with sustained evictions appears in a small number of recurring patterns. Recognising the pattern tells you which fix to reach for.

Working set exceeds cache size. Every slab class is under pressure. evicted_time is low across many classes, not just one. Global bytes is near limit_maxbytes. Hit ratio is declining across the board. The fix is more memory, more instances, or a smaller working set: shorter TTLs, less cache-everything, narrower caching scope.

Slab imbalance. Only one or a few slab classes are evicting, while others have free chunks. Global bytes can sit at 50-70% of limit_maxbytes and look healthy. This is the slab trap: global metrics mislead because memory cannot move between classes without slab_reassign or slab_automove. slab_automove mode 1 is default since 1.5.0 and will eventually rebalance, but it is conservative, moving roughly one page per 10 seconds. Diagnose with stats slabs showing which classes are full (used_chunks at capacity, free_chunks at zero) and which have headroom, plus stats items showing where evicted_time is low.

Item size distribution shift. A deploy changed serialization, added fields, or switched formats. The new items fall into a different slab class that was previously small or empty. That class fills quickly and starts evicting active data. Look for a recent deploy correlated with the drop in evicted_time and a change in which slab classes are full. Adjusting the growth factor (-f, default 1.25) on the next restart can help, but slab_automove will usually catch up given time.

Cache-warming overload. A warming job or stampede pushes a sustained burst of SETs into one class. The LRU maintainer cannot reclaim expired items fast enough and worker threads enter direct reclaim. direct_reclaims (from stats, since 1.4.23, only meaningful with the LRU maintainer active) will be non-zero. evicted_time drops sharply during the burst.

Post-restart cold churn. Immediately after a restart, slab classes are being assigned pages for the first time and the cache is filling from empty. Brief eviction spikes with low evicted_time are expected during warmup and are not a fault. The signal normalises as the working set stabilises.

Common misreadings

• Alerting on a global aggregate. There is no global evicted_time. If your tooling averages or sums per-class values, it will hide the one class that is thrashing. Read per slab class, always.
• Alerting on evicted_time < 60s without checking evictions. When a class has no evictions, evicted_time is 0. An alert on “low evicted_time” alone will fire constantly on idle classes. Combine with a sustained eviction rate for that class.
• Treating it as an average. It is the age of the last evicted item. A single very old item evicted among many young ones can momentarily raise the number. Watch the trend, not a single sample, and correlate with eviction rate.
• Comparing across slab classes with different workloads. A session-cache class and a rendered-page class will have different natural ages. “Low” for one may be normal for the other.
• Ignoring version differences. On the flat per-class LRU (default before 1.5.0, opt-in via -o lru_maintainer thereafter), evicted_time still reports the last evicted item’s age, but the eviction dynamics differ. The thresholds above are calibrated for the segmented model.

Signals to watch alongside evicted_time

evicted_time is never interpreted in isolation. The following signals disambiguate what the number means.

How Netdata helps

Netdata’s memcached collector pulls stats and stats items on a per-second cadence, which is what makes the per-slab reading of evicted_time useful in production. Correlating the following signals shortens the diagnosis path:

• Per-slab evicted_time charted alongside per-slab eviction rate, so you can see which class is thrashing and confirm it is actually evicting before alerting.
• Hit ratio on the same dashboard as eviction rate and evicted_time, so the question “are these evictions hurting?” answers itself visually.
• direct_reclaims and the LRU movement stats (moves_to_cold, moves_to_warm, moves_within_lru) exposed as charts, so you can tell whether the segmented LRU is keeping up or falling behind.
• evicted_unfetched and expired_unfetched tracked as rates, so thrash caused by caching write-only data is visible without a separate investigation.
• Memory utilisation broken out per slab class from stats slabs, so slab imbalance is obvious even when global bytes / limit_maxbytes looks healthy.
• ML anomaly detection on evicted_time and eviction rate per slab class, which catches the slow drift from healthy turnover toward thrash before it crosses a static threshold.

Related guides

• Memcached evicted_unfetched and expired_unfetched: caching data nobody ever reads
• Memcached connection refused: telling a dead process from a hung or full one
• Memcached high miss rate: separating cold start, new key patterns, and memory pressure
• How Memcached actually works in production: a mental model for operators
• Memcached hit ratio dropping: reading get_hits, get_misses, and cache effectiveness
• Memcached monitoring checklist: the signals every production cache needs
• Memcached monitoring maturity model: from survival to expert
• Memcached alive but not responding: the silent process hang
• Memcached unexpected restart: uptime reset, wiped cache, and the cold-start backend spike