Memcached segmented LRU: HOT/WARM/COLD tiers and what the move counters tell you

Since memcached 1.5.0, every slab class runs a segmented LRU instead of a single flat queue. Items land in HOT, WARM, COLD, and optionally TEMP tiers, and a background thread moves them between tiers based on access patterns. The goal is to protect frequently-accessed items from being evicted by one-off scans that touch cold data.

The move counters under stats items describe how well that sorting is working. moves_to_cold counts items aging out of active use, moves_to_warm counts items rescued from COLD by a re-access, and moves_within_lru counts re-ranking within WARM. The ratios between these counters explain why a slab class evicts the way it does, even when global memory looks fine.

Most teams never touch the tier configuration and should not. But if you are debugging per-slab eviction patterns, slab calcification, or a cache with headroom that still thrashes, the move counters are the diagnostic layer beneath the eviction rate.

What it is and why it matters

Before segmented LRU, each slab class had one doubly-linked list. Every access bumped the item to the head; eviction happened at the tail. The problem: a scan that reads many keys once each (a batch job, a crawler, a warm-up script) pushes every scanned item toward the head and evicts genuinely active items at the tail. One pass over cold data can wreck the hit ratio for the entire class.

Segmented LRU, introduced as opt-in in 1.4.23 via -o lru_maintainer and made the default in 1.5.0, splits each slab class into sub-LRUs. A background thread (the LRU maintainer) enforces tier transitions asynchronously, off the request path. A scan that touches cold items once each now drives them toward COLD, not toward the head of a single queue, so they can be evicted without displacing WARM items in the active working set.

How it works

Each slab class maintains up to four sub-LRUs. The LRU maintainer thread iterates them, enforces size and age limits, moves items between tiers, reclaims expired items at tails, and processes asynchronous promotion requests from COLD.

flowchart TD
    NEW["new SET"] --> HOT["HOT: probationary, FIFO"]
    HOT -->|"tail item active"| WARM["WARM: reused items"]
    HOT -->|"tail item inactive"| COLD["COLD: eviction pool"]
    WARM -->|"tail item active, bumped to head"| WARM
    WARM -->|"tail item inactive"| COLD
    COLD -->|"re-accessed: async bump"| WARM
    COLD -->|"evicted at tail"| EVICT["eviction"]

HOT. New items land here on SET. HOT is a FIFO-style probationary queue: items are not bumped to the head on re-access within HOT. When the maintainer reaches the HOT tail, it checks whether that tail item has been re-accessed. Active tail items move to WARM; inactive tail items move to COLD.

WARM. Items that were re-accessed after entering HOT, or rescued from COLD. WARM items are bumped toward the head on re-access, unlike HOT. Inactive items at the WARM tail sink to COLD. WARM buffers workloads where items are read a few times but not continuously.

COLD. The eviction pool. Evictions happen from the COLD tail. If a COLD item is accessed again before eviction, the maintainer queues it for an asynchronous move back to WARM. Under heavy load that bump queue can overflow and some rescues become probabilistic rather than guaranteed.

TEMP. Opt-in via -o temporary_ttl=<N> at startup or lru temp_ttl <N> at runtime. Items with a TTL at or below N seconds bypass HOT, WARM, and COLD entirely. TEMP items are never bumped, never moved between tiers, and are not evictable; the maintainer reaps them on expiry. TEMP exists to stop short-lived items (rate-limit tokens, dedup keys, request-scoped cache entries) polluting the main tiers.

HOT and WARM are each capped at a fraction of per-class memory; current 1.6.x defaults are 20% for HOT and 40% for WARM. A secondary age-based cap also applies: items move out of HOT or WARM when their tail age exceeds a configured factor of COLD’s tail age. These caps stop HOT or WARM monopolizing memory while COLD starves and evictions accelerate.

The move counters

Under stats items, each slab class reports movement counters. These are cumulative since process start; sample twice and compute deltas to get rates.

# Per-slab move counters and tier occupancy
echo "stats items" | nc -q1 localhost 11211 | grep -E "moves_to_(cold|warm)|moves_within_lru|direct_reclaims|number_(hot|warm|cold|temp)"

Three counters describe tier movement, one describes pressure:

The ratio that matters most is moves_to_warm / moves_to_cold. This is the rescue rate: what fraction of items that reach COLD get pulled back before eviction.

• High rescue rate: items in COLD are being re-accessed. Segmentation is doing its job: active items survive in WARM, genuinely cold items age out.
• Low rescue rate: items reach COLD and are never accessed again before eviction. Normal for write-heavy or scan-heavy workloads where most items are touched once. It only becomes a problem if hit ratio is also declining, meaning items you needed were evicted before their next access.
• Near-zero moves_to_warm with high evictions: the cache is filling with one-off items. Active items get evicted alongside cold ones because the segmentation cannot distinguish them fast enough. This pattern often accompanies scans over a working set larger than the cache.

direct_reclaims is the pressure signal. Normally the maintainer handles all eviction work in the background. When a slab’s SET rate exceeds the maintainer’s ability to move items to COLD and evict from there, worker threads start evicting directly. Each direct reclaim is synchronous work on a worker thread, adding latency to the SET that triggered it. Sustained non-zero direct_reclaims means pressure is acute and the background thread cannot keep up.

Per-slab tier occupancy

stats items also reports how many items sit in each tier per slab class: number_hot, number_warm, number_cold, and (if TEMP is enabled) number_temp. Alongside these, age_hot, age_warm, and age_cold report the age of the oldest item in each tier.

# Tier occupancy and age per slab class
echo "stats items" | nc -q1 localhost 11211 | grep -E "number_(hot|warm|cold|temp)|age_(hot|warm|cold)"

These numbers tell you whether the tiering matches the workload:

• Most items in WARM with a healthy moves_to_warm rate: the cache has a stable, active working set. Ideal for read-heavy workloads.
• Most items in COLD: items enter HOT, get one or zero re-accesses, and sink. Expected for write-heavy workloads, but a problem if hit ratio is low.
• HOT near its cap with low moves_to_warm: new items are arriving faster than they can be promoted or aged out. The class is under write pressure.
• age_cold very low relative to the typical interval between accesses to popular keys: items are evicted shortly after their last access. That slab class is too small for the working set.

Where this shows up in production

You will reach for these counters in a few specific situations.

Investigating per-slab eviction patterns. When stats items shows one slab class evicting aggressively while others have free chunks, the move counters tell you whether that class is shedding genuinely cold items or losing active ones. High moves_to_warm with rising evicted_time means healthy turnover. Low moves_to_warm with falling evicted_time means thrashing.

Diagnosing scan-heavy workloads. A batch job or crawler that reads thousands of keys once each drives moves_to_cold up and moves_to_warm down. If that coincides with a hit-ratio drop, the scan is displacing the active working set. Enabling TEMP LRU for short-TTL scan entries, or rethinking the scan pattern, may help.

Understanding why adding memory did not help. If a saturated slab class has low moves_to_warm and high direct_reclaims, more memory gives the maintainer headroom but does not change the access pattern. The items being cached are not being re-read. The fix is application-level: cache fewer one-off items, or accept the eviction rate as the cost of the workload.

Deciding whether to enable TEMP LRU. If your workload sets many items with short TTLs (seconds to a minute) and they churn through HOT and COLD without contributing to hit ratio, TEMP LRU isolates them. Set temporary_ttl to cover your shortest-lived items. Be conservative: TEMP items are not evictable, so a threshold set too high can exhaust memory.

Tuning the tiers

Live tuning is available via the lru command. These change runtime behavior on a live cache; lru mode flat in particular alters eviction immediately, so test on a non-production node first.

# Switch between flat and segmented modes (live, changes eviction behavior immediately)
echo "lru mode flat" | nc -q1 localhost 11211
echo "lru mode segmented" | nc -q1 localhost 11211

# Adjust HOT and WARM percentage caps and age factors
echo "lru tune <hot_pct> <warm_pct> <hot_age_factor> <warm_age_factor>" | nc -q1 localhost 11211

# Enable or adjust TEMP LRU
echo "lru temp_ttl <ttl>" | nc -q1 localhost 11211

lru tune adjusts four parameters: the percentage of per-class memory allotted to HOT, the percentage to WARM, and age factors controlling when items move out of HOT or WARM relative to COLD’s tail age. Current defaults are hot_lru_pct=20, warm_lru_pct=40, hot_max_factor=0.20, and warm_max_factor=2.00; stats settings reports the live values.

Most teams should not tune these. The defaults handle typical read-heavy workloads. Adjust the tier caps only with concrete evidence that segmentation is mismatched to your access pattern, such as consistent low rescue rates with declining hit ratio despite adequate total memory.

Version caveat. The lru tune command was inaccessible in memcached 1.6.40 because the rewritten protocol parser compared a command name without checking its length; commit 5f366ccbbfd040a67d2d649f2ae8b5416a2b7677 fixed the regression in 1.6.41. If you run 1.6.40, tuning commands may silently fail or return errors. Upgrade to 1.6.41 or later before attempting tier adjustments.

TEMP LRU risk. Because TEMP items are not evictable, setting temporary_ttl too high fills memory with items that cannot be reclaimed until they expire. Start with a low threshold (a few seconds) and increase only if you see TEMP items expiring before they were useful.

Signals to watch in production

These signals sit beneath the per-slab eviction rate and evicted_time in the diagnostic stack. If hit ratio is stable and evicted_time is healthy, you do not need the move counters at all.

How Netdata helps

• Per-second collection of move counters exposes ratio shifts before they propagate to hit ratio, which is a lagging indicator.
• Correlating a drop in moves_to_warm with a rise in evictions and a fall in evicted_time pinpoints the moment active items stop being rescued and start being evicted.
• direct_reclaims as a per-slab alert catches maintainer-thread overload before it manifests as client-visible SET latency.
• number_hot, number_warm, and number_cold per slab class, alongside eviction rate, show whether the tiering matches the workload or items are pooling in the wrong tier.
• Anomaly detection on the moves_to_warm / moves_to_cold ratio flags workload shifts the segmentation cannot keep up with, such as a new batch job or a change in key-access patterns.

Related guides

• Memcached evicted_time low: distinguishing healthy turnover from cache thrash
• Memcached evictions climbing: the cache is full and discarding live data
• Memcached eviction cascade: when a full cache overloads the backend
• Memcached evicted_unfetched and expired_unfetched: caching data nobody ever reads
• Memcached cache stampede: a hot key expires and the backend takes the hit
• Memcached command rate anomalies: cmd_get and cmd_set spikes and sudden drops
• Memcached cas_badval climbing: check-and-set contention and lost updates
• Memcached conn_yields rising: one client’s pipeline starving the others
• Memcached connection churn: total_connections racing and TIME_WAIT buildup
• Memcached curr_connections climbing: connection leaks and missing pooling
• Memcached connection limit reached: accepting_conns=0 and clients being refused
• Memcached connection refused: telling a dead process from a hung or full one