Memcached reclaimed vs evictions: reading memory pressure through reclamation

Most operators learn one signal first: evictions. It goes up, the cache is full, hit ratio drops, someone pages. The counter next to it, reclaimed, tells you how often memcached made room for a new item by reusing an expired item’s slot instead of throwing out live data. Read together, they describe the same process from two angles: how memcached finds space for new writes.

Two more counters complete the picture. crawler_reclaimed is the background LRU crawler doing the same work proactively. direct_reclaims counts cases where a worker thread had to reclaim inline because the background thread fell behind. The four together tell you whether your cache is in a healthy steady state, scrambling to keep up, or actively destroying data the application still wants.

This article assumes you understand the slab allocator and segmented LRU at a high level (see How Memcached actually works in production).

What it is and why it matters

Every SET in memcached needs a slab chunk in the right size class. When the class has free chunks, the SET takes one. When the class is full, memcached makes room one of two ways:

Reuse an expired item’s slot. The item’s TTL has elapsed, so it is dead data. This increments reclaimed when a SET reuses the slot, and crawler_reclaimed when the background LRU crawler frees the item ahead of demand.
Evict a valid item. The item still has a future TTL. Removing it throws away data the application might request next. This increments evictions.

reclaimed is a positive signal. evictions is a pressure signal. The ratio that matters is reclaimed / (reclaimed + evictions). High means most room-making is healthy turnover; low means most room-making is destructive.

direct_reclaims is the warning layer on top. It is exposed only when the LRU maintainer thread is active; upstream gates it behind -o lru_maintainer, and the daemon default remains off even in current releases. When the maintainer is enabled and keeping up, background work handles reclamation and worker threads normally avoid inline LRU work. When it falls behind, usually under a sustained high SET rate, worker threads start reclaiming and evicting inline. Each such event increments direct_reclaims. Any sustained non-zero rate there is acute pressure.

How it works

When a SET arrives and the target slab class has no free chunks, memcached falls back to one of several room-making paths:

flowchart TD
    Full[Slab class full,
SET needs a chunk] --> How{How is room found?}
    How -->|SET reuses an
expired item slot| Reclaim[reclaimed++
healthy turnover]
    How -->|Crawler frees
expired items| Crawl[crawler_reclaimed++
background cleanup]
    How -->|Worker reclaims inline,
maintainer fell behind| Direct[direct_reclaims++
acute pressure]
    How -->|No expired items,
live item dropped| Evict[evictions++
useful data lost]

All four counters describe success or failure of the same goal: free a chunk for the incoming SET without throwing away data anyone will read.

reclaimed grows when a SET lands in a slot that held an expired item. No live data lost. This is the cache doing what TTLs were designed to enable.
crawler_reclaimed grows in the background, independent of SETs. High values are healthy housekeeping, not a problem.
direct_reclaims grows when a worker thread could not wait for the background path. The maintainer was too slow, so the worker reached into the LRU itself. This adds latency to that SET and consumes worker capacity that should be serving other commands. Some inline actions reclaim expired items; others evict live data. Either way, direct_reclaims climbing means the background path cannot keep up.
evictions grows when memcached ran out of expired items to reclaim and had to drop a valid one. This is the moment useful data leaves the cache.

direct_reclaims is a leading indicator because it fires before evictions necessarily spikes. A worker thread reclaiming inline is already strained, even if it happens to find an expired item this time. Sustained direct_reclaims means the next SET may not be so lucky, and evictions will follow.

Where it shows up in production

Reading the four counters as ratios and rates maps onto a small number of operating modes.

Healthy steady state. reclaimed grows roughly in step with cmd_set, crawler_reclaimed grows steadily, direct_reclaims is flat at zero, and evictions is zero or a trickle of genuinely cold items. The reclaimed / (reclaimed + evictions) ratio is near 1. TTLs are well-tuned and the working set fits.

TTLs too long, or cache undersized. reclaimed is low, evictions is high, and direct_reclaims may still be zero because the maintainer is keeping up but there is nothing expired to reclaim. Every SET forces an eviction of a live item. evicted_time confirms: if evicted items are young, the cache is thrashing live data.

Acute write spike. direct_reclaims spikes alongside evictions. The SET rate overwhelmed the maintainer’s ability to free expired items, so workers started reclaiming inline. Latency on affected SETs climbs because the worker thread is doing eviction work instead of serving commands. This is the signature of a write-heavy workload hitting a cache sized for reads.

Deployments without lru_maintainer. direct_reclaims is not exposed because the background path is disabled, even where the underlying counter exists in older 1.4.x code. reclaimed still works, but only lazily: expired items sit in the LRU until a SET needs their slot. crawler_reclaimed is zero unless the crawler was explicitly enabled. On these deployments, the reclaimed/evictions ratio is the only reclamation signal you have, and it undercounts because the crawler is not proactively freeing anything.

Per-slab-class divergence. All four counters are also available per slab class via stats items:

echo "stats items" | nc <host> 11211

Look for items:<class>:reclaimed, items:<class>:crawler_reclaimed, items:<class>:direct_reclaims, and items:<class>:evicted. Where upstream exposes direct_reclaims at all, it is available both globally and per class. One slab class can be in acute pressure with high direct_reclaims and low evicted_time while another class is idle with free chunks. Global counters hide this. When the global ratio looks borderline, break it down by class before deciding the cache is uniformly full.

Tradeoffs and common misuses

Absolute values are meaningless without rates. Both are cumulative counters since process start. Always compute deltas over a window. A cache running for a month with 10 million reclaimed and 1000 evictions is healthy; a cache that added 1000 evictions in the last minute is not.

The ratio hides scale. reclaimed / (reclaimed + evictions) = 0.99 looks great, but if the absolute evictions rate is high enough to depress hit ratio, the cache is still losing useful data. Pair the ratio with hit ratio and evicted_time.

direct_reclaims is invisible on some exporters. The Prometheus memcached exporter did not expose direct_reclaims until v0.15.0 (released 2024-11-08). If you rely on an older exporter, this leading indicator is missing, and you will not see acute pressure until evictions and hit ratio degrade. Verify your exporter exports this counter before relying on it.

reclaimed requires the crawler to be meaningful at scale. Without the LRU crawler enabled, expired items are only reclaimed lazily when a SET needs the slot. reclaimed still increments, but the background cleanup that prevents the cache from accumulating dead weight does not happen. The daemon default is lru_crawler off, so verify that your startup configuration enables it.

Evictions are not always bad. A cache sized slightly below the working set will evict genuinely cold items as a matter of course. If hit ratio is stable and evicted_time is high, the evictions are healthy turnover. Alerting on any non-zero evictions rate produces noise and leads to over-provisioning. See Memcached evictions climbing for the full treatment.

Signals to watch in production

Signal	Why it matters	Warning sign
`reclaimed` rate	SETs reusing expired slots instead of evicting	Rate collapses while `cmd_set` holds: TTLs too long or no expired items to find
`evictions` rate	Valid items being removed to make room	Any sustained non-zero rate; correlate with hit ratio and `evicted_time`
`reclaimed / (reclaimed + evictions)`	Share of room-making that is healthy turnover	Dropping toward 0.5 or below while evictions climb
`crawler_reclaimed` rate	Background crawler keeping expired items cleared	Flat at zero while evictions are active: crawler disabled or stuck
`direct_reclaims` rate	Worker threads reclaiming inline	Any sustained non-zero rate; spikes track acute write pressure
`evicted_time` (per slab class)	Seconds since the last access of the most recently evicted item	Under 300 seconds with sustained evictions: recently active data being thrashed
`cmd_set` rate	Write load driving demand for new chunks	Sudden spike correlating with `direct_reclaims` spike

How Netdata helps

Netdata collects reclaimed, evictions, crawler_reclaimed, and direct_reclaims per second, so ratios and rates reflect real workload changes rather than slowly-averaged cumulative counters.
Correlating direct_reclaims against cmd_set rate in the same view makes it obvious when a write spike forces inline reclaims, before evictions and hit ratio follow.
Per-slab visibility from stats items shows which size class is under pressure when global counters look borderline.
Pairing these counters with evicted_time and hit ratio in one dashboard turns the four reclamation signals into a single pressure picture.