$ guides / memcached / memcached-lru-crawler ▌

Operations Guides

Memcached LRU crawler: crawler_reclaimed, disabled crawlers, and lazy expiry

Memcached does not remove items when their TTL expires. An expired item holds its slab slot until something touches it: a client request, an eviction scan, or the LRU crawler. The crawler is a background thread that walks LRU chains, finds expired items, and frees their memory before eviction pressure forces the issue.

A working crawler keeps slab slots available for new writes without evicting live data. Every expired item it reclaims is one fewer item evicted under pressure. Without it, expired items that are never accessed and never reach the eviction tail waste memory indefinitely.

This article covers the two primary counters (crawler_reclaimed and crawler_items_checked), the related lrutail_reflocked diagnostic, how lazy and active expiry interact, and how to tell whether the crawler is healthy, disabled, or stuck.

Lazy and active expiry coexist

Expired items leave memcached through three paths:

Path	Trigger	Counter
Lazy expiry on access	Client GET, SET, or touch hits an expired item	`reclaimed` (if a SET reuses the slot)
Active expiry by crawler	Background crawler scan finds an expired item	`crawler_reclaimed`
Eviction-driven reclaim	Memory pressure forces an LRU tail scan	`reclaimed` or `evictions`

reclaimed counts store operations that reused an expired item’s slot rather than allocating fresh memory. crawler_reclaimed counts items the crawler freed proactively. Both can be nonzero simultaneously, and both should be on a healthy cache with expiring items.

A well-tuned cache with an active crawler shows a healthy reclaimed rate, a modest crawler_reclaimed rate, and low or zero evictions. A cache where evictions climbs while crawler_reclaimed sits flat at zero is the signature of a crawler that is disabled, stuck, or unable to keep up.

How the crawler works

flowchart TD
    A["Item TTL expires"] --> B{"Client accesses it?"}
    B -- "yes" --> C["Lazy expiry: reclaimed++"]
    B -- "no" --> D{"Crawler enabled?"}
    D -- "yes" --> E["Crawler reclaims: crawler_reclaimed++"]
    D -- "no" --> F{"Eviction pressure?"}
    F -- "yes" --> G["Eviction scan: reclaimed++ or evicted++"]
    F -- "no" --> H["Wastes memory until touched"]

In memcached 1.5.0 and later with the default -o modern mode, each slab class has a segmented LRU with HOT, WARM, and COLD tiers. The optional TEMP tier is off unless temporary_ttl is enabled. The LRU maintainer thread moves items between the active tiers based on access patterns. The LRU crawler is a separate thread that walks these same queues looking for expired items.

The crawler inserts crawler items at the tail of each selected sub-LRU, then walks backward from tail to head. For its automatic expiration scans it records a histogram of remaining TTLs in one-minute buckets and uses that history to choose the next scan interval.

Two settings control crawler pacing:

lru_crawler_sleep sets a microsecond sleep after every crawls_persleep items; the default sleep is 100 microseconds and the default batch is 1,000 items.
lru_crawler tocrawl sets the maximum items to examine per slab per run.

These prevent the crawler from consuming excessive CPU on large caches.

The crawler and maintainer are distinct threads. The maintainer moves live items between tiers. The crawler frees expired items. Their stats are reported separately.

Checking whether the crawler is enabled

# Crawler enabled in settings?
echo "stats settings" | nc -q1 localhost 11211 | grep lru_crawler

# Is a crawl currently running?
echo "stats" | nc -q1 localhost 11211 | grep lru_crawler_running

# Crawler work counters
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT crawler_(reclaimed|items_checked)"

stats settings reports lru_crawler as yes or no. If no, the crawler is disabled and crawler_reclaimed stays at zero regardless of how many items expire.

Version history and defaults

The crawler’s default state has changed across versions:

1.4.18: Introduced experimentally. Enable explicitly via lru_crawler enable at runtime or -o lru_crawler at startup.
1.4.23: Recommended startup flags become -o lru_maintainer,lru_crawler.
1.4.31: The default background job launches less aggressively, and crawler rate-limit sleeps now take effect.
1.5.0: Segmented LRU and LRU maintainer become default with -o modern. The crawler works with segmented LRU.
1.6.9: The background item crawler thread walks LRUs and actively reclaims expired items by default. This is a separate default-on decision from the 1.4.31 launch refinements.

On versions that support -o no_modern, that flag disables the crawler. stats settings will show lru_crawler: no.

On AWS ElastiCache for Memcached, the crawler is controlled via the parameter group, not command-line flags. Check the parameter group setting rather than assuming the upstream default applies.

When the crawler is stuck

If lru_crawler shows yes but crawler_reclaimed stays at zero while evictions are active, the crawler may be stuck. Check lru_crawler_running: if it stays true indefinitely without crawler_reclaimed increments, the crawler thread may be wedged.

A stuck crawler is rare. More commonly, crawler_reclaimed is zero because the cache has no expiring items (all items have long or no TTLs), or because the crawler is disabled.

lrutail_reflocked: items the crawler cannot reclaim

lrutail_reflocked (since 1.4.21) counts times the server found an item at the LRU tail with an active reference (refcount greater than zero). An item with a positive refcount is being read or modified by a worker thread and cannot be freed or evicted.

When this counter climbs, items at the eviction end of the LRU are hot: they are accessed frequently enough that the server cannot reclaim their memory. The internal code walks up the LRU tail looking for the next unlocked item; 1.4.21 through 1.6.x use a five-item try limit.

High lrutail_reflocked with active evictions means the cache is undersized for its working set, or large-value reads are holding tail items open during eviction scans.

tail_repair_time: the dangerous escape hatch

The -o tail_repair_time option forcefully reclaims an LRU tail item whose refcount has leaked (stuck at a nonzero value due to a bug, not a legitimate read). It is disabled by default and documented as dangerous. Enabling it can free stuck tail items, but it can also free items legitimately in use, causing data corruption. Do not enable it without understanding the risk.

metadump: inspecting what the crawler sees

lru_crawler metadump dumps metadata for items in the cache: key, expiration time, last accessed time, CAS, fetch status, class, and size. It is the most direct way to inspect what the crawler would find.

# Dump metadata for all items (expensive on large caches)
echo "lru_crawler metadump all" | nc -q1 localhost 11211 | head -20

Two operational warnings:

Cannot be pipelined. The protocol returns ERROR cannot pipeline other commands before metadump if you send other commands first on the same connection.
Connection close crash (pre-1.6.26). Prior to 1.6.26, closing the client connection early during lru_crawler metadump could crash the server; 1.6.26 fixed the corruption. Let the dump complete before closing the connection on older versions.

On large caches, expect significant output and server overhead. Run it during low-traffic periods or on a non-production instance.

Signals to watch

Signal	Why it matters	Warning sign
`crawler_reclaimed`	Items the crawler freed proactively	Flat at zero while `evictions` is active
`crawler_items_checked`	Items the crawler examined	Zero means the crawler is not scanning
`crawler_reclaimed / crawler_items_checked`	Fraction of scanned items that were expired	Very low ratio with high scan rate means TTLs are longer than scan frequency expects
`lrutail_reflocked`	Tail items locked, cannot be reclaimed	Climbing rate under eviction pressure
`reclaimed`	Lazy and eviction-driven slot reuse	High with zero `crawler_reclaimed` suggests crawler is off but lazy expiry works
`evictions`	Live items forcibly removed	Non-zero while `crawler_reclaimed` is zero is the crawler-problem signature
`lru_crawler` (stats settings)	Whether the crawler is enabled	`no` means expired items only leave via lazy expiry or eviction
`lru_crawler_running`	Whether a crawl is in progress	Stuck `true` with no `crawler_reclaimed` progress

The key diagnostic pattern: if evictions climbs and crawler_reclaimed is flat at zero, check lru_crawler in stats settings first. If no, the crawler is disabled. If yes, check lru_crawler_running and crawler_items_checked to see whether it is actually scanning.

How Netdata helps

Netdata collects crawler_reclaimed and crawler_items_checked per second, so you see the crawler’s work rate rather than cumulative counters. A flat line on crawler_reclaimed while evictions rises is visible on a correlated timeline.

Correlating crawler stats with evictions and memory utilization makes the disabled-crawler signature obvious: evictions climbing, crawler_reclaimed flat, reclaimed flat or modest. lrutail_reflocked tracked alongside eviction rate reveals when tail-lock contention contributes to pressure versus when the cache is simply full of live data.

Anomaly detection on crawler_reclaimed rate flags sudden drops to zero without a static threshold, which matters because a healthy crawler’s rate varies with TTL distribution. Polling stats settings surfaces a lru_crawler: no misconfiguration before it becomes an incident, especially after a restart with changed startup flags.