Memcached does not remove items when their TTL expires. An expired item holds its slab slot until something touches it: a client request, an eviction scan, or the LRU crawler. The crawler is a background thread that walks LRU chains, finds expired items, and frees their memory before eviction pressure forces the issue.

A working crawler keeps slab slots available for new writes without evicting live data. Every expired item it reclaims is one fewer item evicted under pressure. Without it, expired items that are never accessed and never reach the eviction tail waste memory indefinitely.

This article covers the two primary counters (crawler_reclaimed and crawler_items_checked), the related lrutail_reflocked diagnostic, how lazy and active expiry interact, and how to tell whether the crawler is healthy, disabled, or stuck.

Lazy and active expiry coexist

Expired items leave memcached through three paths:

PathTriggerCounter
Lazy expiry on accessClient GET, SET, or touch hits an expired itemreclaimed (if a SET reuses the slot)
Active expiry by crawlerBackground crawler scan finds an expired itemcrawler_reclaimed
Eviction-driven reclaimMemory pressure forces an LRU tail scanreclaimed or evictions

reclaimed counts store operations that reused an expired item’s slot rather than allocating fresh memory. crawler_reclaimed counts items the crawler freed proactively. Both can be nonzero simultaneously, and both should be on a healthy cache with expiring items.

A well-tuned cache with an active crawler shows a healthy reclaimed rate, a modest crawler_reclaimed rate, and low or zero evictions. A cache where evictions climbs while crawler_reclaimed sits flat at zero is the signature of a crawler that is disabled, stuck, or unable to keep up.

How the crawler works

flowchart TD
    A["Item TTL expires"] --> B{"Client accesses it?"}
    B -- "yes" --> C["Lazy expiry: reclaimed++"]
    B -- "no" --> D{"Crawler enabled?"}
    D -- "yes" --> E["Crawler reclaims: crawler_reclaimed++"]
    D -- "no" --> F{"Eviction pressure?"}
    F -- "yes" --> G["Eviction scan: reclaimed++ or evicted++"]
    F -- "no" --> H["Wastes memory until touched"]

In memcached 1.5.0 and later with the default -o modern mode, each slab class has a segmented LRU with four tiers: HOT, WARM, COLD, and TEMP. The LRU maintainer thread moves items between tiers based on access patterns. The LRU crawler is a separate thread that walks these same queues looking for expired items.

The crawler inserts crawler items at the tail of each sub-LRU in every slab class, then walks backward from tail to head, examining each item’s expiration time.

Two settings control crawler pacing:

  • lru_crawler sleep sets the delay between items during a scan. The default is 100 microseconds.
  • lru_crawler tocrawl sets the maximum items to examine per slab per run.

These prevent the crawler from consuming excessive CPU on large caches.

The crawler and maintainer are distinct threads. The maintainer moves live items between tiers. The crawler frees expired items. Their stats are reported separately.

Checking whether the crawler is enabled

# Crawler enabled in settings?
echo "stats settings" | nc -q1 localhost 11211 | grep lru_crawler

# Is a crawl currently running?
echo "stats" | nc -q1 localhost 11211 | grep lru_crawler_running

# Crawler work counters
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT crawler_(reclaimed|items_checked)"

stats settings reports lru_crawler as yes or no. If no, the crawler is disabled and crawler_reclaimed stays at zero regardless of how many items expire.

Version history and defaults

The crawler’s default state has changed across versions:

  • 1.4.18: Introduced experimentally. Enable explicitly via lru_crawler enable at runtime or -o lru_crawler at startup.
  • 1.4.23: Recommended startup flags become -o lru_maintainer,lru_crawler.
  • 1.4.33: Improvements to how the crawler’s default background job launches.
  • 1.5.0: Segmented LRU and LRU maintainer become default with -o modern. The crawler works with segmented LRU.
  • 1.6.9: The background item crawler thread walks LRUs and actively reclaims expired items by default.

On any version, -o no_modern disables the crawler. stats settings will show lru_crawler: no.

On AWS ElastiCache for Memcached, the crawler is controlled via the parameter group, not command-line flags. Check the parameter group setting rather than assuming the upstream default applies.

When the crawler is stuck

If lru_crawler shows yes but crawler_reclaimed stays at zero while evictions are active, the crawler may be stuck. Check lru_crawler_running: if it stays true indefinitely without crawler_reclaimed increments, the crawler thread may be wedged.

A stuck crawler is rare. More commonly, crawler_reclaimed is zero because the cache has no expiring items (all items have long or no TTLs), or because the crawler is disabled.

lrutail_reflocked: items the crawler cannot reclaim

lrutail_reflocked (since 1.4.21) counts times the server found an item at the LRU tail with an active reference (refcount greater than zero). An item with a positive refcount is being read or modified by a worker thread and cannot be freed or evicted.

When this counter climbs, items at the eviction end of the LRU are hot: they are accessed frequently enough that the server cannot reclaim their memory. The internal code walks up the LRU tail looking for the next unlocked item, bounded by a limit to avoid hangs.

High lrutail_reflocked with active evictions means the cache is undersized for its working set, or large-value reads are holding tail items open during eviction scans.

tail_repair_time: the dangerous escape hatch

The -o tail_repair_time option forcefully reclaims an LRU tail item whose refcount has leaked (stuck at a nonzero value due to a bug, not a legitimate read). It is disabled by default and documented as dangerous. Enabling it can free stuck tail items, but it can also free items legitimately in use, causing data corruption. Do not enable it without understanding the risk.

metadump: inspecting what the crawler sees

lru_crawler metadump dumps metadata for items in the cache: key, expiration time, last accessed time, CAS, fetch status, class, and size. It is the most direct way to inspect what the crawler would find.

# Dump metadata for all items (expensive on large caches)
echo "lru_crawler metadump all" | nc -q1 localhost 11211 | head -20

Two operational warnings:

  • Cannot be pipelined. The protocol returns ERROR cannot pipeline other commands before metadump if you send other commands first on the same connection.
  • Connection close crash (pre-1.6.26). Prior to 1.6.26, closing the client connection early during lru_crawler metadump could crash the server. Let the dump complete before closing the connection on older versions.

On large caches, expect significant output and server overhead. Run it during low-traffic periods or on a non-production instance.

Signals to watch

SignalWhy it mattersWarning sign
crawler_reclaimedItems the crawler freed proactivelyFlat at zero while evictions is active
crawler_items_checkedItems the crawler examinedZero means the crawler is not scanning
crawler_reclaimed / crawler_items_checkedFraction of scanned items that were expiredVery low ratio with high scan rate means TTLs are longer than scan frequency expects
lrutail_reflockedTail items locked, cannot be reclaimedClimbing rate under eviction pressure
reclaimedLazy and eviction-driven slot reuseHigh with zero crawler_reclaimed suggests crawler is off but lazy expiry works
evictionsLive items forcibly removedNon-zero while crawler_reclaimed is zero is the crawler-problem signature
lru_crawler (stats settings)Whether the crawler is enabledno means expired items only leave via lazy expiry or eviction
lru_crawler_runningWhether a crawl is in progressStuck true with no crawler_reclaimed progress

The key diagnostic pattern: if evictions climbs and crawler_reclaimed is flat at zero, check lru_crawler in stats settings first. If no, the crawler is disabled. If yes, check lru_crawler_running and crawler_items_checked to see whether it is actually scanning.

How Netdata helps

Netdata collects crawler_reclaimed and crawler_items_checked per second, so you see the crawler’s work rate rather than cumulative counters. A flat line on crawler_reclaimed while evictions rises is visible on a correlated timeline.

Correlating crawler stats with evictions and memory utilization makes the disabled-crawler signature obvious: evictions climbing, crawler_reclaimed flat, reclaimed flat or modest. lrutail_reflocked tracked alongside eviction rate reveals when tail-lock contention contributes to pressure versus when the cache is simply full of live data.

Anomaly detection on crawler_reclaimed rate flags sudden drops to zero without a static threshold, which matters because a healthy crawler’s rate varies with TTL distribution. Polling stats settings surfaces a lru_crawler: no misconfiguration before it becomes an incident, especially after a restart with changed startup flags.