Memcached evicting with memory to spare: the slab-class imbalance trap

Evictions are climbing on your memcached instance. Global memory is at 55% of limit_maxbytes. The natural response is to add memory, restart the daemon, or hunt for an eviction storm. None of those help here, because the cache is not out of memory in aggregate. One slab class is at 100% capacity and discarding recently-active items, while several other classes sit mostly empty.

This is slab-class imbalance. The global bytes and limit_maxbytes counters aggregate over a partitioned allocator, so a healthy-looking global number can hide a saturated class. The fix is almost never “add memory”: new pages go to whichever class happens to be allocating next, not to the class that is evicting.

This article covers detecting the imbalance with stats slabs and stats items, distinguishing harmful eviction from healthy turnover, and relieving pressure without a full cache wipe.

What this means

Memcached does not allocate from a single free pool. At startup it pre-allocates a fixed block (-m, default 64 MB), divides it into 1 MB pages, and assigns pages to slab classes. Each class owns items in a specific chunk-size bucket, growing by a factor (-f, default 1.25). Items do not cross class boundaries: a 200-byte object cannot borrow space from a class sized for 96-byte objects.

Memory pressure is therefore per-class. A workload dominated by 1 KB JSON blobs saturates one or two classes, while 96-byte session tokens have plenty of room. The global bytes counter sums every class, so it can read 60% of limit_maxbytes while the active class is pinned at 100% used_chunks, 0 free_chunks, and evicting items the application still wants.

The usual operator instincts backfire:

• Adding memory (-m higher on restart) allocates new pages to whichever slab class receives the next SET. The saturated class does not necessarily receive them. Idle classes grow instead. Runtime memory-limit change: memcached does support cache_memlimit <MB> at runtime (introduced in 1.4.27), but this adjusts the global -m budget rather than a specific slab class. New pages freed by lowering the limit come from whichever class the allocator decides, not specifically from the class that is under pressure, so this does not solve a slab-class imbalance.
• Restarting re-partitions the cache from cold. This works, but it is a full cache wipe and triggers a thundering herd on the backend. It is a blunt instrument for what is usually a sustained workload-shape problem.
• Doing nothing works if slab_automove is enabled and the algorithm keeps up. It fails when automove is disabled or when the imbalance outruns the conservative mode-1 algorithm.

flowchart TD
    A[Workload item-size shift] --> B[One slab class fills]
    B --> C[Class at 100% used_chunks]
    C --> D[Evicting recently-active items]
    D --> E[Hit ratio drops for that size range]
    A --> F[Other classes stay idle]
    F --> G[Global bytes sits at 50-70%]
    G --> H[Operator adds memory]
    H --> I[New pages go to idle classes]
    I --> D

The loop in the diagram is the trap. Adding memory without changing the partition feeds the imbalance instead of relieving it.

Common causes

Quick checks

All read-only and safe against a production instance. None mutate cache state.

# Global memory and eviction overview
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT (bytes |limit_maxbytes|evictions|curr_items)"

# Per-slab allocation: look for a class with free_chunks 0 next to classes with free_chunks > 0
echo "stats slabs" | nc -q1 localhost 11211 | grep -E "(used_chunks|free_chunks|total_pages|chunk_size)"

# Per-slab item health: low evicted_time in an evicting class is the smoking gun
echo "stats items" | nc -q1 localhost 11211 | grep -E "(evicted|evicted_time|age|number)"

# Confirm whether slab_automove is enabled
echo "stats settings" | nc -q1 localhost 11211 | grep "slab_automove"

# Recent restart history, in case the imbalance started at boot
echo "stats" | nc -q1 localhost 11211 | grep "STAT uptime"

The diagnostic signature is the pair: a class with free_chunks == 0 and a non-zero evicted counter, sitting alongside classes with significant free_chunks and zero evictions. Add a low evicted_time in the saturated class and the diagnosis is confirmed.

The -q flag is not portable across all nc implementations. On systems where it is unavailable, use -w 1 (connection timeout) instead.

How to diagnose it

1. Identify the saturated class. From stats slabs, find the slab class (the STAT <id>:<field> <value> lines, where <id> is the class number) where used_chunks equals total_chunks and free_chunks is 0. Note its chunk_size: that is the overflowing item-size bucket.
2. Confirm it is actively evicting. From stats items, look at the same class id. A non-zero evicted counter that is increasing between two samples means live items are being discarded. A stable evicted means the class is full but not under pressure; that state is fine.
3. Check eviction quality. In the same stats items output, read evicted_time for that class. This is the age, in seconds since last access, of the most recently evicted item. Values in hours or days mean the LRU is shedding cold items cleanly. Values under 300 seconds mean recently-active items are being evicted: that is the harmful case.
4. Verify other classes have room. Look for classes with free_chunks well above zero and zero evictions. Their existence proves the problem is partitioning, not aggregate memory. If every class is full, this is a different problem: the cache is genuinely undersized. See Memcached evictions climbing: the cache is full and discarding live data.
5. Check whether automove is helping. stats settings | grep slab_automove reports the current mode. Mode 0 is off. Mode 1 is the conservative algorithm. Mode 2 is aggressive and not recommended for sustained use. If automove is on but the imbalance persists, the algorithm may be unable to find a suitable donor class, or demand is outrunning the one-page-per-10-seconds pace.
6. Quantify the impact. Watch the global hit ratio while the saturated class evicts. A hit-ratio drop combined with stable aggregate memory is the user-visible symptom that justifies intervention.

Metrics and signals to monitor

Fixes

Each fix below targets a different cause. Pick based on whether the imbalance is acute (needs relief now) or structural (needs a config change on next restart).

Enable slab_automove at runtime

If slab_automove is at mode 0, turn it on without restarting:

# Enable conservative automove (mode 1). Safe for production.
echo "slabs automove 1" | nc -q1 localhost 11211

This activates a background thread that watches for classes with sustained evictions and moves 1 MB pages from idle classes. The algorithm is deliberately slow: at most one page every 10 seconds, and only from classes with zero recent evictions. It will not stabilize a violent imbalance instantly, but it relieves chronic drift.

Mode 2 (slabs automove 2) moves pages faster but is not recommended for sustained use; it can introduce jitter as pages are evicted from donor classes.

Manually reassign a page

For acute pressure on a single class, move a page directly from an idle class to the saturated one. Destructive to the donor class: all items on the reassigned page are evicted as part of the move.

# Move one page from class <src> to class <dst>. Evicts all items on the page in <src>.
echo "slabs reassign <src> <dst>" | nc -q1 localhost 11211

Pick a donor class with high free_chunks and zero evictions. The reassigned page is 1 MB, so choose a class with low-value items. The command returns a numeric status; retry with a different donor if it refuses the move.

Per the official protocol documentation, slabs reassign returns one of the following text responses rather than a numeric status code: OK (page scheduled to move), BUSY [message] (a page is already being processed; try again later), BADCLASS [message] (invalid class id), NOSPARE [message] (source class has no spare pages), NOTFULL [message] (destination class must be full to receive pages), UNSAFE [message] (source class cannot move a page right now), or SAME [message] (source and destination must differ).

This is a one-shot fix. It does not prevent the imbalance from returning if the workload keeps the saturated class pinned.

Adjust the growth factor on next restart

If stats slabs shows many near-empty classes between the saturated ones, the default growth factor of 1.25 may be spreading the workload across too many classes. A lower factor (for example -f 1.10) creates more finely-grained classes and reduces internal fragmentation, at the cost of more classes competing for pages.

This requires a restart, which means a full cache wipe. Schedule it for a low-traffic window and have a warming plan ready.

Restart to reset allocation

Restarting re-partitions memory from cold based on the items that arrive first. If the cache warmed with a different size distribution than steady-state traffic, this can break the calcification.

This is the bluntest fix. It loses all cached data and triggers a cold-start cascade on the backend. Prefer runtime automove or manual reassignment unless the imbalance is severe, persistent, and the above options have failed.

Prevention

• Verify slab_automove is enabled. Check with stats settings | grep slab_automove and enable mode 1 if it is at 0. Mode 1 is conservative enough for long-term use.
• Monitor per-slab, not just global. Track free_chunks, used_chunks, evicted, and evicted_time per class. The global bytes counter is necessary but not sufficient.
• Review item-size distribution after deploys. A serialization change is the most common trigger. Compare per-class used_chunks before and after the deploy.
• Size the growth factor to the workload. Default 1.25 is reasonable for mixed workloads but suboptimal when item sizes cluster tightly.
• Track mem_requested per class over time. Drift upward indicates item-size inflation, which eventually saturates the class.
• Watch direct_reclaims per slab. Non-zero values mean worker threads are evicting inline because the LRU maintainer cannot keep up. That is the moment chronic imbalance crosses into acute.

How Netdata helps

• Netdata collects per-slab metrics from stats slabs and stats items at per-second resolution, so the moment a class hits free_chunks == 0 you see it without polling by hand.
• The global bytes / limit_maxbytes ratio and per-slab saturation appear on the same dashboard, making the “looks fine globally, broken per-class” pattern obvious at a glance.
• Anomaly detection flags unusual per-slab eviction spikes even when global counters are still inside their normal range, which is exactly the slab-imbalance signature.
• Correlating per-slab evicted_time with backend latency or hit-ratio drops lets you confirm that the saturated class is the one harming users, not just the one with the most evictions.
• Per-slab direct_reclaims and lrutail_reflocked surface the moment the LRU maintainer falls behind, before the hit-ratio drop becomes visible to clients.

Related guides

• Memcached evicted_unfetched and expired_unfetched: caching data nobody ever reads
• Memcached cas_badval climbing: check-and-set contention and lost updates
• Memcached connection refused: telling a dead process from a hung or full one
• Memcached evicted_time low: distinguishing healthy turnover from cache thrash
• Memcached evictions climbing: the cache is full and discarding live data
• Memcached high miss rate: separating cold start, new key patterns, and memory pressure
• How Memcached actually works in production: a mental model for operators
• Memcached incr/decr misses: evicted counters that silently break rate limiters and locks
• Memcached hit ratio dropping: reading get_hits, get_misses, and cache effectiveness
• Memcached monitoring checklist: the signals every production cache needs
• Memcached monitoring maturity model: from survival to expert
• Memcached alive but not responding: the silent process hang