Memcached slab calcification: pages locked to the wrong item sizes after a workload shift

Hit ratio is dropping. Evictions are climbing. But global memory utilization sits at 50 to 70 percent, and adding memory does nothing. This is the signature of slab calcification: whole megabyte pages are locked into slab classes that serve item sizes the workload no longer produces, while the classes for the new sizes are starved and evicting actively used data.

Memcached’s slab allocator divides its memory budget into 1 MB pages, and each page is permanently assigned to a slab class. Each slab class serves items in a narrow size band, with the bands growing by a configurable factor (default 1.25). Once a page was handed to a class, the traditional behavior was that it never came back. When the item-size distribution shifts after a serialization change, a new application version, or an added field, memory stays trapped in classes serving the old sizes while the new sizes evict.

The defining diagnostic feature is that global memory is not at capacity. If your bytes stat sits at 60 percent of limit_maxbytes while evictions climb and hit ratio falls, do not add memory. Per-slab analysis will show one or two classes at 100 percent used_chunks with zero free_chunks and low evicted_time, while other classes sit idle with high free chunk counts and items that have not been touched in hours.

What this means

Slab calcification means the cache’s effective capacity is far smaller than its allocated memory, but only for specific item sizes. The global picture hides it. A serialization change that grows objects from 200 bytes to 500 bytes moves the working set from one set of slab classes to another, but the pages allocated to the old 200-byte classes do not follow. The result is a cache that behaves as if it is undersized for the new workload while half its memory serves a workload that no longer exists.

This is distinct from genuine undersizing. In true undersizing, global memory is at or near limit_maxbytes and every active slab class is evicting. In calcification, global memory has headroom and only specific classes are under pressure. Adding memory to a calcified cache often makes things worse in the short term, because the new pages get assigned based on incoming write traffic and may not land in the classes that need them.

The modern mitigation is slab_automove, default-on since memcached 1.5.0. It helps, but it is conservative by design and does not eliminate the problem. Understanding its limits matters during an active incident.

flowchart TD
    A["Workload shift: item sizes change"] --> B["New writes land in different slab classes"]
    B --> C["Old slab classes keep their pages"]
    B --> D["New slab classes fill and evict"]
    C --> E["Global bytes stays moderate"]
    D --> F["Hit ratio drops, evicted_time low"]
    E --> G["Global metrics look healthy"]
    F --> H["Per-slab stats reveal imbalance"]
    G --> H

Common causes

Quick checks

These commands are read-only. None of them alter cache state.

# Global memory headroom - if under 85% with evictions, suspect calcification over undersizing
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT (bytes |limit_maxbytes)"

# Per-slab memory allocation: total_pages, used_chunks, free_chunks per class
echo "stats slabs" | nc -q1 localhost 11211 | grep -E "(total_pages|used_chunks|free_chunks)"

# Per-slab eviction detail: evicted count, evicted_time, age (oldest item in seconds)
echo "stats items" | nc -q1 localhost 11211 | grep -E "(evicted|evicted_time|age)"

# Confirm automove setting (1 = conservative default, 2 = aggressive, 0 = off)
echo "stats settings" | nc -q1 localhost 11211 | grep "slab_automove"

# Check version - slab_automove became default-on in 1.5.0
echo "version" | nc -q1 localhost 11211

How to diagnose it

1. Confirm global memory is not the bottleneck. Pull bytes and limit_maxbytes. If bytes / limit_maxbytes is below 85 percent and evictions are non-zero, you are not dealing with simple undersizing. Proceed to per-slab analysis.

2. Identify the starving slab classes. From stats slabs, find classes where free_chunks is zero and used_chunks equals total_chunks. These are saturated. Cross-reference with stats items for the same class IDs: a saturated class with a rising evicted counter and low evicted_time (under 300 seconds) is actively thrashing useful data.

3. Identify the bloated slab classes. Find classes with high free_chunks, zero recent evictions, and high age (oldest item in hours or days). These hold pages the current workload does not need. They are the source pool for rebalancing.

4. Check the automove setting. Run stats settings and look for slab_automove. A value of 1 means the conservative automover is active. A value of 0 means it is off, which is the default only on versions before 1.5.0 or on instances launched with explicit legacy flags.

5. Check automove effectiveness. If automove is on but the imbalance persists, the automover may be unable to find a valid source class. Level 1 only takes pages from classes with zero recent evictions in the last decision window. If every class has some eviction activity, the automover stalls.

6. Correlate with the timeline. Match the onset of evictions to a deploy, a serialization change, or a traffic shift. The item-size distribution change is the root cause. Without addressing it, rebalancing is temporary.

Metrics and signals to monitor

Fixes

Confirm and enable slab_automove

On any instance running 1.5.0 or later with default options, slab_automove should already be at level 1. Verify with stats settings. If it is 0, enable it at runtime without restart:

# Enable conservative automove (default 10s decision window; only classes with no recent eviction activity)
printf "slabs automove 1\r\n" | nc -q1 localhost 11211

This is safe and non-disruptive. Level 1 evaluates a rolling 10-second window by default and selects source classes with no eviction activity in that window; successful page moves are rate limited to roughly one per second. The tradeoff is speed: for a large imbalance, full rebalancing can take minutes to hours.

Avoid level 2 for steady-state use unless you have benchmarked it. Level 2 is still supported in current upstream source: when a COLD-LRU eviction occurs, it triggers a slab reassign, trading cache contents for faster reallocation; level 1 chooses moves from age and eviction windows more conservatively.

Manually reassign a page for immediate relief

When the automover is too slow or cannot find a source class, move a page manually. First identify the source class (high free_chunks, zero evictions) and destination class (zero free_chunks, active evictions) from stats slabs.

# Move one page from slab class <src> to slab class <dst>
# Replace src and dst with actual class IDs from stats slabs
# Response: DONE on success; BUSY, BADCLASS, NOSPARE, or NOTFULL on error
printf "slabs reassign <src> <dst>\r\n" | nc -q1 localhost 11211

Manual reassign is more immediate than level-1 automove. The page mover tries to rescue valid items elsewhere, frees expired or flushed items in place, and can delete items that remain busy for too long. Unlike automove level 2, it does not specifically evict from the COLD-LRU tail.

Check the response and re-run stats slabs after each move to confirm the destination class gained a page and the source still has headroom. Slab reassign and automove were opt-in before 1.5.0; 1.5.0 made them defaults, so manual reassign is the practical option when both are disabled.

Adjust the growth factor on next restart

The -f flag (default 1.25) controls how chunk sizes grow across slab classes. A lower factor creates more, narrower classes, which reduces internal fragmentation but means more classes competing for pages. A higher factor creates fewer, wider classes with more padding waste.

If your workload concentrates items in a narrow size band that falls between two slab classes, a small factor adjustment can land more items cleanly in one class and reduce fragmentation. This requires a restart, so schedule it. Changing -f does not help an active calcification incident, but it prevents recurrence if the item-size distribution is known and stable.

Review the item-size distribution shift

The structural fix is to understand why the distribution changed. Common triggers:

• Serialization format change (JSON to protobuf, added fields, compression removed)
• New application version writing larger or differently-sized objects
• Different data types being cached after a feature launch
• Cache warming that loaded a different distribution than steady-state traffic

If the shift is permanent, the cache needs to reallocate to the new distribution. If it is transient (a bulk load, a one-time migration), rebalancing plus patience may suffice.

The nuclear option: restart

When the automover is stalled, manual reassign is not keeping up, and the imbalance is causing real user impact, a restart redistributes pages from scratch based on current write traffic. This is destructive: every item is lost, hit ratio drops to zero, and the backend absorbs the full load until the cache warms.

The memcached project’s ServerMaint documentation still says that a mismatch between evicting slabs and page distribution may require a restart so memory can be reallocated. Use it when rebalancing tools have failed and the backend can survive the cold start, or when you can pre-warm the cache.

Prevention

• Run 1.5.0 or later with default options. slab_automove at level 1 is the baseline mitigation. Verify it is active on every instance, including those launched from old machine images or container base layers.
• Monitor per-slab utilization, not just global memory. The single most common miss is trusting bytes / limit_maxbytes while one class thrashes. Track free_chunks and evicted_time per class.
• Correlate deploys with slab distribution changes. Capture stats slabs output before and after serialization changes or application version rolls. A shift in total_pages distribution that precedes an eviction spike is the early signal.
• Size the growth factor to the workload. If you know your item-size distribution, tune -f so the dominant sizes land cleanly in slab classes without excessive padding.
• Avoid mode 2 automove for steady-state. It is aggressive and can cause jitter. Reserve manual page reassign for targeted intervention.
• Keep the version current. The memory mover has had significant fixes: 1.6.41 fixed a slab-mover hang/crash, 1.6.42 fixed slabs reassign crashes, and 1.6.45 fixed an invalid automove-window crash plus a slab-picker off-by-one. Older versions have known slab-rebalancer bugs.

How Netdata helps

• Per-slab metrics collection from stats slabs and stats items surfaces used_chunks, free_chunks, evicted, and evicted_time per class, exposing the imbalance that global bytes hides.
• Correlation of global memory utilization with per-class eviction rates in a single view makes the calcification signature visible at a glance: moderate global usage alongside concentrated eviction pressure.
• Hit ratio tracking alongside eviction rate and evicted_time distinguishes healthy cold-item turnover from harmful thrash.
• Anomaly detection on per-slab eviction rates can flag a new class beginning to thrash before the global hit ratio drops enough to trigger a threshold alert.
• Version and settings visibility confirms whether slab_automove is active and whether the instance is running a version with known rebalancer bugs.

Related guides

• Memcached evicted_unfetched and expired_unfetched: caching data nobody ever reads
• Memcached cas_badval climbing: check-and-set contention and lost updates
• Memcached connection refused: telling a dead process from a hung or full one
• Memcached evicted_time low: distinguishing healthy turnover from cache thrash
• Memcached evictions climbing: the cache is full and discarding live data
• Memcached high miss rate: separating cold start, new key patterns, and memory pressure
• How Memcached actually works in production: a mental model for operators
• Memcached incr/decr misses: evicted counters that silently break rate limiters and locks
• Memcached hit ratio dropping: reading get_hits, get_misses, and cache effectiveness
• Memcached monitoring checklist: the signals every production cache needs
• Memcached monitoring maturity model: from survival to expert
• Memcached alive but not responding: the silent process hang