Memcached slab_automove and slab_reassign: rebalancing pages between classes

Memcached’s slab allocator divides its memory budget into 1MB pages and permanently assigns each page to a slab class based on item size. Once assigned, a page traditionally stayed locked to that class forever. If the workload’s item-size distribution shifted after deployment, you got slab calcification: one class full and evicting while others sat idle with free chunks. Global memory utilization looked healthy while cache effectiveness collapsed for the saturated size range.

Two mechanisms address this. slab_reassign (introduced in 1.4.11) moves individual pages between slab classes at runtime. slab_automove (also 1.4.11) automates that decision with a background thread that watches eviction patterns and relocates pages accordingly. Mode 1 is the conservative mover; mode 2 is aggressive and not recommended for sustained use.

These are not free. Page moves are destructive to the source class. The automover cannot help when every slab class is evicting. And the underlying cause, a mismatch between your item-size distribution and the slab class boundaries, often requires a structural fix via the growth factor (-f) on the next restart. This article covers when to rely on runtime rebalancing, when it cannot help, and when you need the restart-time lever instead.

Both are enabled by default since 1.5.0 (slab_reassign on, slab_automove mode 1) and remain so through current 1.6.x; on 1.4.x both were opt-in. The stats settings output shows the effective state.

What slab_reassign and slab_automove actually do

The slab allocator pre-allocates a fixed memory budget (the -m flag, default 64MB) at startup. It divides this into 1MB pages and assigns each page to a slab class. Each class stores items within a specific chunk-size range. Class 1 handles items up to 96 bytes, class 2 up to 120 bytes, class 3 up to 152 bytes, growing by the configurable factor -f (default 1.25).

Before slab_reassign, a page assigned to a class stayed there permanently. If the application’s serialization format changed, or a new feature started caching differently-sized values, the old slab classes held pages they no longer needed while the new size range starved. The only fix was a restart, which wiped all cached data.

slab_reassign breaks that permanence. It lets you, or the automover, move a 1MB page from one slab class to another at runtime. The move is destructive: every item stored in that page within the source class is evicted. Since 1.4.25, the automover attempts to rescue still-valid items from the source page by relocating them to other pages in the same class before the move completes. Items that cannot be rescued are dropped. Freed memory can also be reclaimed into a global page pool and reassigned to new slab classes as needed.

How the automover decides

The automover runs in a background thread and evaluates slab classes on a sliding window. The window tracks eviction activity per class over time.

Mode 0 (disabled): No automatic page movement. Pages stay where they were initially assigned.

Mode 1 (conservative, the default since 1.5.0): The automover evaluates every second over a rolling window of slab_automove_window samples (default 10; -o slab_automove_window). A class becomes a move target if it is evicting and its evictions filled more than half the window’s samples, or it drew over a quarter of all window evictions. The source class is the one with the oldest average item age holding more than 2 pages (MIN_PAGES_FOR_SOURCE = 2 in the source). A page moves when the target’s item age is below slab_automove_ratio (default 0.80) of the source’s age. Spare capacity — more than 2.5 pages’ worth of free chunks — is returned to a global page pool instead. This is deliberately slow. Third-party distributions rarely change this default, but stats settings shows the effective mode.

Mode 2 (aggressive): Moves a page on every eviction event. It reacts instantly to pressure but causes significant latency jitter because each move locks items, rescues what it can, and drops the rest. Not recommended for long-term use.

flowchart TD
    A["Automover evaluates every second"] --> B{"Target class: evicting with young item age?"}
    B -->|yes| C{"Source class: oldest average age, > 2 pages?"}
    B -->|no| Z["No action this cycle"]
    C -->|found| D["Move 1 page from source to target"]
    C -->|"all classes evicting"| E["Cannot help: no donor available"]
    D --> F["Rescue valid items to other pages in source class"]
    F --> G["Drop non-rescued items"]
    G --> H["Reassign page to target class"]

The key constraint is that the automover can only take pages from classes that are not themselves under eviction pressure. If every active slab class is evicting, there is no donor, and the automover has nothing to do. This is the most common reason operators enable rebalancing and see no effect.

Enabling and tuning slab rebalancing

Check whether the automover is active:

# Check automove mode and slab_reassign setting
echo "stats settings" | nc localhost 11211 | grep slab

Enable conservative mode at runtime (no restart needed):

# Enable automove mode 1 (conservative)
echo "slabs automove 1" | nc localhost 11211

Disable it:

# Disable automove
echo "slabs automove 0" | nc localhost 11211

For a manual one-shot page move, identify the source and target slab classes from stats slabs, then issue the reassign command directly:

# Identify which classes have free pages and which are saturated
echo "stats slabs" | nc localhost 11211 | grep -E "(total_pages|free_chunks|used_chunks)"
echo "stats items" | nc localhost 11211 | grep -E "(evicted|evicted_time)"

# Move one page from class 5 to class 12 (destructive to class 5 items in that page).
# The command returns a status string; check it before assuming the move happened.
echo "slabs reassign 5 12" | nc localhost 11211

The response is OK when the move is scheduled. Other responses carry a message: BUSY (a move is already in progress), BADCLASS (invalid class id), NOSPARE (source has no spare pages), NOTFULL (destination class must be full), UNSAFE (source cannot move a page right now), or SAME (source and destination identical).

Manual reassignment is useful when the automover is off or when you need to force a specific move the automover would not make on its own. The tradeoff is the same: items in the source page are evicted unless they can be rescued. For targeted relief during an active slab imbalance incident, a manual move can be faster than waiting for the conservative automover to act. Do not run a manual reassign against a class that is itself evicting; you are stealing memory from a saturated range.

When slab rebalancing cannot help

Every slab class is evicting. The automover’s algorithm requires a donor class with zero recent evictions. If all classes are under pressure, there is no page to take. This typically means the working set genuinely exceeds the allocated memory, and the fix is more RAM or fewer cached items, not page shuffling.

The automover anti-favours a specific class. A documented production issue showed the automover persistently starving one slab class (memcached issue #677, observed on 1.5.12 in May 2020 and closed as fixed in September 2020 after the automover was reworked into the window-and-ratio algorithm described above). The automover would donate pages away from that class to serve other classes, then the class would evict heavily. Manual reassignment was ineffective because the automover immediately took the page back. The counters told the story: slab_reassign_busy_items (2.4 billion) and slab_reassign_evictions_nomem (2.4 billion) ran into the billions, indicating the automover was constantly attempting moves that failed. The workaround was to explicitly steal pages from the class with the most pages, or disable the automover entirely and manage moves manually.

Reassignment is slow relative to workload changes. The automover moves one page per 10 seconds. If a workload shift causes sudden concentrated pressure on a class that needs many pages, the automover cannot keep up. A single page move under contention can require a large number of request cycles to complete while the rescuer waits for locked items to drain. For gradual drift, this is fine. For sudden shifts, it is too slow.

Old versions have a worse automover. The automover in 1.4.x is significantly less effective than in 1.5.x and 1.6.x. Operators running older versions who observe that automove “doesn’t seem to do anything” should consider upgrading before concluding the feature is useless. The 1.4.25 release brought major improvements (global page pool, item rescue), and 1.5.0+ refined the algorithm further.

The structural fix: growth factor (-f)

When slab imbalance recurs after every restart, or when the automover cannot keep up with a persistent distribution mismatch, the root cause is usually the chunk-size boundaries determined by the growth factor.

The -f flag (default 1.25) controls how slab class boundaries grow. With -f 1.25, class boundaries are roughly 96, 120, 152, 192 bytes, and so on. Lower values (for example 1.10 or 1.15) produce more classes with finer granularity between sizes, reducing internal fragmentation (items stored in oversized chunks) but creating more classes competing for the same fixed pool of pages. Higher values produce fewer classes, which means each class gets more pages but items waste more space within their chunks.

# Check current growth factor
echo "stats settings" | nc localhost 11211 | grep "STAT growth_factor"

Changing -f requires a restart, which means total data loss. The decision should be based on analysis of the item-size distribution under the current workload. Use stats slabs to identify which classes are perpetually starved and which have excess pages. If the starved classes are adjacent (for example, 120-byte and 152-byte items both evicting because they share a boundary), a lower growth factor creates more classes in that range and spreads the load.

The tradeoff is straightforward: lower -f reduces fragmentation at the cost of more class competition for pages. With slab_automove enabled, the server can redistribute pages over time to match the new class layout, but the initial allocation on restart will reflect the new boundaries.

Signals to watch

How Netdata helps

Netdata surfaces per-slab eviction counts, evicted_time, free_chunks, and used_chunks as per-second time series, which lets you see slab imbalance developing before it becomes an incident.

• Correlate slabs_moved with per-slab eviction rates to verify the automover is actually addressing the imbalance, or to confirm it is stuck because all classes are evicting.
• Track slab_reassign_busy_items and slab_reassign_rescues rates to distinguish productive page moves (high rescues) from destructive ones (high busy_items, low rescues).
• Watch per-slab evicted_time trends to detect when evictions shift from healthy cold-item turnover to harmful thrashing of recently-accessed data.
• Alert on slab_automove setting changes to catch unexpected disabling of the automover.
• Correlate automove activity with client-observed latency spikes to identify whether mode 2 aggressive moves are causing jitter.
• Compare global memory utilization against per-slab utilization to surface the gap that makes slab calcification invisible at the aggregate level.

Related guides

• Memcached evicted_unfetched and expired_unfetched: caching data nobody ever reads
• Memcached cas_badval climbing: check-and-set contention and lost updates
• Memcached connection refused: telling a dead process from a hung or full one
• Memcached evicted_time low: distinguishing healthy turnover from cache thrash
• Memcached eviction cascade: when a full cache overloads the backend
• Memcached evictions climbing: the cache is full and discarding live data
• Memcached high miss rate: separating cold start, new key patterns, and memory pressure
• How Memcached actually works in production: a mental model for operators
• Memcached incr/decr misses: evicted counters that silently break rate limiters and locks
• Memcached hit ratio dropping: reading get_hits, get_misses, and cache effectiveness
• Memcached memory utilization: bytes vs limit_maxbytes and why the global number lies
• Memcached monitoring checklist: the signals every production cache needs