Memcached SERVER_ERROR out of memory storing object: stores rejected instead of evicting

When a memcached client receives SERVER_ERROR out of memory storing object, the daemon refused a SET because it could not allocate a chunk for the item. The store_no_memory counter increments. Under default operation, memcached evicts LRU items from the relevant slab class to make room. This error means eviction was either disabled or could not produce a free chunk for that allocation.

Two root causes. First, the instance was started with -M, which disables eviction entirely and returns errors instead of evicting. Second, slab-level exhaustion: a specific slab class has no free chunks and no pages can be reassigned to it. The second case is rarer on 1.5.0+ where slab_automove defaults to on, but still happens when all pages are busy or TEMP_LRU holds unevictable items.

The discriminator: is this refusing writes by design (-M), or a sizing or slab imbalance problem?

What this means

The error string SERVER_ERROR out of memory storing object is generated by memcached’s storage command handler when an item allocation fails. It is returned for SET, ADD, CAS, and REPLACE requests that cannot obtain a chunk. The store_no_memory counter in stats output tracks these failures.

The key discriminator is the relationship between two counters:

• store_no_memory climbing while evictions stays at zero: the daemon is running with -M. Writes are refused by design.
• store_no_memory climbing alongside evictions in specific slab classes: a slab class exhausted its free chunks and could not steal a page. This is a sizing or imbalance problem.

flowchart TD
    A["store_no_memory climbing"] --> B{"evict_to_free = 0?"}
    B -->|"Yes: -M active"| C["Stores refused by design"]
    C --> D{"bytes near limit?"}
    D -->|"Yes"| E["Remove -M or increase -m"]
    B -->|"No: eviction on"| F{"Per-slab outofmemory > 0?"}
    F -->|"Yes"| G["Slab class exhausted"]
    G --> H{"automove enabled?"}
    H -->|"No"| I["Enable automove or reassign page"]
    H -->|"Yes, cannot keep up"| J["Increase -m or adjust -f"]
    F -->|"No"| K["Check TEMP_LRU pressure"]

Common causes

Quick checks

These commands are read-only and safe to run during an incident.

# Check if -M (no-eviction) mode is active
# evict_to_free = 0 means -M is active; stores fail instead of evicting
echo "stats settings" | nc -q1 localhost 11211 | grep evict_to_free

# Check store_no_memory alongside evictions and memory usage
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT (store_no_memory|evictions|bytes |limit_maxbytes)"

# Check per-slab memory distribution
echo "stats slabs" | nc -q1 localhost 11211 | grep -E "(used_chunks|free_chunks|total_pages)"

# Check per-slab eviction and OOM details
echo "stats items" | nc -q1 localhost 11211 | grep -E "(evicted|outofmemory|number_temp)"

# Check slab_automove status (1 = default since 1.5.0)
echo "stats settings" | nc -q1 localhost 11211 | grep slab_automove

# Check TEMP_LRU configuration
echo "stats settings" | nc -q1 localhost 11211 | grep -E "(temp_lru|temporary_ttl)"

How to diagnose it

1. Confirm whether -M is active. Run stats settings and look at evict_to_free. A value of 0 means the daemon was started with -M and is refusing writes by design. A value of 1 means eviction is enabled and you have a genuine allocation failure.

2. If -M is active, confirm the cache is actually full. Check bytes against limit_maxbytes. If usage is near the limit, the daemon is behaving correctly for its configuration: it cannot evict, so it refuses writes. The fix is to remove -M or increase -m. If usage is well below the limit with -M on, look for slab-level exhaustion instead.

3. If -M is not active, identify which slab class is failing. Run stats items and look for classes with non-zero outofmemory counts. The outofmemory counter per slab class tracks how many times that class failed to allocate a chunk. This pinpoints the problem to a specific item size range.

4. For each failing slab class, check page availability. Run stats slabs and look at the class’s total_pages, used_chunks, and free_chunks. If free_chunks is zero and total_pages is low relative to other classes, the class is starved. If other classes have many free chunks and zero recent evictions, this is slab calcification.

5. Check whether slab_automove can help. Run stats settings and look for slab_automove. Mode 1 (default since 1.5.0) slowly moves pages from idle classes to evicting ones. Mode 0 means it is off. If it is off, enable it. If it is on but cannot keep up, the cache may be genuinely undersized for the working set in that size class.

6. Check for TEMP_LRU pressure. If temp_lru is yes, items with TTLs at or below temporary_ttl go into a TEMP segment that bypasses normal eviction. Run stats items and look at number_temp per class. If TEMP segments are large relative to HOT, WARM, and COLD, they may be holding memory that cannot be reclaimed through eviction, starving non-temporary stores.

Metrics and signals to monitor

Fixes

If -M is active: remove it or size the cache for it

The -M flag is a deliberate choice: the daemon returns errors instead of evicting. Some teams use it for caches where serving stale data is worse than a miss, and the application handles cache misses by falling through to the backend.

If the application cannot tolerate refused writes, remove -M from the startup flags and restart. This requires a restart; the flag cannot be toggled at runtime. Removing -M means the cache will evict under pressure. If the working set exceeds cache size, you will see evictions and potentially declining hit ratio instead of store_no_memory. The application must handle both cases the same way: miss on an evicted key, miss on a refused write, both fall through to the backend.

If you want to keep -M but stop the errors, increase -m. This also requires a restart. You can use cache_memlimit to raise the memory limit at runtime (since 1.4.27), which helps if the host has spare RAM, but it does not change the -M behavior.

Tradeoff: -M guarantees no eviction-induced stale data, at the cost of refusing writes when full. Without -M, writes always succeed, at the cost of evicting items that may still be useful.

If a slab class is exhausted (no -M)

This is slab calcification or a genuine size-class shortage. The fix depends on whether slab_automove can help.

If slab_automove is off (mode 0), enable it at runtime:

# Enable conservative automatic page rebalancing (safe, moves 1 page per 10s)
echo "slabs automove 1" | nc -q1 localhost 11211

For immediate relief, manually reassign a page from an idle class:

# Destructive: items in the moved page are evicted.
# Only move from classes with zero recent evictions and significant free chunks.
echo "slabs reassign <source_class> <destination_class>" | nc -q1 localhost 11211

If slab_automove is already on (mode 1) and cannot keep up, you can temporarily switch to mode 2 (aggressive), though this can cause latency jitter.

If all classes are saturated rather than just one, the cache is genuinely undersized. Increase -m or add instances. Adjusting the growth factor (-f, default 1.25) on the next restart can reduce internal fragmentation and improve slab class fit for your workload’s item size distribution. A lower factor creates more granular classes with less wasted space per chunk, at the cost of more classes competing for pages.

If TEMP_LRU is filling memory

TEMP_LRU (since 1.4.35) routes items with TTLs at or below temporary_ttl into a TEMP segment that bypasses normal eviction. The LRU maintainer background thread reaps them on expiry. If temporary_ttl is set too high, many items become effectively unevictable for their lifetime and can fill memory, causing store_no_memory on non-temporary items.

Check stats items for number_temp counts per class. If TEMP segments are large relative to HOT, WARM, and COLD, lower temporary_ttl or review whether short-TTL items should bypass eviction at all.

temporary_ttl is set at startup via -o temporary_ttl=<N>, or changed at runtime with lru temp_ttl <N>; use a negative value such as -1 to disable it live.

Prevention

• Audit startup flags. Know whether your instances run with -M. A cache running with -M that is expected to evict will produce store_no_memory under load, and the team may not realize the configuration is the cause. Document the intent behind -M wherever it is used.

• Alert on store_no_memory rate. Pair it with evictions. If evictions is zero and store_no_memory is climbing, the instance is in -M mode and full. This signature is the fastest way to distinguish “by design” from “genuinely broken.”

• Monitor per-slab outofmemory from stats items. This catches slab-level exhaustion before it becomes a client-visible error storm. Poll stats items no more frequently than every 30 seconds; it is heavier than stats.

• Verify slab_automove is enabled (mode 1) on 1.5.0+. This is the default but can be overridden by startup flags or orchestrator configs. Confirm with stats settings.

• Set temporary_ttl conservatively. Items in the TEMP segment are not evictable. A high threshold can crowd out normal items and cause allocation failures that look like undersizing but are actually a TEMP_LRU configuration issue.

• Size the cache with headroom. With slab_automove active, target 15-20% global headroom. The cliff edge is per slab class: once a class is full, stores to that class fail immediately.

How Netdata helps

• The memcached collector surfaces store_no_memory and evictions as per-second charts. The zero-evictions-plus-climbing-store_no_memory signature is visible without manual nc commands.

• Per-slab charts from stats items and stats slabs show which slab class is exhausted and whether free_chunks is at zero.

• Settings like evict_to_free and slab_automove are collected from stats settings, letting you confirm whether -M is active and whether rebalancing is enabled across the fleet.

• Correlating store_no_memory with bytes / limit_maxbytes, per-slab used_chunks, and direct_reclaims in a single view distinguishes design (-M) from sizing from slab imbalance.

Related guides

• Memcached evicted_unfetched and expired_unfetched: caching data nobody ever reads
• Memcached cas_badval climbing: check-and-set contention and lost updates
• Memcached connection refused: telling a dead process from a hung or full one
• Memcached evicted_time low: distinguishing healthy turnover from cache thrash
• Memcached eviction cascade: when a full cache overloads the backend
• Memcached evictions climbing: the cache is full and discarding live data
• Memcached high miss rate: separating cold start, new key patterns, and memory pressure
• How Memcached actually works in production: a mental model for operators
• Memcached incr/decr misses: evicted counters that silently break rate limiters and locks
• Memcached hit ratio dropping: reading get_hits, get_misses, and cache effectiveness
• Memcached monitoring checklist: the signals every production cache needs
• Memcached monitoring maturity model: from survival to expert