Memcached high miss rate: separating cold start, new key patterns, and memory pressure

A high miss rate on memcached is frequently misdiagnosed. The instinct is to assume the cache is too small and add memory. That instinct is usually wrong. Random or unique key access produces a near-0% hit rate no matter how healthy the cache is. A freshly restarted cache legitimately runs near 0% until it warms. A deploy that changed key naming produces instant misses on the new keys without any memory pressure.

The real diagnostic question is not “why is the miss rate high?” but “is the miss rate high because of memory pressure, or because of something the application is doing?” The answer determines whether you resize the cache or fix the application.

The fastest discriminator is the eviction counter. High misses combined with high evictions points to memory pressure: the cache is full and discarding items the application still wants. High misses with zero evictions means the cache has free headroom and the misses are coming from application behavior: cold start, new key patterns, write-only workloads, or natural TTL expiry of items nobody re-reads.

What this means

A miss in memcached is a GET request that did not find a key in cache. The miss rate alone tells you only that lookups are failing. It tells you nothing about why. The global hit ratio formula is:

hit_ratio = get_hits / (get_hits + get_misses)

These are cumulative counters since process start. Compute the ratio from deltas over a recent window, not from absolute cumulative values, which average over the entire lifetime and obscure recent changes.

The three root causes this article separates are:

1. Cold start or flush. The cache was recently emptied by a restart, an OOM kill, or a flush_all. Every request misses until the cache warms.
2. New key patterns. The application started requesting keys that were never stored, changed key naming, or is doing flag-existence checks against non-existent keys. This inflates get_misses without any memory pressure.
3. Memory pressure. The working set exceeds the allocated memory, or one slab class does, and evictions are discarding items the application would have hit. This is the only case where adding memory helps.

The decision tree below shows the diagnostic branch points. The eviction counter is the first fork: it separates memory-pressure causes from everything else.

flowchart TD
    A[High miss rate detected] --> B{uptime recently reset?}
    B -- yes --> C[Cold start after restart]
    B -- no --> D{cmd_flush incremented?}
    D -- yes --> E[flush_all event]
    D -- no --> F{evictions increasing?}
    F -- no --> G[Application behavior]
    G --> H{cmd_set near cmd_get?}
    H -- yes --> I[Write-heavy or unique keys]
    H -- no --> J[New key patterns or existence checks]
    F -- yes --> K{one slab class full?}
    K -- yes --> L[Slab imbalance]
    K -- no --> M{evicted_time low?}
    M -- yes --> N[Working set exceeds cache]
    M -- no --> O[Healthy eviction of cold items]

Common causes

Quick checks

These are read-only and safe to run against production.

# Check daemon responsiveness and uptime
echo "stats" | nc -w 2 localhost 11211 | grep -E "STAT (uptime|version)"

# Hit rate components - compute ratio from deltas between two samples
echo "stats" | nc -w 2 localhost 11211 | grep -E "STAT get_(hits|misses)"

# Eviction counter and memory utilization
echo "stats" | nc -w 2 localhost 11211 | grep -E "STAT (evictions|bytes|limit_maxbytes)"

# Check for flush_all events and their read-side impact
echo "stats" | nc -w 2 localhost 11211 | grep -E "STAT (cmd_flush|get_flushed)"

# Command ratio - read-heavy vs write-heavy
echo "stats" | nc -w 2 localhost 11211 | grep -E "STAT cmd_(get|set)"

# Waste indicators - items evicted or expired without ever being read
echo "stats" | nc -w 2 localhost 11211 | grep -E "STAT (evicted_unfetched|expired_unfetched)"

# Per-slab eviction distribution - which classes are under pressure?
echo "stats items" | nc -w 2 localhost 11211 | grep -E "evicted|evicted_time"

# Per-slab free chunks - which classes have headroom?
echo "stats slabs" | nc -w 2 localhost 11211 | grep -E "used_chunks|free_chunks|total_pages"

# Direct reclaims - is the LRU maintainer falling behind?
echo "stats" | nc -w 2 localhost 11211 | grep "STAT direct_reclaims"

How to diagnose it

Step 1: Check uptime and cmd_flush.

If uptime is low (minutes when it should be hours or days), the cache is warming after a restart. A near-zero hit ratio is expected and will climb. If cmd_flush has incremented, someone or something issued flush_all. Both are transient causes. Focus on whether the backend can survive the warm-up load rather than on fixing the cache.

Step 2: Check the eviction rate.

This is the single most important branch point. If evictions is zero or flat, the cache is not discarding anything. The misses are coming from application behavior, not memory pressure. Skip to step 5.

If evictions is increasing, the cache is full in at least one slab class and discarding valid, non-expired items. Proceed to step 3.

Step 3: Check per-slab eviction distribution.

Global evictions can be driven by a single saturated slab class while others sit idle. Run stats items and look for which classes have active evictions. Then run stats slabs and check free_chunks across all classes. If one class is at zero free chunks and evicting while others have significant free chunks, you have a slab imbalance, not a global memory shortage. Adding memory will not help unless it is accompanied by slab rebalancing.

Step 4: Check evicted_time per slab class.

evicted_time (from stats items, not from plain stats) reports the seconds since last access of the most recently evicted item in each slab class. This is the critical discriminator:

• High values (hours or days): the cache is evicting genuinely cold items. This is healthy LRU turnover.
• Low values (under 300 seconds): the cache is evicting recently-active items. This is harmful thrashing where the working set exceeds the cache size for that class.
• Under 60 seconds: severe thrash. Items are evicted almost immediately after last access.

When evictions are zero in a slab class, evicted_time is 0 and meaningless.

Step 5: If evictions are zero, investigate the application.

Zero evictions with high misses means the application is requesting keys that were never stored or have already expired naturally. Check the following:

• Did a deploy change key naming? Old keys in the cache will never match new key requests.
• Is the application doing existence checks (flag lookups) for keys that may not exist? These inflate get_misses legitimately. This is a common trap in workloads that treat memcached as a presence oracle.
• Is cmd_set high relative to cmd_get? A write-heavy pattern with unique keys, such as per-request tokens or session IDs written but rarely re-read, will never produce hits regardless of cache size.

Step 6: Check waste indicators.

evicted_unfetched counts items evicted without ever being read after their last set. expired_unfetched counts items that expired naturally without ever being read. High values for either mean the application is caching data nobody requests. This inflates miss rates by filling the cache with useless items that displace useful ones, or by contributing to the miss count on lookups that were never going to hit.

Metrics and signals to monitor

Fixes

Cold start or flush

No cache fix is needed. The cache will warm as traffic flows. Focus on the backend:

• Confirm the backend can absorb the full miss load during warm-up.
• If you have cache-warming scripts, run them.
• For future restarts, consider stampede protection such as distributed locks or probabilistic early refresh to prevent thundering herd on the backend.

If flush_all was accidental, investigate who or what issued it. Memcached does not log the source of commands, so check network-level logging or application code paths that call flush_all.

New key patterns or write-only workload

No cache fix will help. Adding memory or instances will not change the hit ratio if the application is requesting keys that were never stored. The fix is in the application:

• If a deploy changed key naming, ensure the application writes and reads with the same key scheme.
• If the application does existence checks for keys that may not exist, consider whether those lookups belong in memcached at all.
• If the workload is inherently write-heavy with unique keys, accept the low hit ratio or reconsider whether memcached is the right store for that data.

Memory pressure (global undersizing)

If bytes is near limit_maxbytes, evictions are sustained across multiple slab classes, and evicted_time is low, the cache is genuinely undersized for the working set:

• Increase the -m memory limit. This requires a restart, which means total data loss and a fresh cold start.
• Consider adding more instances to the cluster rather than making one larger, to distribute connection load and network bandwidth.
• Before resizing, verify via per-slab analysis that the problem is genuinely global and not a slab imbalance masquerading as global pressure.

Slab imbalance

If global memory is moderate (50-70%) but one slab class is full and evicting while others have free chunks, the problem is memory distribution, not total memory:

• Check if slab_automove is enabled: echo "stats settings" | nc -w 2 localhost 11211 | grep slab_automove. It is default-on at mode 1 since 1.5.0.
• If automove is off, enable it at runtime: slabs automove 1. This changes daemon state live; it is safe to run in production but will start moving memory pages between slab classes.
• For urgent relief, manually move a page from an idle class to a starving one: slabs reassign <source_class> <dest_class>. Warning: this is destructive to items in the moved page. Those items are evicted immediately.
• For a structural fix on the next restart, review the slab growth factor (-f, default 1.25). A lower factor creates more classes with finer granularity, reducing internal fragmentation at the cost of more classes competing for pages.

Cache stampede

If the miss spike correlates with a single hot key expiring and backend load spiking, memcached itself is not broken. The fix is application-level stampede prevention:

• Probabilistic early refresh: refresh keys slightly before they expire, with randomness to avoid synchronized refreshes.
• Distributed lock pattern: when a miss occurs, one request acquires a lock to fetch and repopulate the key while others wait or serve stale data.

Prevention

• Alert on hit ratio with context, not in isolation. A raw hit-ratio alert is noisy. Correlate with evictions, uptime, and cmd_flush before paging. A hit-ratio drop with zero evictions is an application signal, not a cache crisis.
• Track the cmd_set / cmd_get ratio. A rising ratio means the workload is becoming more write-heavy, which will depress hit ratio regardless of cache health.
• Monitor per-slab utilization. Global memory metrics hide slab imbalance. Track per-slab free_chunks and evicted to catch calcification before it degrades hit ratio.
• Track evicted_time per slab class. This is the single best signal for distinguishing healthy eviction of cold items from harmful thrashing of active items.
• Monitor evicted_unfetched and expired_unfetched. High values indicate the application is caching data nobody reads, which wastes memory and inflates miss rates.
• Alert on cmd_flush. Any increment in production should trigger investigation. An accidental flush_all is indistinguishable from a cache restart in terms of impact, but harder to detect because the process stays up.
• Have a warming strategy. Every restart is complete data loss. Know how long warm-up takes for your workload and whether the backend can survive the cold period.

How Netdata helps

• Netdata collects get_hits, get_misses, evictions, and computed hit ratio at per-second resolution, letting you see the exact moment the miss rate changes and correlate it with eviction rate, uptime, and cmd_flush in the same time window.
• Per-second eviction and memory utilization charts let you distinguish a gradual eviction-driven decline from an instant cold-start drop within seconds of the event.
• cmd_set and cmd_get are tracked independently, so write-to-read ratio shifts are visible alongside hit-ratio changes without manual sampling.
• ML-based anomaly detection flags unusual changes in hit ratio, eviction rate, and command rates, helping separate expected warm-up curves from genuinely anomalous degradation.
• uptime discontinuities and cmd_flush increments appear as discrete events in the same dashboard, making restart and flush causes immediately visible without manual correlation across tools.

Related guides

• Memcached connection refused: telling a dead process from a hung or full one
• How Memcached actually works in production: a mental model for operators
• Memcached monitoring checklist: the signals every production cache needs
• Memcached monitoring maturity model: from survival to expert
• Memcached alive but not responding: the silent process hang