Memcached hit ratio dropping: reading get_hits, get_misses, and cache effectiveness

The cache hit ratio tells you whether memcached is earning its keep. When it drops, the backend database inherits the miss traffic, and a slow decline can cascade into a failure before anyone pages on “cache.” The ratio is also frequently read wrong: computing it from lifetime counters hides the exact degradation you are trying to catch.

What the hit ratio actually measures

The cache hit ratio is the fraction of GET lookups that found the key in cache:

hit_ratio = get_hits / (get_hits + get_misses)

Both counters come from the stats command:

# Read the hit and miss counters
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT get_(hits|misses)"

The -q flag is not portable across all nc implementations. On systems where it is unavailable, use -w 1 (connection timeout) instead.

Two properties of these counters shape everything else:

• They are cumulative since process start. They never reset except on restart. A memcached that ran at 99% for a week and then dropped to 50% for five minutes still shows roughly 98.9% on the raw counters.
• They count key lookups, not commands. cmd_get counts a multi-get as one command, but get_hits + get_misses reflects the per-key outcome. A single multi-get for a hundred keys can produce both hits and misses.

The ratio measures cache effectiveness, not cache health. A cache can be perfectly healthy and still show a low ratio if the workload is write-heavy or accesses random keys. A cache can be actively failing and still show a high ratio if the failure is recent relative to the process lifetime. The number is only meaningful with context: workload type, time window, and the signals you read alongside it.

How to compute it correctly: deltas, not lifetime averages

The most common operator mistake is reading get_hits and get_misses once and dividing. That gives you the lifetime average since process start, which masks recent degradation.

To get a meaningful ratio over a window, sample the counters twice and compute the delta:

# Compute hit ratio from a 10-second delta, not from lifetime counters
H1=$(echo "stats" | nc -q1 localhost 11211 | awk '/STAT get_hits / {print $3}')
M1=$(echo "stats" | nc -q1 localhost 11211 | awk '/STAT get_misses / {print $3}')
sleep 10
H2=$(echo "stats" | nc -q1 localhost 11211 | awk '/STAT get_hits / {print $3}')
M2=$(echo "stats" | nc -q1 localhost 11211 | awk '/STAT get_misses / {print $3}')
awk -v h1="$H1" -v h2="$H2" -v m1="$M1" -v m2="$M2" \
  'BEGIN { dh=h2-h1; dm=m2-m1; print "hit_ratio =", dh/(dh+dm) }'

This is what monitoring systems do under the hood. If your monitoring tool computes the ratio from raw cumulative counters instead of deltas, the number is close to useless for detecting incidents.

Why absolute thresholds lie and rate-of-change does not

Absolute hit ratio thresholds are workload-dependent, and treating them as universal is a source of noise:

• A session cache should run above 99%. Anything below 95% after warmup is a problem.
• A CDN-style content cache at 60% may be perfectly healthy because the working set is large and access is long-tailed.
• A write-heavy cache used mostly for invalidation may sit at 70% and be fine.

The universal signal is rate-of-change, not the absolute value:

• A drop of more than 10 percentage points over 5 minutes is abnormal regardless of baseline.
• A sustained decline of more than 15 points from a one-hour rolling average warrants investigation.
• After a restart, the ratio starts near 0% and climbs over minutes to hours depending on TTLs and traffic. That curve is expected, not a bug.

The ratio is also a lagging indicator. By the time it moves enough to trigger an alert, the backend may already be absorbing the miss traffic and showing latency or saturation. Alert on the ratio’s rate-of-change and read it alongside eviction and backend signals, not in isolation.

One more trap: a ratio near 100% with very low cmd_get means clients may have stopped reading from the cache and are only writing. That masks a different problem. Always read the ratio next to the command rate.

What moves the ratio

When the ratio drops, one of six mechanisms is usually responsible. Each has a distinct signal signature.

flowchart TD
  A["hit ratio dropping
from delta computation"] --> B{"uptime reset or
cmd_flush incremented?"}
  B -- "yes" --> C["cold start or flush
ratio recovers over warmup"]
  B -- "no" --> D{"evictions increasing?"}
  D -- "no" --> E{"cmd_get spiked?"}
  E -- "yes" --> F["cache stampede
backend is the victim"]
  E -- "no" --> G["TTL misconfig or
application key pattern shift"]
  D -- "yes" --> H{"global bytes
near limit_maxbytes?"}
  H -- "yes" --> I["eviction cascade
cache undersized for working set"]
  H -- "no" --> J["slab calcification
check per-slab evicted_time"]

Eviction cascade

The cache is full and the LRU is discarding items that are still being requested. Signals:

• evictions increasing
• hit ratio declining gradually over minutes
• global bytes near limit_maxbytes
• backend load rising in step with the miss rate

This is the canonical memcached failure. The cache is undersized for the working set, or the working set grew. Adding memory helps here, but only after you confirm the pressure is global and not concentrated in one slab class.

Slab calcification

Memory is partitioned by item size into slab classes. A page assigned to a class was traditionally never returned, so a workload shift (item sizes changed after a deploy) can leave one class starved and evicting while others sit idle. Signals:

• evictions increasing but global bytes at 50 to 70% of limit
• per-slab stats show one class at 100% used_chunks, zero free_chunks, high evictions
• evicted_time low for the saturated class
• other slab classes with significant free chunks

Since 1.5.0, slab_automove mode 1 is on by default and moves pages between classes slowly, which mitigates but does not eliminate this. Before 1.5.0, slab_automove and slab_reassign had to be enabled explicitly. Adding global memory does not help if the problem is distribution, not total size.

# Find the saturated slab class
echo "stats slabs" | nc -q1 localhost 11211 | grep -E "(used_chunks|free_chunks|total_pages)"
echo "stats items" | nc -q1 localhost 11211 | grep -E "(evicted|evicted_time)"

Cache stampede (thundering herd)

A popular key expired or was evicted, and many concurrent requests miss simultaneously, hammering the backend. Signals:

• cmd_get spikes
• get_misses spikes
• evictions is low or zero (memory is not full)
• backend load spikes in lockstep

Memcached itself is fine. The victim is the backend. This is an application architecture problem: no stampede prevention, no distributed lock or lease around recomputation.

TTL misconfiguration

Items expire too quickly, or many items share the same TTL and expire simultaneously. Signals:

• high get_misses with low or zero evictions
• reclaimed rate high (expired slots being reused by new sets)
• ratio recovers in waves that match the TTL interval

No eviction means no memory pressure. The misses come from expiration, not displacement.

Flush (cmd_flush)

Someone or something issued flush_all. Signals:

• cmd_flush incremented
• get_flushed spikes (GETs hitting lazily-invalidated items)
• curr_items declining
• ratio plummets instantly, not gradually

flush_all does not lock the server. It sets a timestamp and items are lazily invalidated on next access. Memory is not freed immediately. Any increment of cmd_flush in production should alert, because an accidental flush causes a cold-cache thundering herd that is indistinguishable from a restart in impact but harder to detect because the process stays up.

# Detect a flush event and measure its read-side impact
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT (cmd_flush|get_flushed)"

Cold start

The process restarted (OOM kill, crash, upgrade). Signals:

• uptime near zero or lower than the previous sample
• curr_items at zero, then climbing
• ratio starts at 0% and climbs over the warmup period

Every restart is total data loss. The ratio will be 0% and climb over minutes to hours depending on TTLs and traffic. This is expected. The risk is the backend absorbing full miss load during warmup, not the ratio itself.

The per-slab blind spot

The aggregate hit ratio hides per-slab behavior, and you cannot compute a per-slab hit ratio directly. Memcached tracks some per-slab-class counters internally, but get_misses is only reported as a global aggregate: a miss does not record which slab class the key would have belonged to. Without per-slab misses, the per-slab ratio is not derivable from memcached’s own stats.

When the aggregate ratio drops and global memory looks fine, the problem is almost always in one slab class. The way to find it is not to chase the ratio but to look at per-slab evictions and evicted_time from stats items. A saturated class shows high evicted, zero free_chunks, and low evicted_time (recently-accessed items being discarded). An idle class shows the opposite.

The key discriminator is evicted_time. If it is low (under 300 seconds) in a class with active evictions, the cache is thrashing on actively-used data. If it is high (hours or days), the LRU is doing healthy cold-item turnover and the evictions are not the cause of the ratio drop.

Signals to watch alongside hit ratio

How Netdata helps

• Netdata collects get_hits and get_misses per second and computes the hit ratio from deltas, so the ratio reflects current effectiveness rather than a lifetime average that masks incidents.
• Per-second granularity means a 10-point drop over five minutes is visible as it forms, not smoothed away by a long polling interval.
• ML anomaly detection flags unusual rate-of-change in the ratio even when the absolute value is still within a nominal range.
• Correlating hit ratio with eviction rate, per-slab utilization, cmd_flush, and uptime in a single view shortens the path from “ratio dropped” to “here is the slab class that is starving.”
• The memcached collector exposes evicted_time, get_flushed, reclaimed, and per-slab counters so the slab calcification and flush cases do not require a manual nc session to triage.
• Backend database metrics on the same host or parent node confirm whether the miss traffic is already stressing the downstream system before the ratio finishes moving.

Related guides

• Memcached connection refused: telling a dead process from a hung or full one
• How Memcached actually works in production: a mental model for operators
• Memcached monitoring checklist: the signals every production cache needs
• Memcached monitoring maturity model: from survival to expert