$ guides / memcached / memcached-incr-decr-misses ▌

Operations Guides

Memcached incr/decr misses: evicted counters that silently break rate limiters and locks

Most memcached operators never look at incr_misses and decr_misses. They sit in the per-operation hit/miss breakdown and stay near zero. When they start climbing, the cache itself usually looks healthy: hit ratio is fine, evictions are modest, memory is not full. The damage is happening somewhere else.

The pattern is specific. The application uses memcached as an atomic counter store. Rate limiters increment a per-client counter and reject when it crosses a threshold. Distributed locks hold a lease token with a TTL and decrement on release. Quota counters track usage per tenant. These counter keys are small, hot, and short-lived. When the slab class they live in comes under memory pressure, the LRU evicts them between operations.

The next incr after an eviction returns NOT_FOUND. Memcached does not auto-create the key. The application has to handle that path, and most rate limiter and lock implementations either skip the increment or re-initialize to zero. Either way the guarantee disappears: the rate limit resets, the lock is released early, the quota restarts. Nothing throws. The incr_misses counter is the only server-side signal that this is happening.

This article covers how to read that signal, attribute it, and handle the overflow and underflow gotchas that bite even when counters survive eviction.

What this means

incr_misses and decr_misses count incr and decr commands issued against keys that do not exist. On a healthy memcached instance, “do not exist” can mean three things:

The key was never created. The application issued incr without first doing add or set.
The key was evicted. The slab class came under memory pressure and the LRU removed it.
The key expired. Its TTL elapsed between the previous operation and this one.

All three increment the same counter. The second and third cases are the silent-break scenarios. A counter key with a 60-second TTL for a rate limiter window can be evicted at second 12 if the slab class it lives in is full. The next incr returns NOT_FOUND. If the rate limiter treats NOT_FOUND as “no prior count, start fresh”, the rate window effectively restarts. A client sending requests just fast enough to keep the slab under pressure gets an effectively unlimited rate.

The red-flag ratio is incr_misses / (incr_hits + incr_misses) > 20%. Above that, counter keys are being evicted or expired faster than the application uses them. Below 5% is usually initialization churn. Between 5% and 20% deserves investigation but is not yet a guarantee failure.

A related gotcha is independent of misses. incr overflow wraps silently at the 64-bit unsigned boundary (2^64-1). The value resets to a small number with no error returned to the client. decr underflow clamps to 0, also silently. Neither is visible in any stat. The only defense is application-level: cap counters, use modular arithmetic, or detect wraparound in client code.

Common causes

Cause	What it looks like	First thing to check
Slab class under eviction pressure	`incr_misses` rising alongside `evictions`; counter keys are small, so they concentrate in one early slab class	`stats items` per-slab `evicted` and `evicted_time` for the small-size classes
TTL shorter than the operation interval	`incr_misses` rising with zero or low global evictions; counters expire between operations	TTL on the counter key vs. expected time between `incr` calls
Missing initial set/add	`incr_misses` high from process start, never comes down; the application never bootstraps counters	Application code path: does it call `add` before `incr`, and does it handle `NOT_FOUND`?
Flush or restart	`incr_misses` spikes once alongside a `cmd_flush` increment or `uptime` reset, then recovers	`cmd_flush` and `uptime`
Overflow misread as miss	Counters reset silently at high values; `incr_misses` is not actually rising	Client-side max value check; the value wraps at 2^64-1

Quick checks

# incr/decr hit and miss counters
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT (incr|decr)_(hits|misses)"

# Evictions in the small-item slab classes
echo "stats items" | nc -q1 localhost 11211 | grep -E "evicted|evicted_time|evicted_nonzero"

# Slab class balance: small counter keys land in early classes
echo "stats slabs" | nc -q1 localhost 11211 | grep -E "chunk_size|used_chunks|free_chunks|total_pages"

# Recent flush or restart as the trigger
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT (cmd_flush|uptime|curr_items)"

# Slab automove state (default since 1.5.0 is mode 1)
echo "stats settings" | nc -q1 localhost 11211 | grep slab_automove

All read-only. None touch cache contents.

How to diagnose it

The diagnostic flow: confirm the miss rate is real, attribute it to one of the three causes (initialization, eviction, expiration), then trace the affected slab class.

flowchart TD
    A[incr_misses rising] --> B{Evictions also rising?}
    B -- Yes --> C[Check stats items per-slab evicted_time]
    B -- No --> D{uptime reset or cmd_flush incremented?}
    D -- Yes --> E[One-time miss spike, recovers]
    D -- No --> F{Misses since process start?}
    F -- Yes --> G[Application never bootstraps counters]
    F -- No --> H[TTL shorter than operation interval]
    C --> I{evicted_time < counter TTL?}
    I -- Yes --> J[Slab pressure evicting live counters]
    I -- No --> K[Healthy eviction of cold items]
    J --> L[Reduce slab pressure or move counters off memcached]

Step by step:

Pull two stats samples a minute apart and compute the rate of incr_misses. If it is zero or near-zero, the rising value was historical and you can stop.
Pull incr_hits over the same interval and compute incr_misses / (incr_hits + incr_misses). Above 20% is the red flag.
Pull evictions over the same interval. If evictions are also rising, the cause is eviction pressure, not expiration. Jump to step 5.
If evictions are flat but incr_misses is rising, the cause is TTL or initialization. Check the TTL the application sets on counter keys. If the TTL is shorter than the typical interval between incr calls, the counter expires legitimately between operations. If the misses have been non-zero since process start, the application is calling incr on keys it never created.
Run stats items and look at the evicted and evicted_time counters per slab class. Counter keys are small, usually landing in slab classes with chunk sizes under 200 bytes. Find the class with rising evicted.
Check evicted_time for that class. If it is low (under 300 seconds, especially under the TTL of your counter keys), the LRU is evicting counters that have not yet expired. This is the silent-break condition.
Check evicted_nonzero for the same class. Counter keys almost always have non-zero TTL, so a rising evicted_nonzero confirms that items with expiration times are being evicted early.
Confirm whether slab_automove is active. If it is on and the class is still evicting, the automover cannot keep up because every other class is also under pressure or because there are no idle pages to move.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`incr_misses` rate	Direct count of counter operations against missing keys	Sustained non-zero rate when the application is supposed to be initializing counters
`incr_misses / (incr_hits + incr_misses)`	Fraction of counter operations that lost their key	Above 20% sustained
`decr_misses` rate	Same pattern for decrement operations, often lock release	Sustained non-zero rate
`evictions` rate	Confirms eviction pressure as the cause	Rising in parallel with `incr_misses`
`stats items` per-slab `evicted_time`	Age of the most recently evicted item per slab class	Under the TTL of your counter keys, especially under 60 seconds
`stats items` per-slab `evicted_nonzero`	Counts items with non-zero TTL evicted early	Rising in the slab class that holds counter keys
`cmd_flush` and `uptime`	Distinguish a one-time miss spike from a sustained problem	`cmd_flush` incrementing or `uptime` resetting
`slab_automove` setting	Whether the server can self-repair slab imbalance	Set to 0 when slab imbalance is the cause

There is no stat that exposes incr overflow or decr underflow directly. Those failures are invisible to memcached monitoring and must be caught in client code.

Fixes

Counter keys are being evicted (slab pressure)

The fastest relief is reducing pressure on the slab class that holds the counters. Options, roughly in order of operational cost:

Enable slab_automove at runtime with slabs automove 1 if it is off. This is the default since 1.5.0 but it may have been disabled. Mode 1 is conservative and safe. Mode 2 is aggressive and not recommended for sustained use.
Manually move a page into the starving class: slabs reassign <source_class> <dest_class>. Pick a source class with free_chunks > 0 and zero recent evictions. This is safe but ephemeral; the page can be reclaimed by automove later.
Increase -m if the working set genuinely exceeds memory and every class is under pressure. This requires a restart, which means cold cache and a backend spike. Plan it.
Move counter keys off memcached entirely. A dedicated small memcached instance just for counters, or a persistent store like Redis, removes the conflict between the counter workload and the rest of the cache. Rate limiter counters are a poor fit for a shared, evictable cache.

Counter keys are expiring between operations

Lengthen the TTL on counter keys to comfortably exceed the interval between operations. For a rate limiter with a 60-second window, the counter TTL should be at least the window length plus margin.
Avoid a TTL of 1 second. Memcached updates internal expiration on second boundaries, which makes a TTL of 1 sometimes expire immediately. Use 2 or higher.
Reset counters explicitly at the window boundary with set rather than relying on TTL expiry. This trades determinism for one extra command per window per key.

Application never bootstraps counters

The classic pattern is add to create the counter (fails if the key exists), then incr to increment. If add succeeds, the counter is initialized. If add fails because the key already exists, the counter is live and incr proceeds. If incr returns NOT_FOUND after a successful add, the counter was evicted between the two commands, which is the eviction case above.

Handle NOT_FOUND explicitly in client code. Either re-initialize with add and retry, or treat it as a known degraded state and fail closed. The default behavior of “ignore the miss and proceed” is what silently disables rate limiters.

The newer meta arithmetic command (ma) can auto-create the item on miss using the N flag (with a TTL) and the J flag (initial value, default 0). This collapses the add-then-incr pattern into one round trip and removes the race. ma is part of the meta protocol, stable in memcached 1.6.0 and present in every release since. Not every client library exposes it, so verify support in yours before relying on it.

Overflow and underflow gotchas

Neither produces an error. incr past 2^64-1 wraps silently to a small value. decr below 0 clamps silently to 0. There is no stat, no log, no client-side notification.

Defenses are application-side only:

For rate limiter counters, cap the value at the threshold and stop incrementing. This makes wraparound impossible because the counter never approaches 2^64.
For lock lease tokens and quota counters, use values that are small relative to 2^64 and treat implausibly small values as a sign of wraparound.
For decr, the clamp-to-0 behavior is usually safe. The risk is a lock that decrements past zero on early release and then cannot be re-acquired at the expected count. Treat any decr result of 0 as authoritative and do not assume the previous value.

Prevention

Monitor incr_misses and decr_misses as first-class signals. They are not noise; they are the only server-side signal that counter-backed guarantees are failing.
Compute the ratio, not just the rate. incr_misses / (incr_hits + incr_misses) is more stable than the raw miss count because it normalizes for traffic.
Track per-slab evicted_time for the small-item classes. Counter keys live there. Low evicted_time in those classes is the early warning.
Reserve enough memory in the small-item slab classes. Counter workloads are small, hot, and numerous. They fit comfortably in early slab classes, but only if those classes have room.
Treat NOT_FOUND from incr as a degraded state in application code. Fail closed: deny the request, re-initialize, or alert. Do not silently proceed.
Consider a dedicated counter store. If the rate limiter or lock is load-bearing, memcached’s evictable memory model is the wrong tool. A small Redis instance or a dedicated memcached instance sized to hold the counter working set with headroom removes the conflict.

How Netdata helps

Netdata’s memcached collector pulls incr_hits, incr_misses, decr_hits, and decr_misses per second, so the miss ratio is visible as a live trend rather than a manual two-sample calculation.
Per-slab evicted, evicted_time, and evicted_nonzero from stats items are correlated against incr_misses in the same dashboard, so the link between slab pressure and counter loss shows up without cross-referencing terminals.
Anomaly detection on the miss rate flags the inflection point where incr_misses starts climbing before the absolute count looks alarming. Rate limiter guarantees break well before miss volume is large.
cmd_flush and uptime are tracked alongside the miss counters, so a one-time miss spike from a flush or restart is distinguishable from a sustained eviction problem.
Eviction rate, hit ratio, and per-slab utilization are pre-correlated, which shortens the path from “counter misses rising” to “which slab class is evicting my counters”.