Memcached cas_badval climbing: check-and-set contention and lost updates

cas_badval tracks check-and-set operations that failed because the CAS unique token changed between your gets read and your cas write. Each increment is a rejected write: another writer modified the key first, and memcached correctly refused the stale update to prevent a lost update.

A sustained cas_badval rate above 10% of total CAS attempts, computed as cas_badval / (cas_hits + cas_misses + cas_badval), indicates meaningful write contention. The application is spending CPU on writes that never land, and unbounded retry logic can amplify the problem: every failed CAS triggers an immediate re-read and re-write, increasing load on both the client and memcached without making progress.

What this means

CAS (check-and-set) is memcached’s optimistic concurrency mechanism. A client calls gets <key> to read a value along with an opaque CAS unique token. Later, it calls cas <key> <flags> <exptime> <bytes> <cas_unique> to write conditionally. Memcached compares the supplied token against the current one stored with the item.

Three outcomes are possible:

• Token matches: write succeeds, response is STORED, cas_hits increments.
• Token does not match: another writer modified the key between the gets and the cas. Response is EXISTS. cas_badval increments.
• Key not found: the key was evicted, expired, or never existed between read and write. Response is NOT_FOUND. cas_misses increments.

The cas_badval counter is not an error condition. Each increment is a correct rejection. The problem is the rate and what it implies about application behavior.

The distinction between cas_badval and cas_misses matters for diagnosis. cas_misses is not contention. It means the key was gone when the CAS arrived, typically because it was evicted or expired between the gets and the cas. High cas_misses points to memory pressure or TTL mismatch. If cas_misses is also climbing, correlate with evictions and evicted_time and see the hit ratio guide.

Applications that never issue cas commands will always read zero for all three counters. That is normal, not missing data. A sudden move from zero to nonzero means a code path started using CAS.

sequenceDiagram
    participant W1 as Writer A
    participant MC as Memcached
    participant W2 as Writer B
    W1->>MC: gets "user:42"
    MC-->>W1: VALUE ... cas_unique=100
    W2->>MC: gets "user:42"
    MC-->>W2: VALUE ... cas_unique=100
    W1->>MC: cas "user:42" ... cas_unique=100
    MC-->>W1: STORED (cas_hits++)
    W2->>MC: cas "user:42" ... cas_unique=100
    MC-->>W2: EXISTS (cas_badval++)
    Note over W2,MC: Writer B update rejected. Retry, backoff, or abandon.

Common causes

Quick checks

# CAS counters from global stats
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT cas_(badval|hits|misses)"

# Per-slab CAS contention to identify contended item sizes
echo "stats slabs" | nc -q1 localhost 11211 | grep -E "cas_(badval|hits)"

# Check if CAS was disabled at startup with -C
ps aux | grep '[m]emcached' | grep -- ' -C'
# No output means CAS is enabled (the default)

# Overall command rates for context
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT cmd_(get|set)"

# Evictions and reclaimed items (evicted/expired keys cause cas_misses, not cas_badval)
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT (evictions|reclaimed)"

# Memcached version
echo "version" | nc -q1 localhost 11211

How to diagnose it

1. Confirm CAS is in use. If cas_badval, cas_hits, and cas_misses are all zero, the application does not issue cas commands and the counter is irrelevant. Nonzero cas_badval confirms CAS is active.

2. Compute the contention ratio. Sample the counters twice with a known interval. Compute delta(cas_badval) / (delta(cas_hits) + delta(cas_misses) + delta(cas_badval)). Above 10% sustained is significant contention. Below 5% is typical background noise in concurrent applications. The threshold is a guideline; some workloads such as counters and rate limiters naturally run higher.

3. Distinguish cas_badval from cas_misses. If cas_misses dominates, keys are being evicted or expiring between read and write. That is a memory or TTL problem, not contention. Check evictions and evicted_time, and see the hit ratio guide. Only cas_badval indicates genuine version conflicts from concurrent writers.

4. Identify hot slab classes. Run stats slabs and look for per-class cas_badval concentration. The slab class with disproportionate cas_badval tells you the item size range of the contended keys, which narrows the application code path. Note that cas_misses is not tracked per-slab because the key was not found and cannot be attributed to a slab class.

5. Check for retry amplification. If cmd_get spikes alongside cas_badval, clients are likely retrying failed CAS operations in a tight loop. Each retry re-reads with gets, re-attempts cas, and fails again if the contention window has not closed. This increases load on memcached and the backend without making progress.

6. Verify client library and protocol. Some client libraries report CAS failures incorrectly under certain protocols or versions. If cas_badval is climbing but application logs show no CAS failures, suspect a client-side reporting issue. Check the client library version and whether the binary protocol is in use.

Metrics and signals to monitor

Fixes

Hot key contention

Multiple application instances writing to the same key concurrently:

• Reduce write frequency. If the cached value changes rarely, batch updates or use a longer TTL with lazy refresh instead of CAS on every write. Fewer writes means fewer conflicts.
• Shard the contended key. Split a single hot key into N shards. For example, user:42:counter becomes user:42:counter:0 through user:42:counter:3. Writers pick a shard by hash or round-robin. Reads aggregate. This reduces conflict probability roughly proportionally to N.
• Move coordination out of the cache. If the CAS pattern implements a distributed counter, rate limiter, or lock, use a system designed for atomic coordination. Memcached CAS is a correctness guard, not a coordination primitive. Redis with atomic INCR, a database with row-level locking, or a dedicated coordination service will handle contention more gracefully.

Unbounded retry loops

The classic CAS pattern is gets, modify locally, cas, retry on EXISTS. Under contention, retries pile up and each one re-enters the race:

• Bound the retries. Cap at 3 to 5 attempts. After the limit, skip the cache write and accept a stale value on next read, or fall through to the backend. An unbounded loop under heavy contention burns CPU on both client and server without progress.
• Add backoff and jitter. Exponential backoff with random jitter prevents synchronized retry storms across multiple application instances hitting the same key simultaneously.
• Measure retry depth. Instrument the application to track how many attempts each CAS operation takes. If most writes succeed on the first attempt, contention is mild. If retries consistently hit the cap, the key pattern needs redesign.

Client library bugs

If cas_badval is climbing but the application does not log CAS failures, the client may be misreporting:

• Check the protocol. Meta commands are the intended replacement for basic storage and retrieval workflows, and existing commands continue to work; the binary protocol was not formally deprecated in memcached 1.6.0. Some client libraries have issues with CAS return values under the binary protocol. If you are on the binary protocol, test with the text protocol to isolate the issue.
• Upgrade the client library. Older versions of some client libraries had issues returning CAS tokens during gets requests. Without a valid token, cas operations behave incorrectly. Upgrade to a current version and verify that gets returns a CAS unique value.
• Verify with a manual test. Issue a gets and cas pair manually against the same instance and key. If the manual test succeeds but the application fails silently, the problem is in the client code path, not in memcached.

Server version bugs

Memcached 1.6.20 incorrectly returned the s flag in an ms response when only the c flag was requested; this was fixed in 1.6.21.

Some memcached versions have shipped bugs affecting meta protocol CAS behavior. If you see unexpected CAS behavior after a server upgrade, check the version and review the changelog for CAS-related fixes. Upgrade to the latest patch release if you are affected.

Prevention

• Monitor the contention ratio, not the raw counter. cas_badval as an absolute rate is meaningless without the denominator. Track cas_badval / (cas_hits + cas_misses + cas_badval) as a percentage. A jump from 2% to 8% is more actionable than “500 failures per second.”
• Design write paths for low contention. CAS is a guard against lost updates, not a coordination mechanism. If multiple writers routinely hit the same key, rethink the access pattern before it shows up in metrics.
• Bound retries in every CAS code path. Every CAS loop should have a maximum attempt count, backoff with jitter, and a defined fallback when the limit is reached.
• Track client library versions and protocols. Document which library and protocol each application uses, and test CAS behavior after any client upgrade.
• Alert on the ratio. Alert on cas_badval / (cas_hits + cas_misses + cas_badval) > 0.10 sustained for 10 minutes to catch contention before retry loops amplify it.

How Netdata helps

• Collects cas_badval, cas_hits, and cas_misses at per-second resolution, so the contention ratio can be computed without manual nc sampling.
• Correlates CAS counters with cmd_get, cmd_set, evictions, and hit ratio in the same dashboard, making it fast to distinguish write contention from eviction-driven cas_misses.
• Per-slab metrics from stats slabs surface whether contention concentrates in a single item-size class.
• ML anomaly detection flags unusual shifts in the cas_badval ratio even when the absolute counter values look normal for the current traffic level.

Related guides

• Memcached connection refused: telling a dead process from a hung or full one
• How Memcached actually works in production: a mental model for operators
• Memcached hit ratio dropping: reading get_hits, get_misses, and cache effectiveness
• Memcached monitoring checklist: the signals every production cache needs
• Memcached monitoring maturity model: from survival to expert
• Memcached alive but not responding: the silent process hang
• Memcached unexpected restart: uptime reset, wiped cache, and the cold-start backend spike