Memcached command rate anomalies: cmd_get and cmd_set spikes and sudden drops

cmd_get, cmd_set, and cmd_touch are cumulative counters in memcached. They increase monotonically from process start and are meaningless as raw numbers. The signal is the derived rate: the delta between two samples divided by the interval. A sudden change in that rate is almost never the root cause. It is a proxy for something that changed upstream (clients stopped or started sending traffic), inside the cache (a hot key expired, the cache was flushed), or in the application logic (a write loop, a deploy with new key patterns).

Command rate anomalies are detection signals, not diagnoses. A spike above 2x baseline or a drop below 10-50% tells you something shifted. The investigation correlates the rate change with hit ratio, eviction rate, connection state, and backend load to find the cause.

One subtlety that catches teams off guard: cmd_get counts every requested key, including each key inside a multiget. A single multiget for 50 keys increments cmd_get by 50, and get_hits and get_misses count those same keys individually. The real lookup volume is always get_hits + get_misses, and over the same interval its delta matches the cmd_get delta.

Reading command rates

These counters come from the stats command. There is no native rate metric from the server; sample twice with a known interval and compute the delta.

# Compute cmd_get rate over a 5-second window
A=$(echo "stats" | nc -w 2 localhost 11211 | awk '/STAT cmd_get/ {print $3}')
sleep 5
B=$(echo "stats" | nc -w 2 localhost 11211 | awk '/STAT cmd_get/ {print $3}')
echo "cmd_get rate: $(( (B - A) / 5 )) ops/sec"

Anomaly thresholds, drawn from operational consensus rather than a hard server limit:

These are heuristics. The real value is anomaly detection against a time-of-day and day-of-week baseline. A cache handling 500K gets/sec with 100 evictions/sec may be perfectly healthy. A cache at 1K gets/sec with the same eviction rate is in crisis. Absolute thresholds without baselines generate false positives and miss real problems.

flowchart TD
    A[Command rate anomaly] --> B{Spike or drop?}
    B -->|Spike| C{cmd_get or cmd_set?}
    B -->|Drop| D{Connections at max?}
    C -->|cmd_get up, misses up| E[Cache stampede]
    C -->|cmd_get up, hit ratio stable| F[Traffic surge or retry storm]
    C -->|cmd_set up, cmd_get flat| G[Write loop or bulk load]
    D -->|accepting_conns=0| H[Connection exhaustion]
    D -->|uptime low or cmd_flush up| I[Restart or flush_all]
    D -->|Stable connections and uptime| J[Upstream failure or DNS change]

Common causes

Quick checks

Run these read-only probes to classify the anomaly. All are safe for production. Avoid polling stats more frequently than every 10 seconds under extreme throughput (above 500K ops/sec); the stats command itself adds load.

# Command counters
echo "stats" | nc -w 2 localhost 11211 | grep -E "STAT cmd_(get|set|touch|flush)"

# Hit and miss breakdown (key-level lookup volume)
echo "stats" | nc -w 2 localhost 11211 | grep -E "STAT get_(hits|misses|flushed)"

# Uptime for restart detection
echo "stats" | nc -w 2 localhost 11211 | grep "STAT uptime"

# Connection state and rejection counters
echo "stats" | nc -w 2 localhost 11211 | grep -E "STAT (curr_connections|max_connections|rejected_connections|accepting_conns|listen_disabled_num)"

# Eviction rate and memory pressure
echo "stats" | nc -w 2 localhost 11211 | grep -E "STAT (evictions|bytes |limit_maxbytes)"

# Network throughput (NIC saturation from large-value reads)
echo "stats" | nc -w 2 localhost 11211 | grep -E "STAT bytes_(read|written)"

# CPU usage
echo "stats" | nc -w 2 localhost 11211 | grep "STAT rusage"

# Connection yields (one client dominating the server)
echo "stats" | nc -w 2 localhost 11211 | grep "STAT conn_yields"

# Item count (mass expiration or flush detection)
echo "stats" | nc -w 2 localhost 11211 | grep "STAT curr_items"

How to diagnose it

1. Confirm the direction and magnitude. Sample cmd_get and cmd_set twice with a known interval. Determine whether the anomaly is a spike (above 2x baseline) or a drop (below 10-50% of baseline). Compare against your time-of-day baseline, not an absolute number.

2. If it is a spike in cmd_get, check hit ratio direction. Pull get_hits and get_misses and compute the ratio from deltas. If get_misses is climbing while get_hits stays flat, you have a cache stampede: a hot key expired or was evicted and every client is missing simultaneously. If get_hits is climbing proportionally with get_misses, traffic genuinely increased. See Memcached cache stampede: a hot key expires and the backend takes the hit.

3. If it is a spike in cmd_set without a proportional cmd_get spike, look for a write loop. Check evicted_unfetched and expired_unfetched. If these are climbing alongside the cmd_set spike, the application is caching data nobody reads. Correlate with recent deploys.

4. If it is a drop, check connections first. Look at accepting_conns, curr_connections, max_connections, and rejected_connections. If accepting_conns = 0 or rejected_connections is incrementing, the cache hit its connection limit and clients are being turned away. See Memcached connection limit reached: accepting_conns=0 and clients being refused.

5. Check uptime and cmd_flush. If uptime reset, the process restarted and the cache is cold. If cmd_flush incremented, someone or something issued flush_all. Both produce a temporary cmd_get and cmd_set drop followed by a cold-cache pattern: high get_misses, climbing curr_items as the cache warms, and backend load spike. See Memcached flush_all: the accidental cache wipe and its cold-start blast radius.

6. If connections are stable and the process is responsive but traffic stopped, investigate upstream. A cmd_get drop to near zero with stable curr_connections, stable uptime, and a responsive version probe means the server is fine but clients stopped sending. Check client-side server lists, DNS resolution, and load balancer configuration. A DNS change or a stray space character in a server name in client config can silently redirect traffic.

7. Correlate with backend load. A cmd_get spike with rising backend load and rising get_misses confirms a stampede. A cmd_get drop with rising backend load suggests clients are bypassing the cache and hitting the database directly. A cmd_get spike with stable backend load means the cache is absorbing the increase.

8. Check conn_yields if one client seems to dominate. High conn_yields during a cmd_get spike means a single connection is pipelining requests faster than the server’s -R limit (default 20 requests per event) allows. The server forces it to yield so other connections get served. See Memcached conn_yields rising: one client’s pipeline starving the others.

Metrics and signals to monitor

Fixes

Cache stampede (cmd_get spike with rising misses)

Identify the hot key through application telemetry if possible. Manually warm the key to reduce the miss storm. For long-term prevention, implement stampede protection in the application layer: distributed locks around cache regeneration, probabilistic early expiration, or stale-while-revalidate patterns. The tradeoff is added application complexity and a small latency cost on the first miss after expiration. See Memcached cache stampede: a hot key expires and the backend takes the hit.

Client retry storm (cmd_get spike with stable hit ratio)

Identify which client is generating the traffic. Check client-side error rates, retry counts, and backoff configuration. A common cause is a client library retrying failed operations with no backoff, amplifying a transient blip into a sustained spike. Reduce retry aggressiveness or add exponential backoff. The tradeoff: reducing retries may increase latency for genuine transient failures.

If conn_yields is high alongside the spike, one client is pipelining aggressively. Check whether the -R limit (default 20) is appropriate for your workload, or throttle the client-side pipeline depth.

Application write loop (cmd_set spike without cmd_get spike)

High evicted_unfetched and expired_unfetched confirm the application is writing data nobody reads. Correlate with recent deploys to identify the code path. Common causes: cache-aside logic that writes to cache after every database write regardless of read demand, background jobs populating keys that are never requested, or a serialization change producing oversized items. Fix at the application level: only cache data that is actually read, and review TTLs for write-heavy patterns. See Memcached evicted_unfetched and expired_unfetched: caching data nobody ever reads.

Connection exhaustion (cmd_get and cmd_set drop, accepting_conns=0)

Verify max_connections and the OS ulimit -n for the memcached process. If curr_connections is stable at max but total_connections rate is very high, the problem is connection churn (missing pooling), not a leak. If curr_connections is monotonically increasing, it is a leak. Increasing -c requires a restart. Fix the root cause first: implement or fix client-side connection pooling. See Memcached connection limit reached: accepting_conns=0 and clients being refused and Memcached curr_connections climbing: connection leaks and missing pooling.

Upstream failure or DNS change (cmd_get drop, connections stable)

The cache is healthy but clients stopped sending traffic. Check client-side server lists, DNS resolution, and load balancer health. A DNS change pointing clients to a different IP, a misconfigured server name (including whitespace), or a load balancer that removed the memcached node from its pool all produce this pattern. Verify with ss -tn | grep :11211 to see which clients are actually connected. The fix is client-side configuration. Note that client-side DNS caching can delay detection of legitimate changes.

Restart or flush_all (uptime reset or cmd_flush increment)

After a restart or flush_all, the cache is cold. Focus on the backend: it is now receiving the full production request load the cache was absorbing. Monitor hit ratio recovery and backend load. Run cache-warming scripts if available. Implement circuit breakers in the application layer if the backend cannot absorb the cold-cache load. Do not restart memcached as a fix unless there is a confirmed process-level issue. See Memcached flush_all: the accidental cache wipe and its cold-start blast radius.

Prevention

• Baseline by time-of-day and day-of-week. Command rates are workload-driven and follow predictable patterns. Alert on deviation from baseline, not absolute thresholds.
• Alert on rate-of-change, not raw counters. The counters are cumulative. The useful signal is the derivative.
• Break the cmd_get delta into hits vs misses. cmd_get counts every requested key; get_hits + get_misses shows whether the movement came from hits or misses.
• Implement stampede prevention. Distributed locks, probabilistic early expiration, or stale-while-revalidate at the application layer prevent the most common cmd_get spike pattern.
• Alert on any cmd_flush increment in production. A single accidental flush_all causes a cold-cache thundering herd.
• Monitor connection state alongside command rates. A command rate drop caused by connection exhaustion looks identical to a client-side failure until you check accepting_conns and rejected_connections.
• Track evicted_unfetched and expired_unfetched. These reveal write-only caching patterns that waste memory and can mask as a cmd_set anomaly.

How Netdata helps

• Per-second rate derivation. Netdata computes cmd_get, cmd_set, and cmd_touch rates from counter deltas automatically, exposing sub-minute anomalies that 10-15 second polling intervals miss.
• ML anomaly detection on command rates. Netdata’s anomaly advisor learns the time-of-day baseline for each command type and flags deviations without manual threshold tuning.
• Correlated hit ratio and miss breakdown. When cmd_get spikes, get_hits, get_misses, and hit ratio appear in the same view, making stampede detection immediate.
• Connection state alongside throughput. accepting_conns, curr_connections, rejected_conns, and conn_yields appear alongside command rates, so a drop caused by connection exhaustion is distinguishable from a client-side failure in one view.
• Flush and restart detection. cmd_flush increments and uptime resets surface as discrete events, making cold-cache incidents immediately classifiable.

Related guides

• Memcached cache stampede: a hot key expires and the backend takes the hit
• Memcached evicted_unfetched and expired_unfetched: caching data nobody ever reads
• Memcached cas_badval climbing: check-and-set contention and lost updates
• Memcached conn_yields rising: one client’s pipeline starving the others
• Memcached connection churn: total_connections racing and TIME_WAIT buildup
• Memcached curr_connections climbing: connection leaks and missing pooling
• Memcached connection limit reached: accepting_conns=0 and clients being refused
• Memcached connection refused: telling a dead process from a hung or full one
• Memcached evicted_time low: distinguishing healthy turnover from cache thrash
• Memcached eviction cascade: when a full cache overloads the backend
• Memcached evictions climbing: the cache is full and discarding live data
• Memcached flush_all: the accidental cache wipe and its cold-start blast radius