Memcached hot key: one key, one thread, and lopsided load

Aggregate cmd_get and cmd_set rates look normal. Hit ratio is healthy. Evictions are zero. CPU across the memcached fleet is moderate. Yet a specific subset of clients reports p99 latency an order of magnitude above baseline, and in a consistent-hashing cluster one node runs noticeably hotter than the rest. Memory pressure and connection exhaustion are absent.

The signature of a memcached hot key is concentration without saturation. A tiny number of keys, sometimes a single key, take a disproportionate share of traffic: a feature flag read on every request, a viral item page, a single bot’s session record, a shared rate-limit counter. All funnel traffic to one slab class on one node.

The standard dashboard tells you nothing is wrong. Aggregate throughput is fine because the hot key is small and fast to serve. Eviction and memory metrics are clean because the working set is tiny. The damage shows up in two places: client-side tail latency, where a small fraction of requests queue behind the others, and cluster-level imbalance, where one node handles a disproportionate share of total bytes and commands.

What this means

Memcached’s listener thread accepts connections and hands them round-robin to worker threads (configured with -t, default 4). Each worker runs its own libevent loop. The connection-to-thread assignment is per connection, not per key: requests for hot_key are not pinned to one worker. The literal reading of “one key, one thread” is misleading. The real serialization point at extreme rates is the per-slab-class lock protecting the LRU and hash table entries for the size class holding the hot key.

At normal rates this lock is invisible. At extreme rates, sustained GETs and SETs to the same slab class from multiple workers converge on the same mutex, and the queueing cost shows up as elevated tail latency for the connections serving those requests. The segmented LRU (default since 1.5.0) splits each slab class into HOT, WARM, and COLD sub-LRUs with separate locks; the optional TEMP sub-LRU is off by default and is enabled by temporary_ttl. item bumps on fetch are asynchronous. This reduces contention compared to the pre-1.5.0 flat per-class LRU but does not eliminate it. Under a single hot key at very high rates, residual lock contention is real.

In a consistent-hashing cluster the bigger problem is structural: a given key always hashes to the same node. You cannot load-balance a single hot key by adding nodes. The node that owns the key takes all traffic for it; the other nodes stay relatively idle. The fix has to come from the client or proxy layer, not from resizing the ring.

flowchart TD
    A[Many clients] -->|GET hot_key| B{consistent hash}
    B -->|same hash, always| C[Node 3]
    B -->|other keys| D[Node 1]
    B -->|other keys| E[Node 2]
    C -->|all workers converge| F[slab class N lock]
    F -->|queueing| G[tail latency spike]
    D -.->|underused| H[low CPU]
    E -.->|underused| I[low CPU]

Common causes

Cause	What it looks like	First thing to check
Feature flag or config blob cached as one key	Every request reads the same key; small value; very high GET rate, low SET rate	Application code: one shared cache key on the request hot path?
Viral item or page	Sudden imbalance on one node; traffic spike to one entity ID	Application telemetry: one entity ID with anomalous read volume
Single bot or high-traffic user session	One session key with extreme GET/SET churn; session slab class disproportionately hot	Per-user or per-session access logs
Shared counter or rate-limit key	Global counter used by every request via incr/decr; high `incr_hits` in one slab class	Application code: any global counter shared across all requests

Quick checks

All read-only and safe in production. The stats family of commands can cause brief latency under extreme throughput (above roughly 500k ops/sec), so sample at 10-second or longer intervals, and poll stats slabs / stats items less frequently than plain stats.

# Confirm aggregate throughput and hit ratio look healthy
echo "stats" | nc -w 2 localhost 11211 | grep -E "STAT (cmd_get|cmd_set|get_hits|get_misses|evictions)"

# Confirm the hot node is not memory- or connection-bound
echo "stats" | nc -w 2 localhost 11211 | grep -E "STAT (bytes |limit_maxbytes|curr_connections|max_connections)"

# Per-slab-class command distribution
echo "stats slabs" | nc -w 2 localhost 11211 | grep -E "(used_chunks|free_chunks|get_hits|cmd_set)"

# Worker thread count
echo "stats settings" | nc -w 2 localhost 11211 | grep -E "STAT num_threads"
# Per-thread CPU (look for one worker near 100% while others are idle)
# ps pcpu is a lifetime average; use top -H for a real-time snapshot
top -b -n 1 -H -p "$(pgrep memcached | head -1)" | head -20

# Compare command and byte throughput across cluster nodes
# These are cumulative counters; sample twice to derive rates
for host in node1 node2 node3; do
  echo "== $host =="
  echo "stats" | nc -w 2 "$host" 11211 | grep -E "STAT (cmd_get|bytes_written)"
done

The most diagnostic single observation is the cluster comparison at the bottom. If one node’s cmd_get and bytes_written counters are climbing several times faster than the others while the cluster as a whole looks normal, you have a concentration problem.

How to diagnose it

Rule out memory and connection pressure first. If evictions is non-zero or curr_connections / max_connections is over 0.8, you are looking at a different failure pattern, not a pure hot key.
Compare per-node command and byte rates across the ring. A single node running 3x to 5x hotter than the median is the cluster-level fingerprint.
Look for client-side tail latency skew. Memcached does not expose per-key or per-operation latency histograms natively; this must come from client library metrics, application tracing, or an external probe such as memtier_benchmark against a representative workload.
Correlate the hot node with a slab class. stats slabs per-class get_hits will be heavily concentrated in one class.
Identify the hot key. Memcached has no native per-key access statistics. Realistic options: application-level telemetry logging key access counts, client library instrumentation, or, for small slab classes only, stats cachedump (limited and effectively deprecated). For larger caches, lru_crawler metadump all dumps key metadata but is expensive; do not run it in production without understanding the impact.
Validate the hypothesis by temporarily replicating the suspected hot key under variant names (see Fixes) and confirming the latency skew drops.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Per-node `cmd_get` and `bytes_written` rate	Cluster balance; a hot key always hashes to one node	One node 3x or more above the cluster median
Per-slab-class `get_hits` from `stats slabs`	Concentration within a single size class	One slab class accounts for a disproportionate share of total gets
`rusage_user` and `rusage_system` rate	Per-thread saturation hides in aggregate stats	A single worker thread near 100% while others are underutilized
Client-observed p99 latency	Memcached has no server-side latency histogram	p99 spikes while p50 stays normal
`conn_yields`	One client dominating the per-event request limit (`-R`, default 20)	Sustained non-zero rate from one client
Cluster-wide `cmd_get` balance	Same signal as per-node, expressed as distribution	Spread across nodes trending wider over time

Fixes

Client-side key replication

The most portable fix. Store the hot value under N variant keys (for example feature_flag:1, feature_flag:2, feature_flag:3) and have clients pick one at random per read. In a consistent-hashing ring the variants hash to different nodes, spreading the read load across the ring. Writes become N times more expensive, so this is only appropriate for keys whose write rate is much lower than the read rate. Invalidation has to update all variants, or you accept short-lived inconsistency windows. This is the only option that works without changing memcached versions or running a proxy.

In-process caching for rarely-changing keys

For keys that change rarely and are read on every request, the right answer is often to not ask memcached at all. Cache the value in the application process with a short TTL (seconds to a minute) and refresh asynchronously. This removes the request from the memcached hot path entirely. Feature flags and config blobs are the canonical case. The tradeoff is a small inconsistency window on config changes, which is usually acceptable for slowly-changing data.

Built-in proxy with prefix-based replication (1.6.23+)

Memcached 1.6.23+ ships a built-in proxy that supports prefix-based routing and replication. The prefix-router pattern uses route_failover with shuffle = true and miss = true to spread reads across replicated backends for a given key prefix. This is the recommended first-line defense if you can run a recent enough version. The proxy is not enabled in default package builds; you must compile from source with proxy support. Mcrouter and Twemproxy are third-party alternatives that provide similar hot-key replication behavior at the proxy layer.

Reshard or add nodes (does not help a single hot key)

Adding nodes to a consistent-hashing ring does not help a single hot key, because the key still hashes to exactly one node. Resharding helps only when many moderately-hot keys are unevenly distributed. For a true single-key hotspot, the fix has to be at the client or proxy layer. Listing this as a non-fix is intentional: it is the most common wrong move operators reach for.

Prevention

Audit shared hot-path keys. Any single key read on every request is a future incident. Feature flags, config blobs, schema versions, and global counters are the usual suspects.
Decide per key: memcached or process memory. If a key changes less often than the request rate, process-local caching with a short TTL is usually better than a remote cache lookup.
Instrument client libraries for per-key access counts or top-N keys. Memcached will not give you this signal server-side.
Track cluster-wide balance as a first-class metric. Per-node cmd_get and bytes_written should be within a small multiple of each other. A widening spread is the earliest signal of a developing hotspot.
Classify each new shared cache key by access pattern. Uniform access is fine; concentrated access needs replication or in-process caching from day one.

How Netdata helps

Per-second collection of cmd_get, cmd_set, bytes_written, and related counters per node makes cluster imbalance visible as soon as it develops, not at the next 60-second polling interval.
Per-slab-class metrics from stats slabs and stats items surface the concentration of get_hits in one size class, which is the instance-level fingerprint of a hot key.
ML-based anomaly detection on per-node throughput flags the lopsided-load pattern before aggregate dashboards would, because the anomaly is in the distribution across nodes, not in the cluster total.
Correlation between client-side latency probes (when instrumented) and per-node rusage rates helps separate a hot-key problem from genuine CPU saturation.
Tracking conn_yields alongside command rates distinguishes a single aggressive client from a true key-concentration problem, which points the fix in a different direction.