Memcached monitoring maturity model: from survival to expert

Memcached exposes most of its internal state through plain text counters. Most teams stop at hit ratio, global memory, and eviction rate, then discover during an incident that the signal they needed was buried in stats items or /proc/<pid>/status. The four-level model below stages the path from “is the process alive” to “is the segmented LRU moving items between HOT and COLD the way the workload expects”.

Use the levels as a checklist, not a sequence. Each level answers a class of question the previous one could not. Skipping straight to Level 4 without Level 2 fundamentals produces dashboards full of internal counters with no baseline for client impact. The most common gap is between Level 2 and Level 3: operational coverage looks healthy on global metrics while one slab class is silently evicting recently-used items, because the slab allocator partitions memory by item size and the global bytes counter averages across all classes.

The boundaries are deliberate. Survival keeps the service nominally up. Operational proves the cache is doing useful work without quietly swapping or wasting memory. Mature is where slab calcification, LRU maintainer starvation, and write contention become visible. Expert covers deployments running extstore, TLS, very high per-thread throughput, or sharded clusters where cross-node balance matters.

flowchart TD
  L4["Level 4 Expert
per-thread CPU, extstore, TLS, LRU segment dynamics"]
  L3["Level 3 Mature
per-slab eviction age, direct_reclaims, crawler, automove"]
  L2["Level 2 Operational
command mix, network bytes, swap, client latency"]
  L1["Level 1 Survival
process alive, hit ratio, evictions, connections"]
  L1 --> L2 --> L3 --> L4

Level 1: survival

The minimum signals that tell you memcached is alive and not actively destroying cache effectiveness. If any of these break, you have an incident regardless of what else is monitored.

Signal	What it answers	First thing to check
Process liveness	Is the daemon responding to commands?	`echo "version" \| nc -q1 localhost 11211`
`uptime`	Did the process restart?	`STAT uptime` from `stats`; a reset implies full data loss
Hit ratio	Is the cache offloading the backend?	`get_hits / (get_hits + get_misses)` from deltas
Eviction rate	Is memory under pressure?	`rate(evictions)` from `stats`
`bytes / limit_maxbytes`	Is global memory near the ceiling?	Both fields from `stats`
`curr_connections / max_connections`	Are we approaching the connection limit?	Both fields from `stats`
`rejected_connections`, `listen_disabled_num`	Have clients been turned away?	Counters from `stats`

The -q1 flag on the liveness probe is openbsd-netcat specific. On systems shipping traditional netcat, substitute an equivalent timeout or use memcapable / a language-specific client. Always probe with an actual command, not a TCP port check: a process that accepts connections but does not respond to commands is effectively down and will pass a naive port check.

Survival-level alerts are blunt by design. A failed version probe sustained for 30 to 60 seconds is a page. A curr_connections ratio above 0.95 or a listen_disabled_num increment is a ticket. Hit ratio and eviction rate are workload-relative and need baselines, but a sudden hit ratio collapse alongside a low uptime is the canonical cold-cache thundering herd pattern and should page only if the backend is also showing load.

One Level 1 trap: global memory utilization can sit at 50% while one slab class is evicting aggressively. That is precisely the gap Level 3 closes.

Level 2: operational

Level 1 told you the service is up. Level 2 tells you whether the cache is doing useful work and whether the process is healthy from the OS perspective.

Signal	Why it matters
`cmd_get`, `cmd_set`, `cmd_touch`, `cmd_flush` rates	Establishes the workload profile. Any `cmd_flush` increment in production is a cache-wide data loss event.
`bytes_read`, `bytes_written` rates	Network I/O volume. `bytes_written` approaching NIC capacity is the hidden bottleneck for large-value workloads.
`conn_yields` rate	A connection hit the `-R` per-event request limit and was forced to yield. Sustained high rates indicate one or two clients dominating the server.
`evicted_unfetched`, `expired_unfetched` rates	Items stored but never read before being evicted or expiring. A high `evicted_unfetched / evictions` ratio means the application is caching write-only data.
`reclaimed` rate	Items whose expired slots were reused. High reclaim with low eviction is the ideal state and confirms the LRU crawler is doing its job.
Process RSS from `/proc/<pid>/status`	Total resident memory including slab memory, hash table, connection buffers. RSS more than roughly 1.4x `limit_maxbytes` suggests overhead growth.
`VmSwap` from `/proc/<pid>/status`	Must be zero. Any nonzero value is a production incident; the in-memory cache is now serving some items at disk speed.
Client-observed latency	Memcached exposes no latency histograms. p99 must be measured at the client. p99 above 5ms for same-datacenter traffic is abnormal.
`cmd_flush` change detection	Alert on any increment.

The composite connection-exhaustion pattern is the only connection-state condition worth paging on in isolation. It combines accepting_conns = 0, curr_connections / max_connections > 0.98, a positive rate on rejected_connections or time_in_listen_disabled_us, sustained for 5 minutes, with max_connections > 50 to exclude tiny instances. The sustained window filters out deploy-time reconnection storms and brief batch spikes.

Differentiate miss causes. High misses with high evictions means memory pressure. High misses with zero evictions means the application is requesting keys that were never set, or the cache is cold. These have completely different fixes and the global hit ratio counter does not distinguish them.

Level 3: mature

Level 3 is where per-slab instrumentation enters the picture. This is the level most teams never reach, and it is where the most common production incidents become diagnosable. Without it you cannot tell the difference between “the cache is too small” and “one slab class is starved while others have free pages”.

Signal	Source	What it tells you
Per-slab `evicted`, `age`, `free_chunks`, `used_chunks`, `mem_requested`	`stats items`, `stats slabs`	Identifies slab calcification: one class at zero free chunks and high evictions while others sit idle.
`evicted_time` per slab class	`stats items`	Seconds since last access of the most recently evicted item. Low values (under 300s) with active evictions mean recently-used data is being discarded.
Slab calcification detection	`stats slabs` cross-class comparison	The composite pattern: global bytes at 50 to 70% of limit, evictions climbing, one class at 100% `used_chunks`, others with significant `free_chunks`.
`direct_reclaims` per slab	`stats items`	Worker threads evicting directly instead of letting the LRU maintainer handle it. Sustained non-zero means the maintainer is falling behind.
`hash_is_expanding`, `hash_power_level`, `hash_bytes`	`stats`	Hash table growth state. Expansion runs in a background thread but doubles hash table memory temporarily.
`crawler_reclaimed`, `crawler_items_checked` ratio	`stats`	LRU crawler effectiveness. A dropping reclaim rate alongside active evictions suggests the crawler is disabled or stuck.
Slab automove status and activity	`stats settings`, `stats slabs`	Whether `slab_automove` is on (default mode 1 since 1.5.0). Mode 2 is aggressive and not recommended for long-term use.
`cas_badval` rate	`stats`	CAS validation failures. `cas_badval / (cas_hits + cas_misses + cas_badval)` above 10% indicates write contention on hot keys.
`incr_misses`, `decr_misses` rates	`stats`	Counter keys being evicted or expired between increments. Breaks rate limiters and distributed counters.
`response_obj_oom`	`stats`	Response buffer allocation failures forcing connection closes. Separate from item eviction memory; can occur with zero evictions.
`store_too_large`, `store_no_memory`	`stats`	`store_too_large` indicates oversized items (application serialization bug). `store_no_memory` indicates `-M` no-eviction mode is refusing writes.
`auth_cmds`, `auth_errors` (if auth enabled)	`stats`	SASL or ASCII auth activity. SASL requires the deprecated binary protocol; zero errors with auth disabled does not mean access is controlled.

The single most important Level 3 skill is reading evicted_time correctly. The counter is per-slab, only meaningful when that slab is actively evicting, and reports only the age of the last evicted item, not an average. A 3-day-old evicted item is healthy cache turnover. A 10-second-old evicted item means the cache is actively thrashing and the items being discarded would have been hit. Distinguishing the two is impossible from the global evictions counter alone.

Level 3 is also where flush_all becomes diagnosable rather than mysterious. cmd_flush increments, and the post-flush state shows curr_items declining and hit ratio dropping. The flush itself does not lock the server and does not free memory immediately; items are lazily invalidated on next access.

Level 4: expert

Level 4 is for deployments where the segmented LRU’s internal dynamics, per-thread CPU saturation, extstore, TLS, or cross-instance balance matter. These signals are added after a specific class of incident has already happened once.

Signal	What it reveals
`moves_to_cold`, `moves_to_warm`, `moves_within_lru` rates	Whether the segmented LRU is moving items between HOT, WARM, COLD, and TEMP the way the workload expects. `moves_to_warm / moves_to_cold` indicates the rescue rate for re-accessed items.
Per-slab fragmentation: `mem_requested / (used_chunks * chunk_size)`	Internal slab waste. Below 0.5 for a class means items are using less than half their allocated chunk, suggesting the `-f` growth factor needs tuning.
Cross-instance balance in sharded clusters	Per-node command rates, hit ratios, and evictions. One hot node indicates uneven key distribution; client libraries do the sharding, so memcached itself has no cluster-wide view.
Per-thread CPU utilization	Single worker thread at 100% while others sit idle. Aggregate `rusage` averages across threads and hides this. Requires OS-level per-thread monitoring of the memcached process.
TCP connection state via `ss -tn`	TIME_WAIT and CLOSE_WAIT accumulation indicating client-side connection handling problems. Memcached stats do not expose this.
`lrutail_reflocked` rate	Items at the LRU tail with nonzero refcount, blocking eviction. High values indicate large-value reads competing with memory pressure.
Item size distribution via `-o track_sizes`	Live histogram of item sizes, useful for sizing the slab growth factor and detecting serialization drift.
`total_malloced` vs `limit_maxbytes`	How much of the configured memory has actually been claimed by slabs. A low ratio means the working set has not yet demanded the full allocation.

Conditional Level 4 signals apply only to specific deployment variants:

extstore enabled : get_extstore, get_aborted_extstore, get_oom_extstore, recache_from_extstore, extstore_page_allocs, extstore_page_evictions, extstore_objects_evicted/read/written/used, extstore_bytes_*, extstore_io_queue. extstore adds disk I/O as a monitored resource.
TLS enabled : ssl_handshake_errors, ssl_proto_errors, ssl_new_sessions, time_since_server_cert_refresh. TLS splits messages above 16KB by default; buffer size is tunable via -o ssl_wbuf_size=. Compile-time opt-in.
SASL authenticated: auth_cmds and auth_errors. SASL requires the deprecated binary protocol, and auth_errors = 0 with auth disabled does not mean access is controlled.

Expert level also includes version-aware behavior changes that affect what the stats mean. Segmented LRU is default since 1.5.0. slab_automove mode 1 is default since 1.5.0. Treating all versions identically will produce wrong conclusions about which features are active.

How Netdata helps

Per-second deltas on cmd_get, cmd_set, cmd_touch, cmd_flush, and the full hit/miss family mean hit ratio is computed from rate windows rather than lifetime cumulative counters.
Per-slab charts surface evicted, age, free_chunks, used_chunks, and mem_requested per class, making slab calcification visible without manually diffing stats slabs output between polls.
evicted_time is tracked per slab class alongside eviction rate, so the “evicting cold items vs evicting recently-used items” distinction shows up as a correlated pair.
OS-level signals (RSS, VmSwap, per-thread CPU, NIC utilization, TCP state) are collected from the same agent, which matters most for silent-degradation patterns where memcached’s own stats look healthy but the process is partially swapped or a single worker thread is pinned.
Anomaly detection on rate-of-change signals catches the gradual hit-ratio decline and the slow per-slab free_chunks trend that absolute thresholds miss.
cmd_flush increments and get_flushed rates land on the same timeline, so post-flush cold-cache events are diagnosable from a single view rather than a log scrape plus a manual stats poll.