Memcached exposes most of its internal state through plain text counters. Most teams stop at hit ratio, global memory, and eviction rate, then discover during an incident that the signal they needed was buried in stats items or /proc/<pid>/status. The four-level model below stages the path from “is the process alive” to “is the segmented LRU moving items between HOT and COLD the way the workload expects”.
Use the levels as a checklist, not a sequence. Each level answers a class of question the previous one could not. Skipping straight to Level 4 without Level 2 fundamentals produces dashboards full of internal counters with no baseline for client impact. The most common gap is between Level 2 and Level 3: operational coverage looks healthy on global metrics while one slab class is silently evicting recently-used items, because the slab allocator partitions memory by item size and the global bytes counter averages across all classes.
The boundaries are deliberate. Survival keeps the service nominally up. Operational proves the cache is doing useful work without quietly swapping or wasting memory. Mature is where slab calcification, LRU maintainer starvation, and write contention become visible. Expert covers deployments running extstore, TLS, very high per-thread throughput, or sharded clusters where cross-node balance matters.
flowchart TD L4["Level 4 Expert
per-thread CPU, extstore, TLS, LRU segment dynamics"] L3["Level 3 Mature
per-slab eviction age, direct_reclaims, crawler, automove"] L2["Level 2 Operational
command mix, network bytes, swap, client latency"] L1["Level 1 Survival
process alive, hit ratio, evictions, connections"] L1 --> L2 --> L3 --> L4
Level 1: survival
The minimum signals that tell you memcached is alive and not actively destroying cache effectiveness. If any of these break, you have an incident regardless of what else is monitored.
| Signal | What it answers | First thing to check |
|---|---|---|
| Process liveness | Is the daemon responding to commands? | echo "version" | nc -q1 localhost 11211 |
uptime | Did the process restart? | STAT uptime from stats; a reset implies full data loss |
| Hit ratio | Is the cache offloading the backend? | get_hits / (get_hits + get_misses) from deltas |
| Eviction rate | Is memory under pressure? | rate(evictions) from stats |
bytes / limit_maxbytes | Is global memory near the ceiling? | Both fields from stats |
curr_connections / max_connections | Are we approaching the connection limit? | Both fields from stats |
rejected_connections, listen_disabled_num | Have clients been turned away? | Counters from stats |
The -q1 flag on the liveness probe is openbsd-netcat specific. On systems shipping traditional netcat, substitute an equivalent timeout or use memcapable / a language-specific client. Always probe with an actual command, not a TCP port check: a process that accepts connections but does not respond to commands is effectively down and will pass a naive port check.
Survival-level alerts are blunt by design. A failed version probe sustained for 30 to 60 seconds is a page. A curr_connections ratio above 0.95 or a listen_disabled_num increment is a ticket. Hit ratio and eviction rate are workload-relative and need baselines, but a sudden hit ratio collapse alongside a low uptime is the canonical cold-cache thundering herd pattern and should page only if the backend is also showing load.
One Level 1 trap: global memory utilization can sit at 50% while one slab class is evicting aggressively. That is precisely the gap Level 3 closes.
Level 2: operational
Level 1 told you the service is up. Level 2 tells you whether the cache is doing useful work and whether the process is healthy from the OS perspective.
| Signal | Why it matters |
|---|---|
cmd_get, cmd_set, cmd_touch, cmd_flush rates | Establishes the workload profile. Any cmd_flush increment in production is a cache-wide data loss event. |
bytes_read, bytes_written rates | Network I/O volume. bytes_written approaching NIC capacity is the hidden bottleneck for large-value workloads. |
conn_yields rate | A connection hit the -R per-event request limit and was forced to yield. Sustained high rates indicate one or two clients dominating the server. |
evicted_unfetched, expired_unfetched rates | Items stored but never read before being evicted or expiring. A high evicted_unfetched / evictions ratio means the application is caching write-only data. |
reclaimed rate | Items whose expired slots were reused. High reclaim with low eviction is the ideal state and confirms the LRU crawler is doing its job. |
Process RSS from /proc/<pid>/status | Total resident memory including slab memory, hash table, connection buffers. RSS more than roughly 1.4x limit_maxbytes suggests overhead growth. |
VmSwap from /proc/<pid>/status | Must be zero. Any nonzero value is a production incident; the in-memory cache is now serving some items at disk speed. |
| Client-observed latency | Memcached exposes no latency histograms. p99 must be measured at the client. p99 above 5ms for same-datacenter traffic is abnormal. |
cmd_flush change detection and get_flushed | Alert on any cmd_flush increment; get_flushed counts GETs that found flushed items. |
The composite connection-exhaustion pattern is the only connection-state condition worth paging on in isolation. It combines accepting_conns = 0, curr_connections / max_connections > 0.98, a positive rate on rejected_connections or time_in_listen_disabled_us, sustained for 5 minutes, with max_connections > 50 to exclude tiny instances. The sustained window filters out deploy-time reconnection storms and brief batch spikes.
Differentiate miss causes. High misses with high evictions means memory pressure. High misses with zero evictions means the application is requesting keys that were never set, or the cache is cold. These have completely different fixes and the global hit ratio counter does not distinguish them.
Level 3: mature
Level 3 is where per-slab instrumentation enters the picture. This is the level most teams never reach, and it is where the most common production incidents become diagnosable. Without it you cannot tell the difference between “the cache is too small” and “one slab class is starved while others have free pages”.
| Signal | Source | What it tells you |
|---|---|---|
Per-slab evicted, age, free_chunks, used_chunks, mem_requested | stats items, stats slabs | Identifies slab calcification: one class at zero free chunks and high evictions while others sit idle. |
evicted_time per slab class | stats items | Seconds since last access of the most recently evicted item. Low values (under 300s) with active evictions mean recently-used data is being discarded. |
| Slab calcification detection | stats slabs cross-class comparison | The composite pattern: global bytes at 50 to 70% of limit, evictions climbing, one class at 100% used_chunks, others with significant free_chunks. |
direct_reclaims per slab | stats items | Worker threads evicting directly instead of letting the LRU maintainer handle it. Sustained non-zero means the maintainer is falling behind. |
hash_is_expanding, hash_power_level, hash_bytes | stats | Hash table growth state. Expansion runs in a background thread but doubles hash table memory temporarily. |
crawler_reclaimed, crawler_items_checked ratio | stats | LRU crawler effectiveness. A dropping reclaim rate alongside active evictions suggests the crawler is disabled or stuck. |
| Slab automove status and activity | stats settings, stats slabs | Whether slab_automove is on (default mode 1 since 1.5.0). Mode 2 is aggressive and not recommended for long-term use. |
cas_badval rate | stats | CAS validation failures. cas_badval / (cas_hits + cas_misses + cas_badval) above 10% indicates write contention on hot keys. |
incr_misses, decr_misses rates | stats | Counter keys being evicted or expired between increments. Breaks rate limiters and distributed counters. |
response_obj_oom | stats (1.6.0+) | Response buffer allocation failures forcing connection closes. Separate from item eviction memory; can occur with zero evictions. |
store_too_large, store_no_memory | stats | store_too_large indicates oversized items (application serialization bug). store_no_memory indicates -M no-eviction mode is refusing writes. |
auth_cmds, auth_errors (if auth enabled) | stats | SASL or ASCII auth activity. SASL requires the deprecated binary protocol; zero errors with auth disabled does not mean access is controlled. |
The single most important Level 3 skill is reading evicted_time correctly. The counter is per-slab, only meaningful when that slab is actively evicting, and reports only the age of the last evicted item, not an average. A 3-day-old evicted item is healthy cache turnover. A 10-second-old evicted item means the cache is actively thrashing and the items being discarded would have been hit. Distinguishing the two is impossible from the global evictions counter alone.
Level 3 is also where flush_all becomes diagnosable rather than mysterious. cmd_flush increments, and the post-flush state shows curr_items declining and hit ratio dropping. The flush itself does not lock the server and does not free memory immediately; items are lazily invalidated on next access.
Level 4: expert
Level 4 is for deployments where the segmented LRU’s internal dynamics, per-thread CPU saturation, extstore, TLS, or cross-instance balance matter. These signals are added after a specific class of incident has already happened once.
| Signal | What it reveals |
|---|---|
moves_to_cold, moves_to_warm, moves_within_lru rates | Whether the segmented LRU is moving items between HOT, WARM, COLD, and TEMP the way the workload expects. moves_to_warm / moves_to_cold indicates the rescue rate for re-accessed items. |
Per-slab fragmentation: mem_requested / (used_chunks * chunk_size) | Internal slab waste. Below 0.5 for a class means items are using less than half their allocated chunk, suggesting the -f growth factor needs tuning. |
| Cross-instance balance in sharded clusters | Per-node command rates, hit ratios, and evictions. One hot node indicates uneven key distribution; client libraries do the sharding, so memcached itself has no cluster-wide view. |
| Per-thread CPU utilization | Single worker thread at 100% while others sit idle. Aggregate rusage averages across threads and hides this. Requires OS-level per-thread monitoring of the memcached process. |
TCP connection state via ss -tn | TIME_WAIT and CLOSE_WAIT accumulation indicating client-side connection handling problems. Memcached stats do not expose this. |
lrutail_reflocked rate | Items at the LRU tail with nonzero refcount, blocking eviction. High values indicate large-value reads competing with memory pressure. |
Item size distribution via -o track_sizes (available since 1.4.27; runtime enable/disable removed in 1.6.25) | Live histogram of item sizes, useful for sizing the slab growth factor and detecting serialization drift. |
total_malloced vs limit_maxbytes | How much of the configured memory has actually been claimed by slabs. A low ratio means the working set has not yet demanded the full allocation. |
Conditional Level 4 signals apply only to specific deployment variants:
- extstore enabled (feature introduced in 1.5.4; compiled in by default since 1.6.0 but still enabled at startup):
get_extstore,get_aborted_extstore,get_oom_extstore,recache_from_extstore,extstore_page_allocs,extstore_page_evictions,extstore_objects_evicted/read/written/used,extstore_bytes_*,extstore_io_queue. extstore adds disk I/O as a monitored resource. - TLS enabled (experimental support introduced in 1.5.13; TLS stats vary by version):
ssl_handshake_errors,ssl_proto_errors,ssl_new_sessions,time_since_server_cert_refresh. TLS splits messages above 16KB by default; buffer size is tunable via-o ssl_wbuf_size=. Compile-time opt-in. - SASL authenticated:
auth_cmdsandauth_errors. SASL requires the deprecated binary protocol, andauth_errors = 0with auth disabled does not mean access is controlled.
Expert level also includes version-aware behavior changes that affect what the stats mean. Segmented LRU is default since 1.5.0. slab_automove mode 1 is default since 1.5.0. UDP is disabled by default since 1.5.6, the release that mitigated CVE-2018-1000115. The binary protocol is officially deprecated since 1.6.0 in favor of the meta and text protocols. Treating all versions identically will produce wrong conclusions about which features are active.
How Netdata helps
- Per-second deltas on
cmd_get,cmd_set,cmd_touch,cmd_flush, and the full hit/miss family mean hit ratio is computed from rate windows rather than lifetime cumulative counters. - Per-slab charts surface
evicted,age,free_chunks,used_chunks, andmem_requestedper class, making slab calcification visible without manually diffingstats slabsoutput between polls. evicted_timeis tracked per slab class alongside eviction rate, so the “evicting cold items vs evicting recently-used items” distinction shows up as a correlated pair.- OS-level signals (RSS, VmSwap, per-thread CPU, NIC utilization, TCP state) are collected from the same agent, which matters most for silent-degradation patterns where memcached’s own stats look healthy but the process is partially swapped or a single worker thread is pinned.
- Anomaly detection on rate-of-change signals catches the gradual hit-ratio decline and the slow per-slab
free_chunkstrend that absolute thresholds miss. cmd_flushincrements andget_flushedrates land on the same timeline, so post-flush cold-cache events are diagnosable from a single view rather than a log scrape plus a manual stats poll.
Related guides
- Memcached monitoring checklist: the signals every production cache needs
- How Memcached actually works in production: a mental model for operators
- Memcached connection refused: telling a dead process from a hung or full one
- Memcached alive but not responding: the silent process hang
- Memcached unexpected restart: uptime reset, wiped cache, and the cold-start backend spike
- Memcached hit ratio dropping: reading get_hits, get_misses, and cache effectiveness
- Memcached high miss rate: separating cold start, new key patterns, and memory pressure
- Memcached evicted_unfetched and expired_unfetched: caching data nobody ever reads
- Memcached incr/decr misses: evicted counters that silently break rate limiters and locks
- Memcached cas_badval climbing: check-and-set contention and lost updates
- Memcached evictions climbing: the cache is full and discarding live data
- Memcached evicted_time low: distinguishing healthy turnover from cache thrash