Memcached monitoring checklist: the signals every production cache needs

A production reference for memcached signals, organized by monitoring maturity. Use it to audit what you collect, identify gaps, and prioritize additions.

Levels are cumulative: Level 2 includes Level 1. Level 1 detects crashes and restarts but misses slab calcification and eviction quality. Level 3 catches those before they hit the backend. Level 4 adds signals teams adopt after being burned by subtle failures.

Memcached does not expose latency histograms, per-key statistics, or source IP attribution natively. Several critical signals come from the OS (process state, swap, file descriptors, NIC counters), not from the stats command. Sources are noted where that applies.

The maturity model

Each level adds signals without dropping the ones below. Make sure your current level is solid before adding depth.

flowchart TD
  L1["Level 1: Survival
process alive, no swap"]
  L2["Level 2: Operational
effective, within capacity"]
  L3["Level 3: Mature
per-slab, leading indicators"]
  L4["Level 4: Expert
deep signals, post-incident"]
  L1 --> L2 --> L3 --> L4

Level 1: survival

The absolute minimum. These signals answer: is the process up, and is it processing commands. A team at Level 1 catches crashes, OOM kills, restarts, and swap incidents. They will not catch slab imbalance, eviction cascades, or silent degradation until hit ratio collapses and the backend complains.

Signal	Source	What to watch for
Process responds to a command	`echo "version" \| nc -w 2 localhost 11211`	No valid response for 3 consecutive checks over 30-60 seconds. A TCP port check is not sufficient: the process can accept connections while frozen.
Uptime stable	`stats` field `uptime`	Uptime dropping below the previous reading. A restart means total data loss and a cold cache.
VmSwap is zero	`/proc/<pid>/status` field `VmSwap`	Any nonzero value. Memcached on swap is disk-speed access for the swapped pages. This is a production incident, not a tuning issue.

Why a command probe, not a port check. The kernel TCP stack will accept connections on behalf of a process that is hung in a deadlock, suspended, or being killed by the OOM killer. A port check returns success while the process does nothing. Send an actual command (version, stats) and confirm a response within 2 seconds. Three consecutive failures over 30-60 seconds eliminates transient blips.

Why VmSwap matters at Level 1. An in-memory cache with data on disk is a contradiction. Even a few megabytes of swap causes millisecond-scale latency spikes for the affected items. Run memcached with -k (calls mlockall) to prevent swapping, but still monitor VmSwap: -k can silently fail without CAP_IPC_LOCK.

Level 2: operational

Signals that tell you whether the cache is effective, whether memory is under pressure, whether connections are healthy, and whether someone flushed it.

Signal	Source	What to watch for
Hit ratio	`stats` fields `get_hits`, `get_misses`. Formula: `get_hits / (get_hits + get_misses)` computed from deltas	Drop >15 percentage points from a 1-hour rolling average. Sustained <75% for a read-heavy workload after warmup.
Eviction rate	`stats` field `evictions` (rate from deltas)	Sustained nonzero rate. Especially urgent if `evicted_time` is low (see Level 3).
Memory utilization	`stats` fields `bytes`, `limit_maxbytes`. Ratio: `bytes / limit_maxbytes`	>0.90 and trending upward. Global utilization is misleading for slab-allocated caches (see Level 3).
Connection saturation	`stats` fields `curr_connections`, `max_connections`	`curr_connections / max_connections > 0.80`. Default `max_connections` (1024) is often too low for production.
Connection rejections	`stats` fields `listen_disabled_num`, `rejected_connections`	Any increment from zero. `listen_disabled_num` counts transitions into maxconns state. `rejected_connections` counts connections actively rejected. Both should be zero.
Flush events	`stats` field `cmd_flush`	Any increment in production. `flush_all` invalidates the entire cache. Each increment is a cache-wide data loss event.
Accept state	`stats` field `accepting_conns`	Flips to 0. The server has hit its connection limit and is refusing new connections.

Compute hit ratio from deltas, not cumulative counters. The get_hits and get_misses counters are cumulative since process start. Dividing raw counters averages hit ratio over the entire lifetime, obscuring recent changes. Sample twice with a known interval and compute (get_hits_delta) / (get_hits_delta + get_misses_delta).

Track cmd_flush as a change event. cmd_flush is a lifetime counter. In a stable production deployment it should never increment. Alert on any delta, not an absolute threshold. Each increment means someone or something issued flush_all, whether intentionally (maintenance, cache poisoning remediation) or accidentally. The get_flushed counter (present since 1.4.x) measures read-side impact: how many GET requests encountered a flushed item.

Correlate eviction rate with hit ratio. Evictions alone do not indicate a problem. A cache sized slightly below the working set will evict cold items while maintaining a stable hit ratio. The signal that matters is sustained evictions combined with a declining hit ratio. High misses with high evictions means memory pressure. High misses with zero evictions means application behavior (new keys, cold start, random access patterns).

Level 3: mature

Per-slab visibility and leading indicators. This is where teams catch slab calcification, distinguish healthy eviction from harmful thrash, and detect the LRU maintainer falling behind. The most common monitoring gap in memcached operations is the absence of per-slab data.

Signal	Source	What to watch for
Per-slab utilization	`stats slabs` per class: `used_chunks`, `free_chunks`, `total_pages`	Any class with `free_chunks == 0` and active evictions while other classes have significant free space. This is slab calcification.
Evicted item age	`stats items` per class: `evicted_time`	`evicted_time < 300` (5 minutes) in any class with active evictions. You are evicting recently-accessed data.
Listen disabled duration	`stats` field `time_in_listen_disabled_us`	Any increase over a 5-minute window. Captures duration of connection saturation events, not just count.
Cache waste	`stats` fields `evicted_unfetched`, `expired_unfetched`	`evicted_unfetched / evictions > 0.5`. More than half of evicted items were never read after being set.
Direct reclaims	`stats` field `direct_reclaims`	Any sustained nonzero rate. Worker threads are bypassing the LRU maintainer to reclaim memory. Pressure is acute.
Response buffer OOM	`stats` field `response_obj_oom`	Any sustained nonzero rate. Connections are being closed because response buffers cannot be allocated. Separate from item storage memory.
Store failures	`stats` fields `store_too_large`, `store_no_memory`	Any nonzero rate. `store_too_large`: application generating items exceeding `-I` (default 1 MB). `store_no_memory`: cache running with `-M` (no-eviction mode) is full.
Flush impact	`stats` field `get_flushed`	Nonzero rate after a `cmd_flush` increment. Measures how actively the flushed data is being requested.

The slab trap. Global bytes can be at 50% of limit_maxbytes while one slab class is 100% full and evicting actively used items. The global eviction counter sums all classes, so a single saturated class drives it while others sit idle. Always correlate global evictions with per-slab data from stats items and stats slabs.

Evicted age is the quality discriminator. evicted_time is the age (seconds since last access) of the most recently evicted item in a slab class. High values (hours, days) mean healthy turnover of cold items. Low values (seconds, minutes) mean the cache is thrashing: evicting items that would have been hit. When evicted_time is less than the typical request interval for popular keys, the cache for that size class provides almost no value.

slab_automove mitigates but does not eliminate calcification. Since 1.5.0, slab_automove mode 1 is the default. It moves one page per 10 seconds at most, which is conservative. If automove is on but not keeping up with a workload shift, investigate whether mode 2 (aggressive) is appropriate for temporary relief, or adjust the growth factor -f on the next restart.

Level 4: expert

Deep signals that teams add after specific incidents. These are not daily monitoring signals for most deployments, but they catch the failure modes that Levels 1-3 miss.

Signal	Source	What to watch for
LRU tail reflock	`stats` field `lrutail_reflocked`	High rate. Items at the LRU tail are locked by active reads while eviction pressure exists. Indicates large-value reads competing with memory pressure.
Crawler efficiency	`stats` fields `crawler_reclaimed`, `crawler_items_checked`	Ratio trending down. The LRU crawler is examining more items but reclaiming fewer expired ones.
Slab reassign activity	`stats` fields `slab_reassign_*` (rescues, chunk_rescues, evictions_nomem, inline_reclaim, busy_items, busy_deletes), `slabs_moved`	Whether automove is actively rebalancing and whether it is effective.
Connection yields	`stats` field `conn_yields`	Sustained rate >100/sec. A client is sending large pipelines and being forced to yield by the `-R` fairness limit (default 20 requests per event).
Hash table state	`stats` fields `hash_is_expanding`, `hash_power_level`, `hash_bytes`	`hash_is_expanding` true for extended periods. Memory doubles during expansion because old and new tables coexist.
Client-side latency	External measurement (not from `stats`)	p99 >5ms for same-datacenter connections. Memcached does not expose latency natively. Must be measured at the client or via external probe.

Deployment-specific signals. If you run extstore (1.5.4+, per the official release notes), track extstore_page_evictions and get_oom_extstore for SSD spill health. If you run TLS (1.5.13+, per the official TLS documentation), track ssl_handshake_errors and ssl_proto_errors. If you run SASL auth, track auth_cmds and auth_errors . If you run a sharded cluster, track per-instance load balance separately: consistent hashing can concentrate traffic on one node.

What most teams get wrong

Trusting global memory utilization. bytes at 50% looks healthy. One slab class can be full and evicting. Monitor per-slab utilization and evicted_time, not just global bytes.
Ignoring connection limits. Default max_connections (1024) is too low for most production deployments. Check OS ulimit -n for the memcached process: if it is lower than max_connections, the OS limit wins.
Monitoring miss rate without context. High miss rate is not always a problem. Cold start, new key patterns, and write-heavy workloads produce high miss rates legitimately. Correlate with evictions, uptime, and evicted_time.
Not distinguishing eviction quality. Evicting a 3-day-old cold item is healthy turnover. Evicting a 10-second-old active item is a crisis. Monitor evicted_time per slab class.
Leaving UDP enabled on old installations. UDP was enabled by default before 1.5.6. CVE-2018-1000115 enabled amplification attacks with up to 51,000x factor. Audit startup flags and verify udpport = 0 in stats settings.
Assuming the cache survives restart. Every restart is complete data loss. Have a cache warming strategy and monitor hit ratio recovery.
Not tracking flush_all events. An accidental flush_all is indistinguishable from a restart in impact but harder to detect because the process stays up. Alert on any cmd_flush increment.
Treating all versions the same. Segmented LRU (1.5.0+), slab_automove defaults (1.5.0+), and UDP default-off (1.5.6+) all change which signals are relevant. Know your version and which features are active.

How Netdata helps

Per-second collection of stats counters means hit ratio and eviction rates are computed from tight deltas, not lifetime cumulative averages that obscure recent changes.
Per-slab data from stats items and stats slabs is collected alongside global counters, making slab calcification visible without manual polling or ad hoc scripts.
VmSwap is collected natively as part of per-process monitoring, so swap detection requires no separate check.
System metric correlation (NIC bytes, per-thread CPU, RSS, swap) catches failure modes that stats alone misses: network saturation, per-thread CPU contention, and memory overhead growth beyond the slab allocator.
cmd_flush increments and uptime discontinuities are tracked as events, making cold-cache incidents visible alongside hit ratio recovery curves and backend load correlation.
Anomaly detection on the full signal set flags deviations from baseline for workload-dependent signals like hit ratio and eviction rate where static thresholds do not generalize.