Memcached is a multi-threaded, in-memory key-value cache daemon built on libevent. No persistence, no replication, no clustering. Clients handle sharding. A restart means total data loss. These are the design, not limitations to work around.

Before you can debug a memcached incident, you need three abstractions: how memory is partitioned (the slab allocator), how items age within each partition (the segmented LRU), and how connections and threads interact under load. Without these, the stats output is noise. With them, the same numbers tell you exactly which resource is saturated.

Daemon architecture

A main listener thread accepts TCP connections and distributes them round-robin across worker threads (configured with -t, default 4). Each worker runs its own libevent event loop. Connections are non-blocking and sticky to their assigned worker for their lifetime. There is a hard connection limit (configured with -c, default 1024); when it is reached, the listen socket is temporarily disabled and listen_disabled_num increments. New connections then queue in the OS backlog and are eventually refused by the kernel, not by memcached itself.

The daemon does not persist data, replicate, or cluster. Treating it as a persistent store, a source of truth, or a strongly consistent system is the most common source of production incidents. Every monitoring signal is downstream of these facts:

  • No persistence means restarts are data-loss events.
  • Clients shard means one hot node indicates uneven key distribution, not a daemon bug.
  • Fixed memory ceiling means evictions are the primary pressure signal, not a defect.

How it works

The slab allocator

At startup, memcached pre-allocates a fixed block of memory (the -m flag, default 64 MB). This memory is divided into pages of 1 MB each. Pages are assigned to slab classes. Each slab class stores items within a specific chunk size range: class 1 handles items up to 96 bytes, class 2 up to 120 bytes, class 3 up to 152 bytes, growing by a configurable factor (the -f flag, default 1.25).

Once a page is assigned to a slab class, it was traditionally never returned. This is the root cause of slab calcification: if your workload’s item size distribution shifts, memory remains locked in classes serving the old distribution while new items evict aggressively in their undersized classes.

Since 1.4.11, slab_reassign allows moving pages between classes (default-on since 1.5.0). Since 1.4.11, slab_automove automates this (mode 1 is the default since 1.5.0). These mitigations reduce calcification but do not eliminate it. The automover is conservative: mode 1 moves at most one page every 10 seconds, and only from classes with no recent evictions.

The segmented LRU

Each slab class maintains a segmented LRU with four tiers (default since 1.5.0):

  • HOT: newly inserted items land here.
  • WARM: items promoted from COLD after a re-access.
  • COLD: eviction candidates. Items that aged out of HOT or WARM without further access.
  • TEMP: items with very short TTLs (opt-in via -o temporary_ttl=<N>) that bypass the full LRU.

A background LRU maintainer thread manages movement between tiers. A separate LRU crawler thread walks the queues reclaiming expired items. Before 1.5.0, a flat per-class LRU was the default and the maintainer was opt-in (-o lru_maintainer).

The segmented design exists to solve a specific problem: scanning workloads (bulk reads of many keys) should not displace the active working set. HOT is probationary; items are never bumped within it. WARM absorbs items that survive COLD. Evictions happen from the tail of COLD. The moves_to_warm and moves_to_cold stats tell you whether the segmentation is working for your workload.

Connections, threads, and the hash table

The main listener thread round-robins connections across worker threads. Each worker has its own libevent loop. There is no key-based routing to workers: the architecture does not create key-to-thread affinity. A hot key accessed through many client connections spreads across workers; a hot key hammered through a single connection concentrates on one worker.

Connection fairness is enforced by the -R limit (max requests per event, default 20), which yields a connection if it sends too many requests in a burst. The conn_yields stat counts these events.

Keys are stored in an expandable hash table. When the table needs to grow, expansion happens in a dedicated background thread with fine-grained locking (since 1.4.x). It is not a stop-the-world operation. The hash_is_expanding stat indicates when this is in progress. During expansion, hash table memory is temporarily doubled.

flowchart TD
  mem["-m memory (default 64MB)"] --> pages["1MB pages"]
  pages -->|"assigned to"| cls["Slab classes by item size"]
  cls --> lru["Segmented LRU per class"]
  lru --> hot["HOT - new inserts"]
  lru --> warm["WARM - promoted from COLD on re-access"]
  lru --> cold["COLD - eviction candidates"]
  lru --> temp["TEMP - opt-in short TTL"]
  cold -->|"class full"| evict["Eviction"]

Where it shows up in production

The design creates specific failure archetypes.

Slab imbalance. One slab class is full and evicting while others have free space. Global memory looks fine, but cache effectiveness collapses for items of the saturated size class. This is the most underdiagnosed memcached problem, and the one that global metrics hide best.

Cache stampede. A popular key expires or is evicted. Thousands of requests miss simultaneously and hammer the backend. Memcached itself is fine; the backend is the victim. Low evictions, high misses, and a correlated backend load spike are the signature.

Eviction storm. The working set exceeds cache size. Constant eviction of useful data. Distinguished from healthy turnover by evicted_time: if recently-accessed items are being evicted (low evicted_time in an actively evicting class), the cache is thrashing rather than turning over cold items.

Connection exhaustion. The -c limit is reached. The listen socket is disabled (listen_disabled_num increments, accepting_conns flips to 0). New connections stall in the OS backlog. Clients perceive memcached as down or slow. The process itself is healthy and may be idle on existing connections.

Silent degradation. The process is alive (the TCP port accepts connections) but not processing commands. This can happen during deadlock, OOM killer activity, or swap thrashing. A simple port check passes. A command probe (version or stats) fails.

Deployment variants change the monitoring posture:

  • Standalone: single instance, all monitoring is local.
  • Client-side sharded cluster: memcached itself does not cluster. Client libraries shard keys via consistent hashing. One hot node indicates uneven key distribution. Node loss reshuffles the entire keyspace.
  • SASL authenticated: auth failures appear as connection failures, not command failures. SASL requires the binary protocol.
  • extstore enabled (1.5.4+): external storage for large items. Adds disk I/O as a monitored resource. Strictly opt-in.
  • TLS enabled (1.5.13+): adds TLS handshake overhead and failure stats. Compile-time opt-in.

Tradeoffs and when to use it

Memcached is the right choice when you need a fast, simple, ephemeral cache and your application can tolerate cold-start misses. It is the wrong choice when you need persistence, replication, clustering, or strong consistency.

Common misuses:

  • Treating it as a persistent store. Any data you cannot afford to lose must live elsewhere. Memcached will lose everything on restart, OOM kill, or flush_all.
  • Relying on it for correctness. Atomic counters (incr/decr) wrap silently on overflow and clamp to zero on underflow. Neither produces an error. CAS failures (cas_badval) are silent. If correctness depends on these semantics, you need a different system.
  • Assuming it survives process restarts. Every restart is a cold-cache event. Have a warming strategy and monitor hit ratio recovery.
  • Ignoring client library behavior. Different client libraries handle failures, timeouts, and retries differently. Some silently fall back to a different server, some throw errors, some queue retries. The client library is half the system.

The no-clustering choice is a feature, not a gap. Clustering logic lives in client libraries (consistent hashing rings), which means cluster behavior depends on which client you use and how it is configured. When a node is added or removed, a fraction of keys remaps, producing a temporary burst of misses proportional to the fraction of keys moved.

Signals to watch in production

SignalWhy it mattersWarning sign
Hit ratio (get_hits / (get_hits + get_misses))Primary effectiveness metric. Decline means more traffic hits the backend.Sustained drop of more than 15 percentage points from rolling average.
Eviction rate (evictions)Memory is full in at least one slab class and the LRU is discarding items.Sustained non-zero rate per slab class, not just globally.
Evicted item age (evicted_time per slab class)Discriminates healthy turnover from harmful thrash. Low value means recently-accessed items are being evicted.evicted_time below 300 seconds in any class with active evictions.
Slab class utilization (stats slabs, stats items)Reveals imbalance hiding behind healthy global memory metrics.One class at 100% used_chunks with evictions while others have significant free_chunks.
Connection saturation (curr_connections, accepting_conns, listen_disabled_num)Approaching the hard limit means imminent rejection.curr_connections above 80% of -c, or accepting_conns = 0, or listen_disabled_num increasing.
UptimeReset means restart means total data loss.Unexpected discontinuity in the uptime counter.
cmd_flushEach increment invalidates the entire cache.Any increment in production.
direct_reclaimsThe LRU maintainer is falling behind. Worker threads are evicting inline.Any sustained non-zero rate.

How Netdata helps

  • Per-second visibility into hit ratio, eviction rate, and memory utilization reveals the slab imbalance pattern (one class evicting while others sit idle) before it cascades into backend overload.
  • Correlating memcached signals with backend database metrics on the same timeline turns the eviction-cascade narrative into a single visible story rather than two disconnected alerts firing minutes apart.
  • Anomaly detection on cmd_get, cmd_set, and bytes_written rates catches workload shifts such as item size inflation or traffic spikes that precede eviction storms.
  • Tracking uptime alongside hit ratio recovery curves distinguishes expected post-restart cold-cache behavior from genuine degradation, preventing false pages during planned maintenance.
  • Connection signals (curr_connections, accepting_conns, listen_disabled_num) correlated with client-side metrics isolate connection-pool leaks from real cache pressure.