How Memcached actually works in production: a mental model for operators

Memcached is a multi-threaded, in-memory key-value cache daemon built on libevent. By default it has no persistence; it also has no replication or clustering. Clients handle sharding. A restart means total data loss. These are the design, not limitations to work around.

Before you can debug a memcached incident, you need three abstractions: how memory is partitioned (the slab allocator), how items age within each partition (the segmented LRU), and how connections and threads interact under load. Without these, the stats output is noise. With them, the same numbers tell you exactly which resource is saturated.

Daemon architecture

A main listener thread accepts TCP connections and distributes them round-robin across worker threads (configured with -t, default 4). Each worker runs its own libevent event loop. Connections are non-blocking and sticky to their assigned worker for their lifetime. There is a hard connection limit (configured with -c, default 1024); when it is reached, the listener is temporarily disabled and listen_disabled_num increments. With the default maxconns_fast, memcached returns ERROR Too many open connections and closes each surplus socket; with maxconns_fast disabled, connections instead queue in the OS backlog.

The daemon does not persist data, replicate, or cluster. Treating it as a persistent store, a source of truth, or a strongly consistent system is the most common source of production incidents. Every monitoring signal is downstream of these facts:

• No persistence means restarts are data-loss events.
• Clients shard means one hot node indicates uneven key distribution, not a daemon bug.
• Fixed memory ceiling means evictions are the primary pressure signal, not a defect.

How it works

The slab allocator

The daemon enforces a fixed memory ceiling (the -m flag, default 64 MB). The allocator divides that ceiling into pages of 1 MB. Pages are assigned to slab classes. Each slab class stores items within a specific chunk size range: class 1 handles items up to 96 bytes, class 2 up to 120 bytes, class 3 up to 152 bytes, growing by a configurable factor (the -f flag, default 1.25).

Once a page is assigned to a slab class, it was traditionally never returned. This is the root cause of slab calcification: if your workload’s item size distribution shifts, memory remains locked in classes serving the old distribution while new items evict aggressively in their undersized classes.

Since 1.4.11, slab_reassign allows moving pages between classes (default-on since 1.5.0). Since 1.4.11, slab_automove automates this (mode 1 is the default since 1.5.0). These mitigations reduce calcification but do not eliminate it. The automover is conservative: mode 1 evaluates a default 10-second activity window and then moves at most about one page per second, choosing from an old, idle class for an actively evicting class.

The segmented LRU

Each slab class can maintain a segmented LRU with up to four tiers. HOT, WARM, and COLD are the tiers in the default segmented LRU since 1.5.0:

• HOT: newly inserted items land here.
• WARM: items promoted from COLD after a re-access.
• COLD: eviction candidates. Items that aged out of HOT or WARM without further access.
• TEMP: items with very short TTLs (opt-in via -o temporary_ttl=<N>) that bypass the full LRU.

A background LRU maintainer thread manages movement between tiers. A separate LRU crawler thread walks the queues reclaiming expired items. Before 1.5.0, a flat per-class LRU was the default and the maintainer was opt-in (-o lru_maintainer).

The segmented design exists to solve a specific problem: scanning workloads (bulk reads of many keys) should not displace the active working set. HOT is probationary; items are never bumped within it. WARM absorbs items that survive COLD. Evictions happen from the tail of COLD. The moves_to_warm and moves_to_cold stats tell you whether the segmentation is working for your workload.

Connections, threads, and the hash table

The main listener thread round-robins connections across worker threads. Each worker has its own libevent loop. There is no key-based routing to workers: the architecture does not create key-to-thread affinity. A hot key accessed through many client connections spreads across workers; a hot key hammered through a single connection concentrates on one worker.

Connection fairness is enforced by the -R limit (max requests per event, default 20), which yields a connection if it sends too many requests in a burst. The conn_yields stat counts these events.

Keys are stored in an expandable hash table. When the table needs to grow, expansion happens in a dedicated background thread with fine-grained locking (since 1.4.x). It is not a stop-the-world operation. The hash_is_expanding stat indicates when this is in progress. During expansion, the old and new tables coexist, so hash_bytes temporarily includes both allocations.

flowchart TD
  mem["-m memory (default 64MB)"] --> pages["1MB pages"]
  pages -->|"assigned to"| cls["Slab classes by item size"]
  cls --> lru["Segmented LRU per class"]
  lru --> hot["HOT - new inserts"]
  lru --> warm["WARM - promoted from COLD on re-access"]
  lru --> cold["COLD - eviction candidates"]
  lru --> temp["TEMP - opt-in short TTL"]
  cold -->|"class full"| evict["Eviction"]

Where it shows up in production

The design creates specific failure archetypes.

Slab imbalance. One slab class is full and evicting while others have free space. Global memory looks fine, but cache effectiveness collapses for items of the saturated size class. This is the most underdiagnosed memcached problem, and the one that global metrics hide best.

Cache stampede. A popular key expires or is evicted. Thousands of requests miss simultaneously and hammer the backend. Memcached itself is fine; the backend is the victim. Low evictions, high misses, and a correlated backend load spike are the signature.

Eviction storm. The working set exceeds cache size. Constant eviction of useful data. Distinguished from healthy turnover by evicted_time: if recently-accessed items are being evicted (low evicted_time in an actively evicting class), the cache is thrashing rather than turning over cold items.

Connection exhaustion. The -c limit is reached. The listener is disabled (listen_disabled_num increments, accepting_conns flips to 0); under the default maxconns_fast, memcached immediately closes surplus sockets. Clients perceive memcached as down or slow. The process itself is healthy and may be idle on existing connections.

Silent degradation. The process is alive (the TCP port accepts connections) but not processing commands. This can happen during deadlock, OOM killer activity, or swap thrashing. A simple port check passes. A command probe (version or stats) fails.

Deployment variants change the monitoring posture:

• Standalone: single instance, all monitoring is local.
• Client-side sharded cluster: memcached itself does not cluster. Client libraries shard keys via consistent hashing. One hot node indicates uneven key distribution. Node loss reshuffles the entire keyspace.
• SASL authenticated: auth failures appear as connection failures, not command failures. SASL uses the binary protocol; separate ASCII authentication is available through -Y. The binary protocol has no formal deprecation status in recent releases.
• extstore enabled (1.5.4+): external storage for large items. Adds disk I/O as a monitored resource. Strictly opt-in.
• TLS enabled (1.5.13+): adds TLS handshake overhead and failure stats. Compile-time opt-in.

Tradeoffs and when to use it

Memcached is the right choice when you need a fast, simple, ephemeral cache and your application can tolerate cold-start misses. It is the wrong choice when you need persistence, replication, clustering, or strong consistency.

Common misuses:

• Treating it as a persistent store. Any data you cannot afford to lose must live elsewhere. Memcached will lose everything on restart, OOM kill, or flush_all.
• Relying on it for correctness. Atomic counters (incr/decr) wrap silently on overflow and clamp to zero on underflow. Neither produces an error. CAS failures (cas_badval) are silent. If correctness depends on these semantics, you need a different system.
• Assuming it survives process restarts. Every restart is a cold-cache event. Have a warming strategy and monitor hit ratio recovery.
• Ignoring client library behavior. Different client libraries handle failures, timeouts, and retries differently. Some silently fall back to a different server, some throw errors, some queue retries. The client library is half the system.

The no-clustering choice is a feature, not a gap. Clustering logic lives in client libraries (consistent hashing rings), which means cluster behavior depends on which client you use and how it is configured. When a node is added or removed, a fraction of keys remaps, producing a temporary burst of misses proportional to the fraction of keys moved.

Signals to watch in production

How Netdata helps

• Per-second visibility into hit ratio, eviction rate, and memory utilization reveals the slab imbalance pattern (one class evicting while others sit idle) before it cascades into backend overload.
• Correlating memcached signals with backend database metrics on the same timeline turns the eviction-cascade narrative into a single visible story rather than two disconnected alerts firing minutes apart.
• Anomaly detection on cmd_get, cmd_set, and bytes_written rates catches workload shifts such as item size inflation or traffic spikes that precede eviction storms.
• Tracking uptime alongside hit ratio recovery curves distinguishes expected post-restart cold-cache behavior from genuine degradation, preventing false pages during planned maintenance.
• Connection signals (curr_connections, accepting_conns, listen_disabled_num) correlated with client-side metrics isolate connection-pool leaks from real cache pressure.

Related guides

• Memcached monitoring checklist: the signals every production cache needs
• Memcached monitoring maturity model: from survival to expert
• Memcached connection refused: telling a dead process from a hung or full one
• Memcached alive but not responding: the silent process hang
• Memcached unexpected restart: uptime reset, wiped cache, and the cold-start backend spike
• Memcached hit ratio dropping: reading get_hits, get_misses, and cache effectiveness
• Memcached high miss rate: separating cold start, new key patterns, and memory pressure
• Memcached evicted_unfetched and expired_unfetched: caching data nobody ever reads
• Memcached incr/decr misses: evicted counters that silently break rate limiters and locks
• Memcached cas_badval climbing: check-and-set contention and lost updates
• Memcached evictions climbing: the cache is full and discarding live data
• Memcached evicted_time low: distinguishing healthy turnover from cache thrash