$ guides / memcached ▌

MEMCACHED · OPERATIONS PLAYBOOK

Memcached's three silent traps: memory partitioned by item size, a cache that forgets on restart, and an open port anyone can flush

An in-memory cache with no persistence, no replication, and a fixed memory ceiling — where memory is pre-split into fixed-size slabs, connections are pinned to worker threads for life, and any client that can reach the port can wipe everything. We trace how that design behaves under load, where a healthy-looking global metric hides a saturated slab class, and what to do when the cache stops helping.

> Start with the monitoring checklist → # Jump to the full guide list

Memcached is trivial to start and unforgiving in production: the defaults hand you a cache that partitions memory by item size, forgets everything on restart, and answers to anyone who can reach the port.

The basics work. Until one slab class fills while bytes/limit_maxbytes still reads 60% and the global number swears there is room — because memory is partitioned by item size, not shared. Until a deploy changes an object's serialized size and pages stay locked to the old size class while the new one starves and evicts. Until the process restarts and every item is gone at once — there is no persistence — the hit ratio falls to zero, and every miss stampedes the backend. Until curr_connections reaches -c, accepting_conns flips to 0, and clients are refused while CPU and memory look fine. Until someone, or something, sends a single unauthenticated flush_all and the whole cache evaporates.

These guides are written for engineers who already run Memcached, not for people learning what a cache is. The goal is the mental model of how the daemon actually behaves under load — the slab allocator, the per-class segmented LRU, the per-thread worker model — plus the failure patterns that keep recurring, the monitoring story that catches them before they page anyone, and the runbooks you wish someone had handed you before your last incident.

How Memcached actually runs in production

Memcached is not a simple key-value box. It is a multi-threaded libevent daemon where a main listener pins each connection to a worker thread for life, memory is pre-split into fixed-size slabs partitioned by item size, and each slab class runs its own segmented LRU. There is no persistence, no replication, and no server-side clustering. Most production failures live between these layers, not inside any one of them.

clients + sharding

There is no server-side cluster. Clients consistent-hash keys across instances and hold their own connection pools, so a single hot key always lands on the same node and one node can run hot while the fleet looks balanced. A missing pool turns into connection churn or a leak.

CLIENT

listener + worker threads

A single libevent listener accepts connections and round-robins them across <code>-t</code> worker threads. A connection is pinned to its worker for its whole life, so one saturated worker gives a subset of clients high latency while aggregate CPU still looks idle.

THREADS

connection buffers

Each connection costs a file descriptor and roughly 10KB of buffer memory that lives OUTSIDE the <code>-m</code> cache budget. <code>-c</code> caps the count; at the ceiling <code>accepting_conns</code> flips to 0 and clients are refused. When a response buffer cannot be allocated, <code>response_obj_oom</code> drops the connection mid-request.

CONN

commands + protocol

get, set, touch, incr, decr, and cas over the ASCII, meta, or (deprecated) binary protocol. A multiget counts as one <code>cmd_get</code> but many key lookups, so <code>get_hits+get_misses</code> is the real work. <code>flush_all</code> is a single, unauthenticated command that invalidates everything.

COMMAND

hash table

A global hash table maps keys to items. As item count grows it doubles on a background thread (<code>hash_is_expanding=1</code>), briefly holding old and new tables at once, and it only ever grows — the table never shrinks back down.

INDEX

slab allocator

The <code>-m</code> budget is carved into 1MB pages, and each page is permanently assigned to a slab class serving one item-size range. Memory is partitioned by size, which is why the global <code>bytes/limit_maxbytes</code> number can look healthy while one class is 100% full and evicting.

SLAB

segmented LRU per class

Each slab class runs its own HOT/WARM/COLD/TEMP chains. The LRU maintainer and crawler threads age items, reclaim expired ones before they must be evicted, and evict from COLD. Evictions are per-class, never global — <code>evicted_time</code> per class says whether they are harmful.

LRU

host + kernel

No persistence means a restart is a total flush; no replication means the node is a single point of loss. Process RSS runs roughly 1.4x the <code>-m</code> limit, any <code>VmSwap</code> negates the cache, and the OS OOM killer and <code>ulimit -n</code> are external ceilings the daemon cannot see.

HOST

Why this matters: 'Memcached is slow' or 'the hit ratio dropped' can come from a single saturated worker thread, a per-class slab imbalance with memory to spare, a hot key concentrated on one node, connection-buffer exhaustion, the process quietly swapping, or an accidental flush. The symptom rhymes but each layer has a different signal — and a different fix.

The failures you'll actually see

Most Memcached incidents fall into a small set of recurring patterns. Recognise the shape, and triage gets dramatically faster.

CRITICAL

Evicting with memory to spare

Evictions climb and recently-used items disappear, yet global bytes/limit_maxbytes sits at 50-70%. Because memory is partitioned by item size, one slab class is 100% full and evicting while others hold idle free chunks. The operator's instinct — add memory — does not help, because the new pages go to the idle classes, not the saturated one.

Evictions rising while global bytes/limit_maxbytes stays well under 1.0
One slab class with free_chunks == 0 and evictions in stats slabs
Low age / evicted_time in the saturated class from stats items
Hit ratio falling even though total memory looks available

Investigate →

CRITICAL

The eviction cascade

The cache genuinely fills, evictions accelerate, and every eviction becomes a future miss that falls through to the database. The backend saturates, latency spikes stack-wide, and slow responses drive retries that add still more load — a positive feedback loop. Memcached itself looks fine: low latency, normal CPU. The victim is the backend, and the hit ratio slides gradually rather than dropping to zero at once.

Evictions per second climbing toward the set rate
Hit ratio declining gradually over minutes (not an instant cliff)
Backend query rate and latency rising in lockstep with misses
Memcached CPU and latency still normal while downstream burns

Investigate →

CRITICAL

Refused, hung, or full

Clients see connection errors. A TCP check alone cannot tell the three cases apart: the process is dead or OOM-killed (connection refused), or it is alive but the kernel accepts the socket while a stats/version probe times out and command rates fall to zero (a silent hang), or it is at the connection ceiling with accepting_conns=0. Only a real command probe distinguishes them, and each has a different fix.

Port refuses connections; dmesg shows the OOM killer
TCP accepts but a stats/version probe times out repeatedly
cmd_get and cmd_set falling to near zero with stable prior uptime
accepting_conns == 0 while memory and CPU look fine

Investigate →

ACTIVE

The connection ceiling

curr_connections reaches the -c limit, accepting_conns flips to 0, the listen socket is disabled, and rejected_connections and listen_disabled_num climb. New clients get connection-refused and fall through to the backend, purely because of a connection-slot shortage — memory and CPU can be entirely healthy. The trigger is a leak, a missing pool, or a -c and ulimit -n left at defaults.

accepting_conns == 0 (the definitive real-time bit)
curr_connections at or near the -c limit
rejected_connections / listen_disabled_num climbing
New clients refused while the cache itself is idle

Investigate →

ACTIVE

The accidental flush

cmd_flush increments and the entire cache is invalidated. flush_all does not lock the server or free memory immediately — it sets a timestamp and items are lazily invalidated on access, and a delay argument makes the effect future-dated and hard to correlate. Anyone with network access can issue it (no auth by default). The result is a cold cache and a thundering herd on the backend.

cmd_flush incrementing at all (almost never intentional)
Hit ratio dropping toward zero across the whole instance
get_flushed rising as reads hit invalidated items
Backend load spiking with no matching eviction growth

Investigate →

IMMINENT

The open reflector

A latent misconfiguration that does not self-resolve. If udpport is non-zero and the instance is reachable from untrusted networks, it can be abused as a DDoS reflector (CVE-2018-1000115, amplification up to ~51,000x) — the mechanism behind the record 1.3-1.7Tbps attacks of 2018. UDP has been off by default since 1.5.6, but older installs and a bind to 0.0.0.0 leave the door open to anyone who can reach the port.

udpport non-zero in stats settings
The instance bound to 0.0.0.0 on a routable interface
Unexpected outbound UDP bursts from port 11211
No firewall allowlist in front of the memcached port

Investigate →

Choosing a tool

Best Memcached Monitoring Tools in 2026: 10 Ranked

A ranked review of the tools teams actually shortlist here, what each one is genuinely good at, and how the pricing behaves as you scale.

Read the buyer's guide →

Memcached monitoring maturity levels

Memcached observability works in four practical levels. Each is a complete operation, not a stepping stone. Pick the level that matches how much your cache matters. Most production caches should land at the second level.

Level 1: Survival

Know that something is wrong

Survival monitoring is the floor. With these signals you can answer one question: is the cache alive and still helping? You will not learn what broke, but you will learn that something broke before users do. Survival is enough for dev caches and non-critical workloads.

Process responds to a command Send stats or version — a TCP accept is not proof of life.
Uptime stable (no unexpected reset) A drop to near-zero means a restart wiped every item.
Hit ratio from deltas get_hits / (get_hits + get_misses) over an interval, not lifetime.
Evictions rate Valid items being discarded — the cache is under pressure.
Memory utilisation bytes / limit_maxbytes as a fill gauge (with slab caveats).
Connections vs the -c limit curr_connections approaching -c means refusals are near.
Rejected connections rejected_connections / listen_disabled_num above zero.

Level 2: Operational

Diagnose most incidents on your own

Operational monitoring is what most production caches should target. Survival tells you something is wrong; operational tells you what. With this coverage your team can usually diagnose an incident on its own: eviction pressure, connection ceilings, an accidental flush, or the process quietly swapping.

Command rates by type cmd_get / cmd_set / cmd_touch deltas reveal the workload shape.
Network bytes in and out bytes_written usually dwarfs bytes_read; watch for NIC saturation.
cmd_flush change detection Any increment is a near-total cache wipe — alert on it.
reclaimed and evicted_unfetched Reclaimed is healthy; evicted_unfetched flags wasted cache.
conn_yields A client's oversized pipeline hitting the -R fairness limit.
Process RSS Roughly 1.4x -m; higher hints at buffers or fragmentation.
VmSwap must be zero Any swap on an in-memory cache is a latency incident.
Client-side latency Memcached exposes no histogram — measure p99 from the client.

Level 3: Mature

Catch problems before they become incidents

Mature monitoring catches problems before they wake anyone up. A single slab class calcifying, evicted items still warm, direct reclaims scrambling to keep up, counter keys evicted out from under a rate limiter. None of these page you on day one. They become page-out incidents on day thirty.

Per-slab evictions and free_chunks Find the one saturated class the global number hides.
evicted_time per class Low age means the cache is evicting recently-active data.
Slab calcification detection Pages locked to old sizes after a workload shift.
direct_reclaims Worker threads reclaiming inline — the maintainer fell behind.
Hash table state hash_is_expanding and hash_power_level during item growth.
LRU crawler progress crawler_reclaimed at zero while evicting means it is stuck.
incr/decr and cas_badval misses Evicted counters and CAS conflicts break app guarantees silently.
slab_automove activity Whether page rebalancing is keeping up or cannot help.

Level 4: Expert

Reactive instrumentation after real incidents

Expert signals enter your stack the day after a specific incident proved you needed them. Per-thread CPU skew, segmented-LRU move counters, per-slab fragmentation, cross-instance balance, TCP state accumulation. Most teams never need every signal here. Add the ones your incident history says you do.

Per-thread CPU One worker pinned at 100% hides inside a calm aggregate.
LRU segment move counters moves_to_cold / moves_to_warm show whether tiers fit the workload.
Per-slab fragmentation Chunk size vs item size waste within a class.
Cross-instance key balance Consistent-hash skew leaving one node hot.
TCP state distribution TIME_WAIT buildup and churn via ss on port 11211.
response_obj_oom Connections dropped for lack of buffer memory at scale.
extstore / TLS stats Flash-tier and encrypted-transport counters where enabled.
auth_errors trend SASL failures — only meaningful when auth is actually enabled.

Operating mistakes worth avoiding

The traps Memcached teams keep falling into. Each has a clear, well-known fix. Most teams only learn it after an incident.

Treating a TCP accept as proof of life

A port check passes even when the process is deadlocked, suspended, or swap-thrashing — the kernel still accepts the socket while no command ever completes. Health checks must send a real <code>stats</code> or <code>version</code> command and require a response, over 3 consecutive failures in 30-60s, before declaring the cache down.

Adding memory the moment evictions climb

The reflex fix is almost always wrong. Evictions with a stable, high hit ratio are the LRU doing its job on cold items; and when they are harmful, the cause is often a saturated slab class, not a global shortage. Adding memory sends new pages to idle classes and does nothing. Check <code>stats slabs</code> for a single full class first.

Trusting the global memory percentage

<code>bytes/limit_maxbytes</code> is the number everyone graphs, and for a slab-allocated cache it lies: it can read 60% while one size class is 100% full and evicting recently-used data. It also includes per-item overhead and allocates in 1MB page steps, so it never reaches the limit smoothly. Read it per slab class, not just globally.

Reading aggregate CPU instead of per-thread

With <code>-t 4</code> workers, one thread pinned at 100% while three idle shows as ~40% aggregate — which looks fine while a quarter of connections see high latency, because each connection is pinned to its worker for life. Watch per-thread CPU, and treat rising <code>conn_yields</code> as confirmation that a worker cannot keep up.

Tolerating any swap on an in-memory cache

Even 50MB of <code>VmSwap</code> looks small but means a real fraction of items now live on disk, and every access to a swapped page is disk-speed — negating the entire point of the cache. The latency spikes are random and hard to trace unless swap is the first thing you check. Alert on any non-zero <code>VmSwap</code>; in containers check the cgroup limit, not just the host.

Not treating a cmd_flush change as an incident

<code>flush_all</code> is almost never intentional in production, invalidates every item at once, and — because anyone with network access can issue it unauthenticated — is easy to trigger by accident or malice. A delay argument makes it future-dated and harder to correlate. Alert on any change to <code>cmd_flush</code>, not on a threshold.

Leaving -c at 1024 and forgetting ulimit -n

The default <code>-c 1024</code> is quickly outgrown once an app scales to dozens of instances each with its own pool, and the OS <code>ulimit -n</code> is a second ceiling that wins if it is lower. Each connection also costs ~10KB of buffer memory outside the <code>-m</code> budget, so raising <code>-c</code> without headroom can push the process toward OS OOM. Size both together.

Assuming an internal-only cache needs no network hardening

Memcached was built for trusted networks and has minimal access control: by default anyone who can reach the port reads every key, writes any value, and can <code>flush_all</code>. Binding to 0.0.0.0 and leaving <code>udpport</code> non-zero turns it into a data-leak and DDoS-reflector risk. Bind to internal interfaces, keep UDP off, and firewall the port.

Memcached runbooks in this section

Each guide is a focused runbook for one symptom or topic. Pick one when you have an incident, or use the categories to learn the area.

▸

Start here

▸

Availability, restarts, and hangs

▸

Hit ratio and cache effectiveness

▸

Evictions and memory pressure

▸

Slab allocator, calcification, and memory limits

▸

Connections, limits, and rejections

▸

Throughput, network, latency, and flush

▸

LRU, hash table, and internal state

▸

Host resources: CPU, memory, swap

▸

Security and network exposure

WHERE TO GO NEXT

Setting up Memcached monitoring, or putting out a fire?

If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, jump straight to the symptom that matches what you're seeing.

> Start with the checklist > Back to Operations Guides