Memcached Operations Guides on Netdata

How Memcached actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

Memcached is a multi-threaded, in-memory key-value cache daemon built on libevent. No persistence, no replication, no clustering. Clients handle sharding. A restart means total data loss. These are the design, not limitations to work around.

Before you can debug a memcached incident, you need three abstractions: how memory is partitioned (the slab allocator), how items age within each partition (the segmented LRU), and how connections and threads interact under load. Without these, the stats output is noise. With them, the same numbers tell you exactly which resource is saturated.

Memcached alive but not responding: the silent process hang

Mon, 01 Jan 0001 00:00:00 +0000

The port check passes. The process shows up in ps. The kernel still accepts TCP on 11211. But every stats and version probe times out, and dashboards show cmd_get and cmd_set falling off a cliff. This is the silent process hang: memcached is effectively down while appearing alive to every shallow health check.

Most liveness checks stop at “is something listening on the port?” That question is answered by the kernel’s TCP stack, not by memcached’s worker threads. A frozen main thread, a stuck slab rebalancer, a process suspended by SIGSTOP, or an OOM kill in progress can all leave the listen backlog accepting connections while no command ever executes. Clients connect, send a command, and wait forever.

Memcached auth_errors: SASL failures, brute force, and the no-auth default

Mon, 01 Jan 0001 00:00:00 +0000

Two counters, present since memcached 1.4.4, are the only native signal the daemon gives you about authentication activity: auth_cmds and auth_errors. They count attempts and failures. They do not tell you who attempted, from where, or whether authentication is even enforced.

The critical interpretation rule: if authentication is not enabled (no -S for SASL, no -Y for ASCII token auth

), both counters read zero forever. Zero does not mean “no unauthorized access.” It means “nothing is enforced.” A cache that reports zero auth_cmds can still be wide open to every host that can reach port 11211.

Memcached bound to 0.0.0.0: unauthenticated access on an open port

Mon, 01 Jan 0001 00:00:00 +0000

Memcached was designed for trusted internal networks. It has no authentication by default, no per-key access control, and no per-connection logging. Anyone who can open a TCP connection to the daemon can read every cached value, overwrite any key, enumerate key names and sizes, and issue flush_all to wipe the entire cache in one command. Session tokens, PII, and application secrets cached in plaintext are all readable by that caller.

Memcached cache stampede: a hot key expires and the backend takes the hit

Mon, 01 Jan 0001 00:00:00 +0000

A cache stampede happens when a popular cached key expires or is evicted and many concurrent requests miss at the same instant. Each miss falls through to the backend independently. The backend, usually sized to handle only the small fraction of traffic that misses the cache, receives the full load for that key at once. Memcached itself is not malfunctioning. It is correctly reporting misses and returning them as fast as it can. The victim is the backend, and the fix lives in the application layer.

Memcached cas_badval climbing: check-and-set contention and lost updates

Mon, 01 Jan 0001 00:00:00 +0000

cas_badval tracks check-and-set operations that failed because the CAS unique token changed between your gets read and your cas write. Each increment is a rejected write: another writer modified the key first, and memcached correctly refused the stale update to prevent a lost update.

A sustained cas_badval rate above 10% of total CAS attempts, computed as cas_badval / (cas_hits + cas_misses + cas_badval), indicates meaningful write contention. The application is spending CPU on writes that never land, and unbounded retry logic can amplify the problem: every failed CAS triggers an immediate re-read and re-write, increasing load on both the client and memcached without making progress.

Memcached command rate anomalies: cmd_get and cmd_set spikes and sudden drops

Mon, 01 Jan 0001 00:00:00 +0000

cmd_get, cmd_set, and cmd_touch are cumulative counters in memcached. They increase monotonically from process start and are meaningless as raw numbers. The signal is the derived rate: the delta between two samples divided by the interval. A sudden change in that rate is almost never the root cause. It is a proxy for something that changed upstream (clients stopped or started sending traffic), inside the cache (a hot key expired, the cache was flushed), or in the application logic (a write loop, a deploy with new key patterns).

Memcached conn_yields rising: one client's pipeline starving the others

Mon, 01 Jan 0001 00:00:00 +0000

conn_yields is climbing on a memcached instance. The name suggests thread contention or a locking bug. It is neither.

conn_yields is memcached’s per-connection fairness throttle. Each worker thread processes requests from its assigned connections in a libevent loop. The -R flag (default 20, since memcached 1.4.0) caps how many sequential requests a worker pulls from a single connection in one event loop pass. When a connection hits that cap, the worker yields it: moves it to the back of the processing queue, serves other connections, then comes back. The conn_yields stat increments on each yield.

Memcached connection churn: total_connections racing and TIME_WAIT buildup

Mon, 01 Jan 0001 00:00:00 +0000

total_connections is a cumulative counter that only moves up. When it climbs fast while curr_connections stays flat, clients are opening a fresh TCP connection per request and closing it immediately instead of pooling. Each cycle costs a syscall pair on both ends, a socket on both ends, and on Linux the closed socket sits in TIME_WAIT on the client host for roughly 60 seconds.

The daemon stays responsive, hit ratio is unaffected, and memory looks fine. The damage shows up as inflated p99 latency, wasted accept-thread CPU, and a slow march toward either the -c ceiling or ephemeral port exhaustion on the client side. When either cliff is reached, a previously quiet cache starts refusing connections and clients cascade to the backend.

Memcached connection limit reached: accepting_conns=0 and clients being refused

Mon, 01 Jan 0001 00:00:00 +0000

Memcached enforces a hard ceiling on concurrent client connections via the -c flag (default 1024). When curr_connections reaches that limit, the daemon flips accepting_conns to 0, disables the listen socket, and turns away every new TCP connection. Clients see connection-refused errors or timeouts, fall through to the backend, and the cache stops working for anyone not already connected.

This is purely a connection-slot problem. Memory can be at 40% of limit_maxbytes, CPU idle, evictions zero, and the process fully serving existing connections. The only thing wrong is that the connection table is full.

Memcached connection refused: telling a dead process from a hung or full one

Mon, 01 Jan 0001 00:00:00 +0000

“Connection refused” from a memcached client is commonly misdiagnosed. Operators run a TCP port check, see that the port is open, and act on that single data point. A port test tells you whether the kernel accepts a TCP SYN on 11211, not whether memcached is processing commands.

Three distinct failure modes all surface to clients as “cannot reach memcached”:

Process dead: crashed, OOM-killed, or never owned the port. The kernel refuses the connection.
Process hung: alive but unresponsive. The kernel accepts the connection because the listen socket is open, but commands never return.
Connection limit reached: healthy but saturated. New connections are rejected, queued indefinitely, or accepted but never serviced.

Each requires a different response. Restarting a hung process without understanding why it hung will reproduce the hang. Raising the connection limit on a dead process does nothing.

Memcached CPU saturation: the per-thread ceiling that aggregate graphs hide

Mon, 01 Jan 0001 00:00:00 +0000

Memcached is rarely CPU-bound. At normal request rates, worker threads spend most of their time in epoll waits. When CPU climbs on a memcached host, the reflex is to check aggregate rusage and conclude the daemon is busy. That reflex misses the actual failure mode.

Memcached dispatches each accepted connection round-robin to one of -t worker threads (default 4). Once assigned, a connection lives on that worker for its lifetime. The worker runs its own libevent loop and serves its assigned connections exclusively. If one worker saturates at 100% of a core, every connection pinned to that worker sees elevated latency. The other workers may be at 20%. The aggregate graph reads roughly 40%. Three quarters of your clients are fine.

Memcached curr_connections climbing: connection leaks and missing pooling

Mon, 01 Jan 0001 00:00:00 +0000

curr_connections trends upward with no corresponding traffic increase, edging toward the -c ceiling (default 1024). When it hits the limit, the listen socket disables: accepting_conns flips to 0 and new client TCP connects are refused. To the application it looks like memcached is down. To memcached it is idle and healthy on the connections it already holds.

Most of the time this is not a memcached bug. It is a client-side problem: connections opened and never returned, a broken or absent pool, or a fleet of application instances each holding more sockets than you accounted for. The server is the victim, not the cause.

Memcached evicted_time low: distinguishing healthy turnover from cache thrash

Mon, 01 Jan 0001 00:00:00 +0000

A high global eviction counter is one of the most over-alerted memcached signals. Operators see evictions climbing and page someone at 3 a.m. or reflexively add memory. Both reactions are usually wrong. Raw eviction count says nothing about whether the evicted items mattered. The discriminator that turns “evictions are high” into “evictions are harmful” is evicted_time, reported per slab class in the output of stats items.

evicted_time is the age, in seconds since last access, of the most recently evicted item in a given slab class. Hours or days means the cache is discarding cold data to make room for new entries. Seconds or low minutes means the cache is thrashing: evicting items that were accessed moments ago and would have been hit again. The same eviction rate can be healthy at one value of evicted_time and catastrophic at another.

Memcached evicted_unfetched and expired_unfetched: caching data nobody ever reads

Mon, 01 Jan 0001 00:00:00 +0000

Memcached exposes two counters that most operators never look at, and both describe the same failure: the application is storing data that nobody ever reads. evicted_unfetched counts items evicted from the LRU before they were ever touched by a read. expired_unfetched counts items that lived their full TTL and expired without ever being read. Both are efficiency signals, not reliability signals. They will not tell you the cache is down. They tell you the cache is being used as a write-only buffer, and that write-only data is displacing items that might actually be read.

Memcached evicting with memory to spare: the slab-class imbalance trap

Mon, 01 Jan 0001 00:00:00 +0000

Evictions are climbing on your memcached instance. Global memory is at 55% of limit_maxbytes. The natural response is to add memory, restart the daemon, or hunt for an eviction storm. None of those help here, because the cache is not out of memory in aggregate. One slab class is at 100% capacity and discarding recently-active items, while several other classes sit mostly empty.

This is slab-class imbalance. The global bytes and limit_maxbytes counters aggregate over a partitioned allocator, so a healthy-looking global number can hide a saturated class. The fix is almost never “add memory”: new pages go to whichever class happens to be allocating next, not to the class that is evicting.

Memcached eviction cascade: when a full cache overloads the backend

Mon, 01 Jan 0001 00:00:00 +0000

The most common memcached-related production incident is not a memcached crash. The process stays up, serves hits at sub-millisecond latency, CPU is flat, and memory looks fine in aggregate. What breaks is the backend.

An eviction cascade starts when the working set outgrows the cache. Memory fills, evictions accelerate, and every evicted item becomes a future miss. Those misses fall through to the database, which was sized for cache-assisted load, not raw traffic. The backend saturates, latency climbs stack-wide, and slow responses drive retries that add still more load. It is a positive feedback loop, and memcached is not where it closes.

Memcached evictions climbing: the cache is full and discarding live data

Mon, 01 Jan 0001 00:00:00 +0000

The evictions counter is climbing. At least one slab class is saturated, and the daemon is removing valid, non-expired items to make room for new sets. The severity question is whether the cache is discarding cold data the application no longer needs, or live data that will be requested again within seconds.

Adding memory is the right response only when the cache is globally undersized. If the problem is slab calcification (memory concentrated in the wrong size classes), adding RAM does nothing useful and may mask the real issue. First determine whether evictions are healthy LRU turnover or harmful thrash, and whether the pressure is global or per-slab.

Memcached flush_all: the accidental cache wipe and its cold-start blast radius

Mon, 01 Jan 0001 00:00:00 +0000

You are paged because hit ratio collapsed and the backend database is saturating. The memcached process is up, uptime is stable, memory looks allocated, and there are no evictions. The signature is an instant hit ratio cliff with no gradual decline. The cause is almost certainly a flush_all command, and the cmd_flush counter will confirm it.

A single flush_all invalidates every item in the cache. It does not lock the server, does not free memory immediately, and does not require authentication by default. Anyone with TCP access to port 11211 can issue it. The result is a cold-cache thundering herd: every subsequent GET misses and hits the backend simultaneously. If the backend is sized for cache-assisted load, it just received a multiple of its designed capacity.

Memcached hash table expansion: hash_is_expanding, memory spikes, and item churn

Mon, 01 Jan 0001 00:00:00 +0000

When you first see hash_is_expanding=1 in memcached stats, it looks like a warning. It is not. The daemon is doubling its internal key hash table to accommodate more items, a routine maintenance operation that runs on a dedicated background thread (since 1.4.0) and does not block request processing. It does temporarily spike memory and add CPU load on the maintenance thread, and in specific cases it can expose underlying problems: item-count oscillation, repeated expansion, or an overstretched memory budget.

Memcached high miss rate: separating cold start, new key patterns, and memory pressure

Mon, 01 Jan 0001 00:00:00 +0000

A high miss rate on memcached is frequently misdiagnosed. The instinct is to assume the cache is too small and add memory. That instinct is usually wrong. Random or unique key access produces a near-0% hit rate no matter how healthy the cache is. A freshly restarted cache legitimately runs near 0% until it warms. A deploy that changed key naming produces instant misses on the new keys without any memory pressure.

Memcached hit ratio dropping: reading get_hits, get_misses, and cache effectiveness

Mon, 01 Jan 0001 00:00:00 +0000

The cache hit ratio tells you whether memcached is earning its keep. When it drops, the backend database inherits the miss traffic, and a slow decline can cascade into a failure before anyone pages on “cache.” The ratio is also frequently read wrong: computing it from lifetime counters hides the exact degradation you are trying to catch.

What the hit ratio actually measures

The cache hit ratio is the fraction of GET lookups that found the key in cache:

Memcached hot key: one key, one thread, and lopsided load

Mon, 01 Jan 0001 00:00:00 +0000

Aggregate cmd_get and cmd_set rates look normal. Hit ratio is healthy. Evictions are zero. CPU across the memcached fleet is moderate. Yet a specific subset of clients reports p99 latency an order of magnitude above baseline, and in a consistent-hashing cluster one node runs noticeably hotter than the rest. Memory pressure and connection exhaustion are absent.

The signature of a memcached hot key is concentration without saturation. A tiny number of keys, sometimes a single key, take a disproportionate share of traffic: a feature flag read on every request, a viral item page, a single bot’s session record, a shared rate-limit counter. All funnel traffic to one slab class on one node.

Memcached incr/decr misses: evicted counters that silently break rate limiters and locks

Mon, 01 Jan 0001 00:00:00 +0000

Most memcached operators never look at incr_misses and decr_misses. They sit in the per-operation hit/miss breakdown and stay near zero. When they start climbing, the cache itself usually looks healthy: hit ratio is fine, evictions are modest, memory is not full. The damage is happening somewhere else.

The pattern is specific. The application uses memcached as an atomic counter store. Rate limiters increment a per-client counter and reject when it crosses a threshold. Distributed locks hold a lease token with a TTL and decrement on release. Quota counters track usage per tenant. These counter keys are small, hot, and short-lived. When the slab class they live in comes under memory pressure, the LRU evicts them between operations.

Memcached latency high: diagnosing slow gets when the server exposes no latency metric

Mon, 01 Jan 0001 00:00:00 +0000

Memcached does not expose a latency histogram, p50, or p99. The standard stats output is all counters and gauges at the moment of query: no time-series, no distribution, no per-operation duration. If your application is reporting slow cache reads, the daemon itself cannot confirm or deny the problem. You must measure latency at the client, then use server-side and OS signals to localize the cause.

On a healthy LAN, memcached serves small-item gets in sub-millisecond time. Sustained p99 above a few milliseconds means something is wrong, and it is often not memcached itself. Network saturation, swap, a single saturated worker thread, hash table expansion, and large values are the usual server-side culprits. Client-side GC pauses, DNS resolution, and connection setup add their own latency on top.

Memcached LRU crawler: crawler_reclaimed, disabled crawlers, and lazy expiry

Mon, 01 Jan 0001 00:00:00 +0000

Memcached does not remove items when their TTL expires. An expired item holds its slab slot until something touches it: a client request, an eviction scan, or the LRU crawler. The crawler is a background thread that walks LRU chains, finds expired items, and frees their memory before eviction pressure forces the issue.

A working crawler keeps slab slots available for new writes without evicting live data. Every expired item it reclaims is one fewer item evicted under pressure. Without it, expired items that are never accessed and never reach the eviction tail waste memory indefinitely.

Memcached maxconns and ulimit: why the default 1024 is too low for production

Mon, 01 Jan 0001 00:00:00 +0000

Memcached is up, the port is open, existing connections work fine. But new clients get connection refused. Their requests fall through to the backend, backend load climbs, and the cache looks healthy from the outside: CPU is low, memory is nowhere near the limit, and the slab allocator is not evicting anything.

The most common cause is the default -c 1024 (maxconns) limit. It looks generous for a single application instance, but becomes a hard cliff-edge once you scale from 10 to 50 instances, each with its own connection pool. Monitoring agents, health checks, and load balancer probes also consume connections against this budget.

Memcached memory utilization: bytes vs limit_maxbytes and why the global number lies

Mon, 01 Jan 0001 00:00:00 +0000

The bytes and limit_maxbytes fields from stats look like a clean fill gauge. Divide one by the other and you know how full the cache is. Many dashboards and alerts are built on exactly that ratio.

This article covers why the global ratio lies, what it actually measures, and which per-slab signals confirm real pressure. bytes includes per-item overhead and never quite reaches the ceiling, allocation happens in 1MB page jumps rather than smoothly, and the pool is partitioned into fixed-size slab classes that saturate independently. A cache at 50% global utilization can be evicting live, actively-requested data.

Memcached monitoring checklist: the signals every production cache needs

Mon, 01 Jan 0001 00:00:00 +0000

A production reference for memcached signals, organized by monitoring maturity. Use it to audit what you collect, identify gaps, and prioritize additions.

Levels are cumulative: Level 2 includes Level 1. Level 1 detects crashes and restarts but misses slab calcification and eviction quality. Level 3 catches those before they hit the backend. Level 4 adds signals teams adopt after being burned by subtle failures.

Memcached does not expose latency histograms, per-key statistics, or source IP attribution natively. Several critical signals come from the OS (process state, swap, file descriptors, NIC counters), not from the stats command. Sources are noted where that applies.

Memcached monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Memcached exposes most of its internal state through plain text counters. Most teams stop at hit ratio, global memory, and eviction rate, then discover during an incident that the signal they needed was buried in stats items or /proc/<pid>/status. The four-level model below stages the path from “is the process alive” to “is the segmented LRU moving items between HOT and COLD the way the workload expects”.

Use the levels as a checklist, not a sequence. Each level answers a class of question the previous one could not. Skipping straight to Level 4 without Level 2 fundamentals produces dashboards full of internal counters with no baseline for client impact. The most common gap is between Level 2 and Level 3: operational coverage looks healthy on global metrics while one slab class is silently evicting recently-used items, because the slab allocator partitions memory by item size and the global bytes counter averages across all classes.

Memcached network saturation: large values, multiget, and a maxed-out NIC

Mon, 01 Jan 0001 00:00:00 +0000

Memcached latency is climbing for every client, but the host looks healthy. CPU is well below saturation, memory utilization is far from the limit, evictions are zero, and curr_connections is well below -c. The daemon answers version and stats instantly. Yet every application reports slow gets, and the slowdown is uniform across keys and clients.

When latency degrades uniformly and the usual suspects are clear, look at the wire. Memcached pulls values from RAM and writes them to a socket. If the response stream exceeds what the NIC can push, the kernel queues, TCP congestion control engages, and every client waits. The cache is fine; the link is the bottleneck.

Memcached reclaimed vs evictions: reading memory pressure through reclamation

Mon, 01 Jan 0001 00:00:00 +0000

Most operators learn one signal first: evictions. It goes up, the cache is full, hit ratio drops, someone pages. The counter next to it, reclaimed, tells you how often memcached made room for a new item by reusing an expired item’s slot instead of throwing out live data. Read together, they describe the same process from two angles: how memcached finds space for new writes.

Two more counters complete the picture. crawler_reclaimed is the background LRU crawler doing the same work proactively. direct_reclaims counts cases where a worker thread had to reclaim inline because the background thread fell behind. The four together tell you whether your cache is in a healthy steady state, scrambling to keep up, or actively destroying data the application still wants.

Memcached response_obj_oom: connections killed for lack of buffer memory

Mon, 01 Jan 0001 00:00:00 +0000

If response_obj_oom is climbing in your stats output, the daemon is closing client connections because it cannot allocate internal response buffers. This is not an item-storage problem and not an eviction problem. A cache can be otherwise healthy (zero evictions, free slab memory, low CPU) and still kill clients over this.

The stat has existed since 1.6.0, when memcached reworked its connection buffer management to allocate buffers on demand instead of reserving them per connection. That cut idle connection overhead but introduced a new failure surface: when the pool of memory for response objects runs out, the daemon closes the connection rather than blocking or queueing. The close is immediate when the allocation fails. There is no retry, no backpressure, no graceful degradation. Applications see timeouts or resets against a daemon that still answers version and stats probes.

Memcached RSS above the -m limit: connection buffers, hash table, and fragmentation

Mon, 01 Jan 0001 00:00:00 +0000

Operators set -m to cap cache memory, then notice RSS is 40 percent or more above that number. This is not a leak. The -m flag bounds only the slab allocator, the pool that stores cached items. RSS also includes the hash table, per-connection buffers, worker thread stacks, and allocator overhead.

The bytes / limit_maxbytes gauge in memcached stats describes cache memory: the keys and values your application stores. RSS, reported as VmRSS in /proc/<pid>/status, describes the entire process. They measure different scopes and will almost never match.

Memcached segmented LRU: HOT/WARM/COLD tiers and what the move counters tell you

Mon, 01 Jan 0001 00:00:00 +0000

Since memcached 1.5.0, every slab class runs a segmented LRU instead of a single flat queue. Items land in HOT, WARM, COLD, and optionally TEMP tiers, and a background thread moves them between tiers based on access patterns. The goal is to protect frequently-accessed items from being evicted by one-off scans that touch cold data.

The move counters under stats items describe how well that sorting is working. moves_to_cold counts items aging out of active use, moves_to_warm counts items rescued from COLD by a re-access, and moves_within_lru counts re-ranking within WARM. The ratios between these counters explain why a slab class evicts the way it does, even when global memory looks fine.

Memcached SERVER_ERROR object too large for cache: items over the max item size

Mon, 01 Jan 0001 00:00:00 +0000

SERVER_ERROR object too large for cache is the protocol-level error memcached returns when a store operation targets a value larger than the daemon’s configured maximum item size. The default ceiling is 1 MB (1048576 bytes), set by the -I flag. The store is rejected cleanly: nothing is written, and any pre-existing item under that key is left untouched. From the application’s side this is usually the first sign that something upstream is serializing more data than intended.

Memcached SERVER_ERROR out of memory storing object: stores rejected instead of evicting

Mon, 01 Jan 0001 00:00:00 +0000

When a memcached client receives SERVER_ERROR out of memory storing object, the daemon refused a SET because it could not allocate a chunk for the item. The store_no_memory counter increments. Under default operation, memcached evicts LRU items from the relevant slab class to make room. This error means eviction was either disabled or could not produce a free chunk for that allocation.

Two root causes. First, the instance was started with -M, which disables eviction entirely and returns errors instead of evicting. Second, slab-level exhaustion: a specific slab class has no free chunks and no pages can be reassigned to it. The second case is rarer on 1.5.0+ where slab_automove defaults to on, but still happens when all pages are busy or TEMP_LRU holds unevictable items.

Memcached slab calcification: pages locked to the wrong item sizes after a workload shift

Mon, 01 Jan 0001 00:00:00 +0000

Hit ratio is dropping. Evictions are climbing. But global memory utilization sits at 50 to 70 percent, and adding memory does nothing. This is the signature of slab calcification: whole megabyte pages are locked into slab classes that serve item sizes the workload no longer produces, while the classes for the new sizes are starved and evicting actively used data.

Memcached’s slab allocator divides its memory budget into 1 MB pages, and each page is permanently assigned to a slab class. Each slab class serves items in a narrow size band, with the bands growing by a configurable factor (default 1.25). Once a page was handed to a class, the traditional behavior was that it never came back. When the item-size distribution shifts after a serialization change, a new application version, or an added field, memory stays trapped in classes serving the old sizes while the new sizes evict.

Memcached slab_automove and slab_reassign: rebalancing pages between classes

Mon, 01 Jan 0001 00:00:00 +0000

Memcached’s slab allocator divides its memory budget into 1MB pages and permanently assigns each page to a slab class based on item size. Once assigned, a page traditionally stayed locked to that class forever. If the workload’s item-size distribution shifted after deployment, you got slab calcification: one class full and evicting while others sat idle with free chunks. Global memory utilization looked healthy while cache effectiveness collapsed for the saturated size range.

Memcached swapping: why any VmSwap on an in-memory cache is an incident

Mon, 01 Jan 0001 00:00:00 +0000

A nonzero VmSwap value in /proc/<pid>/status for a memcached process is a production incident, not a tuning concern. Every access to a swapped page costs milliseconds instead of nanoseconds, so a fraction of your cache lookups runs at disk speed. The point of an in-memory cache is lost for those items, and the resulting latency outliers are random and intermittent, almost impossible to trace from the application layer without checking swap first.

Memcached UDP amplification: udpport, CVE-2018-0115, and internet exposure

Mon, 01 Jan 0001 00:00:00 +0000

You are here because a third party reported your server in a DDoS attack, a security scan flagged UDP 11211, or you are auditing an older deployment. The condition is the same in every case: memcached has UDP enabled (udpport is non-zero) and is reachable from a network you do not control.

This is CVE-2018-1000115. In early 2018 it powered reflection attacks peaking at 1.3 to 1.7 Tbps. A single small UDP GET request with a spoofed source IP can trigger a response of hundreds of kilobytes from an unauthenticated memcached instance, with observed amplification factors around 51,000x. The upstream fix in 1.5.6 (Feb 2018) made udpport default to 0, but that does not retroactively rewrite init scripts, container images, or distro configs.

Memcached unexpected restart: uptime reset, wiped cache, and the cold-start backend spike

Mon, 01 Jan 0001 00:00:00 +0000

The uptime stat in memcached’s stats output is a seconds-since-process-start counter. When it drops from hours or days back to near-zero, the process restarted. Because memcached has no persistence, a restart is a full cache wipe: every item is gone, curr_items falls to zero, and hit ratio collapses to 0%.

The restart is rarely the incident. The incident is the cold-start backend spike that follows. Every cache miss now hits the origin database or service at full production volume. If the backend was sized assuming 90%+ cache offload, it now sees several times its normal read load.