Memcached swapping: why any VmSwap on an in-memory cache is an incident

A nonzero VmSwap value in /proc/<pid>/status for a memcached process is a production incident, not a tuning concern. Every access to a swapped page costs milliseconds instead of nanoseconds, so a fraction of your cache lookups runs at disk speed. The point of an in-memory cache is lost for those items, and the resulting latency outliers are random and intermittent, almost impossible to trace from the application layer without checking swap first.

A value that looks small is still a real problem. 50 MB of swap on a process with a 2 GB cache means a measurable fraction of items is on disk. Operators dismiss the number as negligible and then spend hours chasing phantom latency spikes that only touch requests hitting swapped pages. The memcached project lists swap as the first item to check when investigating timeouts for exactly this reason.

The slab allocator manages memory pressure internally by evicting items within its -m budget. It does not rely on the OS swap subsystem. When the kernel pages out memcached memory, the host is under memory pressure that extends beyond memcached’s allocation. The fix is always at the system or container level, never inside memcached’s eviction or LRU configuration.

What this means

When VmSwap is nonzero for a memcached PID, some pages of the process address space are on disk: cached item data, hash table structures, connection buffers, thread stacks. Any GET touching a swapped page incurs a major page fault, blocking the worker thread until the kernel reads the page back from swap space.

The latency impact is severe and non-uniform. Most requests still hit in-RAM pages and return in under a millisecond. A subset stall for milliseconds or longer, depending on swap I/O contention. From the client, this looks like random latency outliers with no obvious pattern. p99 degrades while median latency stays flat, which makes the problem invisible to dashboards that track only averages.

Memcached exposes no swap metric in its own stats output. The server has no awareness that its memory is on disk. The only reliable signal is VmSwap from /proc/<pid>/status, read from the host.

flowchart TD
    A["Random latency spikes
on memcached"] --> B{"VmSwap nonzero?"}
    B -- No --> C["Investigate network,
CPU, slab evictions"]
    B -- Yes --> D["Swap confirmed.
Find the cause."]
    D --> E{"-k flag present?"}
    E -- No --> F["Missing mlockall.
Add -k on next start"]
    E -- Yes --> G{"Host RAM
oversubscribed?"}
    G -- Yes --> H["Free memory or
relocate instances"]
    G -- No --> I["Check container cgroup
swap limits"]

Note the documented scope: proc(5) defines VmSwap as the “amount of swap used by anonymous private data (shmem swap usage is not included)”. For a memcached process this is the relevant number — its heap is anonymous private memory — but shared-memory swap is out of scope for this counter. If symptoms persist with VmSwap showing zero, cross-check with /proc/<pid>/smaps_rollup and read the Swap field.

Common causes

Quick checks

All commands below are read-only and safe to run in production.

# Check swap for the memcached process - must be zero
PID=$(pgrep memcached)
grep VmSwap /proc/$PID/status

# Check daemon responsiveness and rough latency under potential swap pressure
time (echo "version" | nc -w 2 localhost 11211)

# Check overall system swap usage
free -m
swapon --show

# Check for active swap-in / swap-out activity (si and so columns)
vmstat -w 1 5

# Check if -k (lock memory) is in the memcached process arguments
ps -p $(pgrep memcached) -o args=

# Check RLIMIT_MEMLOCK for the memcached process
prlimit --pid $(pgrep memcached) | grep MEMLOCK

# Check for OOM killer activity in kernel logs
dmesg -T | grep -i -E "oom|killed process"

# Check memcached uptime for recent restarts (OOM kill plus auto-restart cycle)
echo "stats" | nc -w 2 localhost 11211 | grep "STAT uptime"

pgrep memcached returns only one PID. If you run multiple memcached instances per host, iterate over the PID list explicitly (pgrep -a memcached) and check each VmSwap.

How to diagnose it

1. Confirm the swap. Run grep VmSwap /proc/$(pgrep memcached)/status. If the value is nonzero, swap is confirmed and you are in an incident.

2. Assess user impact. Check client-side latency if available. Look for p99 or p99.9 spikes that do not correlate with cmd_get rate changes or eviction storms. If client latency is not instrumented, run time (echo "version" | nc -w 2 localhost 11211) several times against a sub-millisecond baseline to catch intermittent stalls.

3. Check for -k in the process arguments. Run ps -p $(pgrep memcached) -o args=. If -k is absent, memcached has no protection against swapping. This is the most common root cause.

4. Check host memory pressure. Run free -m for available memory, and vmstat -w 1 5 watching the si and so columns. Sustained nonzero values mean the host is actively swapping, not just carrying stale swap pages from a past pressure event.

5. Identify noisy neighbours. Run ps aux --sort=-%mem | head -20 to see top memory consumers. If memcached is not the top consumer, a co-located process is driving the pressure.

6. Check for OOM killer activity. Run dmesg -T | grep -i oom. If the OOM killer has been active, the host has been under severe pressure. Memcached may have been killed and restarted. Check the uptime stat for discontinuities.

7. If containerized, check the cgroup. Host free can look healthy while the container cgroup is constrained. A cgroup memory limit alone does not prevent swap. Check the cgroup memory and swap files directly, or inspect the container runtime configuration.

Metrics and signals to monitor

Fixes

Missing -k flag: enable mlockall

Add -k (also --lock-memory) to the memcached startup command. This calls mlockall() to lock all current and future pages into RAM, preventing the kernel from swapping them out.

Two prerequisites must be met before -k will work:

• Capability. The process needs root privileges or the CAP_IPC_LOCK capability. In systemd, use AmbientCapabilities=CAP_IPC_LOCK in the unit file. In Docker, add --cap-add=IPC_LOCK.

• MEMLOCK rlimit. The kernel default RLIMIT_MEMLOCK is 8 MiB, and distribution, init, and container configurations vary between 64 KB and unlimited — all far below a production cache’s resident size. Check with prlimit --pid $(pgrep memcached) | grep MEMLOCK. Raise it before starting memcached. In systemd, set LimitMEMLOCK=infinity. In /etc/security/limits.conf, add a memlock line for the memcached user.

Tradeoff. With -k active, the kernel cannot reclaim memcached pages under pressure. If the host is oversubscribed, the kernel will swap other processes instead, including system daemons and the application itself. That can cascade into worse failures than the original swap problem. Use -k when the host is properly sized, not as a substitute for adequate RAM.

Host memory overcommitment: free RAM or relocate

If -k is already set and swap still appears, or if enabling -k pushes other critical processes into swap, the host does not have enough physical RAM for its workload.

Options, from least to most disruptive:

• Move noisy neighbours off the host. If co-located processes spike memory, relocate them or schedule them on separate hosts.
• Reduce memcached -m. If the cache allocation is too large for the host after OS overhead and co-located services, shrink it. This trades cache effectiveness for stability.
• Add RAM to the host. The durable fix.
• Disable swap on cache hosts. Running swapoff -a on a host that is actively swapping is destructive and can trigger immediate OOM kills. Only do this when the host has enough free RAM to absorb the swapped pages, or during a maintenance window after adding RAM.

Container without swap limit: constrain the cgroup

A cgroup memory limit alone does not prevent swap. The kernel will use swap when the container hits its memory limit unless you also set a swap limit.

Docker. Set --memory and --memory-swap to the same value. --memory-swap is the memory-plus-swap total, so --memory=2g --memory-swap=2g gives the container 2 GB of RAM and zero bytes of swap headroom. The Docker documentation states this directly: with both set to the same positive value, “the container doesn’t have access to swap”.

cgroup v2. Set memory.swap.max to 0 to disable swap for the cgroup. This is the direct swap limit, separate from memory.max.

cgroup v1. Set memory.memsw.limit_in_bytes equal to memory.limit_in_bytes. The memsw limit is the combined memory-plus-swap ceiling, so setting it equal to the memory limit leaves zero room for swap.

Prevention

• Run with -k on properly sized hosts. This is the primary defense. Ensure RLIMIT_MEMLOCK and CAP_IPC_LOCK are configured before enabling it.
• Monitor VmSwap continuously. It should always read zero for a memcached process. Alert on any nonzero value, no matter how small.
• Size hosts with headroom. Budget for memcached -m, which already includes per-item metadata — each slab chunk carries the item header alongside key and value — but excludes the hash table, connection buffers, and thread stacks. Add those, plus OS memory and co-located processes, on top. The overhead varies with key length, value size, and connection count, so measure RSS against -m on your own workload rather than assuming a fixed percentage. Do not fill the host to capacity.
• Avoid co-locating memcached with variable-memory workloads. Batch jobs, build agents, and JVM processes with large heaps are common drivers of pressure spikes that cascade into cache hosts.
• Set container swap limits explicitly. Never assume that a memory limit prevents swap on its own.
• Disable swap on dedicated cache hosts if the workload and host sizing allow it. This converts silent degradation into a hard OOM kill, which is easier to detect and alert on.

How Netdata helps

• Per-process VmSwap tracking. Netdata’s apps plugin reads /proc/<pid>/status and surfaces VmSwap per process. A nonzero reading for memcached becomes visible immediately rather than at the end of a polling interval.
• Correlation with memcached internals. Netdata’s memcached collector exposes uptime, cmd_get, cmd_set, hit ratio, eviction rate, and connection metrics alongside the OS-level swap data. A VmSwap increase coinciding with an uptime reset (OOM kill) or a hit ratio drop narrows the root cause in seconds.
• System-wide pressure signals. Netdata tracks host swap usage, available memory, and swap-in/swap-out rates from kernel counters. When these rise alongside memcached swap, the problem is host-level overcommitment, not a memcached configuration issue.
• Container cgroup visibility. In containerized deployments, Netdata surfaces cgroup memory and swap metrics directly, so you can see when a container has hit its memory limit and is spilling to swap without SSH access to the host.
• ML-based anomaly detection. Swap-related latency spikes produce unusual patterns in memcached throughput and connection metrics. Netdata’s anomaly advisor flags these deviations before explicit thresholds are crossed.

Related guides

• Memcached cache stampede: a hot key expires and the backend takes the hit
• Memcached evicted_unfetched and expired_unfetched: caching data nobody ever reads
• Memcached cas_badval climbing: check-and-set contention and lost updates
• Memcached command rate anomalies: cmd_get and cmd_set spikes and sudden drops
• Memcached conn_yields rising: one client’s pipeline starving the others
• Memcached connection churn: total_connections racing and TIME_WAIT buildup
• Memcached curr_connections climbing: connection leaks and missing pooling
• Memcached connection limit reached: accepting_conns=0 and clients being refused
• Memcached connection refused: telling a dead process from a hung or full one
• Memcached evicted_time low: distinguishing healthy turnover from cache thrash
• Memcached eviction cascade: when a full cache overloads the backend
• Memcached evictions climbing: the cache is full and discarding live data