A nonzero VmSwap value in /proc/<pid>/status for a memcached process is a production incident, not a tuning concern. Every access to a swapped page costs milliseconds instead of nanoseconds, so a fraction of your cache lookups runs at disk speed. The point of an in-memory cache is lost for those items, and the resulting latency outliers are random and intermittent, almost impossible to trace from the application layer without checking swap first.

A value that looks small is still a real problem. 50 MB of swap on a process with a 2 GB cache means a measurable fraction of items is on disk. Operators dismiss the number as negligible and then spend hours chasing phantom latency spikes that only touch requests hitting swapped pages. The memcached project lists swap as the first item to check when investigating timeouts for exactly this reason.

The slab allocator manages memory pressure internally by evicting items within its -m budget. It does not rely on the OS swap subsystem. When the kernel pages out memcached memory, the host is under memory pressure that extends beyond memcached’s allocation. The fix is always at the system or container level, never inside memcached’s eviction or LRU configuration.

What this means

When VmSwap is nonzero for a memcached PID, some pages of the process address space are on disk: cached item data, hash table structures, connection buffers, thread stacks. Any GET touching a swapped page incurs a major page fault, blocking the worker thread until the kernel reads the page back from swap space.

The latency impact is severe and non-uniform. Most requests still hit in-RAM pages and return in under a millisecond. A subset stall for milliseconds or longer, depending on swap I/O contention. From the client, this looks like random latency outliers with no obvious pattern. p99 degrades while median latency stays flat, which makes the problem invisible to dashboards that track only averages.

Memcached exposes no swap metric in its own stats output. The server has no awareness that its memory is on disk. The only reliable signal is VmSwap from /proc/<pid>/status, read from the host.

flowchart TD
    A["Random latency spikes
on memcached"] --> B{"VmSwap nonzero?"}
    B -- No --> C["Investigate network,
CPU, slab evictions"]
    B -- Yes --> D["Swap confirmed.
Find the cause."]
    D --> E{"-k flag present?"}
    E -- No --> F["Missing mlockall.
Add -k on next start"]
    E -- Yes --> G{"Host RAM
oversubscribed?"}
    G -- Yes --> H["Free memory or
relocate instances"]
    G -- No --> I["Check container cgroup
swap limits"]

/proc/pid/status is documented in proc(5) as potentially inaccurate, undercounting shared pages. A zero reading reduces confidence but does not eliminate the possibility of swapped pages. If symptoms persist with VmSwap showing zero, cross-check with /proc/<pid>/smaps_rollup and read the Swap field.

Common causes

CauseWhat it looks likeFirst thing to check
Missing -k / mlockallmemcached started without -k; swap appears under any host memory pressureps -p $(pgrep memcached) -o args= for the -k flag
Host memory overcommitmentTotal process memory exceeds physical RAM; kernel swaps least-recently-used pages, which may include memcachedfree -m and vmstat -w 1 for si / so columns
Noisy neighboursOther processes on the shared host spike memory; kernel reclaims pages from memcached`ps aux –sort=-%mem
Container without swap limitcgroup memory limit is set but no swap limit; process hits cgroup ceiling and spills to swapcgroup memory.swap.current or Docker inspect for --memory-swap

Quick checks

All commands below are read-only and safe to run in production.

# Check swap for the memcached process - must be zero
PID=$(pgrep memcached)
grep VmSwap /proc/$PID/status

# Check daemon responsiveness and rough latency under potential swap pressure
time (echo "version" | nc -w 2 localhost 11211)

# Check overall system swap usage
free -m
swapon --show

# Check for active swap-in / swap-out activity (si and so columns)
vmstat -w 1 5

# Check if -k (lock memory) is in the memcached process arguments
ps -p $(pgrep memcached) -o args=

# Check RLIMIT_MEMLOCK for the memcached process
prlimit --pid $(pgrep memcached) | grep MEMLOCK

# Check for OOM killer activity in kernel logs
dmesg -T | grep -i -E "oom|killed process"

# Check memcached uptime for recent restarts (OOM kill plus auto-restart cycle)
echo "stats" | nc -w 2 localhost 11211 | grep "STAT uptime"

pgrep memcached returns only one PID. If you run multiple memcached instances per host, iterate over the PID list explicitly (pgrep -a memcached) and check each VmSwap.

How to diagnose it

  1. Confirm the swap. Run grep VmSwap /proc/$(pgrep memcached)/status. If the value is nonzero, swap is confirmed and you are in an incident.

  2. Assess user impact. Check client-side latency if available. Look for p99 or p99.9 spikes that do not correlate with cmd_get rate changes or eviction storms. If client latency is not instrumented, run time (echo "version" | nc -w 2 localhost 11211) several times against a sub-millisecond baseline to catch intermittent stalls.

  3. Check for -k in the process arguments. Run ps -p $(pgrep memcached) -o args=. If -k is absent, memcached has no protection against swapping. This is the most common root cause.

  4. Check host memory pressure. Run free -m for available memory, and vmstat -w 1 5 watching the si and so columns. Sustained nonzero values mean the host is actively swapping, not just carrying stale swap pages from a past pressure event.

  5. Identify noisy neighbours. Run ps aux --sort=-%mem | head -20 to see top memory consumers. If memcached is not the top consumer, a co-located process is driving the pressure.

  6. Check for OOM killer activity. Run dmesg -T | grep -i oom. If the OOM killer has been active, the host has been under severe pressure. Memcached may have been killed and restarted. Check the uptime stat for discontinuities.

  7. If containerized, check the cgroup. Host free can look healthy while the container cgroup is constrained. A cgroup memory limit alone does not prevent swap. Check the cgroup memory and swap files directly, or inspect the container runtime configuration.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
VmSwap from /proc/<pid>/statusDirect measurement of swapped memcached pagesAny nonzero value is an incident
si / so from vmstatActive swap-in and swap-out at the host levelSustained nonzero rate
free -m available memoryHost-level memory headroomAvailable memory approaching zero
Client-observed p99 latencyThe user-visible symptom of swapped pagesp99 spikes while median stays flat
memcached uptime statOOM kill followed by auto-restartSudden drop from the previous check
OOM killer entries in dmesgKernel killed a process under pressureAny recent oom-kill or Killed process lines
rusage_system rateKernel time rises with major page fault servicingUnexpected rise not correlated with request volume

Fixes

Missing -k flag: enable mlockall

Add -k (also --lock-memory) to the memcached startup command. This calls mlockall() to lock all current and future pages into RAM, preventing the kernel from swapping them out.

Two prerequisites must be met before -k will work:

  • Capability. The process needs root privileges or the CAP_IPC_LOCK capability. In systemd, use AmbientCapabilities=CAP_IPC_LOCK in the unit file. In Docker, add --cap-add=IPC_LOCK.

  • MEMLOCK rlimit. The default RLIMIT_MEMLOCK on Linux is typically 64 KB, far below the needs of a production cache. Raise it before starting memcached. In systemd, set LimitMEMLOCK=infinity. In /etc/security/limits.conf, add a memlock line for the memcached user.

Tradeoff. With -k active, the kernel cannot reclaim memcached pages under pressure. If the host is oversubscribed, the kernel will swap other processes instead, including system daemons and the application itself. That can cascade into worse failures than the original swap problem. Use -k when the host is properly sized, not as a substitute for adequate RAM.

Host memory overcommitment: free RAM or relocate

If -k is already set and swap still appears, or if enabling -k pushes other critical processes into swap, the host does not have enough physical RAM for its workload.

Options, from least to most disruptive:

  • Move noisy neighbours off the host. If co-located processes spike memory, relocate them or schedule them on separate hosts.
  • Reduce memcached -m. If the cache allocation is too large for the host after OS overhead and co-located services, shrink it. This trades cache effectiveness for stability.
  • Add RAM to the host. The durable fix.
  • Disable swap on cache hosts. Running swapoff -a on a host that is actively swapping is destructive and can trigger immediate OOM kills. Only do this when the host has enough free RAM to absorb the swapped pages, or during a maintenance window after adding RAM.

Container without swap limit: constrain the cgroup

A cgroup memory limit alone does not prevent swap. The kernel will use swap when the container hits its memory limit unless you also set a swap limit.

Docker. Set --memory and --memory-swap to the same value. For example, --memory=2g --memory-swap=2g gives the container 2 GB of RAM and zero bytes of swap headroom.

cgroup v2. Set memory.swap.max to 0 to disable swap for the cgroup. This is the direct swap limit, separate from memory.max.

cgroup v1. Set memory.memsw.limit_in_bytes equal to memory.limit_in_bytes. The memsw limit is the combined memory-plus-swap ceiling, so setting it equal to the memory limit leaves zero room for swap.

Prevention

  • Run with -k on properly sized hosts. This is the primary defense. Ensure RLIMIT_MEMLOCK and CAP_IPC_LOCK are configured before enabling it.
  • Monitor VmSwap continuously. It should always read zero for a memcached process. Alert on any nonzero value, no matter how small.
  • Size hosts with headroom. Budget for memcached -m plus the overhead for hash tables, connection buffers, thread stacks, and the slab allocator’s per-item metadata, plus OS memory and co-located processes. Do not fill the host to capacity.
  • Avoid co-locating memcached with variable-memory workloads. Batch jobs, build agents, and JVM processes with large heaps are common drivers of pressure spikes that cascade into cache hosts.
  • Set container swap limits explicitly. Never assume that a memory limit prevents swap on its own.
  • Disable swap on dedicated cache hosts if the workload and host sizing allow it. This converts silent degradation into a hard OOM kill, which is easier to detect and alert on.

How Netdata helps

  • Per-process VmSwap tracking. Netdata’s apps plugin reads /proc/<pid>/status and surfaces VmSwap per process. A nonzero reading for memcached becomes visible immediately rather than at the end of a polling interval.
  • Correlation with memcached internals. Netdata’s memcached collector exposes uptime, cmd_get, cmd_set, hit ratio, eviction rate, and connection metrics alongside the OS-level swap data. A VmSwap increase coinciding with an uptime reset (OOM kill) or a hit ratio drop narrows the root cause in seconds.
  • System-wide pressure signals. Netdata tracks host swap usage, available memory, and swap-in/swap-out rates from kernel counters. When these rise alongside memcached swap, the problem is host-level overcommitment, not a memcached configuration issue.
  • Container cgroup visibility. In containerized deployments, Netdata surfaces cgroup memory and swap metrics directly, so you can see when a container has hit its memory limit and is spilling to swap without SSH access to the host.
  • ML-based anomaly detection. Swap-related latency spikes produce unusual patterns in memcached throughput and connection metrics. Netdata’s anomaly advisor flags these deviations before explicit thresholds are crossed.