Memcached alive but not responding: the silent process hang

The port check passes. The process shows up in ps. The kernel still accepts TCP on 11211. But every stats and version probe times out, and dashboards show cmd_get and cmd_set falling off a cliff. This is the silent process hang: memcached is effectively down while appearing alive to every shallow health check.

Most liveness checks stop at “is something listening on the port?” That question is answered by the kernel’s TCP stack, not by memcached’s worker threads. A frozen main thread, a stuck slab rebalancer, a process suspended by SIGSTOP, or an OOM kill in progress can all leave the listen backlog accepting connections while no command ever executes. Clients connect, send a command, and wait forever.

The detection fix is to require a command response, not a port response. Sustained failure of a version or stats probe over 30 to 60 seconds, combined with previously stable uptime, rules out transient blips and fresh restarts. This article walks through what to check first, how to tell the causes apart, and what to do once you confirm the process is hung.

What this means

Memcached’s listener thread accepts TCP connections and hands them round-robin to -t worker threads (default 4), each running its own libevent loop. The kernel’s TCP stack completes the three-way handshake for new connections as long as the listen backlog has room, even if no worker thread ever calls recv() on the new socket. A TCP port check answers “is the kernel accepting SYNs on this port?” rather than “is memcached processing commands?”

When the worker threads stop making forward progress, the symptoms cluster:

• New connections succeed (kernel accepts them into the backlog).
• Existing connections stop receiving responses.
• cmd_get and cmd_set rates collapse toward zero because nothing is being processed.
• Clients that give up and close leave the server side of those connections in CLOSE_WAIT, because memcached never finishes its side of the close.
• CLOSE_WAIT accumulates toward max_connections, at which point listen_disabled_num increments and even new connection attempts start failing.

A plain port check is actively misleading here. It returns “open” right up until the connection limit is hit, then “closed” for an unrelated reason. Neither state reflects the real condition: the process is alive but not doing work.

flowchart TD
    A[Client TCP connect to :11211] --> B[Kernel accepts into backlog]
    B --> C[Client sends stats or version]
    C --> D{Worker thread reads?}
    D -->|No: thread deadlock| E[Probe times out]
    D -->|No: state T SIGSTOP| E
    D -->|No: state D uninterruptible IO| E
    D -->|No: pages swapped to disk| E
    D -->|No: OOM kill in progress| E
    D -->|Yes| F[Response returned]
    E --> G[cmd_get and cmd_set drop to near zero]
    G --> H[CLOSE_WAIT accumulates on server side]
    H --> I[curr_connections climbs toward max_connections]

Common causes

Historical note: upstream issue #446 reported the 1.5.6 TCP/CLOSE_WAIT symptom on an Ubuntu package; the maintainer suggested 1.5.x stability issues and suspected rare hash-expansion lock cases, but did not confirm a root cause or fix. A separate dispatcher/rebalancer deadlock after slabs reassign followed by shutdown was reported in issue #738; the maintainer treated it as rare and shutdown-only. Neither report supports a specific “fixed in” version, though both fit this symptom signature.

Quick checks

Run these before touching the process. All are read-only.

# 1. Probe command response with a hard timeout (not just a port check)
timeout 3 bash -c 'printf "version\r\n" | nc -w 2 localhost 11211'
# Expected on a healthy process: VERSION x.y.z
# Timeout or empty output = the hang signature

# 2. Check process state from ps
ps -o pid,stat,pcpu,pmem,etime,cmd -p $(pgrep -x memcached)
# STAT column: R=running, S=sleeping, T=stopped (SIGSTOP), D=uninterruptible sleep

# 3. Check swap and state from /proc
PID=$(pgrep -x memcached)
grep -E "^(State|VmRSS|VmSwap|VmSize)" /proc/$PID/status
# VmSwap must be 0. Any nonzero is a performance incident for an in-memory cache.

# 4. Look for OOM killer activity in the kernel ring buffer
sudo dmesg -T | grep -iE "oom|memcached" | tail -30

# 5. Check memory pressure (PSI, v4.20+ kernels)
cat /proc/pressure/memory 2>/dev/null || echo "PSI not available on this kernel"
free -m

# 6. Count CLOSE_WAIT connections on the memcached port
ss -tan '( sport = :11211 )' | awk 'NR>1{print $1}' | sort | uniq -c
# CLOSE_WAIT pile-up = clients gave up waiting; server side never closed

# 7. Snapshot all thread states
PID=$(pgrep -x memcached)
for t in /proc/$PID/task/*/stat; do
  awk '{print $1, $3}' "$t"
done | sort | uniq -c
# A cluster of threads all in the same non-R state suggests a lock or stall

# 8. Check pending signals (was SIGSTOP sent and not yet delivered?)
PID=$(pgrep -x memcached)
grep -E "^(SigPnd|ShdPnd|SigBlk|SigIgn|SigCgt)" /proc/$PID/status

How to diagnose it

Work through these in order. The goal is to identify the cause before restarting, because a blind restart destroys both the evidence and the cache.

1. Confirm the hang with a command probe. Run the version probe from Quick check 1 at least three times over 30 to 60 seconds. A single timeout can be a transient blip. Sustained timeout with previously stable uptime is the real signal.

2. Read the process state. If ps shows T, the process received SIGSTOP. Look for the source: a debugger attach, a cgroup freezer operation, or a manual kill -STOP. Check cgroup v2 freezer state at /sys/fs/cgroup/.../cgroup.freeze if applicable. If ps shows D, the process is stuck in an uninterruptible kernel sleep, usually IO. That points at swap thrashing or a filesystem issue (rare for memcached, which does no disk IO unless extstore is enabled).

3. Check swap. VmSwap in /proc/<pid>/status must be zero for memcached. Any nonzero value means some cache pages are being served at disk speed, which can progress so slowly that the process appears hung. Cross-reference with vmstat 1 (watch the si and so columns) and /proc/pressure/memory if available. If memcached was started without -k (which calls mlockall), it is eligible for swapping.

4. Check for OOM killer activity. dmesg -T | grep -i oom shows whether the kernel’s OOM killer targeted memcached. An OOM kill in progress can leave the process in a half-killed state where it still owns the port but is tearing down. This is brief but visible if you catch it in time.

5. Inspect thread states. If the process state is R or S (running or sleeping normally) but command probes still time out, suspect a userspace deadlock. Snapshot all thread states from /proc/<pid>/task/*/stat. If all worker threads are stuck in the same state, that is consistent with lock contention or a deadlock. Capturing pid-targeted thread backtraces before restarting gives upstream maintainers the evidence they need. Note that gdb -p <pid> attaches via PTRACE_ATTACH, which pauses the target; on a process already suspected hung this is low risk, but on a live system it adds latency.

6. Check the command history if you have it. A slabs reassign followed by shutdown is the reported reproduction path for the slab rebalancer versus dispatcher deadlock. If your automation or an operator issued those commands, that is likely the cause.

Metrics and signals to monitor

Fixes

Grouped by cause. Do not restart as a first resort; you lose the cache and the diagnostic evidence.

Process suspended (state T)

Send SIGCONT to resume: kill -CONT <pid>. Then find what sent the SIGSTOP. Common sources are a stray debugger, a cgroup freezer operation, or a manual kill -STOP from an operator or automation. Check shell history, systemd unit files, and container orchestration for freezer hooks. If you cannot find the source, treat it as a security signal: an unauthorized process sent a stop signal.

Swap thrashing

If VmSwap is nonzero and the system is actively thrashing, the process may be making forward progress so slowly that it looks hung. Immediate mitigation is to relieve memory pressure on the host: stop other memory-heavy processes, or drop pagecache with echo 1 > /proc/sys/vm/drop_caches (brief IO stall expected; harmless to data but visible to anything doing buffered IO). Structural fix: ensure memcached starts with -k to call mlockall and lock its pages in RAM, and ensure the host has enough physical memory for memcached plus everything else running there. On kernels older than 4.20, swap thrashing can livelock without the OOM killer firing promptly, because allocations technically succeed via swap. PSI arrived in 4.20; OOM timing remains cgroup- and kernel-configuration-dependent, so absence of an OOM message is not proof that severe thrashing is absent.

OOM killer activity

If dmesg shows the OOM killer targeting memcached, the host is overcommitted. The process may already be dead or dying. Either way, reduce memory pressure: lower memcached’s -m, move other processes off the host, or add RAM. Restart memcached after the host has headroom. Prevent recurrence with monitoring on system memory and dmesg for OOM events, and consider -k plus CAP_IPC_LOCK to make memcached’s memory unswappable.

Thread deadlock

This is the rarest case and the one where a restart is usually unavoidable. Before restarting, capture thread backtraces if you can (gdb -p <pid> -batch -ex "thread apply all bt"), since the deadlock state is the evidence upstream needs. The attach pauses the target briefly. After restart, avoid the known trigger patterns: do not issue slabs reassign immediately followed by shutdown, and keep memcached on a current 1.6.x release. If the deadlock is reproducible, file an issue with the backtraces.

CLOSE_WAIT pile-up as a secondary effect

CLOSE_WAIT accumulation is a symptom, not a cause. Once you fix the underlying hang, existing CLOSE_WAIT connections close as memcached processes its side of the close. If they do not, let them time out or restart the process. Do not raise max_connections to work around CLOSE_WAIT pile-up; that just delays the wall.

Prevention

• Alert on command probes, not port checks. A version or stats probe with a 2-3 second timeout, sustained over 30-60 seconds with stable uptime, is the only reliable liveness signal for this class of failure.
• Run memcached with -k. Calls mlockall, prevents swap. Requires CAP_IPC_LOCK or root. Without it, swap thrashing is always a possible cause of “alive but not responding.”
• Keep memcached on a current stable release. The reports above were never linked to confirmed fixed-in versions; current releases reduce exposure to accumulated fixes but do not make command probes optional. If you are on an older distro package, check the version first.
• Monitor system memory and OOM events. dmesg OOM messages and host-level memory pressure are leading indicators. PSI (/proc/pressure/memory) on v4.20+ kernels gives early warning before thrashing stalls the process.
• Avoid known deadlock triggers. Do not script slabs reassign followed by shutdown. If you need to restart after a slab rebalance, leave a gap.
• Watch CLOSE_WAIT trends. A sudden CLOSE_WAIT increase on the memcached port is a strong secondary signal that clients are giving up on responses.

How Netdata helps

• Per-second command probes with timeout. Netdata’s memcached collector issues stats probes at per-second resolution, so a sustained hang over 30-60 seconds shows as a flat line on the response chart, not a slow-burn miss.
• Correlate process state with throughput. When cmd_get and cmd_set drop to near zero, overlay process CPU, RSS, and per-process VmSwap (collected from /proc) on the same timeline to see whether the drop coincides with a state change, a swap spike, or an OOM event without pivoting between tools.
• VmSwap and PSI on the same dashboard. Netdata surfaces per-process VmSwap and system-level memory pressure (/proc/pressure/memory) alongside cache metrics, so swap-induced hangs are distinguishable from userspace deadlocks at a glance.
• Anomaly detection on throughput. ML-based anomaly detection on cmd_get and cmd_set rates flags the collapse before threshold-based alerts fire, which matters when the hang is intermittent.
• Connection-state visibility. Netdata’s TCP monitoring shows CLOSE_WAIT accumulation on the memcached port, giving the secondary signal that confirms clients are stuck waiting on responses.

Related guides

• Memcached connection refused: telling a dead process from a hung or full one
• How Memcached actually works in production: a mental model for operators
• Memcached monitoring maturity model: from survival to expert