Memcached unexpected restart: uptime reset, wiped cache, and the cold-start backend spike

The uptime stat in memcached’s stats output is a seconds-since-process-start counter. When it drops from hours or days back to near-zero, the process restarted. Because memcached has no persistence, a restart is a full cache wipe: every item is gone, curr_items falls to zero, and hit ratio collapses to 0%.

The restart is rarely the incident. The incident is the cold-start backend spike that follows. Every cache miss now hits the origin database or service at full production volume. If the backend was sized assuming 90%+ cache offload, it now sees several times its normal read load.

Work two tracks in parallel: find out why the process died (OOM kill, segfault, orchestration, package upgrade) so it does not repeat, and manage the cache-warming period to protect the backend until hit ratio recovers.

What this means

There is no WAL, no snapshot, no replay. The diagnostic fingerprint is narrow and specific:

• uptime resets to a small number of seconds
• curr_items is zero, or climbing from zero
• cmd_get continues at normal or elevated rate (clients are still asking)
• get_hits is near zero
• get_misses spikes to match cmd_get
• cmd_set spikes as the application writes miss results back

The backend sees the miss traffic directly. Correlate the memcached uptime discontinuity with a backend load spike at the same timestamp. That correlation is what confirms “unexpected restart caused the backend spike” rather than “backend is slow for an unrelated reason”.

flowchart TD
    A[Process dies: OOM, segfault, signal] --> B[Supervisor restarts memcached]
    B --> C[uptime resets, curr_items = 0]
    C --> D[All GETs miss]
    D --> E[Miss traffic hits backend]
    E --> F{Backend headroom?}
    F -->|Yes| G[Cache warms over minutes/hours]
    F -->|No| H[Backend saturates, latency spikes]
    H --> I[Client timeouts, retries, cascade]

Common causes

Quick checks

Run these read-only checks against the affected host. They are all safe and collect no destructive state.

# Confirm the restart fingerprint
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT (uptime|curr_items|cmd_get|get_hits|get_misses)"

# Check the running version
echo "version" | nc -q1 localhost 11211

# Look for OOM killer activity in the kernel ring buffer
dmesg -T | grep -iE "oom|killed process"

# Check systemd's view of recent restarts and exit status
journalctl -u memcached --since "2 hours ago" --no-pager

# Inspect process state, start time, and RSS
ps -o pid,ppid,stat,lstart,rss,cmd -p $(pgrep -x memcached)

# Check for swap (any nonzero VmSwap is a production incident for memcached)
grep -E "Vm(RSS|Swap)" /proc/$(pgrep -x memcached)/status

# Confirm the configured memory limit and max connections
echo "stats settings" | nc -q1 localhost 11211 | grep -E "STAT (maxbytes|maxconns|udpport)"

For Kubernetes, add:

# Pod restart count and last termination reason
kubectl describe pod <pod> | grep -A5 "Last State"

# Recent events in the namespace
kubectl get events --sort-by=.lastTimestamp -n <namespace> | tail -30

How to diagnose it

1. Confirm it was actually a restart. Compare current uptime against your previous sample. A discontinuity (new value smaller than the last) is definitive. A single low sample without history could be a fresh deploy; check deploy timestamps before paging.
2. Distinguish crash from clean exit. A crash leaves evidence in the kernel log (OOM) or systemd journal (signal, exit code). A clean exit followed by restart suggests orchestration or a manual systemctl restart. The fix path is completely different.
3. Check for OOM kill. Run dmesg -T | grep -i oom. If memcached appears, the kernel killed it for memory. Determine whether the pressure came from memcached’s own overhead growing (connection buffers, hash table) or a neighbor process on the host. Check cgroup limits separately from host free memory.
4. Check for segfault. No OOM evidence plus a non-zero exit code in journalctl suggests a crash. Pull the version (echo "version" | nc ...) and compare against upstream release notes. The 1.6.x line has shipped multiple crash fixes in the network path, proxy code, and memory allocation. Running an older 1.6.x is a credible root cause.
5. Check orchestration. For containerized deployments, the pod’s Last State and the controller event log show whether the restart was initiated by the kubelet, an eviction, a liveness probe failure, a rolling update, or a node reschedule. A liveness probe that sends stats but times out during a legitimate traffic spike can cause a restart loop that looks exactly like a crash.
6. Check systemd restart policy. The systemd default OOMPolicy is stop, meaning the service will not restart after an OOM kill unless explicitly configured. If memcached came back on its own after OOM, someone set Restart=on-failure or Restart=always plus OOMPolicy=continue. If it did not come back, the policy is the reason.
7. Correlate with backend load. Pull backend metrics (DB queries/sec, origin latency, connection count) for the same timestamp window as the uptime reset. A spike that begins within seconds of the reset confirms the cold-start cascade.

Metrics and signals to monitor

Fixes

If the cause was OOM kill

The slab allocator bounds cache memory at -m. OOM kills usually come from process overhead (connection buffers, hash table) or a neighbor process, not from cache growth.

• Check whether the cgroup or host memory limit leaves headroom above limit_maxbytes plus expected overhead. Budget roughly 30-40% on top of -m for the hash table, connection buffers, thread stacks, and internal structures.
• Reduce -c (max connections) if connection buffer memory is the overhead source, or add system memory.
• Verify -k (mlockall) is set if swapping contributed. -k requires CAP_IPC_LOCK or root privileges.
• In containers, raise the cgroup memory limit. Do not raise -m beyond what the cgroup can hold.

If the cause was a segfault

• Upgrade to the latest stable release. The 1.6.x line has accumulated multiple crash fixes in the network path, proxy code, and memory allocation. A version even a few minor releases behind can carry a known crash bug.
• If core dumps are enabled, preserve the core file and the exact binary before upgrading.
• Disable UDP if it is enabled. UDP has been off by default since 1.5.6 and has a history of crash-relevant bugs.

If the cause was orchestration

• Distinguish intentional rolling updates from accidental restarts. A rolling update is expected; a liveness-probe-triggered restart loop is a bug in the probe configuration.
• Tune the liveness probe. A probe that sends stats and times out in one second during a traffic spike will kill healthy pods. Give the probe enough time and verify with a command response, not just a TCP connect.
• For Kubernetes, the warm restart feature (1.5.18+, using -e /tmpfs_mount/memory_file) needs a ram disk that survives the pod’s restart. On an ephemeral container filesystem without a persistent volume or an emptyDir with medium: Memory, warm restart does not help across pod reschedules.

Managing the cold-start backend spike

Regardless of cause, the backend spike is the active incident. Work it in parallel with root cause investigation.

• Do not restart memcached again. A second restart resets warming progress to zero.
• Shed or rate-limit backend traffic if the backend is saturating. Circuit breakers, request shedding, or graceful degradation in the application layer protect the origin while the cache warms.
• Run cache-warming scripts if they exist. Pre-populate the hottest keys (known feature flags, configuration, top content) to short-circuit the stampede.
• Watch hit ratio recovery. A session cache with 30-minute TTLs takes roughly 30 minutes to approach steady state. A content cache with hour-long TTLs takes longer. A stalled hit ratio after the expected warmup window indicates a different problem.
• Expect a temporary cmd_set spike as misses are written back. This is healthy warming behavior, not a runaway write loop.

Prevention

• Track uptime with discontinuity detection. The signal is not “uptime is low” but “uptime went down”. Alert on any new sample smaller than the previous sample, with planned-maintenance suppression.
• Track cmd_flush separately. A flush_all produces the same cold-cache fingerprint as a restart, but the process stays up. Alert on any increment in production.
• Maintain a warming strategy. Know which keys matter, have a script ready, and practice it outside of incidents.
• Fix restart-loop risk. If the supervisor or orchestrator restarts on failure, make sure the failure condition is not self-reinforcing (OOM on restart, immediate reload, repeat).
• Size for overhead, not just -m. Budget process RSS. Account for connection buffers and hash table memory when setting cgroup or host limits.
• Keep current. The maintainer has explicitly not assigned CVEs for recent security and crash fixes, so track release notes rather than CVE databases.

How Netdata helps

• Per-second uptime collection. A restart is visible within one collection cycle, not at the next five-minute scrape.
• Correlated timelines. uptime, curr_items, get_hits, get_misses, cmd_set, and backend metrics share the same timeline. The cold-start cascade appears as a synchronized pattern rather than four separate dashboards.
• ML anomaly detection on hit ratio and command rates. A sudden hit-ratio collapse after a stable period flags as anomalous even before a static threshold would fire.
• Process and cgroup metrics next to memcached stats. RSS, VmSwap, cgroup memory pressure, and OOM-kill events from the kernel appear alongside memcached’s own counters, which is what distinguishes a crash from an OOM kill.
• Backend integration. Database and origin metrics collected by the same agent confirm the cascade without correlating across tools.

Related guides

• Memcached connection refused: telling a dead process from a hung or full one
• How Memcached actually works in production: a mental model for operators
• Memcached monitoring checklist: the signals every production cache needs
• Memcached monitoring maturity model: from survival to expert