Memcached CPU saturation: the per-thread ceiling that aggregate graphs hide

Memcached is rarely CPU-bound. At normal request rates, worker threads spend most of their time in epoll waits. When CPU climbs on a memcached host, the reflex is to check aggregate rusage and conclude the daemon is busy. That reflex misses the actual failure mode.

Memcached dispatches each accepted connection round-robin to one of -t worker threads (default 4). Once assigned, a connection lives on that worker for its lifetime. The worker runs its own libevent loop and serves its assigned connections exclusively. If one worker saturates at 100% of a core, every connection pinned to that worker sees elevated latency. The other workers may be at 20%. The aggregate graph reads roughly 40%. Three quarters of your clients are fine.

Aggregate CPU, aggregate throughput, and even aggregate hit ratio can all look healthy while a quarter of your traffic suffers. The signal you need is per-thread, and memcached does not expose per-thread CPU directly. You derive it from OS-level thread views (top -H, ps -L, /proc/<pid>/task/<tid>/stat) and correlate it with conn_yields, which rises when workers cannot keep up with the request burst on their connections.

What this means

Two mechanisms cause the visible damage.

Per-thread ceiling. Each worker is single-threaded with no intra-thread parallelism. If the connections on worker 3 send requests faster than worker 3 can service them, requests queue inside that worker’s event loop. The clients on worker 3 see increasing latency. Clients on workers 0, 1, and 2 are unaffected.

conn_yields protection. When a single connection tries to run too many requests in one event loop iteration (default 20, set by -R), the worker yields that connection to the back of the queue and serves others. This is a fairness mechanism. A rising conn_yields rate means a worker is protecting itself from one aggressive connection at the cost of latency for that connection.

The two combine. If the workload saturates threads, conn_yields climbs with CPU. If only one client is misbehaving, conn_yields climbs without aggregate CPU saturation. Distinguishing the two is the diagnostic job.

A third CPU consumer runs in the background: the hash table expansion thread. When the cache fills and the hash table needs to grow, hash_is_expanding flips to 1 and the maintenance thread migrates items from the old table to the new one. This runs in a dedicated thread and does not block workers, but it does consume CPU and memory (the old and new tables coexist during migration). Correlate CPU spikes with hash_is_expanding = 1 before assuming request load is the cause.

flowchart TD
    A[Aggregate CPU climbing] --> B{Per-thread CPU balanced?}
    B -- No, one thread at 100% --> C[Per-thread ceiling]
    B -- Yes, all threads high --> D[Insufficient workers]
    B -- Yes, all threads low --> E[CPU spike not from workers]
    C --> F{conn_yields rising?}
    F -- Yes, with CPU --> G[Request volume exceeds capacity]
    F -- Yes, without CPU --> H[One aggressive client]
    F -- No --> I[Check hash_is_expanding]
    I --> J{hash_is_expanding = 1?}
    J -- Yes --> K[Background expansion]
    J -- No --> L[Check VmSwap, NIC, version]

Common causes

Quick checks

# Check daemon responsiveness and version
echo "version" | nc -q1 localhost 11211

# Cumulative CPU (per-process). Sample twice, 10s apart, compute delta.
echo "stats" | nc -q1 localhost 11211 | grep "STAT rusage"

# Connection fairness signal (cumulative). Derive rate across samples.
echo "stats" | nc -q1 localhost 11211 | grep "STAT conn_yields"

# Is the hash table expanding right now?
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT hash_(is_expanding|power_level|bytes)"

# Per-thread CPU at the OS level (press H in top for thread view)
top -H -p $(pgrep memcached)

# Same data, scriptable. Sort by CPU to find the hot thread.
ps -L -o tid,pcpu,comm -p $(pgrep memcached) | sort -k2 -rn | head

# Startup flags: -t (threads), -R (requests per event), -c (maxconns)
ps -o args= -p $(pgrep memcached)

# Rule out swap (any nonzero value is a production incident)
grep VmSwap /proc/$(pgrep memcached)/status

The -q1 flag works with netcat-openbsd (Debian/Ubuntu default). On RHEL/CentOS with nmap-ncat, substitute -w1. Avoid polling stats more frequently than every 10 seconds under high load.

How to diagnose it

1. Confirm aggregate CPU is genuinely climbing. Sample rusage_user and rusage_system twice, 10 seconds apart. The rate is the delta divided by the sample interval. A 4-thread memcached can use up to roughly 4 CPU-seconds per wall second.

2. Check per-thread CPU. This is the step most operators skip. top -H -p $(pgrep memcached) or ps -L -o tid,pcpu,comm -p $(pgrep memcached) shows CPU per thread. If one thread is near 100% of a core and others are much lower, you have a per-thread ceiling problem, not a global CPU problem.

3. Check conn_yields rate. Sample twice and compute the delta. If conn_yields is climbing with CPU, request volume is overwhelming at least one worker. If conn_yields is climbing but CPU is moderate, one client is sending large pipelines and the -R limit is doing its job.

4. Check hash_is_expanding. If this is 1, the hash table is mid-expansion. Check hash_power_level and curr_items for context. Expansion should complete in seconds to minutes. If it persists, suspect a very large item count or a version-specific bug.

5. Check version. Versions >=1.5.14 with hashpower=32 may be affected by an integer overflow in hashsize calculation that drives 100% CPU.

6. Check per-client contribution if possible. Memcached does not expose per-connection command counts natively. Infer from client-side metrics, network-level observation, or by temporarily isolating suspect clients.

7. Rule out adjacent causes. Swap (VmSwap in /proc/<pid>/status) causes CPU burn via page faults and is catastrophic for latency. NIC saturation (bytes_written versus link capacity) causes kernel system time to rise without user time rising proportionally.

Metrics and signals to monitor

Fixes

Per-thread saturation from request volume

Increase worker thread count. Bump -t at startup. The manpage recommends not exceeding the number of CPU cores, and 64 or more threads is not recommended. This requires a restart, which means total cache loss. Weigh against splitting traffic across more memcached instances, which also gives you more listener capacity. There is only one listener thread accepting new connections, and under heavy connection churn it can itself become a bottleneck.

Reduce per-request work. Larger values mean more serialization and more bytes to copy per response. If bytes_written / cmd_get is climbing, item size inflation is contributing to CPU. Compress large values on the client side before storage.

Upgrade to benefit from response batching. Pre-1.6.0 versions lack automatic batching of response syscalls. Upgrading can reduce server CPU.

Aggressive single-client pipelining

Evaluate -R tuning. The default of 20 means a connection can send 20 requests per event loop iteration before being yielded to the back of the queue. If your workload legitimately pipelines more than 20 requests per event and the client is not starving others, raising -R reduces conn_yields overhead. If the client is starving others, do not raise -R. Fix the client instead. Changing -R requires a restart.

Fix the client. A single client sending tight-loop requests or massive unbatched multigets is an application-level bug. Identify it through client-side telemetry or network-level observation and rate-limit or batch at the application layer.

Hash table expansion

Let it finish. Expansion is normal and self-completing. If hash_is_expanding is 1 for a few seconds, do nothing. Monitor hash_power_level and curr_items to understand the growth trajectory.

Presize the hash table on large caches. For deployments with very large item counts, runtime expansion to high hashpower values consumes CPU and temporarily doubles hash table memory. Presizing to the expected maximum at startup avoids runtime expansion entirely.

Upgrade if affected by the overflow bug.

Insufficient workers with idle cores

Increase -t to match the workload. If all worker threads are at high utilization and the host has idle cores, adding workers (up to core count) spreads connections across more event loops. Restart required, with total cache loss.

Prefer more instances over more threads at scale. Beyond a point, adding threads increases lock contention on the item lock table. That table scales with thread count. Multiple smaller instances on the same host, each with its own listener thread and slab allocator, can outperform one large instance with many threads.

Prevention

• Monitor per-thread CPU, not just aggregate. Alert on any single memcached thread sustained above 80% of a core for 5 or more minutes. This catches the problem before clients do.
• Track conn_yields rate as a capacity signal. A sustained non-zero rate means workers are actively throttling. Investigate before it becomes a latency incident.
• Size -t to the host and workload. Match worker count to core count. Do not exceed it.
• Keep memcached current. The hash expansion overflow fix and response batching are both meaningful for CPU behavior. Know which features your version has.
• Monitor hash_is_expanding duration. Expansion persisting beyond 60 seconds or recurring frequently indicates item count instability.
• Measure client-observed latency externally. Memcached exposes no latency histograms natively. Without client-side measurement, you cannot see the per-thread ceiling’s impact on users.

How Netdata helps

• Netdata collects rusage_user and rusage_system per second and derives rates automatically, so you see current CPU consumption without manual delta math.
• Per-thread CPU from the OS level is collected alongside the memcached stats context, letting you correlate aggregate rusage with actual per-thread saturation in the same time window.
• conn_yields rate is tracked alongside cmd_get and cmd_set, so the relationship between request volume, fairness throttling, and CPU is visible on one chart.
• Correlated dashboards let you overlay memcached CPU with NIC utilization and VmSwap, so you can rule out network saturation and swap as the cause.

Related guides

• Memcached cache stampede: a hot key expires and the backend takes the hit
• Memcached evicted_unfetched and expired_unfetched: caching data nobody ever reads
• Memcached cas_badval climbing: check-and-set contention and lost updates
• Memcached command rate anomalies: cmd_get and cmd_set spikes and sudden drops
• Memcached conn_yields rising: one client’s pipeline starving the others
• Memcached connection churn: total_connections racing and TIME_WAIT buildup
• Memcached curr_connections climbing: connection leaks and missing pooling
• Memcached connection limit reached: accepting_conns=0 and clients being refused
• Memcached connection refused: telling a dead process from a hung or full one
• Memcached evicted_time low: distinguishing healthy turnover from cache thrash
• Memcached eviction cascade: when a full cache overloads the backend
• Memcached evictions climbing: the cache is full and discarding live data