$ guides / memcached / memcached-latency-high ▌

Operations Guides

Memcached latency high: diagnosing slow gets when the server exposes no latency metric

Memcached does not expose a latency histogram, p50, or p99. The standard stats output is all counters and gauges at the moment of query: no time-series, no distribution, no per-operation duration. If your application is reporting slow cache reads, the daemon itself cannot confirm or deny the problem. You must measure latency at the client, then use server-side and OS signals to localize the cause.

On a healthy LAN, memcached serves small-item gets in sub-millisecond time. Sustained p99 above a few milliseconds means something is wrong, and it is often not memcached itself. Network saturation, swap, a single saturated worker thread, hash table expansion, and large values are the usual server-side culprits. Client-side GC pauses, DNS resolution, and connection setup add their own latency on top.

The diagnostic split that matters first: latency spikes that appear simultaneously across all clients point to the server or the network. Spikes isolated to one client point to that client’s runtime, connection handling, or GC. Establish that distinction before chasing daemon counters.

What it means

“High memcached latency” is almost always a client-reported symptom. The daemon keeps incrementing cmd_get, get_hits, and bytes_written normally even while individual operations queue behind a saturated worker thread or a swapped page. The absence of a server-side latency signal is not a monitoring gap you can close with more stats polling; it is architectural.

Your investigation has three layers:

Client measurement. Histogram the actual round-trip time of get and set operations in the application or client library. Without this, you are guessing.
Server-side saturation. CPU per worker thread, NIC bytes, swap, hash table expansion, connection queuing.
OS and network. NIC ring drops, retransmits, swap, noisy neighbors, TCP keepalive killing idle connections.

flowchart TD
    A[Client reports high latency] --> B{Spike across all clients?}
    B -- Yes --> C[Server or network]
    B -- No, one client --> D[Client runtime]
    C --> C1[NIC saturation]
    C --> C2[Swap on memcached]
    C --> C3[Worker thread pegged]
    C --> C4[Large value responses]
    D --> D1[GC pauses]
    D --> D2[DNS or connection setup]
    D --> D3[Client retry storms]

Common causes

Cause	What it looks like	First thing to check
NIC saturation	`bytes_written` rate approaching link speed; latency rises uniformly across all clients and all operations; OS NIC counters show TX drops	`ip -s link show <iface>`
Swap on the memcached process	Any nonzero `VmSwap`; random latency spikes on items whose pages were swapped; often dismissed because “only a few MB”	`grep VmSwap /proc/$(pgrep -x memcached \| head -1)/status`
Single saturated worker thread	Aggregate CPU looks fine (e.g., 40% of 4 cores), but a fraction of connections see high latency; `conn_yields` climbing	per-thread CPU; `conn_yields` rate
Hash table expansion	`hash_is_expanding = 1` during the spike; temporary memory and CPU overhead on the maintenance thread	`echo "stats" \| nc localhost 11211 \| grep hash_`
Large value responses	`bytes_written / cmd_get` ratio well above historical average; multiget batches amplifying	`stats items` per-class; average item size trend
Connection queuing	`curr_connections` near `-c` limit; `accepting_conns` flipping to 0; clients retrying	`curr_connections / max_connections` ratio
Client GC / runtime pauses	Latency spike on one client only; correlates with client GC or scheduling events, not server signals	client-side runtime metrics
DNS resolution at client	Spikes correlated with DNS TTL refresh; clients connecting by hostname	client DNS cache; switch to IP or local resolver

Quick checks

Run these read-only. None of them mutate cache state. Replace <iface> with the host’s memcached-facing interface; pick the correct PID if you run more than one daemon instance.

# Confirm the daemon is responsive at all
echo "version" | nc -w 2 localhost 11211

# Crude single-op latency probe (includes nc startup + connect; not a substitute for client histograms)
time (echo -e "get __latency_probe__\r" | nc -w 2 localhost 11211 > /dev/null)

# CPU consumed by the process (cumulative - sample twice for a rate)
echo "stats" | nc localhost 11211 | grep "STAT rusage_"

# Connection fairness: a climbing rate means a connection is being forced to yield
echo "stats" | nc localhost 11211 | grep "STAT conn_yields"

# Hash table expansion state
echo "stats" | nc localhost 11211 | grep -E "STAT hash_(is_expanding|power_level|bytes)"

# Network I/O volume
echo "stats" | nc localhost 11211 | grep -E "STAT bytes_(read|written)"

# Connection pressure
echo "stats" | nc localhost 11211 | grep -E "STAT (curr_connections|max_connections|accepting_conns)"

# Swap on the memcached process - must be zero
grep VmSwap /proc/$(pgrep -x memcached | head -1)/status

# OS-level NIC saturation and drops
ip -s link show <iface>

# Which clients hold the most connections (peer address:port is column 5; IPv6 peers parse differently)
ss -tn | grep ":11211" | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn

If VmSwap is nonzero, stop here. That is the incident. An in-memory cache with any data on disk is running at disk-speed access for the swapped regions, and no amount of daemon tuning will fix it.

How to diagnose it

Confirm the latency is real and measured client-side. If you do not have client-side histograms yet, you are working from application timeouts or APM spans. Wrap the memcached client call with a timing histogram (OpenTelemetry, Micrometer, or a client library that exposes its own). Record operation type, target node, and percentile buckets down to 100us. Without this, you cannot distinguish a 2ms server problem from a 50ms client GC pause.
Determine the blast radius. Pull latency percentiles per client instance and per target memcached node.
- Spikes on every client hitting the same node: server or network on that node.
- Spikes on one client across all nodes: that client’s runtime or network path.
- Spikes on one client hitting one node: usually a hot key or a client-library issue on that path.
Check swap first. VmSwap must be zero. Even a few MB causes random multi-millisecond spikes that look exactly like a server-side latency problem. If swap is present, the fix is host memory pressure, not memcached tuning. Lock memory with -k (which calls mlockall) where the capability allows it.
Check NIC saturation. Compare bytes_written rate against link capacity. Above 70% sustained is concerning; above 85% produces latency spikes and packet loss. Cross-check with OS NIC counters: if the OS shows saturation but memcached bytes_written looks modest, another process on the host is using the NIC. Large values and multiget batches are the usual amplifier: one multiget for 100 keys at 10KB each is a 1MB response.
Check for a single saturated worker thread. Memcached distributes connections round-robin across -t worker threads (default 4). Aggregate CPU can look fine while one thread is pegged and the connections assigned to it queue. conn_yields climbing means the -R fairness mechanism (default 20 requests per event) is actively throttling a connection that is sending large pipelines. If one client is dominating, reduce its batch sizes first, then consider raising -R.
Check hash table expansion. hash_is_expanding = 1 is normal as the cache fills, but it runs on a background thread and competes for CPU. If expansion coincides with latency spikes and persists for minutes on a very large item count, it is a contributor. The hash table never shrinks, so once expansion completes it will not recur unless item count grows again.
Check connection queuing. If curr_connections is near max_connections and accepting_conns has flipped to 0, new connections are being refused and existing clients are retrying, which manifests as latency and timeouts on the client even though the daemon is serving established connections.
Check response buffer pressure. response_obj_oom counts connections closed because an internal response buffer could not be allocated. It is separate from item storage memory. Raising maxconns without enough memory for connection buffers can trigger this.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Client-observed p50/p99 (external)	The only true latency signal; the daemon provides none	p99 above 2ms on a LAN sustained
`VmSwap` for the memcached process	Any swap turns an in-memory cache into a disk-speed cache	Any nonzero value
`bytes_written` rate vs NIC capacity	Large-value reads saturate the link before CPU or memory	Sustained above 70% of link speed
`conn_yields` rate	One client’s pipeline is starving others	Sustained above 1% of total command rate
`rusage_user` + `rusage_system` rate	Aggregate CPU; compute per-thread for hot-thread detection	Approaching worker thread count capacity
`hash_is_expanding`	Background expansion competes for CPU and memory	Persisting more than a few minutes on large caches
`curr_connections / max_connections`	Queuing begins near the limit	Ratio above 0.8 sustained
`response_obj_oom`	Response buffer exhaustion kills connections	Any sustained nonzero rate
`direct_reclaims`	Worker threads doing eviction work the background thread should handle	Any sustained nonzero rate

Fixes

Swap on the memcached process

This is a host memory problem, not a memcached problem. Free memory on the host or move co-located processes off. Lock memcached memory with -k (calls mlockall), which requires CAP_IPC_LOCK or root. Verify VmSwap returns to zero and stays there. Even small swap values are a production incident for a cache.

NIC saturation

Reduce value size: enable compression in the client library for large objects. Reduce multiget batch sizes. If the working set legitimately requires large values, spread the read load across more memcached instances or upgrade the NIC. Confirm the fix by watching bytes_written rate and OS NIC drop counters together.

Single saturated worker thread

If conn_yields is high and one client is dominating, reduce that client’s pipeline depth first. Raising -R lets a single connection send more requests per event loop iteration, which helps that client but can starve others, so tune conservatively. Increasing -t (worker threads) spreads connections across more threads. Default is 4; high thread counts can introduce lock contention, so test before raising significantly.

Hash table expansion

Expansion is a one-time cost as item count grows. If it is causing sustained latency on a very large cache, pre-sizing the hash table at startup avoids runtime expansion. . Stay current: versions before 1.6.14 had an integer overflow in hash table size calculation on servers with more than 1B items, which caused runaway CPU during expansion.

Large value responses

Check stats items for the slab classes holding the largest chunks and correlate with get_hits per class. If a small number of large values dominate bytes_written, compress them, split them, or move them out of memcached. The -I flag controls max item size (default 1MB).

Connection queuing

If curr_connections is at the limit, the real fix is client-side pooling, not just raising -c. Non-persistent connections create rapid churn and TIME_WAIT buildup. Raising -c also raises memory for connection buffers, which can trigger response_obj_oom if pushed too high without corresponding memory.

Client-side causes

If the blast radius is one client, look at its runtime. GC pauses, DNS resolution, and connection setup all add latency that has nothing to do with the daemon. TCP keepalive settings matter: if a load balancer or firewall kills idle connections, the next request on a dead connection fails and the client retries, which looks like latency. Client libraries should set keepalive and handle reconnection explicitly.

Prevention

Instrument client-side latency histograms before you need them. This is the single highest-leverage action. You cannot diagnose a latency incident without the data, and adding the instrumentation mid-incident biases the measurement.
Lock memory and monitor VmSwap. Treat any nonzero swap as a page-worthy incident for a cache host.
Track bytes_written against NIC capacity, not just as a counter. The ratio to cmd_get (average response size) is the leading indicator for large-value drift.
Watch conn_yields as an early signal of pipeline imbalance, before it becomes client-visible latency.
Keep the daemon on a current release. The 1.6.x line has fixed multiple crash and correctness bugs, including the hash table overflow on very large caches (1.6.14) and SASL timing side-channel fixes (1.6.42).
Separate daemon health from client experience. A daemon with a good hit ratio and low CPU can still deliver bad latency if the NIC is saturated or the client is swapping.

How Netdata helps

Netdata surfaces the server-side and OS-level signals you need to localize a client-reported latency problem without polling stats by hand.

Per-second collection of rusage_user, rusage_system, bytes_read, bytes_written, conn_yields, curr_connections, and hash_is_expanding shows the second a saturation event begins and lets you correlate it with the client-reported latency spike.
Process-level memory and swap metrics for the memcached process surface a nonzero VmSwap immediately, which is the highest-priority cause to rule out.
NIC-level counters (drops, retransmits, TX/RX bytes) sit alongside bytes_written, so you can tell whether a latency spike is memcached or the network without switching tools.
Per-slab eviction and age signals distinguish a latency spike caused by direct reclaims and LRU maintainer fallback from one caused by network saturation.
Anomaly detection on the rate-of-change of conn_yields, bytes_written, and connection count can flag a single client’s pipeline imbalance or a large-value drift before it crosses an absolute threshold.

Client-side latency histograms remain something you instrument in the application or client library. Netdata gives you the server and OS half of the correlation.