Memcached conn_yields rising: one client's pipeline starving the others

conn_yields is climbing on a memcached instance. The name suggests thread contention or a locking bug. It is neither.

conn_yields is memcached’s per-connection fairness throttle. Each worker thread processes requests from its assigned connections in a libevent loop. The -R flag (default 20, since memcached 1.4.0) caps how many sequential requests a worker pulls from a single connection in one event loop pass. When a connection hits that cap, the worker yields it: moves it to the back of the processing queue, serves other connections, then comes back. The conn_yields stat increments on each yield.

This protects other clients assigned to the same worker from one noisy connection. A sustained high rate means that protection is firing constantly: one client is sending bursts large enough to monopolize a worker thread. The symptom on other clients is elevated and inconsistent latency, not errors. At high enough rates, client libraries may interpret the latency as a dead connection, triggering timeouts and reconnection storms that compound the problem.

What this means

Memcached distributes connections round-robin across worker threads (default 4 via -t). Each worker runs a single-threaded libevent loop. Without a per-connection cap, one client sending a massive pipeline of commands would consume an entire event loop pass, and every other connection assigned to that worker would wait behind it.

The -R limit is a fairness mechanism, not a performance limiter. The noisy connection is not penalized beyond the requeue. It eventually gets its full request stream processed, just interleaved with everyone else.

flowchart TD
    A["Burst of pipelined requests
in one read buffer"] --> B{"Request count vs -R
default 20"}
    B -->|"below limit"| C["Process all requests
send responses"]
    B -->|"at or above limit"| D["Yield connection"]
    D --> E["conn_yields increments"]
    D --> F["Move to back of queue"]
    F --> G["Serve other queued connections"]
    G --> H["Resume yielded connection
on next event loop pass"]
    H --> B

Two details matter for diagnosis:

Binary protocol changes the shape of the problem. With the binary protocol, commands tend to arrive split across separate packets rather than batched in a single read buffer. This makes it harder to accumulate enough requests in one pass to trigger the limit. Clients using the text protocol and pipelining aggressively are more likely to produce yields. The binary protocol implementation is deprecated as of 1.6.0 but remains supported; meta commands are ASCII additions rather than a replacement implementation.

Multiget is one request, not many. A single multiget for 1000 keys counts as one request against the -R limit, not 1000. TCP may deliver a large ASCII multiget in multiple reads, but the parser waits for the complete command line before running it once. Binary protocol has no multiget command, so a client that “multigets” there issues separate GET requests, each counted once. What triggers yields is many top-level commands arriving for one connection within an event-loop pass.

Common causes

Quick checks

All commands are read-only and safe to run in production.

# Check the current conn_yields counter (cumulative since process start)
echo "stats" | nc -q1 localhost 11211 | grep "STAT conn_yields"

# Sample twice to compute the per-second rate
A=$(echo "stats" | nc -q1 localhost 11211 | awk '/STAT conn_yields/{print $3}')
sleep 5
B=$(echo "stats" | nc -q1 localhost 11211 | awk '/STAT conn_yields/{print $3}')
echo "conn_yields/sec: $(( (B - A) / 5 ))"

# Check overall command rates for the ratio comparison
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT cmd_(get|set) "

# Check the configured -R value (absent means default of 20)
cat /proc/$(pgrep -x memcached | head -1)/cmdline | tr '\0' ' ' | grep -oE '\-R [0-9]+'

# Count open connections by source IP to find the noisy client
ss -tn | grep ":11211" | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn

# Check worker thread count (default 4)
cat /proc/$(pgrep -x memcached | head -1)/cmdline | tr '\0' ' ' | grep -oE '\-t [0-9]+'

# Verify the process is responding (not a hung-process issue)
echo "version" | nc -q1 localhost 11211

# Check CPU utilization
echo "stats" | nc -q1 localhost 11211 | grep "STAT rusage_"

On managed platforms like AWS ElastiCache, you cannot access the process command line. The -R value is exposed as the requests_per_event parameter in the parameter group.

How to diagnose it

1. Confirm the rate is actually high. The conn_yields counter is cumulative since process start. An instance running for weeks with 50 million total yields is not necessarily in trouble. Compute the rate over a 5-second window. The threshold for investigation is sustained conn_yields exceeding roughly 1% of the cmd_get + cmd_set rate, or sustained above 100 per second.

2. Compute the ratio. A cache handling 500,000 ops/sec with 200 yields/sec is at 0.04%. That is fine. A cache handling 5,000 ops/sec with 200 yields/sec is at 4%. That is a problem. The ratio matters more than the absolute number.

3. Identify which clients dominate connections. Memcached does not expose per-client command counts or source IPs in its stats. Use OS-level tools. Run ss -tn | grep ":11211" and count connections by source IP. A single host holding a disproportionate share of connections is your primary suspect.

4. Correlate with deploys and batch schedules. If conn_yields started climbing at a specific time, check whether any application deployed a change around then. A new feature that batches cache writes, or a batch job moved to a tighter schedule, are common triggers.

5. Check the -R setting. If -R is explicitly set low (for example, someone copied an old config with -R 5), the threshold may be too aggressive for a legitimate high-throughput client. The default of 20 is appropriate for most workloads.

6. Check worker thread count. With the default 4 worker threads and many client connections, each worker handles a fraction of the connection pool. A noisy client assigned to one worker starves the other connections on that same worker. More worker threads (via -t) distribute connections more thinly, reducing the blast radius of a single noisy client. This requires a restart and should be tested under load before production rollout.

7. Rule out other causes of latency. Before attributing client-visible latency to conn_yields, verify the instance is not also experiencing CPU saturation (rusage approaching worker thread capacity), network bandwidth saturation (bytes_written approaching NIC capacity), or swap usage (VmSwap in /proc/<pid>/status). These produce similar symptoms but require different fixes.

Metrics and signals to monitor

Fixes

Reduce the noisy client’s pipeline depth or batch size

This is the root-cause fix. If one application instance is pipelining thousands of commands per send, cap the pipeline depth in the client library. Most memcached client libraries expose a connection-level pipeline limit or a batch size setting. The goal is to keep the number of commands landing in a single TCP read buffer at or below the -R threshold per event loop pass.

This requires identifying which client is responsible and coordinating a configuration change on the application side. It does not require a memcached restart.

Increase -R

If the workload is legitimately high-throughput and the yields come from a client that needs to send large pipelines, raising -R reduces the frequency of yields. This is a band-aid, not a fix: it lets one client consume more of each event loop pass, which increases latency for other connections on the same worker.

Changing -R requires a memcached restart, which means total cache data loss and a cold-start thundering herd on your backend. Plan this carefully. If the instance is part of a sharded cluster, drain it from the consistent hashing ring before restarting.

On AWS ElastiCache, modify the requests_per_event parameter in the parameter group and apply it during a maintenance window.

Redistribute the noisy client across more instances

If the noisy client writes to a sharded cluster, ensure its key distribution spreads evenly across nodes. A client whose keys all hash to one node will produce yields on that node while others sit idle. This is a client-side consistent hashing configuration issue, not a memcached server issue.

Fix application tight-loop bugs

If a recent deploy introduced a tight loop that issues cache requests without batching or rate limiting, the fix is in the application code. The signal is a cmd_get or cmd_set spike from one client that correlates exactly with the conn_yields increase.

Prevention

Monitor the yield ratio, not the absolute counter. Alert on rate(conn_yields) / (rate(cmd_get) + rate(cmd_set)) exceeding 0.01 sustained over 5 minutes. This adapts to your workload volume and avoids false positives on high-throughput instances.

Set client-side pipeline limits proactively. Review client library defaults for pipeline depth and batch size before they become a problem. A client that sends 500 commands per pipeline is fine on a quiet instance but will dominate a shared worker under load.

Document which batch jobs talk to memcached. Scheduled jobs that bulk-load or bulk-read cache data are the most common source of sudden conn_yields spikes. Their schedule should be known to the on-call engineer.

Track connection distribution across source IPs. Periodically sample ss -tn output to establish which hosts are the heaviest connection consumers. When yields spike, you already know where to look.

How Netdata helps

• Netdata collects conn_yields at per-second resolution, so you see the rate shape immediately rather than inferring it from sparse polling. The spike pattern (sustained versus bursty) tells you whether the cause is a steady pipeline or a scheduled batch job.
• The cmd_get and cmd_set rates are collected alongside conn_yields, so the yield ratio is visible without manual calculation. A dashboard panel showing both overlaid makes it obvious whether yields are proportional to load or anomalous.
• ML-based anomaly detection flags unusual conn_yields behavior even when the absolute rate is low, which catches problems before they cross a static threshold.
• CPU metrics (rusage_user, rusage_system) are correlated in the same timeline, so you can distinguish yield-induced latency from CPU saturation.
• Per-second memcached metrics are viewable alongside system-level network and CPU collectors, letting you correlate yields with OS-level signals in a single view.

Related guides

• How Memcached actually works in production: a mental model for operators
• Memcached connection limit reached: accepting_conns=0 and clients being refused
• Memcached connection refused: telling a dead process from a hung or full one
• Memcached high miss rate: separating cold start, new key patterns, and memory pressure
• Memcached hit ratio dropping: reading get_hits, get_misses, and cache effectiveness
• Memcached memory utilization: bytes vs limit_maxbytes and why the global number lies