Memcached network saturation: large values, multiget, and a maxed-out NIC

Memcached latency is climbing for every client, but the host looks healthy. CPU is well below saturation, memory utilization is far from the limit, evictions are zero, and curr_connections is well below -c. The daemon answers version and stats instantly. Yet every application reports slow gets, and the slowdown is uniform across keys and clients.

When latency degrades uniformly and the usual suspects are clear, look at the wire. Memcached pulls values from RAM and writes them to a socket. If the response stream exceeds what the NIC can push, the kernel queues, TCP congestion control engages, and every client waits. The cache is fine; the link is the bottleneck.

What this means

Network bandwidth saturation occurs when bytes_written (the cumulative bytes the daemon has pushed to the network) approaches NIC capacity and the kernel cannot drain the send queue as fast as memcached produces responses.

The problem is asymmetric. Memcached responses carry values; requests carry keys. For read-heavy workloads, bytes_written is typically much larger than bytes_read. A workload storing multi-KB values and reading them at even a moderate rate can push more bytes per second than the link can carry, while operations per second remains well below the daemon’s CPU ceiling.

Multiget amplifies the effect. A single multiget for 100 keys at 10 KB each produces a 1 MB response assembled and written as one logical operation. The command rate barely moves, but bytes_written spikes. A few thousand such multigets per second can saturate a 1 Gbps link.

The signature is uniform latency degradation. Every client, key, and operation slows down together. CPU and memory remain healthy. The only signals that move are the bytes_written rate and OS NIC TX counters. Past roughly 85% of link speed, packet drops and retransmissions begin, and latency spikes.

flowchart TD
    A[Large values stored in cache] --> B[High get rate or large multigets]
    B --> C[bytes_written rate climbs]
    C --> D[NIC TX approaches link speed]
    D --> E{Share of link speed}
    E -->|> 70% sustained| F[Latency rises uniformly]
    E -->|> 85%| G[Packet drops and retransmits]
    F --> H[All clients slow down together]
    G --> H
    H --> I[CPU and memory still look fine]

These are application-layer bytes. bytes_written excludes TCP/IP overhead, so actual NIC utilization runs 5-10% higher than the memcached counter suggests. Always confirm against OS-level NIC counters before concluding the link is the limit.

Common causes

Quick checks

All commands below are read-only and safe to run in production.

# Cumulative bytes the daemon has read and written
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT bytes_(read|written)"

# Command counters for computing average response size
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT cmd_(get|set)"

# OS NIC byte and error counters
cat /proc/net/dev

# Per-interface TX bytes, packets, drops, overruns
ip -s link show <interface>

# NIC link speed and duplex mode
ethtool <interface>

# Offload feature state (look for gro, lro, tso)
ethtool -k <interface>

# IRQ distribution for the NIC across cores
grep <interface> /proc/interrupts

# Slab class chunk sizes to see where large values land
echo "stats slabs" | nc -q1 localhost 11211 | grep chunk_size

Use -w 2 if -q1 is unavailable for nc (for example, on macOS or certain BSD-based netcat ports).

How to diagnose it

1. Confirm the symptom is uniform. Check whether latency rose across all clients and keys at the same time. Uniform degradation points to the server or network; per-client or per-key spikes point to the application. Memcached exposes no native latency histograms, so confirm this via client-side instrumentation.
2. Compute the bytes_written rate. Sample bytes_written twice with a known interval. Subtract and divide by the interval to get bytes per second. A single sample is cumulative since process start and useless for rate calculation.
3. Express the rate as a percentage of link speed. For a 1 Gbps link, full-duplex capacity is roughly 125 MB/s in one direction. Above 70% sustained (about 87 MB/s) is concerning; above 85% (about 106 MB/s), expect latency spikes and packet loss. Adjust the denominator for your actual link speed (10 Gbps is roughly 1.25 GB/s per direction).
4. Confirm against OS NIC counters. Pull TX bytes from /proc/net/dev or ip -s link show. The kernel counter includes all interface traffic. If the interface is near saturation but memcached’s bytes_written rate is modest, another process is using the NIC. If both agree, memcached is the dominant producer.
5. Check for drops and overruns. In the ip -s link output, watch TX drops and overruns. Any nonzero rate means the kernel ran out of ring buffer space. This is where latency stops climbing gradually and starts spiking.
6. Determine whether the cause is value size or batch size. Compute bytes_written / cmd_get over the same window. Memcached increments cmd_get per requested key, not per command, so this ratio accurately reflects the average response size per key. A rising ratio over days or weeks means values are inflating. A sudden jump means either new large values or larger multiget batches entered the workload.
7. Rule out NIC misconfiguration. Run ethtool <interface> to confirm the link negotiated at full duplex and expected speed. Half-duplex links cause throughput collapse. Run ethtool -k <interface> to confirm segmentation offloads (TSO, GRO, LRO) are enabled. Disabled offloads push per-packet processing to the CPU and reduce effective throughput.
8. Check IRQ distribution. If all NIC interrupts land on one core, that core bottlenecks before the link does. Inspect /proc/interrupts for the NIC’s queues. Uneven distribution combined with high rusage_system on one core suggests the kernel network stack (softirq or send processing) is the constraint, not the link itself. rusage_system is the memcached process’s system CPU: its own syscalls, including inline TCP send processing. On kernels with IRQ time accounting (CONFIG_IRQ_TIME_ACCOUNTING, enabled on most current distros), softirq processing is accounted system-wide instead — visible as %soft in mpstat or as ksoftirqd usage — so check both before concluding.

Metrics and signals to monitor

Memcached does not expose per-operation latency. The latency signal must come from the client. Without client instrumentation, the server-side signals above are your only diagnostic path.

Fixes

Reduce value size with client-side compression

Most memcached client libraries support transparent compression. The client compresses before set, sets a flag bit, and decompresses on retrieval. Text and binary payloads often compress 3x to 10x, which directly reduces bytes_written for the same cmd_get rate.

Compression adds CPU cost on the client, not the server. If your application servers have CPU headroom and the memcached host is network-bound, this is usually the highest-leverage fix. Measure compression ratios on your actual data before assuming a specific gain.

Shrink multiget batches

If clients issue multigets with dozens or hundreds of keys, splitting them into smaller batches reduces peak response size per operation. This does not reduce total bytes over time, but it smooths bursty transmission and gives the kernel more chances to drain the queue between responses.

This increases round trips per logical operation. It helps when the problem is burst-induced queue buildup, not sustained average throughput. If sustained throughput is the limit, smaller batches will not help.

Spread load across more instances

Adding memcached instances on different hosts distributes egress across multiple NICs. In a client-side sharded cluster, consistent hashing redistributes keys across the new nodes automatically. This is the right structural fix when a single instance is permanently over its link budget.

More hosts mean more operational overhead. Adding instances to the same host does not help because the bottleneck is the host’s NIC, not the memcached process. Sharding across more instances on one box only moves the queue from per-socket buffers to the kernel qdisc layer.

Upgrade the NIC

If the workload is legitimately large-value and the application cannot compress further, a faster link is the direct fix. Moving from 1 Gbps to 10 Gbps gives a 10x ceiling. At 400,000 operations per second with an average value size of 3072 bytes, even a 10 Gbps link saturates.

Confirm the kernel, PCIe, and interrupt handling can actually deliver the new link rate before paying for it. A 25 Gbps NIC on a host that cannot process interrupts fast enough buys nothing.

Verify NIC offloads and IRQ affinity

Before buying hardware, confirm the existing NIC is configured correctly. Distribute NIC receive and transmit interrupts across multiple cores. Misconfiguration can halve effective throughput.

Warning: ethtool -K changes live NIC settings and can cause a brief link flap or packet loss depending on the driver. Test during a maintenance window, not during an active incident.

# WARNING: disruptive on some drivers; verify current state with ethtool -k first
ethtool -K <interface> tso on gro on lro on

TSO is the most relevant offload for egress saturation. GRO and LRO are receive-side offloads. Offloads are the default on most distributions, but verify rather than assume. Some virtualization platforms and cloud providers expose limited offload control.

Prevention

• Track bytes_written as a percentage of link speed. Alert on sustained utilization above 70%. Express it as a rate, not a cumulative counter.
• Track the bytes_written / cmd_get ratio over time. A rising trend means average value size is inflating, even before the link saturates. Catch this in capacity planning, not during an incident.
• Capacity plan against link speed, not just operations per second. Memcached handles hundreds of thousands of operations per second on modern hardware, but a few thousand large-value multigets can saturate a 1 Gbps link. Ops/sec and bytes/sec are independent ceilings.
• Audit value sizes at the application layer. Serialized objects that grow over time (new fields, larger nested structures) silently push the workload toward the link limit. Review the largest cached objects during every major application release.
• Confirm NIC configuration after host provisioning. Duplex mismatches and disabled offloads are provisioning-time errors that persist until someone investigates a slowdown. Build ethtool checks into your host bootstrap.

How Netdata helps

• Per-second bytes_written and bytes_read collection from the memcached stats endpoint, with rates computed automatically. Per-second resolution catches burst-induced saturation that 60-second polling misses.
• OS NIC TX metrics per interface, collected at the same per-second cadence. Correlating memcached bytes_written with kernel TX bytes in one view confirms whether the link is the limit or whether another process shares the NIC.
• Derived bytes_written / cmd_get tracking as a computed dimension, so average response size trends are visible without manual sampling.
• NIC drop and overrun counters surfaced alongside throughput, so the transition from gradual latency rise to packet loss is visible as it happens.
• Anomaly detection on the bytes_written rate and NIC TX rate, flagging deviations from the learned baseline.
• Client-side latency correlation when application instrumentation is available, confirming the uniform-degradation signature that distinguishes network saturation from CPU or memory pressure.

Related guides

• How Memcached actually works in production: a mental model for operators
• Memcached conn_yields rising: one client’s pipeline starving the others
• Memcached eviction cascade: when a full cache overloads the backend
• Memcached connection churn: total_connections racing and TIME_WAIT buildup
• Memcached high miss rate: separating cold start, new key patterns, and memory pressure