Memcached RSS above the -m limit: connection buffers, hash table, and fragmentation

Operators set -m to cap cache memory, then notice RSS is 40 percent or more above that number. This is not a leak. The -m flag bounds only the slab allocator, the pool that stores cached items. RSS also includes the hash table, per-connection buffers, worker thread stacks, and allocator overhead.

The bytes / limit_maxbytes gauge in memcached stats describes cache memory: the keys and values your application stores. RSS, reported as VmRSS in /proc/<pid>/status, describes the entire process. They measure different scopes and will almost never match.

Expect roughly 1.4x the -m limit under normal conditions. RSS above approximately 1.6x signals investigation: a high connection count consuming buffer memory, an oversized or mid-expansion hash table, or glibc malloc fragmentation in a long-running process. RSS approaching host or cgroup limits risks the OOM killer terminating memcached entirely.

What RSS contains beyond the slab allocator

The -m flag controls one component: the slab allocator. At startup, memcached divides this memory into 1 MB pages, assigns pages to slab classes, and stores items in class-specific chunks. The bytes stat tracks how much is in use. The limit_maxbytes stat reflects the configured ceiling. Everything beyond that is overhead the -m flag does not bound.

flowchart TD
    RSS["Process RSS (VmRSS)"]
    Slabs["Slab cache
bounded by -m"]
    Hash["Hash table
hash_bytes"]
    Conn["On-demand network buffers"]
    Threads["Thread stacks
worker + maintenance"]
    Frag["Allocator overhead
glibc fragmentation"]

    RSS --> Slabs
    RSS --> Hash
    RSS --> Conn
    RSS --> Threads
    RSS --> Frag

Slab cache memory. Bounded by -m. This is what bytes and limit_maxbytes report. With -L (large pages), memcached pre-allocates the full -m at startup. Without -L, it allocates on demand as items arrive, so RSS grows as the cache fills.

Hash table. Tracked by the hash_bytes stat. Keys live in an expandable hash table sized as a power of two (hash_power_level). The table never shrinks, even if item count drops. When it grows, a background thread performs incremental expansion while both old and new tables exist simultaneously. During that window, hash_bytes temporarily doubles. Expansion runs on a dedicated thread and does not block request processing, but the memory spike is real.

Connection buffers. Since 1.6.0, network buffers are allocated on demand instead of being assigned statically to every client. An idle client connection uses roughly 400-500 bytes rather than a fixed 10 KB; active requests allocate read and response buffers as needed. That memory is outside the slab allocator and does not appear in bytes or limit_maxbytes. The default connection limit (-c 1024) bounds idle overhead naturally, but high connection counts and stalled pipelines can still make buffer memory material.

Thread stacks. Memcached runs a main listener thread, -t worker threads (default 4), and several maintenance threads (LRU crawler, slab automover, hash table expander). Each carries a stack. On default configurations this is minor, but high -t values add up and each resident stack page counts toward RSS.

Allocator overhead. When memcached is linked against glibc malloc (the default on most Linux distributions), long-running processes accumulate fragmentation. The allocator requests memory via brk and mmap, serves slab and internal allocations from that arena, and frequently cannot return freed memory to the OS. RSS creeps upward over days or weeks even when the working set has not changed. This is a well-known glibc malloc behavior for long-lived daemons, not a memcached-specific defect.

How the contributors scale

Each overhead component scales differently, which is why RSS can jump suddenly or drift slowly.

Hash table growth and the expansion spike

The hash table grows in powers of two as item count increases. Expansion triggers when the load factor crosses a threshold, and hash_is_expanding flips to 1 during the process. The expansion is incremental with fine-grained locking, so requests are never blocked. But both tables exist in memory for the duration, and hash_bytes is effectively doubled.

For a cache with millions of items, the hash table itself can be tens of megabytes. A doubling event adds that much again on top of slab memory. If the process is already close to a memory ceiling, the expansion spike can push it over.

The hash_power_level stat reports the current table size as 2^N. It only increases; the table never shrinks. A cache that once held 50 million items but now holds 5 million still carries the full hash table from its peak. To pre-size the table and avoid runtime expansion spikes, memcached 1.4.8 and later support -o hashpower=N, which sets the initial power level at startup.

# Check hash table state
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT hash_(is_expanding|power_level|bytes)"

Connection buffer scaling

Since 1.6.0, idle connection overhead is roughly 0.4-0.5 KB per connection, not a fixed 10 KB. Active connections vary with request size, pipelining depth, and how long responses wait to be written. Read the daemon-side gauges before sizing:

# Inspect allocated and in-use network buffer memory
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT (read_buf_count|read_buf_bytes|read_buf_bytes_free|response_obj_count|response_obj_bytes|response_obj_oom)"
echo "stats settings" | nc -q1 localhost 11211 | grep -E "(read_buf_mem_limit|resp_obj_mem_limit)"

At very high connection counts, memory can still exceed the cache on small instances. response_obj_oom means the server closed connections because it could not allocate response buffer objects. If you raise -c significantly, measure buffer use under production-shaped load and budget for it in the host or cgroup limit.

glibc fragmentation over time

The slowest contributor. A freshly started memcached has clean allocator state. Over weeks of operation, the pattern of allocations and frees (items set and evicted, connections opened and closed, hash table expansions) fragments the glibc heap. The allocator holds freed regions internally but cannot return them to the kernel because they are interleaved with still-used regions.

The result is RSS that never decreases even when curr_items drops or flush_all is issued. The slab allocator returns pages to its own pool on eviction, but glibc malloc does not necessarily release those pages back to the OS. RSS that climbed to 1.8x over a month, with no corresponding increase in bytes or connection count, is typically fragmentation.

Two mitigations exist. Linking memcached against jemalloc or tcmalloc at build time provides allocators that return memory more aggressively. Setting MALLOC_ARENA_MAX in the environment limits the number of glibc allocation arenas, reducing per-thread fragmentation at the cost of some lock contention. Neither changes memcached’s behavior; they change how the underlying allocator manages the heap.

When RSS is expected vs when to investigate

Expected overhead is approximately 30 to 40 percent above -m, accounting for the hash table, connection buffers, and internal structures. The ratio RSS / (limit_maxbytes * 1.4) should be close to 1.0.

Normal: RSS between 1.0x and 1.4x of -m. The slab cache dominates and overhead is proportionate.

Expected but worth noting: RSS between 1.4x and 1.6x. Common for instances with many connections, large item counts (big hash table), or moderate fragmentation. Not necessarily a problem, but worth identifying which contributor is responsible.

Investigate: RSS above 1.6x of -m. Something is consuming more memory than the typical overhead profile. Check connection count, hash table size, and process age. If none explain it, fragmentation is the likely culprit.

Dangerous: RSS approaching the host or cgroup memory limit. The OOM killer will terminate memcached if it cannot reclaim memory elsewhere. A memcached kill means total cache loss and a cold-start thundering herd on the backend. The -k flag enables mlockall to prevent swapping but does not protect against OOM kills from cgroup limits.

# Compare cache gauge to process RSS
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT (bytes |limit_maxbytes)"
PID=$(pgrep -x memcached | head -1)
grep -E "Vm(RSS|Size|Swap)" /proc/$PID/status

Where this shows up in production

Container and Kubernetes memory limits. The most common incident pattern: operators set a pod memory limit to -m plus 10 percent headroom, assuming RSS tracks limit_maxbytes closely. The container gets OOM-killed within hours or days. Size the cgroup limit for the full RSS envelope: -m plus hash table overhead, connection buffer overhead at expected peak connections, and a fragmentation buffer for long-running processes. A practical starting point is -m * 1.5 for the cgroup limit, adjusted upward for high connection counts.

Long-running bare-metal or VM deployments. RSS drifts upward over weeks. Operators may not notice until monitoring alerts on system memory pressure or the OOM killer fires during an unrelated traffic spike. The drift is fragmentation, not a leak: bytes and curr_connections are stable, but VmRSS keeps climbing. A planned restart with a cache warming strategy resets allocator state but causes temporary cold-cache impact. Migrating to jemalloc eliminates the drift without restarts.

High-connection-count deployments. Applications with many instances, each maintaining its own connection pool, can push curr_connections into the tens of thousands. Each connection adds buffer overhead outside the slab allocator. Idle 1.6.x connections add roughly 0.4-0.5 KB each; active and stalled connections can add much more. Measure read_buf_bytes and response_obj_bytes before converting connection counts into a fixed RSS allowance.

Hash table expansion during cache warming. After a restart, the cache fills from empty. Item count climbs rapidly, potentially triggering multiple hash table expansions in succession. Each expansion temporarily doubles hash_bytes. If the host is sized tightly, this post-restart spike can cause an OOM kill before the cache finishes warming.

Signals to watch

How Netdata helps

• Per-second RSS tracking alongside bytes and limit_maxbytes surfaces the gap between cache memory and process memory without manual polling.
• Correlating RSS with curr_connections at per-second resolution reveals whether connection buffer overhead is driving RSS growth.
• Hash table stats (hash_bytes, hash_is_expanding, hash_power_level) plotted against RSS make expansion spikes immediately visible. A sudden RSS jump coinciding with a hash table doubling event stands out.
• VmSwap monitoring for the memcached process catches any portion of the cache paged to disk.
• ML-based anomaly detection on RSS trends flags slow fragmentation drift before it reaches the OOM threshold.
• Cgroup memory pressure signals alongside process RSS provide early warning when RSS approaches the container or pod limit.

Related guides

• Memcached cache stampede: a hot key expires and the backend takes the hit
• Memcached evicted_unfetched and expired_unfetched: caching data nobody ever reads
• Memcached cas_badval climbing: check-and-set contention and lost updates
• Memcached command rate anomalies: cmd_get and cmd_set spikes and sudden drops
• Memcached conn_yields rising: one client’s pipeline starving the others
• Memcached connection churn: total_connections racing and TIME_WAIT buildup
• Memcached curr_connections climbing: connection leaks and missing pooling
• Memcached connection limit reached: accepting_conns=0 and clients being refused
• Memcached connection refused: telling a dead process from a hung or full one
• Memcached evicted_time low: distinguishing healthy turnover from cache thrash
• Memcached eviction cascade: when a full cache overloads the backend
• Memcached evictions climbing: the cache is full and discarding live data