Operators set -m to cap cache memory, then notice RSS is 40 percent or more above that number. This is not a leak. The -m flag bounds only the slab allocator, the pool that stores cached items. RSS also includes the hash table, per-connection buffers, worker thread stacks, and allocator overhead.

The bytes / limit_maxbytes gauge in memcached stats describes cache memory: the keys and values your application stores. RSS, reported as VmRSS in /proc/<pid>/status, describes the entire process. They measure different scopes and will almost never match.

Expect roughly 1.4x the -m limit under normal conditions. RSS above approximately 1.6x signals investigation: a high connection count consuming buffer memory, an oversized or mid-expansion hash table, or glibc malloc fragmentation in a long-running process. RSS approaching host or cgroup limits risks the OOM killer terminating memcached entirely.

What RSS contains beyond the slab allocator

The -m flag controls one component: the slab allocator. At startup, memcached divides this memory into 1 MB pages, assigns pages to slab classes, and stores items in class-specific chunks. The bytes stat tracks how much is in use. The limit_maxbytes stat reflects the configured ceiling. Everything beyond that is overhead the -m flag does not bound.

flowchart TD
    RSS["Process RSS (VmRSS)"]
    Slabs["Slab cache
bounded by -m"]
    Hash["Hash table
hash_bytes"]
    Conn["Connection buffers
~10KB per conn"]
    Threads["Thread stacks
worker + maintenance"]
    Frag["Allocator overhead
glibc fragmentation"]

    RSS --> Slabs
    RSS --> Hash
    RSS --> Conn
    RSS --> Threads
    RSS --> Frag

Slab cache memory. Bounded by -m. This is what bytes and limit_maxbytes report. With -L (large pages), memcached pre-allocates the full -m at startup. Without -L, it allocates on demand as items arrive, so RSS grows as the cache fills.

Hash table. Tracked by the hash_bytes stat. Keys live in an expandable hash table sized as a power of two (hash_power_level). The table never shrinks, even if item count drops. When it grows, a background thread performs incremental expansion while both old and new tables exist simultaneously. During that window, hash_bytes temporarily doubles. Expansion runs on a dedicated thread and does not block request processing, but the memory spike is real.

Connection buffers. Each client connection consumes buffer memory outside the slab allocator, approximately 10 KB per connection. A server holding 5,000 idle connections allocates roughly 50 MB of non-cache memory that does not appear in bytes or limit_maxbytes. The default connection limit (-c 1024) bounds this naturally, but operators who raise -c without accounting for buffer overhead see RSS climb.

Thread stacks. Memcached runs a main listener thread, -t worker threads (default 4), and several maintenance threads (LRU crawler, slab automover, hash table expander). Each carries a stack. On default configurations this is minor, but high -t values add up and each resident stack page counts toward RSS.

Allocator overhead. When memcached is linked against glibc malloc (the default on most Linux distributions), long-running processes accumulate fragmentation. The allocator requests memory via brk and mmap, serves slab and internal allocations from that arena, and frequently cannot return freed memory to the OS. RSS creeps upward over days or weeks even when the working set has not changed. This is a well-known glibc malloc behavior for long-lived daemons, not a memcached-specific defect.

How the contributors scale

Each overhead component scales differently, which is why RSS can jump suddenly or drift slowly.

Hash table growth and the expansion spike

The hash table grows in powers of two as item count increases. Expansion triggers when the load factor crosses a threshold, and hash_is_expanding flips to 1 during the process. The expansion is incremental with fine-grained locking, so requests are never blocked. But both tables exist in memory for the duration, and hash_bytes is effectively doubled.

For a cache with millions of items, the hash table itself can be tens of megabytes. A doubling event adds that much again on top of slab memory. If the process is already close to a memory ceiling, the expansion spike can push it over.

The hash_power_level stat reports the current table size as 2^N. It only increases; the table never shrinks. A cache that once held 50 million items but now holds 5 million still carries the full hash table from its peak. To pre-size the table and avoid runtime expansion spikes, memcached 1.4.8 and later support -o hashpower=N, which sets the initial power level at startup.

# Check hash table state
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT hash_(is_expanding|power_level|bytes)"

Connection buffer scaling

Connection buffer memory scales linearly with curr_connections. At approximately 10 KB per connection, overhead is modest at default scale but significant at high counts:

curr_connectionsApproximate buffer overhead
100 (baseline)~1 MB
1,024 (default -c)~10 MB
5,000~50 MB
10,000~100 MB
50,000~500 MB

At 50,000 connections, buffer overhead alone can exceed the cache memory on small instances. The playbook warns that oversized connection pools without corresponding buffer memory can trigger response_obj_oom, where the server closes connections because it cannot allocate response buffer objects. If you raise -c significantly, budget for the buffer memory in your host or cgroup limit.

# Check connection count and configured limit
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT (curr_connections|max_connections)"

glibc fragmentation over time

The slowest contributor. A freshly started memcached has clean allocator state. Over weeks of operation, the pattern of allocations and frees (items set and evicted, connections opened and closed, hash table expansions) fragments the glibc heap. The allocator holds freed regions internally but cannot return them to the kernel because they are interleaved with still-used regions.

The result is RSS that never decreases even when curr_items drops or flush_all is issued. The slab allocator returns pages to its own pool on eviction, but glibc malloc does not necessarily release those pages back to the OS. RSS that climbed to 1.8x over a month, with no corresponding increase in bytes or connection count, is typically fragmentation.

Two mitigations exist. Linking memcached against jemalloc or tcmalloc at build time provides allocators that return memory more aggressively. Setting MALLOC_ARENA_MAX in the environment limits the number of glibc allocation arenas, reducing per-thread fragmentation at the cost of some lock contention. Neither changes memcached’s behavior; they change how the underlying allocator manages the heap.

When RSS is expected vs when to investigate

Expected overhead is approximately 30 to 40 percent above -m, accounting for the hash table, connection buffers, and internal structures. The ratio RSS / (limit_maxbytes * 1.4) should be close to 1.0.

Normal: RSS between 1.0x and 1.4x of -m. The slab cache dominates and overhead is proportionate.

Expected but worth noting: RSS between 1.4x and 1.6x. Common for instances with many connections, large item counts (big hash table), or moderate fragmentation. Not necessarily a problem, but worth identifying which contributor is responsible.

Investigate: RSS above 1.6x of -m. Something is consuming more memory than the typical overhead profile. Check connection count, hash table size, and process age. If none explain it, fragmentation is the likely culprit.

Dangerous: RSS approaching the host or cgroup memory limit. The OOM killer will terminate memcached if it cannot reclaim memory elsewhere. A memcached kill means total cache loss and a cold-start thundering herd on the backend. The -k flag enables mlockall to prevent swapping but does not protect against OOM kills from cgroup limits.

# Compare cache gauge to process RSS
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT (bytes |limit_maxbytes)"
PID=$(pgrep -x memcached | head -1)
grep -E "Vm(RSS|Size|Swap)" /proc/$PID/status

Where this shows up in production

Container and Kubernetes memory limits. The most common incident pattern: operators set a pod memory limit to -m plus 10 percent headroom, assuming RSS tracks limit_maxbytes closely. The container gets OOM-killed within hours or days. Size the cgroup limit for the full RSS envelope: -m plus hash table overhead, connection buffer overhead at expected peak connections, and a fragmentation buffer for long-running processes. A practical starting point is -m * 1.5 for the cgroup limit, adjusted upward for high connection counts.

Long-running bare-metal or VM deployments. RSS drifts upward over weeks. Operators may not notice until monitoring alerts on system memory pressure or the OOM killer fires during an unrelated traffic spike. The drift is fragmentation, not a leak: bytes and curr_connections are stable, but VmRSS keeps climbing. A planned restart with a cache warming strategy resets allocator state but causes temporary cold-cache impact. Migrating to jemalloc eliminates the drift without restarts.

High-connection-count deployments. Applications with many instances, each maintaining its own connection pool, can push curr_connections into the tens of thousands. Each connection adds buffer overhead outside the slab allocator. At 10 KB per connection, 10,000 connections costs 100 MB of non-cache RSS. On a cache sized at 512 MB (-m 512), that is 20 percent overhead from connections alone before the hash table or fragmentation is counted.

Hash table expansion during cache warming. After a restart, the cache fills from empty. Item count climbs rapidly, potentially triggering multiple hash table expansions in succession. Each expansion temporarily doubles hash_bytes. If the host is sized tightly, this post-restart spike can cause an OOM kill before the cache finishes warming.

Signals to watch

SignalWhy it mattersWarning sign
VmRSS (/proc/<pid>/status)Total process memory from the OS view. The number that triggers OOM kills.Exceeds 1.6x limit_maxbytes without explanation
bytes / limit_maxbytesCache memory gauge. Distinct from RSS.Does not move while RSS climbs (indicates overhead, not cache growth)
hash_bytesHash table memory consumption. Overhead on top of slab storage.Sudden doubling (expansion in progress) or large value relative to -m
hash_is_expandingWhether the table is mid-expansion. Both tables exist simultaneously.Stuck at 1 for extended periods
hash_power_levelCurrent table size as 2^N. Never decreases.High value relative to curr_items (table sized for a past peak)
curr_connectionsEach connection adds ~10 KB of buffer overhead outside slabs.Count disproportionate to expected client pool size
VmSwap (/proc/<pid>/status)Any swap for an in-memory cache is a performance catastrophe.Any non-zero value
Process uptimeLong uptime correlates with glibc fragmentation accumulation.RSS climbing over weeks with stable bytes and curr_connections

How Netdata helps

  • Per-second RSS tracking alongside bytes and limit_maxbytes surfaces the gap between cache memory and process memory without manual polling.
  • Correlating RSS with curr_connections at per-second resolution reveals whether connection buffer overhead is driving RSS growth.
  • Hash table stats (hash_bytes, hash_is_expanding, hash_power_level) plotted against RSS make expansion spikes immediately visible. A sudden RSS jump coinciding with a hash table doubling event stands out.
  • VmSwap monitoring for the memcached process catches any portion of the cache paged to disk.
  • ML-based anomaly detection on RSS trends flags slow fragmentation drift before it reaches the OOM threshold.
  • Cgroup memory pressure signals alongside process RSS provide early warning when RSS approaches the container or pod limit.