$ guides / memcached / memcached-hash-table-expansion ▌

Operations Guides

Memcached hash table expansion: hash_is_expanding, memory spikes, and item churn

When you first see hash_is_expanding=1 in memcached stats, it looks like a warning. It is not. The daemon is doubling its internal key hash table to accommodate more items, a routine maintenance operation that runs on a dedicated background thread (since 1.4.0) and does not block request processing. It does temporarily spike memory and add CPU load on the maintenance thread, and in specific cases it can expose underlying problems: item-count oscillation, repeated expansion, or an overstretched memory budget.

This article covers what hash table expansion does, what a healthy expansion event looks like, and the three patterns worth investigating: expansion that persists for minutes, expansion that repeats hourly, and expansion whose memory spike pushes the process toward OOM.

What it is and why it matters

Memcached stores all keys in a single hash table. The table size is always a power of two, expressed as hash_power_level, where the bucket count is 2^hash_power_level. When the number of stored items exceeds 150% of the current bucket count, memcached doubles the table.

Three stats fields expose the table state (available since 1.4.8):

hash_is_expanding: boolean, set to 1 while expansion is in progress.
hash_power_level: current table power. Bucket count equals 2^hash_power_level.
hash_bytes: bytes of memory consumed by the hash table or tables.

Two properties are critical for operators:

The memory spike is real and predictable. Both old and new tables coexist until migration completes, so hash_bytes temporarily doubles.
hash_power_level only ever grows. The hash table never shrinks, even if item count later collapses to zero. An instance that once held 50 million items carries a hash table sized for 50 million items for the rest of its process lifetime.

How it works

Expansion triggers when curr_items exceeds (hashsize(hashpower) * 3) / 2 (150% load). At that point the assoc maintenance thread allocates a new table of double the current size and migrates buckets incrementally.

flowchart TD
    A["Normal: single table
2^N buckets"] --> B{"curr_items > 1.5x
bucket count?"}
    B -- no --> A
    B -- yes --> C["Allocate new table
2^(N+1) buckets"]
    C --> D["Migration phase:
old + new tables coexist
hash_bytes = sum of both"]
    D --> E{"All buckets
migrated?"}
    E -- no --> D
    E -- yes --> F["Swap pointers
Free old table"]
    F --> G["Normal: single table
2^(N+1) buckets"]
    G --> B

During migration, both the old and new tables exist simultaneously. hash_bytes reflects the sum of both. Lookups check both tables: if the item’s bucket in the old table has not yet been migrated, the old table is consulted; otherwise the new table holds the item. The maintenance thread migrates a batch of buckets per cycle. The MEMCACHED_HASH_BULK_MOVE environment variable controls the batch size; its default is one bucket per cycle.

Once all buckets are migrated, the old table is freed and hash_bytes drops to the new baseline (the doubled size). hash_is_expanding returns to 0.

The expansion runs on the assoc maintenance thread, not on worker threads. Worker threads continue serving requests during bucket migration. Starting in 1.4.22, upstream briefly pauses all threads while swapping to the completed table. On 1.4.21 and earlier, expect a brief stall correlated with the swap rather than with the bucket-copy work.

Where it shows up in production

Healthy expansion events are fast. You typically notice them only because you are graphing hash_bytes and see a step-function spike that resolves in seconds. Under normal conditions, expansion completes well under a minute.

Three patterns turn expansion into something worth investigating.

Expansion that persists for minutes

If hash_is_expanding stays at 1 for more than a minute or two, the maintenance thread is migrating buckets slower than new items arrive. This happens under sustained high write rates where the insertion rate outpaces the migration rate.

Two responses help:

Increase MEMCACHED_HASH_BULK_MOVE so the thread migrates more buckets per cycle. This trades a short-term CPU spike for faster completion. It is an environment variable set before daemon startup; changing it requires a restart.
Presize the hash table at startup with -o hashpower=N so the instance never needs to expand during peak write load. This also requires a restart.

Expansion that repeats hourly

hash_power_level only grows. If hash_is_expanding flips to 1 every hour or two, item count is oscillating across the 150% threshold. Items arrive in waves, push count above the trigger, then get evicted or expired back below it, then arrive again.

This is item churn, not a hash table problem. The root cause is upstream: a batch job flooding the cache on a schedule, an eviction pattern cycling working set size, or an application storing large numbers of short-lived items in bursts. Investigate curr_items rate-of-change and correlate with cmd_set rate and eviction rate.

The hash table cost of this pattern is cumulative and permanent. Each expansion raises hash_power_level and never lowers it. An instance that has expanded ten times carries a table ten doublings larger than its starting size, even if item count has since collapsed.

Memory spike that risks OOM

During expansion, hash_bytes doubles because both tables coexist. For a table at hash_power_level=24 (16,777,216 buckets at 8 bytes per pointer on 64-bit, or 128 MB per table), expansion temporarily requires 256 MB for the hash tables alone. This is overhead on top of item storage governed by -m.

If the process is already near its memory ceiling, the expansion spike can push RSS past safe limits. Check:

Process RSS vs system or cgroup memory. The hash table is part of process RSS. If RSS is already close to the limit, the doubling spike can trigger the OOM killer.
-m budget vs actual RSS. hash_bytes is overhead on top of slab-allocated item memory. RSS should be roughly limit_maxbytes plus hash_bytes plus connection buffer overhead. If you budgeted only for -m, you underestimated.
Container memory limits. The expansion spike counts against the cgroup memory limit. A process that fits at its current hash table size may breach the limit during expansion.

To avoid repeated expansion spikes during cache warming, set -o hashpower=N at startup. Memcached 1.4.8 introduced the option with a range of 12–64; current 1.5.10+ releases cap it at 12–32. Set hashpower so the table starts large enough that no expansion is needed. Note that hash_power_level cannot be set lower than the item lock table power; memcached will refuse to start if you set it too low.

When this matters

This signal is INFO severity in the signal catalog and appears at Level 3 monitoring maturity, alongside per-slab stats and LRU crawler effectiveness. Most teams never need to act on it. The cases where it warrants attention:

New deployment capacity planning. If you know your steady-state item count, presize with -o hashpower=N to avoid expansion spikes during cache warming.
Tight memory budgets. Account for the doubling of hash_bytes during expansion in your RSS ceiling.
Item churn diagnosis. Repeated expansion is a symptom. The cause is upstream in the application or eviction pattern.
Old versions. Before 1.4.22, hash-table swaps could stall all threads. Upgrade if you see micro-freezes correlated with expansion.

There is a -o no_hashexpand flag that disables expansion entirely. It is labeled “dangerous” in the memcached source: if the hash table fills and cannot grow, lookup performance degrades severely as bucket chains deepen. Do not use it unless you have presized the table to a level you are certain will never be exceeded.

Signals to watch in production

Signal	Why it matters	Warning sign
`hash_is_expanding`	Indicates active expansion	Persists longer than 60 seconds, or triggers more than once per hour
`hash_bytes`	Memory consumed by hash table(s)	Spikes to roughly 2x baseline and does not resolve
`hash_power_level`	Current table size as power of 2	Climbing continuously without item-count growth to justify it
`curr_items`	Working set size	Oscillating across the 150% expansion threshold
Process RSS	Total process memory including overhead	Expansion spike pushes RSS toward system or cgroup limit
`rusage_user`	Maintenance thread CPU work	Sustained spike during expansion that does not resolve after migration completes

How Netdata helps

Netdata collects hash_is_expanding, hash_bytes, and hash_power_level per second from the memcached stats interface. Per-second resolution captures the exact duration and frequency of expansion events, rather than missing short events between polls.
Correlating hash_bytes with process RSS in the same dashboard shows whether the expansion spike pushes memory toward a dangerous threshold or resolves cleanly.
Pairing hash_is_expanding with curr_items rate-of-change reveals item-count oscillation across the expansion threshold.
Correlating expansion events with cmd_set rate and eviction rate distinguishes normal growth-driven expansion from write-heavy churn.
ML-based anomaly detection can flag expansion patterns that deviate from the established baseline without requiring static thresholds.