Memcached flush_all: the accidental cache wipe and its cold-start blast radius

You are paged because hit ratio collapsed and the backend database is saturating. The memcached process is up, uptime is stable, memory looks allocated, and there are no evictions. The signature is an instant hit ratio cliff with no gradual decline. The cause is almost certainly a flush_all command, and the cmd_flush counter will confirm it.

A single flush_all invalidates every item in the cache. It does not lock the server and does not require authentication by default. Anyone with TCP access to port 11211 can issue it. The result is a cold-cache thundering herd: every subsequent GET misses and hits the backend simultaneously. If the backend is sized for cache-assisted load, it just received a multiple of its designed capacity.

What this means

flush_all sets an internal invalidation timestamp. Every item created before that timestamp is considered invalid. The server returns immediately; since 1.6.27 it also walks the LRU lists non-blockingly and unlinks flushed items, so on current versions bytes and curr_items drop right away on an idle cache. Items the walk skips (a concurrent operation held the item lock) are lazily invalidated on next access: when a GET hits a flushed item, the server returns a miss, counts get_flushed, and the slot is reclaimed. Before 1.6.27, invalidation is purely lazy and bytes stays near-full until items are accessed or overwritten by new SET operations.

The critical consequence: the cache is effectively empty from the application’s perspective immediately, whatever bytes shows. On a busy server the flush walk skips locked items, so bytes can stay high for a while — but bytes is the wrong signal either way. Confirm the flush with cmd_flush.

The get_flushed counter (added in 1.4.28) is the direct impact measurement. It counts GET requests that encountered an item invalidated by flush_all — mostly items the flush walk skipped or a delayed flush’s pending invalidations on modern versions. A spike in get_flushed after cmd_flush increments confirms the flush is actively affecting production traffic.

A flush_all with a delay argument (flush_all 3600) makes diagnosis harder. The cmd_flush counter increments immediately when the command is received, but the invalidation timestamp is set one hour in the future. The cache continues serving normally until the delay expires, then all items become invalid at once. The gap between command and impact can be large enough that operators do not correlate them.

flowchart TD
    A["Hit ratio collapsed"] --> B{"cmd_flush increased?"}
    B -- Yes --> C{"uptime stable?"}
    B -- No --> D["Check evictions or restart"]
    C -- Yes --> E["flush_all confirmed"]
    C -- No --> F["Process restarted"]
    E --> G{"get_flushed rising?"}
    G -- Yes --> H["Thundering herd active"]
    G -- No --> I["Delayed flush or cold keys"]
    H --> J["Protect backend now"]

Common causes

Cause	What it looks like	First thing to check
Debug script or admin tool	Single cmd_flush increment, no recurring pattern	Shell history on the host, deploy logs, who had access
Application bug calling flush_all	cmd_flush increments at regular intervals or on specific events (deploys, cron)	Search application code and deployment scripts for flush_all calls
Unauthorized network access	cmd_flush increments from an unknown source	`ss -tn \| grep :11211` for unexpected source IPs
Deployment or automation tooling	cmd_flush increments consistently after deploys	CI/CD pipeline and infrastructure automation for cache-clear steps
flush_all with delay argument	cmd_flush increments but hit ratio stays normal, then collapses	Gap between increment and impact equals the delay value

Quick checks

# Check if cmd_flush has incremented
echo "stats" | nc -w 2 localhost 11211 | grep "STAT cmd_flush"

# Confirm uptime is stable (distinguishes flush from restart)
echo "stats" | nc -w 2 localhost 11211 | grep "STAT uptime"

# Measure the read-side impact of the flush
echo "stats" | nc -w 2 localhost 11211 | grep "STAT get_flushed"

# Check current hit ratio counters (compute ratio from deltas, not cumulative totals)
echo "stats" | nc -w 2 localhost 11211 | grep -E "STAT get_(hits|misses)"

# Verify memory is still allocated (bytes stays high after lazy invalidation)
echo "stats" | nc -w 2 localhost 11211 | grep -E "STAT (bytes |limit_maxbytes)"

# Check curr_items (drops as items are lazily invalidated)
echo "stats" | nc -w 2 localhost 11211 | grep "STAT curr_items"

# List active connections by source IP to identify potential flush source
ss -tn | grep ":11211" | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn

# Verify the process is responding
echo "version" | nc -w 2 localhost 11211

How to diagnose it

Confirm cmd_flush incremented. Any value above the previous baseline is the trigger. If cmd_flush has not changed, the hit ratio collapse has a different cause: eviction cascade, slab imbalance, or restart.
Check uptime to distinguish flush from restart. If uptime is low (seconds to minutes), the process restarted and the cache loss came from the restart, not a flush_all. An accidental flush_all is harder to detect because the process stays up and no crash log exists.
Measure impact with get_flushed. A rising get_flushed rate tells you how much production traffic is hitting invalidated items. If get_flushed is zero despite cmd_flush incrementing, the delay argument was used, the flushed items are not being actively requested, or the flush walk already reclaimed everything (1.6.27+ on an idle cache).
Check for a delay argument. If cmd_flush incremented but hit ratio is still normal, someone may have issued flush_all <delay>. The counter increments immediately but items remain valid until the delay expires. This is the hardest variant to catch because command and impact are separated in time.
Trace the source. Memcached does not attribute individual commands to source IPs: -vv logging shows commands against connection numbers only. Recent versions (1.6.14+) can log connection open/close events with client addresses via watch connevents, but still cannot tie a specific command to a source. Check shell history on the memcached host, CI/CD pipeline logs, deployment automation, and application code. For recurring sources, capture future flush_all commands with tcpdump on port 11211 (requires root or CAP_NET_RAW).
Assess backend impact. The real damage is the thundering herd on the backend. Check backend load, connection counts, and latency. If the backend is saturating, the priority shifts from finding the source to protecting the backend.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
cmd_flush	The trigger counter. Any increment in production is significant.	Any increase from previous value
get_flushed	Direct measurement of flush impact. Counts GETs that hit invalidated items.	Non-zero rate after cmd_flush increment
uptime	Distinguishes flush_all from process restart.	Sudden drop means restart, not flush
get_hits / get_misses	Hit ratio collapse is the primary symptom. Compute from deltas.	Instant drop to near 0%, not gradual decline
bytes	May stay allocated after flush (skipped items; always, before 1.6.27). Wrong signal for detection.	Stays high despite cache being effectively empty
curr_items	Drops as items are lazily invalidated or overwritten.	Sudden decline correlating with cmd_flush
Backend load	The real victim. Backend receives full production load when cache is cold.	Spike correlated with hit ratio collapse
curr_connections	May spike during thundering herd as clients retry or open new connections.	Sudden increase above baseline

Fixes

During the incident: protect the backend

The cache will warm itself as the application writes miss results back. The question is whether the backend survives the warming period. If backend latency is climbing or connections are saturating:

Enable circuit breakers in the application layer to limit concurrent backend queries. Most overload comes from thousands of simultaneous misses for the same popular keys.
Run cache warming scripts if available. Pre-populating the most-requested keys reduces the miss storm.
Rate-limit non-critical traffic temporarily. If the backend is at capacity, shedding load is safer than allowing cascading failure.

Do not restart memcached. A restart does not help (the cache is already cold) and loses items re-cached since the flush.

After the incident: prevent recurrence

The fix depends on the root cause identified during diagnosis.

Debug script or manual command: audit who has network access to port 11211 and restrict it to application servers only. Memcached has no authentication by default, so network access control is the primary defense.

Application code: search for flush_all calls. Common patterns include “clear cache” admin endpoints, deploy hooks that flush as a reset step, and test code that accidentally runs against production. Remove or gate these behind explicit confirmation.

Deployment automation: review CI/CD pipelines for cache-clear steps. Some deployment tools include flush_all as a default action during deploys.

Unauthorized access: the memcached port should never be reachable from untrusted networks. Bind to internal interfaces with -l, firewall port 11211, and consider authentication if the network environment is shared. SASL (-S) requires the binary protocol, which is deprecated as of 1.6.0; ASCII token authentication (-Y <file>, since 1.5.16) works over the text protocol.

Prevention

Alert on any cmd_flush increment in production. This is the single most important prevention step. Default severity TICKET; escalate to PAGE if backend overload signals confirm impact.
Track get_flushed as the impact measurement. When a cmd_flush alert fires, get_flushed tells you whether the flush is actively affecting traffic and how severely.
Restrict network access to port 11211. Memcached relies entirely on network-level access control. Bind to internal IPs with -l, firewall the port, and maintain an allowlist of expected client IPs.
Remove flush_all from automation and admin tooling. Audit deployment scripts, CI/CD pipelines, and admin interfaces for cache-clear operations. flush_all should never be an automated default in production.
Search application code for flush_all calls. Grep for flush_all, flushAll, flush(), and equivalent methods in memcached client libraries.
Use the delay argument for planned invalidation across a pool. If cache invalidation is genuinely needed across multiple servers, flush_all <delay> with staggered delays per server spreads the cold-start impact. This is a deliberate technique, not an accident mitigation.

How Netdata helps

Per-second cmd_flush tracking catches any increment immediately. The difference between detecting a flush_all in 1 second versus 60 seconds matters when the backend is saturating.
Correlating cmd_flush with get_flushed and hit ratio in the same view shows the full causal chain: the command fired, items were invalidated, misses spiked, hit ratio collapsed. This distinguishes a flush_all from an eviction cascade or restart without manual stat sampling.
Backend metrics alongside memcached metrics let you see the thundering herd in real time. Database query rate, connection count, and latency spike at the same moment hit ratio drops.
Anomaly detection on hit ratio and get_misses rate can flag the instant collapse pattern characteristic of a flush_all before an operator checks cmd_flush. The shape of the drop (instant cliff versus gradual decline) is a strong discriminator.
Uptime monitoring with discontinuity detection distinguishes a flush_all (stable uptime, cmd_flush incremented) from a restart (uptime reset, cmd_flush unchanged).