$ guides / redis ▌

REDIS · OPERATIONS PLAYBOOK

Running Redis in production, without the 3 a.m. surprises

A single-threaded event loop, an in-memory dataset, fork-based persistence, and a fixed replication backlog. The mental model of how Redis actually behaves under load, where it tends to break, what to monitor as your operation matures, and the runbooks for the incidents you'll see.

> Start with the monitoring checklist → # Jump to the full guide list

Redis is famously fast to set up and famously unforgiving once the dataset, the write rate, or the replica count grows past what the defaults assumed.

The defaults work. Until maxmemory is never set and the kernel OOM-kills the process. Until a forked child duplicates pages during an RDB save and used_memory_rss doubles. Until someone runs KEYS * against a million keys and the single-threaded event loop freezes every other client. Until the 1 MB default repl-backlog-size overflows during a network blip and a full resync forks the primary all over again. Until a failed background save trips stop-writes-on-bgsave-error and Redis answers every write with MISCONF Redis is configured to save RDB snapshots.

These guides are written for engineers who already run Redis, not for people learning what a key is. The goal is to give you the mental model of how the server actually behaves under load, the failure patterns that keep recurring, the monitoring story that catches problems before they page anyone, and the runbooks you wish someone had handed you before your last incident.

How Redis actually runs in production

Redis is not just a key-value store. It is one event-loop thread serving commands against an in-memory dataset, with a memory allocator that hates giving RAM back, fork-based persistence, and a streaming replication contract. Most production failures live between these layers, not inside any one of them.

clients / connection pool

Application connections, replicas, Sentinel, and cluster peers. Each costs a file descriptor and ~10 KB minimum. The total counts against <code>maxclients</code> — exceed it and new connections are refused with <code>rejected_connections</code>.

CLIENT

single-threaded event loop

One thread executes every command sequentially. Since 6.0, I/O threads handle network read/write, but command execution stays single-threaded. Any slow command — <code>KEYS</code>, a large <code>SORT</code>, a Lua script — blocks everything behind it.

EVENTLOOP

command execution

Commands run to completion with no preemption. An O(N) command on a big key, or a fork that freezes the thread, is paid synchronously by every other client waiting in the queue.

EXEC

in-memory dataset

Keys live in a hash table that rehashes in powers of two. Values are strings, hashes, sets, sorted sets, streams. Encoding downgrades (listpack to hashtable) and a single oversized key can blow up both memory and latency.

DATASET

eviction + expiry

At <code>maxmemory</code> the <code>maxmemory-policy</code> evicts keys or rejects writes with <code>OOM</code>. TTLs are cleaned lazily on access and actively by a sampling cycle (<code>hz</code> times/sec). Mass TTL-aligned expiry causes CPU and latency spikes.

EVICT

persistence (RDB / AOF)

RDB snapshots and AOF rewrites <code>fork()</code> a child to serialize the dataset. <code>appendfsync everysec</code> trades up to a second of durability for latency. A failed save with <code>stop-writes-on-bgsave-error yes</code> makes Redis read-only.

PERSIST

replication (backlog / replicas)

The primary streams writes through a fixed-size circular backlog. A replica that falls behind the backlog forces a full resync — another fork. Each replica also holds an output buffer on the primary.

REPLICA

OS memory + copy-on-write

The kernel OOM killer judges by <code>used_memory_rss</code>, not <code>used_memory</code>. During fork, copy-on-write duplicates dirty pages — Transparent Huge Pages turns a single-byte write into a 2 MB copy and inflates fork latency 10–100x.

KERNEL

disk (fsync)

Local NVMe, EBS, or a container volume under the data directory. AOF <code>fsync</code> latency and RDB write throughput set the floor on how durable — and how fast — persistence can be.

DISK

Why this matters: a latency spike can come from a slow O(N) command, a fork freeze, an fsync stall, an eviction storm, the active expiry cycle, swap, or a saturated single core. The symptom is the same — Redis is slow — but each layer has a different signal and a different fix.

The failures you'll actually see

Most Redis incidents fall into a small set of recurring patterns. Recognise the shape, and triage gets dramatically faster.

CRITICAL

The fork / COW memory storm

Redis calls fork() for an RDB save or AOF rewrite. If writes are heavy during the fork, copy-on-write duplicates dirty pages and used_memory_rss climbs toward 2x used_memory. The kernel OOM killer fires on RSS. This is the single most common cause of Redis getting killed in memory-limited containers.

used_memory_rss spikes while rdb_bgsave_in_progress = 1
rdb_last_cow_size or aof_last_cow_size above 50% of used_memory
OOM kill in dmesg during a scheduled save
latest_fork_usec far above ~20ms per GB (THP enabled)

Investigate →

IMMINENT

The event-loop wedge

One slow command blocks the single thread. KEYS *, SMEMBERS on a million-element set, a large SORT, or an unoptimised Lua script runs to completion while every other command — even a simple GET — queues behind it. instantaneous_ops_per_sec drops, latency compounds across all commands, and PING may stop answering.

slowlog filling rapidly with one or two command types
instantaneous_ops_per_sec drops while clients stay connected
cmdstat_keys showing any calls in production
latency rising on commands that are normally sub-millisecond

Investigate →

ACTIVE

The memory pressure spiral

Redis hits maxmemory and starts evicting. Evicted keys are re-requested, miss, and the application re-populates them — which evicts more keys. Redis does maximum work (evictions, writes, reads) for minimum value (constant misses). With noeviction the writes fail outright with OOM command not allowed.

evicted_keys rate and keyspace_misses rate climbing together
used_memory pinned at maxmemory
errorstat_OOM incrementing under noeviction
CPU rising as eviction runs in the command path

Investigate →

IMMINENT

The replication backlog overflow

A replica falls behind by more than repl-backlog-size (default 1 MB, almost always too small). Partial resync becomes impossible, so the primary forks for a full resync — which spikes latency, which makes other replicas lag past the backlog too. The primary can end up in a perpetual fork-resync loop.

sync_full incrementing on the primary
sync_partial_err incrementing (backlog too small)
connected_slaves fluctuating with rdb_bgsave_in_progress
replication offset lag approaching repl-backlog-size

Investigate →

CRITICAL

The connection exhaustion cascade

Redis hits maxclients and refuses new connections. Applications that can't connect retry aggressively, creating more attempts than are freed. rejected_connections climbs, application error rates spike, and the failure cascades to downstream services. A connection leak or an undersized pool ceiling is usually underneath it.

rejected_connections incrementing
connected_clients + connected_slaves + cluster_connections at maxclients
connection count climbing uncorrelated with traffic (leak)
CLIENT LIST full of idle connections

Investigate →

ACTIVE

The persistence write-stop

A background save fails — disk full, an I/O error, or a fork that can't allocate memory. With stop-writes-on-bgsave-error yes (the default), Redis answers every write with MISCONF Redis is configured to save RDB snapshots. The failure is sticky: it persists until the next successful save, so a transient disk hiccup becomes an availability incident.

MISCONF errors on every write command
rdb_last_bgsave_status:err
Can't save in background: fork: Cannot allocate memory in the log
disk free on the persistence volume near zero

Investigate →

Redis monitoring maturity levels

Redis observability works in four practical levels. Each is a complete operation, not a stepping stone. Pick the level that matches how much your Redis matters. Most production instances should land at the second level.

Level 1: Survival

Know that something is wrong

Survival monitoring is the floor. With these signals you can answer one question: is Redis still functioning? You will not learn what broke, but you will learn that something broke before users do. Survival is enough for dev environments and side-project caches.

PING reachability Does the instance answer PONG within a couple of seconds?
Uptime / unexpected restarts Did uptime_in_seconds reset without your permission?
Loading state Is loading:1 still rejecting data commands after a restart?
used_memory vs maxmemory How close are you to eviction or write rejection?
connected_clients vs maxclients Are you within the connection ceiling?
rdb_last_bgsave_status / aof_last_write_status Is persistence actually succeeding?
master_link_status on replicas Is the replica still attached to its primary?

Level 2: Operational

Diagnose most incidents on your own

Operational monitoring is what most production instances should target. Survival tells you something is wrong; operational tells you what. With this coverage your team can usually diagnose an incident on its own: eviction, fork latency, replication lag, slow commands, connection pressure.

instantaneous_ops_per_sec Is throughput where it should be — or has the event loop stalled?
keyspace hit rate keyspace_hits / (hits + misses) for cache effectiveness.
evicted_keys rate Is Redis deleting data to stay under maxmemory?
total_error_replies / errorstat_OOM Are writes being rejected with OOM?
rejected_connections rate Are clients being refused at maxclients?
mem_fragmentation_ratio Wasted RAM above 1.5, or swap below 1.0?
latest_fork_usec How long does each persistence fork freeze the thread?
slowlog growth rate Are O(N) commands blocking the event loop?
replication offset lag How far behind is each replica, in bytes?
connected_slaves on the primary Are all expected replicas attached?

Level 3: Mature

Catch problems before they become incidents

Mature monitoring catches problems before they wake anyone up. Fork COW size creeping, the replication backlog being too small, fragmentation drifting, AOF outgrowing its base, a Stream consumer group falling behind. None of these will page you on day one. They become page-out incidents on day thirty.

LATENCY LATEST by event category Where is latency coming from — command, fork, aof-fsync?
Client output buffer memory (omem) Is a slow subscriber or a stray MONITOR bloating the heap?
sync_full / sync_partial_err Are replicas doing expensive full resyncs?
rdb_last_cow_size / aof_last_cow_size How much does copy-on-write cost on each fork?
AOF size ratio (current / base) Is the AOF rewrite keeping the file compact?
blocked_clients Are queue consumers healthy, or have they died blocked?
expired_keys rate / expired_stale_perc Is active expiry keeping up with TTL churn?
Keyspace key-count trend Are keys without TTL accumulating unbounded?
Cluster state and slot health Are all 16384 slots covered and OK?

Level 4: Expert

Reactive instrumentation after real incidents

Expert signals enter your stack the day after a specific incident proved you needed them. jemalloc allocator stats, big-key sampling, encoding-downgrade detection, expire-cycle throttling, per-client leak analysis. Most teams never need every signal here. Add the ones your incident history says you do.

allocator_frag_ratio / allocator_rss_ratio Precise jemalloc fragmentation vs RSS padding.
active_defrag_running / hits / misses Is active defrag earning its CPU?
expired_time_cap_reached_count Is the active expiry cycle being throttled?
used_cpu_user_main_thread (6.2+) Precise main-thread CPU against the single-core ceiling.
Big-key sampling (--bigkeys / MEMORY USAGE) Is one giant key monopolising CPU and memory?
OBJECT ENCODING sampling Did a key downgrade from listpack to hashtable?
Stream PEL size and consumer idle time Are pending entries accumulating from dead consumers?
ACL LOG Failed auth and authorization attempts.

Operating mistakes worth avoiding

The traps Redis teams keep falling into. Each has a clear, well-known fix. Most teams only learn it after an incident.

Not setting maxmemory

The single most common production misconfiguration. With <code>maxmemory</code> at 0 (the default) Redis grows until the kernel OOM-kills it, then restarts cold into a thundering herd of reconnections and is immediately under pressure again. Every production instance needs <code>maxmemory</code> and a deliberate <code>maxmemory-policy</code>.

Running KEYS in production

<code>KEYS *</code> is O(N) over the entire keyspace and blocks the single thread for the whole scan. One call against a few million keys freezes every other client. Replace it with <code>SCAN</code> in all application code, and alert on any <code>cmdstat_keys</code> calls.

Leaving Transparent Huge Pages enabled

THP inflates fork latency 10–100x because a single-byte write copies a whole 2 MB page during copy-on-write. It's a kernel default, so it's on almost everywhere. Set <code>echo never > /sys/kernel/mm/transparent_hugepage/enabled</code> and make it persist across reboots.

Running a write-heavy master with RDB and no vm.overcommit_memory

Background saves <code>fork()</code>, and without <code>vm.overcommit_memory = 1</code> the kernel can refuse the fork with <code>Can't save in background: fork: Cannot allocate memory</code>. On a write-heavy master, copy-on-write also doubles RSS. Set overcommit, size for COW headroom, and consider <code>repl-diskless-sync</code>.

Leaving repl-backlog-size at the 1MB default

1 MB is wildly insufficient for any real write rate. A brief network blip accumulating more than 1 MB of writes forces a full resync — a fork, a latency spike, and often a cascade to other replicas. Set it to 100 MB minimum, 512 MB for write-heavy workloads.

appendfsync always for durability

<code>appendfsync always</code> makes every write wait for an fsync, which collapses throughput and exposes you directly to disk-latency spikes. <code>everysec</code> is the right default for almost everyone; bound your data-loss window with replication and backups instead.

Ignoring the RSS / fragmentation gap

Teams watch <code>used_memory</code> but not <code>used_memory_rss</code>. Fragmentation can consume 2–3x the logical data size, and the OOM killer judges by RSS. A ratio above 1.5 wastes RAM; a ratio below 1.0 means Redis is swapping — catastrophic for latency. Alert on both.

Treating persistence as fire-and-forget

Configuring <code>save</code> directives or AOF and never checking that saves complete. <code>rdb_last_bgsave_status:err</code> goes unnoticed for weeks until a crash reveals the last good backup is days old. Monitor save status and freshness, and restore-test on a schedule.

Redis runbooks in this section

Each guide is a focused runbook for one symptom or topic. Pick one when you have an incident, or use the categories to learn the area.

▸

Start here

▸

Memory, maxmemory, and eviction

▸

Persistence: RDB, AOF, and fork

▸

Latency and the single-threaded event loop

▸

Connections, clients, and buffers

▸

Replication, backlog, and resync

▸

Cluster, Sentinel, and failover

▸

Keyspace, expiry, and big keys

▸

Fragmentation, CPU, and the OS

▸

Pub/Sub and Streams

▸

Security and dangerous commands

WHERE TO GO NEXT

Setting up Redis monitoring, or putting out a fire?

If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, jump straight to the symptom that matches what you're seeing.

> Start with the checklist > Back to Operations Guides