$ guides / cassandra ▌

CASSANDRA · OPERATIONS PLAYBOOK

Running Cassandra in production, without the 3 a.m. surprises

A peer-to-peer ring, a log-structured merge tree, fork-free but GC-bound, with compaction running forever in the background and repair quietly racing a 10-day clock. The mental model of how Cassandra actually behaves under load, where it tends to break, what to monitor as your operation matures, and the runbooks for the incidents you'll see.

> Start with the monitoring checklist → # Jump to the full guide list

Cassandra is famously easy to scale horizontally and famously unforgiving once write rate, delete patterns, or compaction debt grow past what the defaults assumed.

The defaults work. Until the JVM heap fills and a multi-second GC pause makes gossip mark the node DN, so clients retry, hints pile up, and the node spirals. Until writes outrun compaction throughput and pending compactions climb for days while read latency creeps up unnoticed. Until a delete-heavy table accumulates tombstones across hundreds of SSTables and a read finally aborts with TombstoneOverwhelmingException. Until an STCS major compaction needs 100% temporary space the disk doesn't have and logs Not enough space for compaction. Until a node sits down longer than max_hint_window_in_ms and the hints simply stop, leaving data silently divergent until the next repair.

These guides are written for engineers who already run Cassandra, not for people learning what a keyspace is. The goal is to give you the mental model of how the cluster actually behaves under load, the failure patterns that keep recurring, the monitoring story that catches problems before they page anyone, and the runbooks you wish someone had handed you before your last incident.

How Cassandra actually runs in production

Cassandra is not just a distributed key-value store. It is a ring of equal peers, each one an LSM-tree engine appending to a commitlog, buffering in memtables, flushing immutable SSTables, and compacting them forever — while gossip, hints, and repair work to keep replicas agreeing. Most production failures live between these layers, not inside any one of them.

clients / coordinator

Any node can be the <code>coordinator</code> for a request. It hashes the partition key, finds the replica nodes that own the token range, and sends to enough of them to satisfy the consistency level. Coordinator-side latency (<code>ClientRequest</code> Read/Write) is the user-visible number.

COORDINATOR

replication / consistency level

Replication factor decides how many copies exist; the consistency level decides how many must answer. If not enough replicas are alive, the request fails immediately with <code>UnavailableException</code>; if they are alive but slow, it fails with a <code>TimeoutException</code>.

REPLICAS

commitlog + memtable (write path)

On every replica, the write is appended to the <code>commitlog</code> for durability before it is acknowledged, and lands in an in-memory <code>memtable</code> (one per table). If the commitlog disk backs up (<code>WaitingOnSegmentAllocation</code>), writes stall. When a memtable crosses <code>memtable_heap_space</code> / <code>memtable_offheap_space</code> it flushes to disk; premature flushes under pressure create many small SSTables and extra compaction work.

WRITEPATH

SSTables + compaction

Flushed memtables become immutable <code>SSTables</code>. Compaction merges them, discards tombstones, and reclaims space. The strategy (STCS, LCS, TWCS, UCS) sets the I/O pattern and the disk headroom you must keep. Falling behind is the compaction death spiral.

STORAGE

read path (bloom filters / merge)

A read checks the memtable, then uses bloom filters to pick candidate SSTables, then merges rows and applies tombstones. More SSTables per table means more disk seeks per read — read amplification — and tombstone-heavy partitions can make a single read scan enormous amounts of dead data.

READ

gossip + failure detector

Every second each node gossips state with peers. The phi accrual failure detector (default <code>phi_convict_threshold</code> 8) marks a node <code>DN</code> after sustained heartbeat absence — a GC pause beyond roughly 18 seconds is enough. Flapping nodes are usually a heap problem misread as a network one.

GOSSIP

hints + repair (anti-entropy)

When a replica is down, the coordinator stores <code>hints</code> and replays them on return — but only within <code>max_hint_window_in_ms</code> (default 3h). Beyond that, only anti-entropy <code>repair</code> reconciles replicas, and it must complete within <code>gc_grace_seconds</code> (default 10 days) or deleted data resurrects.

CONSISTENCY

JVM heap, GC, and the OS

Cassandra is one JVM per node. Heap holds memtables, caches, bloom-filter and compression structures, and in-flight requests; high old-gen occupancy triggers long stop-the-world G1 pauses that freeze gossip, reads, writes, and compaction at once. Underneath, the LSM engine is I/O-bound (commitlog, flush, compaction, and reads all compete), each SSTable opens several file descriptors, bloom filters and buffers live off-heap, and the Linux OOM killer judges by RSS — so a node can die with a healthy-looking heap.

JVM

Why this matters: a read latency spike can come from a compaction backlog, a tombstone-heavy partition, a slow replica over the network, a GC pause, disk I/O saturation, or a single oversized partition. The symptom is the same — Cassandra is slow — but each layer has a different signal and a different fix.

The failures you'll actually see

Most Cassandra incidents fall into a small set of recurring patterns. Recognise the shape, and triage gets dramatically faster.

CRITICAL

The GC death spiral

A trigger — a large partition read, a tombstone scan, an oversized batch — drives a long old-gen GC pause. During the pause the node can't gossip, so peers mark it DN. Clients retry elsewhere, hints accumulate, and when the node recovers it faces a flood of hint replay and read repair on top of the now-higher baseline. That load triggers the next pause. The canonical Cassandra failure mode.

GCInspector pause warnings, G1 Old Generation pauses above 2s
heap usage not recovering after GC (rising post-GC floor)
node flapping UP/DOWN in gossip while reachable between pauses
dropped mutations and client timeouts rising together

Investigate →

IMMINENT

The compaction death spiral

Write workload generates SSTables faster than compaction can merge them. PendingTasks climbs for hours or days. Every read must touch more SSTables (read amplification), so P99 creeps up while writes stay fast and hide the problem. Compaction falls further behind because each merge reads more files, and disk I/O saturates. Operators mistake it for a sudden problem when it was gradual all along.

Compaction PendingTasks rising continuously over many hours
LiveSSTableCount per table growing
read P99 climbing while write latency stays normal
disk I/O utilisation pinned near saturation

Investigate →

ACTIVE

The tombstone storm

A delete-heavy or TTL-heavy access pattern spreads tombstones across many SSTables. Compaction can't purge them efficiently, and they can't be purged at all until repair has run within gc_grace_seconds. Reads scan and merge all that dead data, burning CPU, heap, and I/O. Past tombstone_failure_threshold (100,000) the read aborts with TombstoneOverwhelmingException.

Scanned over N tombstones warnings in system.log
TombstoneScannedHistogram p99 growing on a table
read P99 collapsing while P50 still looks fine
TombstoneOverwhelmingException aborting reads

Investigate →

CRITICAL

The disk space exhaustion

Compaction needs temporary space to write merged SSTables before deleting the originals — STCS can transiently need up to 100% of the table size. As free space shrinks, compaction stops, so old SSTables are never reclaimed. Forgotten snapshots (hard links) and accumulated hints quietly eat the rest. Eventually the commitlog can't allocate segments and writes stop with Not enough space for compaction in the log.

free space below per-strategy headroom and still shrinking
Not enough space for compaction in system.log
commitlog WaitingOnSegmentAllocation above zero
snapshots or hints directory silently consuming gigabytes

Investigate →

IMMINENT

The quorum loss

Enough replicas go offline that no consistency level requiring a majority can be satisfied. Clients get UnavailableException immediately — no waiting, no retry at that CL. This is total outage for the affected token ranges, and unlike a timeout it does not self-resolve. It usually follows multiple node failures, a bad rolling restart, or cascading GC death spirals.

UnavailableException (not TimeoutException) spiking
DownEndpointCount above RF/2 in the same failure domain
read/write failures immediate rather than slow
hint accumulation on the surviving coordinators

Investigate →

WATCHFUL

The hint overflow

A replica is down, so coordinators store hints to replay later. But hints are only kept for max_hint_window_in_ms (default 3 hours). If the node stays down longer, hints stop being generated and every write to that replica during the rest of the outage is simply lost — no error, no retry. The only fix is a full repair, which is frequently forgotten. Hint files also consume coordinator disk on the way there.

hints directory growing on coordinators over hours
HintsFailed or HintsTimedOut incrementing
a node down longer than the 3-hour hint window
hint replay storm re-downing the node when it returns

Investigate →

Cassandra monitoring maturity levels

Cassandra observability works in four practical levels. Each is a complete operation, not a stepping stone. Pick the level that matches how much your cluster matters. Most production clusters should land at the second level.

Level 1: Survival

Know that something is wrong

Survival monitoring is the floor. With these signals you can answer one question: is the cluster still functioning? You will not learn what broke, but you will learn that something broke before users do. Survival is enough for dev clusters and non-critical workloads.

Process alive Is the Cassandra JVM running on each node?
Node status (UN / DN) Does nodetool status show every node Up/Normal?
Native transport running Is the node accepting CQL clients on 9042?
Disk space remaining Is any data or commitlog volume approaching full?
Basic client error rate Are drivers reporting timeouts or unavailables?

Level 2: Operational

Diagnose most incidents on your own

Operational monitoring is what most production clusters should target. Survival tells you something is wrong; operational tells you what. With this coverage your team can usually diagnose an incident on its own: GC pressure, compaction backlog, dropped writes, timeouts, quorum loss.

Client request latency (read/write P99) The user experience — watch P99/P999, not averages.
Client request rate Throughput baseline that gives every other signal context.
Timeouts and Unavailables Slow replicas (timeout) vs not enough replicas alive (unavailable).
Dropped messages (MUTATION) Writes silently shed at a replica — possible data loss.
JVM heap usage and GC pause time Post-GC heap floor and pauses above 500ms / 2s.
Pending compactions Is compaction keeping up, or building debt?
Node count vs expected topology Are all members present, or is one DN/UJ/UL?

Level 3: Mature

Catch problems before they become incidents

Mature monitoring catches problems before they wake anyone up. SSTable count creeping per table, tombstone warnings on a delete-heavy table, hints accumulating during a slow replica, a repair that hasn't completed in eight days. None of these will page you on day one. They become page-out incidents on day thirty.

Per-table SSTable count and latency Read amplification building before P99 spikes cluster-wide.
Thread pool pending / blocked tasks SEDA backpressure — and a backed-up GOSSIP pool is serious.
Hinted handoff status and rate Is a replica down long enough to threaten the hint window?
Tombstone scan warnings Reads scanning past tombstone_warn_threshold (1000).
Repair completion tracking Has every table been repaired within gc_grace_seconds?
Commitlog pending tasks Write-path I/O pressure before mutations drop.
Disk I/O per device (data vs commitlog) Utilisation and await on the volumes Cassandra uses.
File descriptor usage Open FDs vs ulimit — a binary cliff, not a slope.
Schema agreement Does describecluster show a single schema version?
Storage exceptions counter Disk/filesystem/SSTable corruption surfacing.

Level 4: Expert

Reactive instrumentation after real incidents

Expert signals enter your stack the day after a specific incident proved you needed them. Partition size distribution, per-table tombstone density, phi failure-detector values, off-heap accounting, LWT contention. Most teams never need every signal here. Add the ones your incident history says you do.

Per-partition size distribution Catch oversized partitions before they OOM a read.
Tombstone density per table Tombstone-to-live-cell ratio flags data-model problems.
Gossip phi failure-detector values How close is a node to being convicted DOWN?
Off-heap memory usage Bloom filters, buffers, caches the heap metric can't see.
LWT (CAS) latency, tracked separately Paxos adds round-trips — don't hide it in normal latency.
Read repair and speculative retry rates Replica divergence and consistently slow replicas.
Streaming throughput and session status Bootstrap, decommission, and repair streaming health.
Virtual tables / guardrails (4.0+ / 4.1+) CQL-queryable operational data and soft/hard limits.

Operating mistakes worth avoiding

The traps Cassandra teams keep falling into. Each has a clear, well-known fix. Most teams only learn it after an incident.

Never running (or never verifying) repair

The single most common and most dangerous gap in the industry. Teams set Cassandra up, it works, and nobody schedules repair or checks that it completes. Months later <code>gc_grace_seconds</code> passes, tombstones are compacted away on some replicas, and deleted data resurrects. Schedule incremental repair, and alert when any table's last successful repair exceeds 80% of <code>gc_grace_seconds</code>.

Ignoring pending compactions until latency spikes

Teams watch "is it UP?" and read latency but ignore the compaction queue. By the time P99 finally spikes, the root cause is days of accumulated backlog, and recovery takes hours. Trend <code>PendingTasks</code> and SSTable count per table, and alert when they climb continuously.

Not alerting on GC pauses until the node crashes

Sub-second GC pauses destroy latency long before an <code>OutOfMemoryError</code> arrives, and a pause beyond roughly 18 seconds gets the node marked <code>DN</code>. Alert on GC pause duration and the post-GC heap floor — the #1 cause of Cassandra performance degradation — not just on OOM.

Running full repairs during peak hours

Anti-entropy repair generates massive disk and network I/O from Merkle-tree exchange and streaming. Run it during production peak and it starves reads and writes of I/O — causing the very outage repair exists to prevent. Throttle it, schedule it off-peak, and prefer incremental repair.

Treating the Load metric as disk usage

The <code>Load</code> value in <code>nodetool info</code> is only data file size. It excludes commitlog, hints, snapshots, and compaction temporary space, and it can't see how much is garbage awaiting compaction. Always size disk headroom from actual filesystem metrics, and remember STCS may transiently need 100% additional space.

Forgetting NTP and clock synchronisation

Because Cassandra resolves conflicts by timestamp (last write wins), clock drift of even a few seconds silently reorders or overwrites writes — a quiet data-corruption catastrophe with no error in any log. Run NTP/chrony on every node and monitor drift across the ring.

Confusing Unavailable with Timeout

<code>UnavailableException</code> (not enough replicas alive for the CL) and <code>TimeoutException</code> (replicas alive but slow) have completely different causes and responses. Teams that lump them into one "errors" metric miss the distinction and chase the wrong fix mid-incident.

Monitoring only the heap, never off-heap

Bloom filters, compression metadata, direct buffers, and the chunk cache live off-heap and scale with SSTable count. The JVM heap can look perfectly healthy while total RSS exceeds available RAM and the Linux OOM killer terminates Cassandra. Watch process RSS against system memory, especially in containers with cgroup limits.

Cassandra runbooks in this section

Each guide is a focused runbook for one symptom or topic. Pick one when you have an incident, or use the categories to learn the area.

▸

Start here

▸

JVM heap, GC, and the death spiral

▸

Compaction, SSTables, and read amplification

▸

Tombstones, deletes, and gc_grace_seconds

▸

Disk space, snapshots, and storage

▸

Timeouts, dropped messages, and load shedding

▸

Node liveness, gossip, and quorum

▸

Hinted handoff, repair, and consistency

▸

Streaming, bootstrap, and topology changes

▸

Commitlog, memtables, and the write path

▸

File descriptors, off-heap, and OS limits

▸

Data model, large partitions, and batches

WHERE TO GO NEXT

Setting up Cassandra monitoring, or putting out a fire?

If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, jump straight to the symptom that matches what you're seeing.

> Start with the checklist > Back to Operations Guides