Cassandra Operations Guides on Netdata

Cassandra adding and removing nodes safely: vnodes, tokens, and cleanup

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra adding and removing nodes safely: vnodes, tokens, and cleanup

Expanding or contracting a Cassandra cluster redistributes token ownership, triggers bulk streaming, and leaves stale data that cleanup must reclaim. Skip cleanup after a bootstrap, run it while a node is joining, or misconfigure token allocation, and you waste disk space, create hot spots, and risk silent inconsistency.

This guide covers adding and removing nodes with virtual nodes (vnodes) enabled, the default for Cassandra 3.x through 5.x. Commands and paths assume standard packaged Cassandra on Linux. Perform topology changes one node at a time.

Cassandra Batch too large warning: oversized batches and coordinator OOM

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra Batch too large warning: oversized batches and coordinator OOM

Oversized BEGIN BATCH statements cause Batch for [ks.table] is of size N, exceeding specified threshold of M by ... warnings and coordinator OutOfMemoryError. Unlike single-partition batches, which provide atomicity within one partition, multi-partition batches force the coordinator to hold mutation buffers for every affected partition until all replicas acknowledge. When the buffer grows large enough, it triggers heap pressure, long GC pauses, and eventual OOM.

Cassandra clock skew: how NTP drift silently corrupts data

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra clock skew: how NTP drift silently corrupts data

Cassandra’s last-write-wins conflict resolution is simple, fast, and unforgiving. Every write carries a timestamp; when replicas disagree, the highest timestamp wins. The database assumes larger timestamps correspond to later wall-clock events. When node clocks drift, that assumption collapses. A write that happened first can carry a later timestamp, or a later write an earlier one. The result is not a timeout, an UnavailableException, or an ERROR log entry. It is silent data loss, permanent shadowing of valid writes, or the sudden return of deleted rows.

Cassandra commitlog disk full: segment exhaustion and forced flushes

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra commitlog disk full: segment exhaustion and forced flushes

WriteTimeoutException from client drivers, a node still UP in gossip, and a commitlog volume nearing 100% with climbing pending tasks in nodetool info mean commitlog segment exhaustion. Unlike data disk exhaustion, which slows compaction, commitlog pressure blocks the write path directly: every mutation must be durably appended to the WAL before acknowledgment.

Cassandra recycles commitlog segments only after all memtables they reference are flushed to SSTables. When the flush pipeline cannot keep pace, segments accumulate until the total size exceeds commitlog_total_space (or commitlog_total_space_in_mb on older versions) or the filesystem fills. Cassandra then forces flushes of every dirty column family referenced in the oldest segment to free space. If flushes are already backed up, this cascades into blocked segment allocation, dropped mutations, and depending on commit_failure_policy, a node that stops accepting writes entirely.

Cassandra commitlog pending tasks: write-path I/O pressure

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra commitlog pending tasks: write-path I/O pressure

Sustained non-zero CommitLog PendingTasks means a Cassandra node’s write path is backing up. Every write must be appended to the commitlog and synced to disk before the coordinator acknowledges it. When the fsync thread cannot keep up, mutations queue. This starts as elevated write latency; if the queue persists, it forces emergency memtable flushes, overwhelms the flush and compaction pipeline, and produces dropped mutations.

Cassandra compaction death spiral: when writes outrun compaction throughput

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra compaction death spiral: when writes outrun compaction throughput

P99 read latency climbs while disk utilisation on the data volume pins near 100%. nodetool compactionstats shows pending tasks rising hour over hour, and nodetool tablestats reports a growing SSTable count. Writes stay fast; reads slow down. This is the compaction death spiral: writes exceed compaction throughput, SSTables accumulate, and read amplification rises.

Unlike a sudden node crash, this failure is gradual. A background queue grows a little each day. Once disk I/O saturates, the cycle self-reinforces: compaction falls further behind, reads consult more files, latency spikes, and the backlog deepens. By the time client SLAs breach, recovery can take hours.

Cassandra compaction strategies: STCS vs LCS vs TWCS vs UCS

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra compaction strategies: STCS vs LCS vs TWCS vs UCS

Compaction merges immutable SSTables, discards tombstones, and reclaims disk space. The strategy assigned to a table controls the tradeoff between write amplification and read amplification, and it determines how much temporary disk headroom you must preserve. A fit strategy keeps SSTable counts low and latency predictable. A mismatch creates compaction debt: creeping P99 read latency first, then disk space exhaustion, and finally write rejections when compaction cannot reclaim space fast enough to keep up with flushes.

Cassandra compaction stuck: large partitions blocking a compaction thread

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra compaction stuck: large partitions blocking a compaction thread

nodetool compactionstats shows a compaction on one table that has not moved past the same byte offset for hours. The progress percentage is frozen, the pending queue behind it is growing, and read latency on that table is creeping up. This is not a slow disk. A single large partition has monopolized a compaction thread, turning background maintenance into a bottleneck that threatens node stability.

Cassandra consistency levels explained: QUORUM, ONE, LOCAL_QUORUM, and EACH_QUORUM

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra consistency levels explained: QUORUM, ONE, LOCAL_QUORUM, and EACH_QUORUM

Consistency level (CL) balances latency, availability, and correctness. It does not control how many replicas store data; replication factor (RF) does. CL controls how many replicas must acknowledge a read or write before the coordinator returns to the client. The wrong choice produces UnavailableException during rolling restarts that should be safe, or leaves replicas inconsistent for hours after a write is acknowledged at CL=ONE. The four CLs that define most production topologies are ONE, QUORUM, LOCAL_QUORUM, and EACH_QUORUM.

Cassandra CorruptSSTableException and FSError: disk failure and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra CorruptSSTableException and FSError: disk failure and recovery

A Cassandra node stops serving traffic or refuses to start. In system.log you see org.apache.cassandra.io.sstable.CorruptSSTableException, FSError, or a JVM shutdown triggered by a filesystem exception. These indicate disk failure, filesystem corruption, or irreversible SSTable damage, not retryable application bugs.

Because SSTables are immutable, a corrupt file cannot be patched. The node either stops serving the data or shuts down, depending on disk_failure_policy. Recovery requires at least one healthy replica. Without that, corruption is data loss.

Cassandra disk space exhaustion: emergency recovery when the data volume fills

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra disk space exhaustion: emergency recovery when the data volume fills

A Cassandra node that runs out of disk space does not degrade gracefully. Compaction halts because it cannot allocate temporary space to merge SSTables. Old SSTables are never deleted. Writes append to the commitlog until segment allocation blocks. At that point the node rejects mutations and CommitLog.WaitingOnSegmentAllocation climbs. You may see No space left on device errors while the data volume still reports a few percent free, because Cassandra’s internal headroom requirements are stricter than the filesystem.

Cassandra dropped mutations: silent write loss and load shedding

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra dropped mutations: silent write loss and load shedding

Your application logs successful writes, but reads return stale or missing data. An alert fires on DroppedMessage rate for MUTATION scope. The client never received an error, yet a replica discarded the write after it sat in the MutationStage queue past timeout. This is Cassandra load shedding. Silent write loss occurs whenever not enough other replicas succeed to meet the consistency level.

Cassandra dropped reads and other messages: reading nodetool tpstats Dropped

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra dropped reads and other messages: reading nodetool tpstats Dropped

When nodetool tpstats reports non-zero values in the Dropped section, the node discarded internal messages that exceeded their stage timeout. These counters are cumulative since JVM startup, not rates. A non-zero value warrants investigation: the timeout is defined in cassandra.yaml by settings such as read_request_timeout_in_ms and write_request_timeout_in_ms, so a drop means the message sat in the queue for seconds.

Dropped messages are a lagging indicator. Correlate the drop type with the matching thread pool pending count, disk I/O latency, and GC pause duration to find the root cause.

Cassandra frequent memtable flushes: small SSTables and compaction burden

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra frequent memtable flushes: small SSTables and compaction burden

When MemtableFlushWriter pending tasks climb on Cassandra nodes, SSTables often land on disk at only tens of megabytes and multiply quickly. Within hours, PendingCompactions rises, read latency drifts, and disk I/O stays pinned despite flat write throughput. Frequent memtable flushing under memory pressure creates compaction debt and read amplification. Once started, the cluster enters a feedback loop that is hard to unwind without targeted tuning.

Cassandra GC death spiral: long pauses, gossip flapping, and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra GC death spiral: long pauses, gossip flapping, and recovery

You are paged because a Cassandra node is flapping between UP and DOWN in nodetool status, client timeouts are rising, and system logs show GCInspector warnings. The node has not crashed. It is stuck in a GC death spiral: heap pressure produces long pauses, gossip marks the node DOWN, and the resulting retry and hint traffic creates even more heap pressure when the node recovers. It can start with a single large partition read, a misconfigured cache, or an oversized batch statement, and escalates until the node is effectively useless. Catch it early by watching the GC floor and gossip stability together, not just process uptime.

Cassandra GC pauses too long: diagnosing G1 stop-the-world pauses

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra GC pauses too long: diagnosing G1 stop-the-world pauses

ReadTimeoutException and WriteTimeoutException from clients, GCInspector warnings in system.log, and nodes flapping between UP and DOWN in nodetool status without a JVM restart mean G1 is producing long stop-the-world pauses. Root causes include promotion pressure, humongous objects, or allocation bursts. Left unchecked, one node’s pauses trigger gossip failures, retries, and hint replay that drive cluster-wide degradation.

What this means

G1GC is the default collector for Cassandra 4.x on JDK 11+. During a stop-the-world pause, every thread freezes, including gossip, native transport, and compaction. Cassandra logs GCInspector warnings when a pause exceeds the configured threshold, commonly 500 ms . Pauses longer than ~2 seconds cause gossip rounds to be missed; under the default phi accrual failure detector threshold of 8, sustained pauses of tens of seconds result in the node being marked DOWN . While the JVM is paused, mutations queue, reads stall, hints accumulate on peers, and clients retry. On recovery, hint replay and retry bursts raise allocation pressure, creating a self-reinforcing spiral.

Cassandra gossip flapping: nodes bouncing UP and DOWN

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra gossip flapping: nodes bouncing UP and DOWN

nodetool status shows a node flipping between UN and DN, or multiple nodes doing it in sequence. Each transition forces the cluster to replay hints, recalculate read repair, and propagate gossip state. When a node transitions more than three times in thirty minutes, you are dealing with gossip flapping. It is almost always JVM heap pressure or GC pauses misdiagnosed as a network problem.

Cassandra heap pressure: sizing the JVM heap and tuning G1GC

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra heap pressure: sizing the JVM heap and tuning G1GC

Cassandra runs as a single JVM process per node. Every write path allocation, memtable mutation, read merge buffer, and cache entry lives on the heap. When the heap is undersized or GC is left at JVM defaults, stop-the-world pauses freeze gossip, client requests, and compaction. A pause longer than roughly 18 seconds (the default phi accrual threshold is 8) causes peers to mark the node DOWN, which triggers hinted handoff, replay storms, and client retries that worsen memory pressure: the GC death spiral.

Cassandra hint overflow: max_hint_window expiry and silent data divergence

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra hint overflow: max_hint_window expiry and silent data divergence

You restart a node after a four-hour outage. Gossip converges, nodetool status shows UN, and clients reconnect. Reads at consistency level ONE return stale data. The node has never been repaired.

The problem is hint overflow: the outage lasted longer than max_hint_window_in_ms (default three hours), so coordinators stopped saving hints after the window expired. Writes accepted during the final hour of the outage are missing from that replica. Coordinator logs show no errors; write acknowledgments succeeded because other replicas responded. Only anti-entropy repair closes the gap. Without it, the missing data sits on that replica indefinitely, surfacing as inconsistent reads or resurrected deletes.

Cassandra hints accumulating: hinted handoff backlog and replay storms

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra hints accumulating: hinted handoff backlog and replay storms

A coordinator node triggers a disk alert for /var/lib/cassandra/hints, or a replica returning from maintenance flaps UP/DOWN under a write burst it never requested. Hints let writes succeed when a replica is temporarily unreachable, but a large backlog turns that safety net into a secondary failure. Hints consume coordinator disk, expire after max_hint_window_in_ms, and can synchronize into a replay storm that overwhelms a recovering node.

Cassandra hot partition: when one key saturates a replica set

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra hot partition: when one key saturates a replica set

One or two nodes run hot while the rest of the cluster idles. Client P99 latency doubles or triples, but the average looks fine. Timeouts cluster on a subset of hosts, and nodetool status shows uneven load that does not match token ring expectations. When you trace requests, a single partition key consumes a disproportionate share of reads or writes. This is a hot partition: the partitioner mapped one key to a narrow token range, and the replicas owning that range are saturated.

Cassandra java.lang.OutOfMemoryError: Java heap space - causes and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra java.lang.OutOfMemoryError: Java heap space - causes and recovery

Your Cassandra log shows java.lang.OutOfMemoryError: Java heap space. The node stops responding to client requests, gossip marks it DOWN, and the JVM may crash. Because Cassandra runs as a single JVM process, heap exhaustion freezes every in-memory subsystem: memtables, prepared statement caches, bloom filter summaries, and in-flight request buffers.

The heap may climb toward its limit for hours, or a single massive allocation from a large partition read or oversized batch can push it over immediately. Recovery depends on whether the root cause is capacity, data modeling, or a traffic flood.

Cassandra killed by the Linux OOM killer: off-heap memory and RSS

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra killed by the Linux OOM killer: off-heap memory and RSS

The JVM heap chart shows 50% utilization and a flat line. There is no OutOfMemoryError. Then the Cassandra process vanishes. dmesg shows the OOM killer terminated the JVM: Killed process 12345 (java). The JVM heap metric does not include native allocations: bloom filters, compression metadata, index summaries, direct buffers, and chunk cache. When heap plus off-heap RSS exceeds available RAM, the kernel kills the process. This guide covers how to confirm that pattern, reduce off-heap footprint, and prevent recurrence.

Cassandra large partition pathology: Compacting large partition warnings and reads

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra large partition pathology: Compacting large partition warnings and reads

If system.log prints Compacting large partition ks/table:key (N bytes) and reads to the affected table are spiking, a single partition has crossed compaction_large_partition_warning_threshold_mb (default 100 MB). Oversized partitions force the node to deserialize a large in-memory index on reads, rewrite the entire partition during compaction, and move it atomically during streaming or repair. Left alone, the partition grows until it triggers GC pauses, gossip flapping, and cascading retries. Use this guide to confirm the offending key, measure the blast radius, and choose between immediate relief and a data-model fix.

Cassandra lightweight transaction contention: Paxos round-trips and CAS latency

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra lightweight transaction contention: Paxos round-trips and CAS latency

A normal Cassandra write reaches a coordinator, gets appended to the commitlog and memtable on the replicas, and returns. One network round-trip. A lightweight transaction (LWT) using IF NOT EXISTS, IF EXISTS, or IF <condition> triggers Paxos consensus instead: four round-trips of coordination overhead, plus serialization of contending requests against the same partition. If your monitoring only tracks aggregate read and write latency, LWT tail latency is invisible until clients time out.

Cassandra monitoring checklist: the signals every production cluster needs

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra monitoring checklist: the signals every production cluster needs

The four maturity levels below are cumulative. Do not instrument level 2 until level 1 is visible and alerted.

Cassandra’s peer-to-peer architecture and LSM storage engine create failure modes generic infrastructure monitoring misses. A node can be UP in gossip and accepting CQL connections while dropping mutations or accumulating compaction debt that only surfaces hours later. These signals catch liveness, performance, saturation, and consistency failures: GC death spirals, compaction avalanches, tombstone storms, and silent data divergence.

Cassandra monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra monitoring maturity model: from survival to expert

Cassandra exposes JMX MBeans, virtual tables, and log signals. Without priority, teams miss compaction debt or drown in noise. This model structures production monitoring into four cumulative levels. Each level adds signals that reduce mean time to detection and catch the failures that dominate Cassandra incidents: data resurrection from missed repair, compaction death spirals, and GC-induced gossip flapping.

Audit your current instrumentation against these levels. See the Cassandra monitoring checklist for a condensed signal inventory.

Cassandra native transport not running: node UP in gossip but refusing CQL clients

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra native transport not running: node UP in gossip but refusing CQL clients

A node reports UN in nodetool status but rejects CQL connections on port 9042. Gossip and replication are healthy; the failure is isolated to the native transport layer.

Because the node remains in the token ring, it continues to handle internode replication, gossip, and streaming. Applications see it as down; the cluster sees it as up. The JMX attribute NativeTransportRunning on org.apache.cassandra.db:type=StorageService is false while gossip heartbeats continue. The usual triggers are nodetool disablebinary left active after maintenance, or a firewall blocking TCP 9042.

Cassandra network partition and split brain: detection and reconciliation

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra network partition and split brain: detection and reconciliation

A partial network partition does not always stop the cluster. Each isolated subset often continues to serve traffic, accept writes, and report itself healthy while marking the other side as DOWN. By the time you notice contradictory gossip views or resurrected data, the two sides have diverged for hours. A healed partition is not self-resolving. If both sides accepted writes, last-write-wins semantics combined with even modest clock skew can silently overwrite valid data. Cross-DC deployments are most vulnerable because WAN latency already strains the phi accrual failure detector and inter-DC gossip paths have more single points of failure.

Cassandra node showing DN in nodetool status: gossip, phi, and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra node showing DN in nodetool status: gossip, phi, and recovery

You run nodetool status and one of your nodes shows DN (Down/Normal). Before you restart anything, understand that this output is the local node’s opinion, not global truth. In Cassandra’s peer-to-peer architecture, every node runs its own phi accrual failure detector over gossip heartbeats. A DN mark means this specific observer has not heard from the target for roughly 18 seconds at default settings. Another node in the same cluster may still show the same target as UN.

Cassandra node stuck in joining (UJ): bootstrap diagnosis

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra node stuck in joining (UJ): bootstrap diagnosis

You add a node to the ring, run nodetool status, and see it stuck in UJ (Up/Joining) for hours. The cluster sees it in gossip, but it never transitions to UN (Up/Normal). Client drivers do not route traffic to it, so the expansion has not added usable capacity. Until the state changes, the node is a ghost member: visible to the ring but unable to serve reads or writes for its assigned token ranges.

Cassandra Not enough space for compaction: STCS space amplification and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra Not enough space for compaction: STCS space amplification and recovery

Not enough space for compaction in system.log means STCS has hit a structural space-amplification limit. Disk usage may already be above 50%. Cassandra aborts the compaction, skips the tier, and leaves tombstones and old versions unmerged. SSTable count rises, read amplification increases, and free space stops being reclaimed. Left unchecked, this enters a compaction death spiral that ends in write rejection or disk exhaustion. Recovery requires immediate free space, targeted cleanup, and a headroom plan that accounts for transient STCS amplification.

Cassandra OperationTimedOutException: client-side timeouts vs server timeouts

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra OperationTimedOutException: client-side timeouts vs server timeouts

OperationTimedOutException (driver 3.x) or DriverTimeoutException (driver 4.x) means the driver gave up before the coordinator responded. This is distinct from a server-side ReadTimeout or WriteTimeout, where the coordinator explicitly errors because replicas were too slow. Confusing the two leads to tuning the wrong timeout, masking a server capacity problem or adding unnecessary client latency. Use server-side signals and driver behavior to tell them apart.

Cassandra pending compactions growing: the compaction backlog runbook

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra pending compactions growing: the compaction backlog runbook

Pending tasks climbing in nodetool compactionstats is normal after a bulk load under STCS, but when the number trends upward for hours it signals that your node is producing SSTables faster than compaction can merge them.

This is the leading indicator of the compaction death spiral. Left unchecked, the backlog drives read amplification up, saturates disk I/O, and eventually exhausts disk space as temporary compaction files accumulate. Writes often stay fast while reads degrade, masking the problem.

Cassandra quorum loss: when too many replicas are down to satisfy the CL

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra quorum loss: when too many replicas are down to satisfy the CL

Your application is throwing UnavailableException. Writes and reads fail immediately, not timing out. The coordinator checked the token ring, counted live replicas, and rejected the request because it cannot satisfy the consistency level. No client retry will fix this server-side. The cluster is in quorum loss and will not recover until enough replicas are restored.

When the count of down nodes for a token range exceeds floor(RF/2), every request at QUORUM or stronger fails instantly. Surviving nodes may still serve CL=ONE requests for ranges that have a live replica, but any operation requiring a majority is an outage for those partitions. This is a PAGE-level composite failure pattern that demands immediate node recovery followed by repair.

Cassandra read latency spikes: P99 vs P50 and proxyhistograms

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra read latency spikes: P99 vs P50 and proxyhistograms

Your application is timing out on Cassandra reads, but a quick glance at average latency looks acceptable. This is the percentile trap. In Cassandra, read latency follows a long-tail distribution: most requests are fast, but a fraction hit slow replicas, large partitions, or GC pauses and become orders of magnitude slower. The critical first step is knowing whether the spike lives at the coordinator or the replica, and whether it is systemic or isolated to the tail. nodetool proxyhistograms and nodetool tablehistograms answer exactly that.

Cassandra ReadTimeoutException: diagnosing coordinator read timeouts

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra ReadTimeoutException: diagnosing coordinator read timeouts

The driver throws ReadTimeoutException when the coordinator fails to gather enough replica responses within read_request_timeout_in_ms (default 5000 ms). This is a server-side timeout. It maps directly to the JMX metric ClientRequest,scope=Read,name=Timeouts.

This is not OperationTimedOutException, which fires on the driver’s socket timeout. It is also not UnavailableException, which means not enough replicas were alive to attempt the read. Here, replicas are alive but too slow. The read may have executed partially on some replicas, yet the coordinator could not assemble a response that met the consistency level within the window.

Cassandra repair failing or stuck: partial repairs and how to verify completion

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra repair failing or stuck: partial repairs and how to verify completion

nodetool repair returning to the prompt, or hanging at 99 percent, are both dangerous because the worst outcome is silent: a partial repair. A partial repair anti-compacts some token ranges and skips others, leaving inconsistency while looking complete. Because incremental repair marks SSTables as repaired during anti-compaction, a session that fails mid-range leaves later ranges unrepaired while earlier ones are already marked done. No built-in alert fires when only forty percent of ranges were covered. If unrepaired ranges contain tombstones, deleted data resurrects once gc_grace_seconds passes. This guide shows how to verify completion, diagnose stuck sessions, and recover without causing a cascading I/O incident.

Cassandra repair not running: the silent gap that resurrects deleted data

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra repair not running: the silent gap that resurrects deleted data

Deleted rows reappear in application queries, or system_distributed.repair_history shows a last success older than your gc_grace_seconds window. In Cassandra, a repair that is not running, not completing, or not scheduled is an active data integrity risk.

Cassandra uses tombstones to track deletions and TTL expirations. Those tombstones must survive on every replica until anti-entropy repair propagates the delete. Once gc_grace_seconds passes, compaction drops tombstones. If repair has not finished for a table within that window, nodes that missed the original delete retain live data while the rest have discarded the tombstone. The result is zombie data resurrection.

Cassandra repair overload: when anti-entropy repair causes the outage it prevents

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra repair overload: when anti-entropy repair causes the outage it prevents

A full nodetool repair started during peak traffic can spike P99 read latency from milliseconds to hundreds of milliseconds, trigger write timeouts, and cause nodes to flap between UP and DOWN in gossip. The repair job meant to prevent inconsistency becomes the cause of the outage.

Anti-entropy repair is a heavy distributed scan, not a background task. It reads all local data to build Merkle trees, exchanges hashes with replicas, and streams differing ranges. On a multi-terabyte node this means terabytes of sequential disk reads, heavy CPU hashing, and gigabits of network traffic. Without dedicated headroom, repair competes with the commitlog, memtable flushes, compaction, and client requests for disk bandwidth, CPU, and network. The result is thread pool backpressure, dropped messages, GC pressure from Merkle tree construction, and eventually gossip failure as the node becomes unresponsive.

Cassandra Scanned over N tombstones warning: finding the offending query

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra Scanned over N tombstones warning: finding the offending query

Your application latency may still look acceptable at the median, but system.log is filling with warnings like Read 500 live rows and 12000 tombstone cells for query SELECT ... or Scanned over 5000 tombstones. These messages mean a single read is sifting through thousands of delete markers to return a small result set. Each tombstone consumes CPU, disk I/O, and heap memory during the merge phase. Left alone, the same query will eventually cross tombstone_failure_threshold (default 100000) and be aborted by the coordinator. The log line gives you the query text, but the real operational work is deciding whether the root cause is a missing repair, a compaction backlog, or a data model mismatch, and then confirming which partition is the actual source of the dead data.

Cassandra schema disagreement: nodetool describecluster shows multiple versions

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra schema disagreement: nodetool describecluster shows multiple versions

nodetool describecluster should report exactly one schema version UUID cluster-wide. When the Schema versions section lists more than one UUID, the cluster is in schema disagreement. DDL (CREATE, ALTER, DROP) fails or hangs until every node converges. DML and reads against existing tables are unaffected.

Transient disagreement lasting less than five minutes during or immediately after a DDL change is normal. Cassandra propagates schema mutations asynchronously via gossip. If multiple versions persist beyond five minutes, you have a stuck node, a partitioned peer, or a migration stage backlog.

Cassandra secondary index pitfalls: when 2i scatters across the ring

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra secondary index pitfalls: when 2i scatters across the ring

A query on an indexed column behaves differently in Cassandra than in a relational database. Without a partition key to anchor the request, the coordinator cannot hash the value to a token range and choose the right replicas. It fans out to every node in the cluster, waits for each local index scan to complete, and collates the results. Adding nodes can make indexed queries slower because scatter is linear with cluster size.

Cassandra snapshots silently consuming disk: hard links and clearsnapshot

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra snapshots silently consuming disk: hard links and clearsnapshot

df shows usage climbing toward 90%, but nodetool info Load does not explain the gap. If commitlog and hints are normal and compaction backlog is small, the missing space is likely a snapshot taken days or weeks ago. Cassandra snapshots use hard links, so they allocate no additional blocks at creation. After compaction deletes live SSTables, the snapshot links become the sole owners of old blocks and silently consume gigabytes.

Cassandra streaming failures: stalled bootstrap, decommission, and rebuild

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra streaming failures: stalled bootstrap, decommission, and rebuild

Bootstrapping stuck in JOINING for six hours. Decommission streaming with zero byte progress after eight hours. Rebuild failed, leaving incomplete token ranges. These are streaming failures.

Streaming moves SSTables between nodes during topology changes and repair. When a session fails, the topology is left incomplete. When it stalls with no progress for more than thirty minutes, the node stays in a transitional state and may not handle traffic correctly. Root causes are usually network timeouts, source node bottlenecks, or configuration mismatches. Unlike client request timeouts, streaming failures do not always surface as explicit errors; a session can hang silently while the control channel stays open.

Cassandra thread pool pending and blocked tasks: SEDA backpressure

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra thread pool pending and blocked tasks: SEDA backpressure

You run nodetool tpstats and see non-zero values in the Pending or Blocked columns. On a healthy Cassandra node, request-stage pools like MUTATION and READ should show zero pending tasks in steady state. When pending climbs and stays above zero, the SEDA pipeline is backing up. If Blocked also rises, the node has moved from queuing to rejecting work.

This is not a transient spike you can ignore. Sustained pending tasks on the MUTATION or READ stages add latency to every client request. Blocked tasks mean the queue is full and the node is actively shedding load. In the GOSSIP stage, even a small pending backlog is an emergency: it means gossip is falling behind, which leads to false DOWN marking across the cluster.

Cassandra tombstone storm: delete-heavy tables and read latency collapse

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra tombstone storm: delete-heavy tables and read latency collapse

DELETEs and TTL expirations write tombstones. Read latency bifurcates: P50 stays flat while P99 spikes. Logs show queries scanning thousands of tombstones; eventually clients see queries abort after crossing tombstone_failure_threshold. Disk space does not shrink despite deletions. This is a tombstone storm.

Cassandra does not remove deleted data immediately. A DELETE inserts a tombstone marker that persists until compaction purges it after gc_grace_seconds elapses and all replicas have been repaired. When tombstones scatter across many SSTables, every read must scan and merge them, generating temporary heap objects and escalating GC pressure. Only a subset of partitions may be affected, which is why P50 stays flat while tail latency explodes.

Cassandra TombstoneOverwhelmingException: reads aborted by tombstone_failure_threshold

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra TombstoneOverwhelmingException: reads aborted by tombstone_failure_threshold

TombstoneOverwhelmingException is a hard stop. Cassandra aborts the read after scanning more tombstones than tombstone_failure_threshold allows. The default is 100000 tombstones per query. When this exception hits a production table, client reads fail outright.

Without the threshold, a tombstone-heavy read pins cores, saturates disk I/O, and triggers GC pauses that cascade into gossip flapping or OOM. The exception is a circuit breaker. It is also a signal that tombstones are accumulating faster than compaction can purge them, or that the data model is generating them faster than the storage engine can remove them.

Cassandra Too many open files: file descriptor exhaustion

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra Too many open files: file descriptor exhaustion

Your application starts logging connection timeouts or Unable to connect errors, yet nodetool status still marks the node as UP. Inside the Cassandra system log, you see java.io.IOException: Too many open files or compaction tasks failing with Cannot open enough files. This is file descriptor exhaustion. The node operates normally until it hits the process ulimit, then it cannot open new SSTables, accept client sockets, or maintain internode connections. The result is a node that looks alive in gossip but is functionally unable to serve reads, writes, or background compaction.

Cassandra too many SSTables per table: read amplification and how to fix it

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra too many SSTables per table: read amplification and how to fix it

Your Cassandra read P99 latency is climbing. Queries that used to take milliseconds now time out. The application reports intermittent ReadTimeoutException. You check nodetool tablestats and see one table sitting at 200 SSTables. For LCS, that is a catastrophe. For STCS, anything sustained above 50 means compaction has fallen behind.

Each extra SSTable adds a bloom filter check and a potential disk seek to every read. The read path must merge fragments from memtables plus every SSTable that might contain the partition. When SSTable counts balloon, the node spends more time checking filters and seeking than returning data. This is read amplification.

Cassandra TTL tombstone accumulation: time-series tables and TWCS

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra TTL tombstone accumulation: time-series tables and TWCS

You set a TTL on every row expecting old data to vanish. Instead, disk usage climbs, read latency spikes, and your logs fill with tombstone warnings.

An expired TTL does not delete data immediately. Cassandra writes a tombstone that survives until compaction runs and the tombstone exceeds gc_grace_seconds (default 864000 seconds, or 10 days). In time-series workloads ingesting millions of TTL’d rows per hour, the wrong compaction strategy turns this bookkeeping into a production incident.

Cassandra UnavailableException: not enough replicas for the consistency level

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra UnavailableException: not enough replicas for the consistency level

Cassandra threw UnavailableException. The coordinator rejected the request immediately without contacting replicas. No mutation occurred, and retrying at the same consistency level against the same coordinator fails until enough replicas are UN in the coordinator’s gossip view. This is a topology problem, not a performance problem.

This exception is qualitatively different from TimeoutException. A timeout means enough replicas were alive but responded too slowly. An unavailable means the coordinator never sent the request because the topology could not satisfy the consistency level.

Cassandra WriteTimeoutException: coordinator write timeouts and writeType

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra WriteTimeoutException: coordinator write timeouts and writeType

A WriteTimeoutException means the coordinator did not receive enough replica acknowledgments before write_request_timeout_in_ms expired. It does not mean the write failed; one or more replicas may have already persisted the mutation, so the outcome is ambiguous. The writeType field determines whether a client-side retry is safe.

Distinguish between a slow replica and an unavailable one. Understand the idempotency contract for the write type, and correlate coordinator timeouts with replica-side saturation. The default write_request_timeout_in_ms is 2000 ms. If replicas cannot append to the commitlog and update the memtable within that window, the coordinator throws this exception.

Cassandra zombie data resurrection: gc_grace_seconds and unrepaired tombstones

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra zombie data resurrection: gc_grace_seconds and unrepaired tombstones

A query returns rows that were deleted weeks ago. Application logs show no errors. All nodes report UP in nodetool status. Compaction is running and disk usage looks normal. The data is back because a tombstone was compacted away before anti-entropy repair verified that every replica saw the delete. Once the tombstone is gone, live data on unrepaired replicas is treated as authoritative. The next repair or read repair streams it back to the tombstone-less nodes.

How Cassandra actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How Cassandra actually works in production: a mental model for operators

If you operate Cassandra in production, you are managing a distributed, partitioned, replicated log-structured merge tree where every node is a peer. There is no master to restart, no central query planner to tune, and no automatic load balancing that will save you from a hot partition. Every write is a sequential append to a commitlog and an in-memory update on multiple replicas. Every read is a merge of memtables and immutable SSTables, filtered by probabilistic bloom filters and reconciled by timestamp.