Cassandra monitoring checklist: the signals every production cluster needs

The four maturity levels below are cumulative. Do not instrument level 2 until level 1 is visible and alerted.

Cassandra’s peer-to-peer architecture and LSM storage engine create failure modes generic infrastructure monitoring misses. A node can be UP in gossip and accepting CQL connections while dropping mutations or accumulating compaction debt that only surfaces hours later. These signals catch liveness, performance, saturation, and consistency failures: GC death spirals, compaction avalanches, tombstone storms, and silent data divergence.

Level 1 is survival. Levels 2 and 3 add diagnostics and leading indicators. Level 4 is for capacity planning and post-incident analysis.

flowchart TD
    L4["Level 4: Expert
Runway projections, per-partition telemetry, heap-after-GC trends"]
    L3["Level 3: Mature
Repair tracking, tombstone density, off-heap, speculative retries"]
    L2["Level 2: Operational
Latency p99, compaction, thread pools, GC pauses, hints"]
    L1["Level 1: Survival
UN status, native transport, disk space, heap, dropped messages"]

    L1 --> L2
    L2 --> L3
    L3 --> L4

Level 1 - Survival

These six signals tell you if the node is alive, reachable, and has room to breathe. They require only nodetool, filesystem checks, and JMX. If any fire, the cluster is failing or one event away from stopping writes. Check these first during an incident. A node that is DN in nodetool status but still pings is likely in GC thrash or disk hang. Native transport stopped while gossip is UP usually means CQL port bind failure or excessive heap pressure.

Level 2 - Operational

These signals explain why a node is struggling. Coordinator latency, error rates, and compaction state turn “the node is slow” into actionable diagnostics. Thread pool backpressure and disk I/O latency separate CPU problems from storage problems. Check these before tuning caches or adding capacity. P99 latency approaching read_request_timeout_in_ms means requests are about to fail outright. Blocked thread pools indicate backpressure failure and imminent dropped messages.

Level 3 - Mature

These are leading indicators. Repair tracking prevents data resurrection. Tombstone density and bloom filter accuracy catch data model degradation before it becomes an outage. Off-heap monitoring closes the gap that causes OOM kills despite a “healthy” JVM heap. Tombstone scans above 1,000 per read indicate a missing filter or TTL design flaw. Bloom filter false positives above fp_chance mean the filter is undersized or data distribution has changed.

[OUTPUT TRUNCATED: Response exceeded output token limit.]