Cassandra snapshots silently consuming disk: hard links and clearsnapshot

df shows usage climbing toward 90%, but nodetool info Load does not explain the gap. If commitlog and hints are normal and compaction backlog is small, the missing space is likely a snapshot taken days or weeks ago. Cassandra snapshots use hard links, so they allocate no additional blocks at creation. After compaction deletes live SSTables, the snapshot links become the sole owners of old blocks and silently consume gigabytes.

This occurs after backup automation, schema changes, or emergency nodetool snapshot runs that are never cleaned up. With Size-Tiered Compaction Strategy (STCS), major compaction can already require up to 100% additional temporary space. A forgotten snapshot can push a node from comfortable to disk full without any warning in live data metrics.

flowchart TD
    A[Snapshot creates hard links to live SSTables] --> B[True size reported as zero]
    B --> C{Compaction replaces live SSTable?}
    C -->|No| D[Blocks shared with live data]
    C -->|Yes| E[Live directory entry removed]
    E --> F[Snapshot link now sole owner]
    F --> G[True size grows silently]
    G --> H[Disk usage increases without Load changing]

What this means

When you run nodetool snapshot, Cassandra creates hard links to current SSTable files inside <data_dir>/<keyspace>/<table>/snapshots/<tag>/. The snapshot and live SSTable share the same inode, so no data is copied and no additional disk blocks are allocated. nodetool listsnapshots reflects this by showing a True size near zero while the original SSTable still exists.

As compaction runs and merges SSTables, the live versions are deleted. Because the snapshot directory still holds a hard link to the original inode, the blocks are not freed. The snapshot True size grows to match the space it exclusively owns. This consumption is invisible to nodetool info Load, which counts only live data files and excludes snapshots, commitlog, hints, and compaction temporary files. A single forgotten snapshot on a 50 GB table can eventually hold 50 GB of dead data, and multiple snapshots compound the problem.

Common causes

CauseWhat it looks likeFirst thing to check
Forgotten manual or backup snapshotsGradual disk growth after a backup window; old tags in listsnapshotsnodetool listsnapshots
Auto-snapshot on DROP or TRUNCATESudden disk jumps after schema changes; tags prefixed with dropped- or timestampsnodetool listsnapshots
Incremental backups without cleanupbackups/ directories grow indefinitely; space not reclaimed by clearsnapshotfind /var/lib/cassandra/data -name backups -type d
Orphaned snapshots from dropped tableslistsnapshots underreports usage; snapshot directories exist for tables no longer in the schemafind /var/lib/cassandra/data -name snapshots -type d

Quick checks

# List all snapshots and their true sizes
nodetool listsnapshots

# Check whether auto-snapshot is enabled (default true)
grep "auto_snapshot" /etc/cassandra/cassandra.yaml

# Check whether incremental backups are enabled (default false)
grep "incremental_backups" /etc/cassandra/cassandra.yaml

# Find snapshot directories, including orphans from dropped tables
find /var/lib/cassandra/data -name snapshots -type d

# Check keyspace-level disk usage
du -sh /var/lib/cassandra/data/*/

How to diagnose it

  1. Compare nodetool info Load to df output. If the filesystem shows significantly more usage, check snapshots, incremental backups, and compaction temp files.
  2. Run nodetool listsnapshots. High True size means the snapshot exclusively owns blocks that would otherwise be free.
  3. Identify old tags. Snapshots older than your retention window are deletion candidates.
  4. Check for orphaned snapshots. Run find /var/lib/cassandra/data -name snapshots -type d and inspect subdirectories. Snapshot directories for table UUIDs that no longer exist in the schema are leftovers from dropped tables. nodetool listsnapshots does not show snapshots for dropped tables, so they are invisible to standard tooling.
  5. Check cassandra.yaml for auto_snapshot and incremental_backups. If incremental_backups is true, the backups/ subdirectories grow indefinitely unless you manage them externally.
  6. Correlate the start of disk growth with snapshot creation or compaction. Snapshots created before major compaction grow in True size once it finishes.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Disk space available (df)Snapshots consume physical blocks not reflected in Cassandra’s Load metricFree space declining while live data size is stable
nodetool listsnapshots True sizeMeasures disk blocks held exclusively by the snapshotTrue size large or growing week over week
nodetool info Load vs. filesystem usedLoad excludes snapshots; a large gap indicates hidden consumptiondf used significantly larger than Load
Pending compactionsCompaction deletes live SSTables and converts shared hard links into snapshot-owned blocksPending compactions high with simultaneous unexplained disk growth
auto_snapshot enabledCreates automatic snapshots on every DROP and TRUNCATEUnexpected snapshot tags appearing after schema changes

Fixes

Remove forgotten snapshots with clearsnapshot

Warning: This permanently deletes snapshot data. Do not remove snapshots that have not been backed up or verified.

Remove all snapshots on the node:

# Remove all snapshots (destructive)
nodetool clearsnapshot --all

Remove a specific snapshot by tag:

# Remove one snapshot tag from a specific keyspace (destructive)
nodetool clearsnapshot -t <tag> <keyspace>

If your version supports it, remove snapshots older than a retention window:

# Remove snapshots older than 7 days (destructive)
nodetool clearsnapshot --older-than 7d

Clean up orphaned snapshots from dropped tables

nodetool listsnapshots does not show snapshots for dropped tables. Identify orphaned directories with find /var/lib/cassandra/data -name snapshots -type d, inspect the contents, and remove snapshot directories for tables that no longer exist. This is destructive and irreversible.

nodetool clearsnapshot does not remove hard links created by incremental_backups: true. These accumulate in backups/ subdirectories under each table directory. You must script rotation or removal manually. Leaving incremental_backups enabled without cleanup will fill disks predictably.

Disable automatic snapshot creation

If you frequently DROP or TRUNCATE and do not need the safety net, set auto_snapshot: false in cassandra.yaml. This requires a rolling restart. The tradeoff is that dropped or truncated tables will not be recoverable from local snapshots. With auto_snapshot: true, TRUNCATE operations wait for the snapshot to complete before returning.

Use snapshot TTL

If you are on Cassandra 4.1 or later, use nodetool snapshot --ttl <seconds> to set an expiration on new snapshots. This eliminates manual cleanup for snapshots that do not need indefinite retention.

Prevention

  • Automate nodetool clearsnapshot after backup verification completes. Never leave snapshots on production nodes longer than necessary.
  • Monitor nodetool listsnapshots True size as a time series. A steady upward trend is an early warning.
  • If you enable incremental_backups, implement an external rotation job that deletes old hard links from backups/ directories. clearsnapshot will not do this for you.
  • Maintain a snapshot naming convention that includes creation dates in the tag, making it easy to identify stale snapshots.

How Netdata helps

  • Disk space alerts on Cassandra data directories trigger before compaction headroom is breached, giving time to clear snapshots before writes stall.
  • Correlation of disk usage with pending compactions and SSTable count helps distinguish snapshot growth from compaction backlog.
  • Tracking filesystem usage independently of Cassandra Load surfaces hidden consumption from snapshots or incremental backups.
  • Long-term retention of disk usage trends makes it easy to correlate the start of growth with backup schedules or schema changes.