Cassandra streaming failures: stalled bootstrap, decommission, and rebuild

Bootstrapping stuck in JOINING for six hours. Decommission streaming with zero byte progress after eight hours. Rebuild failed, leaving incomplete token ranges. These are streaming failures.

Streaming moves SSTables between nodes during topology changes and repair. When a session fails, the topology is left incomplete. When it stalls with no progress for more than thirty minutes, the node stays in a transitional state and may not handle traffic correctly. Root causes are usually network timeouts, source node bottlenecks, or configuration mismatches. Unlike client request timeouts, streaming failures do not always surface as explicit errors; a session can hang silently while the control channel stays open.

What this means

Cassandra uses streaming to transfer SSTable data during bootstrap, decommission, rebuild, and repair. In Cassandra 4.0+, Zero Copy Streaming (ZCS) transfers entire SSTable files directly off disk, bypassing object reification and reducing CPU overhead versus the partition-based path. A streaming session is a long-lived TCP connection with a control channel and one or more data channels.

A failed session is detected when the control channel closes or two consecutive keep-alive cycles fail. A stalled session is different: the connection is alive, but bytes stop moving. If no progress occurs for more than thirty minutes, the operation is frozen. The node remains in a transitional state (JOINING, LEAVING, or REBUILDING) and may not accept or redirect traffic correctly. There is no automatic abort timeout for a stalled bootstrap; the operator must choose between resume, restart, or a full wipe and retry.

Common causes

Quick checks

# Check active streaming sessions and progress
nodetool netstats

# Check cluster topology and node states
nodetool status

# List active SSTable tasks including streaming (Cassandra 4.0+)
cqlsh -e "SELECT keyspace_name, table_name, task_id, total, completed FROM system_views.sstable_tasks;"

# Search logs for streaming errors or bootstrap state
grep -iE "streaming|bootstrap|decommission|Some data streaming failed" /var/log/cassandra/system.log

# Check streaming throughput trend via JMX
# TotalIncomingBytes / TotalOutgoingBytes should change between samples

# Verify streaming tunables in configuration
grep -E "stream_throughput_outbound_megabits_per_sec|streaming_socket_timeout" /etc/cassandra/cassandra.yaml

# Check if internode encryption is enabled (disables ZCS)
grep -A 5 "server_encryption_options" /etc/cassandra/cassandra.yaml

How to diagnose it

flowchart TD
    A[Streaming stalled >30 min] --> B{nodetool netstats active?}
    B -->|Yes| C{Bytes moving?}
    B -->|No| D[Check logs for session close]
    C -->|No| E[Check firewall vs keep-alive period]
    C -->|Yes slowly| F[Check throttle and source node IO]
    D --> G[Check for DOWN replicas]
    E --> H[Fix network timeout]
    F --> I[Raise throttle or relieve source]
    G --> J[Resume or override bootstrap]

1. Confirm the operation and state. Run nodetool status to check if the node is JOINING, LEAVING, or NORMAL. Run nodetool netstats to confirm active streaming sessions and whether bytes are moving.

2. Determine if the stream is failed or stalled. A failed session disappears from nodetool netstats and usually leaves an error in system.log. A stalled session remains listed but shows no byte progress for more than thirty minutes. On Cassandra 4.0+, query system_views.sstable_tasks to confirm whether active streaming work is registered.

3. Check for network layer drops. Compare the firewall or NAT idle timeout against the streaming keep-alive period. If the firewall timeout is shorter than 600 seconds (two keep-alive cycles), the firewall drops the connection even when the stream is healthy but slow. This is a common cause of mysterious streaming interruptions.

4. Verify throughput expectations. Calculate the dataset size divided by the configured stream_throughput_outbound_megabits_per_sec (default 200 Mbps). If the result is many hours, the operation may not be stalled; it may simply be throttled. However, if nodetool netstats shows no byte increase over a 15-minute window while the session is open, the stream is stalled regardless of throttle.

5. Inspect the source node. Streaming competes with compaction, flushes, and client traffic on the source. Check the source node for pending compactions, blocked thread pools, long GC pauses, and disk IO wait (iostat -x). If the source is in a GC death spiral or compaction backlog, it cannot feed the stream.

6. Check for Zero Copy Streaming fallback. In Cassandra 4.0+, ZCS is enabled by default, but it is automatically disabled when internode_encryption is enabled. There is no warning in nodetool netstats. If encryption is on, expect slower partition-based streaming and plan longer windows.

7. Evaluate CDC and materialized view overhead. If CDC or materialized views are enabled, streamed data may be written through the commitlog on the target. This can significantly slow streaming compared to direct SSTable ingestion.

8. Review timeout configuration. In Cassandra 5.0 and later, streaming_socket_timeout_in_ms is absent. The equivalent control is the internode streaming TCP user timeout parameter. If an operator set the old parameter name, it has no effect. Similarly, the streaming keep-alive period controls the control-channel heartbeat.

9. Check for consistent bootstrap blocking. Since Cassandra 3.0, bootstrap requires all replicas for a token range to be available. If any replica is DOWN, the bootstrap will fail or hang. You can override this with -Dcassandra.consistent.rangemovement=false, but the joining node may miss data from the unavailable replica.

Metrics and signals to monitor

Fixes

Resume or restart a stalled bootstrap

On Cassandra 2.2 and later, a stalled bootstrap can often be resumed:

# Attempt to resume from saved state
nodetool bootstrap resume

If the node resumes but fails again, inspect the source node and network first. Restarting the node also resumes bootstrap automatically in most versions. If the bootstrap state is corrupted and you must start fresh, wipe the progress and restart:

# Wipe bootstrap state