Cassandra OperationTimedOutException: client-side timeouts vs server timeouts

OperationTimedOutException (driver 3.x) or DriverTimeoutException (driver 4.x) means the driver gave up before the coordinator responded. This is distinct from a server-side ReadTimeout or WriteTimeout, where the coordinator explicitly errors because replicas were too slow. Confusing the two leads to tuning the wrong timeout, masking a server capacity problem or adding unnecessary client latency. Use server-side signals and driver behavior to tell them apart.

When the driver timer fires first, the coordinator may still be processing the request. For writes, the mutation can be committed on some replicas even though the client sees an exception. When the server timer fires first, the coordinator returns a READ_TIMEOUT or WRITE_TIMEOUT to the driver. The diagnostic path and fix differ for each case.

flowchart LR
  subgraph ClientSide ["Client JVM"]
    App["Application"]
    Driver["Driver Request Timer"]
  end

  subgraph ServerSide ["Cassandra Cluster"]
    Coord["Coordinator"]
    R1["Replica"]
    R2["Replica"]
  end

  App -->|"1. execute()"| Driver
  Driver -->|"2. CQL request"| Coord
  Coord -->|"3a. replica requests"| R1
  Coord -->|"3b. replica requests"| R2

  Driver -->|"4a. timer expires
client-side timeout"| App
  Coord -->|"4b. READ_TIMEOUT /
WRITE_TIMEOUT response"| Driver

What this means

Client-side timeout: The driver starts a timer when sending a request. If it expires before the coordinator responds, the driver throws. In 3.x this is OperationTimedOutException; the retry policy invokes onRequestError. In 4.x it is DriverTimeoutException. Defaults shifted significantly between versions: 3.x defaults to roughly 12 seconds, 4.x defaults to 2 seconds.

Server-side timeout: The coordinator waits for enough replicas to satisfy the consistency level. If that does not happen within the configured timeout, the coordinator returns READ_TIMEOUT or WRITE_TIMEOUT. The driver surfaces this to onReadTimeout or onWriteTimeout. The coordinator responded; the driver is not guessing.

UnavailableException: If too few replicas are alive to attempt the request, the coordinator fails immediately. This is not a timeout; the fix is node recovery, not timeout tuning.

The driver timeout must exceed the relevant server-side timeout plus network round-trip time. If it is lower, the driver blames Cassandra while the server is still within its own limits.

Common causes

CauseWhat it looks likeFirst thing to check
Driver timeout shorter than server timeoutTimeouts spike under load but server ClientRequest Timeouts stay flatDriver request timeout configuration against read_request_timeout_in_ms / write_request_timeout_in_ms in cassandra.yaml
Replica slowness (GC, I/O saturation)Server ClientRequest Timeouts increasing, high pending tasks or GC pausesnodetool tpstats, GC logs, disk I/O
Network latency between application and coordinatorDriver timeouts correlate with RTT spikes, no replica saturationping, mtr, or TCP RTT on the client-to-coordinator path
Coordinator node failureDriver timeouts cluster on one coordinator IP, node may be DOWN in gossipnodetool status and driver logs for errors to a specific IP
Driver 4.x non-idempotent write behaviorWrite timeouts are not automatically retried by the default policyQuery idempotence flags and retry policy configuration

Quick checks

These read-only checks help determine which side fired first.

# Server-side timeout counters (JMX)
# org.apache.cassandra.metrics:type=ClientRequest,scope=Read,name=Timeouts
# org.apache.cassandra.metrics:type=ClientRequest,scope=Write,name=Timeouts

# Replica saturation and dropped messages
nodetool tpstats

# Coordinator latency percentiles (microseconds)
nodetool proxyhistograms

# Node liveness across the cluster
nodetool status

# Recent stop-the-world GC pauses
grep -i "pause" /var/log/cassandra/gc.log | tail -20

How to diagnose it

  1. Check server-side timeout metrics. Query JMX for org.apache.cassandra.metrics:type=ClientRequest,scope=Read,name=Timeouts and the corresponding Write scope. If these counters are increasing, the coordinator fired first and the driver is surfacing a server-side error. If they are flat while the application sees timeouts, the driver timer fired first.

  2. Compare coordinator latency to timeout walls. Use nodetool proxyhistograms to read coordinator-level latency percentiles. If p99 read latency is approaching half of read_request_timeout_in_ms (default 5000 ms) or spiking beyond it, the server is the bottleneck. If coordinator latency is well below the driver timeout but driver timeouts still occur, the network or the driver threshold is the problem.

  3. Inspect replica saturation. On each replica, run nodetool tpstats and look at ReadStage and MutationStage pending tasks. Sustained pending greater than zero means the replica cannot keep up. Also check CompactionExecutor pending; if compaction is falling behind, reads slow down due to amplification.

  4. Check GC pause duration. Long pauses freeze replicas. Inspect GC logs or JMX GarbageCollector MBeans for collection times. Pauses greater than 2 seconds directly drive coordinator timeouts because replicas stop responding.

  5. Check for node DOWN states. Run nodetool status. Unavailable exceptions are distinct from timeouts, but a DOWN node shrinks the replica set and can overload the remaining nodes, causing them to time out.

  6. Compare driver timeout configuration to server timeouts. The driver-side request timeout should be larger than the relevant server-side timeout (read_request_timeout_in_ms, write_request_timeout_in_ms, range_request_timeout_in_ms) plus the estimated network round-trip time between the application and the coordinator.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
ClientRequest Timeouts (Read/Write)Counts server-side coordinator timeoutsRate greater than 0 sustained
ClientRequest UnavailablesDistinguishes timeout from quorum lossImmediate failures, not slowness
Coordinator read/write latency p99Shows if traffic is approaching timeout thresholdsp99 greater than server timeout divided by 2
Thread pool pending tasks (Read/Mutation)Backpressure on replicasPending greater than 0 sustained
GC pause durationLong pauses freeze replicas and drive timeoutsMax pause greater than 2 seconds
Dropped messages (MUTATION/READ)Server is shedding load before timeoutsNon-zero rate
Driver request timeout rateClient-side perspective for correlationSpikes without matching server metric spikes

Fixes

Driver timeout is lower than server timeout

Increase the driver request timeout so it exceeds the relevant server timeout plus network RTT. In 4.x, adjust the global request timeout. In 3.x, adjust the socket read timeout. Tradeoff: clients wait longer. Ensure connection pools and circuit breakers accommodate the increase. Re-verify defaults after any major driver upgrade.

Server-side replica slowness

If server ClientRequest Timeouts are rising, do not increase the driver timeout without first reducing replica latency. Check for GC pressure, compaction debt, or disk I/O saturation. Increase compaction_throughput_mb_per_sec or add IOPS capacity if compaction is the bottleneck. Increase concurrent_compactors if CPU headroom exists. See Cassandra GC death spiral: long pauses, gossip flapping, and recovery and Cassandra compaction death spiral: when writes outrun compaction throughput.

Non-idempotent writes and driver retries

In driver 4.x, the default retry policy will not retry DriverTimeoutException for non-idempotent queries. This prevents duplicate writes. If your application sees write timeouts that are not retried, mark writes as idempotent when they are safe to replay, or implement explicit application-level handling. Tradeoff: automatic retries on non-idempotent writes risk duplicate data.

Netty timer issues in older driver releases

If you are running driver releases that include Netty 4.1.77 through 4.1.86, a HashedWheelTimer race condition can cause spurious client-side timeouts. Upgrade the Netty dependency or the driver to a fixed version.

Node failure and in-flight requests

When a node goes down, in-flight requests to that coordinator fail with client-side timeouts. The driver eventually marks the node DOWN via gossip, but the in-flight requests are lost. Ensure your application handles these explicitly rather than depending on automatic retry for non-idempotent operations.

Prevention

  • Keep driver timeout greater than server timeout plus network RTT. Re-verify after driver upgrades; defaults shift dramatically between major versions.
  • Monitor server-side ClientRequest Timeouts and Unavailables alongside driver exceptions. Alerting on only one side creates a blind spot.
  • Keep coordinator read latency p99 below half of the server read timeout to leave headroom for tail latency and jitter.
  • Watch ThreadPools pending tasks and GC pause duration as leading indicators. Server-side timeouts follow resource saturation; driver timeouts can appear without warning if the threshold is simply too aggressive.

How Netdata helps

Netdata correlates coordinator read/write latency with ClientRequest timeout and unavailable rates, showing whether the driver or the server fired first. It tracks JVM GC pauses and thread pool pending tasks per node to highlight replica slowness before it becomes client-visible. Disk I/O utilization per device separates commitlog contention from data directory saturation. Per-node timeout and unavailable metrics are shown alongside application error rates.