CockroachDB Operations Guides on Netdata

CockroachDB /health?ready=1: load balancer checks, draining, and impaired nodes

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB /health?ready=1: load balancer checks, draining, and impaired nodes

CockroachDB exposes a readiness endpoint at GET /health?ready=1 on its HTTP port (default 8080). Load balancers and Kubernetes readiness probes use it to decide whether to route SQL traffic to a node. The node returns HTTP 200 when ready and HTTP 503 when not. This signal separates “process is alive” from “this node should receive client connections.”

The most common operator mistake is using a plain TCP check against the SQL port (26257) or the plain /health endpoint instead of /health?ready=1. A TCP check succeeds as long as the port is listening, telling you nothing about whether the node is draining, write-stalled, or GC-thrashing. The plain /health endpoint returns 200 whenever the process is running, regardless of draining state. Neither is safe for routing decisions.

CockroachDB admission control throttling: queue depth, store-write, and capacity headroom

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB admission control throttling: queue depth, store-write, and capacity headroom

When admission control starts queuing requests, p99 latency climbs while throughput stays flat or degrades. The cluster hasn’t crashed, disks aren’t full, and CPU may not be saturated. An internal flow-control system decided the node or store is at capacity and started holding work back to protect itself.

Admission control (v21.2+, enabled by default since v22.1) regulates work through five queues: kv, sql-kv-response, sql-sql-response, elastic-cpu, and store-write. Each gates a different class of work with distinct triggers. Knowing which queue is deep and why is the difference between a five-minute diagnosis and a multi-hour investigation into “why is the database slow.”

CockroachDB backup job failures: RPO breaches, duration trends, and stuck jobs

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB backup job failures: RPO breaches, duration trends, and stuck jobs

CockroachDB scheduled backups run through the internal jobs system, visible via crdb_internal.jobs (backed by system.jobs). When a scheduled backup fails silently, stalls indefinitely, or grows so slowly that it cannot complete within its interval, your recovery point objective (RPO) is at risk. The failure is insidious: backups often appear healthy until you realize the last successful completion was 36 hours ago.

CockroachDB certificate expired: TLS handshake failures and online rotation

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB certificate expired: TLS handshake failures and online rotation

CockroachDB uses mutual TLS for every connection: node-to-node, client-to-node, and admin UI. There is no plaintext fallback and no grace period. When a certificate expires, affected connections fail immediately.

Node certificate expiry prevents nodes from completing TLS handshakes with each other, which can look like a network partition or quorum loss. Client certificate expiry prevents applications from connecting. CA certificate expiry invalidates the entire trust chain at once, breaking every connection simultaneously.

CockroachDB changefeed lag: changefeed_max_behind_nanos and the GC time bomb

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB changefeed lag: changefeed_max_behind_nanos and the GC time bomb

Changefeed lag in CockroachDB starts as a consumer latency problem and ends as a disk space emergency. The changefeed_max_behind_nanos gauge measures how far behind your CDC feeds have fallen, but the real danger is what follows: a stalled changefeed holds a protected timestamp that prevents MVCC garbage collection, and dead data accumulates silently until the disk fills. The cluster appears healthy from the SQL side. Queries are fast, nodes are up, ranges are available. But every deleted row and every overwritten value stays on disk because GC cannot reclaim the space.

CockroachDB clock skew cascade: how shared NTP drift causes quorum loss

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB clock skew cascade: how shared NTP drift causes quorum loss

Multiple CockroachDB nodes crashed overnight. The logs show “clock synchronization error: this node is more than 500ms away from at least half of the known nodes.” You restart them, and they crash again. Some ranges are now unavailable. The cluster is losing quorum.

A shared NTP failure caused multiple nodes to drift past CockroachDB’s self-termination threshold in quick succession. Single-node clock skew is bad but recoverable. Multi-node skew from a shared NTP source can take down quorum faster than the cluster can heal.

CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes

You see this fatal log line on a CockroachDB node:

clock synchronization error: this node is more than 500ms away from at least half of the known nodes

The process exits immediately. If the node is managed by systemd, Kubernetes, or a process supervisor, it restarts and crashes again. The crash-loop continues until the clock problem is fixed.

CockroachDB clock_offset_meannanos high: catching clock drift before self-termination

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB clock_offset_meannanos high: catching clock drift before self-termination

clock_offset_meannanos measures the mean clock offset between a CockroachDB node and its peers. When it climbs, you are on a path that ends in silent performance degradation from widened read uncertainty windows, or a node self-terminating to preserve data consistency.

The thresholds are unforgiving. CockroachDB uses a default --max-offset of 500ms. A node self-terminates when its mean offset exceeds 80% of that value (400ms) relative to a majority of peers. But a constant 200ms offset, stable and below any threshold, still doubles the uncertainty interval for every read. Transactions silently restart more often, P99 read latency creeps up, and nobody suspects the clock.

CockroachDB compaction backlog growing: when Pebble can't keep pace with writes

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes

Pebble’s background compaction threads sometimes fall behind foreground writes. In CockroachDB, this does not cause immediate failure. SSTable files accumulate in Level 0 and the compaction queue, read amplification rises, and the node drifts toward write stalls. Because the database continues to serve traffic, the backlog is easy to miss until it becomes severe.

The earliest visible sign is a gentle upward trend in Level 0 file counts or marked-for-compaction files over hours. These metrics fluctuate with workload bursts, but a sustained upward slope means the store is consuming headroom. Healthy operation requires compaction throughput at least twice the write ingestion rate. Less than that leaves no margin for bursts, MVCC garbage collection, or rebalancing.

CockroachDB connection storm after failover: reconnect stampedes and surviving-node overload

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB connection storm after failover: reconnect stampedes and surviving-node overload

When a CockroachDB node dies, every client connected to it reconnects at the same time. Without jittered backoff in client connection pools, hundreds or thousands of new connections land on the surviving nodes within seconds. The survivors are not overwhelmed by additional query load. They are overwhelmed by connection overhead: per-connection goroutines, session memory allocations, TLS handshakes, and SQL planner initialization.

CockroachDB consistency check failed: checksum mismatches and confirmed corruption

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB consistency check failed: checksum mismatches and confirmed corruption

When the background consistency checker detects a checksum mismatch between replicas of a range, CockroachDB logs a fatal error and the node terminates. This is not a transient condition. The database is telling you that data integrity is compromised.

The definitive message: “consistency check failed with N inconsistent replicas” or a Pebble-level “checksum mismatch” error. Both originate from CockroachDB’s consistency checker subsystem, not from unrelated log noise. This article covers the failure mechanism, the recovery path, and the signals that help you scope the damage.

CockroachDB context deadline exceeded: timeouts, slow ranges, and overload

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB context deadline exceeded: timeouts, slow ranges, and overload

“context deadline exceeded” tells you something is slow. It does not tell you what. The same error appears whether a leaseholder is overloaded, a network path is congested, L0 sublevels are climbing past write-stall territory, or the application set a 2-second statement timeout on a query that normally takes 50 milliseconds.

Treat the error as a symptom, not a diagnosis. The diagnostic question is: which layer is slow?

CockroachDB CPU saturation: Raft ticking, SQL execution, and the per-node ceiling

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB CPU saturation: Raft ticking, SQL execution, and the per-node ceiling

CockroachDB is CPU-hungry by design. Every range runs its own Raft state machine. Every SQL statement parses, plans, and executes through Go. Compaction, encryption-at-rest, and checksumming all burn cycles. When CPU saturates, the failure mode is not a clean slowdown: Raft heartbeats get delayed, admission control starts queuing, GC pauses lengthen, and the node risks losing liveness. The cluster average can look healthy while one leaseholder melts.

CockroachDB detecting hot ranges: per-range QPS, CPU asymmetry, and the Hot Ranges page

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB detecting hot ranges: per-range QPS, CPU asymmetry, and the Hot Ranges page

Hot ranges are the most common performance bottleneck in CockroachDB that does not surface in aggregate metrics. The leaseholder model routes all reads and writes for a range through a single node. When one range receives disproportionate traffic, that node saturates while the rest of the cluster idles. The cluster-wide CPU average looks healthy. The per-node breakdown tells a different story.

CockroachDB disk space running out: capacity_available trends and the 20% rule

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB disk space running out: capacity_available trends and the 20% rule

When a CockroachDB store runs out of disk space, the node cannot accept writes, cannot compact its LSM tree (the operation that would reclaim space), and enters a downward spiral that typically requires operator intervention. The capacity_available metric tracks remaining free space per store, but raw free space alone does not tell the full story. MVCC garbage, protected timestamps, compaction space amplification, and replica rebalancing all consume space faster than headline data growth suggests.

CockroachDB disk stall detected: storage_disk_stalled and node self-termination

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB disk stall detected: storage_disk_stalled and node self-termination

When storage_disk_stalled goes nonzero on a CockroachDB node, the Pebble storage engine has detected that disk I/O is no longer completing within the expected time window. The node is on a countdown to self-termination. This is not a performance degradation warning. It is a safety mechanism preparing to fire.

CockroachDB writes every committed transaction through a write-ahead log (WAL). The WAL fsync is on the critical path for every write: Raft cannot acknowledge a commit until the log entry is persisted to disk. If that fsync blocks for long enough, the node cannot process Raft heartbeats, cannot commit writes, and cannot renew its liveness record. Rather than continue operating in a state that could produce data inconsistency, CockroachDB terminates the process.

CockroachDB error 53200: SQL memory budget exhausted and query rejection

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB error 53200: SQL memory budget exhausted and query rejection

Error 53200 is CockroachDB’s PostgreSQL-compatible signal that the per-node SQL memory budget has run out. Applications see SQLSTATE 53200 (“insufficient resources”) with text such as “memory budget exceeded” and byte counts showing what was requested, what is allocated, and what the budget allows.

The SQL memory budget is enforced per-node, bounded by --max-sql-memory (default 25% of system RAM). When a node’s SQL execution layer exhausts its budget, queries on that node either spill to temporary disk storage (slow) or are rejected outright with 53200. One large analytical query on a single gateway node can starve every other session connected to that node, even if the rest of the cluster has memory to spare.

CockroachDB file descriptor exhaustion: SSTables, connections, and ulimit

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB file descriptor exhaustion: SSTables, connections, and ulimit

When CockroachDB hits its file descriptor limit, multiple subsystems fail at once. New SQL client connections are refused. SSTable file opens fail with I/O errors. Inter-node gRPC connections fail, destabilizing Raft consensus. The node may crash, stall, or refuse to start. In the logs you will see “too many open files” errors and, if the limit is below the startup minimum, the message “open file descriptor limit of X is under the minimum required Y”.

CockroachDB Go GC pauses high: when garbage collection threatens Raft heartbeats

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB Go GC pauses high: when garbage collection threatens Raft heartbeats

GC pause time spiking, node liveness flapping, lease transfers churning. SQL latency oscillates between acceptable and terrible. The cluster is alive but unstable, and the oscillation pattern is the tell.

CockroachDB runs on the Go runtime, which performs stop-the-world garbage collection pauses. When those pauses grow long enough, they block the Raft heartbeat loop. The node stops responding to heartbeats for the duration of the pause. If the pause is long enough, the cluster declares the node dead and redistributes its leases. Then the pause ends, the node recovers, leases come back, the heap grows again, and the cycle repeats.

CockroachDB hot range bottleneck: one leaseholder saturated while the cluster idles

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB hot range bottleneck: one leaseholder saturated while the cluster idles

One node runs hot while the rest of the cluster idles. CPU on that node is 80%+, but others hover around 20-30%. SQL latency is elevated, but only for specific tables. Transaction retry rates climb with writetooold as the dominant restart cause. No range is unavailable, no node has lost liveness, and disk I/O is within normal bounds. The cluster has aggregate capacity, but a single range is funneling all its traffic through one leaseholder.

CockroachDB intent accumulation cascade: abandoned transactions and intentcount growth

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB intent accumulation cascade: abandoned transactions and intentcount growth

When intentcount and intentbytes climb and refuse to come down, your cluster is accumulating unresolved write intents from transactions that never committed or rolled back. Every subsequent transaction touching those keys must stop, resolve the intent, then proceed. At scale, the cluster spends more CPU and I/O resolving old intents than executing new work.

This is the intent accumulation cascade. It is one of CockroachDB’s subtler failure modes because the cluster looks healthy on infrastructure metrics: nodes are live, ranges are available, disk I/O is within bounds. The damage shows up in transaction latency and throughput, not in availability signals.

CockroachDB LSM compaction death spiral: L0 sublevels, read amplification, and write stalls

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB LSM compaction death spiral: L0 sublevels, read amplification, and write stalls

SQL P99 latency jumps from milliseconds to seconds. KV write latency climbs. Nodes transfer leases. Logs show Pebble write stall messages. This is the LSM compaction death spiral: writes outpace the storage engine’s ability to compact data from Level 0 down the LSM tree. L0 sublevels stack up, read amplification rises, and the node eventually stalls writes to protect itself. By the time write stalls appear, the node is already at risk of losing Raft leases and appearing partially unavailable. This guide shows how to diagnose the spiral, stop it, and prevent it.

CockroachDB memory pressure, GC thrashing, and Raft liveness failure

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB memory pressure, GC thrashing, and Raft liveness failure

A CockroachDB node loses liveness in a repeating pattern: it drops out, the cluster redistributes its leases, it recovers, reacquires leases, then drops again. Each cycle lasts seconds to minutes. Application queries see intermittent timeouts, ambiguous results, and latency spikes that correlate with the node’s oscillation. The DB Console shows the node flapping between live and not-live states.

This is the memory pressure to GC thrashing to Raft liveness failure cascade. The Go runtime heap grows until garbage collection pauses become long enough to prevent the node from renewing its liveness heartbeat. Once the heartbeat interval lapses, the cluster marks the node dead and moves its leases. When GC completes and memory is freed, the node recovers and reacquires leases, restarting the cycle.

CockroachDB monitoring checklist: the signals every production cluster needs

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB monitoring checklist: the signals every production cluster needs

CockroachDB layers SQL execution on a replicated KV store backed by Pebble LSM trees, Raft consensus, and MVCC concurrency control. Each subsystem has distinct failure modes, and interactions between them create cascades that single-signal monitoring cannot catch. A cluster can show healthy CPU, adequate disk space, and sub-millisecond SQL latency while L0 sublevels climb toward write stalls or clock offset drifts toward the self-termination threshold.

CockroachDB monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB monitoring maturity model: from survival to expert

CockroachDB failures rarely announce themselves through a single metric. A slow disk turns into L0 compaction debt, which stalls writes, which drops Raft proposals, which makes ranges unavailable. A drifting clock raises transaction retry rates long before any node self-terminates. To run this database safely, you need layered observability that matches the system’s own layers: storage engine, Raft replication, distributed SQL, and transaction execution.

CockroachDB MVCC garbage growing: tombstones, GC lag, and silent disk growth

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB MVCC garbage growing: tombstones, GC lag, and silent disk growth

MVCC garbage accumulation produces no errors, no latency spikes, and no user-visible symptoms until the disk fills. By then, you may be hours away from a compaction death spiral that takes the node or cluster offline.

CockroachDB retains old MVCC versions of every key until garbage collection removes them after the GC TTL window (default 25 hours). Deletes and updates create tombstones: markers that a key was removed at a specific timestamp. These tombstones persist in the LSM tree until MVCC GC runs and Pebble compaction physically removes them downstream. If anything blocks that cleanup, dead data accumulates silently.

CockroachDB node liveness failure: heartbeats, lease redistribution, and flapping

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB node liveness failure: heartbeats, lease redistribution, and flapping

Lease transfer spikes, briefly unavailable ranges, and client errors such as ambiguous results or connection resets indicate node liveness failure. In the logs, nodes transition to not-live and back within seconds. When the cluster decides a node cannot renew its liveness heartbeat, it redistributes leases. If the node recovers fast enough to renew but not fast enough to stay healthy, it flaps: an oscillating state more destructive than a clean outage because it repeatedly interrupts in-flight work and prevents stable failover.

CockroachDB out of memory: sys_rss, --cache, --max-sql-memory, and the OOM killer

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB out of memory: sys_rss, –cache, –max-sql-memory, and the OOM killer

A CockroachDB node disappears. No graceful shutdown, no drain sequence, no error in the SQL layer. The process is gone, and dmesg shows the kernel OOM killer selected it. Or in Kubernetes, the pod restarts with reason OOMKilled.

The root cause is almost always a mismatch between what CockroachDB thinks it can allocate and what the container or host actually allows. CockroachDB partitions its memory into two manually-sized pools: the Pebble block cache (--cache) and the SQL execution budget (--max-sql-memory). The Go garbage collector only manages the Go heap. CGo allocations, primarily the Pebble block cache and memtables, are manually managed. When the sum of these pools plus runtime overhead exceeds the container or host limit, the OOM killer intervenes.

CockroachDB Pebble write stalls: when the storage engine refuses writes

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB Pebble write stalls: when the storage engine refuses writes

When application writes time out or return ambiguous errors, check CockroachDB logs for pebble: write stall and watch storage_write_stalls. Pebble pauses writes to a store when Level 0 compaction debt exceeds safe thresholds. Until compaction drains L0, that store cannot accept new writes.

Write stalls are the most severe storage signal in CockroachDB. A brief stall during bulk loading may be harmless, but sustained stalls at one per second mean the node cannot meet its Raft obligations. The node may still serve reads, but it can lose Raft leadership because it cannot append log entries. That cascades into lease transfers and temporary range unavailability.

CockroachDB protected timestamp GC stall: stalled changefeeds that silently fill the disk

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB protected timestamp GC stall: stalled changefeeds that silently fill the disk

Disk space is growing on your CockroachDB cluster and nothing explains it. capacity_available trends downward hour after hour, but live data size has not changed. No bulk imports, no schema changes, no write spike. The SQL layer looks healthy: latency is fine, throughput is normal. But the disk is filling.

A CDC changefeed, a backup job, or another internal operation has created a protected timestamp (PTS) record that prevents MVCC garbage collection from reclaiming old data versions. The job has stalled, paused, or entered a retry loop, but the PTS record persists. Dead MVCC data accumulates underneath it. Disk usage grows linearly until free space drops below what compaction needs, at which point L0 sublevels spike, write stalls begin, and nodes lose liveness.

CockroachDB Raft liveness failure cascade: slow node, lost leases, rolling unavailability

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB Raft liveness failure cascade: slow node, lost leases, rolling unavailability

You notice brief, repeating SQL timeouts or ambiguous result errors across your application. In the CockroachDB DB Console, one node is still marked as running, yet its range lease count is oscillating and lease transfers are spiking. This is not a clean node crash. It is a Raft liveness failure cascade: a node has become slow enough to miss Raft heartbeats, but not dead enough to fail permanently. The cluster keeps trying to move leases away and back, creating rolling windows of unavailability.

CockroachDB Raft snapshot storm: when followers fall too far behind

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB Raft snapshot storm: when followers fall too far behind

Raft snapshots are CockroachDB’s fallback when incremental log replication is no longer possible. A leader sends a full copy of a range (up to 512 MiB by default) to a follower that has fallen so far behind that the log entries it needs have already been truncated. In steady state, this almost never happens. When it starts happening at scale, it creates a resource-draining cycle that can degrade an entire cluster.

CockroachDB range count per node too high: Raft ticking as a scaling dimension

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB range count per node too high: Raft ticking as a scaling dimension

CPU utilization is climbing across one or more CockroachDB nodes, and the usual explanations do not fit. Query throughput is flat. Disk I/O looks healthy. Admission control is not queuing. L0 sublevels are in single digits. CPU keeps trending upward, and the only change is that the database holds more data.

The likely cause is the scaling dimension most teams never instrument: range count per node. Every range in CockroachDB is an independent Raft consensus group. Even idle, each range’s Raft state machine ticks at a fixed interval, consuming CPU separate from query execution, compaction, or any workload-driven process. This baseline overhead grows linearly with range count per node.

CockroachDB range unavailable: diagnosing ranges_unavailable and recovering quorum

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB range unavailable: diagnosing ranges_unavailable and recovering quorum

The Prometheus gauge ranges_unavailable is the hardest of hard signals in CockroachDB. When it rises above zero and stays there, some slice of your keyspace has no leaseholder or has lost Raft quorum. Reads and writes to those ranges block or fail, and applications see errors, retries, or timeouts. If the unavailable ranges include system keyspaces, the impact is not limited to user tables. The cluster may lose the ability to heartbeat node liveness, execute schema changes, or route requests.

CockroachDB ReadWithinUncertaintyInterval restarts: the near-diagnostic signal of clock skew

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB ReadWithinUncertaintyInterval restarts: the near-diagnostic signal of clock skew

When CockroachDB reports readwithinuncertainty as a transaction restart cause, you have a clock synchronization problem. This restart cause does not appear in meaningful quantities for any other reason. Any sustained nonzero rate, even well below the self-termination threshold, indicates that NTP or the underlying clock source is not keeping node clocks aligned.

The error stems from CockroachDB’s Hybrid Logical Clock (HLC) design. Each transaction receives a timestamp from its gateway node’s HLC, which combines physical wall-clock time with a logical counter. When a read on one node encounters a write from a transaction that started on a different node, and the two nodes’ clocks are not aligned, the database cannot determine which transaction began first. CockroachDB conservatively restarts the reading transaction at a higher timestamp. Under SERIALIZABLE isolation, this restart requires client-side retry logic and adds directly to tail latency.

CockroachDB replica unavailable: lost quorum and stuck Raft groups

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB replica unavailable: lost quorum and stuck Raft groups

ranges_unavailable has gone nonzero. Clients are seeing “replica unavailable” errors. Some portion of your keyspace cannot be read or written. This is an active availability incident.

In CockroachDB, every range (a ~512 MiB slice of the keyspace) is replicated across multiple nodes. A write requires quorum acknowledgment from a majority of replicas before it commits. When too few replicas are reachable, the range loses quorum and cannot serve reads or writes. The ranges_unavailable metric tracks exactly this condition: ranges with no leaseholder or with lost Raft quorum.

CockroachDB result is ambiguous: ambiguous commit errors and how to handle them

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB result is ambiguous: ambiguous commit errors and how to handle them

Your application received result is ambiguous (PostgreSQL error code 40003, statement_completion_unknown) from CockroachDB. The transaction may have committed. It may have been aborted. The database does not know, and neither do you.

This is not a bug. It is a correctness property of CockroachDB’s distributed commit protocol. When the transaction coordinator loses contact with the leaseholder during the final commit step, or during the last statement of an implicit transaction, it cannot determine whether the commit succeeded. Rather than guess and risk a silent double-apply, CockroachDB surfaces the ambiguity.

CockroachDB RETRY_WRITE_TOO_OLD: the most common contention restart and how to kill it

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB RETRY_WRITE_TOO_OLD: the most common contention restart and how to kill it

Your cluster is healthy: nodes are live, ranges are available, disk space is fine, L0 sublevels are low. But transaction commit latency is climbing, P99 is widening, and the txn_restarts metric shows a rising rate tagged writetooold. Whether applications surface errors to users depends on whether their retry logic works.

RETRY_WRITE_TOO_OLD is the most common transaction restart cause under CockroachDB’s default SERIALIZABLE isolation. It is not an infrastructure problem. It is a contention problem rooted in schema design, key patterns, and transaction scope. Adding nodes does not help because the bottleneck is logical serialization, not physical capacity. Once you identify the conflicting keys and the transaction patterns driving them, writetooold restarts are fixable.

CockroachDB sequential primary key hotspot: SERIAL, timestamps, and write skew

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB sequential primary key hotspot: SERIAL, timestamps, and write skew

One node in your CockroachDB cluster runs hot while the others sit near idle. Write latency for specific tables climbs into hundreds of milliseconds. Transaction restarts tagged writetooold increase steadily alongside the insert rate. The cluster has plenty of aggregate CPU, memory, and disk headroom, but the workload cannot use it.

CockroachDB stores data in an ordered keyspace divided into ranges of approximately 512 MiB. Each range has a single leaseholder that serves all reads and coordinates all writes for keys in that range. When primary keys are sequential (SERIAL, auto-incrementing integers, timestamps), every new insert lands near the end of the keyspace, concentrated in the same range. That range’s leaseholder becomes a single-node bottleneck.

CockroachDB SQL error rate: XX000 internal errors and error-code triage

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB SQL error rate: XX000 internal errors and error-code triage

sql_failure_count is a single Prometheus counter with no error-code labels. A serialization conflict (40001) is expected noise in a contended workload. An internal error (XX000) is a database fault that needs immediate escalation. Treating the aggregate counter as one signal guarantees the wrong response.

This article covers how to break that aggregate apart by error code, when XX000 internal errors warrant paging, and how to triage the other common error classes.

CockroachDB SQL latency high: sql_service_latency, per-fingerprint diagnosis, and the layers

Mon, 01 Jan 0001 00:00:00 +0000

P99 sql_service_latency is climbing. Applications are timing out or retrying. The DB Console overview chart shows the spike, but the aggregate histogram cannot tell you which query is slow or which subsystem is adding the latency.

The most common diagnostic mistake is stopping at aggregate P99. A single slow query hiding among thousands of fast ones barely moves aggregate P99 but can take down an endpoint. A system-wide storage problem lifts all fingerprints together. Without per-fingerprint breakdown, you cannot distinguish “one query regressed” from “everything got slower.”

CockroachDB storage_l0_sublevels climbing: the earliest warning of write stalls

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB storage_l0_sublevels climbing: the earliest warning of write stalls

storage_l0_sublevels climbing on a single store is the earliest actionable signal for CockroachDB storage distress. The cluster may still serve traffic with slightly elevated SQL latency and no active write stalls. L0 sublevels typically provide 10-30 minutes of warning before write stalls begin, but only when monitored per-store.

As L0 sublevels rise, Pebble consults more SSTables per read, read amplification increases nonlinearly, and compaction falls behind. Admission control shapes regular traffic at 5 sublevels and elastic traffic at 1 sublevel. Past 20 sublevels, write stalls become imminent. Because this metric is per-store, a node with multiple stores can mask a hot disk behind healthy node-level aggregates.

CockroachDB store is full: emergency disk recovery and why deletes don't free space

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB store is full: emergency disk recovery and why deletes don’t free space

A CockroachDB node’s store directory is full or nearly full, and the node may have already shut itself down to protect data integrity.

Deleting data will not help immediately. CockroachDB uses MVCC: a SQL DELETE writes tombstones at new timestamps rather than removing old data. Until those tombstones propagate through Pebble compaction, the old versions stay on disk. A large-scale delete can temporarily increase disk usage because the tombstones are written before old data is reclaimed.

CockroachDB too many connections: sql_conns, pooling, and goroutine pressure

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB too many connections: sql_conns, pooling, and goroutine pressure

CockroachDB has no hard built-in connection limit comparable to PostgreSQL’s max_connections. Connections accumulate until something else breaks: goroutine pressure, memory exhaustion, or file descriptor limits.

The sql_conns gauge tracks active pgwire (SQL) connections per node. With properly configured client-side pooling, this number stays modest. When it climbs past 1000-2000 per node, you almost certainly have a pooling problem upstream.

CockroachDB transaction contention storm: retries, intents, and tail latency

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB transaction contention storm: retries, intents, and tail latency

The cluster looks healthy. Node liveness is stable, ranges_unavailable is zero, disk I/O is within bounds, CPU is moderate across all nodes. But P99 SQL latency is climbing, the transaction abort rate is up, and applications are reporting timeout errors. The database is spending most of its energy waiting, retrying, and resolving conflicts rather than doing useful work.

This is a transaction contention storm. The infrastructure is fine; the workload is the problem. Multiple transactions are contending for the same keys, creating serialized hot paths. Each conflict generates retries under CockroachDB’s default SERIALIZABLE isolation, which in turn leaves write intents that block subsequent transactions. Under sustained load, the retry-intent-latency cycle becomes a positive feedback loop.

CockroachDB transaction retry rate high: breaking down txn_restarts by cause

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB transaction retry rate high: breaking down txn_restarts by cause

When application latency creeps upward without an obvious cause, or you start seeing PostgreSQL error code 40001 (serialization failure) in application logs, the transaction retry rate is likely climbing. CockroachDB exposes txn_restarts as a set of counters, each tagged by cause. The aggregate number tells you retries are happening. It does not tell you why.

The five dominant causes are writetooold, serializable, readwithinuncertainty, txnaborted, and txnpush. Three additional sub-metrics (asyncwritefailure, commitdeadlineexceeded, unknown) cover edge cases. A spike dominated by readwithinuncertainty is a clock infrastructure problem. A spike dominated by writetooold is a schema or application contention problem. The fix for one does nothing for the other.

CockroachDB TransactionRetryWithProtoRefreshError: RETRY_SERIALIZABLE causes and fixes

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB TransactionRetryWithProtoRefreshError: RETRY_SERIALIZABLE causes and fixes

The application logs show TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to a conflict: committed value on key /Table/.../0). SQLSTATE is 40001. Clients are timing out or failing outright. The cluster looks healthy: nodes are live, ranges are available, disk space is fine. The problem is in the transaction layer.

Under SERIALIZABLE isolation (CockroachDB’s default), the database uses optimistic concurrency. It proceeds as if no conflict exists, then detects conflicts at commit or statement boundary. When a transaction’s preemptive refresh fails because another transaction already committed a write to a key the first transaction read, CockroachDB emits RETRY_SERIALIZABLE. This is expected behavior, not a bug.

CockroachDB under-replicated ranges: ranges_underreplicated and the healing margin

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB under-replicated ranges: ranges_underreplicated and the healing margin

When ranges_underreplicated rises above zero, CockroachDB has ranges with fewer live replicas than the configured replication factor. The replicate queue is already working to heal them by transferring Raft snapshots to new target nodes, but until healing completes, those ranges have reduced or zero fault tolerance margin.

The key operational question is not whether under-replicated ranges exist (they will, transiently, after almost any node event), but whether the cluster is healing fast enough to stay ahead of the next failure. This is the healing margin: the buffer between the current replication state and the point where a range loses quorum and becomes unavailable.

CockroachDB WAL fsync latency high: the most direct write-path health signal

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB WAL fsync latency high: the most direct write-path health signal

Elevated storage_wal_fsync_latency means the storage device is taking too long to durably persist write-ahead log entries. Every write in CockroachDB must complete a WAL fsync before Raft can acknowledge it. Fsync latency is the gating signal for the entire write path.

On healthy local SSDs, WAL fsync P99 should be under 10ms. Sustained values above 50ms mean the device is struggling. Above 200ms, the node is approaching disk stall territory, where CockroachDB’s built-in stall detection may terminate the process to prevent data inconsistency. Disk stalls cause self-termination before utilization metrics react: a volume reporting 60% utilization can stall on an individual fsync for seconds if the underlying hardware is failing, the I/O scheduler is misbehaving, or a cloud provider is throttling IOPS.

How CockroachDB actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How CockroachDB actually works in production: a mental model for operators

CockroachDB is a distributed, strongly-consistent SQL database built on a replicated key-value store. It layers SQL execution on top of a transactional KV engine that uses Raft consensus for replication and MVCC for concurrency control. To reason about its failures, you must hold several interacting subsystems in your head simultaneously.

This is not a tutorial. It is the mental model that experienced operators use when diagnosing slow queries, unavailable ranges, clock skew, and compaction death spirals. Each subsystem has its own failure modes, but the most damaging incidents happen when failures cascade across layers.