CockroachDB monitoring maturity model: from survival to expert

CockroachDB failures rarely announce themselves through a single metric. A slow disk turns into L0 compaction debt, which stalls writes, which drops Raft proposals, which makes ranges unavailable. A drifting clock raises transaction retry rates long before any node self-terminates. To run this database safely, you need layered observability that matches the system’s own layers: storage engine, Raft replication, distributed SQL, and transaction execution.

This guide organizes monitoring into four maturity levels. Level 1 keeps you alive during obvious outages. Level 2 catches workload and latency regressions. Level 3 adds the storage-engine and garbage-collection signals that precede most performance cliffs. Level 4 exposes the per-range, per-queue, and proposal-level details that explain why a seemingly healthy cluster suddenly degrades. Use the model to audit your current coverage and decide what to instrument next.

flowchart TD
    A[Level 1 Survival] --> B[Level 2 Operational]
    B --> C[Level 3 Mature]
    C --> D[Level 4 Expert]
    A -- liveness, ranges, disk, SQL probe, certs --> B
    B -- latency, retries, L0, clock, RPC --> C
    C -- read amp, stalls, cache, GC, intents, MVCC --> D
    D -- raft drops, hot ranges, closed ts, queue errors --> E[Predictive incident response]

Level 1: survival

Survival monitoring answers one question: is the cluster up? These signals are binary, cheap to collect, and unambiguous. If any of them fire outside a maintenance window, you have a production incident.

With these six signals you will know when the cluster is broken, but not why it is slow or why it is about to break.

Level 2: operational

Operational monitoring adds workload-facing and infrastructure signals. This is the layer that catches latency regressions, contention, and the first signs of storage-engine pressure. If your team runs CockroachDB in production, these should be on dashboards and alertable.

Level 2 is where most production teams stop. It is enough to diagnose many incidents, but it misses the slow-burn failures that fill disks with garbage or let compaction debt accumulate for weeks.

Level 3: mature

Mature monitoring tracks the storage engine, runtime, and replication internals. These signals are leading indicators: they warn you before Level 2 latency spikes or Level 1 unavailability.

This level turns operators from firefighters into investigators. You stop asking “why is P99 high?” and start asking “why is read amplification rising while garbage bytes grow?”

Level 4: expert

Expert monitoring targets the low-level signals that explain edge-case behavior. These metrics are noisy in isolation and expensive to collect, but they are decisive during deep incidents and capacity planning.

These signals are best consumed through exploratory dashboards and ad-hoc queries rather than high-noise paging alerts. Their value is giving you the last few data points that separate one root cause from another.

Moving between levels

Do not try to implement all four levels at once. A practical progression:

• Start with Level 1 plus Level 2’s L0 sublevel count and SQL latency. Those three classes of signals catch the majority of production incidents.
• Add Level 2 clock offset, retry cause breakdown, and admission control queue depth before your first multi-region deployment.
• Move to Level 3 when you have recurring latency tickets that Level 2 cannot explain. Focus on read amplification, compaction backlog, and MVCC garbage first.
• Add Level 4 signals after you have experienced at least one incident where you needed per-range request rates or Raft proposal drop metrics to close the diagnosis.

Keep scrape intervals short enough to catch sub-second write stalls; a 30-second scrape will miss brief events that still hurt users. Reserve crdb_internal queries for manual diagnosis, not automated alerting, because they are unsupported and can be expensive during incidents.

How Netdata helps

Netdata can reduce the gap between Level 2 and Level 3 monitoring without forcing you to build custom Prometheus recording rules for every storage-engine signal:

• Netdata collects CockroachDB’s Prometheus-formatted metrics from /_status/vars out of the box and correlates them with node-level CPU, memory, disk I/O, and network on the same charts.
• Per-second granularity makes brief Pebble write stalls, Raft commit latency spikes, and admission-control queue flushes visible instead of averaging them away.
• Composite dashboards let you overlay L0 sublevel count, KV write latency, and disk I/O utilization to confirm an LSM compaction death spiral in one view.
• Anomaly detection on slowly moving signals such as MVCC garbage bytes, protected timestamp age, and read amplification can surface the “silent but catastrophic” trends that static thresholds miss.
• Correlating CockroachDB process RSS with Go GC pause duration and goroutine count helps distinguish a SQL memory leak from a storage-engine cache sizing issue.

Related guides

• How CockroachDB actually works in production: a mental model for operators