Ceph monitoring checklist: the signals every production cluster needs

This checklist defines the signals a production Ceph cluster needs, organized by monitoring maturity. Each level is a superset of the previous one; higher-level signals rely on lower-level context to distinguish real failures from normal churn. The four levels are survival, operational, mature, and expert. You cannot skip levels in practice.

Two caveats apply across all levels. First, ceph_health_status is an umbrella, not a complete picture: HEALTH_OK does not guarantee performance, and HEALTH_WARN covers both expected churn (active recovery) and structural problems (nearfull, noout trap, scrub inconsistency). Drill into ceph_health_detail before treating WARN as noise. Second, per-OSD and per-pool metrics matter more than cluster averages. A cluster at 60% average capacity with one OSD at 85% is closer to trouble than the average suggests.

flowchart TD
    L4["Level 4 - expert
BlueStore, RGW, MDS internals"]
    L3["Level 3 - mature
flags, outliers, scrub debt, CRUSH"]
    L2["Level 2 - operational
PG states, latency, slow ops, recovery"]
    L1["Level 1 - survival
health, quorum, OSD up/in, capacity"]
    L1 --> L2 --> L3 --> L4

Level 1: survival

Minimum viable monitoring. These signals tell you whether the cluster is alive, whether data is at risk, and whether you are about to run out of space.

Gotchas that bite at this level:

• A single OSD can be full while the cluster average is moderate. Watch the maximum, not the mean.
• The quorum formula must compare against the actual MON count, not a hardcoded 3 or 5. For a 5-MON deployment, 2 MONs out is a TICKET, 3 is a PAGE.
• ceph_health_status == 0 does not mean the cluster is fast. Add latency and PG-state signals in Level 2.

Level 2: operational

Detects degradation before users notice. The signals here distinguish transient recovery from stuck states and surface slow I/O before it cascades into client timeouts.

Gotchas that bite at this level:

• During normal rebalancing, degraded and recovering PGs are expected. Alert on degraded AND stalled recovery, not on degraded alone.
• Recovery and client I/O share the same disks. A recovery storm can present as a client-side latency incident. Correlate recovery rate with ceph_healthcheck_slow_ops.
• ceph_num_objects_unfound can briefly appear during cold start as OSDs rejoin. The 300s sustain helps but is not restart-aware; if your monitoring knows an OSD restarted in the last 10 minutes, gate the alert on that.
• ceph_osd_commit_latency_ms and ceph_osd_apply_latency_ms are point-in-time snapshots, not histograms. A single bad sample in a 60s scrape window can be misleading.

Level 3: mature

Proactive capacity, performance, and configuration-drift detection.

A read-only check for the noout trap, runnable on any node with a ceph.conf and admin keyring:

# Check global OSD flags
ceph osd dump | grep flags

Gotchas that bite at this level:

• The noout trap is the most common preventable Ceph outage. noout plus any OSD down equals reduced redundancy with no recovery in progress.
• noscrub and nodeep_scrub set indefinitely is silently catastrophic. HEALTH_OK does not mean data is being verified. Track PG_NOT_DEEP_SCRUBBED separately from cluster health.
• Clock skew tightens at 0.05s by default. VMs and bare metal behave very differently here; chrony/NTP must be solid on MON hosts.
• Per-OSD latency outliers require a per-device-class baseline. HDD, SSD, and NVMe expectations are not comparable. Pool OSDs by device class before computing the median.
• Scrub recency matters even when scrubs are completing. Heavy client I/O can starve scrubs indefinitely if osd_scrub_load_threshold is too low for your workload.

Level 4: expert

Subsystem-specific observability. Most of these come from ceph-exporter (per-daemon perf counters) rather than the MGR prometheus module. They matter once you are chasing specific failure classes: OMAP storms, BlueStore DB spillover, MDS cache pressure, RGW GC falling behind.

Gotchas that bite at this level:

• Reef 18.x changed how daemon-level perf counters are exported. The MGR prometheus module’s exclude_perf_counters defaults to true, and many BlueStore and OSD latency metrics now come from ceph-exporter instead. After upgrade, verify both scrape targets before assuming metrics disappeared.
• RBD per-image IO statistics are not gathered unless mgr/prometheus/rbd_stats_pools is configured. Enabling rbd_stats_pools "*" can impact manager performance on clusters with many RBD images.
• ceph_rgw_failed_req counts aborted requests (typically client disconnects), not HTTP 4xx/5xx. HTTP error breakdown requires RGW access log analysis.
• CephFS signals are irrelevant if you do not run CephFS. The same applies to RGW signals. Track only what your deployment uses.
• ceph_mds_mem_rss is a leading indicator for MDS_ALL_DOWN via OOM kill, which is itself a leading indicator for CephFS client disruption. Treat memory pressure on MDS with the same urgency as OSD memory pressure.

Cross-cutting gotchas worth their own alert

These operational errors recur across teams and clusters. Each deserves a dedicated check that combines multiple signals rather than relying on any single metric.

The noout trap. Alert if ceph_osd_flag_noout == 1 for more than 24 hours. noout set for maintenance, then forgotten: OSDs fail without triggering recovery, degraded PGs accumulate silently, and the next failure causes data loss. This is the single most common preventable Ceph outage.

Nearfull dismissal. Alert on both cluster-wide and per-OSD nearfull. A cluster at 85% that loses 10% of OSDs jumps past backfillfull (90%) and can no longer heal itself. Model the failure scenario, not just current state.

HEALTH_WARN as noise. Gate silences on ceph_health_detail, not on the umbrella status. WARN covers routine recovery but also dangerous conditions: nearfull, OSD down with noout set, PG_NOT_DEEP_SCRUBBED overdue, scrub inconsistencies.

Recovery rate blindness. The alert is ceph_pg_degraded > 0 AND ceph_pool_recovering_bytes_per_sec ~= 0, not either signal alone. Degraded PGs with zero recovery is the signature of a stuck cluster.

Scrub inconsistency indifference. Treat any nonzero ceph_pg_inconsistent as urgent. If the consistent replica fails before repair, data is lost.

BlueStore DB spillover. When the dedicated RocksDB DB device fills, metadata spills to the slow data device. Performance collapses but the OSD stays up and raw capacity metrics look fine. Only BlueStore internals (commit latency spiking, bluefs slow device usage going nonzero) reveal this. If you run OSDs with separate DB/WAL devices, Level 4 monitoring is not optional.

How Netdata helps

Netdata’s Ceph collector surfaces these signals at per-second resolution, which matters when correlating brief OSD flaps, recovery stalls, or peering storms that minute-granularity scraping flattens out.

• The cluster overview correlates ceph_health_status against per-pool PG state counts and recovery rate, so you can distinguish transient WARN from structural WARN without switching contexts.
• Per-OSD latency charts make outliers obvious against the cluster median, the practical way to catch a single dying disk before it cascades.
• Anomaly detection flags simultaneous shifts across ceph_healthcheck_slow_ops, OSD commit latency, and recovery rate, the signature of a recovery storm or BlueStore DB spillover.
• The ceph_osd_flag_* gauges surface forgotten maintenance flags, including the noout trap, before they translate into silently degraded PGs.
• Host-level CPU, memory, disk I/O, and network metrics on the same timeline let you correlate RGW OMAP storms or MDS cache pressure with underlying device saturation.