Ceph monitoring maturity model: from survival to expert

Ceph’s health surface is unusually broad for a storage system. A cluster can report HEALTH_OK while a single OSD runs at 20x the latency of its peers, while the BlueStore DB partition spills to slow media, or while the PG autoscaler splits placement groups under live client load. The gap between “is the cluster up” and “is the cluster observable enough to operate” is where most preventable outages live.

This article maps a four-level monitoring maturity model for Ceph. It is not an upstream Ceph concept. It is an operator-facing framework for reasoning about coverage gaps: what each level detects, what it misses, and which signals to add next when triaging blind spots after an incident. The levels are cumulative. Each builds on the previous one rather than replacing it.

Use this as a self-assessment, not a maturity score to chase. A three-OSD homelab does not need Level 4. A production cluster serving RBD volumes to a hypervisor fleet needs at least Level 3, and Level 4 wherever a specific subsystem (RGW, CephFS, EC pools) is in the data path.

flowchart TD
    L4["Level 4 - Expert
BlueStore, RGW OMAP, MDS caps, EC, autoscaler"]
    L3["Level 3 - Mature
Flags, outliers, quotas, clock skew, scrub recency"]
    L2["Level 2 - Operational
PG states, OSD latency, slow ops, recovery"]
    L1["Level 1 - Survival
Health, OSD up/in, capacity, quorum"]
    L4 --> L3 --> L2 --> L1

How to use this model

Each level below lists the signals that define it, the failure modes those signals catch, and the blind spots that motivate moving to the next level. Signals are named by their Prometheus metric or health check code as exposed by the Ceph manager Prometheus module and ceph_exporter. The same names apply whether you collect them through the manager module, ceph_exporter, or a third-party collector such as Netdata’s Ceph integration.

The model assumes BlueStore as the OSD backend. Filestore is deprecated and its metric prefixes (filestore_*) are out of scope here. If your cluster still runs Filestore, treat migration to BlueStore as a prerequisite for Level 3 and above.

Level 1: survival

Level 1 answers four questions: is the cluster up, are OSDs participating, is there capacity headroom, and do monitors hold quorum. With just these signals you can detect a hard outage but not a degrading one.

Level 1 blind spots:

• HEALTH_OK is not a performance signal. A cluster with terrible latency can still be OK.
• Cluster-wide capacity hides per-OSD imbalance. One OSD at 94% while the average is 70% will trigger backfillfull behavior that the aggregate hides.
• No visibility into recovery progress. You know PGs are degraded but not whether they are healing.
• No scrub integrity signal. Silent corruption goes undetected until it becomes data loss.

Level 2: operational

Level 2 is where monitoring becomes useful for incident response. You can see degradation before users complain and distinguish transient recovery from stuck states.

PG states, individually. Move beyond aggregated “PGs not active+clean” counts. Track the specific states: ceph_pg_degraded, ceph_pg_undersized, ceph_pg_stale, ceph_pg_down, ceph_pg_incomplete, ceph_pg_inconsistent, ceph_pg_recovering, ceph_pg_backfilling, ceph_pg_backfill_toofull, ceph_pg_recovery_unfound. The state tells you the failure mode. down and incomplete are availability failures. inconsistent is corruption. backfill_toofull is a capacity problem masquerading as a recovery problem.

OSD latency, per OSD. ceph_osd_commit_latency_ms reflects WAL and journal device performance. ceph_osd_apply_latency_ms reflects the main data device. These are point-in-time snapshots, not histograms, so collect them at per-second resolution to see spikes. HDD baselines: commit 10-50ms, apply 50-200ms under load. NVMe: both under 5ms. An OSD 2-3x above these baselines during steady-state load warrants a look.

Slow ops. ceph_healthcheck_slow_ops is the count of operations exceeding osd_op_complaint_time (default 30s). These are stuck, not merely slow. Nonzero is always worth investigating. Cluster-wide slow ops across many OSDs is far more concerning than isolated slow ops on one.

Recovery rate. ceph_pool_recovering_bytes_per_sec per pool. The critical correlation is degraded PG count against recovery rate. Degraded PGs with zero recovery is stalled healing. Degraded PGs with active recovery is normal self-repair.

Object counts. ceph_num_objects_unfound (potential data loss), ceph_num_objects_degraded (reduced redundancy), ceph_num_objects_misplaced (wrong OSDs per CRUSH, data still safe).

Scrub inconsistency. ceph_pg_inconsistent and ceph_health_detail{name="OSD_SCRUB_ERRORS"}. Any nonzero value means scrub found replica divergence. Treat ceph_pg_failed_repair as urgent: Ceph attempted automatic repair and could not complete it.

Level 2 blind spots:

• Latency outliers buried in aggregates. A single OSD at 5x cluster median latency is invisible if you only alert on averages.
• No flag awareness. noout, norecover, noscrub set during maintenance and forgotten will silently disable recovery or integrity verification.
• No quota tracking. A pool can hit its quota and reject writes while cluster-wide capacity looks fine.
• No scrub recency. You see errors when scrub finds them, but not whether scrubs are running at all.

Level 3: mature

Level 3 is proactive. The signals here detect conditions that will become incidents if left alone: forgotten flags, capacity skew, clock drift, scrub debt, flapping.

OSD cluster flags. ceph_osd_flag_noout, ceph_osd_flag_norecover, ceph_osd_flag_nobackfill, ceph_osd_flag_noscrub, ceph_osd_flag_nodeep-scrub. The noout trap is the single most common preventable Ceph outage: noout set for maintenance, never unset, OSDs fail, and recovery never starts because the cluster believes the OSDs are intentionally out. Alert on noout set for more than 24 hours. Alert on norecover or nobackfill set while degraded PGs exist.

Per-OSD latency outliers. Compare each OSD’s commit and apply latency to the median for the same device class. Alert at 5x median sustained for 300 seconds. This catches the dying-disk case that cluster averages hide.

Per-pool capacity and quotas. ceph_pool_percent_used per pool. ceph_pool_quota_bytes and ceph_pool_quota_objects (0 means no quota). ceph_pool_stored and ceph_pool_objects for current usage. Alert when pool usage exceeds 80% of a configured quota.

Clock skew. No dedicated Prometheus metric exists. The condition surfaces as ceph_health_detail{name="MON_CLOCK_SKEW"}. The default mon_clock_drift_allowed is 0.05s, which is tight. VM monitors are particularly prone. Correlate with NTP or chrony offset metrics on MON hosts.

Recovery-flag correlation. Tie recovery rate to flag state. Zero recovery with degraded PGs and norecover set is an operational error, not a cluster fault. Zero recovery with degraded PGs and no flags set is a real stall worth investigating.

Scrub recency. No direct metric for “PGs overdue for scrub.” The condition surfaces as ceph_health_detail{name="PG_NOT_SCRUBBED"} or ceph_health_detail{name="PG_NOT_DEEP_SCRUBBED"}. Alert if either is active for more than 24 hours. Correlate with noscrub and nodeep-scrub flag state: if the flags are set, overdue-scrub alerts are expected and should be suppressed or routed differently.

OSD flapping. No direct flap-count metric. The condition surfaces as ceph_health_detail{name="OSD_FLAPPING"}. Each up/down transition generates an OSD map epoch and forces re-peering across every PG on that OSD. The peering overhead on the rest of the cluster exceeds the impact of the OSD simply being down.

CRUSH placement safety. ceph_health_detail{name="POOL_NO_REDUNDANCY"} and ceph_health_detail{name="TOO_FEW_OSDS"}. These indicate CRUSH rules that cannot satisfy the configured failure domain. A 3x replicated pool requiring host-level isolation on a cluster with OSDs on only two hosts will never reach active+clean, and only these checks will tell you why.

Level 3 blind spots:

• No BlueStore internals. DB spillover, compaction stalls, and cache pressure are invisible.
• No subsystem depth. RGW OMAP storms, MDS cap pressure, and EC recovery overhead are not visible at this level.
• No per-device-class capacity. SSD and HDD tiers look uniform in aggregate.

Level 4: expert

Level 4 adds subsystem-specific observability. These signals matter when a specific subsystem is in the critical path: RGW for S3 workloads, CephFS for file, EC pools for cold storage, BlueStore DB devices for any cluster with dedicated WAL or DB partitions.

BlueStore internals. Via ceph_exporter or the admin socket: BlueFS slow-device usage (when slow_used_bytes is greater than zero, RocksDB has spilled from the fast DB partition to the slow data partition, a severe performance cliff), RocksDB compaction statistics, cache hit ratios (bluestore_cache_hit vs bluestore_cache_miss). BlueStore DB spillover is the classic mystery latency degradation: capacity metrics look fine, the OSD does not crash, but commit latency jumps 10-100x during compaction.

RGW OMAP and latency. ceph_health_detail{name="LARGE_OMAP_OBJECTS"} signals bucket index shards with too many OMAP entries; the default threshold is osd_pg_object_limit (200000) scaled by osd_pg_omap_ratio_threshold (0.9). ceph_rgw_req, ceph_rgw_failed_req, ceph_rgw_qlen, ceph_rgw_qactive, and per-operation latency counters (ceph_rgw_op_put_obj_lat_sum and _count, ceph_rgw_op_get_obj_lat_sum and _count). Note that ceph_rgw_failed_req counts aborted requests (client disconnects), not HTTP 4xx or 5xx. For HTTP error breakdown, RGW access logs are required.

MDS cap pressure. ceph_mds_caps, ceph_mds_inodes_with_caps, ceph_mds_server_cap_revoke_eviction, ceph_mds_server_session_recall_throttle, ceph_mds_server_global_recall_throttle. Health checks MDS_HEALTH_CACHE_OVERSIZED and MDS_CLIENT_RECALL. Cap pressure is the precursor to client eviction: the MDS cannot reclaim caps fast enough, latency climbs, and eventually clients are blacklisted.

Per-device-class capacity. ceph_cluster_by_class_total_bytes (label device_class). Lets you track SSD, HDD, and NVMe tiers independently. A cluster can be healthy in aggregate while one device class sits at backfillfull.

Erasure coding recovery. EC pools require reading k chunks to reconstruct missing data, making recovery slower and more CPU-intensive than replicated recovery. . Track recovery rate on EC pools separately from replicated pools, and expect longer healing windows after OSD failures.

PG autoscaler activity. The autoscaler splits and merges PGs based on usage. Splits cause brief I/O stalls as PGs re-peer. . Monitor ceph_pg_creating and related state counters for autoscaler-driven churn, especially on pools with rapidly changing data volumes.

Cross-level signal reference

How Netdata helps

Netdata’s Ceph collector exposes the manager Prometheus metrics at per-second resolution, which matters for the Level 2 and Level 3 signals where spikes are short and point-in-time snapshots miss them.

• Per-second OSD latency surfaces commit-latency spikes during RocksDB compaction that 15-second polling smooths away.
• Anomaly detection on per-OSD metrics flags the outlier OSD without you writing a 5x-median recording rule. The Level 3 dying-disk case is exactly the kind of single-series outlier that benefit from this.
• Correlation of PG states with recovery rate and flag state in a single view shortens the Level 2 stalled-recovery versus normal-healing diagnosis. You see degraded PGs, recovery rate, and norecover / nobackfill flags together rather than in separate dashboards.
• Health-check decomposition via ceph_health_detail with the name and severity labels lets you alert on specific checks (OSD_FLAPPING, MON_CLOCK_SKEW, PG_NOT_DEEP_SCRUBBED) rather than the umbrella HEALTH_WARN that operators learn to ignore.
• Per-pool and per-device-class capacity in the same dashboard as cluster-wide utilization surfaces the imbalances that Level 1 aggregates hide.

Related guides

• Ceph monitoring checklist: the signals every production cluster needs
• How Ceph actually works in production: a mental model for operators
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph OSD_FULL: all writes stopped at the 95% full ratio
• Ceph nearfull: the 85% warning that decides whether the cluster can heal
• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph OSD fullness imbalance: one OSD full while the cluster average looks fine
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph OSD down: telling a dead disk apart from a network blip
• Ceph OSD flapping: OSDs cycling up and down and the peering storm that follows