Ceph Prometheus on Netdata

Best Ceph Monitoring Tools: 7 Options Ranked (2026)

Wed, 12 Aug 2026 00:00:00 +0000

Ceph backfill_toofull: recovery blocked because target OSDs are full

Mon, 01 Jan 0001 00:00:00 +0000

A PG in backfill_toofull is not making progress toward active+clean. The source OSD has the data, CRUSH has chosen the target, but the target refuses the reservation because it is above backfillfull_ratio. Client reads and writes still succeed; the redundancy gap is not closing.

The cluster is usually not at the hard full ratio (0.95 default). It is at backfillfull (0.90 default) on one or more individual OSDs, and that is enough to halt recovery for every PG whose up set includes them. Symptoms: HEALTH_WARN (or HEALTH_ERR on older releases), a flat degraded-object count, and recovery bytes/sec near zero.

Ceph blocked ops: client I/O stuck behind a single slow OSD

Mon, 01 Jan 0001 00:00:00 +0000

A client writes an object. The primary OSD forwards sub-ops to its replicas, then waits for every replica to ack before it acks the client. If any one OSD in the acting set stalls, the whole op stalls. After osd_op_complaint_time (default 30 seconds) the OSD logs a slow op and the monitor raises SLOW_OPS. Past that boundary the op is effectively blocked, not merely slow. Clients time out and retry, which puts more ops in flight against the same stuck OSD.

Ceph BLUEFS_SPILLOVER: RocksDB metadata spilling onto the slow device

Mon, 01 Jan 0001 00:00:00 +0000

A small subset of OSDs shows periodic commit-latency spikes and slow ops while the rest of the cluster looks healthy. Capacity metrics are normal. SMART is clean. ceph -s reports HEALTH_WARN, and ceph health detail returns something like:

BLUEFS_SPILLOVER
 3 OSDs spilled over ~18 GiB metadata from 'db' device
 (e.g. osd.12 spilled over 6.1 GiB metadata from 'db' device)

BlueStore’s RocksDB metadata has outgrown its dedicated fast DB partition (SSD/NVMe) and is spilling onto the slow HDD data partition. Compaction that took milliseconds on flash now takes seconds on spinning disk. Between compaction cycles the OSD looks fine; during compaction it stalls. This is a cliff edge, not gradual degradation: the moment slow_used_bytes goes nonzero, latency steps up by one to two orders of magnitude on the affected OSDs.

Ceph capacity death spiral: an OSD fails and recovery has nowhere to go

Mon, 01 Jan 0001 00:00:00 +0000

An OSD fails on a cluster you have been running at 80% or higher. ceph -s shows recovery starting, then ceph health detail lists OSD_NEARFULL and BACKFILL_TOOFULL. The degraded PG count climbs, then flatlines. Recovery bytes per second sits near zero. Nothing is healing, and you are one more failure away from data loss.

This is the Ceph capacity death spiral. It is not a single fault: it is the intersection of tight capacity, CRUSH imbalance, and the hard thresholds Ceph uses to protect itself. Recovery needs spare space on the surviving OSDs. When those OSDs are already past the backfillfull ratio (default 0.90), Ceph refuses to push more data onto them, and backfill stalls for every PG that would target them.

Ceph degraded objects: reduced redundancy and the race against a second failure

Mon, 01 Jan 0001 00:00:00 +0000

Degraded objects in Ceph are the most direct measure of reduced redundancy: somewhere in the cluster, at least one object has fewer live replicas (or fewer erasure-coded fragments) than its pool requires. The cluster is still serving I/O, but the protection against a follow-on failure has thinned for those objects.

ceph_num_objects_degraded is not alarming on its own. Every OSD failure, restart, and CRUSH reweight produces a transient spike that recovery is designed to heal. The real signal is the slope: is the count falling back toward zero, or has it plateaued while the cluster is still degraded?

Ceph HEALTH_ERR: reading the umbrella status and finding the real fault

Mon, 01 Jan 0001 00:00:00 +0000

ceph_health_status == 2 is the most severe top-level status Ceph reports, and one of the most over-paged signals in production. The reason is structural: HEALTH_ERR is an umbrella aggregation. It tells you that one or more child health checks crossed into ERR severity. It does not tell you which subsystem failed, whether I/O is actually blocked, or whether the condition will self-resolve when PGs finish peering.

Treating the umbrella as the page to action creates two failure modes. First, it duplicates the more specific child signals (OSD_FULL, PG down, MON quorum loss) that already have their own PAGE conditions with tighter gating, producing parallel escalations for the same incident. Second, it fires on transient cold-start states. A cluster-wide OSD restart will briefly push PGs through peering and incomplete, which trips HEALTH_ERR for tens of seconds before clearing. Operators who page on HEALTH_ERR without context spend their nights chasing ghosts.

Ceph monitor quorum lost: the cluster can no longer update its maps

Mon, 01 Jan 0001 00:00:00 +0000

Ceph monitor quorum loss is a PAGE condition. Without a majority of MONs agreeing through Paxos, the cluster cannot commit any map update: no OSD up/down transitions, no PG state changes, no pool edits, no CRUSH adjustments. Existing clients keep running on cached maps for a while, but new client connections fail immediately, and as soon as those cached maps go stale, in-flight I/O stalls.

The trigger formula is sum(ceph_mon_quorum_status) < floor(count(ceph_mon_quorum_status) / 2) + 1. For 3 MONs you need 2 in quorum. For 5 MONs you need 3. The standard alert sustain is 300 seconds, long enough to ride out a normal sub-second leader election and short enough to page before clients start timing out en masse.

Ceph monitoring checklist: the signals every production cluster needs

Mon, 01 Jan 0001 00:00:00 +0000

This checklist defines the signals a production Ceph cluster needs, organized by monitoring maturity. Each level is a superset of the previous one; higher-level signals rely on lower-level context to distinguish real failures from normal churn. The four levels are survival, operational, mature, and expert. You cannot skip levels in practice.

Two caveats apply across all levels. First, ceph_health_status is an umbrella, not a complete picture: HEALTH_OK does not guarantee performance, and HEALTH_WARN covers both expected churn (active recovery) and structural problems (nearfull, noout trap, scrub inconsistency). Drill into ceph_health_detail before treating WARN as noise. Second, per-OSD and per-pool metrics matter more than cluster averages. A cluster at 60% average capacity with one OSD at 85% is closer to trouble than the average suggests.

Ceph OSD down: telling a dead disk apart from a network blip

Mon, 01 Jan 0001 00:00:00 +0000

OSD_DOWN fires. ceph_osd_up flipped to 0. The 600-second countdown to OUT and recovery has begun.

Your first job: figure out whether this is a dead disk (act now, plan a replacement) or a network blip (wait, verify, let the OSD come back). Acting on the wrong diagnosis wastes disk and network I/O on a needless backfill, or worse, leaves a failing disk in service past the point where SMART was already warning you.

Ceph OSD flapping: OSDs cycling up and down and the peering storm that follows

Mon, 01 Jan 0001 00:00:00 +0000

OSD flapping is a failure cascade. One OSD misses heartbeats, peers mark it down, its PGs start peering and recovering elsewhere, the OSD comes back, peering reverses, and the cycle repeats. Each flap mints a new OSD map epoch that every OSD in the cluster must process. The peering overhead from a single flapping OSD can slow dozens of healthy OSDs enough that they also miss heartbeats, and the cascade spreads.

Ceph OSD_FULL: all writes stopped at the 95% full ratio

Mon, 01 Jan 0001 00:00:00 +0000

The cluster suddenly stops accepting writes. Client applications report ENOSPC errors. ceph status returns HEALTH_ERR. ceph health detail shows OSD_FULL active. Reads still succeed, but every write, update, and delete fails cluster-wide.

This is a hard stop, not a throttle. Ceph refuses all write operations once any OSD crosses the configured full_ratio (default 0.95). CRUSH spreads every PG across multiple OSDs, so a single full OSD can block writes to hundreds of PGs even when cluster-average utilization looks moderate.

Ceph PG down: no surviving replica for reads or writes

Mon, 01 Jan 0001 00:00:00 +0000

A placement group in the down state has no replica that can serve I/O. Reads and writes to objects in that PG fail. Clients see EIO or ENODEV, and ceph health detail reports PG_AVAILABILITY with one or more PGs flagged down. Data is unavailable, not just degraded.

A PG can briefly pass through down during peering after an OSD failure, before activating on a surviving replica. The 300-second sustain on sum(ceph_pg_down) > 0 filters that transient window. Once a PG has been down past the sustain, no OSD in the acting set can serve it, and the cluster will not heal it without operator action.

Ceph PG incomplete: placement groups that cannot serve I/O

Mon, 01 Jan 0001 00:00:00 +0000

An incomplete placement group means the PG cannot find enough authoritative data to serve reads or writes. This is a data-availability failure, and it does not self-resolve the way a transient peering or recovering state does.

Alert on sum(ceph_pg_incomplete) > 0 sustained for more than 300 seconds, summed across all pools. The 300 second sustain filters out cold-start peering after a cluster-wide restart, which typically completes within 60-120 seconds. Anything still incomplete after five minutes is genuinely stuck.

Ceph PG inconsistent (OSD_SCRUB_ERRORS): scrub found replica divergence

Mon, 01 Jan 0001 00:00:00 +0000

A scrub or deep-scrub finished comparing replicas for a placement group and found they disagree. The cluster surfaces this as OSD_SCRUB_ERRORS (often paired with PG_DAMAGED) and the affected PG sits in active+clean+inconsistent. Client reads still succeed because Ceph serves them from a consistent replica, but at least one copy in the acting set is corrupt.

The danger is not the symptom. Reads work, and Ceph did what it was designed to do: detect silent divergence. The danger is that the corruption was found, not fixed, and the window during which an uncorrupted replica survives is your margin of safety. If the OSD holding the good copy fails before you repair, the object becomes unreadable or unwritable.

Ceph slow requests (SLOW_OPS): operations blocked past osd_op_complaint_time

Mon, 01 Jan 0001 00:00:00 +0000

ceph_healthcheck_slow_ops greater than zero means operations on the cluster have crossed osd_op_complaint_time (default 30s) and are stuck, not merely slow. The SLOW_OPS health check surfaces them as a warning, and the metric itself is a live gauge pulled from ceph health detail.

Slow ops are a symptom of something downstream blocking I/O: a failing disk, a saturated BlueStore RocksDB DB, network timeouts between OSDs, or heavy deep-scrub on HDD during a maintenance window. The right first move is to read where in the pipeline each slow op is stuck before changing any config.

Ceph unfound objects: the cluster cannot locate a surviving copy

Mon, 01 Jan 0001 00:00:00 +0000

When a placement group enters recovery_unfound or backfill_unfound, the cluster has objects it knows should exist but cannot locate on any OSD that is currently up and in. ceph health detail surfaces this as the OBJECT_UNFOUND check, and the cluster-wide gauge ceph_num_objects_unfound rises above zero. Every known replica or erasure-coded chunk is on an OSD that is down, destroyed, or has not yet been probed.

Recovery of the affected PGs is blocked at the unfound object. Client reads and writes to those objects stall indefinitely rather than erroring out. The only paths forward are to bring back the missing data or to explicitly accept the loss.