Ceph Operations Guides on Netdata

Ceph backfill_toofull: recovery blocked because target OSDs are full

Mon, 01 Jan 0001 00:00:00 +0000

Ceph backfill_toofull: recovery blocked because target OSDs are full

A PG in backfill_toofull is not making progress toward active+clean. The source OSD has the data, CRUSH has chosen the target, but the target refuses the reservation because it is above backfillfull_ratio. Client reads and writes still succeed; the redundancy gap is not closing.

The cluster is usually not at the hard full ratio (0.95 default). It is at backfillfull (0.90 default) on one or more individual OSDs, and that is enough to halt recovery for every PG whose up set includes them. Symptoms: HEALTH_WARN (or HEALTH_ERR on older releases), a flat degraded-object count, and recovery bytes/sec near zero.

Ceph blocked ops: client I/O stuck behind a single slow OSD

Mon, 01 Jan 0001 00:00:00 +0000

Ceph blocked ops: client I/O stuck behind a single slow OSD

A client writes an object. The primary OSD forwards sub-ops to its replicas, then waits for every replica to ack before it acks the client. If any one OSD in the acting set stalls, the whole op stalls. After osd_op_complaint_time (default 30 seconds) the OSD logs a slow op and the monitor raises SLOW_OPS. Past that boundary the op is effectively blocked, not merely slow. Clients time out and retry, which puts more ops in flight against the same stuck OSD.

Ceph BLUEFS_SPILLOVER: RocksDB metadata spilling onto the slow device

Mon, 01 Jan 0001 00:00:00 +0000

Ceph BLUEFS_SPILLOVER: RocksDB metadata spilling onto the slow device

A small subset of OSDs shows periodic commit-latency spikes and slow ops while the rest of the cluster looks healthy. Capacity metrics are normal. SMART is clean. ceph -s reports HEALTH_WARN, and ceph health detail returns something like:

BLUEFS_SPILLOVER
 3 OSDs spilled over ~18 GiB metadata from 'db' device
 (e.g. osd.12 spilled over 6.1 GiB metadata from 'db' device)

BlueStore’s RocksDB metadata has outgrown its dedicated fast DB partition (SSD/NVMe) and is spilling onto the slow HDD data partition. Compaction that took milliseconds on flash now takes seconds on spinning disk. Between compaction cycles the OSD looks fine; during compaction it stalls. This is a cliff edge, not gradual degradation: the moment slow_used_bytes goes nonzero, latency steps up by one to two orders of magnitude on the affected OSDs.

Ceph BlueStore allocator fragmentation: rising latency at moderate fullness

Mon, 01 Jan 0001 00:00:00 +0000

Ceph BlueStore allocator fragmentation: rising latency at moderate fullness

Write latency climbs on OSDs that are only moderately full. Capacity dashboards look healthy, the cluster sits well below nearfull, and SMART is clean. But a subset of OSDs shows rising commit and apply latency, slow ops begin accumulating, and the affected OSDs are the long-lived ones carrying heavy overwrite workloads. This is BlueStore allocator fragmentation: the free-space structure BlueStore walks on every allocation has become shredded into small, non-contiguous free extents, so each new write needs a longer search before a suitable run of blocks is found.

Ceph BlueStore RocksDB compaction stalls: periodic latency spikes

Mon, 01 Jan 0001 00:00:00 +0000

Ceph BlueStore RocksDB compaction stalls: periodic latency spikes

Periodic latency spikes on Ceph OSDs that appear and clear on their own schedule are a signature of BlueStore RocksDB compaction. The cluster reports brief bursts of slow ops on a small set of OSDs, commit latency climbs for seconds to minutes, then returns to baseline without intervention. Health checks may briefly show SLOW_OPS before clearing.

Ceph capacity death spiral: an OSD fails and recovery has nowhere to go

Mon, 01 Jan 0001 00:00:00 +0000

Ceph capacity death spiral: an OSD fails and recovery has nowhere to go

An OSD fails on a cluster you have been running at 80% or higher. ceph -s shows recovery starting, then ceph health detail lists OSD_NEARFULL and BACKFILL_TOOFULL. The degraded PG count climbs, then flatlines. Recovery bytes per second sits near zero. Nothing is healing, and you are one more failure away from data loss.

This is the Ceph capacity death spiral. It is not a single fault: it is the intersection of tight capacity, CRUSH imbalance, and the hard thresholds Ceph uses to protect itself. Recovery needs spare space on the surviving OSDs. When those OSDs are already past the backfillfull ratio (default 0.90), Ceph refuses to push more data onto them, and backfill stalls for every PG that would target them.

Ceph CephFS client eviction (EBLACKLISTED): blocklisted clients and failed I/O

Mon, 01 Jan 0001 00:00:00 +0000

Ceph CephFS client eviction (EBLACKLISTED): blocklisted clients and failed I/O

CephFS clients suddenly see EBLACKLISTED errors on open file handles. Reads and writes that worked seconds ago fail; new operations against the same mount return the same error code. The MDS has forcibly evicted the client and added its address to the cluster-wide OSD blocklist. Every OSD now refuses traffic from that client address, and the mount is effectively dead until the blocklist is cleared or expires.

Ceph client latency vs OSD latency: fast disks, slow clients

Mon, 01 Jan 0001 00:00:00 +0000

Ceph client latency vs OSD latency: fast disks, slow clients

You ran ceph osd perf, the commit_latency and apply_latency columns all look healthy, and you declared the cluster fast. Then a client team opens a ticket: writes are taking 50ms, 100ms, sometimes timing out. You re-check ceph osd perf. Still fast. The OSDs are not the bottleneck, but the clients are still slow.

ceph osd perf, the ceph_osd_*_latency_ms Prometheus metrics, and rbd perf image iostat all measure time inside the OSD data path. They do not measure the round trip the client experiences. Client-visible latency is the sum of OSD processing, public network RTT, replication round-trips to peer OSDs, CRUSH map recalculation during flapping, and queuing delay on the OSD’s public-facing messenger. A cluster can show 2ms commit latency on every OSD while clients see 50ms response times because the public network is saturated, or because public and cluster traffic share one NIC.

Ceph deep scrub performance impact: I/O saturation that mimics an incident

Mon, 01 Jan 0001 00:00:00 +0000

Ceph deep scrub performance impact: I/O saturation that mimics an incident

A Ceph cluster suddenly shows elevated apply latency across many OSDs. Slow ops tick up. Client write latency degrades. The dashboard looks like the start of a real incident. But cluster health is HEALTH_OK and all PGs are active+clean. Before you start chasing a failing disk or a network partition, check whether a deep scrub is running.

Deep scrub reads and checksums every byte of every object in a PG. On HDD-backed OSDs, this is a sequential read pass across the entire device. With default scheduling, it happens to every PG roughly once a week. If the schedule concentrates scrubs onto the same OSDs at the same time, or if your scrub window is narrow, deep scrub will saturate disk I/O and produce symptoms that look identical to a real performance incident.

Ceph degraded objects: reduced redundancy and the race against a second failure

Mon, 01 Jan 0001 00:00:00 +0000

Ceph degraded objects: reduced redundancy and the race against a second failure

Degraded objects in Ceph are the most direct measure of reduced redundancy: somewhere in the cluster, at least one object has fewer live replicas (or fewer erasure-coded fragments) than its pool requires. The cluster is still serving I/O, but the protection against a follow-on failure has thinned for those objects.

ceph_num_objects_degraded is not alarming on its own. Every OSD failure, restart, and CRUSH reweight produces a transient spike that recovery is designed to heal. The real signal is the slope: is the count falling back toward zero, or has it plateaued while the cluster is still degraded?

Ceph failed_repair: when ceph pg repair cannot fix the inconsistency

Mon, 01 Jan 0001 00:00:00 +0000

Ceph failed_repair: when ceph pg repair cannot fix the inconsistency

A failed_repair PG state means Ceph attempted to repair a data inconsistency that scrub or deep-scrub detected, and could not. The PG is now active+clean+inconsistent+failed_repair, the cluster is at HEALTH_WARN (typically via the OSD_SCRUB_ERRORS health check), and the inconsistency will not heal on its own. Reads still succeed from consistent replicas, but the cluster has stopped trying to fix the divergence until you intervene.

Ceph FS_DEGRADED: standby MDS failed to take over a rank

Mon, 01 Jan 0001 00:00:00 +0000

Ceph FS_DEGRADED: standby MDS failed to take over a rank

FS_DEGRADED fires when at least one CephFS rank is failed or damaged and a standby did not promote. Clients can usually still reach the filesystem through surviving ranks, but you are running without the failover reserve the MDS cluster was sized to provide.

FS_DEGRADED is the precursor to MDS_ALL_DOWN. If the last active rank fails before you restore a healthy standby, CephFS becomes fully unavailable. The playbook classifies FS_DEGRADED active for more than 120 seconds as a TICKET. If CephFS is a primary storage interface, treat 120 seconds as the upper bound on response time, not a soft target.

Ceph health detail: mapping ceph_health_detail checks to a cause

Mon, 01 Jan 0001 00:00:00 +0000

Ceph health detail: mapping ceph_health_detail checks to a cause

ceph_health_detail is the bridge between the umbrella status an alert fires on (ceph_health_status: 0=HEALTH_OK, 1=HEALTH_WARN, 2=HEALTH_ERR) and the specific fault you have to fix. Each health check is a separate gauge with name, severity, and message labels; value 1 means active, 0 means inactive. The name label is the check code (OSD_FULL, PG_AVAILABILITY, MON_CLOCK_SKEW, etc.) and is what you build alert routing, dashboards, and post-incident timelines around.

Ceph HEALTH_ERR: reading the umbrella status and finding the real fault

Mon, 01 Jan 0001 00:00:00 +0000

Ceph HEALTH_ERR: reading the umbrella status and finding the real fault

ceph_health_status == 2 is the most severe top-level status Ceph reports, and one of the most over-paged signals in production. The reason is structural: HEALTH_ERR is an umbrella aggregation. It tells you that one or more child health checks crossed into ERR severity. It does not tell you which subsystem failed, whether I/O is actually blocked, or whether the condition will self-resolve when PGs finish peering.

Ceph HEALTH_WARN: which warnings are noise and which are structural

Mon, 01 Jan 0001 00:00:00 +0000

Ceph HEALTH_WARN: which warnings are noise and which are structural

HEALTH_WARN is an umbrella status, not a single condition. It fires during normal recovery after an OSD restart, and it also fires when the cluster is one step from data loss. Both look identical if you only watch the top-level status. The common mistake is blanket-silencing WARN because it fires too often during recovery, then missing the structural warnings that signal real danger.

Ceph LARGE_OMAP_OBJECTS: the RGW bucket-index OMAP storm and resharding

Mon, 01 Jan 0001 00:00:00 +0000

Ceph LARGE_OMAP_OBJECTS: the RGW bucket-index OMAP storm and resharding

The LARGE_OMAP_OBJECTS health warning fires when a deep scrub finds a RADOS object carrying more OMAP keys than the configured threshold. On clusters running the RADOS Gateway (RGW), the most common trigger is the bucket-index shard: a single shard that has accumulated too many entries because a bucket holds millions of objects with too few index shards.

Once a shard crosses the threshold, the symptoms cluster around two areas. RGW clients see slow LIST responses and slow multipart coordination on the affected bucket. The OSD hosting the shard shows elevated commit latency, slow ops, and in the worst case BlueStore RocksDB pressure that resembles a compaction stall. The OSD usually remains up+in, so cluster-level health stays at WARN rather than ERR, which makes the problem easy to overlook until a user complains about a stuck ListObjects call.

Ceph MDS_ALL_DOWN: CephFS is completely unavailable

Mon, 01 Jan 0001 00:00:00 +0000

Ceph MDS_ALL_DOWN: CephFS is completely unavailable

MDS_ALL_DOWN means no active Metadata Server (MDS) rank exists for a CephFS filesystem. Every CephFS client depending on that filesystem blocks on metadata operations. This is a hard outage for CephFS, not a degradation. The health check ceph_health_detail{name="MDS_ALL_DOWN"} goes active the moment the Monitor has no active rank to assign for that filesystem.

The metadata itself is journaled in a RADOS pool. Unless MDS_DAMAGED is also firing, the metadata is intact. The outage is availability, not loss. Recovery means restoring at least one active MDS, after which clients reconnect and resume.

Ceph MDS_CACHE_OVERSIZED: cache above limit and the cap recall that follows

Mon, 01 Jan 0001 00:00:00 +0000

Ceph MDS_CACHE_OVERSIZED: cache above limit and the cap recall that follows

MDS_CACHE_OVERSIZED fires when an active Metadata Server daemon’s in-memory inode and capability cache crosses mds_cache_memory_limit * mds_health_cache_threshold. It is almost always workload-driven: a client has touched enough files in a short enough window that the MDS is caching more metadata than its budget allows.

Ceph MDS_DAMAGED: metadata journal or cache corruption

Mon, 01 Jan 0001 00:00:00 +0000

Ceph MDS_DAMAGED: metadata journal or cache corruption

MDS_DAMAGED means the CephFS Metadata Server has found damaged metadata, either in its journal or in on-disk structures read from the metadata pool, and has deliberately refused to continue serving the rank. CephFS may be partially or fully unavailable, and recovery is not the usual automatic failover path. A standby that takes over would replay the same suspect journal, so the cluster parks the rank until a human intervenes.

Ceph MON_CLOCK_SKEW: clock drift between monitors and election churn

Mon, 01 Jan 0001 00:00:00 +0000

Ceph MON_CLOCK_SKEW: clock drift between monitors and election churn

The MON_CLOCK_SKEW health check fires when the leader monitor detects clock drift beyond mon_clock_drift_allowed (default 0.05 seconds, 50 milliseconds) on any monitor in the quorum. It raises HEALTH_WARN, not HEALTH_ERR, and on its own it does not stop client I/O. But it is the precursor to a class of failure that does: repeated Paxos elections, slow map distribution, and, if the skew grows, monitor quorum loss.

Ceph MON_DOWN: a monitor out of quorum and reduced redundancy

Mon, 01 Jan 0001 00:00:00 +0000

Ceph MON_DOWN: a monitor out of quorum and reduced redundancy

MON_DOWN fires when one or more monitor daemons are not part of the active quorum. The common case is a 3-monitor cluster with one monitor gone: quorum still holds with 2 of 3, but the cluster has lost redundancy and tolerates zero further monitor loss before consensus collapses. The Prometheus Ceph mixin surfaces this as CephMonDown (warning) and escalates to CephMonDownQuorumAtRisk (critical) when the number of down monitors equals the minimum quorum count.

Ceph monitor election storm: monitors that cannot hold a stable quorum

Mon, 01 Jan 0001 00:00:00 +0000

Ceph monitor election storm: monitors that cannot hold a stable quorum

A Ceph monitor election storm is what happens when the MON cluster cannot complete and hold an election. Each round of Paxos leader election starts, partially completes, then restarts before the new leader can commit any map updates. The election epoch counter climbs, the leader name changes moment to moment, and ceph -s itself starts taking several seconds to return because even reading the current map requires a responsive leader.

Ceph monitor quorum lost: the cluster can no longer update its maps

Mon, 01 Jan 0001 00:00:00 +0000

Ceph monitor quorum lost: the cluster can no longer update its maps

Ceph monitor quorum loss is a PAGE condition. Without a majority of MONs agreeing through Paxos, the cluster cannot commit any map update: no OSD up/down transitions, no PG state changes, no pool edits, no CRUSH adjustments. Existing clients keep running on cached maps for a while, but new client connections fail immediately, and as soon as those cached maps go stale, in-flight I/O stalls.

Ceph monitoring checklist: the signals every production cluster needs

Mon, 01 Jan 0001 00:00:00 +0000

Ceph monitoring checklist: the signals every production cluster needs

This checklist defines the signals a production Ceph cluster needs, organized by monitoring maturity. Each level is a superset of the previous one; higher-level signals rely on lower-level context to distinguish real failures from normal churn. The four levels are survival, operational, mature, and expert. You cannot skip levels in practice.

Two caveats apply across all levels. First, ceph_health_status is an umbrella, not a complete picture: HEALTH_OK does not guarantee performance, and HEALTH_WARN covers both expected churn (active recovery) and structural problems (nearfull, noout trap, scrub inconsistency). Drill into ceph_health_detail before treating WARN as noise. Second, per-OSD and per-pool metrics matter more than cluster averages. A cluster at 60% average capacity with one OSD at 85% is closer to trouble than the average suggests.

Ceph monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Ceph monitoring maturity model: from survival to expert

Ceph’s health surface is unusually broad for a storage system. A cluster can report HEALTH_OK while a single OSD runs at 20x the latency of its peers, while the BlueStore DB partition spills to slow media, or while the PG autoscaler splits placement groups under live client load. The gap between “is the cluster up” and “is the cluster observable enough to operate” is where most preventable outages live.

Ceph nearfull: the 85% warning that decides whether the cluster can heal

Mon, 01 Jan 0001 00:00:00 +0000

Ceph nearfull: the 85% warning that decides whether the cluster can heal

OSD_NEARFULL or POOL_NEAR_FULL in ceph health detail means HEALTH_WARN. Clients are still reading and writing, and nothing looks broken yet. The nearfull ratio (default 0.85) is not a polite reminder to plan storage. It is the point where Ceph warns it is running out of the spare space it needs to heal itself.

One OSD failure at 85% can push surviving OSDs past backfillfull (0.90), at which point backfills refuse to start and recovery stalls. If another OSD fails in that window, you are left with degraded PGs that cannot be recovered, and the next stop is OSD_FULL at 95% where all client writes return ENOSPC.

Ceph noout flag left set: the most common preventable outage

Mon, 01 Jan 0001 00:00:00 +0000

Ceph noout flag left set: the most common preventable outage

The noout flag is the single most common source of preventable Ceph outages. An operator sets it before maintenance to stop Ceph from marking down OSDs as out, then forgets to unset it. The cluster keeps serving I/O, health stays at HEALTH_WARN (not ERR), and nothing visible breaks. But recovery is now disabled by policy. The next OSD failure stacks on top of the first, and degraded PGs that should have healed days ago are still degraded.

Ceph norecover and nobackfill: recovery intentionally, or accidentally, stopped

Mon, 01 Jan 0001 00:00:00 +0000

Ceph norecover and nobackfill: recovery intentionally, or accidentally, stopped

The symptom is familiar: PGs are stuck in degraded or undersized states, the cluster is HEALTH_WARN, but recovery throughput is zero. OSD load, network, and capacity all look normal. The cause is often two cluster-wide flags sitting in the OSD map: norecover and nobackfill. These flags are legitimate tools for protecting client I/O during recovery storms or maintenance, and a common source of “forgotten flag” incidents alongside noout.

Ceph OSD commit and apply latency: reading per-OSD latency outliers

Mon, 01 Jan 0001 00:00:00 +0000

Ceph OSD commit and apply latency: reading per-OSD latency outliers

Two metrics tell you more about per-OSD health than almost anything else in Ceph: ceph_osd_commit_latency_ms and ceph_osd_apply_latency_ms. They are per-OSD gauges exported by the manager’s Prometheus module, labeled by ceph_daemon, and they measure the OSD’s internal I/O time, not the latency your clients experience end-to-end. A cluster can show healthy aggregate latency while one OSD quietly runs 5x slower than its peers of the same device class. The clients whose objects land on that OSD see tail latency spikes; everyone else sees normal performance. Cluster-wide averages hide this. Per-OSD comparison surfaces it.

Ceph OSD down: telling a dead disk apart from a network blip

Mon, 01 Jan 0001 00:00:00 +0000

Ceph OSD down: telling a dead disk apart from a network blip

OSD_DOWN fires. ceph_osd_up flipped to 0. The 600-second countdown to OUT and recovery has begun.

Your first job: figure out whether this is a dead disk (act now, plan a replacement) or a network blip (wait, verify, let the OSD come back). Acting on the wrong diagnosis wastes disk and network I/O on a needless backfill, or worse, leaves a failing disk in service past the point where SMART was already warning you.

Ceph OSD flapping: OSDs cycling up and down and the peering storm that follows

Mon, 01 Jan 0001 00:00:00 +0000

Ceph OSD flapping: OSDs cycling up and down and the peering storm that follows

OSD flapping is a failure cascade. One OSD misses heartbeats, peers mark it down, its PGs start peering and recovering elsewhere, the OSD comes back, peering reverses, and the cycle repeats. Each flap mints a new OSD map epoch that every OSD in the cluster must process. The peering overhead from a single flapping OSD can slow dozens of healthy OSDs enough that they also miss heartbeats, and the cascade spreads.

Ceph OSD fullness imbalance: one OSD full while the cluster average looks fine

Mon, 01 Jan 0001 00:00:00 +0000

Ceph OSD fullness imbalance: one OSD full while the cluster average looks fine

Cluster writes are blocked. ceph status shows HEALTH_ERR with OSD_FULL active. But ceph df reports the cluster at 65% utilized. Both are correct.

The Ceph full ratio (default 0.95) is enforced per-OSD, not cluster-wide. When any single OSD crosses that threshold, Ceph refuses writes for every PG that OSD serves. Because CRUSH distributes PGs across OSDs, one full OSD can block writes to a large fraction of PGs even when ninety-nine other OSDs have ample free space.

Ceph OSD heartbeat timeouts: the osd_heartbeat_grace precursor to flapping

Mon, 01 Jan 0001 00:00:00 +0000

Ceph OSD heartbeat timeouts: the osd_heartbeat_grace precursor to flapping

Ceph OSDs declare each other down based on a simple rule: if a peer does not respond to heartbeat pings within osd_heartbeat_grace (default 20 seconds), it is reported to the monitors as unresponsive. Two peers from different failure domains must agree before the monitor marks the OSD down. This is the mechanism that turns a transient stall into an OSD state change, and repeated near-misses against this 20-second window are the most reliable leading indicator of OSD flapping.

Ceph OSD up/down vs in/out: the four states and mon_osd_down_out_interval

Mon, 01 Jan 0001 00:00:00 +0000

Ceph OSD up/down vs in/out: the four states and mon_osd_down_out_interval

Most Ceph operations references describe an OSD as “down” as if it were a single state. It is not. An OSD carries two independent flags: up/down (is the daemon alive?) and in/out (does CRUSH place data on it?). Combined, that produces four states, and the most dangerous one, down+in, is invisible if you only look at one flag.

The two flags are linked by a clock. After an OSD goes down, the monitors wait mon_osd_down_out_interval (default 600 seconds) before automatically flipping it to out. That 10-minute window is the gap between “OSD stopped” and “recovery starts.” That timer, and the flags on either side of it, are the vocabulary every recovery, flapping, and noout runbook assumes.

Ceph OSD_FULL: all writes stopped at the 95% full ratio

Mon, 01 Jan 0001 00:00:00 +0000

Ceph OSD_FULL: all writes stopped at the 95% full ratio

The cluster suddenly stops accepting writes. Client applications report ENOSPC errors. ceph status returns HEALTH_ERR. ceph health detail shows OSD_FULL active. Reads still succeed, but every write, update, and delete fails cluster-wide.

This is a hard stop, not a throttle. Ceph refuses all write operations once any OSD crosses the configured full_ratio (default 0.95). CRUSH spreads every PG across multiple OSDs, so a single full OSD can block writes to hundreds of PGs even when cluster-average utilization looks moderate.

Ceph osd_memory_target: cache eviction, RSS growth, and OSD OOM kills

Mon, 01 Jan 0001 00:00:00 +0000

Ceph osd_memory_target: cache eviction, RSS growth, and OSD OOM kills

An OSD repeatedly killed by the kernel OOM killer, or a host where OSDs flap up and down shortly after systemd restarts them, very often traces back to one knob: osd_memory_target. The same knob set too low for the working set produces a quieter failure: read latency climbs on HDD-backed OSDs as BlueStore evicts onodes and RocksDB block-cache pages the workload actually needs.

Ceph PG degraded: fewer replicas than the pool size, and when it matters

Mon, 01 Jan 0001 00:00:00 +0000

Ceph PG degraded: fewer replicas than the pool size, and when it matters

A degraded placement group in Ceph has fewer copies of some objects than the pool’s configured size. When an OSD goes down or is removed, every PG that had a replica on that OSD drops below target and enters the degraded state. The cluster can still serve I/O as long as the surviving replica count is at or above min_size, but redundancy is reduced until recovery rebuilds the missing copies.

Ceph PG down: no surviving replica for reads or writes

Mon, 01 Jan 0001 00:00:00 +0000

Ceph PG down: no surviving replica for reads or writes

A placement group in the down state has no replica that can serve I/O. Reads and writes to objects in that PG fail. Clients see EIO or ENODEV, and ceph health detail reports PG_AVAILABILITY with one or more PGs flagged down. Data is unavailable, not just degraded.

A PG can briefly pass through down during peering after an OSD failure, before activating on a surviving replica. The 300-second sustain on sum(ceph_pg_down) > 0 filters that transient window. Once a PG has been down past the sustain, no OSD in the acting set can serve it, and the cluster will not heal it without operator action.

Ceph PG incomplete: placement groups that cannot serve I/O

Mon, 01 Jan 0001 00:00:00 +0000

Ceph PG incomplete: placement groups that cannot serve I/O

An incomplete placement group means the PG cannot find enough authoritative data to serve reads or writes. This is a data-availability failure, and it does not self-resolve the way a transient peering or recovering state does.

Alert on sum(ceph_pg_incomplete) > 0 sustained for more than 300 seconds, summed across all pools. The 300 second sustain filters out cold-start peering after a cluster-wide restart, which typically completes within 60-120 seconds. Anything still incomplete after five minutes is genuinely stuck.

Ceph PG inconsistent (OSD_SCRUB_ERRORS): scrub found replica divergence

Mon, 01 Jan 0001 00:00:00 +0000

Ceph PG inconsistent (OSD_SCRUB_ERRORS): scrub found replica divergence

A scrub or deep-scrub finished comparing replicas for a placement group and found they disagree. The cluster surfaces this as OSD_SCRUB_ERRORS (often paired with PG_DAMAGED) and the affected PG sits in active+clean+inconsistent. Client reads still succeed because Ceph serves them from a consistent replica, but at least one copy in the acting set is corrupt.

The danger is not the symptom. Reads work, and Ceph did what it was designed to do: detect silent divergence. The danger is that the corruption was found, not fixed, and the window during which an uncorrupted replica survives is your margin of safety. If the OSD holding the good copy fails before you repair, the object becomes unreadable or unwritable.

Ceph PG stale: the monitor has not heard from the PG's OSDs

Mon, 01 Jan 0001 00:00:00 +0000

Ceph PG stale: the monitor has not heard from the PG’s OSDs

A stale placement group means the monitor cluster has stopped trusting the last status it received for that PG. The acting primary OSD that should be reporting state has gone silent, so the MON cannot confirm whether the PG is active, recovering, or unavailable. The recorded state is preserved but flagged as untrusted.

This differs from down. A down PG means the cluster has positively confirmed that no replica can serve I/O. A stale PG means the cluster does not know the current state because the reporting chain broke. Both can co-occur, but the response differs: down is confirmed replica loss; stale is a communication or reporting failure.

Ceph PG undersized: fewer copies than the pool wants to place

Mon, 01 Jan 0001 00:00:00 +0000

Ceph PG undersized: fewer copies than the pool wants to place

A Ceph cluster reports ceph health detail listing PGs in an undersized state. The cluster serves I/O at reduced redundancy. The warning does not resolve on its own and recovery stalls. A structural block prevents CRUSH from placing the configured number of replicas.

Unlike peering or recovering, undersized is not transient. The acting set has fewer OSDs than the pool’s configured size (the replication factor for replicated pools, or the k+m sum for erasure-coded pools). Ceph runs the PG at reduced redundancy because CRUSH cannot find enough distinct failure domains to satisfy the rule. The fix is rarely to wait. It requires changing capacity, topology, or the CRUSH rule.

Ceph PG_NOT_DEEP_SCRUBBED: scrub verification debt and undetected bit rot

Mon, 01 Jan 0001 00:00:00 +0000

Ceph PG_NOT_DEEP_SCRUBBED: scrub verification debt and undetected bit rot

PG_NOT_DEEP_SCRUBBED means Ceph’s proactive integrity check is falling behind. Deep scrub is the only mechanism that reads every byte of every object on every replica and verifies byte-for-byte consistency. When PGs miss their deep-scrub window repeatedly, silent corruption from bit rot, DRAM errors, firmware bugs, and incomplete writes after power loss accumulates without any signal.

The companion check PG_NOT_SCRUBBED covers the lighter daily scrub, which compares object metadata across replicas. Both checks surface through ceph health detail and the ceph_health_detail Prometheus metric exposed by the MGR module. There is no per-PG “overdue” gauge in the standard metrics pipeline, which is why scrub debt often goes unmonitored until someone runs ceph -s and sees a HEALTH_WARN they do not recognize.

Ceph PGs stuck peering: the slow peering loop after a mass restart

Mon, 01 Jan 0001 00:00:00 +0000

Ceph PGs stuck peering: the slow peering loop after a mass restart

After a full cluster power cycle, mass OSD restart, or wide maintenance event, peering should finish within a minute or two. When it does not, ceph -s shows a long list of PGs in peering or peering+activating, the count is not decreasing, and client I/O for those PGs is hung. HEALTH_WARN or HEALTH_ERR can sit there for 20, 30, or 60 minutes with no visible progress.

Ceph recovery stalled: degraded PGs that are not healing

Mon, 01 Jan 0001 00:00:00 +0000

Ceph recovery stalled: degraded PGs that are not healing

ceph -s shows HEALTH_WARN or HEALTH_ERR, the cluster reports non-zero degraded or undersized placement groups, and clients still appear served. The degraded PG count is not climbing, which feels like progress. It is not. A flat degraded count with zero recovery rate is one of the most dangerous states a Ceph cluster can sit in: nothing is healing, and the operator assumes the system is working through it.

Ceph recovery storm: rebuild traffic starving client I/O

Mon, 01 Jan 0001 00:00:00 +0000

Ceph recovery storm: rebuild traffic starving client I/O

A Ceph recovery storm occurs when the cluster’s self-healing machinery starves the workloads it is supposed to serve. After an OSD failure, host loss, or bulk OSD addition, recovery and backfill traffic floods the same disks, network links, and OSD CPU that client I/O depends on. Client latency climbs 10x to 100x, applications time out, and retries add more load on top of the recovery stream.

Ceph RGW failed requests: aborted request rate and what it actually counts

Mon, 01 Jan 0001 00:00:00 +0000

Ceph RGW failed requests: aborted request rate and what it actually counts

ceph_rgw_failed_req is one of the most commonly misread RADOS Gateway metrics. Operators see “failed” in the name and assume it counts HTTP 4xx and 5xx responses. It does not. It counts requests where the client connection was aborted before the response completed. That distinction changes who you page, where you look, and which signals you correlate with.

What it is and why it matters

ceph_rgw_failed_req is a per-instance counter exposed by each RGW daemon, scraped from the admin socket and labeled with instance_id (alongside ceph_daemon, instance, job). Its sibling counter is ceph_rgw_req, the total request count. The operational signal is the ratio between the two: a sustained failed-request rate above 5% of total traffic for more than 5 minutes is the playbook’s TICKET condition.

Ceph RGW garbage-collection backlog: deleted data still consuming space

Mon, 01 Jan 0001 00:00:00 +0000

Ceph RGW garbage-collection backlog: deleted data still consuming space

You deleted several terabytes of S3 objects, but ceph df shows raw usage barely moving. The cluster is approaching nearfull and the write freeze is coming. The deletes returned 204 to clients, so they succeeded from the S3 layer’s perspective, but the underlying RADOS objects are still on disk, queued behind the RADOS Gateway garbage collector.

This is one of the quietest contributors to “the cluster is full but we deleted everything”. RGW does not free object data inline on delete. It marks the head object as deleted, enqueues the data objects (tail segments, multipart parts) for asynchronous garbage collection, and relies on a background GC thread on each gateway to drain the queue. When that thread stops making progress, deleted data keeps consuming capacity indefinitely.

Ceph RGW GET/PUT latency: S3 request latency and queue length

Mon, 01 Jan 0001 00:00:00 +0000

Ceph RGW GET/PUT latency: S3 request latency and queue length

When users report slow S3 GET or PUT responses, the RGW daemon is rarely the root cause. RADOS Gateway is a stateless HTTP frontend that translates REST calls into RADOS object operations. Its observed latency is dominated by the time those underlying operations take, plus whatever queuing happens when the gateway has more in-flight work than it can drain.

The RGW perf counters expose two distinct kinds of signal: per-operation latency accumulators (ceph_rgw_op_get_obj_lat_sum/_count and ceph_rgw_op_put_obj_lat_sum/_count) for the S3 operations themselves, plus queue gauges (ceph_rgw_qlen and ceph_rgw_qactive) that show whether requests are piling up inside the daemon. Treating those signals together is the difference between “S3 is slow” and “S3 is slow because one OSD hosting a bucket index shard is in OMAP collapse.”

Ceph RGW orphaned multipart uploads: space that disappears from view

Mon, 01 Jan 0001 00:00:00 +0000

Ceph RGW orphaned multipart uploads: space that disappears from view

Pool capacity climbs toward nearfull but bucket listings and radosgw-admin bucket stats cannot account for it. The obvious explanations (client growth, snapshots, OMAP bloat) do not fit. This pattern is frequently caused by orphaned multipart upload parts in the RADOS Gateway (RGW) data pool.

S3 multipart uploads create a manifest, upload parts into the bucket’s data pool, then call CompleteMultipartUpload to assemble the final object. When that final step never happens, when the client retries uploads of the same key, or when RGW aborts an upload incompletely, the individual parts remain in RADOS. They occupy raw space but no live bucket index entry references them.

Ceph slow requests (SLOW_OPS): operations blocked past osd_op_complaint_time

Mon, 01 Jan 0001 00:00:00 +0000

Ceph slow requests (SLOW_OPS): operations blocked past osd_op_complaint_time

ceph_healthcheck_slow_ops greater than zero means operations on the cluster have crossed osd_op_complaint_time (default 30s) and are stuck, not merely slow. The SLOW_OPS health check surfaces them as a warning, and the metric itself is a live gauge pulled from ceph health detail.

Slow ops are a symptom of something downstream blocking I/O: a failing disk, a saturated BlueStore RocksDB DB, network timeouts between OSDs, or heavy deep-scrub on HDD during a maintenance window. The right first move is to read where in the pipeline each slow op is stuck before changing any config.

Ceph too many PGs per OSD: mon_max_pg_per_osd, peering cost, and sizing

Mon, 01 Jan 0001 00:00:00 +0000

Ceph too many PGs per OSD: mon_max_pg_per_osd, peering cost, and sizing

PG count per OSD is one of the few Ceph sizing decisions with direct cost in both directions. Too many PGs per OSD means more memory and CPU spent on peering, recovery scans, and PG log maintenance across more logical units. Too few means CRUSH cannot spread data evenly, producing hotspots on specific OSDs while others sit idle.

mon_max_pg_per_osd (default 250) is the cluster’s failsafe against runaway PG counts. When an OSD approaches this number, Ceph raises TOO_MANY_PGS and blocks new pool creation, pg_num increases, and replication factor changes. This is a guard rail, not a performance target. The existing cluster keeps serving I/O; only topology changes that would add PGs are blocked.

Ceph unfound objects: the cluster cannot locate a surviving copy

Mon, 01 Jan 0001 00:00:00 +0000

Ceph unfound objects: the cluster cannot locate a surviving copy

When a placement group enters recovery_unfound or backfill_unfound, the cluster has objects it knows should exist but cannot locate on any OSD that is currently up and in. ceph health detail surfaces this as the OBJECT_UNFOUND check, and the cluster-wide gauge ceph_num_objects_unfound rises above zero. Every known replica or erasure-coded chunk is on an OSD that is down, destroyed, or has not yet been probed.

How Ceph actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How Ceph actually works in production: a mental model for operators

Most Ceph incidents become legible the moment you stop reasoning about RBD, CephFS, and RGW as separate products and start reasoning about one system: RADOS placing objects on OSDs. The three client interfaces are thin translations. Underneath them, every write is an object, every object lives in a placement group, and every placement group is mapped to a set of OSDs by a deterministic algorithm.