Ceph PG degraded: fewer replicas than the pool size, and when it matters

A degraded placement group in Ceph has fewer copies of some objects than the pool’s configured size. When an OSD goes down or is removed, every PG that had a replica on that OSD drops below target and enters the degraded state. The cluster can still serve I/O as long as the surviving replica count is at or above min_size, but redundancy is reduced until recovery rebuilds the missing copies.

Degraded is the PG state that generates the most false-positive alerts in Ceph environments. It is the expected, transient state during normal healing. Every OSD failure, every CRUSH weight change, and every PG autoscaler split produces degraded PGs while recovery runs. Alerting on ceph_pg_degraded > 0 floods ticket queues during ordinary operations and trains operators to ignore the signal.

The dangerous version is not degraded alone. It is degraded with recovery making no progress. When the cluster has fewer replicas than configured and is not rebuilding them, the window of reduced redundancy is not closing. A second failure during that window can permanently lose data. The signal worth alerting on is the combination: degraded count above zero and recovery rate near zero, sustained long enough to rule out a topology change in progress.

This article defines the degraded state precisely, walks through the lifecycle of a PG entering and leaving it, and shows how to tell the expected transient case from the stuck case that needs operator attention.

What “degraded” actually means

A replicated Ceph pool has a configured size (the number of copies the cluster tries to maintain) and a min_size (the minimum number of copies required to acknowledge writes). For a typical size=3 pool with the default min_size of 2, the cluster wants three copies of every object but will accept writes as long as two copies are reachable.

When a PG is active+degraded, at least one object in that PG has fewer than size copies on reachable OSDs. The most common cause is an OSD in the PG’s acting set going down. The objects that lived on that OSD still exist on the surviving replicas, so reads succeed, and writes still commit if enough replicas remain to satisfy min_size. The PG is operational, just at reduced redundancy.

Degraded is distinct from undersized, which the cluster reports when the PG’s acting set itself has fewer OSDs than the pool size. A PG can be degraded without being undersized (some objects missing replicas on a full-size acting set) and can be undersized without every object being degraded. The two often co-occur after an OSD loss, but they are not the same condition.

Degraded is also distinct from down and incomplete, which are the conditions that actually take data offline. A down PG has no reachable replica to serve I/O. An incomplete PG cannot reconstruct enough state to peer. Both are PAGE-candidate conditions. Degraded is not, because the PG is still serving I/O.

How degraded PGs are created and cleared

stateDiagram-v2
    [*] --> active_clean
    active_clean --> degraded: OSD down or CRUSH change
    degraded --> recovering: peers up, targets have space
    recovering --> active_clean: missing replicas rebuilt
    degraded --> degraded: recovery stalled (norecover, toofull, unfound)
    degraded --> down: second failure removes remaining replicas

The normal lifecycle is short and one-directional: a topology change produces degraded PGs, recovery runs, the PGs return to active+clean. The two failure modes worth catching are the self-loop, where recovery is blocked, and the bottom transition, where a second failure collapses a degraded PG into a down PG.

The primary OSD for a PG maintains a log of recent operations. When a peer comes back after a brief outage, recovery replays that log to bring the returning replica up to date. This is fast. When the outage was long enough that the log does not cover it, or when a new OSD replaces one that was marked out, the cluster falls back to backfill, which scans and copies the full PG contents. Backfill is much slower and competes directly with client I/O on the disks involved.

Both recovering and backfilling PGs are normally still flagged degraded until the missing replicas are fully reconstructed. The flag clears when the acting set is complete and every object has the configured number of copies.

Where it shows up in production

Degraded PGs appear in any of these situations:

• Planned OSD maintenance. You mark an OSD down, reduce its CRUSH weight, or stop the daemon. Its PGs go degraded, recovery backfills the data elsewhere, and they clear. Expected, transient, and the entire reason the cluster has redundancy.
• PG autoscaler activity. When the autoscaler splits or merges PGs, brief periods of degraded state appear during remapping as pgp_num steps to match pg_num. Also expected.
• Unplanned OSD failure. A disk dies, the OSD goes down, its PGs go degraded. Recovery should start after the OSD is automatically marked out, which happens after mon_osd_down_out_interval (default 600s).
• Host or rack failure. Multiple OSDs disappear at once. Many PGs go degraded simultaneously. Recovery load is high and competes with client I/O. This is the scenario where recovery rate matters most, because a second failure domain going down during recovery can cause data loss.
• Stuck recovery. The cluster has degraded PGs but recovery is not progressing. This is the case that needs operator intervention.

The first four are normal and self-resolving. The fifth is what your monitoring needs to catch.

When it matters: the recovery rate correlation

The defining question for any degraded state is whether recovery is making progress. These signals answer that together:

The alert that catches the dangerous case is:

sum(ceph_pg_degraded) > 0 AND sum(ceph_pool_recovering_bytes_per_sec) < threshold, sustained for more than 300 seconds.

The exact threshold for “near zero” depends on the cluster. The intent is “recovery has stopped,” not “recovery is slow.” A cluster with a single degraded PG and one HDD target may recover at well under 1 MB/s and still be making progress. Compare against the cluster’s own baseline recovery rate during normal healing, not an absolute number.

The 300-second sustain filters out normal topology transitions. PGs cycle through degraded during peering after any OSD state change, and recovery rate momentarily drops while the new OSD map propagates and peering completes.

Alerting on ceph_pg_degraded > 0 alone floods tickets because during any non-trivial recovery, that condition is true for hours. The recovery-rate correlation is not a refinement. It is the only version of this alert that produces signal instead of noise.

Common reasons recovery stalls while degraded

If the combined alert fires, the cause is almost always one of these.

Recovery flags left set. Someone set norecover or nobackfill during an incident or maintenance window and forgot to clear them. Check with ceph osd dump | grep flags, or via the ceph_osd_flag_norecover and ceph_osd_flag_nobackfill metrics. This is the most common cause of stuck recovery, the easiest to fix, and the operational error most teams make repeatedly.

Target OSDs too full. Recovery needs somewhere to write the rebuilt replicas. If the OSDs that CRUSH wants to place them on are above the backfillfull ratio (default 0.90), backfill is refused and the PGs sit in backfill_toofull or recovery_toofull. This is a capacity problem, not a recovery problem, and usually means the cluster does not have enough spare space to heal itself. See the capacity death spiral pattern for the cascade this triggers.

Unfound objects. Recovery needs a source copy of every object it rebuilds. If every copy of some object is on OSDs that are down or destroyed, the object is “unfound.” Recovery of that PG blocks until the operator decides what to do. ceph_num_objects_unfound rising alongside stuck degraded PGs is a serious signal. Resolving it may require:

# DATA LOSS: this command destroys or reverts unfound objects.
# Only run after confirming the objects are truly unrecoverable.
ceph pg <pgid> mark_unfound_lost revert|delete

This is a data-loss decision. Do not run it without understanding what revert versus delete does in your specific case.

Recovery tail effect. Late in a large recovery, only a few PGs remain degraded, and only a few OSDs are participating. Throughput drops because there is less parallelism, not because anything is broken. The degraded count still trends down, just slowly. If the count is moving, this is not a stall. If operators mistake the tail effect for a stall and start “fixing” things, they can make it worse.

Throttling too aggressive. osd_max_backfills (default 1) and osd_recovery_max_active (default 3) cap how much recovery work each OSD does. Set too low for the cluster’s failure rate, recovery cannot keep up with incoming failures. The fix is tuning, not clearing flags.

Why a second failure is the real risk

Degraded and stalled deserves a ticket not because the cluster is broken now, but because it is one failure away from data loss. Every degraded PG has at least one object with fewer replicas than configured. If the OSD holding a surviving replica of that object also fails before recovery completes, the object may have no remaining copies.

This is why topology matters when reading the degraded count. A cluster with three OSDs down on three different hosts, with each pool using host-level failure domains, has many degraded PGs but is still tolerating one more host failure per PG. A cluster with three OSDs down on the same host, in a pool using rack-level failure domains, may already be at the edge of what its redundancy can absorb. The raw count of degraded PGs does not tell you which case you are in. The OSD topology, the CRUSH rules, and the failure domain configuration do.

This is also why “degraded but recovering” belongs at INFO severity. The cluster is closing the exposure window as fast as its throttles allow. Paging on it trains people to mute the alert, which defeats the purpose of having the signal.

How Netdata helps

• The ceph_pg_degraded metric, summed across all pools, gives you the count of PGs in the state. Per-pool labels show whether degradation is concentrated in one pool or spread across the cluster.
• ceph_pool_recovering_bytes_per_sec at per-second resolution lets you correlate recovery rate with the degraded count immediately, rather than reconstructing it after the fact. This correlation is the single most useful thing you can do for this signal.
• ceph_osd_flag_norecover and ceph_osd_flag_nobackfill expose administrative flags as metrics. A dashboard showing degraded PGs next to these flags makes the “someone forgot to clear a flag” case obvious in seconds.
• ceph_pg_backfill_toofull, ceph_pg_recovery_toofull, ceph_pg_recovery_unfound, and ceph_pg_backfill_unfound surface the specific sub-states that block recovery. Each has a different remediation, and seeing which is active directs the response instead of sending you on a generic recovery-tuning hunt.
• Anomaly detection on the recovery rate baseline catches the “slower than it should be” case before the absolute threshold fires. A cluster whose recovery rate has dropped to a tenth of its normal baseline is worth investigating even if it is not technically zero.
• Correlating degraded state with OSD down events, OSD commit and apply latency, and per-OSD capacity in a single view shortens the path from “degraded PGs exist” to the specific OSD that failed and the specific OSDs too full to receive recovery.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph monitoring checklist: the signals every production cluster needs
• Ceph monitoring maturity model: from survival to expert
• Ceph OSD down: telling a dead disk apart from a network blip
• Ceph OSD flapping: OSDs cycling up and down and the peering storm that follows
• Ceph OSD_FULL: all writes stopped at the 95% full ratio
• Ceph OSD fullness imbalance: one OSD full while the cluster average looks fine