Ceph degraded objects: reduced redundancy and the race against a second failure

Degraded objects in Ceph are the most direct measure of reduced redundancy: somewhere in the cluster, at least one object has fewer live replicas (or fewer erasure-coded fragments) than its pool requires. The cluster is still serving I/O, but the protection against a follow-on failure has thinned for those objects.

ceph_num_objects_degraded is not alarming on its own. Every OSD failure, restart, and CRUSH reweight produces a transient spike that recovery is designed to heal. The real signal is the slope: is the count falling back toward zero, or has it plateaued while the cluster is still degraded?

What this means

A degraded PG has fewer live replicas than its pool’s size (replicated pools) or fewer data and parity chunks than its erasure-coded profile requires. The PG can still be active+degraded, serving reads and writes, but each accepted write goes to fewer-than-intended copies. The safety margin for a second failure inside that PG is reduced.

This is distinct from misplaced, the other common non-clean state. Misplaced objects have full redundancy. They are simply on OSDs that do not match the current CRUSH up set, typically because OSDs were added, removed, or reweighted. Misplaced is data-safe. Degraded is not.

The Prometheus MGR module exposes both as cluster-wide gauges:

• ceph_num_objects_degraded: objects with fewer replicas than pool size.
• ceph_num_objects_misplaced: objects on the wrong OSDs but with full copies.

Both rise and fall during normal healing, and both can be present simultaneously. Treating them as the same signal is the first mistake operators make.

The “race against a second failure” is the operational consequence. Every minute the cluster has degraded objects, the failure domain that would push a PG from degraded to incomplete (or to permanent data loss) is one disk, host, or rack failure closer. Recovery rate determines the length of that exposure window.

flowchart TD
    A[OSD goes down] --> B[PGs marked degraded]
    B --> C{Recovery progressing?}
    C -- Yes, bytes/sec nonzero --> D[Degraded count falls]
    C -- No, backfill_toofull --> E[Blocked: target OSDs full]
    C -- No, unfound objects --> F[Blocked: data possibly lost]
    C -- No, norecover/nobackfill --> G[Blocked: admin flag set]
    C -- No, noout trap --> H[Blocked: OSD never marked out]
    E --> I[Race window stays open]
    F --> I
    G --> I
    H --> I
    D --> J[Window closes, PGs active+clean]
    I --> K{Second failure before close?}
    K -- Yes --> L[PGs go incomplete or data is lost]
    K -- No --> J

Common causes

Quick checks

# Cluster health and degraded/misplaced object counts
ceph health detail

# Recovery rate and PG state summary
ceph -s

# Object counts and PG state at the metric layer
ceph pg stat

# List PGs stuck in degraded state beyond a threshold (seconds)
ceph pg dump_stuck degraded 600

# Per-PG detail for a specific degraded PG
ceph pg <pgid> query

# Check global recovery and maintenance flags
ceph osd dump | grep flags

# Hierarchical view with per-OSD utilization and variance
ceph osd df tree

The single most useful diagnostic when a PG refuses to heal is ceph pg <pgid> query. It shows the acting set, recovery state, blocking reason, and PG log.

How to diagnose it

1. Confirm degraded count and trajectory. Read ceph health detail for the degraded/misplaced fraction, then sample ceph pg stat a few minutes apart. A falling count means recovery is working and this is not an incident. A flat or climbing count means recovery is stalled and the race window is open.

2. Confirm recovery rate is nonzero. Watch ceph -s recovery line or the per-pool ceph_pool_recovering_bytes_per_sec. Zero bytes per second with nonzero degraded objects is the actual alert condition, not the degraded count by itself.

3. Rule out the easy admin-flag causes. Check ceph osd dump | grep flags for norecover, nobackfill, and noout. noout combined with OSDs down is the classic “noout trap”: recovery never triggers because the down OSD is never marked out.

4. Check for capacity blocking. Run ceph pg dump_stuck degraded alongside ceph osd df. If any OSDs are above the backfillfull ratio (default 0.90), recovery for PGs that need to land on those OSDs is blocked, producing backfill_toofull or recovery_toofull states. This is the leading indicator of a capacity death spiral.

5. Check for unfound objects. ceph health detail names specific PGs in recovery_unfound or backfill_unfound state when the cluster cannot find a known object on any live OSD. ceph_num_objects_unfound > 0 is the strongest signal that the race is being lost: at least one object may already be gone.

6. Check for OSD flapping. A flapping OSD oscillates the degraded set. Each down transition creates degraded PGs, each up transition starts peering, and recovery never makes net progress. ceph health detail surfaces an OSD_FLAPPING check when this is happening.

7. Query the specific stuck PG. ceph pg <pgid> query shows the acting set, recovery state, and blocking reason. Use it on any PG that stays degraded across multiple samples.

Metrics and signals to monitor

Fixes

Degraded with active recovery: leave it alone

If ceph_num_objects_degraded is trending down and ceph_pool_recovering_bytes_per_sec is nonzero, recovery is doing its job. Do not throttle unless client impact is unacceptable. Reef-era Ceph (18.2.0 release notes) states that degraded-object recovery is given higher priority than misplaced-object recovery, so a cluster with both should see the degraded count fall first.

Recovery stalled by noout

Unset the flag once maintenance is complete:

# Triggers data movement once the down OSD is marked out
ceph osd unset noout

Verify recovery starts. If the OSD is genuinely failed and will not return, plan to destroy or purge it after confirming data has been recovered elsewhere. Setting noout and forgetting it is the most common preventable Ceph outage, so any noout > 24 hours condition should be a ticket.

Recovery stalled by norecover / nobackfill

Same fix. Unset the flags:

ceph osd unset norecover
ceph osd unset nobackfill

These are maintenance flags. If you set them to free I/O during a sensitive operation and forgot, the cluster has been running at reduced redundancy since that moment.

Recovery blocked by full target OSDs

This is structural and the most dangerous stall. Options in order of preference:

1. Add capacity: new OSDs, a new host, or a new device class.
2. Delete data: expired snapshots, abandoned RGW multipart uploads, or non-critical pools. Check ceph df detail for snapshot space accounting, since deleted data inside snapshots still consumes raw space.
3. Reweight the fullest OSDs down so CRUSH places less new data on them: ceph osd reweight <osd-id> <new_weight>. This does not move existing data off; it only stops the OSD from receiving new placements.
4. As a last resort, temporarily raise mon_osd_backfillfull_ratio. This buys time only. Without added capacity the cluster fills again, and the next OSD failure still has nowhere to recover to.

A cluster that cannot heal from a single OSD failure because the surviving OSDs are too full has effectively zero failure tolerance. Track this proactively, not at incident time.

Recovery blocked by unfound objects

The cluster knows an object should exist but cannot find any live copy. Check the specific PGs in ceph health detail, then ceph pg <pgid> query to see which OSDs the recovery process is waiting on.

If the missing OSD can be brought back online (restarted, host recovered), the object reappears and recovery completes. If the OSD is permanently gone, the remaining options accept data loss:

# Destructive: declares the object permanently gone
ceph pg <pgid> mark_unfound_lost revert|delete

revert uses an older version of the object if one exists; delete removes it entirely. Both are data loss. Review carefully and escalate before running this in production.

OSD flapping

Stabilize the OSD first; recovery will follow.

1. ceph osd add-noout osd.<id> to stop the CRUSH remap churn.
2. Stop the daemon if it keeps cycling: ceph osd down osd.<id> then systemctl stop ceph-osd@<id>.
3. Investigate the root cause: SMART attributes, network errors, OOM kills in dmesg, or BlueStore DB spillover causing compaction stalls.

Prevention

• Alert on the combination, not the count. Page or ticket on ceph_num_objects_degraded > 0 AND sum(ceph_pool_recovering_bytes_per_sec) ~ 0 sustained > 600s. Alerting on the degraded count alone fires during every normal heal and trains operators to ignore the signal.
• Track the slope, not the value. Dashboards that chart ceph_num_objects_degraded over time make a stalled recovery visually obvious. A flat line is the problem; a falling line is recovery working.
• Watch the noout trap. Any ceph_osd_flag_noout == 1 for more than 24 hours is a ticket, full stop. Pair it with down-OSD detection so the alert fires when noout is set and OSDs are actually down.
• Project capacity against failure scenarios. Model what happens to the most-full OSD if you lose the largest host. If the answer crosses backfillfull, you have no recovery headroom.
• Separate degraded from misplaced in dashboards. Misplaced objects are data-safe and should not escalate. Treating them as degraded creates constant noise during every CRUSH change.
• Verify recovery behavior periodically. Stop a non-critical OSD in a test window and measure how fast the cluster heals with current throttle settings. Recovery tuning that was safe for a 10-OSD cluster is wrong for a 100-OSD cluster.

How Netdata helps

• Per-second ceph_num_objects_degraded and ceph_num_objects_misplaced side by side makes the slope-vs-value distinction immediate. A flat degraded line with active recovery is visually obvious within seconds.
• Correlating ceph_pool_recovering_bytes_per_sec with ceph_num_objects_degraded turns the “is recovery stalled?” question into a single chart read. The anomaly advisor flags the combination where degraded is flat and recovery is zero.
• Per-pool ceph_pg_* state metrics let you slice degraded PGs by pool and see whether the stall is concentrated on one OSD’s PGs or distributed across the cluster.
• ceph_osd_flag_* metrics surface the noout trap as a clean binary signal alongside OSD down state, so a forgotten flag becomes visible without parsing ceph osd dump.
• ceph_num_objects_unfound and the ceph_pg_*_unfound states put the worst-case outcome, possible data loss, on the same dashboard as the recovery rate, making escalation decisions obvious.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph blocked ops: client I/O stuck behind a single slow OSD
• Ceph BlueStore RocksDB compaction stalls: periodic latency spikes
• Ceph BLUEFS_SPILLOVER: RocksDB metadata spilling onto the slow device
• Ceph BlueStore allocator fragmentation: rising latency at moderate fullness
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph client latency vs OSD latency: fast disks, slow clients
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph MON_CLOCK_SKEW: clock drift between monitors and election churn