Ceph recovery stalled: degraded PGs that are not healing

ceph -s shows HEALTH_WARN or HEALTH_ERR, the cluster reports non-zero degraded or undersized placement groups, and clients still appear served. The degraded PG count is not climbing, which feels like progress. It is not. A flat degraded count with zero recovery rate is one of the most dangerous states a Ceph cluster can sit in: nothing is healing, and the operator assumes the system is working through it.

Recovery that is slow produces a descending degraded count. Recovery that is progressing produces non-zero bytes per second on the recovery line. Recovery that is stalled produces a flat line on both: the PG count does not move, and ceph_pool_recovering_bytes_per_sec sits at or near zero. The window of reduced redundancy stays open indefinitely, and the next OSD failure lands on a cluster that cannot absorb it.

This article covers the specific causes of stalled recovery, the read-only checks that localize them, and the fixes with their tradeoffs. It assumes you understand PG states and CRUSH basics; for the broader model, see how Ceph actually works in production.

What this means

A degraded PG has fewer copies of some or all objects than the pool’s configured size. An undersized PG has fewer OSDs in its acting set than size. Both mean the cluster has scheduled rebuild work. When the count stays flat while recovery throughput is zero, that work is not executing.

There are only a few ways this happens:

• Recovery is administratively disabled (norecover, nobackfill).
• The targets for recovery refuse to accept data because they are at or above the backfillfull ratio.
• The PG is blocked on objects the cluster cannot find (recovery_unfound, backfill_unfound).
• The recovery throttle is set so low that effective throughput rounds to zero.
• A scheduler or software bug is preventing the recovery queue from draining.
• The cluster network between source and target OSDs is saturated or partitioned.

The diagnostic procedure is the same regardless of which cause you suspect: enumerate the flags, the target capacity, the unfound objects, and the recovery rate, and the answer falls out.

Common causes

Quick checks

These are safe read-only commands. Run them in this order; in most incidents the first three identify the cause.

# Cluster-level recovery state and degraded count
ceph -s

# Enumerate global cluster flags - look for norecover, nobackfill, noout, noup, nodown
ceph osd dump | grep flags

# Per-OSD capacity; flag column shows nearfull, backfillfull, full
ceph osd df

# List PGs in each stuck category; the time argument is seconds (default 300)
ceph pg dump_stuck degraded 300
ceph pg dump_stuck undersized 300
ceph pg dump_stuck unclean 300
ceph pg dump_stuck inactive 60
ceph pg dump_stuck stale 60

# Surface unfound objects and capacity-driven health checks
ceph health detail

# Count recovery-blocked PGs by sub-state
ceph pg ls backfill_toofull
ceph pg ls recovery_toofull
ceph pg ls recovery_unfound
ceph pg ls backfill_unfound

# Recovery rate per pool
ceph osd pool stats

# Effective recovery throttle config across all OSDs
ceph config dump | grep -E 'osd_max_backfills|osd_recovery_max_active|osd_recovery_sleep|osd_op_queue|osd_mclock_override_recovery_settings'

If ceph osd dump | grep flags shows norecover or nobackfill, the rest of the procedure is moot until the flag is cleared. If those flags are not set and the recovery rate is still zero, walk through the remaining causes.

How to diagnose it

flowchart TD
    A[Degraded PGs flat, recovery rate zero] --> B{norecover or nobackfill set?}
    B -- yes --> F1[Administrative block]
    B -- no --> C{PG_BACKFILL_FULL or backfill_toofull PGs?}
    C -- yes --> F2[Target OSDs too full]
    C -- no --> D{Unfound objects reported?}
    D -- yes --> F3[Recovery blocked on missing data]
    D -- no --> E{Throttle effectively zero or mClock overriding?}
    E -- yes --> F4[Throttle or scheduler]
    E -- no --> G{Cluster network saturated?}
    G -- yes --> F5[Network bottleneck]
    G -- no --> H[Scheduler bug or query specific PG]

1. Confirm the stall is real. Sample ceph -s and ceph pg stat at 30-second intervals for two or three samples. Degraded counts during active recovery should be monotonically decreasing. A flat count across three samples is a stall. Confirm ceph_pool_recovering_bytes_per_sec is also zero or near-zero.

2. Check global OSD flags. Run ceph osd dump | grep flags. norecover or nobackfill halts recovery completely. These are set intentionally during maintenance or incident response and then forgotten; they are the single most common cause of zero-throughput stalls. noout does not stop recovery itself; it only prevents DOWN OSDs from being marked OUT. Recovery to replacement OSDs still proceeds normally.

3. Check target OSD capacity. Run ceph osd df and look at the status or flag column for backfillfull or nearfull. The defaults are mon_osd_backfillfull_ratio 0.90 and mon_osd_full_ratio 0.95. When any OSD that CRUSH would choose as a recovery target is at or above backfillfull, PGs land in the backfill_toofull or recovery_toofull state and stay there. List them explicitly with ceph pg ls backfill_toofull. This is the failure mode described in Ceph backfill_toofull.

4. Check for unfound objects. Run ceph health detail and search for unfound. Unfound objects are objects the cluster believes should exist but cannot locate on any surviving OSD; they block the entire PG from completing recovery. PGs in this state show up as recovery_unfound or backfill_unfound. To list specific objects, query the PG: ceph pg <pgid> query and inspect the recovery_state section for unfound object details.

5. Check the recovery throttle and scheduler. Run ceph config dump | grep osd_recovery and check osd_max_backfills (default 1), osd_recovery_max_active (default 3 ), and osd_recovery_sleep. If osd_max_backfills is set to 0, recovery cannot start.

   On Quincy and later, the mClock scheduler is the default and ignores osd_max_backfills, osd_recovery_max_active, and osd_recovery_op_priority unless osd_mclock_override_recovery_settings = true. This is the most common operator gotcha on modern Ceph: legacy recovery tuning is set, expected to take effect, and produces no change in recovery rate. Check with ceph config get osd osd_op_queue; if the value is not wpq, mClock is active. Then check ceph config get osd osd_mclock_override_recovery_settings.

6. Check for a software bug if everything else is clean. No flags, no full OSDs, no unfound objects, no throttle misconfiguration, no obvious network saturation, and recovery is still zero. Tentacle (v20.2.0, released 2025-11-18) fixed a recovery/backfill hang caused by improper handling of items in the dmclock background cleanup thread. If you are on Squid or Reef and see this exact pattern, a scheduler bug is a real candidate.

7. Query the specific stuck PG. For each PG identified in step 1, run ceph pg <pgid> query. The recovery_state section names the specific blocker: waiting for backfill reserve, waiting on unfound, peering blocked, and so on. This is the definitive answer when higher-level signals are ambiguous.

Metrics and signals to monitor

The essential correlation is ceph_pg_degraded > 0 AND sum(ceph_pool_recovering_bytes_per_sec) ~= 0. Each signal in isolation is ambiguous; together they are diagnostic.

Fixes

Forgotten norecover or nobackfill

Clear the flag:

# Confirm before clearing - was maintenance actually completed?
ceph osd dump | grep flags

# Clear both
ceph osd unset norecover
ceph osd unset nobackfill

Recovery should begin within seconds. Watch the recovery rate climb on ceph -s. If you intentionally set the flag during an incident and the original problem is not yet resolved, leave it set and document why; clearing it prematurely can trigger a recovery storm that lands you in Ceph blocked ops.

Target OSDs at backfillfull

This is a capacity problem. Recovery cannot proceed until target OSDs have space to receive data.

• Add OSDs if you can. This is the only durable fix.
• Reweight the fullest OSDs down to push data onto less-full peers: ceph osd reweight osd.<id> 0.9. Watch ceph osd df to confirm the reweight is producing movement.
• Delete snapshots, RGW orphaned objects, or non-critical data to free space.
• As a temporary unblock, raise the backfillfull ratio: ceph osd set-backfillfull-ratio 0.95. This is a calculated risk: it lets Ceph write recovery data closer to the full ratio, leaving less headroom for the next failure. Revert it after recovery completes.

The deeper problem is that the cluster was run too close to capacity. A cluster that cannot tolerate losing one OSD without hitting backfillfull is on the edge of the capacity death spiral described in Ceph capacity death spiral.

Unfound objects

Unfound objects mean the cluster cannot find any surviving copy. This is potential data loss. Do not resolve this casually.

1. Identify the affected PGs: ceph health detail | grep unfound.

2. For each PG, query for unfound object details: ceph pg <pgid> query and inspect the recovery_state section.

3. Check whether any down OSD might still have the data. If an OSD is down rather than destroyed, bring it back up before declaring the objects lost. Many unfound conditions resolve when the missing OSD rejoins.

4. If all replicas truly are gone, mark the objects lost:

   # Destructive - the action argument (revert or delete) is required
   ceph pg mark_unfound_lost <pgid> revert
   # or
   ceph pg mark_unfound_lost <pgid> delete
   

   revert rolls back to a previous version of the object if one exists. delete removes the object entirely. Both are data-loss operations. Document what you did and why.

Recovery throttled too low

If osd_max_backfills is 0 or osd_recovery_max_active is 0, recovery cannot proceed. Raise them:

ceph config set osd osd_max_backfills 1
ceph config set osd osd_recovery_max_active 3

ceph config set persists across restarts. For runtime application without restart, ceph tell 'osd.*' injectargs is available but deprecated on newer releases; prefer ceph config set.

mClock ignoring legacy recovery tuning (Quincy+)

If you have set osd_max_backfills or osd_recovery_max_active and seen no change, mClock is overriding them. Two options:

# Option A: let legacy tuning take effect under mClock
ceph config set osd osd_mclock_override_recovery_settings true

# Option B: switch back to the WPQ scheduler entirely
ceph config set osd osd_op_queue wpq

Option A is preferred if you want to keep mClock for client I/O prioritization. Option B is the closer match to pre-Quincy behavior. Either way, the change is not instantaneous; recovery scheduling picks it up on the next PG scan. This is the most common stalled-recovery cause on Quincy and later, and it is a configuration misunderstanding, not a fault.

Cluster network bottleneck

If recovery rate spikes briefly then stalls, and OSD commit latency rises across multiple OSDs, the cluster network is saturated. This is most common when public and cluster networks share a NIC or when a single link carries both client and recovery traffic.

• Confirm with interface counters (ip -s link show, /proc/net/dev) on the cluster network interfaces.
• Check TCP retransmits in /proc/net/snmp (TCPRetransSegs).
• Throttle recovery to leave bandwidth for clients: lower osd_max_backfills to 1, set osd_recovery_sleep to a small value.
• Long term, separate public and cluster networks onto different physical links.

Software bug

If the diagnostic procedure rules out every other cause and you are on Squid or Reef, consider upgrading to Tentacle (20.2.x), which contains the dmclock recovery/backfill hang fix. Before upgrading, confirm the bug pattern matches: no flags, no full OSDs, no unfound objects, throttle is non-zero, mClock override is set correctly, network is fine, and recovery is still zero. Upgrading a Ceph cluster is a non-trivial operation; do not undertake it on a hunch.

Prevention

• Alert on the combination, not the parts. A degraded > 0 alert fires during every normal recovery. A recovery rate == 0 alert fires whenever nothing is recovering. The useful alert is degraded > 0 AND recovery_bytes_per_sec == 0 sustained > 600s. This catches the dangerous state without noise.
• Track cluster flags as first-class signals. norecover, nobackfill, noout, noscrub, and nodeep-scrub should each alert when set for longer than a maintenance window.
• Keep cluster utilization below 80% on the fullest OSD, not the average. Recovery needs spare capacity on the OSDs CRUSH actually selects as targets, which are not the average OSD.
• Verify recovery configuration after upgrades. Upgrading to Quincy or later switches the scheduler to mClock, and pre-existing recovery tuning stops taking effect. After any major upgrade, confirm osd_op_queue and osd_mclock_override_recovery_settings match your intent.
• Test recovery periodically. Take an OSD out (ceph osd out osd.<id>) on a non-critical pool during a planned window and confirm recovery progresses. This catches misconfiguration before a real failure does.

How Netdata helps

• The Ceph collector surfaces per-second ceph_pg_degraded, ceph_pg_undersized, ceph_pg_backfill_toofull, and ceph_pg_recovery_unfound counts, so a flat degraded PG line is visible as a sustained plateau rather than a slow-drifting average.
• ceph_pool_recovering_bytes_per_sec is collected per pool, making the degraded-and-not-healing correlation visible on a single chart rather than requiring two separate ceph commands.
• Cluster flags (ceph_osd_flag_norecover, ceph_osd_flag_nobackfill, ceph_osd_flag_noout) appear as their own metrics, so a forgotten flag is a constant signal rather than something discovered by re-running ceph osd dump.
• Health checks PG_BACKFILL_FULL, PG_RECOVERY_FULL, and unfound-object states are exposed via ceph_health_detail with the name label, allowing per-check anomaly detection rather than umbrella HEALTH_WARN noise.
• Per-OSD utilization is tracked per OSD, so a single target OSD above the backfillfull ratio is visible against its peers rather than hidden by the cluster average.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph blocked ops: client I/O stuck behind a single slow OSD
• Ceph BlueStore RocksDB compaction stalls: periodic latency spikes
• Ceph BLUEFS_SPILLOVER: RocksDB metadata spilling onto the slow device
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph client latency vs OSD latency: fast disks, slow clients
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph MON_CLOCK_SKEW: clock drift between monitors and election churn
• Ceph MON_DOWN: a monitor out of quorum and reduced redundancy