Ceph norecover and nobackfill: recovery intentionally, or accidentally, stopped

The symptom is familiar: PGs are stuck in degraded or undersized states, the cluster is HEALTH_WARN, but recovery throughput is zero. OSD load, network, and capacity all look normal. The cause is often two cluster-wide flags sitting in the OSD map: norecover and nobackfill. These flags are legitimate tools for protecting client I/O during recovery storms or maintenance, and a common source of “forgotten flag” incidents alongside noout.

The risk is operational, not technical. Nothing in ceph -s explicitly announces “recovery is paused by administrative action.” A degraded cluster with these flags set looks very similar to a degraded cluster where recovery is stalled for some other reason. The distinguishing signal is that the recovery rate is exactly zero while the flags are present.

What this means

norecover and nobackfill are cluster-wide OSD map flags that halt recovery machinery.

• norecover blocks log-based recovery. When an OSD returns after a brief outage, Ceph normally uses the PG log to identify and replay only changed objects. With norecover set, this does not happen.
• nobackfill blocks full PG content migration. Backfill copies entire PG contents rather than just changed objects, and is used when an OSD has been missing too long for the PG log to cover the gap, or when OSDs are added or removed.

Both flags are set and unset with ceph osd set and ceph osd unset. They take effect cluster-wide. In-progress recovery pauses when the flag is set and resumes from where it left off when the flag is cleared.

The companion flag norebalance is narrower. It prevents data movement when a PG is remapped but not degraded, and by design does NOT block recovery of genuinely degraded PGs. It is still worth checking all three flags together, because a PG can be both remapped and degraded, and the precise interaction depends on which replicas are missing.

These flags exist for good reasons. The two most common legitimate uses are recovery storm protection and maintenance windows. Multiple OSD failures can trigger massive data movement that competes with client I/O; setting norecover and nobackfill temporarily pauses that movement while you throttle or investigate. During planned OSD work, the flags prevent the cluster from overreacting to temporary state changes.

The health check OSDMAP_FLAGS fires when any of these flags are set. Treat that warning as a pointer, not noise.

Common causes

Quick checks

# Show current OSD map flags - authoritative check
ceph osd dump | grep -E "norecover|nobackfill|norebalance|noout"

# Cluster summary - look for degraded/recovering PG counts
ceph -s

# Health detail - OSDMAP_FLAGS fires when these flags are set
ceph health detail | grep -E "OSDMAP_FLAGS|norecover|nobackfill|norebalance"

# PG state summary - degraded count with zero recovery is the signature
ceph pg stat

# Stuck PGs by category - confirms degraded PGs are not progressing
ceph pg dump_stuck degraded
ceph pg dump_stuck undersized

# Per-pool recovery rate - confirms recovery is truly stalled
ceph osd pool stats

# OSD capacity state - rules out backfill_toofull as the actual cause
ceph osd df

How to diagnose it

1. Confirm the flag is set. Parsing ceph osd dump for the flag names is more reliable than reading ceph -s. Any of norecover, nobackfill, or norebalance appearing here is a candidate cause.

2. Verify recovery is actually stalled. Cross-check that degraded or undersized PGs exist (ceph pg stat) and that the recovery rate is zero or near-zero (ceph osd pool stats). If PGs are healthy, the flag is harmless. If PGs are degraded but recovery is progressing, the flag is not your bottleneck.

3. Rule out other stall causes. Recovery can also be blocked by backfill_toofull (target OSDs too full), recovery_unfound or backfill_unfound (objects cannot be located), or downstream issues like network partitions. Check ceph health detail for these specific PG states.

flowchart TD
  A[Degraded PGs, recovery rate near zero] --> B{norecover or nobackfill set?}
  B -- Yes --> C{Maintenance window open?}
  C -- Yes --> D[Track ticket, unset after window]
  C -- No --> E[Throttle, then unset flags]
  B -- No --> F{backfill_toofull or unfound?}
  F -- Yes --> G[Capacity or unfound root cause]
  F -- No --> H[Check norebalance, network, OSD load]

4. Check norebalance alongside the others. If norebalance is also set and you intend full recovery to resume, include it in the unset.

5. Quantify exposure. Count degraded PGs and estimate the redundancy loss. While the flag is set, every degraded PG is one failure away from data loss. Flag-state plus degraded PGs is a ticket-worthy condition regardless of intent.

Metrics and signals to monitor

Fixes

If the flag is accidentally set

Before unsetting, especially if the flag has been set for a long time, consider throttling recovery to avoid releasing a wave of accumulated backfill pressure. Operators have reported OSDs crashing immediately when nobackfill is unset after a long pause, because every pending backfill rushes to start at once.

# Throttle recovery before unblocking - safer for clusters with accumulated backlog
ceph config set osd osd_recovery_max_active 1
ceph config set osd osd_recovery_op_priority 1

# Unset the flags
ceph osd unset norecover
ceph osd unset nobackfill

# If norebalance was also set, unset it too
ceph osd unset norebalance

After unsetting, watch the recovery rate climb and the degraded PG count fall. Once recovery completes and the cluster returns to HEALTH_OK, return the throttle values to their previous settings.

If the flag is intentionally set

Document why. Record the expected unset time in your runbook or change management system. The alert for norecover or nobackfill set while degraded PGs exist should fire as a reminder, not be silenced permanently.

If the flag was set to weather a recovery storm and the storm has passed, unset it. The recovery storm pattern, where multiple OSD failures cascade into client-visible latency, is handled in more depth in Ceph capacity death spiral.

If OSDs crash on unset

This is a sign that accumulated backfill pressure is overwhelming target OSDs. Re-set the flag immediately, throttle more aggressively, and consider reweighting the fullest OSDs down to spread load across more targets.

# Re-set if OSDs crash on unset
ceph osd set nobackfill

# Throttle harder
ceph config set osd osd_recovery_max_active 1
ceph config set osd osd_recovery_op_priority 1

# Reweight the fullest OSDs down to spread backfill targets
# NOTE: osd id is the integer ID, not osd.<id>
ceph osd reweight <osdid> 0.9

Prevention

• Treat OSD map flags as state, not commands. Every ceph osd set should have an accompanying ticket with an expected unset time. This is the same discipline that prevents the noout trap.
• Alert on flag state correlated with PG state. The condition to alert on is not “flag is set” but “flag is set while degraded PGs exist.” That combination is always actionable.
• Include flag checks in your runbook verification step. After any maintenance, the closing checklist should verify ceph osd dump | grep -E "norecover|nobackfill|norebalance|noout" returns only expected flags.
• Monitor norebalance too. It is less obvious than norecover and nobackfill but can silently block recovery of remapped PGs.
• Track flag dwell time. A flag set for more than 24 hours with no open maintenance ticket is almost certainly forgotten.

How Netdata helps

Netdata’s Ceph collector surfaces OSD map flags as ceph_osd_flag_{name} metrics, so you can alert on norecover and nobackfill state without parsing CLI output.

• The correlation that matters most is ceph_osd_flag_norecover == 1 OR ceph_osd_flag_nobackfill == 1 combined with sum(ceph_pg_degraded) > 0. That composite is ticket-worthy regardless of whether the flag is intentional.
• Pair flag state with ceph_pool_recovering_bytes_per_sec. If recovery rate is zero while the flag is set and degraded PGs exist, recovery is paused by administrative action rather than by a structural failure.
• The ceph_health_detail{name="OSDMAP_FLAGS"} check surfaces as a labeled metric, giving you a single signal for “any interesting OSD map flag is currently set.”
• ML anomaly detection on recovery rate helps distinguish a flag-paused flatline from a structurally stalled recovery. Both produce near-zero throughput, but the surrounding context differs.
• Per-second collection means you see the exact moment a flag is set or unset, useful for correlating with change events and incident timelines.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph blocked ops: client I/O stuck behind a single slow OSD
• Ceph BlueStore RocksDB compaction stalls: periodic latency spikes
• Ceph BLUEFS_SPILLOVER: RocksDB metadata spilling onto the slow device
• Ceph BlueStore allocator fragmentation: rising latency at moderate fullness
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph client latency vs OSD latency: fast disks, slow clients
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph MON_CLOCK_SKEW: clock drift between monitors and election churn