Ceph PG_NOT_DEEP_SCRUBBED: scrub verification debt and undetected bit rot

PG_NOT_DEEP_SCRUBBED means Ceph’s proactive integrity check is falling behind. Deep scrub is the only mechanism that reads every byte of every object on every replica and verifies byte-for-byte consistency. When PGs miss their deep-scrub window repeatedly, silent corruption from bit rot, DRAM errors, firmware bugs, and incomplete writes after power loss accumulates without any signal.

The companion check PG_NOT_SCRUBBED covers the lighter daily scrub, which compares object metadata across replicas. Both checks surface through ceph health detail and the ceph_health_detail Prometheus metric exposed by the MGR module. There is no per-PG “overdue” gauge in the standard metrics pipeline, which is why scrub debt often goes unmonitored until someone runs ceph -s and sees a HEALTH_WARN they do not recognize.

Treat any sustained PG_NOT_DEEP_SCRUBBED as a real integrity risk. PGs that go weeks past their window are effectively unmonitored for corruption. If a corrupt replica’s OSD fails before the next deep scrub runs, recovery can rebuild from the corrupt copy and propagate the damage.

What this means

Ceph schedules two classes of scrub per PG:

• Light scrub (osd_scrub_min_interval, default 86400 seconds = 1 day) compares object metadata across replicas. Cheap and fast.
• Deep scrub (osd_deep_scrub_interval, default 604800 seconds = 7 days) reads every object on every replica and verifies byte-for-byte checksums. I/O-intensive; competes directly with client traffic and recovery.

When a PG has not been deep-scrubbed within its configured window, the PG_NOT_DEEP_SCRUBBED health check fires. The check is computed from the PGMap independently of the OSDs that actually run the scrub.

flowchart TD
    A[Deep scrub window expires] --> B{Scrub scheduled?}
    B -- No: noscrub / nodeep-scrub --> C[Flags block all scrubs]
    B -- No: PG not clean --> D[Peering / degraded / recovery]
    B -- No: OSD load too high --> E[Scrub deferred by scheduler]
    B -- Yes but never completes --> F[Reservation or chunk stall]
    C --> G[PG_NOT_DEEP_SCRUBBED]
    D --> G
    E --> G
    F --> G
    G --> H[Verification debt accumulates]
    H --> I[Corruption undetected]

Common causes

Quick checks

Run these before changing anything. All are read-only.

# Confirm the health check and see which PGs are overdue
ceph health detail | grep -A 20 'PG_NOT_DEEP_SCRUBBED\|PG_NOT_SCRUBBED'

# Check for flags that intentionally suppress scrubbing
ceph osd dump | grep flags
# Expect: no flags, or only noout during maintenance

# Find the oldest un-deep-scrubbed PGs (sorted by last deep scrub timestamp)
# Note: ceph pg dump can be slow on large clusters
ceph pg dump -f json 2>/dev/null | jq -r '.pg_stats[] | "\(.last_deep_scrub_stamp) \(.pgid)"' | sort | head -30

# See what is scrubbing right now
ceph pg dump | grep -E 'scrubbing|deep'

# Check recovery and backfill activity that competes with scrub
ceph pg stat

# OSD latency to see if scrub is being throttled by load
ceph osd perf

# Confirm configured intervals
ceph config dump | grep -E 'osd_scrub|osd_deep_scrub|osd_max_scrubs'

# Check whether the cluster is in active recovery
ceph -s

How to diagnose it

1. Confirm the flags are clear. ceph osd dump | grep flags is the fastest check. If noscrub or nodeep-scrub appears, scrubs are intentionally blocked. These flags are commonly set during peak hours or maintenance and then forgotten. Clear with ceph osd unset noscrub and ceph osd unset nodeep-scrub.

2. Quantify the debt. Sort PGs by last_deep_scrub_stamp and look at the tail. If the oldest PG is a few days past the interval, the cluster is mildly behind. If the oldest is weeks past, the integrity exposure is real. Prioritize catching up over client performance concerns.

3. Correlate with recovery. With osd_scrub_during_recovery at its default of false, any active recovery or backfill blocks new scrub scheduling. Check ceph pg stat for recovering or backfilling PGs. If recovery is the cause, the debt resolves when recovery completes, but verify recovery is actually making progress rather than stalled.

4. Check OSD load. Light scrubs respect osd_scrub_load_threshold (default 0.5 normalized load). On busy clusters, light scrubs may be deferred indefinitely during business hours. Deep scrubs that are overdue are scheduled regardless, but they still compete with client I/O on the same disks. Use ceph osd perf to find OSDs whose apply or commit latency is elevated.

5. Look for stuck scrubs. If PGs appear in scrubbing+deep state in ceph pg dump but the deep-scrub timestamps never advance, the scrub is hung. This is a different failure mode from “scrub never scheduled” and requires OSD-level investigation.

6. Verify the config actually applied. If you extended osd_deep_scrub_interval to accommodate a slower cluster, confirm the daemon computing the health check sees the new value. Setting the interval only under [osd] may leave the checker using the default, producing a persistent false warning. Set under [global] or explicitly for both [osd] and [mgr].

Metrics and signals to monitor

Fixes

Forgotten noscrub or nodeep-scrub flag

The single most common cause. Clear the flags and let the scheduler resume.

ceph osd unset noscrub
ceph osd unset nodeep-scrub

Watch ceph pg dump | grep -E 'scrubbing|deep' to confirm scrubs are scheduling. Track the count of overdue PGs over the next several hours. If you routinely suppress scrubs during peak hours, automate the set/unset so the flag does not get forgotten.

Recovery or backfill is starving scrub

With osd_scrub_during_recovery at its default of false, this is expected behavior during rebalancing. Do not enable scrub-during-recovery blindly on a busy cluster; it adds I/O contention to an already loaded system.

Options, in order of preference:

1. Let recovery finish. Verify it is making progress with ceph_pool_recovering_bytes_per_sec. If recovery is stalled, fix that first (see Ceph backfill_toofull: recovery blocked because target OSDs are full and Ceph degraded objects: reduced redundancy and the race against a second failure).
2. If recovery will run for days and you cannot tolerate the scrub debt, temporarily widen the scrub window under [global] so the health check stops warning, then restore the original value when recovery completes:

   ceph config set global osd_deep_scrub_interval <seconds>
   

3. As a last resort on a cluster with spare IOPS, enable scrubs during recovery. This increases client-visible latency; watch it closely:

   ceph config set osd osd_scrub_during_recovery true
   

OSD load is deferring scrubs

Deep scrubs that are overdue bypass osd_scrub_load_threshold, but they still yield to client I/O and can be slow enough that throughput cannot keep up. If the cluster has grown beyond what the scrub scheduler can clear in a week:

• Widen the deep-scrub interval under [global] to a value the cluster can sustain. A verified-but-monthly deep scrub is better than a weekly target that is never met.
• Schedule scrubs into a low-traffic window with osd_scrub_begin_hour and osd_scrub_end_hour.
• For pools with different integrity requirements, consider per-pool intervals.

Scrub is stuck and never completes

PGs showing scrubbing+deep with no progress for hours or days indicate a reservation or scheduling problem, not a throughput problem.

• Identify the stuck PGs: ceph pg dump | grep 'scrubbing.*deep' and compare timestamps across samples.
• Query the PG: ceph pg <pgid> query to see where in the scrub state machine it is parked.
• Check the OSD logs on the primary for scrub reservation messages.

If a scrub is genuinely hung, restarting the primary OSD or repeering the PG can unblock it. This is disruptive: it causes client I/O to stall on that PG and may trigger recovery. Snapshot the OSD logs first and prefer a maintenance window.

Config mismatch after interval change

If you extended osd_deep_scrub_interval and the warning will not clear, the daemon computing the health check is likely still using the old default. Set the value globally so all components see the same value:

ceph config set global osd_deep_scrub_interval <seconds>

Confirm with ceph config dump. Most config changes are picked up dynamically; if the warning does not clear on the next health evaluation, restart the active MGR (this triggers an MGR failover to standby).

Prevention

• Alert on the health check directly. Page or ticket when ceph_health_detail{name="PG_NOT_DEEP_SCRUBBED"} is active for more than 24 hours. Shorter windows fire during normal recovery.
• Alert on suppression flags. ceph_osd_flag_noscrub or ceph_osd_flag_nodeep-scrub set for more than 24 hours without a maintenance ticket is a structural risk.
• Track the oldest un-deep-scrubbed PG. Scrape ceph pg dump periodically and record the maximum age. Trending upward week over week means the cluster is losing the race.
• Automate flag set/unset. If you suppress scrubs during peak hours, drive it from a scheduler that always unsets, never relies on a human remembering.
• Size the scrub window to reality. A weekly target that is never met is worse than a monthly target that is always met. Measure actual scrub throughput and set the interval accordingly.
• Correlate scrub debt with recovery. Recovery is the most common legitimate cause of transient scrub debt. Make sure alerting distinguishes “debt during active recovery” from “debt with no recovery in progress”.

Monitoring with Netdata

• The ceph_health_detail metric with the name label exposes PG_NOT_DEEP_SCRUBBED and PG_NOT_SCRUBBED directly. Alert when the value is 1 for more than 24 hours.
• ceph_osd_flag_noscrub and ceph_osd_flag_nodeep-scrub let you gate the scrub-debt alert. If a flag is set during maintenance, the debt alert is expected.
• Per-second OSD latency (ceph_osd_apply_latency_ms, ceph_osd_commit_latency_ms) shows whether scrubs are running and how much client impact they cause. Correlate latency spikes with scrub windows.
• Recovery rate (ceph_pool_recovering_bytes_per_sec) distinguishes “scrub debt because recovery is running” from “scrub debt with no recovery in progress”. The two cases have different fixes.
• ceph_pg_inconsistent and ceph_pg_failed_repair are the downstream signals when scrub debt has already produced undetected corruption. Correlate with scrub history to determine whether the integrity window has been breached.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph blocked ops: client I/O stuck behind a single slow OSD
• Ceph BlueStore RocksDB compaction stalls: periodic latency spikes
• Ceph BLUEFS_SPILLOVER: RocksDB metadata spilling onto the slow device
• Ceph BlueStore allocator fragmentation: rising latency at moderate fullness
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph client latency vs OSD latency: fast disks, slow clients
• Ceph degraded objects: reduced redundancy and the race against a second failure
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators