Ceph PG undersized: fewer copies than the pool wants to place

A Ceph cluster reports ceph health detail listing PGs in an undersized state. The cluster serves I/O at reduced redundancy. The warning does not resolve on its own and recovery stalls. A structural block prevents CRUSH from placing the configured number of replicas.

Unlike peering or recovering, undersized is not transient. The acting set has fewer OSDs than the pool’s configured size (the replication factor for replicated pools, or the k+m sum for erasure-coded pools). Ceph runs the PG at reduced redundancy because CRUSH cannot find enough distinct failure domains to satisfy the rule. The fix is rarely to wait. It requires changing capacity, topology, or the CRUSH rule.

This guide covers common root causes (typically configuration or capacity issues, not runtime faults), read-only checks to localize the cause, and the tradeoffs of remediation paths.

What this means

When a PG is undersized, CRUSH maps it to an acting set with fewer OSDs than size. For a replicated pool with size=3 and min_size=2, the acting set typically has two OSDs instead of three. The PG serves I/O at min_size, but another failure in the same PG pushes it below min_size and blocks writes or loses data.

undersized frequently co-occurs with degraded and either active or peered:

• active+undersized+degraded: Serving I/O at reduced redundancy, missing one or more replicas.
• peered+undersized+degraded: The acting set is below min_size, so the PG cannot serve writes.
• active+clean+undersized: The PG is clean given the smaller acting set, but Ceph still reports the set is below size. This occurs when the pool’s size cannot be satisfied by the available failure domains.

The state is sticky. Unlike recovering or backfilling, it does not clear on its own once the topology is wrong. It clears only when CRUSH can place the PG on enough distinct failure domains.

The relevant metric is ceph_pg_undersized (per pool_id). Alert on sum(ceph_pg_undersized) > 0 sustained for more than 300 seconds, correlated with the TOO_FEW_OSDS health check and CRUSH placement signals.

Common causes

Quick checks

All commands here are read-only and safe during production traffic. ceph pg dump on a large cluster is slow and generates MON load, so prefer targeted JSON queries over the full dump when triaging.

# Cluster summary and current health checks
ceph status
ceph health detail

# List PGs currently stuck in undersized state
ceph pg dump -f json | jq '.pg_stats[] | select(.state | contains("undersized")) | {pgid: .pgid, state: .state}'

# OSD topology and status
ceph osd tree

# Per-OSD utilization and CRUSH weights
ceph osd df tree

# Cluster flags that could be blocking recovery
ceph osd dump | grep flags

# CRUSH rule for the affected pool (replace <pool>)
ceph osd pool get <pool> crush_rule
ceph osd crush rule dump

# Per-PG query for a specific stuck PG (replace <pgid>)
ceph pg <pgid> query

How to diagnose it

flowchart TD
    A["ceph_pg_undersized > 0"] --> B{"All PGs in one pool
affected?"}
    B -- "Yes" --> C["Structural: CRUSH rule,
failure domain count, pool size"]
    B -- "No" --> D{"Recovery rate
non-zero?"}
    D -- "Yes" --> E["Topology event.
Wait for healing."]
    D -- "No" --> F{"Flags set?"}
    F -- "norecover / nobackfill" --> G["Unset flags"]
    F -- "None" --> H["Check capacity
(backfill_toofull)
or query PG peering"]

1. Confirm the scope. If every PG in a single pool is undersized, the cause is structural: CRUSH rule, failure domain count, or pool size. If only a subset is affected, the cause is a topology event (OSD down, host failure) or a localized recovery block.

2. Check the pool’s configured size and the acting set size. For each affected PG, ceph pg <pgid> query shows acting (the current set) and up (the intended set). If acting is smaller than size, that is the undersized condition. Check the recovery_state block in the JSON output for wait_* conditions, which indicate exactly what is blocking progress (for example, wait_backfill_reserve or wait_acting_changed).

3. Compare available failure domains to the CRUSH rule. A pool with size=3 and a CRUSH rule using host as the failure domain needs at least three hosts that are up and in. ceph osd tree shows the hierarchy; count the failure-domain buckets that contain up OSDs.

4. Check for stuck recovery. If OSDs were lost but PGs are still undersized hours later with no recovery progress, check ceph osd dump | grep flags for norecover or nobackfill, check ceph osd df tree for OSDs at or above backfillfull_ratio (look at the VAR and PGS columns to identify imbalances), and check ceph health detail for TOO_FEW_OSDS.

5. For a single stubborn PG with zero recovery activity and no obvious topology cause, inspect the peering state. The Red Hat knowledge base documents ceph pg repeer <pgid> as a workaround for PGs that never re-enter recovery after a peering glitch. Treat this as a last resort after structural causes are ruled out.

Metrics and signals to monitor

The single most important correlation is between ceph_pg_undersized and ceph_pool_recovering_bytes_per_sec. Undersized PGs with active, progressing recovery heal themselves as OSDs come back or capacity is added. Undersized PGs with stalled recovery do not, and the cause of the stall is usually visible in a separate signal: capacity, flags, or TOO_FEW_OSDS.

Fixes

Fixes split into two groups: structural (CRUSH, topology, or pool configuration) and runtime (OSD loss, recovery block). Identify the right group before changing anything.

Not enough failure domains

This is the most common cause and produces permanent undersized. Ceph needs at least size distinct failure domains of the type named in the CRUSH rule. A pool with size=3 and failure-domain=host needs three hosts. A two-host cluster will have every PG in that pool undersized forever.

Options, in increasing order of risk:

• Add capacity. Add another host (or rack, if the rule is rack-aware) so CRUSH has enough failure domains. This is the only fix that preserves the intended redundancy.
• Reduce size. ceph osd pool set <pool> size 2 plus ceph osd pool set <pool> min_size 1. This is a permanent reduction in fault tolerance. Acceptable for test clusters, dangerous for production. With min_size 1, a single OSD failure in the PG means writes proceed against one copy.
• Change the CRUSH failure domain. For single-node or two-host clusters, configure the CRUSH rule to replicate across OSDs instead of hosts (type osd). This defeats host-failure protection and is appropriate only for single-node test clusters or stretched clusters with explicit external redundancy.

Single-node and two-host clusters

The default CRUSH rule places replicas across different hosts. On a single host with multiple OSDs and size > 1, every PG is permanently undersized because CRUSH cannot place two replicas on different hosts.

For single-node test clusters, create and apply a CRUSH rule that uses OSD as the failure domain:

# Create a new CRUSH rule using 'osd' as the failure domain
ceph osd crush rule create-replicated <rule-name> default osd

# Apply the new rule to the pool
ceph osd pool set <pool> crush_rule <rule-name>

Applying a new CRUSH rule triggers backfill across the pool. This is expected and generates heavy I/O. Do not perform this during peak production hours.

For two-host clusters with size=3, the only safe production fix is to add a third host. Reducing to size=2 works but halves your fault tolerance: any second host failure causes data unavailability for the affected PGs.

OSDs down or out

If the cause is a topology event, undersized should resolve as recovery completes. Verify:

• ceph osd tree shows the expected OSDs up and in.
• Recovery is progressing: ceph_pool_recovering_bytes_per_sec is non-zero.
• No flags are blocking recovery: ceph osd dump | grep flags should not list norecover or nobackfill.

If OSDs were marked out due to mon_osd_down_out_interval expiring (default 600s) and the original OSDs are coming back online, ceph osd in <osd-id> brings them back into CRUSH. Recovery then prefers PG log replay over full backfill, provided the OSD’s PG logs are still valid and have not been trimmed.

Recovery blocked by capacity

If ceph health detail shows BACKFILL_TOOFULL, or ceph_pg_backfill_toofull is greater than zero, recovery is blocked because target OSDs are above backfillfull_ratio (default 0.90). This is a structural capacity problem, not a transient one.

Options:

• Add OSDs to give recovery somewhere to go. This is the correct fix.
• Reweight overly-full OSDs down: ceph osd reweight <osd-id> 0.9. This redistributes data away from the full OSD. Use cautiously on clusters already under heavy load, as it triggers additional recovery I/O.
• Delete data (RBD snapshots, abandoned RGW multipart uploads, expired CephFS snapshots) to free raw space.
• As a last resort, temporarily raise the backfill ratio: ceph osd set-backfillfull-ratio 0.92. This is risky because it lets recovery push OSDs closer to full_ratio (0.95), where all writes stop. Lower it again once recovery completes.

See the related guide on the capacity death spiral for the full cascade pattern.

Recovery blocked by flags

If ceph osd dump | grep flags shows norecover or nobackfill, recovery is intentionally stopped. These flags are commonly set during maintenance and forgotten. Clear them:

# Re-enable recovery and backfill
ceph osd unset norecover
ceph osd unset nobackfill

Recovery should begin within seconds. Verify with ceph_pool_recovering_bytes_per_sec.

PG autoscaler interactions

When pg_autoscale_mode is on, the autoscaler may split or merge PGs. During the split or merge window, a small number of PGs may briefly show undersized alongside creating, peering, or remapped. This is expected and self-resolves as peering completes.

If you see persistent undersized from autoscaler activity, check ceph osd pool autoscale-status and the target PG count. Public operator discussions report that setting bulk true on a pool suppresses transient autoscaler-induced undersized for bulk pools, but the exact interaction is version-dependent.

Prevention

• Validate CRUSH topology before creating pools. The CRUSH rule’s failure domain and the available failure-domain count must satisfy size. This is the single most common cause of permanent undersized.
• Monitor TOO_FEW_OSDS and POOL_NO_REDUNDANCY as labeled health-check signals, not just ceph_health_status. They fire before undersized becomes chronic.
• Track recovery rate alongside PG state. Undersized PGs with zero recovery are an incident; undersized PGs with active recovery are expected during topology events.
• Keep capacity headroom below backfillfull_ratio (default 0.90). Recovery needs somewhere to go. Stay at least 20 percent below backfillfull_ratio to absorb the loss of one failure domain.
• Review cluster flags daily. noout, norecover, and nobackfill set for more than 24 hours without an active maintenance window is a defect, not a feature.
• Plan for scrub debt on undersized PGs. Ceph typically does not schedule scrubs on PGs that are not active+clean, so undersized PGs may go unverified. This is a hidden cost of leaving undersized PGs unaddressed.

How Netdata helps

• Per-second ceph_pg_undersized per pool makes the difference between a brief topology blip and a chronic structural problem obvious in seconds, not minutes.
• Correlating ceph_pg_undersized with ceph_pool_recovering_bytes_per_sec distinguishes “healing” from “stuck” without running ceph pg dump repeatedly.
• ceph_health_detail{name="TOO_FEW_OSDS"} and ceph_osd_flag_norecover / ceph_osd_flag_nobackfill as labeled metrics let you see whether the cause is structural (CRUSH), administrative (flag), or operational (OSD loss) from one pane.
• ML anomaly detection on the undersized PG count flags slow drifts, such as a small number of PGs becoming persistently undersized after a quiet OSD loss, earlier than static thresholds.
• Per-OSD capacity metrics against the configured backfillfull_ratio surface the backfill_toofull deadlock pattern before it becomes a recovery block.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph monitoring checklist: the signals every production cluster needs
• Ceph monitoring maturity model: from survival to expert
• Ceph OSD down: telling a dead disk apart from a network blip
• Ceph OSD flapping: OSDs cycling up and down and the peering storm that follows
• Ceph OSD_FULL: all writes stopped at the 95% full ratio
• Ceph OSD fullness imbalance: one OSD full while the cluster average looks fine