Ceph blocked ops: client I/O stuck behind a single slow OSD

A client writes an object. The primary OSD forwards sub-ops to its replicas, then waits for every replica to ack before it acks the client. If any one OSD in the acting set stalls, the whole op stalls. After osd_op_complaint_time (default 30 seconds) the OSD logs a slow op and the monitor raises SLOW_OPS. Past that boundary the op is effectively blocked, not merely slow. Clients time out and retry, which puts more ops in flight against the same stuck OSD.

The blast radius is what makes this painful. A single slow OSD, whether primary or replica, can block requests across many PGs and therefore many clients. Cluster-wide latency aggregates can still look acceptable while a meaningful slice of writes sits in a 30-second-plus queue. Operators who alert only on averages miss this until users complain.

What this means

Blocked ops are the client-visible face of SLOW_OPS. The OSD’s op pipeline has several stages: the op is received, queued for the PG, dispatched to the local store, replicated to peer OSDs, committed, and applied. When dump_blocked_ops reports an op, the description tells you which stage it is stuck in. That stage is the single most useful diagnostic signal. Count the blocked ops, but read their descriptions.

Three things to internalize:

1. Replication is synchronous from the client’s perspective. The primary does not ack until every replica has persisted the write. One slow replica blocks the op even when the primary’s local disk is fast.
2. The 30-second osd_op_complaint_time is a hard threshold, not a latency target. Once an op crosses it, clients are already timing out at typical application SLAs.
3. BLUESTORE_SLOW_OP_ALERT is a separate health check that fires on sustained BlueStore-level slow ops. It is not the same signal as SLOW_OPS, although they often co-fire.

flowchart TD
    Client[Client write] --> Primary[Primary OSD receives op]
    Primary --> Sub1[Sub-op to replica 1]
    Primary --> Sub2[Sub-op to replica 2]
    Sub1 --> Ack1[Replica 1 acks fast]
    Sub2 --> SlowRep[Replica 2 stalls]
    Ack1 --> Wait[Primary waits for ALL acks]
    SlowRep --> Wait
    Wait --> Blocked[Op crosses osd_op_complaint_time 30s]
    Blocked --> SlowOps[SLOW_OPS health warning]
    Blocked --> Retry[Client times out and retries]
    Retry --> Load[More ops queue on slow OSD]

The feedback loop in the diagram is the failure mode that turns a single slow OSD into a cluster-wide incident. Retries pile onto whatever resource is already saturated.

Common causes

Quick checks

These are read-only and safe to run during an incident.

# See the SLOW_OPS warning and which OSDs are involved
ceph health detail | grep -A 20 SLOW_OPS

# Check for the separate BlueStore slow op alert
ceph health detail | grep BLUESTORE_SLOW_OP

# List currently blocked ops on a suspect OSD
ceph daemon osd.<id> dump_blocked_ops

# See all in-flight ops, including those not yet past complaint time
ceph daemon osd.<id> dump_ops_in_flight

# Read the slowest recent ops with full event timelines
ceph daemon osd.<id> dump_historic_ops

# Per-OSD commit and apply latency, sorted to find outliers
ceph osd perf

# Per-OSD capacity and variance
ceph osd df tree

# OS-level disk latency on the OSD's block device
iostat -xz /dev/<dev> 1 5

# Confirm quorum is stable so map updates are not the bottleneck
ceph quorum_status

If you do not know which OSD is the culprit, ceph health detail names the OSDs with slow ops and shows the oldest blocked duration. Start there.

How to diagnose it

1. Identify the OSDs named in ceph health detail. The SLOW_OPS block lists the affected OSDs and the duration of the oldest blocked op. If only one or two OSDs appear, the problem is localized. If many OSDs appear, treat it as a cluster-wide saturation event and look at network, recovery, or capacity first.

2. Run dump_blocked_ops on each named OSD. Output includes the complaint_time, the num_blocked_ops, and per-op details. Read the description field for every op. The description tells you where the op is parked.

3. Interpret the op stage. Common patterns:

   • waiting for subops from <osd.id> means replication is the bottleneck. The named peer OSD, not the one you are inspecting, is the slow one. Pivot to that OSD.
   • waiting on pg means the PG is in a non-active state, typically peering. Check ceph pg <pgid> query and the cluster-wide peering count.
   • waiting for rw locks indicates contention on object locks, often from overlapping writes to the same object or from snapshot activity.
   • reached pg or commit sent with no further progress suggests a local BlueStore stall, usually RocksDB compaction or WAL/DB device saturation.

4. Correlate with device signals. If the stage points local, run iostat -xz on the OSD’s block device and look at await, %util, and avgqu-sz. For SSD and NVMe, trust await over %util because parallel devices can show 100% util at low latency. For HDD, %util above 85% sustained is saturation.

5. Check replication peers. If the op is waiting on subops, inspect the named peer OSD with the same dump_blocked_ops and ceph osd perf checks. The slow OSD is frequently the replica, not the primary that first surfaces in the health warning.

6. Rule out network. Look at /proc/net/snmp TCPRetransSegs on both OSD hosts and confirm MTU consistency across the cluster network. A single switch port with a lower MTU causes silent fragmentation that presents as disk slowness.

7. Check BlueStore DB health. ceph daemon osd.<id> bluefs stats reports slow_used_bytes. Any nonzero value means RocksDB metadata has spilled from the fast DB partition to the slow data partition. This is a performance cliff, not gradual degradation, and it produces periodic latency spikes during compaction.

Metrics and signals to monitor

Fixes

Failing or degraded disk

If SMART shows reallocated sectors, current pending sectors, or offline uncorrectable sectors, and disk await is climbing, the device is failing. Mark the OSD down and out so recovery moves the data.

# Destructive: marks the OSD down and out, triggers recovery and rebalancing
ceph osd down osd.<id>
ceph osd out osd.<id>

# Verify recovery is progressing
ceph pg dump_stuck degraded

If noout is set globally, clear it or set per-OSD exceptions. Recovery will not start otherwise. Plan replacement once the OSD is empty.

BlueStore DB spillover

Confirm with ceph daemon osd.<id> bluefs stats. If slow_used_bytes is nonzero, the metadata partition has overflowed. Short term, reweight the OSD lower to reduce its PG count and omap pressure:

# Disruptive: shifts I/O away from this OSD, redistributes load to peers
ceph osd reweight osd.<id> 0.9

Long term, migrate the DB to a larger fast device with ceph-bluestore-tool. This requires OSD downtime. Undersized DB partitions are a deployment-time mistake that surfaces months later as periodic latency cliffs.

Network issues

If ops are stuck in waiting for subops from <peer>, the bottleneck is between OSDs. Check physical links, switch port counters, MTU consistency, and TCP retransmits on both hosts. Jumbo frames configured on some ports but not others cause silent fragmentation. Bonded interfaces can saturate a single member link depending on the hashing algorithm.

RocksDB compaction stall

If commit latency spikes correlate with compaction events and L0 file count is elevated, the OSD is hitting compaction stalls. Short term, you can wait it out if the workload is transient. If the pattern is persistent, the DB device is too slow or too full. Manual compaction via ceph-kvstore-tool is a temporary relief, not a fix. The real answer is a faster or larger DB device.

Op queue saturation

On Quincy and later, the default scheduler is mclock_scheduler. If the OSD’s CPU is saturated and ops are parking in waiting for rw locks or at the PG queue, recovery and client I/O may be competing. You can temporarily throttle recovery to give client I/O priority:

# Cluster-wide: affects recovery on every OSD, not just the slow one
ceph tell 'osd.*' injectargs '--osd_max_backfills 1 --osd_recovery_max_active 1'

Changing osd_op_queue itself requires an OSD restart and should not be done mid-incident. If you suspect an mclock misconfiguration for HDD shards, document the workload and revisit the reservation, weight, and limit settings outside the incident window.

Short-term containment

When you cannot immediately identify the root cause and client impact is severe, mark the suspect OSD out so CRUSH redistributes its PGs:

# Destructive: triggers recovery I/O across the cluster
ceph osd out osd.<id>

This trades recovery I/O for unblocking client writes. Use noout only if you are confident the OSD will recover quickly, otherwise you extend the degraded window.

Prevention

• Monitor per-OSD latency, not averages. A single OSD at 200ms commit latency disappears in a cluster median of 10ms. Alert on the worst OSD by device class, not the mean.
• Track SMART attributes continuously. Reallocated and pending sector counts are leading indicators. A nonzero and rising count means plan replacement now, not after the OSD starts timing out.
• Size BlueStore DB partitions for object growth. The 4% of data partition rule is a floor, not a target. RGW bucket indexes and CephFS metadata pools generate heavy omap load and need more.
• Keep public and cluster networks separate. Replication, recovery, and heartbeat traffic on the same link as client I/O guarantees that recovery events degrade clients.
• Verify MTU consistency end to end. A single misconfigured port causes silent fragmentation that mimics disk failure.
• Review recovery throttle settings as the cluster grows. Defaults safe at 10 OSDs can cause cascading failures at 100. Test recovery behavior by intentionally failing an OSD during a maintenance window.

How Netdata helps

• Netdata collects ceph_healthcheck_slow_ops per second, so you see the exact moment ops start crossing osd_op_complaint_time rather than sampling minutes after the fact.
• Per-OSD ceph_osd_commit_latency_ms and ceph_osd_apply_latency_ms are collected with per-OSD labels, which lets you spot the outlier OSD against its device class peers without manual ceph osd perf polling.
• The BLUESTORE_SLOW_OP_ALERT health check is surfaced alongside SLOW_OPS, so you can distinguish general slow ops from sustained BlueStore-level stalls.
• Per-device disk metrics (await, %util, avgqu-sz) on the same host as the OSD let you correlate a slow op spike with a specific block device without switching tools.
• Correlating slow ops with TCP retransmit counters and network interface saturation on the same dashboard shortens the disk-vs-network decision.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph MON_CLOCK_SKEW: clock drift between monitors and election churn
• Ceph MON_DOWN: a monitor out of quorum and reduced redundancy
• Ceph monitor election storm: monitors that cannot hold a stable quorum
• Ceph monitor quorum lost: the cluster can no longer update its maps
• Ceph monitoring checklist: the signals every production cluster needs
• Ceph monitoring maturity model: from survival to expert