Ceph slow requests (SLOW_OPS): operations blocked past osd_op_complaint_time

ceph_healthcheck_slow_ops greater than zero means operations on the cluster have crossed osd_op_complaint_time (default 30s) and are stuck, not merely slow. The SLOW_OPS health check surfaces them as a warning, and the metric itself is a live gauge pulled from ceph health detail.

Slow ops are a symptom of something downstream blocking I/O: a failing disk, a saturated BlueStore RocksDB DB, network timeouts between OSDs, or heavy deep-scrub on HDD during a maintenance window. The right first move is to read where in the pipeline each slow op is stuck before changing any config.

Cluster-wide slow ops across many OSDs are far more serious than isolated slow ops on a single OSD. A handful that clear within a minute or two while deep-scrub or recovery is running is often transient noise on HDD. Sustained slow ops across many OSDs for more than 120 seconds is a structural problem worth paging on.

What this means

Each OSD reports operations that have been in flight past osd_op_complaint_time. The monitor aggregates them and surfaces the count as SLOW_OPS in ceph health detail. The count is the current set of ops that have crossed the threshold. When they complete or are cancelled, the count drops. The metric ceph_healthcheck_slow_ops mirrors that count, so a value of zero means no operation is currently past 30s in flight.

The severity distinction matters. The playbook flags slow ops as TICKET (ceph_healthcheck_slow_ops > 0 sustained > 120s) rather than PAGE, because deep scrub and recovery on HDD clusters can produce transient slow ops that resolve on their own. PAGE-worthy escalation comes when slow ops cascade into PG down or incomplete states, which have their own dedicated alerts.

There is a related but distinct health check, BLUESTORE_SLOW_OP_ALERT, that tracks BlueStore-internal slow operations. If you see it alongside SLOW_OPS, the cause is inside BlueStore rather than at the OSD op queue.

flowchart TD
  A[SLOW_OPS active] --> B{How many OSDs?}
  B -- One or two --> C[Check device:
iostat, SMART]
  B -- Many across failure domains --> D{Recovery or scrub running?}
  D -- Yes --> E[Throttle:
osd_max_backfills,
noscrub and nodeep-scrub]
  D -- No --> F{Slow op description?}
  F -- waiting for subops --> G[Network:
cluster NIC, TCP retransmits]
  F -- reached pg, stalled --> H[BlueStore:
bluefs slow_used_bytes,
RocksDB compaction]
  C --> I[Replace disk or reweight OSD]
  E --> J[Recovery resumes,
slow ops clear]
  G --> K[Fix link or split networks]
  H --> L[Migrate DB or compact RocksDB]

Common causes

Quick checks

# Triage: is SLOW_OPS the active check, and how many ops are stuck?
ceph health detail | grep -A 5 SLOW_OPS

# Live count from the monitor, machine-readable
ceph health detail -f json | jq '.checks.SLOW_OPS'

# Event timeline for every in-flight operation on a named OSD
ceph daemon osd.<id> dump_ops_in_flight

# Operations stuck past the blocked threshold
ceph daemon osd.<id> dump_blocked_ops

# Read the "where it's stuck" description in the OSD log
grep "slow request" /var/log/ceph/ceph-osd.<id>.log | tail -n 50

# Commit and apply latency per OSD, comparable across device classes
ceph osd perf

# Per-pool PG state, including deep-scrub and recovery activity
ceph pg stat

# Confirm whether recovery or scrub flags are silently set
ceph osd dump | grep flags

How to diagnose it

1. Confirm scope. ceph health detail | grep -A 5 SLOW_OPS shows how many slow ops are in flight and which OSDs they belong to. If only one or two OSDs are listed, focus on those hosts. If the list spans many OSDs across multiple failure domains, treat it as a cluster-wide resource problem.

2. Read where the ops are stuck. For each named OSD, ceph daemon osd.<id> dump_ops_in_flight shows the event timeline for every in-flight operation. The most recent event description tells you what each op is waiting on:

   • waiting for subops from <osd>: replication stall. The named replica is the bottleneck, not the primary.
   • waiting on pg: the PG is in a non-active state, usually peering or recovery.
   • waiting for rw locks: contention on a busy object, often many writers to the same PG.
   • reached pg followed by a long stall: the local device or BlueStore pipeline is the bottleneck.

3. Correlate with per-OSD latency. ceph osd perf prints commit and apply latency per OSD. Compare each slow OSD to its peers on the same device class. An OSD whose latency is many times its peers’ is your failing component.

4. Check what else the OSDs are doing. Slow ops that arrive during recovery or deep-scrub are usually resource contention, not failure. ceph pg stat shows how many PGs are recovering, backfilling, scrubbing, or deep-scrubbing. ceph -s shows the recovery I/O rate. If those line up with the slow ops, throttle or schedule.

5. Look at BlueStore internals. If a single OSD has periodic latency spikes with no obvious device fault, inspect ceph daemon osd.<id> perf dump under the bluefs and bluestore sections, and ceph daemon osd.<id> bluefs stats. A nonzero slow_used_bytes in bluefs means RocksDB has spilled from the fast DB partition to the slow data device. This is a cliff-edge failure with a 10-100x latency increase that is invisible in capacity metrics.

6. Check the network. If slow op descriptions repeatedly say waiting for subops from <osd>, the named replica is the bottleneck. Use ip -s link show <cluster_iface> and cat /proc/net/snmp (the TCPRetransSegs counter) to see whether the cluster network is saturated or losing packets.

Metrics and signals to monitor

Fixes

Single failing disk

If slow ops are isolated to one OSD and its latency is 5x or more of its peers, the device is likely failing or saturated.

• Confirm with iostat -xz /dev/<dev>: high %util, growing await, growing avgqu-sz.
• Check SMART: smartctl -A /dev/<dev>. Nonzero Reallocated_Sector_Ct, Current_Pending_Sector, or Offline_Uncorrectable indicates active media errors.
• If the disk is failing, drain and replace. Mark the OSD out and down cleanly, let recovery complete, then replace the media and redeploy.

Do not just restart the OSD. Restarting does not fix a failing disk and may disrupt clients further.

Recovery or scrub contention

If slow ops arrive while recovery or deep-scrub is running on HDD OSDs, throttle before considering anything else.

• Reduce recovery pressure: ceph tell 'osd.*' injectargs '--osd_max_backfills 1 --osd_recovery_max_active 1'. This is a runtime change that takes effect immediately but does not persist across OSD restart. Use ceph config set osd <key> <value> for persistence.
• Defer scrub during the incident: ceph osd set noscrub and ceph osd set nodeep-scrub. Unset them when the incident is over; leaving them set silently disables data integrity verification.
• Confirm whether noout, norecover, or nobackfill are set and forgotten: ceph osd dump | grep flags.

RocksDB compaction stall or BlueStore kv_sync block

Periodic latency spikes on a specific OSD with no disk errors often point to RocksDB compaction. The bluefs and rocksdb sections of ceph daemon osd.<id> perf dump show compaction activity.

Manual compaction is a short-term lever:

# Compact the BlueStore RocksDB. The OSD must be stopped first;
# running this on a live OSD corrupts the DB.
systemctl stop ceph-osd@<id>
ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-<id> compact
systemctl start ceph-osd@<id>

This is a temporary mitigation, not a root-cause resolution. The underlying cause is often a mismatch between DB load and slow hardware, or DB spillover.

BlueStore DB spillover to slow device

If ceph daemon osd.<id> bluefs stats shows nonzero slow device usage, RocksDB has spilled from the fast DB partition to the slow data partition. Latency jumps 10-100x on every metadata operation.

• Confirm with ceph daemon osd.<id> bluefs stats and check the slow device usage line.
• Migrate the DB to a larger device using ceph-bluestore-tool. This requires OSD downtime and careful planning.
• Short term, reweight the affected OSD down to reduce its object count: ceph osd reweight <id> 0.9.

Inter-OSD network timeout

If slow op descriptions consistently point to waiting for subops from <osd> across a failure domain:

• Check whether the cluster network is shared with the public network. If so, recovery traffic is competing with client I/O.
• Verify MTU consistency. Jumbo frames configured on some ports but not others cause silent fragmentation that looks like disk slowness.
• Look for rising TCP retransmits in /proc/net/snmp (TCPRetransSegs).

mClock shard misconfiguration on HDD

In Squid 19.2.1 the mClock HDD shard defaults changed: osd_op_num_shards_hdd moved from 5 to 1 and osd_op_num_threads_per_shard_hdd moved from 1 to 5. Clusters on older point releases with the old defaults can exhibit slow requests on HDD OSDs under mClock scheduling. If you are on Squid below 19.2.1, manually apply the new defaults via ceph config set after validating the change in a non-production tier.

Tuning osd_op_complaint_time

The default 30 seconds is appropriate for most deployments. Some operators lower it to 5-10 seconds for latency-sensitive workloads to get earlier visibility, but lowering the threshold increases noise from legitimate transient stalls on HDD clusters. Treat this as observability tuning, not a fix.

Prevention

• Monitor per-OSD latency, not cluster averages. A single OSD with 5x the latency of its peers shows up only in per-OSD signals. Cluster-average latency looks fine while one OSD silently degrades.
• Track bluefs slow_used_bytes. Spillover is a cliff-edge failure with no warning in capacity metrics.
• Track RocksDB compaction stats. Compaction stalls are a common root cause of latency spikes but are almost never proactively monitored.
• Separate public and cluster networks. Recovery traffic can saturate a shared NIC; the fix is physical separation, not throttling alone.
• Schedule deep-scrub windows explicitly. osd_scrub_begin_hour and osd_scrub_end_hour exist for a reason. Document them and alarm on noscrub and nodeep-scrub set for more than 24 hours.
• Validate DB partition sizing at deployment. Aim for 4-5% of the data partition for replicated pools, larger for EC or heavy omap usage. Revisit when workload changes.
• Watch for forgotten noout. The “noout trap” is the single most common preventable Ceph outage.

How Netdata helps

• ceph_healthcheck_slow_ops is collected per second, so you can pinpoint the exact second the count crosses zero and correlate it with what else changed.
• Per-OSD ceph_osd_apply_latency_ms and ceph_osd_commit_latency_ms let you isolate the failing OSD against its device-class peers instead of staring at a cluster average.
• PG state metrics (ceph_pg_recovering, ceph_pg_backfilling) and recovery rate metrics let you confirm or rule out recovery contention before changing config.
• Co-located host-level disk metrics (%util, await, avgqu-sz) and network metrics (TCP retransmits, interface saturation) appear next to the Ceph metrics, so you can attribute slow ops to a device, a network, or a scheduler without pivoting tools.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph monitor quorum lost: the cluster can no longer update its maps
• Ceph monitoring checklist: the signals every production cluster needs
• Ceph monitoring maturity model: from survival to expert
• Ceph OSD down: telling a dead disk apart from a network blip
• Ceph OSD flapping: OSDs cycling up and down and the peering storm that follows
• Ceph OSD_FULL: all writes stopped at the 95% full ratio