NVMe queue depth saturation: command slots, io_timeout, and deep queues

You are looking at a host where NVMe latency has climbed, iostat shows the device busy, and nothing is erroring. No media errors, no kernel I/O error lines, SMART looks clean. The question is whether the drive is simply working as hard as it can, or whether something is wrong inside it. Queue depth is the signal that separates those two cases.

NVMe was designed for deep parallelism: each CPU core typically gets its own submission queue (SQ) and completion queue (CQ) pair, and a single queue can hold up to 64K command entries. Most enterprise drives reach peak throughput somewhere between QD 64 and QD 256 across all queues. Past that point, more outstanding commands buy nothing but latency: every extra command sits in a queue slot waiting for the controller to drain the ones ahead of it, and completion time grows linearly with how deep you stack.

When the controller cannot drain queues fast enough for long enough, the situation escalates from slow to broken: individual commands age past the kernel’s I/O timeout (default 30 seconds), the driver logs a timeout, aborts the command, and eventually resets the controller. That escalation has a specific signature in the kernel log, and the queue depth telemetry before it tells you why it happened.

What this means

An NVMe command occupies a slot in a submission queue from the moment the host writes it until the controller posts a completion entry. “Command slots” is just that capacity: queue depth per queue times the number of queues. Saturation means outstanding commands are piling up faster than the controller completes them, so the queue stays full and new commands wait.

There are two fundamentally different reasons a queue can stay full:

1. The device is at its legitimate ceiling. The workload is issuing more parallel I/O than the drive’s rated parallelism. Throughput is at or near spec, latency rises smoothly with depth. Nothing is wrong with the drive; the fix is at the workload or capacity layer.
2. The drive is draining slowly. Garbage collection, thermal throttling, SLC cache exhaustion, or a firmware problem has cut the controller’s internal completion rate. The queue backs up as a symptom. The giveaway: host-visible throughput drops while the queue stays deep.

The diagnostic value of queue depth is the relationship between depth, latency, and throughput, not any one of them alone:

flowchart TD
  A[QD rising] --> B{Latency behavior}
  B -->|Latency stable, throughput rising| C[Healthy: exploiting more parallelism]
  B -->|Latency rising, throughput at ceiling| D[Saturated by legitimate load]
  B -->|Latency rising, throughput LOW| E[Sick drive: GC, thermal, SLC cliff, firmware]
  D --> F{Sustained beyond io_timeout?}
  E --> F
  F -->|Yes, default 30s| G[Command timeout, abort, controller reset]
  F -->|No| H[Steady-state degradation, no errors logged]

Note the branch on the right: a deep queue by itself raises latency without producing a single error anywhere. If you only alert on errors, you can run for months in state D or E before the first timeout fires.

Common causes

Quick checks

All read-only, all safe to run during an incident.

# Instantaneous in-flight I/O count (field 9 of stat, ios_in_progress)
awk '{print $9}' /sys/block/nvme0n1/stat

# In-flight reads and writes separately
cat /sys/block/nvme0n1/inflight

# Per-hardware-context queue occupancy (one hctx per CPU queue set)
for f in /sys/kernel/debug/block/nvme0n1/hctx*/busy; do echo "$f: $(cat $f)"; done

# Current I/O timeout the driver is enforcing
cat /sys/module/nvme_core/parameters/io_timeout

# Look for timeout and reset escalation in the kernel log
dmesg | grep -i "nvme.*timeout\|Resetting controller"

# Average latency and I/O rate from block counters (two samples)
iostat -xp nvme0n1 1 3

# Is the controller itself busy when the host is not?
nvme smart-log /dev/nvme0 | grep -i "controller_busy_time\|temperature\|media_errors"

# Device-reported maximum queue capabilities
nvme id-ctrl /dev/nvme0 | grep -i "sqes\|cqes\|nn"

Two caveats on tools you may already be reaching for. iostat’s %util is not a saturation signal for NVMe: it measures the fraction of time the device had at least one request outstanding, and a massively parallel device can pin at 100% util while still having deep headroom. Likewise avgqu-sz often reads implausibly low on multi-queue devices. Use ios_in_progress and inflight instead.

How to diagnose it

1. Establish the QD-latency-throughput triple. Sample ios_in_progress frequently (sub-second; it is an instantaneous snapshot, so single samples mislead) alongside computed average latency from /sys/block/nvme0n1/stat deltas and throughput from iostat. The rule: QD rising with stable latency is healthy parallelism; QD rising with rising latency is saturation; QD rising with rising latency and falling throughput is a sick drive.

2. Do the load-shed test. If you can pause or throttle the workload briefly, watch latency as QD falls toward single digits. Latency that collapses back to baseline at low QD means the drive itself is fine and you were over-driving it. Latency that stays high at QD 1-4 means the problem is internal to the device (GC, thermal, firmware, media). This single test resolves most of the ambiguity.

3. Check per-queue balance. Compare hctx*/busy values across hardware contexts. One queue doing all the work while others idle points at IRQ affinity or NUMA placement problems on the host, not the drive. Every queue equally deep points at device-level limits.

4. Check the sick-drive candidates. Composite temperature climbing alongside latency (thermal throttle), drive over 80-90% full with discard disabled (GC thrash), a step-function write throughput cliff (SLC cache exhaustion), controller_busy_time high while host IOPS is low (internal contention). Any of these explains a slow drain rate.

5. Check for timeout escalation. I/O <N> QID <N> timeout lines mean commands aged past io_timeout (default 30 seconds): the queue was effectively stuck, not just deep. Count occurrences and look at what preceded them: a reset loop, thermal events, or a load spike.

6. Compare against device limits. nvme id-ctrl reports the controller’s queue entry capabilities (SQES/CQES). The kernel-side per-queue depth is set by the driver (the io_queue_depth module parameter; the default has been 1024 since the parameter was introduced in kernel 4.13, and the driver’s queue depth was 1024 before that). Saturation at the host queue layer looks different from saturation at the device layer: host-side slots filling while the device reports low utilization means the bottleneck is above the drive.

Metrics and signals to monitor

Fixes

Workload is legitimately over-driving the device

Reduce application-side concurrency to land in the device’s optimal band. For a database, that means tuning I/O thread counts and async I/O depth; for fio-based validation, cap iodepth. The target is the QD where throughput stops improving: past roughly QD 64-256 on most drives you are paying latency for zero throughput. If the workload genuinely needs more, the fix is capacity (more devices, a higher-class drive), not deeper queues.

You can also trim host-side queueing so backpressure lands on the application instead of inflating device latency. nr_requests in /sys/block/nvme0n1/queue/ controls how many requests the block layer stages per direction (on current kernels it defaults to the driver’s queue depth, i.e. 1024 for NVMe; the classic block-layer default used to be 128). Lowering it makes overload fail fast and visibly at the application layer instead of hiding as device-side latency.

Drive is draining slowly (GC, thermal, SLC cliff)

Attack the drain rate, not the queue:

• GC thrash: enable continuous discard or schedule fstrim, and get the drive below roughly 80% fill. TRIM is what tells the FTL which blocks are free; without it the drive treats itself as perpetually full.
• Thermal throttle: fix airflow or heatsinking. Throttling is self-protecting, so performance returns when temperature drops, but a drive that throttles daily is degrading.
• SLC write cliff: this is by design. If sustained write throughput matters, either reduce burst duration or use a drive whose native TLC/QLC write rate meets your floor.

Host-side queue misconfiguration

Confirm the scheduler is none (cat /sys/block/nvme0n1/queue/scheduler); any other scheduler adds latency in front of a device that manages its own queues. Fix IRQ affinity so MSI-X vectors spread across cores, and keep I/O submission on the NUMA node local to the device. A single saturated hctx with idle siblings is a configuration problem, not a hardware one.

io_timeout is firing

Raising io_timeout (module parameter on nvme_core; 30s default) stops the resets but fixes nothing: the commands are still waiting 30+ seconds for completion. Treat a raised timeout as blast-radius control for workloads that would rather stall than see a reset (some cloud block devices recommend the maximum value for exactly this reason), never as a fix. If timeouts fire under normal load, the drain rate is the problem; work the sections above. If timeouts fire in a repeating reset cycle with no thermal or PCIe correlate, suspect firmware and check the vendor’s advisories for your firmware version.

Prevention

• Baseline the QD-latency curve per drive model at deploy time, so “latency at QD 128” has a known-good reference. Deviation from your own curve is a far better alert than any absolute threshold.
• Alert on the combination, not the depth. QD alone is noisy; QD persistently high with latency rising and throughput flat or falling is the actionable condition. Reserve paging for QD pinned near device maximum with completions stalled (pre-timeout state) or actual io_timeout firings.
• Track the sick-drive precursors independently: drive fill level, discard configuration, composite temperature trend, and controller_busy_time ratio. These move before the queue backs up.
• Size for the optimal band. If steady-state load needs more than QD 64-256 of parallelism to hit its throughput target, the device class is wrong for the workload and every future growth step will buy latency instead of IOPS.

How Netdata helps

• Per-second block-layer I/O rates and derived average latency per NVMe namespace, so the QD-latency relationship is visible continuously rather than reconstructed from two iostat samples during an incident.
• Device SMART signals (composite temperature, thermal management transitions, controller busy time, media errors) on the same timeline as throughput, which is exactly the correlation that separates a saturated drive from a sick one.
• Temperature-to-throughput correlation that makes thermal throttling obvious: throughput declining in lockstep with rising composite temperature, with no errors.
• controller_busy_time collected alongside host-visible IOPS, exposing the “controller busy, host idle” internal-contention state that block stats alone cannot show.
• Kernel log monitoring context for io_timeout and controller reset events, so escalation from deep queues to timeouts is captured and alertable rather than buried in dmesg.

Related guides

• NVMe high I/O latency: reading block-layer latency and the outliers that matter
• nvme nvme0: I/O timeout, Resetting controller: what an NVMe controller reset means
• NVMe controller reset loop: repeated resets from a firmware hang
• NVMe controller state not live: reading resetting, deleting, and dead from sysfs
• blk_update_request: I/O error, dev nvme0n1: reading NVMe I/O errors in the kernel log
• NVMe critical_warning is nonzero: decoding the SMART critical warning bitmask
• How NVMe actually works in production: a mental model for operators
• NVMe available spare below threshold: critical warning bit 0 and end-of-life wear
• NVMe available spare declining: watching the wear trajectory before the threshold
• NVMe endurance runway: projecting time-to-replacement from wear signals
• NVMe error log entries growing: num_err_log_entries beyond media errors
• NVMe device disappeared: nvme0: Removing and a drive that fell off the PCIe bus