SQL Server runnable tasks backlog: the in-engine CPU queue OS metrics miss

Your monitoring says the host is at 45% CPU. Users say the database is slow. Both are right, because SQL Server does not use the OS scheduler for query execution. It runs its own cooperative scheduling layer, SQLOS, with one scheduler per logical CPU, its own run queues, and its own worker thread pool. OS CPU percent measures what the host kernel sees. It does not see tasks sitting in SQLOS queues waiting for a scheduler slot, and on virtualized hosts it does not see time the hypervisor stole from the guest.

The signal that closes this gap is runnable_tasks_count in sys.dm_os_schedulers: tasks that are ready to execute right now but queued waiting for CPU time on a scheduler. It is the cleanest CPU-queue signal the engine exposes, and the one most OS-level dashboards miss entirely.

What the runnable queue actually is

Every logical CPU visible to SQL Server gets one SQLOS scheduler. Each scheduler owns:

• A runnable queue: tasks that have everything they need (locks granted, I/O complete, memory granted) and are waiting only for their turn on the CPU.
• A waiter list: tasks blocked on a resource (lock, latch, I/O, network). Not runnable.
• A work queue: tasks accepted but with no worker thread assigned yet. This is the worker-exhaustion queue, a different and more severe problem.
• An I/O completion list, which matters less here.

Workers run cooperatively. A worker runs until it voluntarily yields at a known yield point or exhausts its quantum, then moves to the back of the runnable queue. When a worker calls out to external code (CLR, extended stored procedures, some OS calls), it switches to preemptive mode and is tracked separately.

The monitoring consequence: a task in the runnable queue is losing latency to CPU contention, but it is not “using CPU” in any way the OS can attribute, and it is not waiting on a resource either. It only shows up as queue depth inside SQLOS, or as signal wait time in wait stats. Nothing at the OS layer counts it.

flowchart LR
    req[New request] --> wq{Worker available?}
    wq -- yes --> run[Running on scheduler
4 ms quantum]
    wq -- no
work_queue_count gt 0 --> pool[THREADPOOL wait
worker exhaustion]
    run -- needs resource --> wait[Waiter list
resource wait]
    wait -- resource ready --> rq[Runnable queue
runnable_tasks_count]
    run -- quantum exhausted
or yield point --> rq
    rq -- scheduler picks next --> run

The two numbers you alert on come straight off this diagram: runnable_tasks_count is the runnable queue depth, work_queue_count is the work queue depth. Everything else is context.

Reading the signal

The query is short. The filter matters: hidden schedulers (dedicated admin connection, resource monitor, and other internal schedulers) and offline schedulers pollute the results, so restrict to VISIBLE ONLINE.

-- Runnable and work queue depth per scheduler
SELECT scheduler_id, cpu_id,
       current_tasks_count,
       runnable_tasks_count,
       active_workers_count,
       work_queue_count
FROM sys.dm_os_schedulers
WHERE status = 'VISIBLE ONLINE'
ORDER BY runnable_tasks_count DESC;

Requires VIEW SERVER STATE.

Sustained is the load-bearing word. This DMV is a point-in-time snapshot. A single sample of 3 on one scheduler during a parallel query burst means nothing. The same value repeated across samples taken every 15 to 60 seconds over several minutes means queries are genuinely queuing.

Why OS CPU metrics miss this

Three mechanisms make the in-engine queue more truthful than host CPU percent.

1. Low OS CPU with a saturated engine. You can have low OS CPU and still have scheduler pressure when the workload is worker-constrained or when a subset of schedulers is hot. Thread placement across schedulers is not round-robin, and threads do not migrate between schedulers once assigned, so one scheduler can back up while its neighbors idle. Per-CPU averages at the OS layer flatten exactly the skew you care about.

2. CPU steal on VMs. On an oversubscribed hypervisor, the guest is descheduled without knowing it. Workers exhaust their quantum without receiving the equivalent real CPU time. The classic symptom is inflated SOS_SCHEDULER_YIELD waits with low reported CPU: threads yield on schedule, they just do not get scheduled back promptly. Steal primarily shows up as phantom yield waits and elongated queue residence time, so corroborate with the runnable queue and the signal wait ratio rather than any single counter.

3. Time accounting vs queue depth. CPU percent tells you how busy the processors were, not whether work lined up behind them. A scheduler at 100% with an empty runnable queue is busy but healthy. A scheduler at 70% with a runnable queue of 6 is a latency problem no utilization graph will surface.

This is also why CPU percent must never page by itself: high CPU with an empty runnable queue is usually just SQL Server using the cores you gave it.

Signals to correlate before you conclude “CPU pressure”

The runnable queue is necessary but not sufficient. Confirm with these before tuning queries or adding cores:

The pattern that closes the case: sustained runnable_tasks_count above 1 on multiple schedulers, signal wait ratio creeping above 20%, and batch throughput degrading at the same time. That is scheduler starvation. The fix is query/plan work or capacity, not more monitoring.

The pattern that points elsewhere: runnable queue near zero, signal waits low, but work_queue_count above 0 or THREADPOOL waits appearing. That is worker exhaustion, usually from a blocking cascade holding workers in a suspended state, and adding CPU does nothing. Find the head blocker.

On NUMA systems, run the same analysis per node. One node’s schedulers can run hot while the instance-wide average looks calm. Check per-scheduler rows, not a single aggregate.

Sampling it correctly

Because sys.dm_os_schedulers is point-in-time, collection discipline determines whether the signal is useful or noise.

• Sample on an interval, persist the samples. Every 15 to 60 seconds is typical. A single ad-hoc query during an incident tells you about that instant only.
• Filter to VISIBLE ONLINE every time. Hidden and offline schedulers skew aggregates.
• Keep per-scheduler granularity. Do not average at collection time; you can aggregate later, but you cannot recover a hot scheduler from an average.
• Track max and per-NUMA distribution, not just the mean. A max of 8 with a mean of 0.4 is a real story.
• Capture wait-stat deltas alongside. sys.dm_os_wait_stats is cumulative since startup, so snapshot it and compute deltas over the same interval to get a comparable signal-wait ratio.
• Do not rely on DMV state surviving a restart. It resets with the instance. For a post-restart incident you need samples persisted outside SQL Server.

One adjacent use of the same DMV: yield_count increments every time a worker yields on that scheduler. If yield_count does not change across a sampling interval on a scheduler that has work, a thread is holding that scheduler without yielding. SQL Server logs a non-yielding scheduler error after roughly 60 seconds, but watching yield_count go flat gives you the early warning before the dump.

Common misreads

Alerting on any nonzero value. Parallel query bursts, compilation spikes, and checkpoint activity all produce transient runnable queue depth on healthy systems. Alert on sustained depth (multiple consecutive samples above threshold), never on one sample.

Treating SOS_SCHEDULER_YIELD as proof of CPU pressure. It is not. It means threads are hitting yield points and cycling through the queue, which happens constantly on busy systems and is amplified by VM steal. The CPU-pressure indicators are runnable queue depth and the signal wait ratio. Use the yield wait as corroboration, not the trigger.

Averaging across schedulers. Per-scheduler skew is the signal. Averages erase it.

Confusing the two queues. A runnable backlog is a CPU problem. A nonzero work queue is a worker-thread problem with completely different causes (blocking chains, external waits, parallel query worker exhaustion). The fix paths do not overlap.

Paging on OS CPU. CPU above 90% alone must never page, because backup compression, ETL, and index maintenance legitimately drive it there. The composite that justifies a page is high CPU plus sustained runnable backlog plus user-visible timeout or throughput impact.

How Netdata helps

• Netdata collects per-scheduler runnable_tasks_count and work_queue_count from sys.dm_os_schedulers at per-second granularity, removing the point-in-time sampling problem and making transient bursts visible next to sustained backlog.
• Wait statistics are trended with deltas computed for you, so the signal wait ratio can be graphed directly against runnable queue depth on the same timeline.
• Correlating the runnable backlog with batch requests/sec, compilations/sec, and CPU utilization in one view separates “busy engine” from “starved engine” without manual DMV snapshots.
• Metrics persist outside the instance, so history survives a restart, which is exactly when DMV-only monitoring goes dark.
• Anomaly detection on the runnable queue flags the unusual-for-this-hour backlog that fixed thresholds miss on workloads with strong daily cycles.

Netdata’s Microsoft SQL Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• Microsoft SQL Server Operations Guides for the full signal catalog, failure patterns, and monitoring maturity model this signal belongs to.