SQL Server worker thread exhaustion: when the instance stops accepting work

The application reports that the database is down. CPU is low, disk I/O is low, memory looks fine, the sqlservr process is running. A TCP connection to port 1433 even succeeds. But queries hang, logins time out, and nothing completes. The instance is alive and refusing to work.

This is worker thread exhaustion. Every active request in SQL Server needs a worker thread from a bounded pool. When every worker is occupied, usually suspended waiting on something, new requests cannot be scheduled at all. They queue on the THREADPOOL wait, which from the client’s perspective is indistinguishable from the server being down.

The trap is that all your usual saturation signals look healthy. CPU is low because the workers are suspended, not running. I/O is low because nothing is executing. The system is not overloaded. It is paralyzed.

What this means

SQL Server schedules its own work through SQLOS, one scheduler per logical CPU, with a bounded pool of worker threads. The default maximum on 64-bit systems with 4 to 64 logical CPUs is 512 + ((logical_CPUs - 4) * 16): an 8-core instance gets 576 workers, a 64-core instance gets 1472. Your exact ceiling is in sys.dm_os_sys_info.max_workers_count.

Each active request consumes at least one worker. Parallel queries consume one worker per thread per degree of parallelism, so a single MAXDOP 8 query can hold up to 8 workers. Workers waiting on locks, latches, I/O, or external calls stay allocated until the wait ends.

When the pool drains, new work queues. The failure progression is consistent:

flowchart TD
  A[Head blocker holds a lock
or parallel queries fan out
or connection flood arrives] --> B[Workers suspend on LCK_M_*
or get consumed by DOP]
  B --> C[Active workers climb
toward max_workers_count]
  C --> D[Batch requests still arrive
but transactions/sec falls]
  D --> E[CPU drops - workers are
suspended, not running]
  E --> F[THREADPOOL waits
work_queue_count > 0]
  F --> G[Instance stops accepting work
probes and logins fail]

Two details matter for diagnosis. First, THREADPOOL waits may not appear in sys.dm_os_wait_stats during full exhaustion, because recording a wait requires a thread. Monitor the active worker count directly. Second, the Dedicated Admin Connection (DAC) reserves its own scheduler and bypasses the normal pool. It is your way in when everything else hangs.

Common causes

The blocking cascade is the most common and the most dangerous, because a sleeping head blocker with an uncommitted transaction will never resolve on its own.

Quick checks

All of these are read-only. If normal connections hang, run them over the DAC (see below).

-- 1. How big is the pool?
SELECT max_workers_count FROM sys.dm_os_sys_info;

-- 2. How many requests are active right now?
SELECT COUNT(*) AS active_requests
FROM sys.dm_exec_requests
WHERE status IN ('running', 'runnable', 'suspended');

-- 3. Per-scheduler pressure: work_queue_count > 0 means requests are
--    queued with no worker available. This is the direct exhaustion signal.
SELECT scheduler_id, current_tasks_count, runnable_tasks_count,
       active_workers_count, work_queue_count
FROM sys.dm_os_schedulers
WHERE status = 'VISIBLE ONLINE';

-- 4. THREADPOOL waits (may be absent during full exhaustion).
SELECT wait_time_ms, waiting_tasks_count
FROM sys.dm_os_wait_stats
WHERE wait_type = 'THREADPOOL';

-- 5. Blocking chains, longest waits first.
SELECT r.session_id AS blocked_session,
       r.blocking_session_id AS blocker,
       r.wait_type,
       r.wait_time / 1000 AS wait_seconds,
       DB_NAME(r.database_id) AS database_name
FROM sys.dm_exec_requests r
WHERE r.blocking_session_id <> 0
ORDER BY r.wait_time DESC;

-- 6. Find the head blocker: a session that blocks others but is not
--    itself blocked.
SELECT DISTINCT r.blocking_session_id AS head_blocker
FROM sys.dm_exec_requests r
WHERE r.blocking_session_id <> 0
  AND r.blocking_session_id NOT IN (
      SELECT session_id FROM sys.dm_exec_requests
      WHERE blocking_session_id <> 0);

Interpretation guide:

• active_requests near max_workers_count plus work_queue_count > 0: exhaustion is happening now.
• Many suspended sessions on LCK_M_* and a sleeping head blocker: blocking cascade. Go to recovery.
• High active worker count, no blocking chain, CXPACKET/CXCONSUMER climbing: parallelism is consuming the pool.
• User Connections far above baseline with flat batch throughput: the app tier is leaking or retry-storming connections.

How to diagnose it

1. Confirm the symptom shape. Low CPU, low I/O, instance unresponsive, probe or login timeouts. If CPU is pegged instead, you likely have scheduler starvation, not thread exhaustion. See SQL Server runnable tasks backlog.

2. Connect over the DAC. Normal connections queue behind the exhausted pool. Use sqlcmd -A -S <server> (or the admin: prefix in SSMS). The DAC runs on a reserved scheduler and will get in even when the instance refuses everything else.

3. Measure pool pressure. Run checks 1-3 above. active_workers_count summed across online schedulers approaching max_workers_count, with any work_queue_count > 0, confirms active exhaustion.

4. Classify the cause. Run checks 5-6. If there is a blocking chain, check the head blocker’s state: a session that is sleeping (no active request) with an open transaction is the classic application bug or abandoned SSMS window. If there is no blocking chain, look at what active requests are doing: parallel plans burning workers, or external calls stuck in preemptive waits.

5. Check the throughput split. Batch requests arriving but transactions/sec falling is the signature that work is accepted but nothing completes. This separates worker exhaustion from an upstream traffic drop.

6. Decide: kill or wait. A sleeping head blocker will not self-resolve. An active head blocker that is genuinely running (a large update, a restore) may finish. Killing a session triggers rollback, which can take as long as the original transaction took to execute. Factor that into the decision.

Metrics and signals to monitor

Alert at worker utilization above 60-80% of max as an early warning. Page on sustained THREADPOOL waits or work_queue_count > 0 for 60 seconds or more combined with connection or query probe failures. That composite means actual refusal of work, not a transient spike.

Fixes

Blocking cascade: kill the head blocker

Connect over the DAC, confirm the head blocker is sleeping with an open transaction, assess what it was doing, then kill it:

-- Over the DAC, after assessing the session:
KILL <session_id>;

Killing rolls back the transaction. Rollback duration roughly tracks how much work the transaction did, and on a busy system this is disruptive: warn the on-call channel before you do it. Do not restart the instance as a first response: restart forces recovery on every database, destroys all diagnostic state in the DMVs, and converts a minutes-long incident into a much longer one.

If the head blocker is active and doing legitimate work (a large batch update that escalated to a table lock, a schema deployment holding SCH_M locks), the fix is to wait or to abort that operation deliberately, then fix the process that launched it.

Parallel query explosion

If workers are being consumed by parallel plans rather than blocking, the lever is MAXDOP and cost threshold for parallelism, not the worker pool. Reducing MAXDOP cuts the per-query worker cost directly. See SQL Server CXPACKET and CXCONSUMER waits for the diagnostic detail. As an emergency measure, throttling the offending workload (Resource Governor on Enterprise Edition, or killing the burst queries) restores pool headroom.

Connection leak or retry storm

SQL Server cannot fix a misbehaving app tier from inside. Identify the source (the session list grouped by host and program name shows it clearly), then restart or throttle the offending application instances. Leaked sessions are usually idle, so they hold connections rather than workers; they become a thread problem only when they all wake up at once, which is exactly what a retry storm does.

What not to do

Do not raise max worker threads to make the symptom go away. The pool size is not the problem; something is consuming the workers. A larger pool delays exhaustion slightly, consumes more memory per thread, and makes the eventual failure bigger. Find the consumer first.

Prevention

• Alert on pool utilization, not just exhaustion. Trend active workers against max_workers_count. A workload drifting from 30% to 60% of the pool over months is telling you something. The playbook headroom target is below 70% under peak load.
• Catch head blockers early. A 30-second check for blocking chains deeper than a few sessions, or blocks older than 30-60 seconds, catches cascades while they are still cheap to fix. Sleeping sessions with open transactions deserve their own alert.
• Right-size parallelism. An unbounded MAXDOP on a busy OLTP instance is a worker pool denial-of-service waiting for a traffic spike. Set MAXDOP and cost threshold for parallelism deliberately.
• Fix application transaction hygiene. Most blocking cascades trace back to connections returned to the pool with open transactions. That is an app bug; push it back to the owning team with the session evidence.
• Test the DAC before you need it. Verify that DAC access works, is permitted through your network path, and that the on-call runbook says to use it. The middle of a threadpool outage is the wrong time to learn it is firewalled.
• Persist DMV snapshots externally. Wait stats, scheduler state, and worker counts reset on restart. If you only have in-instance history, a post-incident restart erases your forensics.

How Netdata helps

Worker thread exhaustion is a composite signal problem: no single metric says “the pool is draining,” but the combination is unambiguous. Netdata surfaces the pieces on one timeline:

• Worker utilization against max_workers_count, so you see the pool filling as a trend, not just at the cliff edge.
• THREADPOOL waits and scheduler queue depth, the direct confirmation that work is queued with no thread to run it.
• The throughput split: batch requests/sec holding steady while transactions/sec collapses, with CPU dropping at the same time. That three-line correlation is the exhaustion signature, and it separates a paralyzed instance from an overloaded one.
• Wait statistics broken down by type, so LCK_M_* rising ahead of THREADPOOL points you at the blocking cascade before the pool empties.
• Connection counts and per-database transaction rates to distinguish a connection flood from a parallelism burst as the consumer.

Microsoft SQL Server monitoring with Netdata brings these signals together with per-second metrics and anomaly detection.

Related guides

• SQL Server CPU utilization high: telling query load apart from a bad plan
• SQL Server CXPACKET and CXCONSUMER waits: parallelism, MAXDOP, and what is actually wrong
• SQL Server high compilations per second: plan cache pollution and CPU burn
• SQL Server runnable tasks backlog: the in-engine CPU queue OS metrics miss