Tomcat thread pool exhaustion: currentThreadsBusy at maxThreads and requests hanging

Tomcat is up: the JVM process is running, the HTTP port is open, health checks pass. But requests hang or return 503. The dashboard shows currentThreadsBusy pinned at maxThreads (default 200). CPU is low. Heap is stable. No OutOfMemoryError in the logs.

This is thread pool exhaustion. Every worker thread is occupied and no thread is available to process new requests. The JVM is healthy; the application is just not being served.

The cause is almost always a slow or hung backend, not Tomcat itself. Threads block waiting for a database query, a downstream HTTP call, or a JDBC connection that never arrives. Each blocked thread is held far longer than normal, and the pool drains until nothing is left.

Do not restart Tomcat as a first response. Threads re-accumulate the moment traffic returns, because the underlying cause is unchanged. Take a thread dump first.

What this means

With the NIO connector (the default since Tomcat 8.5), there are two layers of buffering between a client and a worker thread:

• Worker thread pool (maxThreads, default 200): every active request occupies one thread for its entire duration. When all threads are busy, new requests wait for a free thread.
• NIO poller (maxConnections, default 8192 for NIO): a small number of poller threads multiplex thousands of TCP connections. The poller keeps accepting connections even when the thread pool is full. Idle keepalive connections consume a poller slot and a file descriptor, but not a worker thread.

This two-layer design means thread exhaustion does not immediately cause connection refusal. Clients can still open TCP connections (the poller accepts them), but their requests sit waiting for a worker thread. Only when maxConnections is also reached do connections queue in the OS accept queue (acceptCount, default 100). When that fills, the kernel sends RST and clients see “connection refused”.

From the outside, the service appears up at the TCP level but is unresponsive at the HTTP level. From inside the JVM, everything looks healthy: low CPU, stable heap, no exceptions. The threads are just waiting.

flowchart TD
    Client["Client connects"] --> Acceptor["Acceptor thread
accepts TCP"]
    Acceptor --> Poller["NIO poller
cap: maxConnections 8192"]
    Poller -->|dispatch request| Workers["Worker pool
cap: maxThreads 200"]
    Workers -->|process| Backend["Backend: DB, API, cache"]
    Backend -.->|slow or hung| Workers
    Workers -.->|currentThreadsBusy = maxThreads| Full["Pool exhausted"]
    Full -.->|requests queue for thread| Poller
    Poller -.->|maxConnections reached| Queue["OS accept queue
acceptCount 100"]
    Queue -.->|queue full| Refused["Connection refused / RST"]

The signature of this failure: low JVM CPU with all threads busy. If CPU is high, you are looking at a CPU-bound workload or GC death spiral, not thread starvation on I/O. See the Tomcat production mental model for the broader failure pattern catalogue.

Virtual threads caveat: On JDK 21+ with StandardVirtualThreadExecutor (Tomcat 10.1+/11, opt-in via server.xml), the thread pool model changes fundamentally. currentThreadsBusy reports -1 and maxThreads is not meaningful. Connection-based monitoring (connectionCount minus keepAliveCount) is the proxy for active virtual threads.

Async servlet caveat: If the application uses async servlets (AsyncContext), currentThreadsBusy undercounts actual concurrent requests. The thread is released to the pool during async processing while the request remains in-flight.

Common causes

Quick checks

These commands are read-only and safe to run during an active incident. pgrep -f 'catalina.startup.Bootstrap' matches the main class. If multiple Tomcat JVMs run on the same host, pin the PID explicitly to avoid acting on the wrong instance.

# Check thread pool state via Manager app (requires Manager app enabled with auth)
curl -s -u $USER:$PASS 'http://localhost:8080/manager/status?XML=true' | \
  grep -oP '(maxThreads|currentThreadsBusy|currentThreadCount)="[0-9]+"'

# Check via JMX (requires -Dcom.sun.management.jmxremote.port=9090 at JVM start)
java -jar jmxterm.jar -l localhost:9090 -n -v silent -e \
  "get -b Catalina:type=ThreadPool,name=\"http-nio-8080\" currentThreadsBusy currentThreadCount maxThreads"

# Take a thread dump BEFORE any restart - this is the most important diagnostic step
PID=$(pgrep -f 'catalina.startup.Bootstrap' | head -1)
jstack "$PID" > /tmp/tomcat-threads-$(date +%s).txt

# Check accept queue depth (Recv-Q = current backlog, Send-Q = max backlog = acceptCount)
ss -tnl 'sport = :8080'

# Check established connection count (compare to maxConnections=8192)
ss -tn state established '( sport = :8080 )' | wc -l

# Check JVM CPU - low CPU with all threads busy confirms I/O wait, not CPU saturation
ps -p "$PID" -o %cpu,%mem,etime

# Count http-nio worker threads in thread dump
jstack "$PID" | grep -c "http-nio-8080-exec"

# Check GC activity if CPU is high (distinguishes GC death spiral from thread starvation)
jstat -gcutil "$PID" 1000

# Check catalina.out for exceptions around the incident window
grep -cE "Exception|Error" /var/log/tomcat/catalina.out

How to diagnose it

1. Confirm the symptom. Verify currentThreadsBusy == maxThreads sustained for more than 30 seconds. A brief spike during cold start or a traffic burst is normal. Gate on uptime > 120 seconds to avoid false positives from JVM warmup.

2. Take a thread dump immediately. Run jstack <pid> before doing anything else. If you restart first, you lose the diagnostic evidence. Save the output to a file. Take two dumps 10 to 20 seconds apart to distinguish truly stuck threads (identical stack) from slow threads (stack changes between dumps).

3. Group threads by stack trace. Look for patterns. If 150 of 200 threads show the same stack frame, that is your blocked code path. Common signatures:

   • java.net.SocketInputStream.socketRead0: blocked on an HTTP call to a backend with no read timeout
   • getConnection or pool acquire: JDBC connection pool exhaustion
   • Object.wait or BLOCKED on a specific lock: application deadlock or contention
   • Database driver internals: slow query or database unavailability

4. Check whether threads are I/O-bound or CPU-bound. Low JVM CPU with all threads busy means threads are waiting on network, disk, or database I/O. High CPU means either a compute-heavy workload or GC thrashing. Run jstat -gcutil <pid> 1000 and look for Full GC frequency if CPU is high.

5. Identify the specific backend. The thread dump tells you what the threads are waiting on. Cross-reference with backend monitoring: database latency, downstream service response times, JDBC pool metrics. The backend that became slow at the same time the thread pool filled is your root cause.

6. Check the JDBC connection pool if applicable. If threads are blocked in getConnection(), check numActive vs maxActive and waitCount via JMX. A pool at capacity with waitCount > 0 confirms pool exhaustion. This can happen even when the database itself is healthy (connection leak, misconfigured maxActive).

Metrics and signals to monitor

Fixes

Backend slowdown (the most common cause)

If the thread dump shows threads blocked on a specific backend, fix that backend. Restarting Tomcat provides temporary relief, but threads re-accumulate when traffic returns. The real fixes are downstream: optimize slow queries, scale the backend, or add a circuit breaker that fails fast instead of holding threads indefinitely.

Missing timeouts on outbound connections

The default socket timeout in java.net.HttpURLConnection, Apache HttpClient, and most JDBC drivers is effectively infinite. A single hung downstream service will consume threads forever. Set explicit connect and read timeouts on every outbound call. This is the single highest-leverage prevention measure.

For JDBC: set connectTimeout and socketTimeout in the connection URL or pool configuration. For HTTP clients: set connect and read timeouts explicitly. Any finite timeout is better than infinite.

JDBC connection pool exhaustion

If the pool is at capacity with waitCount > 0:

• Increase maxActive if the database can handle more concurrent connections
• Fix connection leaks by enabling removeAbandoned=true and logAbandoned=true to identify the leaking code path
• Size maxActive relative to what the database server can handle across all Tomcat instances

Insufficient thread pool sizing

Increasing maxThreads can help if the pool is genuinely undersized for legitimate concurrent traffic. But this is the wrong first move if threads are stuck on a slow backend. More threads means more concurrent pressure on the already-slow backend, potentially making things worse. Increase maxThreads only after confirming the backend is healthy and the pool is truly undersized.

Each thread reserves stack memory (default 512KB to 1MB via -Xss). With maxThreads=200 and -Xss1m, that is roughly 200MB of off-heap virtual address space reserved. Only pages actually touched get committed to RSS, so real memory usage is typically lower. Doubling maxThreads roughly doubles the reservation.

Immediate mitigation

If the service is down and you need to buy time:

1. Block or shed traffic at the load balancer to reduce incoming load
2. Take the thread dump first if you have not already
3. Restart Tomcat as a last resort, knowing the problem will recur if the backend is still slow

Prevention

• Set timeouts on every outbound call. Connect timeout, read timeout, query timeout. No exceptions. This is the most effective prevention measure.
• Configure StuckThreadDetectionValve. Not enabled by default. Add it to server.xml with a threshold appropriate to your application (default is 600 seconds; 60 seconds is more useful for most services). It logs stuck threads, providing early warning before the pool drains.
• Add processing time to the access log. The default AccessLogValve pattern omits timing. Add %D (milliseconds) to the pattern. Without it, per-request latency analysis is impossible.
• Monitor the thread pool ratio. Alert on currentThreadsBusy / maxThreads > 0.80 sustained for 5 minutes. Page on == maxThreads with maxThreads > 50 and uptime > 120 seconds, sustained for more than 120 seconds.
• Monitor the JDBC connection pool. Track numActive / maxActive and waitCount. Pool exhaustion cascades directly into thread exhaustion.
• Gate alerts on uptime. Thread pool busy ratio spikes briefly on cold start as minSpareThreads (default 10) handles the first burst before more threads are created. Ignore alerts for the first 120 seconds after process start.

How Netdata helps

• Per-second thread pool metrics. currentThreadsBusy, currentThreadCount, and maxThreads are collected every second. Thread exhaustion can develop in under a minute, and per-second resolution catches the ramp before it becomes a full outage.
• Correlation with backend latency. When thread pool saturation and downstream service latency appear on the same timeline, the root cause is visible immediately rather than requiring a manual cross-reference between separate dashboards.
• JVM CPU alongside thread pool utilization. Low CPU with high thread utilization is the I/O-wait signature. Seeing both signals together confirms “backend problem” vs “Tomcat problem” in seconds.
• ML anomaly detection on request throughput and processing time. A slow drift in processing time, the leading indicator of future thread exhaustion, can trigger an alert before the pool fills.
• OS-level accept queue and connection count visibility. These signals are invisible to JMX. Surfacing them alongside thread pool metrics shows the full cascade from thread saturation to connection refusal.

Related guides

• How Tomcat actually works in production: a mental model for operators
• Tomcat process not running: crashes, OOM-kills, and failed restarts
• Tomcat monitoring checklist: the signals every production instance needs
• Tomcat monitoring maturity model: from survival to expert