Tomcat thread dumps: reading jstack to find what the pool is waiting on

When currentThreadsBusy reaches maxThreads, Tomcat stops processing new requests even though the JVM is healthy. The thread pool gauge tells you the pool is full. It does not tell you why. The only diagnostic that shows exactly what every worker thread is doing at that moment is a JVM thread dump.

jstack <pid> (or the modern equivalent jcmd <pid> Thread.print) is the primary first response for thread-related Tomcat failures. A single dump shows whether each http-nio-exec thread is idle and parked in the task queue, blocked in a native socket read against a slow backend, waiting to acquire a database connection, or contending on a Java monitor. Many threads parked in the same stack frame is the stuck path. A cluster of threads BLOCKED on the same lock is contention, or a deadlock.

Thread count alone is not diagnostic. A pool of 200 threads can be at maxThreads because traffic is genuinely heavy, because a backend is slow, because the application is deadlocked, or because someone leaked database connections. The dump distinguishes these in seconds.

What this captures

A JVM thread dump is a snapshot of every live thread, its name, its java.lang.Thread.State, and its current stack trace. On a Tomcat NIO connector with maxThreads=200, the dump typically contains roughly 200 http-nio-<port>-exec-<n> worker threads, where <port> is the connector port (for example http-nio-8080-exec-42). It also contains the acceptor thread, poller threads, GC threads, JIT threads, and any threads your application or frameworks started.

For an exhausted-pool investigation, the dump answers three questions:

• Are the workers actually busy? Idle workers park in TaskQueue.take or TaskQueue.poll. If 180 of 200 threads sit there, the pool is not exhausted, and you should look at the acceptor, poller, OS accept queue, or GC instead.
• If busy, where are they? The top application frame (or the native frame just above it) names the resource: socketRead0 against a backend, getConnection against a JDBC pool, Object.wait on a monitor, a frame inside your own code.
• Are they contending or deadlocked? A thread in state BLOCKED waiting on a specific lock ID, with a known owning thread, is lock contention. A cycle of such threads is a deadlock.

jstack -l (long listing) additionally prints ownable synchronizers (for example ReentrantLock) per thread, which is what reveals CompletableFuture or ReentrantLock contention that ordinary monitor dumps miss.

Prerequisites

• The JDK tooling on the Tomcat host. jstack and jcmd ship with the JDK, not the JRE. On container images that ship a JRE only, neither tool is present; use kill -3 <pid> to dump to the JVM stdout instead.
• Attach permission. The dumping process must run as the same UID as the Tomcat process, or as root. This is the standard HotSpot attach permission model.
• The Tomcat PID. pgrep -f 'org.apache.catalina.startup.Bootstrap' for standalone Tomcat; for embedded (Spring Boot) use the application jar name or container PID 1.
• Confirmation of exhaustion first. A thread dump is most useful when currentThreadsBusy == maxThreads. Confirm via the Manager status XML or the Catalina:type=ThreadPool,name="http-nio-8080" MBean before dumping.

Procedure

1. Confirm the pool is actually exhausted

# Find the Tomcat PID
pgrep -f 'org.apache.catalina.startup.Bootstrap'

# Confirm pool state via Manager status XML
curl -s -u $USER:$PASS 'http://localhost:8080/manager/status?XML=true' | \
  grep -oP '(maxThreads|currentThreadsBusy|currentThreadCount)="[0-9]+"'

If currentThreadsBusy is well below maxThreads, the symptom is not pool exhaustion and the dump answers a different question. Look at the acceptor thread, the NIO poller, GC pause time, or the OS accept queue instead.

2. Capture the dump

# Preferred: jcmd, lower overhead, JDK 8+
jcmd $(pgrep -f 'catalina.startup.Bootstrap') Thread.print -l > /tmp/tomcat-threads-1.txt

# Equivalent: jstack with long listing (ownable synchronizers included)
jstack -l $(pgrep -f 'catalina.startup.Bootstrap') > /tmp/tomcat-threads-1.txt

# JRE-only hosts (no jstack/jcmd): SIGQUIT prints the dump to JVM stdout.
# On HotSpot this triggers a thread dump; it does not stop the process.
kill -3 $(pgrep -f 'catalina.startup.Bootstrap')
# Read it from catalina.out (or wherever stdout is redirected)

Take at least two dumps, 5 to 10 seconds apart, before restarting or failing over. A single snapshot cannot distinguish “permanently stuck” from “momentarily in a slow call.” Three dumps a few seconds apart is the minimum useful series; the same stack across all three is a stuck thread.

3. Locate the worker threads

# Count http-nio-exec threads in the dump
grep -c 'http-nio-.*-exec-' /tmp/tomcat-threads-1.txt

# Show thread name and state for each worker
awk '/http-nio-.*-exec-/{name=$0} /java.lang.Thread.State/{print name, "=>", $0}' \
  /tmp/tomcat-threads-1.txt

Expect roughly currentThreadCount matches. If the count is far below maxThreads, either load has not yet forced thread creation, or the connector is sharing an executor (in which case thread attributes on the Connector are ignored).

4. Classify workers by state

The state distribution tells the story before you read a single stack frame.

flowchart TD
    A[http-nio-exec worker] --> B{Thread.State}
    B -->|WAITING / TIMED_WAITING| C{Top frame?}
    B -->|RUNNABLE| D{Native or app frame?}
    B -->|BLOCKED| E[Lock contention or deadlock]
    C -->|TaskQueue.take / poll| F[Idle: pool not exhausted]
    C -->|socketRead0| G[Stuck on backend I/O]
    C -->|getConnection / pool acquire| H[DB pool exhaustion]
    C -->|Object.wait| I[Monitor or condition wait]
    D -->|socketRead0| G
    D -->|app frame| J[CPU-bound in application]

Idle workers in WAITING (parking) at TaskQueue.take are normal. A pool reported as exhausted should not be dominated by this frame; if it is, the exhaustion is upstream of the pool.

5. Group stacks to find the dominant blocking pattern

# Top frames across all workers, line numbers stripped so duplicates collapse
grep -A 30 'http-nio-.*-exec-' /tmp/tomcat-threads-1.txt | \
  grep -E '^\s+at ' | awk '{print $2}' | sed 's/([^()]*)//' | \
  sort | uniq -c | sort -rn | head -20

The frame with the highest count is the dominant path. If 140 of 200 workers share the same top application frame, that frame is where the pool is being held. Map the frame to its resource:

• java.net.SocketInputStream.socketRead0 (or equivalent NIO read): blocked on a network read with no read timeout. The backend is slow or unreachable. Check outbound latency, retransmits, and the backend’s own saturation.
• GenericObjectPool.borrowObject / HikariPool.getConnection: waiting for a database connection. Cross-check the JDBC pool (numActive == maxActive, waitCount > 0). This is the most common secondary exhaustion in Tomcat.
• ReentrantLock / synchronized / Object.wait: lock contention. Find the matching - locked <0x...> (or - <0x...> (a ...) line and identify which thread holds it. A cycle of such waits is a deadlock.
• A frame inside your own application package: a long-running or infinite loop in business logic. Common causes are unbounded loops, missing pagination, busy spins, or unintentional Thread.sleep in a request path.

6. Check for deadlock explicitly

# jstack prints deadlock info at the bottom when detected
jstack $(pgrep -f 'catalina.startup.Bootstrap') | grep -A 30 -i 'deadlock'

jstack (and Thread.print) reports Java-level monitor deadlocks, and with -l it also reports AbstractOwnableSynchronizer deadlocks, automatically at the end of the output. If the JVM prints “Found … deadlock”, the only safe recovery is a restart; finer triage will not change the outcome.

Verifying it works

A successful dump against a healthy but loaded Tomcat should show:

• A timestamp line near the top.
• A worker thread count close to what currentThreadCount reports.
• A spread of states: some RUNNABLE workers actively processing requests, some WAITING (parking) workers in TaskQueue.take waiting for the next request.
• No BLOCKED workers, or only a small number contending on a known lock.

If the dump shows almost every worker in WAITING (parking) at TaskQueue.take, the pool is idle and the perceived outage is elsewhere: the acceptor, the poller, the OS accept queue (Recv-Q on the listen socket), GC, or a connector that never started.

Common pitfalls

jstack -l is slow on large heaps. The -l flag walks the heap to enumerate ownable synchronizers. On heaps in the tens of GB or larger it can take tens of seconds and may itself stall application threads. If the JVM is already in distress, capture a plain jstack <pid> first (monitors only), then follow with -l only if you need lock detail.

jstack -F is a last resort. The force flag attaches to an unresponsive JVM and can leave the target unstable. Try jcmd Thread.print and kill -3 first; both go through the normal attach or signal path.

Idle WAITING is not stuck. A worker parked in TaskQueue.take is healthy and waiting for work. Operators frequently misread a row of these as a stuck pool. The stuck signature is a non-TaskQueue top frame, repeated across snapshots.

One dump is a snapshot, not a verdict. A thread in socketRead0 for the 50 ms you happened to capture is normal. The same frame across three dumps 10 seconds apart is stuck. Always capture a series.

Shared executors change the thread name. If a named <Executor> is referenced by a Connector, the worker threads are named for the executor, not the connector, and Connector thread attributes are ignored. Confirm the actual prefix in the dump before grepping.

Virtual threads change the model. On JDK 21+, virtual threads are enabled with the useVirtualThreads="true" connector attribute (available since Tomcat 9.0.76, 10.1.10, and 11.0.0-M7) or an explicit <Executor class="org.apache.catalina.core.StandardVirtualThreadExecutor">; both use org.apache.tomcat.util.threads.VirtualThreadExecutor. currentThreadsBusy reports -1, and maxThreads is no longer meaningful. The worker name prefix is tomcat-virt- when using StandardVirtualThreadExecutor, or <connector-name>-virt- (e.g. http-nio-8080-virt-) with the internal executor. Carrier threads appear in the dump alongside virtual threads; the relevant stacks are on the virtual threads themselves. Use connectionCount - keepAliveCount as the proxy for active work in this configuration.

Signals to monitor

How Netdata helps

Netdata surfaces the per-second signals that tell you when to capture a dump and which frame to expect when you do.

• Per-second currentThreadsBusy, currentThreadCount, and maxThreads from the Catalina:type=ThreadPool MBeans, so the climb to saturation is visible the second it begins.
• JVM CPU, GC pause time, and post-GC heap baseline alongside thread pool gauges, so the CPU-low-and-threads-high pattern reads at a glance.
• JDBC pool metrics (numActive, numIdle, waitCount) when the pool is exposed via JMX, so the getConnection stack you find in the dump has a matching metric timeline.
• Anomaly detection on thread pool utilization that flags excursions even when the static threshold has not tripped.
• OS-side listen-socket queue depth and file descriptor count to confirm or rule out the layers below the pool.

Related guides

• Tomcat connection refused: maxConnections and acceptCount both exhausted
• Tomcat accepts connections but never responds: the TCP-connect trap
• Tomcat accept queue overflow: acceptCount, somaxconn, and Recv-Q
• Tomcat heap dump before restart: capturing evidence with jmap and jstack
• Tomcat GC death spiral: full GCs dominating and throughput collapsing
• Tomcat frequent Full GC: pause time, G1, and the 5% overhead rule
• Tomcat average latency lies: why you need p95/p99 from the access log
• Tomcat 5xx error rate: separating server failures from crawler 404s
• Tomcat classloader leak on redeploy: why the old WebappClassLoader never dies
• Tomcat access log setup: adding %D and %T for per-request latency
• Tomcat java.net.BindException: Address already in use: the connector never starts