Tomcat threads busy but CPU idle: telling a blocked backend from a GC spiral

When users report that Tomcat is “hung” but the JVM is up, ports are open, and the operating system looks fine, the fastest split you can make is to look at one number alongside the thread pool state: JVM CPU. Two different failure modes produce currentThreadsBusy == maxThreads, and they have opposite CPU signatures.

If CPU is low while every worker thread is busy, those threads are parked on I/O: a slow database, a hung downstream HTTP service, an unresolvable DNS lookup, or a stalled NFS mount. The JVM is healthy; a backend is not. Restarting Tomcat is the wrong move, because the threads re-block the moment traffic returns.

If CPU is high while every worker thread is busy, the threads are not parked on the network. They are either burning real cycles in application code or, more commonly, the GC threads have taken over. The JVM is the problem, and the diagnostic path goes through heap state and GC activity, not backend latency.

What this means

The thread pool reports currentThreadsBusy == maxThreads when every worker thread (default cap 200) is occupied. With the NIO connector, Tomcat still accepts new TCP connections up to maxConnections (default 8192 for NIO), so clients do not see connection refused yet. Once the pool is saturated, no new request can be picked up until a thread is released, and throughput drops to whatever rate threads free up at.

The split happens one level down:

• Low CPU (typically under 30% of available capacity): threads are in native I/O waits. They consume almost no CPU. The bottleneck is downstream.
• High CPU (typically over 70% sustained): threads are computing, either in application code or in GC. The bottleneck is inside the JVM.

The single fastest triage move is top -H -p <pid> next to a thread dump. The combination tells you, in under a minute, which world you are in.

flowchart TD
    A["currentThreadsBusy ~= maxThreads"] --> B{"JVM CPU level"}
    B -->|"Low - under 30%"| C["Threads blocked on I/O"]
    B -->|"High - over 70%"| D{"GC threads on top?"}
    C --> C1["jstack: socketRead0, borrowObject, recvfrom"]
    C1 --> C2["Backend is the root cause"]
    D -->|"Yes"| E["GC death spiral"]
    D -->|"No"| F["CPU-bound app code"]
    E --> E1["Heap dump before restart"]
    F --> F1["CPU profile, recent change"]

Common causes

The first five causes present as “low CPU + threads busy.” The last two present as “high CPU + threads busy.” Mixing them up wastes time and triggers the wrong response.

Quick checks

Run these in order. They are all read-only.

# 1. Confirm thread pool state (assumes Manager app is enabled)
curl -s -u $USER:$PASS 'http://localhost:8080/manager/status?XML=true' | \
  grep -oP '(currentThreadsBusy|currentThreadCount|maxThreads)="[0-9]+"'

# 2. JVM CPU and memory snapshot
TOMCAT_PID=$(pgrep -f 'catalina.startup.Bootstrap')
ps -p $TOMCAT_PID -o %cpu,%mem,etime

# 3. Per-thread CPU breakdown - which threads are actually hot?
top -H -p $TOMCAT_PID -bn1 | head -30

# 4. GC state (FGC = full GC count, FGCT = full GC time in seconds)
jstat -gcutil $TOMCAT_PID 1000 5

# 5. Thread dump - the single most important artifact
jstack $TOMCAT_PID > /tmp/tomcat-threads-$(date +%s).txt

# 6. Established connections to the connector
ss -tn state established '( sport = :8080 )' | wc -l

Steps 3 and 5 disambiguate the two worlds. If top -H shows the JVM’s hot threads as GC workers (typically named with GC in the thread name), you are in the GC spiral. If top -H is mostly idle and the thread dump shows http-nio-8080-exec-* threads parked in socketRead0 or borrowObject, you are in the backend-blocked world.

The jstack RUNNABLE trap

The most common misread in this diagnosis: a thread parked in java.net.SocketInputStream.socketRead0(Native Method) shows up in jstack as java.lang.Thread.State: RUNNABLE. The JVM considers a thread waiting on native I/O to be runnable, because from the JVM’s perspective the thread is executing in native code. Operators new to thread dumps routinely read this as “the thread is doing CPU work” and chase the wrong problem.

The correct read: a RUNNABLE thread whose top frame is socketRead0 (or epollWait, recvfrom, similar) is parked on I/O. It is consuming no CPU. If most of your http-nio-8080-exec-* threads look like this, you have a backend problem.

How to diagnose it

1. Capture the thread dump before doing anything else. If you restart first, you destroy the evidence. Save it to disk: jstack $TOMCAT_PID > /tmp/threads-$(date +%s).txt. Take two dumps a few seconds apart so you can tell static waits from active compute.

2. Group the http-nio-* worker threads by stack signature. Count how many sit in each call site. A single stack frame appearing 150+ times out of 200 workers is your smoking gun.

   grep -A 1 'http-nio-8080-exec' /tmp/threads-*.txt | grep 'at ' | \
     sort | uniq -c | sort -rn | head -20
   

3. Match the dominant stack frame to a backend. Common signatures:

   • java.net.SocketInputStream.socketRead0 called from a JDBC driver (org.postgresql.*, com.mysql.*, oracle, etc.) - database.
   • java.net.SocketInputStream.socketRead0 called from an HTTP client (org.apache.http.*, okhttp.*, java.net.HttpURLConnection) - downstream HTTP service.
   • org.apache.commons.pool2.impl.GenericObjectPool.borrowObject or similar - connection pool exhausted (often database, sometimes HTTP).
   • java.net.InetAddress.getAddressFromNameService - DNS.
   • Native frames below a filesystem call - NFS or disk stall.

4. If CPU is high instead of low, check whether GC is the consumer. Use jstat -gcutil $TOMCAT_PID 1000 and watch the FGC (full GC count) and FGCT (full GC time) columns over five seconds. If FGC is incrementing or FGCT is climbing, GC is dominating execution. Cross-check with top -H -p $TOMCAT_PID and confirm the hot threads are GC workers.

5. If CPU is high but it is not GC, profile the application. A handful of http-nio-8080-exec-* threads pinned at the top of top -H with no I/O in their stack means CPU-bound application code. Common culprits: regex evaluation (ReDoS), serialization of large object graphs, tight loops, or JIT deoptimization. Pull a CPU profile with async-profiler or Java Flight Recorder.

6. Take a heap dump before any restart if GC is involved. Once you restart, the leak evidence is gone. Use jmap -dump:live,format=b,file=/tmp/heap.hprof $TOMCAT_PID. WARNING: -dump:live triggers a Full GC and pauses the JVM; do not run this on an already-bleeding node without a plan to fail traffic over. Analyze offline with Eclipse MAT or VisualVM to find the dominant retainer.

Metrics and signals to monitor

Fixes

Threads blocked on a backend (low CPU)

• Fix the backend. This is the root cause. Chasing Tomcat tuning while the database is at 100% CPU is wasted effort.
• Verify every outbound call has a timeout. The default socket timeout for many JDBC drivers and HTTP clients is infinite. A hung downstream service will consume threads forever. Set explicit connect and read timeouts on every outbound client.
• Shed load at the load balancer as a temporary measure if the backend is unreachable. This prevents thread accumulation while you fix the backend.
• Do not restart Tomcat as a first move. If the backend is still slow, the new JVM will fill its thread pool again within seconds. Restart buys time and destroys the thread dump evidence.

Connection pool exhaustion

• Enable abandoned connection reclamation on the pool: removeAbandoned=true, removeAbandonedTimeout=60. This forcibly reclaims leaked connections.
• Enable logAbandoned=true to capture the stack trace where each leaked connection was borrowed. That stack trace is the pointer to the bug.
• Verify pool sizing. maxActive too low starves the app under load; too high overwhelms the database. Size relative to what the database can handle multiplied by your Tomcat instance count.

GC death spiral (high CPU)

• Capture the heap dump before restarting. Non-negotiable. Without the dump you cannot diagnose the leak.
• Restart to recover service. Accept that you will lose in-memory state (sessions, caches).
• Analyze the heap dump offline. Eclipse MAT’s “dominator tree” identifies the single object retaining most of the heap. Common culprits: unbounded caches, session accumulation (bot traffic creating sessions), static collections that are never cleared.

CPU-bound application (high CPU, not GC)

• Pull a CPU profile (async-profiler or JFR) during the incident, not after.
• Look at recent deploys. Sudden CPU-bound behavior almost always correlates with a code change. Check regression in regex patterns, serialization paths, or new loops.
• Check JIT deoptimization if you run with -XX:+PrintCompilation. A deoptimized hot method can cause sudden latency regression without a code change.

Prevention

• Set explicit timeouts on every outbound call. JDBC connect and query timeouts, HTTP client connect and read timeouts, DNS lookup timeouts. The default of “infinite” is what turns a slow backend into a thread pool exhaustion.
• Configure StuckThreadDetectionValve with a threshold appropriate to your application (default 600 seconds is too high for most user-facing services). 60 seconds is generous. This valve is not enabled by default.
• Monitor post-GC heap baseline, not raw heap usage. Alerting on “heap over 80%” produces constant false positives during normal sawtooth behavior. Alert on rising valleys instead.
• Set -XX:MaxMetaspaceSize. Without it, a classloader leak grows silently until the OS OOM-kills the process with no JVM-level error.
• Wire backend latency into the same dashboard as thread pool state. The fastest diagnosis comes from seeing the database latency spike and the thread pool fill at the same moment.
• Use circuit breakers on outbound calls so a failing backend does not consume all threads. A fast fail under load is better than an indefinitely hung thread.

How Netdata helps

• Per-second resolution on currentThreadsBusy, maxThreads, and JVM CPU makes the low-CPU-or-high-CPU split obvious in seconds. You can see the exact moment the pool saturated and whether CPU climbed with it.
• GC collection count, collection time, and heap utilization are collected at the same cadence, so correlating a thread pool saturation event with a GC pause or rising post-GC baseline is a single dashboard glance.
• JDBC pool metrics (numActive, numIdle, waitCount) appear alongside thread pool metrics, which exposes the common cascade where database pool exhaustion drives thread pool exhaustion.
• Anomaly detection on thread pool ratio, CPU, and GC time surfaces drift toward saturation before currentThreadsBusy hits maxThreads.

Related guides

• Tomcat thread pool exhaustion: currentThreadsBusy at maxThreads and requests hanging
• Tomcat accepts connections but never responds: the TCP-connect trap
• Tomcat HTTP Status 503 Service Unavailable: the connector is out of threads
• How Tomcat actually works in production: a mental model for operators
• Tomcat monitoring checklist: the signals every production instance needs
• Tomcat process not running: crashes, OOM-kills, and failed restarts