Tomcat stuck threads: StuckThreadDetectionValve and 'may be stuck' warnings

The warning shows up in catalina.out: a worker thread “has been active for [N] milliseconds … and may be stuck.” By the time you see it, the thread has already been blocked for the full threshold duration. If the valve is running with its default 600-second threshold, that is 10 minutes of a worker thread doing nothing useful. With maxThreads=200, a handful of these threads eats real capacity.

The StuckThreadDetectionValve is Tomcat’s built-in canary for thread-pool exhaustion. It is not enabled by default. When enabled, it logs a WARN-level message whenever a request-processing thread exceeds a configurable threshold, and it includes the blocking stack trace. That stack trace is the single most useful piece of evidence you will get during a thread-starvation incident, because it tells you exactly where the thread is parked.

What this means

A “may be stuck” warning means a request-processing thread (one of the http-nio-<port>-exec threads) has been executing a single request for longer than the valve’s threshold. The valve wraps the request pipeline and times each thread from the moment it is dispatched until the response is committed. When the elapsed time crosses the threshold, the valve fires the warning with a stack trace of the offending thread.

Two constraints matter when interpreting these warnings.

First, the valve only monitors request-processing threads. Background threads (ContainerBackgroundProcessor, catalina-utility threads, application-spawned schedulers) are invisible to it. A deadlock or hang in a background thread will not trigger this warning. If your symptom is a scheduled job that stopped running or a cluster heartbeat that went silent, the valve will not help.

Second, the warning is a lagging indicator. With the default 600-second threshold, the thread has been stuck for 10 minutes before you hear about it. The valve’s detection runs on the container’s background processor thread, so the effective resolution is bounded by backgroundProcessorDelay (default 10 seconds). If you set the threshold below that interval, detection will lag further. Lower the threshold to 60 to 120 seconds for production use.

The valve’s companion message matters too. When a previously-stuck thread completes, it logs that the thread “was previously reported to be stuck but has completed,” with the total active time and a count of threads still stuck. Watch for threads that are reported stuck and never complete. Those are the ones consuming pool slots permanently.

flowchart TD
    A[Request arrives at connector] --> B[Worker thread dispatched]
    B --> C{Backend responds?}
    C -- No timeout / hangs --> D[Thread blocks on I/O]
    C -- Responds in time --> E[Thread returns to pool]
    D --> F[Elapsed time exceeds valve threshold]
    F --> G[Valve fires WARN with stack trace]
    D --> H[Thread stays occupied]
    H --> I[More requests hit same path]
    I --> J[Pool drains toward maxThreads]
    J --> K[Accept queue fills, connections refused or proxy 503s]

Common causes

Quick checks

Run these read-only. None of them modify Tomcat state.

# Check whether the valve is configured at all
grep -ri StuckThreadDetectionValve $CATALINA_BASE/conf/

# Look for active or recent "may be stuck" warnings
grep "may be stuck" $CATALINA_BASE/logs/catalina.out | tail -20

# Take a thread dump immediately (the diagnostic goldmine)
# Run as the same OS user that owns the JVM. In containers, use jcmd or kill -3 <pid>.
# If multiple Tomcat instances run, narrow the pgrep pattern or specify the PID directly.
jstack -l $(pgrep -f 'catalina.startup.Bootstrap' | head -1) > /tmp/tomcat-threads.txt

# Count worker threads by state
grep "http-nio" /tmp/tomcat-threads.txt | grep "java.lang.Thread.State" | sort | uniq -c

# Check thread pool saturation via Manager status (requires the manager webapp deployed)
curl -s -u $USER:$PASS 'http://localhost:8080/manager/status?XML=true' | \
  grep -oP '(currentThreadsBusy|currentThreadCount|maxThreads)="[0-9]+"'

# Check JVM CPU (low CPU + stuck threads = waiting on I/O, not computing)
top -p $(pgrep -f 'catalina.startup.Bootstrap' | head -1) -bn1 | tail -2

# If the valve is configured, read stuckThreadCount via JMX.
# The ObjectName varies by where the valve is deployed (Engine, Host, or Context).
# Adjust host and context accordingly.
java -jar jmxterm.jar -l localhost:9090 -n -v silent -e \
  "get -b Catalina:type=Valve,name=StuckThreadDetectionValve stuckThreadCount"

How to diagnose it

1. Confirm the valve is active. If grep finds nothing in server.xml or context.xml, the valve is not configured. The warnings you are chasing came from a different source, or you are looking at the wrong host. Add the valve before the next incident so you actually get the data.

2. Read the stack trace in the warning. The valve logs the stuck thread’s stack trace as part of the WARN record. This is the fastest path to root cause. The top frames tell you what the thread is blocked on: socketRead0 means network I/O, Object.wait or park means waiting on a monitor or connection pool, ThreadPoolExecutor.getTask means waiting for work. Match the frame to the cause table above.

3. Take a fresh jstack if the warning is stale. The valve captures the stack at the moment the threshold was crossed, which is usually what you want. But if the thread has moved on or the situation is evolving, a fresh dump shows current state. Take two dumps 5 to 10 seconds apart and diff them to confirm the thread is truly parked in the same place.

4. Check the backend the thread is waiting on. If the stack shows a JDBC call, check the database. If it shows an outbound HTTP client call, check that downstream service. Stuck threads are almost never a Tomcat bug. They are a backend call with no timeout.

5. Check thread pool utilization. currentThreadsBusy approaching maxThreads confirms the stuck threads are consuming real capacity. If busy is low despite warnings, the stuck threads are clearing on their own, which points to intermittent backend slowness rather than a permanent hang.

6. Check CPU. Low CPU with threads stuck means I/O-bound waiting, which is the common case. High CPU means an infinite loop or pathological computation. The fix is entirely different.

Metrics and signals to monitor

One nuance on stuckThreadCount: the valve exposes this as a JMX attribute on the Valve MBean, and it is pollable, but it is a derived count of threads currently in the valve’s monitored set. The WARN log line remains the primary operational signal because it includes the stack trace. The count tells you how many; the log tells you where and why.

Fixes

Capture evidence before restarting

Do not restart Tomcat as a first response. The stack traces are your evidence, and a restart destroys them. Capture jstack output first, identify the blocking backend, then decide whether a restart is warranted. Wire jstack capture into your restart runbook so this happens automatically.

Backend call with no timeout

This is the most common cause and the one with the cleanest fix. Find the client library in the stack trace and configure an explicit timeout:

• Apache HttpClient: set setSocketTimeout and setConnectTimeout on the RequestConfig (HttpClient 4.x) or setResponseTimeout / setConnectTimeout (HttpClient 5.x).
• OkHttp: call readTimeout and connectTimeout on the OkHttpClient.Builder. OkHttp has a 10-second default since 3.x, but verify your builder is not overriding it to 0.
• JDBC: set query timeout on the Statement, or set a socket timeout in the connection URL (for example socketTimeout=30000 for the MySQL connector).
• java.net.HttpURLConnection: call setConnectTimeout and setReadTimeout. Both default to 0 (infinite).

The default socket timeout in java.net.HttpURLConnection, Apache HttpClient, and most JDBC drivers is infinite (0). A single hung backend call will consume a thread forever. Every outbound call on the request path needs an explicit timeout that is shorter than your tolerance for user-facing latency.

Deadlock

If jstack prints “Found one Java-level deadlock” at the bottom of the dump, you have a lock-ordering bug in application code. This requires a code fix. The only immediate operational response is a restart, but the deadlock will recur until the code is fixed. Capture the full jstack (the deadlock graph identifies the participating threads and locks) and route it to the application owners.

Infinite loop

A thread stuck in RUNNABLE state with high CPU is in a loop. The stack trace identifies the method. Common culprits are regex without input bounds (ReDoS), unbounded recursion degenerating into a loop, or a busy-wait missing a wakeup condition. This is a code fix. Restarting clears it until the same input triggers it again.

JDBC pool exhaustion

If the stack shows the thread waiting to borrow a connection, the database connection pool is the bottleneck, not the query itself. Check the pool MBean: numActive versus maxActive for DBCP, activeConnections versus maximumPoolSize for HikariCP. If connections are borrowed but sitting idle (a leak), enable leak detection:

• Apache Commons DBCP / Tomcat JDBC Pool: set removeAbandoned=true and logAbandoned=true to reclaim leaked connections and log the stack trace where each was checked out.
• HikariCP: set leakDetectionThreshold (in milliseconds) to log a warning when a connection is held longer than expected.

If the pool is genuinely undersized, raise the max, but verify the database server can handle the additional concurrent connections.

interruptThreadThreshold (use with caution)

The valve has a second parameter, interruptThreadThreshold, that calls Thread.interrupt() on a thread stuck longer than this secondary threshold. The default is -1 (disabled). Enabling it sets the thread’s interrupt flag, which works for threads blocked on I/O or locks because they will throw InterruptedException or wake from a wait. It does nothing useful for threads in CPU-bound loops or certain native calls. The official docs warn there is no guarantee the thread will stop. Treat this as a last-resort circuit breaker, not a real fix. It may let a stuck thread release its pool slot without a full restart, but it will not solve the underlying missing timeout or deadlock.

Prevention

• Enable the valve in every production Tomcat. It is one XML element and it gives you the stack trace you need during incidents. Add it inside the Engine, Host, or Context in server.xml:

  <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60" />
  

• Set the threshold to 60 seconds, not 600. The default 10-minute window is too long for incident response. The valve’s detection runs on the container background processor, which ticks every backgroundProcessorDelay seconds (default 10), so the effective detection resolution is bounded by that interval. 60 seconds leaves comfortable margin.

• Configure explicit timeouts on every outbound call. This is the single highest-leverage prevention step. No outbound HTTP, JDBC, or other I/O call in a request path should rely on an infinite default timeout.

• Monitor currentThreadsBusy / maxThreads as your primary saturation signal. Stuck threads are a subset of busy threads. The ratio tells you whether they are consuming meaningful capacity before the pool hits the wall.

• Capture jstack automatically before any restart. Wire this into your restart runbook and your container preStop hook so evidence is never lost. Without the stack trace, you are guessing at the cause.

If you run embedded Tomcat under Spring Boot, there is no server.tomcat.stuck-thread-detection.* property. Register the valve programmatically via a WebServerFactoryCustomizer<TomcatServletWebServerFactory> bean that calls factory.addEngineValves(new StuckThreadDetectionValve()).

How Netdata helps

• Correlate stuck-thread log bursts with thread pool utilization. When currentThreadsBusy climbs toward maxThreads at the same time the valve fires, you see the cascade forming on one timeline rather than reconstructing it from separate tools.
• Distinguish I/O-bound stalls from CPU-bound loops. JVM CPU utilization alongside thread pool saturation tells you immediately whether threads are waiting on a backend (low CPU) or spinning (high CPU), which changes the entire response.
• Track request throughput against thread pool saturation. Throughput dropping while busy threads rise is the signature of threads blocked rather than processing.
• Surface connection refusals and proxy 503 spikes alongside GC pause time. Pool exhaustion and a GC pause look different when you can see both signals per second on the same chart.
• Alert on the currentThreadsBusy / maxThreads ratio with per-second resolution, so you catch pool saturation before the valve’s threshold even fires.

Related guides

• Tomcat accept queue overflow: acceptCount, somaxconn, and Recv-Q
• Tomcat java.net.BindException: Address already in use: the connector never starts
• Tomcat average latency lies: why you need p95/p99 from the access log
• Tomcat classloader leak on redeploy: why the old WebappClassLoader never dies
• Tomcat connection refused: maxConnections and acceptCount both exhausted
• Tomcat accepts connections but never responds: the TCP-connect trap
• Tomcat frequent Full GC: pause time, G1, and the 5% overhead rule
• Tomcat GC death spiral: full GCs dominating and throughput collapsing
• Tomcat heap dump before restart: capturing evidence with jmap and jstack
• Tomcat heap usage: watch the post-GC baseline, not the sawtooth peak
• How Tomcat actually works in production: a mental model for operators
• Tomcat HTTP Status 503 Service Unavailable: the connector is out of threads