Tomcat threads blocked forever: the missing outbound timeout

The JVM is healthy. Heap is fine, no Full GCs, CPU sits at 5%, and the process answers a TCP connect on 8080 within milliseconds. But every HTTP request hangs, the access log has gone quiet, and clients are timing out. When you finally grab a thread dump, most of the http-nio-8080-exec threads are parked in the same stack frame: java.net.SocketInputStream.socketRead0, blocked against a backend that will never reply.

This is the missing-outbound-timeout failure. It is one of the most common causes of Tomcat thread pool exhaustion, and it is almost never Tomcat’s fault. The threads are waiting forever because somewhere in the application or its libraries, an outbound call was made with no connect timeout and no read timeout. In Java, “no timeout” does not mean a sensible default. It means infinite.

Every request occupies one worker thread for its entire duration, and the default pool caps at maxThreads=200. When threads stop returning, the pool drains toward zero idle slots. Long before the JVM shows any distress, your service is effectively dead: the listener keeps accepting connections (up to maxConnections, default 8192 for NIO), requests queue behind the missing threads, and once the OS accept queue (acceptCount, default 100) fills, clients see connection refused.

What this means

Tomcat’s worker thread pool is a fixed, bounded resource. A request is dispatched to a worker thread the moment the NIO poller sees data on the socket, and that thread is held until the response is committed. If the request handler makes a synchronous outbound call, the same thread blocks inside that call. The connector’s own connectionTimeout (default 60000ms) governs only the client-to-Tomcat side: how long Tomcat waits for the request line and body. It has no effect on calls your application makes to other services.

The defaults are the problem. java.net.HttpURLConnection ships with setConnectTimeout and setReadTimeout both defaulting to 0, and the JDK documents 0 as an infinite timeout. Spring’s RestTemplate, when backed by the default SimpleClientHttpRequestFactory, inherits those infinite defaults. Most JDBC drivers ship with socketTimeout=0 (infinite): MySQL Connector/J has connectTimeout=0 and socketTimeout=0 by default, and PostgreSQL’s JDBC driver defaults socketTimeout to 0 even though connectTimeout defaults to 10 seconds. A thread that blocks on any of these will not return until the OS gives up on the TCP connection, which under a half-open or silent peer can be effectively never.

Tomcat has no built-in mechanism to interrupt a thread that is processing a request. StuckThreadDetectionValve can warn you, but it does not free the thread unless you explicitly set interruptThreadThreshold, which is disabled (-1) by default. The stuck thread stays stuck. If enough threads accumulate in that state, thread pool exhaustion becomes mathematically inevitable.

flowchart LR
  A["Backend stops responding"] --> B["Worker blocks in socketRead0"]
  B --> C["Thread held forever, no read timeout"]
  C --> D["Pool drains toward maxThreads"]
  D --> E["New requests queue for a free thread"]
  E --> F["Accept queue (acceptCount) fills"]
  F --> G["Kernel refuses connections, 503s rise"]
  G --> H["Service down: JVM healthy, CPU low"]

Common causes

Quick checks

# Check thread pool utilization via Manager Status XML
curl -s -u "$USER:$PASS" 'http://localhost:8080/manager/status?XML=true' | \
  grep -oP '(maxThreads|currentThreadsBusy|currentThreadCount)="[0-9]+"'

# Read-only JMX: thread pool saturation
java -jar jmxterm.jar -l localhost:9090 -n -v silent -e \
  "get -b Catalina:type=ThreadPool,name=\"http-nio-8080\" currentThreadsBusy currentThreadCount maxThreads"

# Thread dump - the diagnostic goldmine (read-only, safe on a live JVM)
jstack "$(pgrep -f 'catalina.startup.Bootstrap')" > /tmp/tomcat-threads.txt

# How many exec threads are blocked in a native socket read?
grep -c 'socketRead0' /tmp/tomcat-threads.txt

# Group exec threads by their top application stack frame
grep -A 5 'http-nio-8080-exec' /tmp/tomcat-threads.txt | grep -E '^\s+at ' | sort | uniq -c | sort -rn | head

# JVM CPU should be LOW if threads are stuck on I/O (differentiates from GC/CPU spiral)
ps -p "$(pgrep -f 'catalina.startup.Bootstrap')" -o %cpu,%mem,etime

# Accept queue depth (last buffer before the kernel refuses connections)
ss -tnl 'sport = :8080'

# Is StuckThreadDetectionValve even configured?
grep -i 'StuckThreadDetectionValve' "$CATALINA_BASE/conf/server.xml" || echo "NOT CONFIGURED"

# JDBC pool pressure (bean name varies by pool implementation)
java -jar jmxterm.jar -l localhost:9090 -n -v silent -e \
  "get -b tomcat.jdbc:type=ConnectionPool,name=\"jdbc/mydb\" numActive numIdle maxActive waitCount"

How to diagnose it

1. Confirm it is thread starvation, not GC or CPU. currentThreadsBusy should be at or near maxThreads, JVM CPU should be low, and there should be no Full GC storm. If CPU is high and threads are busy, you are looking at a CPU-bound or GC spiral instead.

2. Take two thread dumps 30 to 60 seconds apart. jstack is read-only and safe on a live JVM. Threads that appear in the same stack frame in both dumps are genuinely stuck. Threads that move between dumps are just slow.

3. Look for the signature frame. The classic stuck-on-outbound-call pattern is java.net.SocketInputStream.socketRead0 (Native Method). Group all http-nio-8080-exec threads by their top few stack frames. When dozens are parked in the same outbound call site, that is your culprit path.

4. Read up the stack to identify the outbound target. The frames above socketRead0 tell you whether the call is HTTP (HttpClient, OkHttp, HttpURLConnection, RestTemplate) or JDBC (a driver-specific socket read). The application frame above that tells you which service or database the call was aimed at.

5. Correlate with backend latency. If you have per-backend latency metrics, the stuck Tomcat threads line up in time with a latency spike or outage on exactly one downstream. Thread pool exhaustion is almost always a backend problem wearing a Tomcat costume.

6. Verify the missing timeout. Once you know the call site, inspect the client configuration. Look for new RestTemplate(), an HttpClient built without a RequestConfig, a JDBC URL missing socketTimeout, or a DataSource whose validationQueryTimeout is -1. The fix is almost always “the timeout was never set,” not “the timeout was set too high.”

Metrics and signals to monitor

Fixes

Set HTTP client connect and read timeouts

For HttpURLConnection, set both before connecting:

conn.setConnectTimeout(5000);   // 5s to establish TCP
conn.setReadTimeout(10000);     // 10s to receive any byte

A value of 0 means infinite. Always set both; a missing read timeout is the one that bites in production.

For Spring RestTemplate, the default SimpleClientHttpRequestFactory inherits HttpURLConnection’s infinite timeouts. Configure the factory explicitly, or switch to a client (Apache HttpClient, OkHttp) and configure timeouts on the builder. OkHttp defaults connect, read, and write timeouts to 10 seconds, which is why OkHttp-based services rarely exhibit this failure.

For Apache HttpClient 5, set both connect and response (socket) timeouts explicitly via RequestConfig or ConnectionConfig; note that RequestConfig.Builder.setConnectTimeout is deprecated in 5.x in favour of ConnectionConfig.Builder.setConnectTimeout. The response timeout is easy to miss: if it is left undefined, the connection-level socket timeout applies, and if that is also unset, reads can block indefinitely.

Set JDBC socket and query timeouts

Driver-level socket timeouts are the real fix. They cap every statement, including the ones your ORM generates that you never see. Add them to the JDBC URL or DataSource properties.

• MySQL Connector/J: connectTimeout and socketTimeout both default to 0 (infinite). Set them explicitly, for example connectTimeout=5000&socketTimeout=30000. Note that socketTimeout is a hard ceiling on every query, so set it above your slowest legitimate query or you will get false positives during long-running operations.
• PostgreSQL JDBC: connectTimeout defaults to 10 seconds, but socketTimeout defaults to 0 (infinite). Set socketTimeout explicitly.
• Tomcat JDBC pool: maxWait (default 30000ms) caps how long a thread waits to borrow a connection, and removeAbandonedTimeout (default 60s) reclaims connections held too long. Neither protects you from a statement that hangs after checkout; the driver socketTimeout does.

A statement-level setQueryTimeout is complementary but requires the application to set it per statement. The driver socketTimeout is global and is the one that catches uninstrumented code.

Configure StuckThreadDetectionValve

This does not free threads by itself, but it makes the failure visible. Add it to server.xml (or context.xml) inside the relevant Host or Context:

<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve"
       threshold="60"
       interruptThreadThreshold="600" />

The default threshold is 600 seconds (10 minutes). For most user-facing services, 60 seconds is more useful as an early signal. interruptThreadThreshold defaults to -1, which means Tomcat will log the stuck thread but never interrupt it. Setting it to a finite value lets Tomcat attempt to interrupt threads that have been stuck for far too long, which can recover a pool that has drained into a single bad code path. Treat interruption as a safety net, not a strategy: a thread interrupted mid-call can leave resources in an inconsistent state.

Do not confuse server.tomcat.connection-timeout with a backend timeout

A common misdiagnosis: an operator “sets the Tomcat timeout” via Spring Boot’s server.tomcat.connection-timeout, threads still block, and the team concludes timeouts do not help. That property governs how long Tomcat waits for the client to send the HTTP request line after the connection is accepted. It has no effect on calls your servlet makes to a database or downstream HTTP service. The outbound timeout has to be set on the outbound client.

Circuit breakers and bulkheads

Timeouts stop the bleeding; circuit breakers stop the cascade. Once a backend is failing, a circuit breaker lets subsequent calls fail fast instead of consuming a thread for the full timeout window. Bulkheads (separate thread pools per downstream) prevent one bad backend from consuming the entire HTTP worker pool. These are application-level concerns, but they are the structural fix that turns “one slow dependency takes down the whole service” into “one slow dependency degrades one feature.”

Prevention

• Every outbound call has a connect timeout and a read timeout. No exceptions for internal services; internal services hang too.
• JDBC socketTimeout is set in every DataSource. Audit every JDBC URL and pool configuration, not just the application code.
• StuckThreadDetectionValve is configured in every server.xml. With a threshold matched to your latency budget, not the 10-minute default.
• Per-backend latency is monitored. If you cannot see each downstream’s latency, you cannot correlate it with the thread drain.
• No code path constructs an HTTP client or RestTemplate without explicit timeouts. Treat new RestTemplate() and HttpURLConnection.openConnection() without timeouts as a code-review failure.
• Thread dumps are taken during incidents, not just postmortems. jstack is cheap and read-only; the stack frames are the evidence.

How Netdata helps

• Per-second thread pool utilization (currentThreadsBusy / maxThreads) makes the drain visible long before the pool hits the wall.
• The composite signature (threads at max, throughput collapsing, processing time rising, JVM CPU low) is the exact fingerprint of a stuck-on-backend pattern, and per-second resolution lets you watch it form in real time.
• StuckThreadDetectionValve stuckThreadCount surfaces directly once the valve is configured, turning a log-only signal into an alertable metric.
• JDBC connection pool metrics (numActive, numIdle, maxActive, waitCount) let you distinguish a database pool exhaustion cascade from a pure HTTP-backend hang.
• Correlating Tomcat thread state with downstream service latency, when Netdata is also monitoring the backend, pinpoints which backend owns the stuck threads. That is the part of this diagnosis that takes longest to do by hand.
• ML anomaly detection on thread-pool and throughput signals flags the drain even when no single static threshold has been crossed yet.

Related guides

• Tomcat 5xx error rate: separating server failures from crawler 404s
• Tomcat accept queue overflow: acceptCount, somaxconn, and Recv-Q
• Tomcat access log setup: adding %D and %T for per-request latency
• Tomcat java.net.BindException: Address already in use: the connector never starts
• Tomcat average latency lies: why you need p95/p99 from the access log
• Tomcat classloader leak on redeploy: why the old WebappClassLoader never dies
• Tomcat connection refused: maxConnections and acceptCount both exhausted
• Tomcat accepts connections but never responds: the TCP-connect trap
• Tomcat frequent Full GC: pause time, G1, and the 5% overhead rule
• Tomcat GC death spiral: full GCs dominating and throughput collapsing
• Tomcat heap dump before restart: capturing evidence with jmap and jstack
• Tomcat heap usage: watch the post-GC baseline, not the sawtooth peak