Tomcat accepts connections but never responds: the TCP-connect trap

A client opens a TCP connection to Tomcat on port 8080. The three-way handshake completes. The client sends an HTTP request. Nothing comes back. The connection hangs until the client times out. Meanwhile, the load balancer health check still reports the instance as healthy, because its check is a bare TCP connect that succeeds every time.

A successful connect() only proves the kernel accepted the socket. It says nothing about whether Tomcat has a worker thread available, whether the JVM is mid-garbage-collection, or whether any deployed application can serve a response. On NIO Tomcat, the default connector since 8.5, the OS accept queue and the connector poller can hold thousands of connections even when the worker thread pool is completely exhausted. A port check is nearly useless as a health signal.

The fix is structural. Health checks must exercise the servlet pipeline, and monitoring must distinguish three failure classes that all produce the same symptom: thread pool exhaustion, a GC death spiral, and a connector that failed to bind. All three look identical from a TCP perspective but require different responses.

What this means

Tomcat’s request path has multiple buffers that decouple “TCP connect succeeded” from “request is being processed”.

1. The OS accept queue holds sockets that completed the handshake but have not yet been accepted by the connector’s acceptor thread. Its depth is bounded by acceptCount (default 100) and capped by the kernel’s somaxconn.
2. The NIO poller multiplexes accepted sockets and dispatches ready ones to the worker thread pool. Its capacity is bounded by maxConnections (default 8192 for NIO).
3. The worker thread pool processes requests. Its capacity is bounded by maxThreads (default 200).

A TCP connect succeeds as long as the OS accept queue has room. The request can then sit indefinitely if every worker thread is busy, if the JVM is paused for garbage collection, or if the acceptor thread itself is blocked. No JMX counter records this wait, and Tomcat logs nothing until the request is either picked up or a configured timeout fires.

flowchart LR
  Client -->|TCP SYN| OS["OS accept queue
acceptCount = 100"]
  OS -->|accept| Acceptor["Acceptor thread"]
  Acceptor -->|register| Poller["NIO poller
maxConnections = 8192"]
  Poller -->|dispatch| Pool["Worker thread pool
maxThreads = 200"]
  Pool -->|process| App["Application code"]
  Pool -.->|no free thread| Wait["Request waits until timeout"]

Each layer can absorb a burst independently. A slow backend can fill all 200 threads while thousands of connections sit in the poller and more pile up in the accept queue. From the network side, every connect succeeds. From the application side, no request has been processed in minutes.

Common causes

The connector-failed-to-bind case is the inverse of the trap: the client gets an immediate refusal, not a hang. It belongs in the differential because operators often chase the wrong layer.

Quick checks

Run these read-only. None mutate state. Some require the same user as the JVM process or root.

# TCP connect vs HTTP request: the core of the trap.
# A successful connect followed by a hung request confirms the diagnosis.
# Replace /health with your actual application health endpoint.
time curl --max-time 5 -v http://localhost:8080/health

# Is the JVM process alive?
pgrep -f 'catalina.startup.Bootstrap' || echo "DOWN"

# Is the port bound, and is the accept queue backing up?
# No LISTEN line means the connector failed to start.
# Recv-Q non-zero means sockets are waiting to be accepted.
# Send-Q is the configured backlog (acceptCount).
ss -tnl 'sport = :8080'

# Thread pool state via Manager (requires Manager app enabled).
curl -s -u "$USER:$PASS" 'http://localhost:8080/manager/status?XML=true' | \
  grep -oP '(maxThreads|currentThreadsBusy|currentThreadCount)="[0-9]+"'

# GC activity. FGC is Full GC count. High and rising is bad.
# If multiple JVMs run, specify the PID explicitly.
jstat -gcutil "$(pgrep -f 'catalina.startup.Bootstrap' | head -1)" 1000

# File descriptor pressure, another saturation path.
PID="$(pgrep -f 'catalina.startup.Bootstrap' | head -1)"
ls "/proc/$PID/fd" | wc -l
grep "Max open files" "/proc/$PID/limits"

How to diagnose it

1. Confirm the trap. Run a TCP connect test and an HTTP request test against the same port. If connect succeeds and HTTP hangs, you are inside the trap.
2. Check the port. ss -tnl 'sport = :8080'. No LISTEN line means the connector failed to bind. Read catalina.out for the bind exception.
3. Check the worker pool. Compare currentThreadsBusy to maxThreads. At or near maxThreads sustained means every request is queuing behind occupied threads.
4. Take a thread dump immediately. jstack <pid>. Look at what the http-nio-8080-exec-* threads are doing. A large cluster of threads parked on the same socket read, lock, or database call points directly at the guilty dependency.
5. Check GC. A stop-the-world pause freezes the acceptor and every worker simultaneously. A Full GC during your incident is a prime suspect if threads are not individually blocked on a backend.
6. Check the accept queue. ss -tnl 'sport = :8080' Recv-Q. A sustained non-zero value means the acceptor cannot keep up. Combined with a full thread pool, this is the end state of the cascade.
7. Check connections. connectionCount approaching maxConnections indicates either a slow-client pattern or a load balancer holding an oversized keepalive pool.

The decisive split is thread dump content. If worker threads are parked on the same downstream call, the backend is the cause and restarting Tomcat only buys you minutes. If worker threads are scattered and GC is hammering, the JVM is the cause. If threads are healthy and the port is not bound, the connector is the cause.

Metrics and signals to monitor

Fixes

Thread pool exhaustion

The immediate mitigation is to stop the inflow: drain the node at the load balancer, then let the queued work complete or fail. A blind restart is the wrong move because the same load will refill the pool the moment traffic returns.

The real fix is upstream of Tomcat. Look at the thread dump: the dominant blocking call is your culprit. Add explicit timeouts to every outbound HTTP client and JDBC call. The default socket timeout on most Java HTTP clients and JDBC drivers is infinite, which is what turns a slow dependency into a permanent thread sink. Configure StuckThreadDetectionValve with a threshold appropriate to your latency budget, because the default of 600 seconds is too long for most user-facing services. Only consider raising maxThreads after timeouts are in place: more threads without timeouts just means more threads stuck.

GC death spiral

Capture a heap dump before doing anything destructive: jmap -dump:live,format=b,file=/tmp/heap.hprof <pid>. The :live flag triggers a Full GC and may take time on a large heap. If the service must be recovered immediately, restart, then analyze the dump offline with Eclipse MAT to find the dominant retainer.

Longer term, the post-GC heap baseline (the valley of the sawtooth, not the instantaneous usage) is the metric that matters. A rising valley over hours or days is a leak. Adjust -Xmx, set -XX:MaxMetaspaceSize if it is unset, and verify that container memory limits are not lower than -Xmx or the OS OOM killer will strike before the JVM does.

Connector failed to bind

This is a startup problem, not a runtime one. Read catalina.out for the bind exception. Common causes are port conflicts, TLS certificate problems, or a binding failure caused by an already-running instance. The fix is configuration, not a restart. Confirm with ss -tnl after correction.

Slow client or connection leak

If connectionCount is high and bytes received are low, you have either a Slowloris-style attack or a misbehaving client. Tighten connectionTimeout (verify the value shipped in your server.xml), and ensure the reverse proxy rather than Tomcat is enforcing client-side timeouts. If a specific source IP dominates the connection table, block it upstream.

Prevention

• Health check the pipeline, not the port. Use an endpoint that exercises the servlet path. Tomcat ships org.apache.catalina.valves.HealthCheckValve, which responds at a configurable path and verifies container availability. In Spring Boot, prefer /actuator/health over a TCP check.
• Alert on thread pool ratio, not port liveness. Page when currentThreadsBusy / maxThreads is sustained at 1.0 for more than two minutes, with a maxThreads > 50 gate and an uptime gate to avoid cold-start noise.
• Set timeouts on every outbound call. Infinite timeouts are the single largest contributor to thread starvation. JDBC query timeout, HTTP client socket timeout, and connection validation should all have explicit values.
• Configure StuckThreadDetectionValve. It is not enabled by default. Its output is a log message and a JMX counter, both of which make stuck threads visible before they drain the pool.
• Watch the accept queue. It has no JMX representation. Monitor ss -tnl Recv-Q or client-side connection error rates.
• Track post-GC heap, not instantaneous heap. Instantaneous usage is supposed to spike. Alerting on it produces constant false positives and hides real leaks.

How Netdata helps

• Per-second thread pool metrics expose currentThreadsBusy climbing toward maxThreads before the trap closes.
• Correlating thread pool saturation with GC pause time separates a backend-driven stall from a JVM-driven stall in the same chart view.
• JVM heap and memory pool breakdowns show the rising post-GC baseline that predicts a GC death spiral before users are affected.
• OS-level socket and file descriptor metrics cover the accept queue depth and FD pressure that JMX cannot see.
• Anomaly detection on request throughput and error rate flags the throughput collapse that signals thread starvation without static thresholds.

Related guides

• How Tomcat actually works in production: a mental model for operators
• Tomcat process not running: crashes, OOM-kills, and failed restarts
• Tomcat monitoring checklist: the signals every production instance needs
• Tomcat monitoring maturity model: from survival to expert