Tomcat HTTP Status 503 Service Unavailable: the connector is out of threads

A 503 in a Tomcat topology means something between the user and the servlet gave up. The instinct is to blame the connector thread pool, and that is often the right place to look, but the mechanism is more layered than it appears. Treating “503” and “out of threads” as the same condition leads to misdiagnosis.

With the default NIO connector, Tomcat decouples worker threads from TCP connections. A request thread pool can be saturated while the connector keeps accepting connections, and the connection pool can be saturated while the JVM still passes a TCP health check. The path from a full thread pool to a visible 503 passes through two more buffers before any client sees a failure.

What this means

Key mechanics from the Tomcat connector model (covered in depth in the mental model hub):

• NIO is the default since Tomcat 8.5. A small set of poller threads multiplexes connections; request processing is dispatched to a bounded worker pool. Default sizing: maxThreads=200, minSpareThreads=10, maxConnections=8192, acceptCount=100, connectionTimeout=60000ms.
• Worker threads and connections are separate limits. Every active request occupies one worker thread for its full duration. Idle keepalive connections occupy a poller slot and a file descriptor, but not a thread.
• Saturation cascades through two queues. When all worker threads are busy, new connections are still accepted up to maxConnections. When maxConnections is reached, additional connections land in the OS accept queue (bounded by acceptCount). When the accept queue fills, the kernel rejects new connections with RST.

The crucial implication: when the worker thread pool is exhausted, the NIO connector does not synthesize an HTTP 503 response. The failure surfaces at the TCP layer as connection refused or, upstream of that, as a timeout. The 503 that users see is generated by an upstream load balancer or reverse proxy translating the failed backend connection into a 5xx response, or by the application itself returning 503 from servlet code. Tomcat itself only synthesizes a 503 while the connector is paused (for example during a graceful stop); thread-pool exhaustion surfaces as queuing and, at the limit, TCP-level refusal rather than an HTTP 503. The dominant pattern in production is proxy-synthesized 503 triggered by Tomcat refusing or timing out on the backend connection.

This distinction matters because the fix differs. If the 503 is proxy-synthesized from a saturated thread pool, raising maxThreads or fixing the slow backend resolves it. If the 503 is application-emitted, the thread pool may be fine and the problem is in servlet code. If the 503 is from connection exhaustion (poller full, not thread pool full), adding threads does nothing.

Common causes

Quick checks

Safe read-only commands. Run these on the Tomcat host. JMX commands assume a JMX remote endpoint (for example, -Dcom.sun.management.jmxremote.port=9090) is configured.

# Worker thread pool state via JMX
java -jar jmxterm.jar -l localhost:9090 -n -v silent -e \
  "get -b Catalina:type=ThreadPool,name=\"http-nio-8080\" currentThreadsBusy currentThreadCount maxThreads connectionCount maxConnections"

# Accept queue depth (Recv-Q is current backlog, Send-Q is the configured max)
ss -tnl 'sport = :8080'

# Error count from JMX (lumps 4xx + 5xx, cannot isolate 503)
java -jar jmxterm.jar -l localhost:9090 -n -v silent -e \
  "get -b Catalina:type=GlobalRequestProcessor,name=\"http-nio-8080\" errorCount requestCount"

# 5xx rate from the access log (assumes default field positions, $9 is status)
awk '$9 ~ /^5/' /var/log/tomcat/localhost_access_log.$(date +%Y-%m-%d).txt | wc -l

# 503 specifically
awk '$9 == 503' /var/log/tomcat/localhost_access_log.$(date +%Y-%m-%d).txt | wc -l

# Thread dump: count http-nio worker threads and inspect their state
# If multiple Tomcat JVMs run on the host, narrow the pgrep pattern or use the specific PID
jstack "$(pgrep -f 'catalina.startup.Bootstrap' | head -1)" | grep -c "http-nio-8080-exec"

# Manager status XML (if Manager app is enabled and reachable)
curl -s -u "$USER:$PASS" 'http://localhost:8080/manager/status?XML=true' | \
  grep -oP '(maxThreads|currentThreadsBusy|currentThreadCount|connectionCount|maxConnections)="[0-9]+"'

How to diagnose it

1. Confirm where the 503 originates. Parse the access log on the Tomcat host. If Tomcat’s access log records the request with status 503, Tomcat (or the application) generated it. If the access log shows no matching request at all, the 503 was synthesized upstream and the Tomcat-side symptom is a refused or timed-out connection.

2. Check thread pool saturation. Read currentThreadsBusy and maxThreads. If busy is at or near max, the pool is the constraint. With virtual threads (JDK 21+), currentThreadsBusy reports -1 and maxThreads is not meaningful; the connector’s ThreadPool MBean still exposes connectionCount and keepAliveCount, so monitor connectionCount (minus keepAliveCount) as a proxy instead.

3. Check connection saturation. Read connectionCount against maxConnections. If connections are at the limit while threads still have headroom, the poller is the bottleneck, not the worker pool. Adding maxThreads will not help.

4. Check the accept queue. Run ss -tnl 'sport = :8080'. A non-zero Recv-Q means connections are backing up in the OS backlog. If Recv-Q equals Send-Q (the acceptCount), the kernel is refusing connections. This is the layer that produces the TCP RST an upstream proxy turns into a 503.

5. Take a thread dump. jstack shows exactly what each worker thread is doing. Many threads in the same WAITING or BLOCKED state, on the same socket read or connection acquire, points at the slow backend.

6. Correlate CPU. If currentThreadsBusy == maxThreads and CPU is low, threads are blocked on I/O (database, downstream HTTP, DNS). If CPU is high, the application is compute-bound or GC is thrashing. Low CPU with full threads is the classic thread-exhaustion signature.

The cascade below shows where each layer fails and what surfaces to the client:

flowchart TD
  A[Inbound request] --> B{Free worker thread?}
  B -- yes --> C[Dispatch and process]
  B -- no --> D{connectionCount less than maxConnections?}
  D -- yes --> E[Queued, awaiting worker thread]
  D -- no --> F{accept queue less than acceptCount?}
  F -- yes --> G[Queue in OS backlog]
  F -- no --> H[Kernel sends RST]
  H --> I[Proxy synthesizes 502/503/504]

Metrics and signals to monitor

Fixes

Thread pool exhaustion from a slow backend

The durable fix is on the backend, not in Tomcat. Use the thread dump to identify the dependency, then add or tighten timeouts on the outbound call (JDBC query timeout, HTTP client socket timeout, DNS lookup timeout). Raising maxThreads buys headroom but does not solve the underlying problem; if the backend is slow, more threads just means more threads waiting. If you do raise maxThreads, watch file descriptor usage and connectionCount, since each thread and connection consumes an FD.

Thread pool exhaustion from insufficient sizing

If the thread dump shows threads processing quickly (not blocked) and currentThreadsBusy is still pegged at maxThreads under peak load, the pool is undersized for the workload. Increase maxThreads and verify the backend (especially the database connection pool) can absorb the additional concurrent load. A common mistake is raising Tomcat maxThreads past the database pool maxActive, which just moves the bottleneck downstream.

Connection (poller) saturation

If connectionCount is at maxConnections but threads have headroom, the issue is too many open connections, typically from upstream keepalive pools or a slowloris-style pattern. Tune keepAliveTimeout down from the 60s default (it defaults to connectionTimeout if unset), coordinate keepalive timers with the reverse proxy so the proxy does not reuse connections Tomcat has already closed (for example, nginx holding connections 75s while Tomcat closes at 60s causes resets), and investigate per-source connection counts with ss.

Accept queue overflow

If Recv-Q is at Send-Q, the OS is refusing connections. Increase acceptCount in server.xml, but verify net.core.somaxconn on the host, since the kernel caps the actual backlog at somaxconn regardless of what Tomcat requests. Also investigate why the acceptor thread is not keeping up; a stop-the-world GC pause blocks the acceptor along with every other thread.

Stuck threads

If jstack shows identical BLOCKED or WAITING stack traces across many worker threads, those threads are not coming back without intervention. Configure StuckThreadDetectionValve (not enabled by default; default threshold is 600s, consider 60s for user-facing services) for visibility. The fix is the blocking call: add a timeout or break a deadlock. Restarting Tomcat clears the threads but they will re-accumulate if the root cause remains.

Application-emitted 503

If Tomcat’s access log shows the 503 and the thread pool and connection metrics are healthy, the 503 is coming from servlet code. Grep the codebase for sendError(503), SC_SERVICE_UNAVAILABLE, or framework-level “service unavailable” responses (for example, a circuit breaker or a rate limiter tripping). The fix is in the application.

Prevention

• Monitor currentThreadsBusy / maxThreads as a first-class signal. Sustained >0.80 is a ticket. == 1.0 for >120s with maxThreads > 50 and uptime >120s is a page. Gate on uptime to avoid cold-start false positives.
• Set explicit timeouts on every outbound call. Default socket timeouts in JDBC drivers and HTTP clients are infinite. A hung downstream service will consume threads forever without them.
• Configure StuckThreadDetectionValve with a realistic threshold. The default 600s is too long for most user-facing services. Page when stuckThreadCount > maxThreads * 0.5.
• Coordinate keepalive timers across the proxy and Tomcat. Mismatched keepAliveTimeout and proxy keepalive produces connection resets that look like intermittent 503s.
• Monitor the accept queue with ss. It is invisible to JMX. A non-zero Recv-Q is the leading indicator of connection refusal.
• Parse the access log for 5xx specifically. JMX errorCount lumps 4xx and 5xx; it cannot tell you whether a 503 is happening.
• Confirm acceptCount against net.core.somaxconn. Setting acceptCount higher than somaxconn has no effect.

How Netdata helps

• Correlating currentThreadsBusy, maxThreads, connectionCount, and maxConnections on one per-second timeline makes it obvious whether the constraint is the worker pool or the poller, which determines the fix.
• ML anomaly detection on request throughput and processing time surfaces a slow backend draining the thread pool before currentThreadsBusy reaches maxThreads.
• The OS-level accept queue (ss Recv-Q) and file descriptor counts sit alongside the JVM metrics, so connection-layer saturation is not hidden behind a healthy JVM dashboard.
• Per-second resolution means the thread-exhaustion cascade (busy threads rising, throughput collapsing, errors following) is visible as a sequence rather than a single aggregated point.
• CPU and GC metrics next to thread pool state distinguish I/O-blocked exhaustion (low CPU) from compute-bound or GC-thrashing saturation (high CPU), which changes the response.
• Virtual-thread deployments, where currentThreadsBusy reports -1, still surface connection-count signals so the exhaustion profile remains observable.

Related guides

• How Tomcat actually works in production: a mental model for operators
• Tomcat process not running: crashes, OOM-kills, and failed restarts
• Tomcat monitoring checklist: the signals every production instance needs
• Tomcat monitoring maturity model: from survival to expert