Tomcat JDBC connection pool exhaustion: numActive at maxActive and threads blocking

HTTP requests start timing out, currentThreadsBusy climbs toward maxThreads, CPU stays low, and the database looks healthy. The Tomcat JDBC pool has hit its ceiling: numActive equals maxActive, waitCount is greater than zero, and every request that needs a database connection is parked in getConnection().

The JDBC pool and the HTTP worker thread pool are coupled. A request holds a worker thread for its entire duration. When it blocks in getConnection(), that worker stays occupied. Database pool exhaustion drives thread pool exhaustion, and thread pool exhaustion drives accept queue buildup and 503s. By the time users see errors, both resources are already exhausted.

The recovery path depends on why the pool is full. The two dominant causes produce the same JMX signature but need opposite fixes: a connection leak (connections borrowed and never returned) versus a slow or overloaded database (connections held legitimately but for too long). This article covers distinguishing them and acting on the right one.

What this means

The Tomcat JDBC pool (tomcat-jdbc) is a bounded set of database connections exposed as a JNDI DataSource. Requests borrow a connection, run a query or transaction, and return it. The ceiling is maxActive. The four counters that matter: numActive (currently borrowed), numIdle (available), maxActive (ceiling), and waitCount (threads blocked waiting for a connection). The size attribute reports total connections held, active plus idle.

Exhaustion is numActive == maxActive with waitCount > 0. When a thread calls getConnection() and no idle connection is available, it blocks up to maxWait (default 30000ms). Tomcat JDBC is designed to be starvation-proof: a returned connection goes to a waiting thread rather than back to the idle pool. But if no connection is returned within maxWait, the caller gets a SQLException and the request fails.

The cascade turns a pool problem into an outage. A request holding a worker thread that blocks in getConnection() occupies that thread for the full maxWait window, and the database connection is unavailable to any other request. Under load the thread pool fills with parked threads, currentThreadsBusy climbs toward maxThreads, and from there the failure cascade is the standard thread exhaustion one. See how Tomcat actually works in production for the connector and thread pool model that feeds this.

flowchart TD
    A[HTTP request arrives] --> B[Worker thread acquired]
    B --> C{getConnection}
    C -->|numActive less than maxActive| D[Query runs]
    D --> E[Connection returned]
    E --> F[Thread released]
    C -->|numActive equals maxActive| G[Block up to maxWait]
    G --> H[Worker thread held]
    H --> I[Thread pool fills]
    I --> J[Accept queue builds]
    J --> K[503 or timeout]

Common causes

The leak and the slow backend produce the same JMX signature at the peak. The distinguishing test is what happens when load stops. A leak keeps numActive pinned at maxActive. A slow backend drains it.

Quick checks

All read-only and safe to run during an incident.

# Confirm pool exhaustion: numActive, numIdle, size, maxActive, waitCount
java -jar jmxterm.jar -l localhost:9090 -n -v silent -e \
  "get -b tomcat.jdbc:type=ConnectionPool,name=\"jdbc/mydb\" numActive numIdle size maxActive waitCount"

# Correlate with worker thread saturation
java -jar jmxterm.jar -l localhost:9090 -n -v silent -e \
  "get -b Catalina:type=ThreadPool,name=\"http-nio-8080\" currentThreadsBusy maxThreads"

# Capture a thread dump to see where threads are parked.
# Run as the same user that owns the JVM process (or root).
jstack $(pgrep -f 'catalina.startup.Bootstrap') > /tmp/jdbc-threads.txt
grep -c "getConnection\|ConnectionPool" /tmp/jdbc-threads.txt

# Look for pool timeout or abandoned-connection logging
grep -iE "pool.*exhaust|cannot get a connection|abandoned" /var/log/tomcat/catalina.out | tail -50

# Check the database side: are Tomcat's connections sleeping or active?
# MySQL example:
# mysql -e "SELECT user,host,state,COUNT(*) FROM information_schema.processlist GROUP BY state;"

If the JMX query returns nothing, verify which pool implementation is in use. Spring Boot 2.0 and later default to HikariCP, not tomcat-jdbc. The bean names differ.

# HikariCP exposes different attributes under a different object name
java -jar jmxterm.jar -l localhost:9090 -n -v silent -e \
  "get -b com.zaxxer.hikari:type=Pool\ (HikariPool-1) ActiveConnections IdleConnections TotalConnections ThreadsAwaitingConnection"

On HikariCP, the equivalent exhaustion signature is ActiveConnections == TotalConnections with ThreadsAwaitingConnection > 0. The diagnostic flow is otherwise the same.

How to diagnose it

1. Confirm exhaustion via JMX. numActive == maxActive with waitCount > 0 is the smoking gun. If waitCount is zero, the pool is full but requests are not yet blocking. If numActive is below maxActive, the problem is elsewhere, likely the worker thread pool or the database itself.

2. Take a thread dump immediately. The stack traces tell you whether threads are blocked in getConnection() or in query execution. A large cluster of threads in org.apache.tomcat.jdbc.pool.ConnectionPool waiting on a connection points to the pool. Threads deeper in JDBC, in a socket read, or in a driver call point to the database.

3. Distinguish leak from slow backend. Stop or shed load briefly and watch numActive. If it drops back to near baseline within seconds, connections were held by legitimate work. If it stays pinned at maxActive after traffic stops, you have a leak: connections were borrowed and never returned.

4. If a leak is suspected, find the borrower. Enable logAbandoned=true (with removeAbandoned=true and removeAbandonedTimeout=60) and reproduce. Tomcat JDBC will log the stack trace of each abandoned connection when it is reclaimed. That stack trace is the code path that failed to call close().

5. If a slow backend is suspected, measure it. Check the database’s own process list or slow query log during the incident. Correlate query latency with the rise in numActive. A query that went from 5ms to 500ms will hold each connection 100 times longer and exhaust the pool at a hundredth of the previous traffic.

6. Check the configuration last, not first. Raising maxActive feels productive but is usually wrong. If the pool is exhausted because of a leak, more connections just delay the failure and leak more. If the database is overloaded, more connections make it worse. Fix the cause before resizing the pool.

Metrics and signals to monitor

Track peak numActive / maxActive during the busiest hour. A healthy deployment stays below 0.50 at peak, leaving room for slow query spikes and traffic increases. maxActive should be sized against what the database can handle multiplied by the number of Tomcat instances pointing at it, not against what the application would like.

Fixes

Connection leak

A leak is a code bug. Connections are borrowed and never returned, typically because an exception path skips the close() call or a transaction is left dangling.

Immediate mitigation. Set removeAbandoned=true with removeAbandonedTimeout=60 and logAbandoned=true. The pool will forcibly reclaim connections held longer than the timeout and log the stack trace of the borrower. This unblocks the pool within a minute without a restart.

Real fix. Find the leaking code path from the logAbandoned traces and switch it to try-with-resources or an equivalent try-finally that guarantees close(). Check ORM session management too: an unclosed Hibernate session can hold a JDBC connection just as effectively as a raw getConnection() leak.

Tradeoffs. removeAbandoned can close a legitimately long transaction if it exceeds the timeout. Pair it with the ResetAbandonedTimer interceptor so the timer resets on query activity, which prevents reclaiming connections that are slow but genuinely in use. You can also set abandonWhenPercentageFull so abandoned reclaim only kicks in when the pool is under real pressure, reducing false positives during low load.

Slow or overloaded database

When the pool is full because queries take too long, the fix is on the database side, not the pool side.

Query tuning. Check the slow query log, missing indexes, and lock contention. A single query that regressed from 5ms to 200ms can exhaust a pool of 100 connections at modest traffic.

Query timeouts. Set a JDBC statement timeout (Statement.setQueryTimeout or a JDBC URL parameter) so a runaway query fails fast instead of holding a connection indefinitely. Without it, the default is infinite.

Tradeoffs. Timeouts surface as errors to users, but they also free the connection and the worker thread. A request that fails in 5 seconds is better than a request that hangs until maxWait and then fails anyway. If slow queries are intermittent, a circuit breaker or bulkhead pattern upstream is more effective than tuning the pool.

Undersized pool

If peak numActive / maxActive routinely exceeds 0.80 at normal load and the database has headroom, raise maxActive.

Size against the database, not the wish list. maxActive times the number of Tomcat instances must not exceed what the database server can serve concurrently. A database that tops out at 200 concurrent connections cannot safely serve four Tomcat instances each configured with maxActive=100.

Tradeoffs. Too high and you overload the database, which then slows every query, which holds connections longer, which exhausts the pool faster. This is the same positive feedback loop as the thread pool death spiral, one layer down.

Stale or dead connections

After a database failover or a network blip, connections in the pool may be dead at the TCP level but unknown to the pool.

Validation on borrow. Set testOnBorrow=true with an appropriate validationQuery (SELECT 1 for most databases). The pool validates each connection before handing it out and discards bad ones.

Avoid the validation storm. validationInterval defaults to 30000ms, meaning a connection validated recently is not re-validated on every borrow. Verify it has not been set to 0, which forces validation on every checkout and makes validation latency dominate checkout time under load.

Tradeoffs. testOnBorrow adds a round trip per checkout. For most workloads the cost is acceptable because the alternative is serving errors until stale connections cycle out. validationInterval keeps the cost bounded.

Prevention

• Always use try-with-resources or try-finally for connections. A missing close() on an exception path is the single most common cause of leaks.
• Set a statement or query timeout on every query. The JDBC default is infinite, and an infinite timeout means one slow query can hold a connection forever.
• Monitor numActive / maxActive and waitCount continuously. Trending the ratio over days catches the slow drift toward exhaustion before waitCount ever goes non-zero.
• Enable StuckThreadDetectionValve so threads blocked in getConnection() show up as stuck threads, not just as busy threads.
• Decide the pool implementation deliberately. tomcat-jdbc and HikariCP expose metrics under different bean names. Confirm which one your app uses before you build dashboards.
• Size maxActive against database capacity, then leave headroom. Peak ratio below 0.50 is the target.
• Reproduce leaks in staging with logAbandoned on. Catching the borrower stack trace in CI is far cheaper than catching it at 3 a.m.

How Netdata helps

Netdata turns the double cliff into a single correlated view rather than two separate incidents.

• Per-second collection of numActive, numIdle, maxActive, and waitCount shows the exact moment the pool saturates and whether waitCount follows, which is the leading indicator that worker threads are about to block.
• Correlating the JDBC pool panel with currentThreadsBusy / maxThreads makes the cascade visible. When the two ratios rise together, the database pool is the root cause. When threads climb but the pool is healthy, look at the worker pool or a different backend.
• ML anomaly detection on the pool ratio catches the slow drift toward exhaustion days before waitCount goes non-zero.
• Pairing the pool view with JVM GC pause time and CPU distinguishes a database-driven stall from a GC-driven stall, since both manifest as rising thread counts but need opposite fixes.
• Database-side latency alongside the pool counters closes the loop: if query latency rises before numActive, the backend is the cause; if numActive rises with no latency change, the pool itself is the constraint.

Related guides

• Tomcat accept queue overflow: acceptCount, somaxconn, and Recv-Q
• Tomcat java.net.BindException: Address already in use: the connector never starts
• Tomcat average latency lies: why you need p95/p99 from the access log
• Tomcat classloader leak on redeploy: why the old WebappClassLoader never dies
• Tomcat connection refused: maxConnections and acceptCount both exhausted
• Tomcat accepts connections but never responds: the TCP-connect trap
• Tomcat frequent Full GC: pause time, G1, and the 5% overhead rule
• Tomcat GC death spiral: full GCs dominating and throughput collapsing
• Tomcat heap dump before restart: capturing evidence with jmap and jstack
• Tomcat heap usage: watch the post-GC baseline, not the sawtooth peak
• How Tomcat actually works in production: a mental model for operators
• Tomcat HTTP Status 503 Service Unavailable: the connector is out of threads