$ guides / tomcat / tomcat-jdbc-connection-leak ▌

Operations Guides

Tomcat JDBC connection leak: numActive stuck at max even when idle

The signature of a Tomcat JDBC connection leak is specific: numActive sits at maxActive and refuses to fall, even when request rate is near zero. New requests that need a database connection block on getConnection() until maxWait elapses, then throw a pool exhaustion error. The database reports a flock of idle Sleep connections from Tomcat that never close.

The cascade is what takes the service down. Each request blocking on getConnection() also holds an HTTP worker thread. As leaked connections accumulate, more threads stall on the pool, the thread pool fills, and requests queue in the acceptor. The JVM is healthy, the database is healthy, CPU is low, and the service is effectively down.

The smoking gun is the combination: numActive == maxActive sustained at low request rates plus idle Sleep connections on the database side. If you see both, you have a connection leak, not a slow query problem.

What this means

A connection leak happens when application code borrows a connection from the pool but never returns it. The pool marks the connection active and never recycles it. The database sees the connection as open but idle. Over time every connection leaks and the pool is permanently exhausted.

The Tomcat JDBC pool (org.apache.tomcat.jdbc.pool) exposes four attributes that tell the whole story:

Attribute	Meaning
`numActive`	Connections currently borrowed by application code
`numIdle`	Connections in the pool, available for borrowing
`maxActive`	Hard cap on total connections the pool will open
`waitCount`	Threads currently blocked waiting to borrow a connection

In a healthy pool, numActive tracks request rate: it rises under load and falls back toward numIdle when traffic drops. In a leak, numActive climbs to maxActive and stays there regardless of load. waitCount starts climbing as new requests queue behind the leaked connections.

flowchart TD
    A["Error path skips connection.close"] --> B["numActive pins at maxActive"]
    B --> C["getConnection blocks up to maxWait"]
    C --> D["HTTP worker thread held waiting"]
    D --> E["Thread pool exhausts"]
    E --> F["Requests queue, then 503"]
    C --> G["Pool exhaustion error"]
    G --> F
    H["DB shows idle SLEEP connections from Tomcat"] -. confirms .-> B

The database confirms the diagnosis from the other side: connections from Tomcat’s IP that are idle (in MySQL, command = 'Sleep'). If those connections were executing long queries, you would have a slow-backend problem. Idle connections mean Tomcat borrowed them and forgot to return them.

Common causes

Cause	What it looks like	First thing to check
Missing `close()` in an error path	`numActive` climbs slowly, one connection at a time, often correlated with a specific exception type in logs	grep application logs for the exception that triggers the leaky path
No `try-with-resources` on `getConnection()`	Multiple code paths can leak; harder to pin to one trigger	Code review of every `DataSource.getConnection()` call site
Unclosed ORM session (Hibernate)	`numActive` climbs even though app code never calls `getConnection()` directly	Check session and transaction management configuration, including `OpenSessionInViewFilter`
Transaction timeout leaves dangling transaction	Connections stuck after timeout-driven rollbacks that skip cleanup	Check transaction manager timeout settings versus pool reclaim settings

Quick checks

All read-only and safe to run during an incident.

# Check pool state via JMX (Tomcat JDBC pool)
java -jar jmxterm.jar -l localhost:9090 -n -v silent -e \
  "get -b tomcat.jdbc:type=ConnectionPool,name=\"jdbc/mydb\" numActive numIdle size maxActive waitCount"

# Check thread pool correlation: are HTTP threads stuck waiting?
java -jar jmxterm.jar -l localhost:9090 -n -v silent -e \
  "get -b Catalina:type=ThreadPool,name=\"http-nio-8080\" currentThreadsBusy maxThreads"

# Thread dump: find HTTP threads parked waiting for pool checkout
jstack $(pgrep -f 'catalina.startup.Bootstrap') | grep -A 5 "tomcat.jdbc.pool\|getConnection"

-- MySQL: count idle (Sleep) connections from Tomcat's host
SELECT host, COUNT(*) FROM information_schema.processlist
  WHERE command = 'Sleep' GROUP BY host;

Diagnostic logic:

numActive == maxActive and waitCount > 0: pool exhausted. Now determine if it is a leak or a slow-backend problem.
Database shows Sleep connections: leak. Connections are borrowed but unused.
Database shows connections running long queries: not a leak. Slow-query or lock-contention problem exhausting the pool legitimately.
Thread dump shows HTTP threads in getConnection or pool wait: confirms the cascade. Those threads hold HTTP worker slots while waiting for a DB connection that will never arrive.

How to diagnose it

Confirm the leak. Verify numActive == maxActive at low request rates. If numActive tracks traffic (drops when traffic drops), you do not have a leak; you have a capacity or slow-backend problem.
Confirm from the database side. Check for idle connections from Tomcat that persist:
```
SELECT user, host, db, command, time, state
  FROM information_schema.processlist
  WHERE command = 'Sleep' ORDER BY time DESC;
```
Connections sleeping for a long time and never closing are leaked.

Enable logAbandoned to capture the borrowing stack trace. This is how you find the exact code path that leaked. Configure the pool Resource:

<Resource name="jdbc/mydb" auth="Container"
          type="javax.sql.DataSource"
          factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
          maxActive="100"
          removeAbandoned="true"
          removeAbandonedTimeout="60"
          logAbandoned="true"/>

When the pool reclaims an abandoned connection, it logs the stack trace of the code that originally borrowed it. That trace points directly at the leaky call site.

Set removeAbandoned for immediate recovery. With removeAbandoned="true" and removeAbandonedTimeout="60", the cleaner thread forcibly reclaims connections held longer than 60 seconds. The default abandonWhenPercentageFull=0 means reclamation is eligible immediately, not gated on pool fill level.
Restart to apply the config and clear the current leak. A restart drops all borrowed connections immediately. If you have already set removeAbandoned, the restart also activates the new configuration going forward.
Read the logAbandoned output. The stack trace tells you which method borrowed the connection and never returned it. Fix that code path.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`numActive / maxActive`	Core utilization ratio for the pool	Sustained at 1.0 at low request rates
`waitCount`	Threads actively blocked on `getConnection()`	Any value greater than 0 sustained for more than a few seconds
`numIdle`	Headroom in the pool	Drops to 0 while `numActive == maxActive`
Thread pool `currentThreadsBusy`	Cascade indicator	Rising in lockstep with `waitCount`
Database idle connections from Tomcat	Leak confirmation from the DB side	Count climbing and never dropping
`maxTime` (longest single request)	Requests blocked on pool acquire inflate this	Spikes to minutes

Fixes

Immediate triage: removeAbandoned and logAbandoned

Set removeAbandoned="true", removeAbandonedTimeout="60", and logAbandoned="true" on the pool Resource. This reclaims leaked connections so the pool recovers without a restart, and logs the borrowing stack trace so you can find the code bug.

Tradeoffs to understand:

logAbandoned overhead. It generates a stack trace on every connection borrow, adding CPU cost per checkout. Disable it once the leak is fixed.
Timeout sizing. removeAbandoned can reclaim connections legitimately held by long-running queries if the timeout is set too low. Size it to your longest legitimate query. The ResetAbandonedTimer interceptor resets the timer on each database operation, protecting genuinely long-running queries that are actively executing.
Safety net, not a fix. removeAbandoned reclaims the connection object, but the underlying database-side resources (open statements, result sets) may still need cleanup. The real fix is in the code.

Code fix: try-with-resources

The real fix is ensuring every getConnection() call is wrapped in try-with-resources or an equivalent try-finally that guarantees close() on every path:

try (Connection conn = dataSource.getConnection()) {
    // use connection
}

This returns the connection even if an exception is thrown. Every manual close() inside a finally block should be audited: the error path that skips it is the one that leaks.

ORM session management

If the application uses Hibernate or JPA, the leak may be at the session level rather than the raw JDBC level. Unclosed sessions hold connections. Check:

OpenSessionInViewFilter configuration in web applications. If a view-tier exception skips session cleanup, the connection leaks.
Transaction boundaries that do not match session boundaries.
@Transactional methods that catch exceptions internally without re-throwing, preventing the transaction manager from rolling back and releasing the connection.

HikariCP note

If you are using HikariCP (the default in Spring Boot since 2.0), the JMX attribute names differ: ActiveConnections, IdleConnections, TotalConnections, ThreadsAwaitingConnection. HikariCP has no removeAbandoned; it has its own leak detection mechanism. The diagnostic logic (active pinned at max, idle connections on the DB side) is the same regardless of pool implementation.

Prevention

Adopt try-with-resources as a hard rule. Every getConnection() call must be inside a try-with-resources block so the connection is closed on every code path, including exception paths.
Keep removeAbandoned enabled as a safety net. Set the timeout larger than your longest legitimate query. It will not prevent leaks, but it will stop a leak from taking down the service while you fix the code.
Monitor numActive / maxActive continuously. A slow climb over hours or days is the early warning. The leak is detectable long before numActive reaches maxActive.
Set maxWait to a finite value. The default of 30000ms ensures requests fail fast with a pool exhaustion error rather than blocking indefinitely. If maxWait is set to -1, leaked connections cause threads to hang forever instead of failing loudly.
Run leak detection in CI. Integration tests that exercise error paths and then assert numActive == 0 after cleanup can catch leaks before they reach production.

How Netdata helps

Pool utilization at per-second resolution. Netdata’s JMX collector surfaces numActive, numIdle, maxActive, and waitCount with one-second granularity. A slow leak that takes hours to reach maxActive shows up as a gradual upward trend, not a sudden cliff.
Cascade correlation. When numActive pins at maxActive, Netdata overlays JDBC pool metrics with thread pool utilization, request latency, and error rate in the same view. The cascade from pool exhaustion to thread starvation to 503s reads as a single timeline.
Anomaly detection on pool metrics. Netdata’s ML flags unusual numActive behavior, such as climbing when request rate is dropping, before it crosses the exhaustion threshold.
Ruling out confounders. Overlaying GC pause time and CPU utilization confirms the stall is pool-driven rather than GC-driven, avoiding a wrong root-cause chase.