Tomcat 'appears to have started a thread but has failed to stop it': undeploy leak warnings

The exact log line is the symptom. When you undeploy or redeploy a Tomcat webapp and see this in catalina.out:

SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
The web application [myapp] appears to have started a thread named [AbandonedConnectionCleanupThread]
but has failed to stop it. This is very likely to create a memory leak.

that is not generic noise. During Context stop, WebappClassLoaderBase.clearReferencesThreads() walked every live thread in the JVM, found at least one whose context classloader is still the webapp’s WebappClassLoader, and named it. The bracketed thread name is the breadcrumb back to the library or application code that started it.

Hot redeploy repeatedly with this uncorrected and each redeploy pins the previous WebappClassLoader (and every class it loaded) in Metaspace. After enough redeploys you hit OutOfMemoryError: Metaspace, or if -XX:MaxMetaspaceSize is unset, the OS OOM killer ends the process with no JVM-level error.

What this means

Each web application (Context) in Tomcat gets its own WebappClassLoader. On undeploy, that classloader and all its loaded classes should be garbage collected. That GC only happens if nothing still references the classloader.

Threads are the most common anchor. A thread’s contextClassLoader is set to the webapp classloader when the thread is created inside that webapp. If the thread is still alive at undeploy time, it pins the classloader, which pins every class the classloader loaded, which pins that slab of Metaspace. ThreadLocals, JDBC DriverManager registrations, shutdown hooks, and static references in shared libraries do the same thing.

Tomcat’s defense is detection, not prevention. clearReferencesThreads() enumerates every thread in the JVM and logs the warning for any thread whose context classloader matches the webapp being stopped. The companion ThreadLocalLeakPreventionListener renews threads in Tomcat’s own Executor pools when renewThreadsWhenStoppingContext="true", but neither mechanism can reach into application or library code to actually terminate a foreign thread.

A warning does not guarantee a leak. The flagged thread may be a daemon that exits on its own shortly after, or the classloader may still be collectable if the thread dies before the next GC cycle. Whether it actually leaks is answered by watching Metaspace across the next few redeploys.

flowchart TD
    A["Webapp undeploy / redeploy"] --> B["clearReferencesThreads scans JVM threads"]
    B --> C{"Thread context classloader
== WebappClassLoader?"}
    C -- yes --> D["Log warning naming the thread"]
    C -- no --> E["No warning for that thread"]
    D --> F["Thread stays alive, pins classloader"]
    F --> G["All loaded classes pinned in Metaspace"]
    G --> H["Metaspace grows one app worth per redeploy"]
    H --> I["OutOfMemoryError: Metaspace
or OS OOM kill"]

Common causes

Quick checks

# Find every occurrence of the warning and extract the thread name
grep -oE 'appears to have started a thread named \[[^]]+\]' \
  $CATALINA_BASE/logs/catalina.out | sort | uniq -c | sort -rn

# Confirm whether the warning fires on undeploy/redeploy or only on full shutdown
grep -B2 'appears to have started a thread' \
  $CATALINA_BASE/logs/catalina.out | grep -E 'undeploy|redeploy|Stopping|destroy'

# Check Metaspace usage via jcmd (read-only)
jcmd $(pgrep -f 'catalina.startup.Bootstrap') VM.metaspace

# Check loaded class count (should not grow across redeploys)
java -jar jmxterm.jar -l localhost:9090 -n -v silent -e \
  "get -b java.lang:type=ClassLoading LoadedClassCount TotalLoadedClassCount UnloadedClassCount"

# Capture a thread dump to locate the named thread and see what it is doing
jstack $(pgrep -f 'catalina.startup.Bootstrap') > /tmp/tomcat-threads.txt
grep -A 30 'AbandonedConnectionCleanupThread' /tmp/tomcat-threads.txt

# Use the Manager app's leak detector after an undeploy (calls System.gc, may pause)
curl -s -u $USER:$PASS 'http://localhost:8080/manager/text/findleaks'

The findleaks endpoint calls System.gc() and reports any webapp whose classloader failed to be collected after stop. A non-empty list confirms the leak is real, not just a warning.

How to diagnose it

1. Confirm the trigger is undeploy or redeploy, not full JVM shutdown. If the warning only appears when you stop the entire Tomcat process, it is harmless. The JVM is terminating and the leaked reference does not matter. The dangerous case is a warning during a redeploy while the JVM keeps running.

2. Extract the thread name verbatim from the warning. The bracketed name (AbandonedConnectionCleanupThread, Timer-2, lettuce-timer-1) is the most important piece of evidence. Different libraries use recognizable naming patterns.

3. Capture a thread dump and locate the thread by name. jstack $(pgrep -f 'catalina.startup.Bootstrap') shows every live thread, its state, and its stack. The stack tells you which class started it. For MySQL Connector/J, the stack runs through the connector’s AbandonedConnectionCleanupThread.run. For Quartz, through org.quartz.simpl.SimpleThreadPool. For Lettuce, through io.netty.util.HashedWheelTimer.

4. Measure Metaspace before and after a redeploy. If Metaspace jumps by roughly the same amount on each redeploy and never drops back, the classloader is pinned and the warning is describing a real leak.

5. Run the Manager findleaks check after an undeploy. If it lists the webapp, the classloader was not collected. That confirms the warning translated into an actual leak.

6. Correlate with LoadedClassCount. Pull java.lang:type=ClassLoading before and after each redeploy cycle. A monotonically rising LoadedClassCount after undeploy is the leading indicator of classloader accumulation.

7. Check whether MaxMetaspaceSize is set. Without it, Metaspace grows silently until the OS OOM killer ends the process. There is no JVM-level OutOfMemoryError to alert you. Verify with jcmd $(pgrep -f 'catalina.startup.Bootstrap') VM.flags | grep -i metaspace.

Metrics and signals to monitor

Fixes

MySQL Connector/J AbandonedConnectionCleanupThread

This is the single most common offender. The thread starts on the first JDBC request and holds a reference to the webapp classloader that loaded the connector. Three fixes, in order of preference:

1. Move the connector JAR to $CATALINA_HOME/lib. It is then loaded by the common classloader, not the webapp classloader. The thread still exists, but its context classloader is the common one, so it no longer pins the webapp. Simplest fix and applies to any JDBC driver with the same pattern.

2. Register a ServletContextListener that calls the shutdown method on undeploy. In contextDestroyed():

   com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.checkedShutdown();
   

3. Pre-initialize the offending class via JreMemoryLeakPreventionListener. In server.xml, add classesToInitialize="com.mysql.jdbc.NonRegisteringDriver" to the listener element. This forces the class to load under the common classloader at Tomcat startup.

Other library-managed threads

Quartz, RxJava, Lettuce/Netty, async-http-client, Atomikos, and Guice each have their own shutdown API. The diagnostic is the same: read the thread name from the warning, find it in the thread dump, identify the owning library, then call its shutdown or close method from a ServletContextListener. There is no general fix. Each library must be cleaned up explicitly.

For Timer and ScheduledExecutorService threads that the application created directly: cancel them in contextDestroyed(). Tomcat’s clearReferencesStopTimerThreads="true" Context attribute will forcibly stop TimerThread instances, but only enable this after confirming no other webapp shares the same Timer. Forcing a shared Timer to stop can break unrelated apps.

clearReferencesStopThreads: do not enable

The Apache documentation explicitly warns that clearReferencesStopThreads="true" is unsafe. Forcibly stopping application threads can leave locks held, files and sockets open, and shared state corrupted. The flag exists for diagnostic scenarios where you accept the consequences. It is not a production fix. Prefer the explicit library shutdown path above.

Full JVM restart as the reliable fallback

If you cannot identify or fix the source in time, the safe operational response is to stop hot redeploying and use full JVM restarts for deployments. A restart reclaims all Metaspace. Many teams run long-lived Tomcats in production with the discipline that every deploy is a full process restart, and reserve hot redeploy for development. This sidesteps the entire class of leak.

Prevention

• Set -XX:MaxMetaspaceSize. Without it, a classloader leak has no JVM-level safety net; the OS OOM killer ends the process. Pick a value with comfortable headroom over steady-state usage.
• Prefer full JVM restart over hot redeploy in production. This is the only fully reliable prevention. Containers and immutable deploys make this the default.
• Move shared JDBC drivers and other libraries that start background threads into $CATALINA_HOME/lib. Anything loaded by the common classloader cannot pin a webapp classloader.
• Audit every ServletContextListener for cleanup. Any library that starts a thread, registers a driver, or registers an MBean must have a matching unregister in contextDestroyed().
• Track Metaspace per deploy as a first-class metric. The leak is invisible until you graph Metaspace across redeploys. A step plot that grows monotonically is the signature.

How Netdata helps

• Per-second JVM memory pool metrics, including Metaspace, capture the exact step at each redeploy rather than a coarse poll interval that may miss the jump.
• LoadedClassCount from java.lang:type=ClassLoading is collected continuously, so monotonic growth across undeploys is visible as a trend rather than a one-off snapshot.
• Correlation between deploy events (log parsing) and the Metaspace step turns the warning into a confirmed leak without manual timestamp matching.
• JVM thread count tracking surfaces thread accumulation across redeploys, which often accompanies classloader leaks from the same source.
• Anomaly detection on Metaspace and LoadedClassCount flags the deviation from baseline even without an explicit threshold.

Related guides

• Tomcat java.net.BindException: Address already in use: the connector never starts
• Tomcat accepts connections but never responds: the TCP-connect trap
• Tomcat frequent Full GC: pause time, G1, and the 5% overhead rule
• Tomcat GC death spiral: full GCs dominating and throughput collapsing
• Tomcat heap usage: watch the post-GC baseline, not the sawtooth peak
• How Tomcat actually works in production: a mental model for operators
• Tomcat HTTP Status 503 Service Unavailable: the connector is out of threads
• Tomcat process not running: crashes, OOM-kills, and failed restarts
• Tomcat maxThreads and minSpareThreads: sizing the executor correctly
• Tomcat monitoring checklist: the signals every production instance needs
• Tomcat monitoring maturity model: from survival to expert
• Tomcat OutOfMemoryError: GC overhead limit exceeded: GC running but freeing nothing