Tomcat classloader leak on redeploy: why the old WebappClassLoader never dies

Every hot redeploy leaves the previous WebappClassLoader in memory. After enough redeploys the JVM hits OutOfMemoryError: Metaspace, or if -XX:MaxMetaspaceSize is unset, the OS OOM-kills the process with no JVM error at all. Heap looks flat. GC looks fine. RSS climbs until the process dies.

Each Tomcat webapp gets its own WebappClassLoader for isolation. On undeploy, the classloader and every class it loaded should be collected. If anything still holds a reference to a class loaded by the webapp, a ThreadLocal value, a JDBC driver registered with DriverManager, a Timer thread that was never cancelled, a log appender, or a static field in a shared library, the classloader is pinned. All its classes stay in Metaspace forever.

flowchart TD
    A[Hot redeploy] --> B[Old WebappClassLoader marked for GC]
    B --> C{Pinned by live reference?}
    C -- No --> D[Classes collected, Metaspace reclaims]
    C -- Yes --> E[Classloader retained]
    E --> F[All loaded classes retained in Metaspace]
    F --> G[Next redeploy adds another full copy]
    G --> H[OutOfMemoryError: Metaspace]

Two things make this hard to catch:

Metaspace is not heap. Standard heap dashboards show nothing.
Without -XX:MaxMetaspaceSize, there is no JVM-level alert. Metaspace grows until the kernel kills the process. No OutOfMemoryError, just a dead JVM and a kernel OOM log entry.

Common causes

Cause	What it looks like	First thing to check
`ThreadLocal` not cleared	Tomcat logs a SEVERE ThreadLocal leak warning at undeploy	`catalina.out` for `checkThreadLocalMapForLeaks` warnings
JDBC driver not deregistered	Driver registered by the app survives undeploy	Whether the app calls `DriverManager.deregisterDriver` in a `ServletContextListener`
`Timer` / `ScheduledExecutorService` not cancelled	Warning in `catalina.out`: “appears to have started a thread … but has failed to stop it”	Thread dump for threads named after the app or its timer
Log appender holding classloader	Logging framework retains references to webapp classes	Appender lifecycle in the logging config
JMX MBean not unregistered	MBeans accumulate across redeploys	`jconsole` or MBean server query for the app’s MBeans after undeploy
Static field in shared library	Library loaded by common classloader references an app class	Heap histogram for retained classes from the old webapp

Quick checks

# Check whether the JVM is leaking Metaspace (sample every 5s, watch the M column)
jstat -gcutil $(pgrep -f 'catalina.startup.Bootstrap') 5000

# Detailed Metaspace breakdown from jcmd
jcmd $(pgrep -f 'catalina.startup.Bootstrap') VM.metaspace

# Loaded class count via JMX (should plateau, not grow per redeploy)
java -jar jmxterm.jar -l localhost:9090 -n -v silent -e \
  "get -b java.lang:type=ClassLoading LoadedClassCount UnloadedClassCount"

# Tomcat's leak-detection warnings from the most recent undeploy
grep -iE "memory leak|failed to stop|thread" $CATALINA_BASE/logs/catalina.out | tail -50

# Ask the Manager app which stopped webapps still have live classloaders
# WARNING: triggers System.gc(); expect a pause on a loaded JVM
curl -s -u $MANAGER_USER:$MANAGER_PASS http://localhost:8080/manager/text/findleaks

# Confirm the process RSS is growing, not just heap
ps -p $(pgrep -f 'catalina.startup.Bootstrap') -o rss,vsz,etime

# Check whether MaxMetaspaceSize is set
ps -ef | grep '[c]atalina.startup.Bootstrap' | grep -o -- '-XX:MaxMetaspaceSize=[^ ]*'

How to diagnose it

Confirm the leak. Capture Metaspace usage before and after a clean undeploy/redeploy cycle. If Metaspace does not return to roughly its pre-undeploy value, the classloader is pinned.
Set -XX:MaxMetaspaceSize. This converts a silent OS OOM kill into a JVM OutOfMemoryError: Metaspace you can catch and alert on. Use a value comfortably larger than the steady-state size of one deployed app.
Run Tomcat’s leak detection. Tomcat 8.5+ logs memory leak warnings at undeploy, including ThreadLocal detection. Grep catalina.out for these. The message tells you which class of leak it is.
Use the Manager “Find Leaks” command. It calls System.gc() and reports which stopped webapps still have live classloaders. It only catches direct leaks; indirect leaks through non-webapp ThreadLocal values may not surface.
Take a heap dump and analyze it. When Tomcat’s detection does not pin down the cause, trigger a dump with jmap -dump:live,format=b,file=/tmp/heap.hprof <pid>. The :live option triggers a full GC, so expect a pause on production heaps. Open the dump in Eclipse MAT, run “Leak Suspects,” and look for WebappClassLoader instances that should already be gone. MAT’s “Path to GC Roots” for the classloader shows exactly what is pinning it.
Inspect thread dumps for leaked threads. Run jstack <pid> and look for threads whose names reference the old webapp, or whose context classloader is the old WebappClassLoader. These are usually timer threads, scheduled executors, or connection-cleanup threads spawned by libraries.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Metaspace usage (`java.lang:type=MemoryPool,name=Metaspace`)	Class metadata lives here; pinned classloaders keep classes here permanently	Step increase after each redeploy that never reverts
`LoadedClassCount` (`java.lang:type=ClassLoading`)	Counts classes currently loaded; a leak keeps old classes loaded	Monotonic growth across undeploy/redeploy cycles
Process RSS	Total memory including non-heap; grows when Metaspace grows	RSS climbing while heap stays flat
Deploy/undeploy events in `catalina.out` (`deployWAR`, etc.)	Correlates Metaspace steps to deploy events	Metaspace jump coincident with each event
Tomcat leak-detection warnings in `catalina.out`	Tomcat reports many leak types at undeploy	Any SEVERE entry mentioning ThreadLocal, threads, or JDBC
`OutOfMemoryError: Metaspace` in logs	Terminal signal; only fires if `MaxMetaspaceSize` is set	A single occurrence means the JVM is already compromised

Fixes

Switch to immutable-container deploys

The only fully reliable fix is to stop hot-redeploying. Build an image per deploy, stop the old container, start the new one. Each JVM starts with a clean classloader hierarchy. This eliminates the leak class entirely.

Fix the leaking reference in the application

When you must hot redeploy, remove whatever the application leaves behind. In a ServletContextListener:

ThreadLocals: clear them in contextDestroyed. Tomcat’s detection logs which threads have them; use that as your hit list.
JDBC drivers: iterate DriverManager.getDrivers() and call DriverManager.deregisterDriver for drivers loaded by the webapp classloader. Tomcat 8.5+ does this for you via clearReferencesJdbc , but verify it is enabled.
Timer and ScheduledExecutorService: cancel them in contextDestroyed. Set clearReferencesStopTimerThreads="true" on the Context to have Tomcat forcibly stop java.util.Timer threads the app left running .
Log appenders: ensure the logging framework releases appenders on undeploy. For log4j and logback this typically means attaching the appender lifecycle to the ServletContext.
JMX MBeans: unregister every MBean the app registered.
Threads spawned by libraries: clearReferencesStopThreads="true" forcibly stops them, but this is unsafe and can corrupt application state. Tomcat leaves it off by default . The safer fix is making the library’s threads daemon threads or shutting them down cleanly via the library’s own shutdown API.

Let Tomcat detect and mitigate

Ensure the JreMemoryLeakPreventionListener is configured. It pre-initializes known JRE singletons at Tomcat startup so they cannot capture the webapp classloader later. Treat the SEVERE warnings Tomcat emits at undeploy as a continuous audit signal. Note that some JreMemoryLeakPreventionListener properties were removed in later Tomcat versions as the underlying JDK bugs were fixed .

As a stopgap when the leak cannot be fixed immediately, schedule periodic full JVM restarts based on Metaspace growth. This is duct tape, not a fix.

Prevention

Set -XX:MaxMetaspaceSize on every Tomcat instance that may be hot-redeployed. The default unbounded Metaspace produces a silent OOM kill. A JVM OutOfMemoryError is always preferable to a kernel OOM kill.
Monitor Metaspace and LoadedClassCount across deploy cycles. The signature is a step function: each redeploy adds the same amount and never gives it back.
Prefer immutable-container deploys when the platform allows.
Treat Tomcat’s undeploy warnings as failures. Any SEVERE log entry at undeploy is a leak to fix.
Audit ServletContextListener implementations in every app that hot-redeploys. If contextDestroyed does not clean up what contextInitialized created, the app leaks.

How Netdata helps

Per-second Metaspace metrics from java.lang:type=MemoryPool,name=Metaspace show the step increase after each redeploy and confirm whether the old classloader was reclaimed.
LoadedClassCount tracking reveals monotonic class accumulation across deploy cycles.
Process RSS alongside JVM heap makes the non-heap nature of the leak obvious: RSS climbs while heap stays flat.
Correlation with deploy events lets you line up the Metaspace step with the deploy log entry.
ML anomaly detection on Metaspace growth flags the redeploy that breaks the previous pattern, even when no static threshold has been crossed.
JMX integration collects these signals without custom instrumentation or scripting.