Tomcat classloader leak on redeploy: why the old WebappClassLoader never dies

Every hot redeploy leaves the previous WebappClassLoader in memory. After enough redeploys the JVM hits OutOfMemoryError: Metaspace, or if -XX:MaxMetaspaceSize is unset, the OS OOM-kills the process with no JVM error at all. Heap looks flat. GC looks fine. RSS climbs until the process dies.

Each Tomcat webapp gets its own WebappClassLoader for isolation. On undeploy, the classloader and every class it loaded should be collected. If anything still holds a reference to a class loaded by the webapp, a ThreadLocal value, a JDBC driver registered with DriverManager, a Timer thread that was never cancelled, a log appender, or a static field in a shared library, the classloader is pinned. All its classes stay in Metaspace forever.

flowchart TD
    A[Hot redeploy] --> B[Old WebappClassLoader marked for GC]
    B --> C{Pinned by live reference?}
    C -- No --> D[Classes collected, Metaspace reclaims]
    C -- Yes --> E[Classloader retained]
    E --> F[All loaded classes retained in Metaspace]
    F --> G[Next redeploy adds another full copy]
    G --> H[OutOfMemoryError: Metaspace]

Two things make this hard to catch:

1. Metaspace is not heap. Standard heap dashboards show nothing.
2. Without -XX:MaxMetaspaceSize, there is no JVM-level alert. Metaspace grows until the kernel kills the process. No OutOfMemoryError, just a dead JVM and a kernel OOM log entry.

Common causes

Quick checks

# Check whether the JVM is leaking Metaspace (sample every 5s, watch the M column)
jstat -gcutil $(pgrep -f 'catalina.startup.Bootstrap') 5000

# Detailed Metaspace breakdown from jcmd
jcmd $(pgrep -f 'catalina.startup.Bootstrap') VM.metaspace

# Loaded class count via JMX (should plateau, not grow per redeploy)
echo 'get -b java.lang:type=ClassLoading LoadedClassCount UnloadedClassCount' | \
  java -jar jmxterm.jar -l localhost:9090 -n -v silent

# Tomcat's leak-detection warnings from the most recent undeploy
grep -iE "memory leak|failed to stop|thread" $CATALINA_BASE/logs/catalina.out | tail -50

# Ask the Manager app which stopped webapps still have live classloaders
# WARNING: triggers System.gc(); expect a pause on a loaded JVM
curl -s -u $MANAGER_USER:$MANAGER_PASS http://localhost:8080/manager/text/findleaks

# Confirm the process RSS is growing, not just heap
ps -p $(pgrep -f 'catalina.startup.Bootstrap') -o rss,vsz,etime

# Check whether MaxMetaspaceSize is set
ps -ef | grep '[c]atalina.startup.Bootstrap' | grep -o -- '-XX:MaxMetaspaceSize=[^ ]*'

How to diagnose it

1. Confirm the leak. Capture Metaspace usage before and after a clean undeploy/redeploy cycle. If Metaspace does not return to roughly its pre-undeploy value, the classloader is pinned.
2. Set -XX:MaxMetaspaceSize. This converts a silent OS OOM kill into a JVM OutOfMemoryError: Metaspace you can catch and alert on. Use a value comfortably larger than the steady-state size of one deployed app.
3. Run Tomcat’s leak detection. Tomcat 8.5+ logs memory leak warnings at undeploy, including ThreadLocal detection. Grep catalina.out for these. The message tells you which class of leak it is.
4. Use the Manager “Find Leaks” command. It calls System.gc() and reports which stopped webapps still have live classloaders. It only catches direct leaks; indirect leaks through non-webapp ThreadLocal values may not surface.
5. Take a heap dump and analyze it. When Tomcat’s detection does not pin down the cause, trigger a dump with jmap -dump:live,format=b,file=/tmp/heap.hprof <pid>. The :live option triggers a full GC, so expect a pause on production heaps. Open the dump in Eclipse MAT, run “Leak Suspects,” and look for WebappClassLoader instances that should already be gone. MAT’s “Path to GC Roots” for the classloader shows exactly what is pinning it.
6. Inspect thread dumps for leaked threads. Run jstack <pid> and look for threads whose names reference the old webapp, or whose context classloader is the old WebappClassLoader. These are usually timer threads, scheduled executors, or connection-cleanup threads spawned by libraries.

Metrics and signals to monitor

Fixes

Switch to immutable-container deploys

The only fully reliable fix is to stop hot-redeploying. Build an image per deploy, stop the old container, start the new one. Each JVM starts with a clean classloader hierarchy. This eliminates the leak class entirely.

Fix the leaking reference in the application

When you must hot redeploy, remove whatever the application leaves behind. In a ServletContextListener:

• ThreadLocals: clear them in contextDestroyed. Tomcat’s detection logs which threads have them; use that as your hit list.
• JDBC drivers: iterate DriverManager.getDrivers() and call DriverManager.deregisterDriver for drivers loaded by the webapp classloader. Tomcat deregisters webapp-loaded JDBC drivers automatically on undeploy (WebappClassLoaderBase runs this on stop in Tomcat 8.5+), but drivers registered through a shared or static classloader are not covered; verify the DriverManager driver list after undeploy.
• Timer and ScheduledExecutorService: cancel them in contextDestroyed. Set clearReferencesStopTimerThreads="true" on the Context to have Tomcat forcibly stop java.util.Timer threads the app left running (the default is false).
• Log appenders: ensure the logging framework releases appenders on undeploy. For log4j and logback this typically means attaching the appender lifecycle to the ServletContext.
• JMX MBeans: unregister every MBean the app registered.
• Threads spawned by libraries: clearReferencesStopThreads="true" forcibly stops them, but this is unsafe and can corrupt application state. Tomcat leaves it off by default. The safer fix is making the library’s threads daemon threads or shutting them down cleanly via the library’s own shutdown API.

Let Tomcat detect and mitigate

Ensure the JreMemoryLeakPreventionListener is configured. It pre-initializes known JRE singletons at Tomcat startup so they cannot capture the webapp classloader later. Treat the SEVERE warnings Tomcat emits at undeploy as a continuous audit signal. Note that Tomcat 10.1 removed the AWTThreadProtection, forkJoinCommonPoolProtection, gcDaemonProtection, ldapPoolProtection, tokenPollerProtection, and xmlParsingProtection properties of JreMemoryLeakPreventionListener as the underlying JVM issues were fixed, and Tomcat 11.0 exposes no properties for the listener at all.

As a stopgap when the leak cannot be fixed immediately, schedule periodic full JVM restarts based on Metaspace growth. This is duct tape, not a fix.

Prevention

• Set -XX:MaxMetaspaceSize on every Tomcat instance that may be hot-redeployed. The default unbounded Metaspace produces a silent OOM kill. A JVM OutOfMemoryError is always preferable to a kernel OOM kill.
• Monitor Metaspace and LoadedClassCount across deploy cycles. The signature is a step function: each redeploy adds the same amount and never gives it back.
• Prefer immutable-container deploys when the platform allows.
• Treat Tomcat’s undeploy warnings as failures. Any SEVERE log entry at undeploy is a leak to fix.
• Audit ServletContextListener implementations in every app that hot-redeploys. If contextDestroyed does not clean up what contextInitialized created, the app leaks.

How Netdata helps

• Per-second Metaspace metrics from java.lang:type=MemoryPool,name=Metaspace show the step increase after each redeploy and confirm whether the old classloader was reclaimed.
• LoadedClassCount tracking reveals monotonic class accumulation across deploy cycles.
• Process RSS alongside JVM heap makes the non-heap nature of the leak obvious: RSS climbs while heap stays flat.
• Correlation with deploy events lets you line up the Metaspace step with the deploy log entry.
• ML anomaly detection on Metaspace growth flags the redeploy that breaks the previous pattern, even when no static threshold has been crossed.
• JMX integration collects these signals without custom instrumentation or scripting.

Related guides

• Tomcat java.net.BindException: Address already in use: the connector never starts
• Tomcat accepts connections but never responds: the TCP-connect trap
• Tomcat frequent Full GC: pause time, G1, and the 5% overhead rule
• Tomcat GC death spiral: full GCs dominating and throughput collapsing
• Tomcat heap usage: watch the post-GC baseline, not the sawtooth peak
• How Tomcat actually works in production: a mental model for operators
• Tomcat HTTP Status 503 Service Unavailable: the connector is out of threads
• Tomcat process not running: crashes, OOM-kills, and failed restarts
• Tomcat maxThreads and minSpareThreads: sizing the executor correctly
• Tomcat monitoring checklist: the signals every production instance needs
• Tomcat monitoring maturity model: from survival to expert
• Tomcat OutOfMemoryError: GC overhead limit exceeded: GC running but freeing nothing