Tomcat java.lang.OutOfMemoryError: Metaspace: the classloader leak hot redeploys cause

A java.lang.OutOfMemoryError: Metaspace after a redeploy is the canonical classloader leak symptom in long-running Tomcat instances. The JVM still has heap headroom, GC looks healthy, and the process may have been up for weeks. Then a deploy lands and Tomcat dies, or in containerized setups the kernel OOM-kills it with no JVM-level error at all.

Recovery is the same in every case: restart the JVM. That clears Metaspace and brings the application back, but it does not fix anything. If you hot-deploy again, the leak returns and Metaspace climbs one step higher per cycle until the next crash.

This article covers how to confirm a classloader leak is the cause, find the leaked reference, and apply the configuration changes that buy time while you wait on a code fix. It assumes you already understand the broad Tomcat failure pattern catalogue. See the mental model for operators if you need that context first.

What this means

Metaspace is native memory used by the JVM to store class metadata: class definitions, method bytecode, constant pools, and reflection data. It is not part of the heap and is not bounded by -Xmx. Each web application in Tomcat runs inside its own WebappClassLoader, and every class that classloader touches lives in Metaspace until the classloader itself becomes unreachable and is garbage collected.

When you hot-redeploy, Tomcat creates a new WebappClassLoader for the new version of the app. The old classloader should become unreachable and be collected, freeing all of its classes. That collection only happens if nothing still references the old classloader. If anything holds a reference, a thread that was started by the app, a ThreadLocal, a JDBC driver registered with DriverManager, a java.util.Timer, a logging appender, a static field in a shared library, the old classloader and every class it loaded are pinned in Metaspace permanently.

Each redeploy that leaks adds approximately one app’s worth of classes to Metaspace. After N redeploys, Metaspace holds N copies of the application’s class space. The growth is step-wise and monotonic. That is the signature you are looking for in monitoring.

flowchart TD
  A[Hot redeploy] --> B[New WebappClassLoader]
  B --> C[Old classloader eligible for GC]
  C --> D{Leaked reference?
ThreadLocal / Timer / JDBC / static}
  D -->|No| E[Old classloader collected
Metaspace stable]
  D -->|Yes| F[Old classloader pinned
with all its classes]
  F --> G[Metaspace grows by
one app per redeploy]
  G --> H[After N redeploys:
OOM: Metaspace or
silent OS OOM-kill]

There are two distinct failure modes depending on whether -XX:MaxMetaspaceSize is set:

• MaxMetaspaceSize set: the JVM hits the limit and throws java.lang.OutOfMemoryError: Metaspace. You get a stack trace, a heap dump attempt, and a logged error before the JVM exits or the offending thread dies.
• MaxMetaspaceSize unset (the default): Metaspace grows without bound. In a VM it eventually exhausts RSS and the kernel OOM-kills the process silently. The Tomcat logs show nothing useful, only dmesg does. In a container with a cgroup memory limit, the same thing happens against the cgroup ceiling.

Most teams only set MaxMetaspaceSize after the first silent kill. Set it so the failure mode is a logged JVM error rather than a vanishing process.

Common causes

Quick checks

All commands below are read-only except where noted. They are safe to run on a production Tomcat.

# 1. Confirm the OOM is Metaspace, not heap
grep -E "OutOfMemoryError" $CATALINA_BASE/logs/catalina.out | tail -20

# 2. Check whether MaxMetaspaceSize is set
ps -ef | grep '[c]atalina.startup.Bootstrap' | tr ' ' '\n' | grep -i 'Metaspace'

# 3. Current Metaspace usage and limit
TOMCAT_PID=$(pgrep -f 'catalina.startup.Bootstrap')
jcmd $TOMCAT_PID VM.metaspace

# 4. Loaded class count snapshot
# NOTE: GC.class_histogram forces a full GC. Expect a brief STW pause.
jcmd $TOMCAT_PID GC.class_histogram | head -5

# 5. Tomcat's own leak detection on running webapps
# Requires Manager app enabled and credentials
curl -s -u $MANAGER_USER:$MANAGER_PASS http://localhost:8080/manager/text/findleaks?statusLine=true

# 6. Kernel OOM kills (the silent failure mode)
dmesg -T | grep -iE 'oom|killed process' | tail

# 7. Live thread count vs expected
jcmd $TOMCAT_PID Thread.print | grep -c "http-nio-"

# 8. Tomcat leak detection warnings during the last undeploy
grep -iE "ThreadLocal|failed to stop|memory leak" $CATALINA_BASE/logs/catalina.out | tail -50

If MaxMetaspaceSize is not in the JVM arguments, treat the silent-kill failure mode as the active risk. The fix is to set the flag. The band-aid is to lower the container memory limit so the container is killed before the JVM can consume all node memory and destabilize co-located workloads.

How to diagnose it

1. Correlate the crash with a deploy. Check deploy timestamps in $CATALINA_BASE/logs/catalina.out for HostConfig.deployWAR or HostConfig.undeploy. If every OOM follows a deploy within minutes to hours, you almost certainly have a classloader leak. Long-running instances that never hot-deploy do not hit this failure mode.

2. Graph Metaspace against deploy events. Pull the java.lang:type=MemoryPool,name="Metaspace" MBean usage.used over time, overlay deploy markers, and look for the step pattern. A healthy redeploy produces a small temporary rise that returns to roughly the previous baseline. A leaking redeploy produces a permanent step up of approximately the size of the application’s class space.

3. Use Tomcat’s leak detector before the JVM dies. The Manager app’s findleaks endpoint lists contexts whose classloaders are still reachable after undeploy. Any non-empty result is a confirmed leak. Run it right after a redeploy while the old classloader is still pinned.

4. Inspect catalina.out for Tomcat’s warnings. On undeploy, Tomcat logs diagnostics for threads it could not stop, ThreadLocals it could not clear, and other leak patterns. The exact wording is “The web application […] appears to have started a thread […] but has failed to stop it.” These messages are your fastest triage signal and they are emitted by default.

5. Take a heap dump before restart. When the next OOM hits, capture a heap dump immediately with jmap -dump:format=b,file=/tmp/heap.hprof $TOMCAT_PID before restarting. Warning: this triggers a stop-the-world pause proportional to heap size and produces a file roughly the size of live heap. Ensure adequate disk space. The dump will show which GC root is pinning the old WebappClassLoader. Eclipse MAT’s “Path to GC Roots” or “Duplicate Classes” query is the fastest way to identify the offending library. On JDK 8+, you can also use jcmd $TOMCAT_PID GC.heap_dump /tmp/heap.hprof.

6. Count WebappClassLoader instances. In MAT or VisualVM, search for instances of org.apache.catalina.loader.WebappClassLoader. In a clean state there should be exactly one per deployed app. Each extra instance is a leaked classloader holding an app-sized chunk of Metaspace.

Metrics and signals to monitor

The key correlation is Metaspace step-up versus deploy event. If you only look at heap, GC, and CPU, this failure is invisible until the process dies.

Fixes

Immediate: restart the JVM

A full JVM restart clears Metaspace and resets LoadedClassCount to baseline. This is the only reliable recovery once the leak has accumulated. Hot undeploy and redeploy do not help, because the leaked classloader is by definition unreachable from Tomcat’s lifecycle but still reachable from whatever is pinning it.

Restart is not a fix. It is the recovery step you take while the underlying leak is being tracked down.

Containment: stop hot-deploying

The single most effective mitigation is to stop hot-redeploying in production. A full JVM restart per deploy does not leak classloaders, because the entire JVM including all classloaders is destroyed. CI/CD pipelines that deploy by replacing the WAR and letting Tomcat’s autoDeploy reload are the most common source of this failure in the wild.

If your release process requires rapid iteration without a JVM bounce, accept that you will eventually need either a fix in code or scheduled preventive restarts. Calculate the runway by dividing remaining Metaspace headroom by growth-per-redeploy.

Containment: set MaxMetaspaceSize

Setting -XX:MaxMetaspaceSize=NNNm converts the silent kernel OOM-kill into a logged JVM OutOfMemoryError. Pick a value comfortably above steady-state usage, typically 256m to 512m for a single-app Tomcat. The downside is that you will hit JVM OOM sooner than you would have hit the OS limit, but the trade is worth it because the failure is observable.

Containment: enable Tomcat’s clearReferences* attributes

The Context element supports attributes that force Tomcat to attempt cleanup on undeploy when the application fails to do so. The relevant ones are clearReferencesThreadLocals, clearReferencesStopThreads, and clearReferencesStopTimerThreads. These are off by default because they can mask application bugs and have side effects. Forcibly stopping threads can leave resources in inconsistent states. But for an app you cannot immediately fix, they are the most impactful single configuration change.

Add them in context.xml or per-context in server.xml:

<Context clearReferencesThreadLocals="true"
         clearReferencesStopThreads="true"
         clearReferencesStopTimerThreads="true">
</Context>

Containment: JreMemoryLeakPreventionListener

This listener is enabled by default in modern Tomcat and mitigates several JDK-internal leak spots, including java.beans.Introspector caches and javax.imageio service registries. It cannot fix application-level leaks, but verify it is present in server.xml:

<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"/>

Containment: ThreadLocalLeakPreventionListener

This listener renews threads in Executor pools when a context is stopped, so that ThreadLocal maps holding webapp class references are replaced. Configure it in server.xml:

<Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"/>

It is most useful when your leak is ThreadLocal-based and you cannot change the offending library.

Long-term: fix the leaked reference

The actual fix is always in the application or a library it depends on. Common patterns:

• ThreadLocal without remove(): ensure every ThreadLocal.set() is paired with remove() in a finally block, ideally in a ServletRequestListener or Filter cleanup path.
• JDBC driver registration: deregister drivers in a ServletContextListener.contextDestroyed method. Tomcat 7.0+ does some of this automatically, but app-managed drivers still leak.
• Timer and ScheduledExecutorService: cancel and shutdown in contextDestroyed. Tomcat will warn if it sees these threads still alive.
• Logging appenders: invoke the framework’s shutdown hook. Log4j 2 and logback both provide explicit shutdown APIs.
• JMX MBeans: unregister any MBeans the app registered during contextDestroyed.

For library-level leaks you cannot patch, the options are upstream a fix, pin the library version, or remove hot-deploy for that app.

Prevention

• Set -XX:MaxMetaspaceSize everywhere, even on instances that do not hot-deploy today. The cost is a hard ceiling. The benefit is observability of the failure.
• Monitor Metaspace and LoadedClassCount at second-level resolution. Alert on step-up after deploy events that does not decay within 5 minutes.
• Run manager/text/findleaks after every staging redeploy as a CI check. Any non-empty result fails the deploy.
• Stop hot-deploying in production. Prefer full JVM restarts in the deploy pipeline. This eliminates the entire class of failure.
• Track Metaspace growth-per-redeploy as a capacity metric. If it is non-zero, file a bug, not a config change.
• Schedule periodic full restarts as a stopgap. If you cannot fix a leak immediately, restart at an interval computed from growth-per-redeploy and MaxMetaspaceSize. Document the runway.

How Netdata helps

• The JVM collector surfaces MemoryPool[name=Metaspace] usage.used and usage.max per second, so the step-up pattern after a redeploy is visible at the granularity it actually happens, not just as a slowly-rising hourly aggregate.
• LoadedClassCount, TotalLoadedClassCount, and UnloadedClassCount from java.lang:type=ClassLoading are collected per second. A leaking classloader shows up as LoadedClassCount climbing without corresponding UnloadedClassCount.
• Netdata’s ML anomaly detection flags the step discontinuity in Metaspace after a redeploy, even when the absolute value is still well below any static threshold. This is exactly the regime where the leak is fixable before it becomes an outage.
• Correlating Metaspace against the deploy event timeline, RSS, and GC behavior in a single chart makes the “Metaspace is leaking, heap is fine” pattern immediately readable rather than something you discover by cross-referencing three tools.
• Anomaly advisors on OutOfMemoryError log lines from catalina.out close the loop on the failure mode where MaxMetaspaceSize is set and the JVM throws rather than getting kernel-killed.

Related guides

• Tomcat GC death spiral: full GCs dominating and throughput collapsing
• Tomcat heap usage: watch the post-GC baseline, not the sawtooth peak
• Tomcat OutOfMemoryError: GC overhead limit exceeded: GC running but freeing nothing
• Tomcat process not running: crashes, OOM-kills, and failed restarts
• How Tomcat actually works in production: a mental model for operators
• Tomcat monitoring checklist: the signals every production instance needs