Tomcat NoClassDefFoundError after redeploy: stale classloaders serving half-loaded classes

A hot redeploy on Tomcat succeeds, the new context reports STARTED, and within minutes requests start failing with java.lang.NoClassDefFoundError for classes that are visibly present in the new WAR. Sometimes it is ClassNotFoundException. Sometimes the error names a class that was renamed or removed between versions. Restarting the JVM clears it. The next redeploy brings it back.

The root mechanism is the WebappClassLoader lifecycle. Tomcat creates a new classloader for each web application context and replaces it on every redeploy. The old classloader should be garbage collected along with every class it loaded. When something holds a strong reference to it (a thread, a ThreadLocal, a JDBC driver registration, a logging appender, a static field, a shutdown hook), it stays alive. Threads still bound to it try to resolve classes against it. Those classes either no longer exist in the new context or conflict with what the old classloader already loaded, and the JVM throws NoClassDefFoundError or ClassNotFoundException.

This is the classloader leak failure pattern. NoClassDefFoundError is one companion symptom. OutOfMemoryError: Metaspace is the other. Both share the same root cause and the same long-term fix.

What this means

A NoClassDefFoundError after a hot redeploy means the JVM has two live classloaders for the same context at the same time, and some thread or cache is using the wrong one. The class that is failing to load is not necessarily missing. It is being looked up in a classloader that was supposed to be retired.

Two clues distinguish this from a genuine missing-class error:

• A clean JVM restart makes the error disappear. If the class were truly absent from the WAR, the restart would not help.
• The error appears only after at least one previous deployment of the same context. A fresh Tomcat with a single first-time deploy does not show this pattern.

Metaspace growth is the companion signal. Each leaked classloader retains every class it ever loaded. Metaspace grows in a staircase, one step per redeploy, and never drops. Eventually it hits MaxMetaspaceSize and the JVM dies with OutOfMemoryError: Metaspace. If MaxMetaspaceSize is unset, the OS OOM killer ends the process with no JVM-level error.

There is a second, distinct cause worth separating early: a library or version conflict packaged in the WAR itself. Here the class exists, but two incompatible versions are on the classpath (typically one in WEB-INF/lib and one in Tomcat’s lib), and the classloader resolves the wrong one. The method signatures do not match and the JVM throws NoClassDefFoundError, NoSuchMethodError, or LinkageError. This form is not a leak. It reproduces on a clean restart, which is the fastest way to tell the two apart.

flowchart TD
    A["Hot redeploy"] --> B["New WebappClassLoader created"]
    B --> C["Old classloader should be GC'd"]
    C --> D{"Reference held?"}
    D -- "Yes: thread, ThreadLocal, JDBC, hook" --> E["Old classloader stays live"]
    E --> F["Threads serve classes from old loader"]
    F --> G["NoClassDefFoundError on missing/renamed class"]
    E --> H["Metaspace grows one step"]
    H --> I["OutOfMemoryError: Metaspace after N redeploys"]
    D -- "No" --> J["Clean GC, no error"]

Common causes

Quick checks

These are read-only and safe to run on a live production Tomcat.

# Identify the failing class from recent errors
grep -nE "NoClassDefFoundError|ClassNotFoundException" $CATALINA_BASE/logs/catalina.out | tail -50

# Look for Tomcat's own leak-detection warnings during the last undeploy
grep -iE "threadLocal|failed to stop|memory leak|forcibly" $CATALINA_BASE/logs/catalina.out | tail -50

# Check application context state via Manager (running, stopped, failed)
curl -s -u $MANAGER_USER:$MANAGER_PASS http://localhost:8080/manager/text/list

# Use Manager's leak finder. Calls System.gc and reports stopped webapps with live classloaders.
curl -s -u $MANAGER_USER:$MANAGER_PASS http://localhost:8080/manager/text/findleaks

# Current Metaspace usage and limit (JDK 11+)
jcmd $(pgrep -f 'catalina.startup.Bootstrap') VM.metaspace

# Per-second GC + Metaspace view (MC = capacity, MU = used, in KB)
jstat -gc $(pgrep -f 'catalina.startup.Bootstrap') 1000

# Look for application threads that survived the undeploy
jstack $(pgrep -f 'catalina.startup.Bootstrap') > /tmp/threads.txt
grep -ciE "scheduler|timer|shutdown|ThreadLocal" /tmp/threads.txt

# Verify the failing class is actually in the new WAR
unzip -l /path/to/app.war | grep -i "NameOfFailingClass"

# Find duplicate jars across Tomcat lib and the WAR (version-conflict check)
ls $CATALINA_BASE/lib | sort > /tmp/tomcat-lib.txt
unzip -l /path/to/app.war | awk '/WEB-INF\/lib\// {n=split($4,a,"/"); print a[n]}' | sort > /tmp/war-lib.txt
comm -12 /tmp/tomcat-lib.txt /tmp/war-lib.txt
# Note: this only catches exact filename matches. Tomcat ships short names
# (servlet-api.jar) that differ from Maven artifact names (javax.servlet-api-4.0.1.jar).
# Manually inspect for servlet-api, jsp-api, el-api, and annotations-api in WEB-INF/lib.

How to diagnose it

1. Confirm the error is post-redeploy only. Restart the JVM cleanly and replay the same request. If the error disappears, you are in classloader-leak territory. If it reproduces, you are in version-conflict, failed-deploy, or namespace-mismatch territory.

2. Identify the failing class. The first line of the stack trace names the class the JVM tried to load. Note whether it is an application class, a framework class, or a javax.* / jakarta.* namespace class. A javax.servlet.* error on Tomcat 10.x usually means the WAR was compiled against Java EE 8 (Jakarta EE 8) and not Jakarta EE 9+.

3. Pull Metaspace history. Compare Metaspace usage before the redeploy, immediately after, and ten minutes later. A leak shows a step up that does not come back down. The java.lang:type=MemoryPool,name="Metaspace" MBean is the cleanest source.

4. Run Manager findleaks. Any non-empty output strongly suggests a leak. It calls System.gc and then lists stopped webapps whose classloaders are still reachable.

5. Mine catalina.out for Tomcat’s leak warnings. On undeploy Tomcat logs specific, actionable messages: ThreadLocal keys not removed, threads started but not stopped, JDBC drivers registered but not deregistered. These messages name the leak source.

6. Take a thread dump. Look for application-spawned threads (Timer, ScheduledExecutorService, framework schedulers) that survived the undeploy. Their context classloader still points at the old WebappClassLoader.

7. If the logs do not name the leak, take a heap dump before restarting. Use jmap -dump:format=b,file=/tmp/heap.hprof <pid> and open the result in Eclipse MAT. Run the “Leak Suspects” report and search for WebappClassLoader instances and their GC roots. Warning: both jmap -dump and jcmd GC.heap_dump stop the JVM for the full dump duration. On a large heap this can be seconds of downtime. For the safe capture procedure before a forced restart, see the related heap-dump guide below.

8. If the error reproduces on clean restart, switch to version-conflict diagnosis. List jars in both Tomcat’s lib and the WAR’s WEB-INF/lib. A class present in both locations is ambiguous. The classloader delegation order decides which one wins, and it may not be the one the application expects.

Metrics and signals to monitor

Fixes

Fix: clean JVM restart (immediate)

A clean restart clears all leaked classloaders and is the only reliable immediate remediation. Hot-redeploying again is not the fix. It adds another leaked classloader on top of the existing ones.

Tradeoff: restart loses in-flight sessions unless you run a PersistentManager or an external session store. Schedule it; do not bounce under load without a reason.

Fix: stop hot-redeploying in production

Most production Tomcat instances should not hot-redeploy. The reliable deployment model is a full JVM restart per deploy. This eliminates the leak mechanism entirely.

If you must hot-redeploy, cap the number of redeploys before a scheduled restart based on Metaspace growth per redeploy. Compute it as (MaxMetaspaceSize - current Metaspace) / growth-per-redeploy.

Fix: find and remove the leak source

This is the long-term fix. Common vectors, in rough order of frequency:

• ThreadLocals not cleared in contextDestroyed. Add a ServletContextListener that removes entries from your own ThreadLocals. Tomcat logs the offending key on undeploy.
• Threads started by the application and never stopped (java.util.Timer, ScheduledExecutorService, framework schedulers). Cancel them in contextDestroyed. Tomcat’s clearReferencesStopThreads Context attribute can forcibly stop them, but this is unsafe and should not be relied on in production.
• JDBC drivers in WEB-INF/lib auto-register with DriverManager, which is JVM-global. Tomcat attempts deregistration on undeploy, but a driver also loaded by the common classloader may not clean up fully. Move shared JDBC drivers to Tomcat’s lib, or deregister them manually in contextDestroyed.
• Logging frameworks (log4j, logback) hold classloader references via appenders and repository selectors. Shut them down explicitly in contextDestroyed.
• Shutdown hooks registered by the application or framework. These are JVM-global and pin the classloader. A ServletContextListener that removes the hook, by reflection if necessary, is the workaround. Frameworks have shipped regressions in this area; verify behaviour on every framework upgrade.

Fix: library or version conflict

If the error reproduces on clean restart, the class is on the classpath twice with incompatible versions. Remove the duplicate. The safer rule: framework jars that Tomcat itself provides (servlet API, JSP API, EL) should not also be in WEB-INF/lib. Use scope=provided in Maven or compileOnly in Gradle.

Fix: set MaxMetaspaceSize

Always set -XX:MaxMetaspaceSize. Without it, a classloader leak grows Metaspace until the OS OOM killer ends the process, with no JVM-level error. The limit turns a silent kill into a recoverable OutOfMemoryError: Metaspace and bounds the blast radius.

Prevention

• Production deploys are clean restarts, not hot redeploys. This is the single highest-leverage change.
• Set -XX:MaxMetaspaceSize. Always. The default is unbounded.
• Keep the JreMemoryLeakPreventionListener enabled (it is the default). It pre-loads known JRE singletons at startup so they do not capture a webapp classloader later.
• Audit ServletContextListener cleanup on every framework upgrade. Logging, scheduling, and data-source libraries have all shipped regressions that leaked classloaders.
• Run Manager findleaks after every staging redeploy as a CI check. Any non-empty output fails the build.
• Verify WAR integrity before deploy. A truncated or partially-written WAR produces a half-loaded context that mimics a classloader leak.
• Watch for the javax to jakarta migration on Tomcat 10.x. A WAR compiled against javax.servlet.* throws NoClassDefFoundError on the namespace classes. The fix is recompiling against jakarta.servlet.*, not a classloader cleanup.

How Netdata helps

• The Java JMX integration collects java.lang:type=MemoryPool,name="Metaspace" and java.lang:type=ClassLoading (LoadedClassCount, UnloadedClassCount) at per-second resolution, making the post-redeploy step pattern visible.
• Per-second GC metrics from java.lang:type=GarbageCollector show whether the classloader leak has progressed to triggering Full GCs or pause-time inflation.
• ML anomaly detection on Metaspace can flag the staircase pattern before the absolute value approaches MaxMetaspaceSize.
• The Tomcat integration surfaces per-connector thread counts and error rates, so a 5xx spike lines up against the deploy timestamp.
• Correlating the deploy event with Metaspace, LoadedClassCount, and error rate on one timeline confirms whether the failure is a classloader leak or a WAR packaging issue.

Related guides

• Tomcat java.net.BindException: Address already in use: the connector never starts
• Tomcat accepts connections but never responds: the TCP-connect trap
• Tomcat frequent Full GC: pause time, G1, and the 5% overhead rule
• Tomcat GC death spiral: full GCs dominating and throughput collapsing
• Tomcat heap dump before restart: capturing evidence with jmap and jstack
• Tomcat heap usage: watch the post-GC baseline, not the sawtooth peak
• How Tomcat actually works in production: a mental model for operators
• Tomcat HTTP Status 503 Service Unavailable: the connector is out of threads
• Tomcat process not running: crashes, OOM-kills, and failed restarts
• Tomcat maxThreads and minSpareThreads: sizing the executor correctly
• Tomcat monitoring checklist: the signals every production instance needs
• Tomcat monitoring maturity model: from survival to expert