Tomcat OutOfMemoryError: unable to create new native thread: the JVM can't spawn threads

When Tomcat throws java.lang.OutOfMemoryError: unable to create new native thread, do not reach for heap dumps or GC tuning. This is not a heap problem. The JVM called pthread_create and the kernel refused. The heap can be 20% full when this fires.

Heap OOMs need -Xmx increases, leak hunting, or GC tuning. Native thread OOMs need ulimit -u increases, cgroup pids.max adjustments, -Xss tuning, or finding the code that creates threads without bound.

Each Java thread reserves roughly 512KB to 1MB of off-heap memory for its stack on 64-bit Linux (default ThreadStackSize=1024 KB on JDK 21). A few thousand threads consume gigabytes of RSS before heap pressure registers. A Tomcat with -Xmx2g can be OOM-killed at 4GB RSS with no heap pressure at all.

On JDK 21+, the error string may read unable to create native thread: possibly out of memory or process/resource limits reached. The diagnosis is the same.

What this means

The JVM creates threads by calling pthread_create. That call can fail for several reasons, and the error message does not tell you which one:

1. Process/thread limit (ulimit -u / nproc): the user hit the RLIMIT_NPROC ceiling. On RHEL-based distributions the default soft nproc for non-root users is 1024. The kernel refuses because the process already owns too many tasks.
2. Memory for thread stacks: the process has run out of address space or RSS budget for another stack. Each thread needs ThreadStackSize bytes (default 1024 KB on 64-bit Linux, configurable via -Xss).
3. cgroup pids.max: in containers (Docker, Kubernetes), the pids cgroup controller caps the number of tasks per cgroup. The limit varies by runtime and daemon configuration; 4096 is a common Docker default.
4. System-wide limits: /proc/sys/kernel/threads-max and /proc/sys/kernel/pid_max cap total tasks across the whole kernel. The effective limit is the minimum of the two.
5. Native memory leak starving stack allocation: an off-heap leak (the JDK 21.0.3+ C2 compiler arena leak is the current headline example) can consume RSS until the JVM can no longer allocate thread stacks, even when thread count is modest.

The heap is not involved. Adding -Xmx does nothing. Identify which ceiling was hit, then raise the ceiling or stop the leak.

flowchart TD
    A["OOM: unable to create new native thread"] --> B{"Thread count plateaued?"}
    B -- "Yes, near 1024" --> C["ulimit -u / nproc"]
    B -- "Yes, near pids.max in container" --> D["cgroup pids.max"]
    B -- "Yes, near threads-max or pid_max" --> E["System-wide limit"]
    B -- "No, still climbing" --> F["Thread leak"]
    B -- "Count normal, RSS climbing" --> G["Stack memory or native leak"]
    G --> H{"JDK 21.0.3 to 21.0.5?"}
    H -- "Yes" --> I["Suspect C2 leak, run VM.native_memory"]
    H -- "No" --> J["Check ThreadStackSize vs RSS budget"]

Common causes

Quick checks

# Process thread limit (RLIMIT_NPROC) for the current shell
ulimit -u

# System-wide thread and PID ceilings
cat /proc/sys/kernel/threads-max
cat /proc/sys/kernel/pid_max

# cgroup pids limit and current usage (cgroup v2)
cat /sys/fs/cgroup/pids.max 2>/dev/null
cat /sys/fs/cgroup/pids.current 2>/dev/null
# cgroup v1 equivalent
cat /sys/fs/cgroup/pids/pids.max 2>/dev/null

# Current thread count of the Tomcat process
TOMCAT_PID=$(pgrep -f 'org.apache.catalina.startup.Bootstrap')
ls /proc/$TOMCAT_PID/task | wc -l

# Process RLIMIT from proc (works when you cannot attach a shell as the user)
grep -E 'Max processes|Threads' /proc/$TOMCAT_PID/limits

# JVM thread stack size (default 1024 KB on 64-bit Linux, JDK 21)
java -XX:+PrintFlagsFinal -version 2>&1 | grep ThreadStackSize

# Count live threads via the JVM
jcmd $TOMCAT_PID Thread.print | grep -c '"'

# Check the kernel log for OOM kills (requires root or CAP_SYSLOG)
dmesg -T | grep -iE 'oom|killed process' | tail

How to diagnose it

1. Capture a thread dump before doing anything else. The dump is the evidence. If the JVM is still alive, grab two dumps 30 seconds apart:

   TOMCAT_PID=$(pgrep -f 'org.apache.catalina.startup.Bootstrap')
   
   jstack $TOMCAT_PID > /tmp/threads_1.txt
   sleep 30
   jstack $TOMCAT_PID > /tmp/threads_2.txt
   diff <(grep -oE '"[^"]+"' /tmp/threads_1.txt | sort) \
        <(grep -oE '"[^"]+"' /tmp/threads_2.txt | sort) | head
   

2. Determine which ceiling was hit. Compare the current thread count (ls /proc/$TOMCAT_PID/task | wc -l) against every limit. The plateau value tells you the cause:

   • Near 1024 and runs as non-root: ulimit -u.
   • Near the container’s pids.max (commonly 4096): pids.max.
   • Near /proc/sys/kernel/threads-max or pid_max: system-wide limit.
   • No plateau, still climbing: thread leak.

3. Count threads by name pattern. A thread dump makes the leak source obvious:

   # Tomcat worker threads
   grep -c 'http-nio' /tmp/threads_1.txt
   # Application-created pool threads
   grep -c 'pool-' /tmp/threads_1.txt
   # Scheduled or timer threads
   grep -cE 'Scheduled|Timer' /tmp/threads_1.txt
   

   Tomcat’s own workers cap at maxThreads (default 200) plus internal threads. If you see thousands of pool-N-thread-M names, the application is the leak.

4. Check RSS against the container memory limit. If thread count is not the binding limit, the cause may be stack memory exhaustion:

   grep VmRSS /proc/$TOMCAT_PID/status
   # Container memory limit (cgroup v2)
   cat /sys/fs/cgroup/memory.max 2>/dev/null
   # cgroup v1 equivalent
   cat /sys/fs/cgroup/memory/memory.limit_in_bytes 2>/dev/null
   

   Approximate stack cost: thread_count multiplied by ThreadStackSize. 4000 threads at 1MB each is 4GB of off-heap memory.

5. Check the JDK version if RSS is climbing without thread growth. A C2 compiler arena leak (OpenJDK bug JDK-8343322) affects JDK 21.0.3 through 21.0.5 and JDK 23 and 24. It presents as RSS growing into multiple gigabytes with -Xmx as low as 1GB, eventually triggering the native thread error when the JVM can no longer allocate stacks. Confirm with:

   java -version
   jcmd $TOMCAT_PID VM.native_memory summary
   

   Look for the Compiler section growing without bound. Native memory tracking must be enabled at JVM startup (-XX:NativeMemoryTracking=summary) for this command to work. If NMT is not enabled, there is no way to enable it at runtime; restart the JVM with the flag.

Metrics and signals to monitor

Fixes

Raise the process/thread limit

If the plateau is at ulimit -u, raise it. For a systemd-managed Tomcat, set it in the unit file:

[Service]
LimitNPROC=16384

For non-systemd setups, edit /etc/security/limits.d/ (for example, a file like 90-nproc.conf):

tomcat  soft  nproc  16384
tomcat  hard  nproc  16384

PAM limits only apply to login sessions. systemd services need LimitNPROC. Verify after restart with grep 'Max processes' /proc/$TOMCAT_PID/limits.

Raise the cgroup pids limit

In Docker, pass --pids-limit at run time (use --pids-limit=-1 for unlimited, or pick a number that fits your workload). In Kubernetes, set the pod-level pids limit or the kubelet --pod-pids-limit. A Docker pids limit of 4096 (a common default) is often too low for a Tomcat with maxThreads=200 plus internal thread pools. Verify with cat /sys/fs/cgroup/pids.max from inside the container.

Reduce thread stack size

If RSS is the binding constraint and the thread count is legitimately high (for example, a large thread pool or a workload migrating toward virtual threads), lowering -Xss reduces per-thread cost. The default is 1024 KB. Not every workload needs that much stack:

-Xss512k

Measure before and after. Deep recursion or large frame sizes will produce StackOverflowError if -Xss is too small. The JDK 13+ flag -XX:+AdjustStackSizeForTLS (disabled by default, Linux only) accounts for glibc on-stack TLS allocation that otherwise eats into the requested stack size.

Fix the thread leak

Raising limits buys time, not a fix. Find the unbounded thread creator. Common offenders in Tomcat applications:

• Executors.newCachedThreadPool() with no upper bound and a caller that submits work per request.
• new Thread() instances started in request scope without a pool.
• @Scheduled tasks on Spring’s default scheduler that accumulate across hot redeploys.
• java.util.Timer instances created per request or per session.
• HTTP clients that create a thread or connection per call without pooling.

Count threads by name in the dump. A growing pool-N-thread-M family points at an ExecutorService creating threads without reusing them. A growing set of HTTP client threads points at a client library with no connection pool.

Handle the JDK 21.0.3+ C2 leak

If the JDK is 21.0.3 through 21.0.5, 23, or 24, and VM.native_memory summary shows the Compiler arena growing into gigabytes, the C2 JIT is the leak. Workaround options: downgrade to JDK 21.0.2, switch to JDK 17 LTS, or temporarily disable C2 with -XX:TieredStopAtLevel=1 (this sacrifices peak throughput and is not recommended for production beyond a stopgap).

Prevention

• Monitor ThreadCount from java.lang:type=Threading and alert on monotonic growth. A healthy Tomcat holds a stable count: maxThreads plus GC, JIT, and internal threads, roughly 250 to 300 for a default configuration.
• Set LimitNPROC explicitly in the systemd unit. Do not rely on inherited defaults, especially for non-root Tomcat users.
• Set --pids-limit deliberately in containers. A common Docker default of 4096 is a surprise under load.
• Bound every ExecutorService the application creates. newCachedThreadPool is almost never the right choice for request-scoped work.
• Set -Xss explicitly rather than relying on the JVM default, so the per-thread memory cost is known and reproducible.
• Track RSS alongside heap. A flat heap with climbing RSS is the signature of off-heap growth, whether from thread stacks, Metaspace, direct buffers, or native leaks.
• Pin the JDK version and track OpenJDK bug fixes. The C2 leak in 21.0.3+ affected production deployments for months before it was widely diagnosed.

How Netdata helps

• JVM ThreadCount and PeakThreadCount from java.lang:type=Threading expose thread leaks at per-second resolution, so you see the climb before it hits the ceiling.
• currentThreadsBusy and maxThreads from the Tomcat connector bean separate connector saturation from a general thread leak, so you do not chase the wrong root cause.
• cgroup pids.current and pids.max surface the container limit directly, so you can see the plateau before the OOM fires.
• RSS and cgroup memory metrics let you correlate off-heap growth (thread stacks, native allocations) with thread count, distinguishing stack memory exhaustion from a count limit.
• ML anomaly detection on thread count flags monotonic growth even when the absolute count is below any configured limit.
• dmesg OOM killer events appear alongside the JVM metrics, confirming when the OS killed the process rather than the JVM exiting on its own.

Related guides

• Tomcat 5xx error rate: separating server failures from crawler 404s
• Tomcat accept queue overflow: acceptCount, somaxconn, and Recv-Q
• Tomcat access log setup: adding %D and %T for per-request latency
• Tomcat java.net.BindException: Address already in use: the connector never starts
• Tomcat average latency lies: why you need p95/p99 from the access log
• Tomcat threads blocked forever: the missing outbound timeout
• Tomcat classloader leak on redeploy: why the old WebappClassLoader never dies
• Tomcat connection refused: maxConnections and acceptCount both exhausted
• Tomcat accepts connections but never responds: the TCP-connect trap
• Tomcat frequent Full GC: pause time, G1, and the 5% overhead rule
• Tomcat GC death spiral: full GCs dominating and throughput collapsing
• Tomcat heap dump before restart: capturing evidence with jmap and jstack