$ guides / tomcat / tomcat-too-many-open-files ▌

Operations Guides

Tomcat java.net.SocketException: Too many open files: file descriptor exhaustion

java.net.SocketException: Too many open files in a Tomcat log is a hard cliff, not gradual degradation. The JVM goes from serving traffic to unable to accept a new TCP connection, open a log file, or load a JAR resource. The error is frequently misread as a disk fault (log writes break) or a network fault (accept() fails), when the real constraint is the per-process file descriptor limit.

Every socket, open file, NIO selector, and JAR handle the JVM touches consumes one file descriptor against the process ulimit -n ceiling. A busy Tomcat with thousands of keepalive connections, a deep classpath, and several rotating log files can sit at hundreds of FDs at idle and climb into the thousands under load. When the count hits the limit, the kernel refuses the next open() or accept() syscall and the JVM throws the exception.

The default limits on many Linux distributions, 1024 soft and 4096 hard, were chosen for generic user shells and are too low for a production servlet container. Note that systemd-managed services use their own defaults (1024 soft / 524288 hard since systemd 240), higher than the classic shell pair. Production Tomcat should start at 65536 or higher. Raising the limit is the first action to restore service, but it is rarely the durable fix: a genuine FD leak will eventually hit any ceiling you set.

What this means

A file descriptor is a kernel handle to an open resource. For the Tomcat JVM, the consumers are:

TCP sockets. Every accepted connection, including idle keepalive connections, is one FD. Usually the dominant consumer.
NIO selector / epoll. The NIO connector uses a java.nio.Selector, backed by an epoll FD, to multiplex connections. A small fixed cost on top of the per-connection FDs.
JAR handles. The JVM holds FDs for JARs on the classpath. A large classpath (many webapps, many shared libraries) can consume hundreds of FDs at startup.
Log files. catalina.out, per-app logs, and access log files each hold an FD while open. Misconfigured rotation that leaves old handles open compounds this.
Temp files. Multipart upload handling creates temporary files that consume FDs until they are cleaned up.

The limit is per-process and comes from two values: a soft limit and a hard limit. The JVM can raise its soft limit up to the hard limit on startup, so the value reported by ulimit -n in your shell may not match the limit the running process actually has. /proc/<PID>/limits is the source of truth for the live process.

The failure is non-discriminating at the cliff: accept() fails (looks like a network problem), open() on a log file fails (looks like a disk problem), and loading a class from a not-yet-open JAR fails (looks like a classloader bug). This is why the error is so often misdiagnosed.

flowchart TD
    Sockets["TCP sockets (1 per connection)"] --> Pool["Per-process FD pool"]
    Selector["NIO selector / epoll FDs"] --> Pool
    Jars["JAR handles on classpath"] --> Pool
    Logs["Log files (catalina.out, app logs)"] --> Pool
    Temp["Temp upload files"] --> Pool
    Pool --> Check{"At FD limit?"}
    Check -- No --> OK["Normal operation"]
    Check -- Yes --> Cliff["Hard cliff: next FD op fails"]
    Cliff --> AcceptFail["accept() fails: SocketException"]
    Cliff --> LogFail["open() fails: log writes break"]

Common causes

Cause	What it looks like	First thing to check
Low ulimit	FD count is modest (a few thousand) but already at the process limit; error appears under normal load	`cat /proc/<PID>/limits`
Application FD leak	FD count grows monotonically regardless of traffic; sockets or files accumulate and never close	`lsof -p <PID>` for repeated patterns
Connection surge	FD count tracks connection count; idle keepalive or slow clients hold sockets open	`ss` connection count vs request rate
Log rotation failure	File-type FDs to old rotated logs accumulate; count rises at rotation boundaries	count REG entries in `lsof` output
Temp file accumulation	File-type FDs under the Tomcat work directory grow under upload-heavy traffic	check `work/` and temp dirs

Quick checks

These are read-only and safe to run on a production JVM. On a process with tens of thousands of FDs, lsof can take several seconds and produce large output; consider redirecting to a file.

# Identify the Tomcat JVM process.
# For Spring Boot embedded Tomcat, adjust the pattern to match your entry point.
TOMCAT_PID=$(pgrep -f 'org.apache.catalina.startup.Bootstrap')

# Check the actual per-process FD limit (source of truth, not ulimit -n)
grep "Max open files" /proc/$TOMCAT_PID/limits

# Count currently open FDs (compare to the limit above)
echo "FDs in use: $(ls /proc/$TOMCAT_PID/fd | wc -l)"

# Classify open FDs by type (IPv4, IPv6, REG, PIPE, etc.)
lsof -p $TOMCAT_PID | awk 'NR>1{print $5}' | sort | uniq -c | sort -rn

# Count socket-type FDs specifically
ls -l /proc/$TOMCAT_PID/fd | grep -c socket

# Inspect the largest consumers by target path or inode
# Most useful for REG entries; socket entries show address/state instead
lsof -p $TOMCAT_PID | awk '{print $NF}' | sort | uniq -c | sort -rn | head -20

# Current connection count to the HTTP port (compare to socket FD count)
ss -tn state established '( sport = :8080 )' | tail -n +2 | wc -l

# Confirm the error string in the Tomcat log
# CATALINA_BASE is typically not set in your shell; find the log dir from the process
grep -m 5 "Too many open files" /path/to/tomcat/logs/catalina.out

How to diagnose it

Confirm FD exhaustion is the actual cause. Grep the log for the exact error string, then compare ls /proc/<PID>/fd | wc -l against grep "Max open files" /proc/<PID>/limits. If the used count is at or near the max, the diagnosis is confirmed. If it is well below the limit, the error is coming from something else: a per-user limit, a container cgroup limit, or a different resource constraint.
Verify the real limit, not your shell’s. Run cat /proc/<PID>/limits. The value here is what the JVM is actually bound by. A common trap is editing /etc/security/limits.conf, logging in, checking ulimit -n, seeing 65536, and assuming Tomcat has it. Under systemd, that file is ignored for services, and the Tomcat process may still be running with the systemd default.
Classify the FDs. Run the lsof type classification. If IPv4/IPv6 (sockets) dominate, the driver is connections. If REG (regular files) dominate, the driver is files: logs, temp files, or JARs. If the split is mixed, look at the top paths.
If sockets dominate, separate load from leak. Compare the socket FD count against the established connection count from ss. If they are close, the FDs are legitimate live connections and the problem is capacity or slow clients. If sockets far exceed live connections, connections are being accepted but not closed: a connection leak or a slow-client attack.
If files dominate, find which files. Look for repeated paths in the lsof output. Old rotated log files still held open point to a rotation bug. Many FDs into a single JAR or into the Tomcat work/ directory point to classloading or temp file handling.
Decide leak vs capacity. Watch the FD count over a few minutes during a traffic dip. If it drops proportionally with traffic, you have a capacity problem. If it stays flat or keeps rising while traffic falls, you have a leak.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`OpenFileDescriptorCount` / `MaxFileDescriptorCount` (JMX)	The direct ratio of used to allowed FDs; the only signal that catches the cliff before it arrives	Ratio above 0.8 of the limit
Connection count (`ss` or JMX `connectionCount`)	Sockets are usually the largest FD consumer; separates connection-driven growth from file leaks	Rising without a matching request rate increase
Request throughput	Throughput collapsing while FDs are high confirms the exhaustion is user-impacting	Sudden drop coinciding with FD errors in the log
HTTP error rate	`accept()` failures surface as connection errors or 5xx responses	Spike that lines up with FD pressure
FD count growth rate	A monotonic upward trend independent of traffic is the signature of a leak	Steady increase during low-traffic periods

A practical threshold is FD used divided by FD limit above 0.8. Below 0.5 at peak is healthy headroom.

Fixes

Raise the ulimit (restore service first)

This is the immediate action. It treats the symptom, not the cause, but it restores service while you investigate.

For a systemd-managed Tomcat, /etc/security/limits.conf has no effect. Use a drop-in override:

# Edit the service override (do not edit the installed unit file directly)
systemctl edit tomcat
# Add under [Service]:
#   LimitNOFILE=65536

The service name in the systemctl commands is distribution-dependent (tomcat, tomcat9, tomcat10, …); adjust the commands to your unit name.

Warning: systemctl restart tomcat drops all in-flight connections and causes a full outage window. Do this only if the service is already failing, or schedule a controlled restart.

systemctl restart tomcat

Verify after restart with grep "Max open files" /proc/<PID>/limits.

For Tomcat in Docker, pass the ulimit at run time or in Compose:

# Docker run
docker run --ulimit nofile=65536:65536 ...

# Docker Compose
# ulimits:
#   nofile:
#     soft: 65536
#     hard: 65536

Set both soft and hard to the same value to avoid surprises where the JVM negotiates a different limit than you expect. Always verify the live value via /proc/<PID>/limits, because the JVM can raise its soft limit to the hard limit on startup.

Tradeoff: raising the limit only buys time. If there is a leak, a higher ceiling just delays the cliff. Pair this with leak detection.

Find and fix the FD leak

If FDs grow monotonically regardless of traffic, the application is opening resources and not closing them. Common sources:

Unclosed streams. InputStream, OutputStream, or Reader opened on a file or socket and never closed, especially in exception paths. Use try-with-resources.
JDBC connections. Connections borrowed from the pool and not returned. Enable removeAbandoned=true and logAbandoned=true on the pool to reclaim and trace them.
HTTP client connections. Outbound HTTP clients whose response bodies or connections are not closed.
Multipart temp files. Tomcat’s multipart handling creates temp files that are cleaned up on GC, not on request completion. Under heavy upload concurrency they can accumulate.

Use lsof -p <PID> to find the repeated file or socket patterns, then map them back to the code path that creates them. A thread dump (jstack <PID>) taken while the leak is active often shows the stack frames holding the open handles. Note: jstack briefly pauses the target JVM; run it during low traffic if latency-sensitive.

Reduce FD consumption

If the FD count is legitimate (real traffic, no leak) but the limit is still too tight:

Tune keepalive. Idle keepalive connections hold FDs. Lowering connectionTimeout closes idle connections sooner. Coordinate this with any upstream reverse proxy keepalive setting so the proxy does not reuse a connection Tomcat has already closed.
Fix log rotation. Ensure the logging framework releases handles to rotated files. A rotation that moves or compresses a file without closing the Tomcat-side handle leaks an FD per rotation.
Right-size maxConnections. The NIO connector default is 8192 in current Tomcat (the 10000 figure is from older pre-9.0 documentation), and each accepted connection is an FD. If your ulimit cannot absorb that many connections plus overhead, lower maxConnections or raise the ulimit accordingly. Do not lower it below what peak traffic needs or you will trade FD exhaustion for connection refusal.

Prevention

Set the ulimit to 65536 or higher in production. Verify it via /proc/<PID>/limits, not ulimit -n. For systemd services, use LimitNOFILE in a drop-in override. For containers, set both soft and hard.
Alert on FD used / limit above 0.8. Track the ratio, not the absolute count, because the absolute count is meaningless without the limit.
Monitor FD count and connection count together. A rising FD count with a flat connection count is a leak. A rising FD count with a rising connection count is load. You need both signals to tell them apart.
Watch the growth rate during low-traffic windows. A leak that is invisible at peak becomes obvious at 3 a.m. when traffic drops and FDs keep climbing.
Load test to establish the FD baseline. Know how many FDs your deployment consumes at peak before it hits production, so a real leak is distinguishable from normal load.
Review IO code for try-with-resources. Every open() needs a matching close() in a finally block or try-with-resources. This is the most common leak source in application code.

How Netdata helps

Netdata collects OpenFileDescriptorCount and MaxFileDescriptorCount from the JVM JMX interface every second, so the FD-to-limit ratio is visible live without running lsof during an incident.
Correlating FD count with Tomcat connection count separates a connection-driven surge from a file-handle leak in a single view, which is the hardest judgment call to make from the command line.
Per-second resolution shows whether FD growth tracks request throughput (load) or is monotonic (leak), the key diagnostic distinction.
Anomaly detection can flag unusual FD growth patterns earlier than a static 0.8 threshold, giving lead time before the hard cliff.
Pairing FD saturation with error rate and accept-queue depth shows the full failure cascade, confirming the FD cliff is the root cause and not a downstream symptom.