$ guides / tomcat / tomcat-file-descriptor-exhaustion ▌

Operations Guides

Tomcat file descriptor usage: OpenFileDescriptorCount vs the ulimit

Tomcat’s JVM holds file descriptors for every open socket, every log file, every JAR on the classpath, the NIO selector itself, and various internal pipes. The pool is finite and process-scoped. When it runs out, Tomcat stops accepting connections and stops writing to logs in the same instant. The error is java.net.SocketException: Too many open files, and by the time you see it the failure is already total.

Operators have two views into this resource. JMX exposes OpenFileDescriptorCount and MaxFileDescriptorCount from the java.lang:type=OperatingSystem MBean. The OS exposes the same count through /proc/<pid>/fd and the limit through ulimit -n and /proc/<pid>/limits. The two views should agree within one descriptor. When they do not, you are usually looking at the wrong process, the wrong container namespace, or at lsof output (which counts more than file descriptors).

What it is and why it matters

File descriptors are the kernel’s handle for open files, sockets, pipes, and on Linux a long list of other resources. For a Tomcat process the dominant consumers are:

Consumer	Typical count	Notes
HTTP/HTTPS/AJP sockets	one per accepted connection	includes idle keepalive connections
NIO selector and registered sockets	a handful, plus one per registered socket	the poller’s epoll FD
Access and application logs	one or two per active log file	grows if rotation leaves handles open
Classpath JARs	hundreds on a large Spring classpath	baseline cost, not a leak
Internal pipes	a few	between JVM subsystems

The limit is hit suddenly. There is no graceful degradation, no queueing, no backpressure. The next accept(), open(), or socket() call returns EMFILE, and Java surfaces it as SocketException: Too many open files. Accepting new HTTP connections fails. Writing to the access log fails. Opening a freshly rotated log file fails. All of these happen in the same instant.

The operational signal is the ratio OpenFileDescriptorCount / MaxFileDescriptorCount, and the trajectory of OpenFileDescriptorCount relative to connection count. The playbook’s headroom rule: keep peak FD usage under 50% of the limit, and size production Tomcat with an ulimit of at least 65535. Default distro ulimits of 1024 or 4096 are routinely too low and fail under modest keepalive load.

How it works

flowchart TD
  subgraph C["FD consumers in the Tomcat JVM"]
    Sockets["Sockets"]
    NIO["NIO selector"]
    Logs["Logs"]
    JARs["Classpath JARs"]
    Pipes["Pipes"]
  end

  C --> Pool["Open file descriptors"]

  Pool -->|JMX view| JMXView["OpenFileDescriptorCount
java.lang:type=OperatingSystem"]
  Pool -->|OS view| ProcView["/proc/pid/fd"]

  JMXView -.->|agree within 1| ProcView

  Pool -->|bounded by| Limits["Limit chain:
soft ulimit,
fs.nr_open,
fs.file-max"]

The JMX view

The java.lang:type=OperatingSystem platform MBean exposes two long attributes:

OpenFileDescriptorCount - the current count of file descriptors held by the JVM process.
MaxFileDescriptorCount - the process’s current soft RLIMIT_NOFILE ceiling, not the hard ceiling.

On Linux, OpenFileDescriptorCount is computed by listing /proc/self/fd. The OpenJDK implementation uses opendir(), which itself consumes a descriptor, so JMX and ls /proc/<pid>/fd | wc -l agree within one.

MaxFileDescriptorCount reflects the soft limit at the time of the query. If the JVM or a wrapper script raised the soft limit below the hard limit, JMX reports the raised value. That is the limit the process actually operates under. Do not confuse it with the hard ceiling.

The OS view

ls /proc/<pid>/fd | wc -l is the authoritative count. cat /proc/<pid>/limits | grep "Max open files" shows the soft and hard RLIMIT_NOFILE for that process.

Avoid lsof for FD budget accounting. lsof lists more than file descriptors: it includes memory-mapped files, the current working directory, the root directory, executable text segments, and other entries that do not consume FD slots. lsof | wc -l systematically overcounts relative to /proc/<pid>/fd. For pure FD budget work, /proc/<pid>/fd is the truth and JMX mirrors it.

To identify what is consuming descriptors during a suspected leak:

# Count FDs grouped by target
ls -l /proc/<pid>/fd | awk '{print $NF}' | sed 's/\[.*\]//' | sort | uniq -c | sort -rn | head -20

# Show socket states for FDs that are sockets
ls -l /proc/<pid>/fd | grep socket | wc -l
ss -tanp | grep <pid> | awk '{print $1}' | sort | uniq -c

The first command groups FDs by what they point to. A large count against a single log file or a growing count of socket:[nnnn] entries narrows the leak source. The second cross-references sockets against TCP state; a rising CLOSE_WAIT count is the dominant leak pattern.

The limit chain on Linux

Three limits interact, and the smallest one wins:

Per-process soft limit (ulimit -n, RLIMIT_NOFILE cur) - the value the process operates under and can raise up to the hard limit. This is what MaxFileDescriptorCount reports.
Per-process hard limit (RLIMIT_NOFILE max, bounded by fs.nr_open) - the ceiling the process can raise to.
System-wide limit (/proc/sys/fs/file-max) - the total FDs the kernel will allocate across all processes.

For Tomcat the per-process soft limit is almost always the binding constraint. fs.file-max defaults to a large value on modern kernels and is rarely the bottleneck for a single JVM. The binding constraint shifts to fs.file-max only when many JVMs share a host and each has a generous per-process limit.

In containers the limit comes from the runtime

In containers the FD limit comes from the container runtime, not the host. The host’s ulimit -n is not what the container process sees.

For Docker and containerd, the runtime’s own systemd unit typically sets LimitNOFILE=1048576, and containers inherit this unless overridden. This is generous, sometimes problematically so: tools that iterate over all possible FDs on fork/exec (Python’s subprocess, rpm) can slow down noticeably when the limit is six figures. To narrow the limit per container, set default-ulimits in /etc/docker/daemon.json or pass --ulimit nofile=65536:65536 at run time.

For Kubernetes there is no native pod spec field for FD limits. The container runtime’s defaults apply. To change them, configure ulimits in containerd’s config.toml or in crio.conf. cgroup v2 has no file descriptor controller: pids.max caps processes and threads, not FDs, so you cannot constrain FDs through the cgroup layer.

Where it shows up in production

Default ulimit too low

The most common failure is shipping Tomcat with the distro default of 1024 or 4096. With NIO, every keepalive connection consumes a descriptor, the selector consumes a few, the classpath JARs consume hundreds, and the access log consumes a couple. Under moderate keepalive load a 1024 limit is exhausted within minutes. The fix is to set the ulimit at the layer that owns the process: a systemd unit LimitNOFILE=65535, an init script ulimit -n, or the container runtime’s defaults.

For systemd-managed Tomcat, verify after restart:

cat /proc/<pid>/limits | grep "Max open files"

FD growth without connection growth

The playbook’s leak heuristic: FD count growing without proportional connection count growth is an FD leak. Sockets or files are being opened and never closed.

The dominant Tomcat pattern is CLOSE_WAIT sockets. The remote peer sends FIN, the kernel moves the socket to CLOSE_WAIT, and the application never reads EOF and never calls close(). The socket stays in CLOSE_WAIT indefinitely, consuming an FD. Other recurring patterns: log file rotation that leaves the old handle open, JDBC connections checked out but never returned, and HTTP client responses whose bodies are not fully consumed before the connection is reused or closed.

Distinguishing signal: OpenFileDescriptorCount rises monotonically while connectionCount (the JMX metric on Catalina:type=ThreadPool,name="http-nio-8080") stays flat or oscillates with traffic. If both rise together, the FD growth is explained by real connections; the fix is capacity, not a leak hunt.

Container FD surprises

Two container-specific surprises recur:

The container sees a much higher ulimit -n (1048576) than the host. Operators comparing the host’s 1024 to the container’s reading misdiagnose a configuration change. The container reading is correct and comes from the runtime.
The container hits the host’s fs.file-max. Because the runtime’s per-container limit is so generous, many containers on one host can collectively exhaust the system-wide pool. Check with cat /proc/sys/fs/file-nr; the first column is allocated, the third is the max. This is rare but worth checking when MaxFileDescriptorCount is high but open() still fails with EMFILE.

NIO and classpath descriptors are non-obvious consumers

NIO uses descriptors for the selector itself, plus one per registered socket. A Tomcat instance with maxConnections=8192 is structurally capable of holding 8192 connection descriptors plus the selector FDs plus the classpath. The JVM also holds descriptors for JARs on the classpath; on a large Spring deployment this can be several hundred. These are baseline cost, not leaks. They compress headroom under a low ulimit and explain why a Tomcat that “isn’t doing much” still reports a non-trivial OpenFileDescriptorCount at idle.

Common misuses

Alerting on absolute FD count. A count of 5000 is fine on a host with ulimit -n 65535 and fatal on a host with ulimit -n 4096. Alert on the ratio OpenFileDescriptorCount / MaxFileDescriptorCount, not the raw count.
Using lsof | wc -l as the FD count. It overcounts. Use /proc/<pid>/fd or JMX.
Reading the host’s ulimit -n and assuming the container sees the same value. It does not. Read MaxFileDescriptorCount from inside the JVM, or cat /proc/<pid>/limits for the container’s actual limit.
Treating MaxFileDescriptorCount as the hard ceiling. It is the soft limit. If you raised the soft limit at JVM startup, JMX reports the raised value, which is correct, but the hard ceiling is RLIMIT_NOFILE max (bounded by fs.nr_open).
Assuming FD growth is always a leak. Rule out legitimate growth from connection count, log file count, or classpath changes first. Correlate with connectionCount.

Signals to watch in production

Signal	Why it matters	Warning sign
`OpenFileDescriptorCount / MaxFileDescriptorCount` ratio	Primary saturation signal	Sustained above 70%, or any value above 90%
`OpenFileDescriptorCount` growth rate	Leak indicator	Monotonic growth not matched by `connectionCount`
`connectionCount` on `Catalina:type=ThreadPool`	Explains legitimate FD consumption	FDs rising without this rising means suspect a leak
`java.net.SocketException: Too many open files` in logs	The failure signature	Even one occurrence means you hit the cliff
Access log write failures	FD exhaustion corollary	Errors writing to the access log coincide with socket errors
`CLOSE_WAIT` socket count	Dominant leak pattern	`ss -tan state close-wait` count rising

How Netdata helps

Netdata’s Java and Tomcat collectors surface OpenFileDescriptorCount and MaxFileDescriptorCount at per-second resolution alongside connectionCount, thread pool, and GC metrics. The value is correlation across these signals, not any single number:

Plot OpenFileDescriptorCount against connectionCount from the same connector. A leak shows up as FD count climbing while connection count oscillates with traffic.
Plot the FD ratio against currentThreadsBusy and accept queue depth. A simultaneous rise across all three points to a connection-level event; an isolated FD rise points to a leak.
ML anomaly detection flags FD growth that deviates from the instance’s normal pattern, which catches slow leaks before the 70% threshold trips.
Per-second resolution matters because FD exhaustion is a cliff. The warning window between 70% and 100% can be minutes long, and minute-level polling can step over it entirely.
OS-level /proc collectors provide independent corroboration of the JMX count, so you can verify the two views agree rather than trusting one instrument.
Container-aware collection means the FD ratio reflects the runtime’s actual limit, not the host’s, which removes the most common false positive in containerized Tomcat.