ActiveMQ Too many open files: file descriptor exhaustion and refused connections

The broker log shows Too many open files on the accept path, new clients cannot connect, and existing clients may start timing out. In the worst case the broker also fails to open a new KahaDB journal file during rotation, and now you have a store integrity problem that looks, at first glance, like a disk failure. It is not. The broker has run out of file descriptors.

The root cause is usually boring: the broker is running with the default Linux per-process limit of 1024 open files. Every client connection, every KahaDB journal file, every temp store file, every log file, and every network bridge connection consumes one descriptor. A modest production broker blows through 1024 without trying.

The dangerous part is the cliff-edge failure curve. The broker works perfectly at limit-minus-one descriptors and fails abruptly at the limit, refusing connections and failing store writes in the same moment.

What this means

A file descriptor is the kernel handle the broker’s JVM holds for every open socket and every open file. ActiveMQ Classic consumes them in four places:

• Client connections. One socket per TCP connection on each transport connector (OpenWire 61616, AMQP 5672, STOMP 61613, MQTT 1883, WebSocket 61614).
• KahaDB files. One descriptor per db-*.log journal file, plus the db.data index. Journal files accumulate whenever consumption lags production, because a file is only reclaimable when every message in it has been acknowledged. A backlog directly increases FD usage.
• Temp store and logs. Files under data/tmp_storage and the broker’s own log files.
• Network bridges. In a Network of Brokers, each bridge connection is another socket.

When the process hits its limit, three things happen roughly together: the accept loop starts failing (new connections refused), journal rotation or log writes start failing (store corruption risk), and any monitoring or management connection that needs a new socket also fails. That is why the broker often looks “up” in process checks while being functionally dead.

flowchart TD
  A[Client connections] --> D[Broker FD usage]
  B[KahaDB journal files pinned by backlog or DLQ] --> D
  C[Temp store and log files] --> D
  D --> E{FDs at process limit?}
  E -->|no| F[Normal operation]
  E -->|yes| G[Accept failures: new connections refused]
  E -->|yes| H[Journal open/write failures: store corruption risk]
  G --> I[Reconnection storms make FD pressure worse]
  I --> D

Common causes

Quick checks

All of these are read-only and safe to run during an incident.

# 1. Find the broker JVM (if this returns multiple PIDs, pick the java process)
BROKER_PID=$(pgrep -f activemq)

# 2. Current open FDs vs the process limit
echo "Open: $(ls /proc/$BROKER_PID/fd | wc -l)"
grep 'Max open files' /proc/$BROKER_PID/limits

# 3. What the FDs actually are (sockets, files, pipes)
lsof -p $BROKER_PID | awk '{print $5}' | sort | uniq -c | sort -rn | head

# 4. Established client connections on the OpenWire port
ss -tn state established '( sport = :61616 )' | wc -l

# 5. Sockets stuck in CLOSE_WAIT (client vanished, broker still holding)
ss -tn state close-wait '( sport = :61616 )' | wc -l

# 6. KahaDB journal files, each holding an FD
ls /opt/activemq/data/kahadb/db-*.log | wc -l

And via Jolokia, if the management endpoint still has spare FDs to answer you:

# JMX view of FD usage
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/java.lang:type=OperatingSystem/OpenFileDescriptorCount'
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/java.lang:type=OperatingSystem/MaxFileDescriptorCount'

# Broker-side view of connections
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/CurrentConnectionsCount'

If MaxFileDescriptorCount comes back as 1024, you have already found the root cause. Everything else is triage.

How to diagnose it

1. Confirm the limit and the usage. Compare ls /proc/<pid>/fd | wc -l against the Max open files line in /proc/<pid>/limits. If usage is at or near the limit, the symptom is confirmed. Cross-check with OpenFileDescriptorCount and MaxFileDescriptorCount over JMX.

2. Classify the descriptors. Run lsof -p <pid> and group by type. You are answering one question: is this dominated by sockets, or by files? Sockets point at connections. Files under the KahaDB directory point at journal accumulation.

3. If sockets dominate, check connection behavior. Compare the ss established count against the broker’s CurrentConnectionsCount and against your known baseline. A count that only ever goes up is a leak. A stable count with high churn (clients reconnecting in tight loops) shows up in the accept rate and in the broker log. Also count CLOSE_WAIT sockets: a large CLOSE_WAIT population means clients disappeared without a clean close and the broker is still holding the socket.

4. If files dominate, check journal accumulation. Count db-*.log files and compare against baseline, then find what is pinning them: check QueueSize and DLQ depth over JMX. One unacknowledged message pins an entire journal file (32MB by default), and DLQ messages have no TTL by default, so a neglected DLQ is a common way to accumulate hundreds of pinned journal files and the FDs that go with them.

5. Check thread count alongside. With the default TCP transport, each connection also gets a transport thread, so FD exhaustion and thread growth usually travel together. If you are on nio://, threads are pooled and will not track connections; that is expected, not a contradiction.

6. Check the log for store-side damage. Too many open files on the accept path is recoverable. The same error while opening a journal file is not necessarily so. If the broker failed store writes during the exhaustion window, plan for a KahaDB recovery check on the next restart and treat startup time as suspect. See KahaDB corruption after an unclean shutdown if the broker does not come back cleanly.

Metrics and signals to monitor

Threshold guidance from practice: ticket at 70% of the limit, page at 90% sustained with accept failures increasing. During normal operation, open FDs should sit below 50% of the limit so you have headroom for reconnection storms, which are exactly when FD usage spikes hardest.

Fixes

Raise the limit to at least 65536

This is the mandatory fix. The default of 1024 is not a production value for a message broker.

For traditional init or direct startup, set it in /etc/security/limits.conf for the user running the broker:

# /etc/security/limits.conf
activemq  soft  nofile  65536
activemq  hard  nofile  65536

For systemd deployments, limits.conf is ignored for services. Use an override on the unit instead:

# /etc/systemd/system/activemq.service.d/override.conf
[Service]
LimitNOFILE=65536

Applying either requires a broker restart, which is disruptive: plan it, and verify afterward with /proc/<pid>/limits or MaxFileDescriptorCount that the new limit actually took effect. A restart that comes back with 1024 again means the limit was set in the wrong place for how the broker is launched.

Reduce connection pressure

If lsof showed sockets dominating, raising the limit buys time but does not fix the growth. Enforce connection pooling on clients so applications reuse JMS connections instead of opening one per operation, and hunt down any client reconnecting in a tight loop without backoff. For legitimately high connection counts, switch the transport connector from tcp:// to nio:// so connections share a thread pool instead of a thread each. That does not reduce FDs (every connection is still a socket) but it removes the parallel thread-exhaustion failure.

Drain what is pinning journal files

If KahaDB files dominated, the FD problem is a backlog problem. Identify which destinations are holding unacknowledged messages: check per-queue QueueSize, InFlightCount, and DLQ depth. Purge or export the DLQ after investigating why messages landed there, and put a TTL on DLQ messages so the leak cannot recur. See ActiveMQ.DLQ growing and DLQ never expires for those procedures.

Do not just restart as the fix

A restart drops all sockets and closes all files, so it clears the symptom instantly. It also destroys every diagnostic signal, risks KahaDB recovery on a store that may have taken failed writes, and the FDs will grow back to the same ceiling if the cause was a leak or a low limit. Restart only after you have captured lsof output, the journal file count, and the JMX readings above.

Prevention

• Set the limit in the service definition, not just the shell. Bake LimitNOFILE=65536 (or higher, sized to peak connections plus journal files plus headroom) into the systemd unit or init script, and assert it in configuration management so a rebuild cannot silently revert to 1024.
• Alert on FD percentage, not the error string. By the time Too many open files appears in the log, you are at the cliff. Alert at 80% of MaxFileDescriptorCount and track the growth rate; FD count rising without connection growth is a leak and deserves a ticket even at low absolute values.
• Monitor journal file count and DLQ depth as FD-adjacent signals. Store backlog is an FD consumer that most teams never connect to this incident.
• Baseline connection count and churn. Know your normal CurrentConnectionsCount and what a reconnection storm looks like, so a leak stands out.
• Include FD headroom in capacity planning. Peak connections + expected journal files + temp store + logs should stay under 50% of the configured limit.

How Netdata helps

• Netdata charts OpenFileDescriptorCount against MaxFileDescriptorCount from the broker’s JMX java.lang:type=OperatingSystem MBean, so FD pressure is visible as a percentage of the limit rather than a log surprise.
• Per-second collection catches the short spikes that minute-interval polling misses, which matters because reconnection storms can exhaust a marginal FD budget in seconds.
• Correlating FD usage with CurrentConnectionsCount, thread count, and journal file count on one dashboard is what separates “too many clients” from “leaked sockets” from “backlog pinning files” without three separate investigations.
• Broker log alerting on the Too many open files string gives you the accept-failure timestamp to align against the metric timeline.
• Alerts at 70% (ticket) and 90% (page) of the descriptor limit match the escalation model above and fire before clients are refused.

Related guides

• ActiveMQ consumers connected but not acknowledging: the zombie consumer
• ActiveMQ disk full on the KahaDB partition: write failures and store corruption risk
• ActiveMQ.DLQ growing: dead letter queue accumulation and poison messages
• ActiveMQ DLQ never expires: setting TTL so the dead-letter queue stops leaking storage
• ActiveMQ offline durable subscriber pending messages: the silent storage leak
• ActiveMQ enqueue outpacing dequeue: reading the rate imbalance before the backlog
• ActiveMQ expired message count climbing: TTL expiry and silent correctness loss
• How ActiveMQ Classic actually works in production: a mental model for operators
• ActiveMQ InFlightCount high: prefetch full, acks stalled, and zombie consumers
• ActiveMQ JVM heap exhaustion: OutOfMemoryError and the OOM kill
• ActiveMQ KahaDB corruption: the broker won’t start after an unclean shutdown
• ActiveMQ KahaDB db.data index bloat: slow lookups and slow startup recovery