$ guides / activemq / activemq-connection-leak ▌

Operations Guides

ActiveMQ connection and session leak: clients that never close

The broker’s connection count used to sit around 400. Last month it was 600. This week it is 1,100, and nobody deployed anything new. The JVM thread count and open file descriptor count are climbing in lockstep. Eventually, usually at the worst time, the broker hits its FD limit and starts rejecting every new connection, including the healthy clients.

This is the classic ActiveMQ connection and session leak: clients that open JMS Connections (or Sessions) and never close them. Each leaked connection holds a socket, a file descriptor, one or more transport threads on the default TCP transport, and broker-side state. One JMS Connection can hold many Sessions, and each Session can hold many consumers and producers, so the leak compounds at each layer.

The signature is a monotonically increasing CurrentConnectionsCount with no matching increase in workload. A reconnection storm shows a sawtooth pattern. Normal growth plateaus. A leak only goes up.

What this means

Every accepted connection on an ActiveMQ Classic transport connector costs the broker real resources:

One file descriptor per client socket. FDs are also consumed by KahaDB journal files, so connections compete with the store for the same limit.
One or two transport threads per connection on the default tcp:// transport. The nio:// transport uses a shared pool and decouples thread count from connection count.
Broker-side state per connection, per session, and per subscription: MBeans, dispatch state, prefetch accounting.

The leak is rarely in the broker. It is almost always in client code: a missing close() in a finally block, a Connection created per message instead of reused, a pool configured so large it looks like a leak, or a framework lifecycle bug where restarting a route or redeploying an app abandons connections the broker still sees as alive. Dead clients that vanished without a clean close are usually reaped by the InactivityMonitor (default wireFormat.maxInactivityDuration of 30000 ms), so connections that persist for hours are connections the broker still believes are in use.

The cascade ends in connection and thread exhaustion: FD or thread limits reached, the broker stops accepting new connections, and every client, including the well-behaved ones, loses service.

flowchart TD
  A[Client never closes Connection or Session] --> B[CurrentConnectionsCount grows monotonically]
  B --> C[FD usage climbs toward ulimit]
  B --> D[Thread count climbs with connections]
  C --> E["Too many open files: accepts fail"]
  D --> F[Context switching, stack memory pressure]
  E --> G[All clients rejected, broker effectively down]
  F --> G

Common causes

Cause	What it looks like	First thing to check
Client code creates a Connection per message and never closes it	Connection count grows roughly linearly with message rate	Client-side send path: is `connection.close()` called in a finally block?
Missing Session/consumer close on an error path	Slower growth; sessions and subscriptions accumulate per connection	Per-destination `ConsumerCount` drifting up without new consumer instances
Connection pool sized too large or never evicting idle entries	Stepwise jumps to a high plateau that never comes back down	Pool configuration: max connections, idle timeout, whether eviction is enabled
Framework lifecycle bug (route restarts, redeploys)	Stepwise jumps at each deploy or route restart	Correlate connection count jumps with deployment or restart timestamps
Temporary destinations pinned by long-lived pooled connections	Temp destination count growing alongside connections	Temp destination trend; temp destinations are only deleted when the creating connection closes
Failover transport holding half-dead connections	Connections persist from client IPs that no longer exist	`ss` output vs. known live client hosts

Quick checks

All read-only. Run against the broker host and its JMX interface.

# Current connection count from the broker MBean
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/CurrentConnectionsCount'

# Cumulative connections since startup (rate = churn, separate from level)
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/TotalConnectionsCount'

# OS view: established connections on the OpenWire port
ss -tn state established '( sport = :61616 )' | wc -l

# Which client IPs hold the most connections (column 5 is the peer address)
ss -tn state established '( sport = :61616 )' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head

# FD usage and limit for the broker process
BROKER_PID=$(pgrep -f activemq)
echo "Open: $(ls /proc/$BROKER_PID/fd | wc -l)"
grep 'Max open files' /proc/$BROKER_PID/limits

# JVM thread count
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/java.lang:type=Threading/ThreadCount'

# FD usage via JMX (useful when /proc access is restricted)
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/java.lang:type=OperatingSystem/OpenFileDescriptorCount'

# Temporary destinations: returns the list of temp-queue MBeans; count the entries
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/search/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=TemporaryQueue,destinationName=*'

Two readings of TotalConnectionsCount a minute apart give the churn rate. High churn with a stable level is a reconnect loop, not a leak. Low churn with a rising level is the leak.

How to diagnose it

Confirm the leak pattern. Pull CurrentConnectionsCount over several hours. A monotonic upward trend with no matching workload growth is the leak. A sawtooth is a reconnection storm, which is a different problem (see the GC pause and inactivity guides below).
Rule out the pool. Ask whether the growth is actually PooledConnectionFactory behavior. The pool keeps idle connections open by design, and by default its eviction thread is disabled, so pooled connections can sit open indefinitely and look exactly like a leak. The distinguishing test: growth that plateaus at a fixed multiple of the pool’s max size per client instance is the pool working as configured; growth that never plateaus is a real leak.
Attribute connections to clients. Use ss grouped by client IP (above) to find which hosts hold the most connections. One host holding hundreds of connections when it should hold a handful is your suspect. Thread names in a thread dump (jstack <pid>) include the client address in the transport thread name, which ties threads back to specific clients.
Check the per-connection fan-out. One Connection can carry many Sessions, each with consumers and producers. If ConsumerCount on your destinations is also drifting up, the leak includes sessions and subscriptions, not just connections. If temp destinations are growing too, suspect request-reply clients whose pooled connections never close, pinning their TemporaryQueues.
Correlate with deploys and restarts. If the count jumps in steps, line the jumps up against deployment, route-restart, or container-reschedule timestamps. A known pattern: stopping a framework route that uses a pooled connection factory can close pooled connections while the listener container immediately creates replacements the pool no longer tracks, leaking one connection per restart cycle.
Measure resource runway. Compare open FDs against the process limit (/proc/<pid>/limits). The default Linux ulimit of 1024 is far too low for a production broker; it should be at least 65536. Above 70 percent of the limit and climbing, you are in the TICKET window. Above 90 percent with accept failures, you are in the PAGE window.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`CurrentConnectionsCount` (Broker MBean)	The primary leak indicator	Monotonic growth without workload growth
`TotalConnectionsCount` (derive rate)	Separates churn from level	High create rate with stable level = reconnect loop, not leak
Per-connector connection counts	Isolates which protocol/port is leaking	One connector growing while others are flat
`OpenFileDescriptorCount` vs. `MaxFileDescriptorCount`	FD exhaustion is how the leak kills the broker	Above 70 percent of limit; any growth not explained by connections
`ThreadCount`	Default TCP transport uses 1-2 threads per connection	Count exceeding baseline (~50 + 1-2 x connections)
`ConsumerCount` per destination	Leaked sessions show up as phantom consumers	Drifting up without new consumer deployments
Temporary queue/topic MBean count	Temp destinations only die with their creating connection	Sustained growth; count above ~100
Connection count deviation from baseline	The alerting condition	Deviation above 50 percent from established baseline

Fixes

Fix the client code

The durable fix is always client-side. Every Connection, Session, MessageProducer, and MessageConsumer must be closed, in a finally block or try-with-resources. Create one Connection per application (or per thread), not per message; JMS Connections are expensive and designed to be long-lived, with Sessions created from them as needed.

Tradeoff: code changes take a release cycle. While waiting, you can mitigate broker-side.

Right-size the connection pool

If the pool is the problem, configure it deliberately: cap max connections to what the application actually needs, set an idle timeout, and enable the pool’s eviction thread so idle pooled connections are actually closed rather than held forever. Make sure the application closes the pooled wrapper, not the underlying physical connection; closing the delegate breaks the pool’s tracking and leaks the physical socket.

Broker-side mitigation

Raise the FD limit to at least 65536. This is a prerequisite for survival, not a fix; it buys runway.
Do not disable the InactivityMonitor to “stop disconnects.” The monitor is what reaps truly dead clients. If clients are being disconnected by it, the right fix is why they stopped sending keepalives, not the monitor.
Restarting the broker clears the leaked connections but is purely symptomatic relief: the leak returns at the same rate. Treat a restart only as an emergency measure when FD exhaustion is imminent, and warn stakeholders it is disruptive.

Tradeoff: any broker-side cap or restart treats symptoms. If the client keeps leaking, you are on a treadmill.

Clean up phantom consumers and temp destinations

If leaked sessions have left phantom consumers or orphaned temp destinations, removing the owning connection (via JMX or the web console) reclaims them, since temp destinations are deleted when their creating connection closes. Verify against the client inventory first so you do not kill a legitimate consumer.

Prevention

Baseline the connection count. Alert on deviation above 50 percent from baseline, and separately on the trend: a sustained positive slope in CurrentConnectionsCount over hours should page someone before the FD limit does.
Alert on FD percentage, not just connection count. FDs are the cliff edge; connections are the leading indicator.
Track churn separately from level. Alert on connection create rate so reconnect loops and leaks are distinguishable at a glance.
Audit pools in code review. Any new service using a pooled JMS connection factory should state its max size, idle timeout, and eviction settings.
Correlate deploys with connection jumps as a standing dashboard annotation, so the framework-lifecycle leak pattern is caught in the first hour.
Keep temp destination count under watch if any application uses request-reply.

How Netdata helps

JMX metric collection tracks CurrentConnectionsCount, TotalConnectionsCount, and per-connector counts continuously, so the monotonic trend is visible long before the FD limit is reached.
FD and thread correlation places OpenFileDescriptorCount, its process limit, and JVM ThreadCount on the same timeline as connection count, which is what separates a connection leak from a journal-file FD leak.
Churn vs. level is visible by graphing the rate derived from the cumulative TotalConnectionsCount next to the current gauge, distinguishing reconnect storms from leaks without manual sampling.
OS-level context from the same agent (per-port established sockets, per-process FD counts, context switch rate) confirms what JMX reports and attributes load to the broker process.
Anomaly detection on the trend flags a slow leak that sits below any static threshold, which is exactly the pattern fixed thresholds miss.