ActiveMQ JVM thread count climbing: transport threads, NIO, and thread leaks

The broker’s JVM thread count has been climbing for hours or days. Nothing has broken yet, but the trend only goes one direction, and it ends with context-switch overhead, native memory pressure from thread stacks, and eventually a broker that cannot accept new connections or gets OOM-killed despite a healthy heap.

A rising thread count on ActiveMQ Classic is not one problem. With the default blocking TCP transport, thread count is a proxy for connection count, so growth usually means connections are growing. When thread count climbs without connection growth, you have an actual thread leak, and the diagnostic path is completely different. Telling the two apart takes about two minutes with JMX.

This article covers the split, the attribution (transport thread names carry client IPs, which makes blame assignment fast), the known leak patterns, and when switching to the NIO transport is the right structural fix.

What this means

With the default tcp:// transport connector, ActiveMQ Classic creates roughly one to two threads per accepted connection: a transport thread that reads frames off the socket, plus associated task threads. A healthy broker sits at a baseline of about 50 threads (JVM, GC, scheduler, KahaDB, Jetty) plus one to two per connection. So at 400 connections, 850 threads is normal, and 900 is not a leak.

The NIO transport (nio://) decouples threads from connections. Connections are handled by a shared SelectorManager pool instead of a dedicated thread per socket, so thread count stays roughly flat as connections grow. If you are running NIO and threads still climb linearly, that is a strong leak signal, because the normal scaling mechanism is not in play.

The failure curve is gradual, then a cliff. More threads means more context switching and more native memory for stacks (order of 1MB per thread, so 5,000 threads is roughly 5GB of native memory outside the heap). The gradual part degrades broker latency. The cliff arrives when you hit a thread or memory limit and new connections start failing outright.

flowchart TD
  A[ThreadCount climbing] --> B{Connection count climbing too?}
  B -->|Yes, ~1-2 threads per conn| C[Connection growth]
  B -->|No, diverging| D[Thread leak]
  C --> C1[Sawtooth: reconnection storm]
  C --> C2[Monotonic: connection leak]
  D --> D1[Thread dump: group by name prefix]
  C1 --> E[Fix client reconnect backoff or GC pauses]
  C2 --> F[Fix client pooling or connection-per-message code]
  D1 --> G[Match known leak patterns or upgrade]

Common causes

Quick checks

All read-only. Jolokia paths assume the default web console on 8161; adjust the broker name.

# Current thread count and peak
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/java.lang:type=Threading/ThreadCount'
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/java.lang:type=Threading/PeakThreadCount'

# Current connection count across all connectors
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/CurrentConnectionsCount'

# OS-level thread count (sanity check against JMX)
BROKER_PID=$(pgrep -f activemq)
ls /proc/$BROKER_PID/task | wc -l

# Context switch counters: is the gradual degradation already biting?
grep ctxt_switches /proc/$BROKER_PID/status

# Established connections on the OpenWire port, grouped by client IP
# (peer address is column 5 in ss output; column 4 is the broker's own endpoint)
ss -tn state established '( sport = :61616 )' | awk 'NR>1 {print $5}' | sed 's/:[0-9]*$//' | sort | uniq -c | sort -rn | head

Then take a thread dump. It is the single most valuable artifact for this symptom:

# Capture a thread dump (read-only, but expect a brief pause on very large dumps)
jstack $BROKER_PID > /tmp/broker-threads-$(date +%s).txt

# Group threads by name prefix to see what is accumulating
grep '^"' /tmp/broker-threads-*.txt | sed 's/[0-9]\{2,\}.*//' | sort | uniq -c | sort -rn | head -30

# Attribute transport threads by client IP (prefix grouping above strips IPs)
grep '^"ActiveMQ Transport' /tmp/broker-threads-*.txt | \
  grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' | sort | uniq -c | sort -rn | head

Take two dumps a few minutes apart and compare the grouped counts. A leak shows up as a prefix whose count only grows.

How to diagnose it

1. Confirm the trend is real. Compare ThreadCount now against PeakThreadCount and against your monitoring history. Rule out a one-time reconnection burst after a deploy or network event. A leak climbs and never comes back down; a storm oscillates.

2. Correlate with connection count. Pull CurrentConnectionsCount over the same window. If threads and connections rise together at roughly 1-2 threads per connection, this is connection growth, not a thread leak. Go to step 3. If connections are flat while threads climb, skip to step 5.

3. Characterize the connection growth. A sawtooth pattern (drop then spike) means reconnection storms: check GC pauses against the default wireFormat.maxInactivityDuration of 30000ms, and check clients for aggressive reconnect with no backoff (older Spring JMS configurations are a repeat offender). Monotonic growth means clients are opening connections and never closing them.

4. Attribute the connections. Transport thread names embed the client IP, in the form ActiveMQ Transport: tcp:///192.168.86.35:15871@61616. Use the IP extraction command above to find the offending application, then check that application for connection-per-message code or a misconfigured pool. A client that opens a new JMS connection per message and closes it still costs you: the socket sits in TIME_WAIT and the broker-side transport thread lingers in socketRead0.

5. For diverging thread count, profile the thread dump. Group by thread name prefix across two dumps. Match what you find against the known leak patterns: ActiveMQ InactivityMonitor Worker accumulating points at the websocket leak fixed in 5.14.2/5.15.0 (AMQ-6482). ActiveMQ Connection Executor accumulating on a JCA deployment (WildFly, JBoss EAP) points at AMQ-6700, where a failed initial connection leaks the executor thread. Transport threads for connections that never completed wire-format negotiation point at the old negotiation leak fixed in the 5.9 era; on a current version, check whether anything is port-scanning or health-checking 61616 with bare TCP connects that never speak OpenWire, because those half-open sockets still cost a transport thread until reaped.

6. Check what transport you are actually running. If the connector is nio:// and threads still scale with connections, something is off, because NIO decouples the two. NIO is a broker-side setting; client URLs do not control it.

7. Measure the cost while you decide. Watch context switches in /proc/<pid>/status and estimate native stack memory as roughly thread count times 1MB. Above about 1,000 threads you should be actively investigating; above 5,000 you are in the zone where stack memory alone can trigger the OOM killer with a healthy heap.

Metrics and signals to monitor

Fixes

Reconnection storm

Fix the reason clients disconnect. If GC pauses exceed the inactivity timeout, tune the heap and collector first; the reconnect storm is a symptom, not the disease. On the client side, enforce reconnect backoff. Some older client libraries reconnect in tight loops and will hammer the accept path indefinitely.

Client connection leak or connection-per-message

Fix the client. Reuse JMS connections (they are heavy, thread-spawning objects; sessions and producers are the lightweight parts), or use a properly bounded PooledConnectionFactory. Pooled connections staying open when idle is normal and fine. There is no broker-side fix that makes careless connection usage cheap; you can only move the cost around.

Broker-side thread leaks

Match the leak pattern to the fixed versions and upgrade. The websocket InactivityMonitor leak is fixed in 5.14.2/5.15.0. The wire-format negotiation leak is fixed from 5.9 onward, so if you see it on a current version, look at what is connecting to the port (scanners, dumb TCP health checks) and whether those half-open connections are ever reaped. For the JCA executor leak (AMQ-6700), check whether your deployed version carries the fix and whether the resource adapter is hitting initial connection failures. A broker restart clears accumulated threads but is temporary if the trigger remains. There is also a reported but unresolved pattern on 5.18.3 with BrokerService Task-* and InactivityMonitor threads growing under NIO; if you match that, capture thread dumps and connection-count history before restarting, and treat it as an open investigation.

Structural fix: move to NIO

If the honest answer is “we legitimately have thousands of connections,” switch the transport connector from tcp:// to nio://. This replaces thread-per-connection with a shared SelectorManager pool, so thread count stops scaling with connections. Since 5.15.0 the pool is tunable via system properties for core size, max size, queue capacity, and rejection behavior. Tradeoffs: the failure mode changes from thread exhaustion to pool saturation and accept backpressure, which you must monitor differently, and per-connection read behavior under a shared pool differs from dedicated threads, so test with your workload. Client URLs do not need to change.

Prevention

• Alert on the divergence, not just the number. Track ThreadCount and CurrentConnectionsCount on the same dashboard. Ticket when thread count exceeds 2x baseline; treat “threads climbing without connections” as a leak alarm regardless of absolute value.
• Baseline your ratio. Know your normal threads-per-connection figure (1-2 on TCP transport, roughly flat on NIO) so deviation is obvious.
• Track connection churn, not just connection count. Create/destroy rate catches reconnect loops while the absolute count still looks fine.
• Keep the broker patched. Multiple thread-leak classes are fixed bugs. Running old 5.x means re-experiencing solved problems.
• Use NIO for high connection counts as the default posture, with pool tuning reviewed under load.
• Thread dumps on demand. Wire jstack capture into your incident runbook before you need it at 3 a.m. Two dumps five minutes apart, grouped by thread name, answer most of these incidents.

How Netdata helps

• Netdata collects JVM thread count alongside connection count, so the key correlation (threads vs connections diverging) is visible on one screen without manual JMX polling.
• Per-second collection catches reconnection storms that minute-resolution monitoring flattens into an unremarkable average connection count.
• GC pause and heap metrics sit next to thread metrics, making the GC-pause-to-mass-reconnect cascade directly observable as a timeline.
• Host context switch rate and CPU correlate with thread count, showing the gradual degradation before any hard limit is hit.
• Anomaly detection on thread count flags slow leaks that stay under static thresholds for weeks before becoming a problem.

Related guides

• ActiveMQ JVM heap exhaustion: OutOfMemoryError and the OOM kill
• ActiveMQ consumers connected but not acknowledging: the zombie consumer
• ActiveMQ InFlightCount high: prefetch full, acks stalled, and zombie consumers
• ActiveMQ enqueue outpacing dequeue: reading the rate imbalance before the backlog
• How ActiveMQ Classic actually works in production: a mental model for operators
• ActiveMQ.DLQ growing: dead letter queue accumulation and poison messages
• ActiveMQ disk full on the KahaDB partition: write failures and store corruption risk
• ActiveMQ KahaDB db.data index bloat: slow lookups and slow startup recovery