ActiveMQ reconnection storm: transport accept spikes and CPU on handshakes

The broker is up. Queues are draining. But the accept rate on the OpenWire connector is running at ten times baseline, JVM thread count and open file descriptors are climbing in a sawtooth, and CPU is pinned even though message throughput is flat or down. Clients are connecting, disconnecting, and reconnecting in a tight loop, and every one of those connections costs the broker real work: a TCP accept, an optional TLS handshake, wire-format negotiation, and a new transport thread.

This is a reconnection storm. Connection setup is one of the most expensive things ActiveMQ Classic does per unit of useful work, and the broker has essentially no built-in defense against a client that reconnects aggressively. A single misbehaving application can degrade the broker for every other client.

The storm is highly visible once you know which signals to look at, and transport thread names carry the remote peer IP, so finding the noisy client is usually a five-minute job.

What this means

Under normal operation, client connections to ActiveMQ are long-lived. A JMS connection is established once and held for hours or days, so the accept rate on each transport connector hovers near zero most of the time. The working rule of thumb: a sustained accept rate more than 5x baseline is a reconnection storm; zero accepts for more than 30 seconds while the broker is up is a different problem (a hung connector).

During a storm, the connection count often looks deceptively normal. Fifty clients reconnecting every second still shows “about 50 connections” at any instant. What actually moves is the churn: accept rate, thread creation, FD allocation, and CPU spent on handshakes and wire-format negotiation. If TLS is in play, the CPU cost per connection is much higher, because every reconnect pays for a full handshake.

Storms have four common triggers, and they look different:

1. A client crash loop with no backoff. The application connects, hits an error (auth failure, client ID conflict, a bug in its own startup), exits or resets the connection, and retries immediately.
2. A load balancer failover or network flap that drops all client connections at once. Every client reconnects simultaneously.
3. A deployment. Rolling restarts of a large consumer fleet produce a synchronized reconnect wave.
4. The GC pause death spiral. A long GC pause exceeds wireFormat.maxInactivityDuration (default 30000 ms), clients time out en masse, reconnect simultaneously, and the reconnection load increases heap pressure, which lengthens the next GC pause. This variant is self-sustaining and is covered in depth in ActiveMQ GC pause death spiral.

flowchart TD
  A[Trigger: crash loop, LB failover, deploy, GC pause] --> B[Clients disconnect en masse]
  B --> C[All clients reconnect immediately, no backoff]
  C --> D[Accept rate spikes over 5x baseline]
  D --> E[CPU burns on TCP/TLS handshake and wire-format negotiation]
  D --> F[Transport threads and FDs climb]
  E --> G[Message dispatch slows for healthy clients]
  F --> G
  G --> H{Broker resource pressure}
  H -->|heap or GC pressure| I[Longer GC pauses trigger more timeouts]
  I --> B

Common causes

Quick checks

All of these are read-only. Paths and ports assume the common defaults (OpenWire on 61616, Jolokia on 8161); adjust for your install.

# 1. Count established connections on the OpenWire port
ss -tn state established '( dport = :61616 or sport = :61616 )' | wc -l

# 2. Sample inbound connection setup a few times over 30-60s during the storm
ss -tn state syn-recv '( sport = :61616 )' | wc -l

# 3. Current connection count via JMX/Jolokia
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/CurrentConnectionsCount'

# 4. JVM thread count
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/java.lang:type=Threading/ThreadCount'

# 5. Open file descriptors vs limit
BROKER_PID=$(pgrep -f activemq)
echo "Open: $(ls /proc/$BROKER_PID/fd | wc -l)"
grep 'Max open files' /proc/$BROKER_PID/limits

# 6. Broker CPU
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/java.lang:type=OperatingSystem/ProcessCpuLoad'

# 7. Who is connecting? Per-IP view of established connections (peer address column)
ss -tn state established '( sport = :61616 )' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head

# 8. Check for recent GC activity correlating with disconnects
jstat -gcutil $BROKER_PID 1000 10

A note on check 2: sport matches the broker-side local port, which is what you want for inbound client connections. During a real storm you will see a continuous stream of short-lived connections. If syn-recv stays empty but the established count churns, the loop is connect-then-disconnect at the application layer rather than a TCP-level flood, which points at client logic, not network.

How to diagnose it

1. Confirm the storm, not just high connection count. Take two readings of accept-side activity 60 seconds apart (checks 1-3 above, plus TotalConnectionsCount deltas if exposed in your version). A stable connection count with high churn is the signature. Most teams monitor connection count but not connection create/destroy rate, which is exactly the blind spot a storm exploits.

2. Identify the noisy peer. Capture a thread dump while the storm is running:

   # Capture three thread dumps 5s apart
   BROKER_PID=$(pgrep -f activemq)
   for i in 1 2 3; do jstack $BROKER_PID > /tmp/amq-threads-$i.txt; sleep 5; done
   grep -h 'ActiveMQ Transport' /tmp/amq-threads-*.txt | sort | uniq -c | sort -rn | head
   

   Transport thread names carry the remote peer address, so the IP dominating the dump is your noisy client. If one or two IPs account for most transport threads, you have a point problem. If the churn is spread evenly across your whole fleet, you have a systemic trigger (deploy, LB, GC).

3. Classify the trigger. Line the accept-rate timeline up against deploy events, load balancer failover events, and GC pause timestamps. A storm that starts exactly at a full-GC pause and repeats on each subsequent pause is the death spiral variant, and fixing clients will not end it; you have to fix the heap pressure. A storm that starts when one application instance boots is a client bug.

4. Check what the client is failing on. Look in the broker log for the reason connections drop: authentication failures, InvalidClientIDException (duplicate client ID, a classic cause of two instances fighting over one durable identity and bouncing each other), or InactivityIOException. The log line tells you whether the client is being kicked or is leaving on its own.

5. Check the damage to broker resources. Compare thread count, FD usage, and CPU against your baselines. If FD usage is above 70% of the limit, the storm is approaching a hard failure where the broker starts rejecting connections for everyone, including healthy clients.

Metrics and signals to monitor

Fixes

Fix the client backoff (the real fix)

The broker cannot fix this; the loop lives in the client. For clients using the failover transport, verify the reconnect parameters. The failover transport supports exponential backoff (useExponentialBackOff, on by default in current versions), initialReconnectDelay (default 10 ms), maxReconnectDelay (default 30000 ms), and reconnectDelayExponent (default 2.0). With defaults, repeated failures do back off, but only up to 30 seconds between attempts, and maxReconnectAttempts defaults to retrying forever, so a permanently failing client retries indefinitely.

The trap: a client that fails during connection setup (auth failure, client ID conflict) may not be governed by the failover transport’s reconnect backoff at all, depending on the client stack. Some older Spring JMS versions reconnect aggressively with no backoff. If the offending client is Spring-based, check how the connection factory and listener container handle connection loss, and make sure reconnect attempts are delayed and capped. Also check for duplicate client IDs across instances: two instances with the same client ID will kick each other off the broker in an infinite loop.

Stop the bleeding at the broker

• Throttle at the edge. If the noisy client is identifiable by IP and not business-critical, block it at the firewall or security group while you fix it. This is blunt but immediately restores capacity for healthy clients.
• Enforce connection limits. The TCP transport connector supports maximumConnections to cap concurrent connections per connector (commonly 1000 in the Apache distribution default config). This protects the broker from unbounded connection growth, but it is a concurrent-connection cap, not a rate limit, so it will not slow a churn loop by itself.
• Raise FD limits if you are close. If the storm is pushing FD usage toward the limit, the correct ceiling for a production broker is at least 65536, not the default 1024. This buys headroom but does not fix the loop.
• Switch to NIO transport for high connection counts. The default tcp:// connector creates one transport thread per connection; nio:// uses a shared pool, decoupling thread count from connection count. This makes the broker far more tolerant of churn. It is a connector config change requiring a broker restart, so plan it.

If the trigger is the GC death spiral

Client-side fixes will not help because the disconnects are broker-induced. Follow the GC pause death spiral guide: reduce connection load where possible, address heap pressure, tune GC, and only then expect the reconnect waves to stop.

What not to do

Do not restart the broker as a first response. If the storm is client-driven, the reconnect wave will hit the broker the moment it comes back, now compounded with KahaDB recovery. Restart only if broker-side resources (threads, FDs, heap) are already exhausted and the broker is unresponsive, and only after you have identified and throttled the noisy client.

Prevention

• Baseline accept rate and churn, not just connection count. Alert on sustained accept rate >5x baseline. This catches storms in the first minute instead of the first hour.
• Make backoff mandatory in client standards. Any client template or shared library your teams use should set bounded reconnect attempts with exponential backoff and a sane maxReconnectDelay. Failover send timeouts (the timeout option) prevent producers from blocking forever during broker outages, which reduces panic-restart behavior upstream.
• Unique client IDs per instance. Generate client IDs from hostname plus instance ID so two instances can never fight over one identity.
• Stagger deploys. Rolling restarts of consumer fleets should be batched so the reconnect wave is spread over minutes, not seconds.
• Size for churn, not just steady state. FD limits at 65536+, NIO transport where connection counts are high, and thread headroom above the 2x-baseline alert threshold.
• Fix the GC root causes. A broker that never pauses for 30 seconds never triggers the death spiral variant.

How Netdata helps

• Accept rate vs connection count, together. Netdata’s per-second collection makes the churn visible: you see the accept rate spike and the sawtooth in connections that per-minute polling smooths away.
• Correlation across the failure chain. Thread count, FD usage, CPU, and JVM GC metrics are on the same dashboard as broker JMX metrics, so the jump from “accept spike” to “GC-driven or client-driven” is one glance instead of three tools.
• Per-IP and per-process context. System-level socket and process metrics help confirm which remote peers and which local resources (threads, FDs) the storm is consuming.
• ML anomaly detection on churn metrics. Accept rate and thread count have strong baselines; anomaly flags on these fire on the first wave of a storm rather than after a sustained threshold breach.
• GC pause timelines. Overlaying JVM GC collection time with connection drops confirms or rules out the death spiral variant in seconds.

Related guides

• ActiveMQ GC pause death spiral: long pauses, heartbeat timeouts, and reconnect storms
• ActiveMQ InactivityIOException: Channel was inactive for too long
• How ActiveMQ Classic actually works in production: a mental model for operators
• ActiveMQ JVM heap exhaustion: OutOfMemoryError and the OOM kill
• ActiveMQ consumers connected but not acknowledging: the zombie consumer
• ActiveMQ InFlightCount high: prefetch full, acks stalled, and zombie consumers