ActiveMQ InactivityIOException: Channel was inactive for too long

Your producers or consumers are dropping JMS connections with this in the stack trace:

org.apache.activemq.transport.InactivityIOException: Channel was inactive for too long

Sometimes it is one client. Sometimes every client on the broker disconnects within the same second and reconnects in a burst. The exception points at the connection, but the connection is almost never the root cause. Something on one side of the wire stopped producing traffic for longer than the OpenWire inactivity timeout, and the other side declared it dead.

The timeout is wireFormat.maxInactivityDuration, default 30000ms. A stop-the-world GC pause, a network stall, or a hung broker thread that exceeds 30 seconds of silence trips it on every open connection at once. That is why this exception arrives in waves: it is the visible symptom of the GC pause death spiral or a stalled broker, not a client bug.

This guide covers how the inactivity monitor works, how to tell which side stalled, and what to fix before you touch the timeout.

What this means

ActiveMQ Classic runs an InactivityMonitor on each end of an OpenWire connection. Both sides send periodic keep-alive traffic, and both sides watch for inbound traffic. If nothing arrives within maxInactivityDuration, the watching side assumes the peer is dead and closes the connection with InactivityIOException.

Three behaviors matter for diagnosis:

1. Both ends monitor independently. The client can kill the connection because the broker went quiet, and the broker can kill it because the client went quiet. The exception can appear in client logs, broker logs, or both.
2. The negotiated value is the shorter one. At connection startup, client and broker negotiate maxInactivityDuration and the initial delay. The shorter duration wins. Changing the timeout on only one side may have no effect.
3. The timer measures silence, not health. A broker frozen in a 35-second full GC is alive but produces no traffic, so every client declares it dead. When the GC finishes, all of those clients reconnect simultaneously, allocate new sessions and subscriptions, and increase heap pressure, which makes the next GC pause longer. That is the GC pause death spiral, and this exception is its signature.

flowchart TD
  A[Stall on one side: GC pause, network stall, hung thread] --> B[No traffic for maxInactivityDuration 30s default]
  B --> C[Peer fires InactivityIOException and closes connection]
  C --> D[All clients reconnect simultaneously]
  D --> E[New sessions, subscriptions, MBeans allocated]
  E --> F[Heap pressure rises, next GC pause longer]
  F --> A

The exception is a smoke alarm. Treat the timeout as the last thing to tune, after you have found and fixed whatever stopped the traffic.

Common causes

One historical note: before ActiveMQ 5.3.0 the inactivity monitor only updated its activity flag after a full message was assembled, so large messages over slow links could trip the monitor mid-transfer. This was fixed in 5.3.0 (AMQ-2088). On any current release, this is not your problem.

Quick checks

All read-only. Run on the broker host unless noted.

# 1. Confirm the exception and see whether it is one client or many
grep -c "InactivityIOException" /opt/activemq/data/activemq.log
grep "InactivityIOException" /opt/activemq/data/activemq.log | tail -20

# 2. Check GC pauses on the broker JVM (the prime suspect)
#    If pgrep returns more than one PID, pass them one at a time
jstat -gcutil $(pgrep -f activemq | head -1) 1000 10
tail -100 /opt/activemq/data/gc.log   # if GC logging is enabled

# 3. Current heap state
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/java.lang:type=Memory/HeapMemoryUsage'

# 4. Connection count: did it drop and spike (sawtooth = GC-driven)?
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/CurrentConnectionsCount'

# 5. Is the broker actually serving, or hung?
ss -tn state established '( sport = :61616 )' | wc -l

# 6. Disk latency on the KahaDB device (a store stall freezes the broker)
df /opt/activemq/data/kahadb/ | awk 'NR==2{print $1}'
iostat -xd 1 5

# 7. What was the broker doing around the disconnect window
grep -i "memory limit\|flow control" /opt/activemq/data/activemq.log | tail -20

Adjust the broker log path and the brokerName in the Jolokia query to your installation. On the affected client, check its own GC log or add GC logging if none exists. A client-side 30-second pause produces the same exception with a healthy broker.

How to diagnose it

1. Scope the blast radius. Did one client disconnect, a group of clients, or everything? One client points at that client (its GC, its host, its network path). Everything at once points at the broker or shared network. A correlated subset points at shared infrastructure between them.

2. Align timestamps. Take the exact time of the InactivityIOException and line it up against the broker’s GC log. A full GC spanning the disconnect window is your answer. If GC logging is not enabled on the broker, enable it now (for JDK 11+, -Xlog:gc*:file=gc.log:time) because you will need it the next time this happens. A JVM restart is required to add the flag.

3. Check for the death spiral pattern. Pull connection count over the incident window. A sudden drop followed by a spike, repeated, with heap usage staying high after GC, confirms the GC pause death spiral. The fix is heap and GC work, not heartbeat tuning.

4. Rule out a store stall. If GC is clean, check disk latency on the KahaDB partition during the window. Persistent sends block on journal fsync, and a storage stall can freeze broker progress long enough to trip inactivity timers. Look for await and w_await above 50ms sustained.

5. Rule out flow control. If broker memory hit 100% around the incident, producer flow control blocked sends and the broker stopped reading sockets. That backpressure can cascade into silence on connections. Check MemoryPercentUsage history and the broker log for “memory limit” messages.

6. Verify the negotiated timeout. Check the client connection URI and the broker transport connector for wireFormat.maxInactivityDuration. The shorter value wins. Confirm the option uses the wireFormat. prefix; without it, it is silently ignored. If clients connect over the HTTP transport rather than OpenWire over TCP, wireFormat.* options do not apply there at all.

7. Capture a thread dump if the broker hung. If the broker was unresponsive during the window and GC and disk are clean, jstack <pid> during the next occurrence is the single most valuable artifact. jstack pauses the JVM only briefly and is safe on a production broker, but capture several dumps a few seconds apart to distinguish a stuck thread from a busy one. Transport thread names contain client IPs, which identifies which peer was involved. One caution: there is a long-standing deadlock pattern between the Failover transport and the InactivityMonitor worker thread (tracked upstream as AMQ-6313, reported against 5.13.0 and since closed as incomplete with no code fix, so the risk still applies on current 5.x). If you use failover: URIs, look for the inactivity monitor thread blocked against failover transport locks in the dump.

Metrics and signals to monitor

Fixes

Fix the GC pause (most common root cause)

If the broker is pausing past 30 seconds, the heap is too small, leaking, or badly tuned. Increase -Xmx (requires restart), switch to G1GC for heaps over 4GB, and investigate why heap stays high after GC: too many destinations, messages held in VM cursors, or a leak. The same applies to the application JVM on the client side. Do not mask a 35-second GC pause with a longer heartbeat; the broker is still frozen for 35 seconds and nothing flows during that time.

Fix the store stall

If disk latency on the KahaDB partition spiked, move KahaDB to dedicated storage, away from OS and log I/O. Persistent send throughput is bounded by journal fsync latency, so storage contention shows up first as stalls, then as inactivity timeouts.

Fix the network path

For a correlated subset of clients, investigate the shared hop: load balancer idle timeouts that drop quiet TCP connections, firewall state timeouts, or retransmission-heavy links. A load balancer that silently drops idle connections produces exactly this exception pattern. OpenWire keep-alive traffic (the inactivity monitor’s useKeepAlive option, enabled by default) helps, but the LB’s idle timeout must exceed the keep-alive interval or the connection still gets reaped.

Tune the timeout deliberately, last

Only after the underlying stall is fixed (or accepted as unavoidable), adjust the heartbeat:

• Set wireFormat.maxInactivityDuration on the client URI, the broker transport connector, or both. The shorter of the two negotiated values wins, and the option needs the wireFormat. prefix or it is ignored.
• Raising it (for example to 60000) buys tolerance for jitter but delays detection of genuinely dead peers.
• Setting it to 0, or setting useInactivityMonitor=false on the transport, disables the monitor entirely. This is sometimes appropriate on trusted LAN links, but dead connections are then only reaped by TCP, which can take much longer.
• wireFormat.maxInactivityDurationInitalDelay (default 10000ms; the misspelling is in the ActiveMQ code) controls the grace period before monitoring starts. Under heavy load, wire format negotiation can exceed this and the broker closes brand-new connections; raise the initial delay, not the main duration, in that case.
• On the HTTP transport, wireFormat.* options do not apply. Use useInactivityMonitor=false in the connection URI if the monitor is causing noise there.

Prevention

• Enable GC logging on every broker and every JMS client JVM. Without it, every incident like this starts from zero evidence.
• Alert on GC pauses approaching, not exceeding, the inactivity timeout. A page threshold tied to pauses over 30s with correlated connection drops catches the death spiral; a ticket at pauses over 1-2s catches the buildup.
• Size heap for the workload and keep ActiveMQ’s memoryUsage limit at 60-70% of JVM max heap so the broker’s own accounting and the JVM do not fight.
• Keep KahaDB on dedicated, low-latency storage and alert on write latency, not just disk space.
• Baseline connection count per broker and alert on deviation. The sawtooth pattern is obvious on a graph and invisible in logs until you grep for it.
• If you use failover: client URIs, test reconnection behavior under load before you need it in production, given the known failover/InactivityMonitor deadlock risk.

How Netdata helps

• Per-second JVM heap and GC collection metrics on the broker host make long pauses impossible to miss and easy to align with disconnect timestamps.
• Connection count and accept-rate charts show the sawtooth drop-and-reconnect pattern that distinguishes GC-driven mass disconnection from a single client failure.
• Disk latency per device on the KahaDB partition correlates store stalls with the inactivity timeout window, separating storage problems from JVM problems.
• Enqueue/dequeue rate collapse during the incident window confirms whether the broker was actually frozen or only the connection was.
• Anomaly detection on connection count catches the reconnection storm even when the absolute count returns to baseline within minutes.

Related guides

• How ActiveMQ Classic actually works in production: a mental model for operators
• ActiveMQ consumers connected but not acknowledging: the zombie consumer
• ActiveMQ InFlightCount high: prefetch full, acks stalled, and zombie consumers
• ActiveMQ enqueue outpacing dequeue: reading the rate imbalance before the backlog
• ActiveMQ disk full on the KahaDB partition: write failures and store corruption risk
• ActiveMQ KahaDB corruption: the broker won’t start after an unclean shutdown
• ActiveMQ KahaDB db.data index bloat: slow lookups and slow startup recovery
• ActiveMQ KahaDB journal files not deleted: one unacked message pinning a 32MB log
• ActiveMQ.DLQ growing: dead letter queue accumulation and poison messages
• ActiveMQ DLQ never expires: setting TTL so the dead-letter queue stops leaking storage
• ActiveMQ offline durable subscriber pending messages: the silent storage leak
• ActiveMQ expired message count climbing: TTL expiry and silent correctness loss