$ guides / activemq / activemq-channel-inactive-too-long ▌

Operations Guides

ActiveMQ InactivityIOException: Channel was inactive for too long

Your producers or consumers are dropping JMS connections with this in the stack trace:

org.apache.activemq.transport.InactivityIOException: Channel was inactive for too long

Sometimes it is one client. Sometimes every client on the broker disconnects within the same second and reconnects in a burst. The exception points at the connection, but the connection is almost never the root cause. Something on one side of the wire stopped producing traffic for longer than the OpenWire inactivity timeout, and the other side declared it dead.

The timeout is wireFormat.maxInactivityDuration, default 30000ms. A stop-the-world GC pause, a network stall, or a hung broker thread that exceeds 30 seconds of silence trips it on every open connection at once. That is why this exception arrives in waves: it is the visible symptom of the GC pause death spiral or a stalled broker, not a client bug.

This guide covers how the inactivity monitor works, how to tell which side stalled, and what to fix before you touch the timeout.

What this means

ActiveMQ Classic runs an InactivityMonitor on each end of an OpenWire connection. Both sides send periodic keep-alive traffic, and both sides watch for inbound traffic. If nothing arrives within maxInactivityDuration, the watching side assumes the peer is dead and closes the connection with InactivityIOException.

Three behaviors matter for diagnosis:

Both ends monitor independently. The client can kill the connection because the broker went quiet, and the broker can kill it because the client went quiet. The exception can appear in client logs, broker logs, or both.
The negotiated value is the shorter one. At connection startup, client and broker negotiate maxInactivityDuration and the initial delay. The shorter duration wins. Changing the timeout on only one side may have no effect.
The timer measures silence, not health. A broker frozen in a 35-second full GC is alive but produces no traffic, so every client declares it dead. When the GC finishes, all of those clients reconnect simultaneously, allocate new sessions and subscriptions, and increase heap pressure, which makes the next GC pause longer. That is the GC pause death spiral, and this exception is its signature.

flowchart TD
  A[Stall on one side: GC pause, network stall, hung thread] --> B[No traffic for maxInactivityDuration 30s default]
  B --> C[Peer fires InactivityIOException and closes connection]
  C --> D[All clients reconnect simultaneously]
  D --> E[New sessions, subscriptions, MBeans allocated]
  E --> F[Heap pressure rises, next GC pause longer]
  F --> A

The exception is a smoke alarm. Treat the timeout as the last thing to tune, after you have found and fixed whatever stopped the traffic.

Common causes

Cause	What it looks like	First thing to check
Broker-side GC pause over 30s	All clients disconnect at once, sawtooth connection count, reconnect burst right after	GC log and `CollectionTime` on the broker JVM
Client-side GC pause	One application disconnects while others stay connected; exception only in that app’s logs	GC log on the client JVM
Network stall or partition	A subset of clients (same subnet, same host, same LB path) drops simultaneously	`ss` on the broker, switch/LB logs, retransmit counters
Hung broker (store stall, disk latency)	Broker process alive, port open, but dispatch and heartbeats stop; persistent enqueue stalls	`iostat -x` on the KahaDB device, thread dump
Slow network with large messages	Exceptions during large message transfer on constrained links	Message sizes on the affected destinations, broker version
Deliberately aggressive tuning	`maxInactivityDuration` set very low (for example for fast failover detection) and tripped by normal jitter	Connection URIs on both client and broker transport connector

One historical note: ActiveMQ 5.3 and 5.4.0 only updated the activity flag after a full message was assembled, so large messages over slow links could trip the monitor mid-transfer. This was fixed in 5.4.2. On any current release, this is not your problem.

Quick checks

All read-only. Run on the broker host unless noted.

# 1. Confirm the exception and see whether it is one client or many
grep -c "InactivityIOException" /opt/activemq/data/activemq.log
grep "InactivityIOException" /opt/activemq/data/activemq.log | tail -20

# 2. Check GC pauses on the broker JVM (the prime suspect)
#    If pgrep returns more than one PID, pass them one at a time
jstat -gcutil $(pgrep -f activemq | head -1) 1000 10
tail -100 /opt/activemq/data/gc.log   # if GC logging is enabled

# 3. Current heap state
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/java.lang:type=Memory/HeapMemoryUsage'

# 4. Connection count: did it drop and spike (sawtooth = GC-driven)?
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/CurrentConnectionsCount'

# 5. Is the broker actually serving, or hung?
ss -tn state established '( sport = :61616 )' | wc -l

# 6. Disk latency on the KahaDB device (a store stall freezes the broker)
df /opt/activemq/data/kahadb/ | awk 'NR==2{print $1}'
iostat -xd 1 5

# 7. What was the broker doing around the disconnect window
grep -i "memory limit\|flow control" /opt/activemq/data/activemq.log | tail -20

Adjust the broker log path and the brokerName in the Jolokia query to your installation. On the affected client, check its own GC log or add GC logging if none exists. A client-side 30-second pause produces the same exception with a healthy broker.

How to diagnose it

Scope the blast radius. Did one client disconnect, a group of clients, or everything? One client points at that client (its GC, its host, its network path). Everything at once points at the broker or shared network. A correlated subset points at shared infrastructure between them.
Align timestamps. Take the exact time of the InactivityIOException and line it up against the broker’s GC log. A full GC spanning the disconnect window is your answer. If GC logging is not enabled on the broker, enable it now (for JDK 11+, -Xlog:gc*:file=gc.log:time) because you will need it the next time this happens. A JVM restart is required to add the flag.
Check for the death spiral pattern. Pull connection count over the incident window. A sudden drop followed by a spike, repeated, with heap usage staying high after GC, confirms the GC pause death spiral. The fix is heap and GC work, not heartbeat tuning.
Rule out a store stall. If GC is clean, check disk latency on the KahaDB partition during the window. Persistent sends block on journal fsync, and a storage stall can freeze broker progress long enough to trip inactivity timers. Look for await and w_await above 50ms sustained.
Rule out flow control. If broker memory hit 100% around the incident, producer flow control blocked sends and the broker stopped reading sockets. That backpressure can cascade into silence on connections. Check MemoryPercentUsage history and the broker log for “memory limit” messages.
Verify the negotiated timeout. Check the client connection URI and the broker transport connector for wireFormat.maxInactivityDuration. The shorter value wins. Confirm the option uses the wireFormat. prefix; without it, it is silently ignored. If clients connect over the HTTP transport rather than OpenWire over TCP, wireFormat.* options do not apply there at all.
Capture a thread dump if the broker hung. If the broker was unresponsive during the window and GC and disk are clean, jstack <pid> during the next occurrence is the single most valuable artifact. jstack pauses the JVM only briefly and is safe on a production broker, but capture several dumps a few seconds apart to distinguish a stuck thread from a busy one. Transport thread names contain client IPs, which identifies which peer was involved. One caution: there is a long-standing deadlock pattern between the Failover transport and the InactivityMonitor worker thread (tracked upstream as AMQ-6313, reported on 5.13.0 and confirmed by users on 5.16.0). If you use failover: URIs, look for the inactivity monitor thread blocked against failover transport locks in the dump.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
GC pause duration and frequency	Pauses over `maxInactivityDuration` (30s default) guarantee mass disconnection	Any pause >10s; full GC more than once per 5 minutes
JVM heap after major GC	The death spiral starts with heap pressure	>85% after GC, or rising floor across GC cycles
Connection count	Sawtooth drop-spike pattern is the fingerprint of GC-driven disconnects	Sudden drop to zero or >50% deviation from baseline
Transport accept rate	Reconnection storm shows here first	Sustained rate >5x normal
Broker MemoryPercentUsage	Flow control at 100% stalls producers and can silence connections	Climbing >80%, or 100% with active producers
Disk I/O latency on KahaDB device	Store stalls freeze persistent messaging and broker progress	`w_await` >10ms sustained, >50ms critical
Enqueue/dequeue rates	Both drop to zero during a broker-wide stall	Simultaneous collapse with connections still established

Fixes

Fix the GC pause (most common root cause)

If the broker is pausing past 30 seconds, the heap is too small, leaking, or badly tuned. Increase -Xmx (requires restart), switch to G1GC for heaps over 4GB, and investigate why heap stays high after GC: too many destinations, messages held in VM cursors, or a leak. The same applies to the application JVM on the client side. Do not mask a 35-second GC pause with a longer heartbeat; the broker is still frozen for 35 seconds and nothing flows during that time.

Fix the store stall

If disk latency on the KahaDB partition spiked, move KahaDB to dedicated storage, away from OS and log I/O. Persistent send throughput is bounded by journal fsync latency, so storage contention shows up first as stalls, then as inactivity timeouts.

Fix the network path

For a correlated subset of clients, investigate the shared hop: load balancer idle timeouts that drop quiet TCP connections, firewall state timeouts, or retransmission-heavy links. A load balancer that silently drops idle connections produces exactly this exception pattern. TCP keepalive (the useKeepAlive transport option, enabled by default) helps, but the LB’s idle timeout must exceed the keep-alive interval or the connection still gets reaped.

Tune the timeout deliberately, last

Only after the underlying stall is fixed (or accepted as unavoidable), adjust the heartbeat:

Set wireFormat.maxInactivityDuration on the client URI, the broker transport connector, or both. The shorter of the two negotiated values wins, and the option needs the wireFormat. prefix or it is ignored.
Raising it (for example to 60000) buys tolerance for jitter but delays detection of genuinely dead peers.
Setting it to 0, or setting useInactivityMonitor=false on the transport, disables the monitor entirely. This is sometimes appropriate on trusted LAN links, but dead connections are then only reaped by TCP, which can take much longer.
wireFormat.maxInactivityDurationInitalDelay (default 10000ms; the misspelling is in the ActiveMQ code) controls the grace period before monitoring starts. Under heavy load, wire format negotiation can exceed this and the broker closes brand-new connections; raise the initial delay, not the main duration, in that case.
On the HTTP transport, wireFormat.* options do not apply. Use useInactivityMonitor=false in the connection URI if the monitor is causing noise there.

Prevention

Enable GC logging on every broker and every JMS client JVM. Without it, every incident like this starts from zero evidence.
Alert on GC pauses approaching, not exceeding, the inactivity timeout. A page threshold tied to pauses over 30s with correlated connection drops catches the death spiral; a ticket at pauses over 1-2s catches the buildup.
Size heap for the workload and keep ActiveMQ’s memoryUsage limit at 60-70% of JVM max heap so the broker’s own accounting and the JVM do not fight.
Keep KahaDB on dedicated, low-latency storage and alert on write latency, not just disk space.
Baseline connection count per broker and alert on deviation. The sawtooth pattern is obvious on a graph and invisible in logs until you grep for it.
If you use failover: client URIs, test reconnection behavior under load before you need it in production, given the known failover/InactivityMonitor deadlock risk.

How Netdata helps

Per-second JVM heap and GC collection metrics on the broker host make long pauses impossible to miss and easy to align with disconnect timestamps.
Connection count and accept-rate charts show the sawtooth drop-and-reconnect pattern that distinguishes GC-driven mass disconnection from a single client failure.
Disk latency per device on the KahaDB partition correlates store stalls with the inactivity timeout window, separating storage problems from JVM problems.
Enqueue/dequeue rate collapse during the incident window confirms whether the broker was actually frozen or only the connection was.
Anomaly detection on connection count catches the reconnection storm even when the absolute count returns to baseline within minutes.