ActiveMQ Operations Guides on Netdata

ActiveMQ advisory topics: hidden destinations consuming resources

Mon, 01 Jan 0001 00:00:00 +0000

List every topic on a busy ActiveMQ Classic broker and count how many start with ActiveMQ.Advisory.. On a broker with a thousand queues, you can easily find a thousand or more advisory topics, each a real destination with its own JMX MBeans, each receiving non-persistent messages for broker events.

This is not a bug. Advisory topics are how the broker publishes internal events: connections opening and closing, consumers and producers starting and stopping, destinations hitting their memory limit, messages expiring. Tools and clients can subscribe to observe the broker. The problem is that the default behavior scales with destination count, not with how much of this information anyone consumes, and the cost is paid in MBean count, heap, GC pressure, and inflated throughput counters.

ActiveMQ authentication failures: credential rotation fallout and brute force

Mon, 01 Jan 0001 00:00:00 +0000

Your broker log starts filling with SecurityException: User name [X] or password is invalid or Authentication failed for user. Sometimes it is a slow drip from a single client. Sometimes it is a flood from dozens of source IPs. Both look similar in the log, but they are very different incidents: one is usually a forgotten service after a credential rotation, the other is either a production outage (real consumers and producers dropped) or an active attack.

ActiveMQ authorization denied: authenticated users hitting Not authorized

Mon, 01 Jan 0001 00:00:00 +0000

Your ActiveMQ broker log is filling with lines like User orders-svc is not authorized to write to: queue://PAYMENTS.INBOUND or Not authorized to create: topic://ActiveMQ.Advisory.Connection. The client connected fine. Authentication passed. But every send, consume, or destination creation attempt is rejected.

This is an authorization failure, not an authentication failure. The user proved who they are; the broker is saying they cannot do the specific thing they attempted. That distinction drives the whole diagnosis: you are not chasing bad passwords, you are chasing a mismatch between the user the client authenticated as, the operation it attempted, and the authorization entries the broker loaded.

ActiveMQ broker down: telling a crashed broker from a hung one

Mon, 01 Jan 0001 00:00:00 +0000

Your pager says the ActiveMQ broker is down. The first question is not “how do I restart it” but “what kind of down is this.” A crashed broker (process gone, port closed) and a hung broker (process alive, port possibly open, nothing moving) have different causes, different evidence, and different safe responses. Restarting a hung broker without capturing diagnostics destroys the evidence. Assuming a crashed broker will come straight back up ignores the KahaDB recovery window, where the process runs and the port may even bind, but no client gets served for minutes to hours.

ActiveMQ broker won't start: port conflicts, store recovery, and lock contention

Mon, 01 Jan 0001 00:00:00 +0000

The broker process is running, or maybe it is not, and clients cannot connect. You restart the service, and it still does not come up. Or worse: the process starts, the log goes quiet, and ten minutes later you are still staring at a broker that has not opened its transport connectors.

“Won’t start” in ActiveMQ Classic covers four distinct failure modes that look identical from the outside: something else is holding port 61616 or 8161, KahaDB is replaying a large journal and simply is not done yet, the store is corrupted after an unclean shutdown and the broker is refusing to start, or a store lock in a shared-storage HA pair is held by the wrong broker and this one is waiting forever. Each has a different fix, and applying the wrong one (especially deleting store files or force-clearing locks) can turn a slow start into permanent message loss.

ActiveMQ connection and session leak: clients that never close

Mon, 01 Jan 0001 00:00:00 +0000

The broker’s connection count used to sit around 400. Last month it was 600. This week it is 1,100, and nobody deployed anything new. The JVM thread count and open file descriptor count are climbing in lockstep. Eventually, usually at the worst time, the broker hits its FD limit and starts rejecting every new connection, including the healthy clients.

This is the classic ActiveMQ connection and session leak: clients that open JMS Connections (or Sessions) and never close them. Each leaked connection holds a socket, a file descriptor, one or more transport threads on the default TCP transport, and broker-side state. One JMS Connection can hold many Sessions, and each Session can hold many consumers and producers, so the leak compounds at each layer.

ActiveMQ consumers connected but not acknowledging: the zombie consumer

Mon, 01 Jan 0001 00:00:00 +0000

The queue has consumers. ConsumerCount is exactly where it should be. But dequeue rate has collapsed to near zero, InFlightCount is pinned at the prefetch limit, and the backlog keeps growing. From the broker’s perspective everything is subscribed; from the business’s perspective nothing is being processed.

This is the zombie consumer: a consumer that holds a live connection and a full prefetch buffer but never acknowledges. ActiveMQ dispatched the prefetch window of messages to it, and the broker will not send that consumer more until acks come back. If it is the only consumer, the queue stalls while looking fully staffed.

ActiveMQ CVE-2023-46604: the OpenWire deserialization RCE and how to detect exposure

Mon, 01 Jan 0001 00:00:00 +0000

CVE-2023-46604 is a CVSS 10.0 unauthenticated remote code execution vulnerability in the ActiveMQ Classic OpenWire protocol marshaller. If an attacker can open a TCP connection to your OpenWire transport connector (default port 61616), they can send a crafted packet that causes the broker to instantiate an arbitrary class on the classpath. No credentials are required. It was exploited in the wild before and immediately after disclosure in October 2023, and unpatched, internet-reachable brokers are still being found.

ActiveMQ default credentials and exposed web console: admin/admin on port 8161

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ Classic ships with a web console on port 8161 and well-known default credentials: admin/admin. Combined with a management interface reachable from the network, that equals full broker control for anyone who finds the port: browse and purge queues, move messages, read message bodies, create destinations, and in some versions reach the Jolokia JMX REST API for much worse.

Newer releases have improved the defaults. Since 5.16.x and 5.18.x, the embedded Jetty console binds to 127.0.0.1 and the remote JMX connector is disabled (createConnector="false") out of the box. These defaults are routinely undone: operators reconfigure the bind address for remote management, container images override it, and older brokers never had the protection. A large share of production brokers still expose 8161 with credentials that are in every scanner’s wordlist.

ActiveMQ deserialization errors: ClassNotFoundException and the serializable packages whitelist

Mon, 01 Jan 0001 00:00:00 +0000

ClassNotFoundException, InvalidClassException, or StreamCorruptedException in the ActiveMQ broker log means messages are failing to process. Sometimes this is mundane: a producer and consumer compiled against different versions of a class, or an ObjectMessage payload class that is not on the broker’s classpath. Sometimes it is the signature of a Java deserialization exploit attempt, and the correct response is a security incident, not a config tweak.

The two situations look similar in the log. The difference is in which class names appear, how often, and from where. This guide is about making that call quickly, fixing the benign cases correctly, and locking down the whitelist so the dangerous cases fail closed.

ActiveMQ destination explosion: dynamic destinations, MBean bloat, and GC pressure

Mon, 01 Jan 0001 00:00:00 +0000

An ActiveMQ Classic broker that has been running for months shows a specific signature: heap usage climbs steadily even when message volume is flat, GC pauses get longer and more frequent, the web console and JMX queries feel sluggish, and eventually the broker OOMs or slides into a GC death spiral. Queue depths look normal. Enqueue and dequeue rates look normal. Nothing about the messaging workload explains it.

The explanation is usually the destination registry. ActiveMQ Classic creates a destination automatically the first time a client produces to or consumes from a name that does not exist. If an application generates destination names dynamically (per request ID, per user, per session, per tenant) and nothing removes them, the registry grows without bound. Every destination registers at least four JMX MBeans and holds in-memory structures in the broker’s heap. At thousands of destinations, JMX itself becomes slow. At tens of thousands, the accumulated metadata is a first-order heap consumer, and the broker degrades or dies.

ActiveMQ disk full on the KahaDB partition: write failures and store corruption risk

Mon, 01 Jan 0001 00:00:00 +0000

The KahaDB partition is at 100%. The broker log shows journal write failures, persistent producers have stopped, and you are in the worst ActiveMQ failure mode: the one where freeing space may not be enough, because an in-flight write at the moment the disk filled can leave the journal or index corrupt.

This is not the same incident as StorePercentUsage hitting 100%. StorePercentUsage measures KahaDB against the configured storeUsage limit in activemq.xml. Disk full measures the partition against physical capacity with df. They are independent limits, and either can fire first. If storeUsage is larger than the partition, the OS runs out of space while the broker still thinks it has headroom, and the failure arrives with no warning from any ActiveMQ metric. This guide covers the OS-level case; for the configured-limit case, see ActiveMQ store is full.

ActiveMQ DLQ never expires: setting TTL so the dead-letter queue stops leaking storage

Mon, 01 Jan 0001 00:00:00 +0000

Your ActiveMQ.DLQ has been sitting at some non-zero depth for months. Nobody looks at it because “it’s just the DLQ.” Meanwhile StorePercentUsage creeps up a fraction of a percent a day, the db-*.log journal files keep accumulating, and one morning the broker hits 100% store usage and blocks every persistent producer on the bus.

This is the default behavior of ActiveMQ Classic, not a bug. Messages sent to the dead-letter queue have no TTL. They accumulate forever, and every one of them is a live reference that pins KahaDB journal files against garbage collection. The fix has two parts: configure expiration on the DLQ so the leak stops, and drain-and-investigate the existing backlog instead of just deleting it. Every DLQ message is a failed business transaction; purging blind throws away the evidence.

ActiveMQ enqueue outpacing dequeue: reading the rate imbalance before the backlog

Mon, 01 Jan 0001 00:00:00 +0000

On a healthy ActiveMQ Classic broker, enqueue rate and dequeue rate track each other within normal burst variability. When enqueue outpaces dequeue for more than a few minutes, the broker is accumulating a backlog whether or not QueueSize has moved enough to alarm yet. The rate delta is the leading indicator; QueueSize, MemoryPercentUsage, and StorePercentUsage are the lagging confirmations. If you wait for the backlog gauges to fire, you have already lost your runway.

ActiveMQ expired message count climbing: TTL expiry and silent correctness loss

Mon, 01 Jan 0001 00:00:00 +0000

A queue looks fine. QueueSize is stable, dequeue rate is non-zero, no alerts on memory or store. Then someone asks why a customer order never processed, and you find ExpiredCount on the destination has been climbing for days. Every tick of that counter is a message the broker threw away or shuffled into the DLQ because its TTL ran out before a consumer got to it. Nothing crashed. Nothing paged. The data is just gone.

ActiveMQ GC pause death spiral: long pauses, heartbeat timeouts, and reconnect storms

Mon, 01 Jan 0001 00:00:00 +0000

The broker is up. The port is listening. But clients keep dropping, reconnecting, dropping again, and every cycle makes things worse. Connection count charts show a sawtooth: a sudden drop, a spike, another drop. In the JVM, heap usage sits above 90% after every major GC and full GC pauses are climbing into the multi-second range.

This is the GC pause death spiral, a positive-feedback loop and one of the characteristic failure archetypes of ActiveMQ Classic. Heap pressure produces long GC pauses. A long pause freezes every broker thread, including the ones answering client keepalives. Clients exceed wireFormat.maxInactivityDuration (default 30000 ms) and disconnect. After the pause ends, every client reconnects at once, and each reconnect allocates new connection, session, consumer, and subscription objects. That allocation spike raises heap pressure further, the next GC pause is longer, and more clients time out. The broker oscillates between being frozen in GC and being hammered by reconnect storms until it is effectively down.

ActiveMQ HA failover stuck: standby cannot acquire the store lock

Mon, 01 Jan 0001 00:00:00 +0000

The active broker died. The standby broker’s process is running, the JVM looks healthy, and yet no client can connect anywhere. ss shows nothing listening on 61616 on either node. The failover you designed for is not happening.

This is the shared-storage master/slave stall: the standby is blocked waiting to acquire the lock file in the KahaDB directory, and until it gets that lock it will not start its transport connectors. That behavior is by design. What is not by design is the lock never becoming available. The usual suspects are a stale lock file left behind by a crashed active, an unhealthy NFS lock daemon, or a SAN mount that is unavailable on the standby.

ActiveMQ InactivityIOException: Channel was inactive for too long

Mon, 01 Jan 0001 00:00:00 +0000

Your producers or consumers are dropping JMS connections with this in the stack trace:

org.apache.activemq.transport.InactivityIOException: Channel was inactive for too long

Sometimes it is one client. Sometimes every client on the broker disconnects within the same second and reconnects in a burst. The exception points at the connection, but the connection is almost never the root cause. Something on one side of the wire stopped producing traffic for longer than the OpenWire inactivity timeout, and the other side declared it dead.

ActiveMQ InFlightCount high: prefetch full, acks stalled, and zombie consumers

Mon, 01 Jan 0001 00:00:00 +0000

A queue shows QueueSize near zero, consumers are connected, and nothing is being processed. Dequeue rate is flat, message age is rising upstream, and the only number that looks wrong is InFlightCount, sitting at a suspiciously round value. This is the zombie consumer pattern: the broker dispatched messages into consumer prefetch buffers, and those messages are never coming back as acknowledgments.

Operators misread this state constantly. They watch QueueSize, see an “empty” queue, and conclude the broker is fine while hundreds or thousands of messages sit in consumer limbo: charged against memory, invisible to other consumers, and pinning journal files. This article covers how to read InFlightCount correctly, how to tell a genuinely stuck consumer from one that is high by design, and how to fix it without guessing.

ActiveMQ JVM heap exhaustion: OutOfMemoryError and the OOM kill

Mon, 01 Jan 0001 00:00:00 +0000

The broker was fine an hour ago. Now the process is gone, or it is alive but frozen: clients disconnected, the web console hangs, and the log shows java.lang.OutOfMemoryError or, if it runs in a container, nothing at all because the kernel killed the JVM with SIGKILL (exit code 137, i.e. 128+9).

This is ActiveMQ JVM heap exhaustion. It is distinct from the broker’s internal memoryUsage hitting 100%. That limit triggers producer flow control and blocks send() calls; this one kills or freezes the JVM itself. The confusing part is that the two are independent counters. You can have MemoryPercentUsage at 50% with plenty of headroom while JVM heap is at 97% and the next full GC never finishes. Non-message allocations (destination metadata, MBeans, connection and session state, cursor overhead, transport buffers) live on the heap too, and ActiveMQ’s flow control accounting does not protect them.

ActiveMQ JVM thread count climbing: transport threads, NIO, and thread leaks

Mon, 01 Jan 0001 00:00:00 +0000

The broker’s JVM thread count has been climbing for hours or days. Nothing has broken yet, but the trend only goes one direction, and it ends with context-switch overhead, native memory pressure from thread stacks, and eventually a broker that cannot accept new connections or gets OOM-killed despite a healthy heap.

A rising thread count on ActiveMQ Classic is not one problem. With the default blocking TCP transport, thread count is a proxy for connection count, so growth usually means connections are growing. When thread count climbs without connection growth, you have an actual thread leak, and the diagnostic path is completely different. Telling the two apart takes about two minutes with JMX.

ActiveMQ KahaDB corruption: the broker won't start after an unclean shutdown

Mon, 01 Jan 0001 00:00:00 +0000

The broker was killed (kill -9, power loss, OOM killer, container limit hit) and now it will not come back. Either the process exits during startup, or it sits there with the log stuck partway through KahaDB recovery and port 61616 never accepts a client. The log mentions db.data, a db-*.log journal file, or an IOException about a missing data file.

This failure became rarer after 5.14, but it still happens, and it carries the highest stakes of the classic ActiveMQ failures: every recovery path trades downtime against message loss. The wrong move at 3 a.m. is deleting the store to get the port back, then discovering you wiped the backlog you were supposed to preserve.

ActiveMQ KahaDB db.data index bloat: slow lookups and slow startup recovery

Mon, 01 Jan 0001 00:00:00 +0000

You restart the broker after an unclean shutdown and it sits there for 20, 30, 40 minutes before it accepts a single client connection. Or the broker is running, but persistent message dispatch feels sluggish and disk reads on the KahaDB partition are elevated for no obvious reason. You look at the KahaDB directory and the journal files are not the problem. The problem is db.data, the B-tree index, sitting at multiple gigabytes.

ActiveMQ KahaDB journal files not deleted: one unacked message pinning a 32MB log

Mon, 01 Jan 0001 00:00:00 +0000

Your application queues are empty or nearly empty. Consumers are connected and processing. Yet the KahaDB directory keeps growing, db-*.log files pile up, StorePercentUsage creeps upward, and disk free space trends toward zero. This is the classic KahaDB journal pinning problem, and it confuses operators precisely because the visible queue state looks healthy.

The mechanism is simple and unforgiving: a KahaDB journal file (default 32MB) is only reclaimable when every message stored in that file has been acknowledged. One unacknowledged message anywhere in the file pins the entire file on disk. If that one message is sitting in the DLQ, held by an offline durable subscriber, or stuck in a dead consumer’s prefetch buffer, the file stays, and new files keep being written behind it.

ActiveMQ KahaDB journal write latency: fsync as the persistent-throughput ceiling

Mon, 01 Jan 0001 00:00:00 +0000

A producer is sending persistent messages to ActiveMQ Classic and throughput is far below what the network, CPU, and broker configuration suggest should be possible. Sends are not failing. There are no exceptions. Each send() just takes a few milliseconds, or tens of milliseconds, and the aggregate rate caps out no matter how many producers you add.

This is almost always the KahaDB journal fsync. For persistent messages, ActiveMQ Classic writes the message to the KahaDB write-ahead journal and acknowledges the producer only after the fsync completes. That fsync is the critical path for every persistent send, and its latency sets a hard ceiling on persistent throughput. When the underlying storage gets slow, persistent messaging gets slower in direct proportion to the fsync latency.

ActiveMQ memory limit reached: MemoryPercentUsage at 100% and the flow-control cliff

Mon, 01 Jan 0001 00:00:00 +0000

Your producers have stopped sending. No exceptions, no timeouts, no errors on the producer side. The send() calls just hang, and upstream services pile up threads waiting on the broker. In the broker log you find the line operators search for at 3 a.m.:

Usage Manager Memory Usage ... reached memory limit

MemoryPercentUsage on the Broker MBean is at 100%. This is ActiveMQ Classic’s own memory accounting, not JVM heap, and 100% is not a slowdown. It is a cliff edge. At 99% everything works. At 100% every producer is flow-controlled: the broker stops reading from producer sockets, TCP backpressure builds, and sends block silently until memory frees up.

ActiveMQ MemoryPercentUsage climbing: reading the flow-control leading indicator

Mon, 01 Jan 0001 00:00:00 +0000

Your ActiveMQ broker’s MemoryPercentUsage gauge is at 65% and climbing. Nothing is broken yet. Producers are sending, consumers are consuming, queue depths look tolerable. But the trend line only goes one direction, and you know what happens at 100%: producer flow control activates, every producer’s send() call blocks silently, and upstream services start hanging with no error message anywhere.

This is the right moment to act, and the gauge gives you more information than most operators extract from it. The absolute value tells you where you are. The rate of climb tells you how much time you have. The surrounding signals tell you why. This article is about reading all three before the cliff edge.

ActiveMQ memoryUsage vs JVM heap: the two memory budgets teams confuse

Mon, 01 Jan 0001 00:00:00 +0000

An ActiveMQ Classic broker can hit 100% MemoryPercentUsage and silently block every producer while the JVM heap sits at 40%. The same broker, on a different day, can die from OutOfMemoryError while MemoryPercentUsage shows 50% and every queue-depth dashboard looks calm. Both incidents look contradictory until you understand that ActiveMQ tracks two separate memory budgets, and only one of them is the JVM’s.

Teams monitor one budget, assume it represents the other, and get surprised by whichever one they ignored. This article covers what each budget counts, how they drift apart, how to size them relative to each other, and which signals detect the drift.

ActiveMQ monitoring checklist: the signals every production broker needs

Mon, 01 Jan 0001 00:00:00 +0000

Most ActiveMQ monitoring setups fail in one of two ways. Either they only watch process liveness and queue depth, and discover producer flow control from angry users. Or they export every JMX attribute into a dashboard nobody reads, and the one signal that mattered is buried on page four.

This checklist organizes the signals that matter into four maturity levels, from “is the broker alive” to “can you prove end-to-end message flow works.” Each level lists the signal, where it comes from, and the threshold that makes it actionable. Scope is ActiveMQ Classic (5.x and 6.x Classic stream). Artemis has a different store, memory model, and MBean tree; do not apply these thresholds to it.

ActiveMQ monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Most ActiveMQ outages are not caused by exotic failure modes. They are caused by signals nobody was watching: a DLQ that grew for three weeks, a memory limit that hit 100% and silently blocked every producer, an offline durable subscription that pinned journal files until the disk filled. The signals were available the whole time. The team had not instrumented them yet.

This article lays out a four-level maturity model for monitoring ActiveMQ Classic 5.x. It is a map of which signals to add, in which order, so that each level closes the blind spots the previous level leaves open. Use it to audit your current coverage and to prioritize instrumentation work after your next incident review.

ActiveMQ network bridge connected but not forwarding: broken demand forwarding

Mon, 01 Jan 0001 00:00:00 +0000

The bridge is up. JMX shows the network bridge MBeans, the broker log shows a successful connection to the remote broker, and yet messages produced on broker A sit in the queue while consumers on broker B wait idle. No errors on either side. Each broker’s local dashboard looks healthy.

This is broken demand forwarding in an ActiveMQ Classic Network of Brokers. A network bridge is not a pipe. It forwards messages only when it knows a consumer exists on the remote side, and it learns that through advisory messages. When the bridge is connected but the demand signal is not flowing, you get a silent stall: backlog on one broker, starvation on the other, nothing alarming on a per-broker view.

ActiveMQ network bridge down: a Network of Brokers partition and store-and-forward backlog

Mon, 01 Jan 0001 00:00:00 +0000

In a Network of Brokers, messages cross between ActiveMQ Classic brokers over network bridges. When a bridge drops, the failure is asymmetric: the origin broker keeps accepting messages for destinations whose consumers live on the remote broker, and those messages pile up locally. The remote broker keeps running with consumers connected and nothing to consume. Both brokers look healthy in isolation. Only cross-broker correlation reveals the partition.

The second phase is often worse than the first. When the bridge reconnects, the store-and-forward backlog replays in a burst. The receiving broker gets the entire accumulated backlog at wire speed, which can spike its memory into producer flow control and take down traffic that was fine all along.

ActiveMQ Network of Brokers replay storm: burst forwarding after a bridge reconnect

Mon, 01 Jan 0001 00:00:00 +0000

A network bridge between two ActiveMQ brokers drops. Producers keep sending to the origin broker, and because ActiveMQ networks do reliable store-and-forward, the messages pile up locally. When the bridge reconnects, the accumulated backlog is replayed across the bridge in a burst. The receiving broker, idle a second ago, is now ingesting the backlog at wire speed. Its memory usage spikes, and if the spike reaches 100%, producer flow control activates on the receiver and producers there block silently.

ActiveMQ offline durable subscriber pending messages: the silent storage leak

Mon, 01 Jan 0001 00:00:00 +0000

StorePercentUsage has been climbing for three weeks. Your queues are draining, consumer counts look right, dequeue rates track enqueue rates, and yet the KahaDB directory keeps growing and journal files keep stacking up. Nothing in the queue-depth metrics explains it.

The usual suspect is not a queue at all. It is a durable topic subscription whose subscriber went away and never came back. Every persistent message published to that topic is still being written to the store on behalf of that subscription, and it will keep happening forever: ActiveMQ Classic has no automatic expiration for offline durable subscribers by default.

ActiveMQ oldest message age: the queue latency depth alone cannot show

Mon, 01 Jan 0001 00:00:00 +0000

Every ActiveMQ dashboard has queue depth on it. Almost none show the age of the oldest pending message, which is the number that actually maps to your SLA. A queue holding 50 messages whose head has been waiting 45 minutes is a much bigger problem than a queue holding 50,000 messages whose head is 2 seconds old and draining fast. Depth alone cannot tell you which of those two situations you are looking at.

ActiveMQ per-destination DLQ: IndividualDeadLetterStrategy vs the shared ActiveMQ.DLQ

Mon, 01 Jan 0001 00:00:00 +0000

Out of the box, ActiveMQ Classic routes every undeliverable message from every queue and topic into a single queue: ActiveMQ.DLQ. One noisy destination with a poison-message problem mixes its failures with everyone else’s, and the only way to attribute a message back to its source is to browse the DLQ and inspect the JMSDestination property on each entry. On a busy broker that is slow, expensive, and usually done too late.

ActiveMQ per-destination memory usage: one noisy queue blocking every producer

Mon, 01 Jan 0001 00:00:00 +0000

Every producer on the broker is stuck in send(). Broker-level MemoryPercentUsage reads 100, the broker log shows the usage manager hitting its memory limit, and upstream services are timing out. But when you list queue depths, almost every queue is empty. One queue holds nearly all the pending messages.

That one queue has consumed the broker’s shared memory pool. Every pending message is charged against its destination’s memory accounting and against the broker-wide system memory limit. If either limit is reached, flow control activates. With no per-destination memory limit set, ActiveMQ Classic does not throttle just that queue’s producers: it throttles everyone’s. This is the default behavior, not a bug.

ActiveMQ poison message loop: redelivery storms and DLQ growth

Mon, 01 Jan 0001 00:00:00 +0000

A queue looks busy. Consumers are connected, dequeue counters are moving, and yet the business workflow has stalled: orders are not completing, messages are getting older, and somewhere in the broker a queue called ActiveMQ.DLQ is quietly filling up. This is the poison message loop, and it is one of the most misread failure modes in ActiveMQ Classic because the headline metrics look healthy while useful work has stopped.

The mechanism is simple. A message arrives that the consumer cannot process: a deserialization error, a schema mismatch after a producer deploy, a consumer bug, a downstream dependency returning something the consumer never handles. The consumer rolls back (or the session recovers), and the broker redelivers the message. The client-side RedeliveryPolicy allows this up to maximumRedeliveries, which defaults to 6, before the broker moves the message to the Dead Letter Queue. If the poisoned input is a class of messages rather than a single message, every one of them burns through its redelivery budget, and the broker spends its dispatch capacity retrying failures instead of doing work.

ActiveMQ prefetch limit: unacked hoarding versus idle consumers

Mon, 01 Jan 0001 00:00:00 +0000

You have four consumers on a queue, producers are sending steadily, and yet throughput is a quarter of what you expect. ConsumerCount says 4. QueueSize looks small. DequeueCount is crawling. The broker looks healthy and the consumers look connected, but three of them are doing nothing.

This is the classic prefetch mismatch. The ActiveMQ prefetch limit controls how many messages the broker pushes to a consumer before it requires acknowledgments back. It is a client-side buffer the broker fills eagerly. With the default of 1000 for queues, the first consumer to connect can absorb the entire visible backlog into its prefetch buffer, where those messages are invisible to load balancing, no longer read as pending depth, and pinned against broker memory until they are acked.

ActiveMQ producer flow control: why send() hangs and producers block silently

Mon, 01 Jan 0001 00:00:00 +0000

Your application is healthy. No exceptions, no error logs, no timeouts. But requests are piling up and every thread that touches the message producer is stuck inside send(). Thread dumps show all your producer threads parked in the JMS client, waiting on a socket write that never completes. Nothing is wrong on the client. Nothing looks wrong on the broker either, until you check one number: MemoryPercentUsage is at 100.

ActiveMQ queue with zero consumers: ConsumerCount at zero and a growing backlog

Mon, 01 Jan 0001 00:00:00 +0000

A production queue shows ConsumerCount=0, QueueSize is climbing, and EnqueueCount keeps ticking up while DequeueCount is flat. Nobody is draining the queue. Every message that arrives stays in the broker, charged against destination memory and, for persistent messages, written into the KahaDB journal. Left alone, this ends in one of two places: broker memory hits 100% and producer flow control silently blocks every producer, or the store fills and persistent messaging halts entirely.

ActiveMQ QueueSize growing: reading the broker's backlog gauge correctly

Mon, 01 Jan 0001 00:00:00 +0000

QueueSize is the first number every operator looks at, and the one most often misread. The common mental model is “messages waiting in the queue,” but that is not what the broker reports. QueueSize is a broker-maintained gauge, not enqueued-minus-dequeued, and it includes messages already dispatched to consumers but not yet acknowledged. Two queues showing QueueSize=5000 can be in completely different states, and a queue showing QueueSize=0 can still be in trouble.

ActiveMQ reconnection storm: transport accept spikes and CPU on handshakes

Mon, 01 Jan 0001 00:00:00 +0000

The broker is up. Queues are draining. But the accept rate on the OpenWire connector is running at ten times baseline, JVM thread count and open file descriptors are climbing in a sawtooth, and CPU is pinned even though message throughput is flat or down. Clients are connecting, disconnecting, and reconnecting in a tight loop, and every one of those connections costs the broker real work: a TCP accept, an optional TLS handshake, wire-format negotiation, and a new transport thread.

ActiveMQ redelivery rate climbing: rollbacks, nacks, and retry storms

Mon, 01 Jan 0001 00:00:00 +0000

Your dequeue rate looks busy, but useful work is not getting done. Messages are dispatched, rolled back or nacked, and dispatched again. The broker spends its cycles retrying instead of making progress, and the redelivery counters climb. This is the poison message replay storm pattern in its early stage, and it leads two things you do not want: dead letter queue growth and silent correctness loss.

Redelivery is normal in small doses. A consumer restart, a transient downstream timeout, a transacted session rolled back once: all bump redelivery counters briefly. What matters is a sustained pattern. High dispatch rate combined with an equally high redelivery rate is net-zero progress, and if you only watch dequeue-adjacent throughput the broker can look healthy while nothing completes.

ActiveMQ shared-storage HA split-brain: two brokers holding the store lock

Mon, 01 Jan 0001 00:00:00 +0000

In a shared-storage HA pair, exactly one ActiveMQ Classic broker is supposed to hold the KahaDB store lock and run active. The second broker sits in a polling loop, waiting for the lock, with its transport connectors down. Split-brain is the state where that invariant breaks: both brokers believe they are active, both accept client connections, and both write to the same KahaDB store on shared storage.

Two writers on one store means message duplication, ordering violations, and real risk of journal and index corruption. Clients connected to the “wrong” broker get messages the other broker also delivers. Recovery is not just “restart one node”: you have to decide which broker’s view of the store is authoritative, and the store itself may already be damaged.

ActiveMQ store is full: StorePercentUsage at 100% and persistent messaging halted

Mon, 01 Jan 0001 00:00:00 +0000

StorePercentUsage has hit 100% on the broker MBean and persistent producers have stopped. Their send() calls are either blocked in flow control or being rejected, depending on your destination policy. Non-persistent traffic may still flow, which makes the outage look partial from the outside.

This is the disk-side equivalent of the memory wall. Where MemoryPercentUsage at 100% blocks producers against the configured memory limit, StorePercentUsage at 100% blocks persistent producers against the configured storeUsage limit in activemq.xml. The broker log shows a “Persistent store is Full” message, and producers stall silently unless you configured sendFailIfNoSpaceAfterTimeout.

ActiveMQ store usage climbing: the store exhaustion spiral

Mon, 01 Jan 0001 00:00:00 +0000

StorePercentUsage was 62% last month. Last week it was 74%. This morning it crossed 80% and your alert fired. Queue depths look normal, consumers are connected, dequeue rates look healthy. Nothing is obviously on fire, but the persistent store keeps growing and nobody knows why.

This is the store exhaustion spiral, one of the most common slow-burn failure patterns in ActiveMQ Classic. Unlike the memory-pressure cascade, which develops in minutes, the store spiral develops over days or weeks. Messages accumulate in the KahaDB journal faster than they are acknowledged, journal files pile up on disk, and the store limit creeps closer. When StorePercentUsage reaches 100%, the broker stops accepting persistent messages entirely and every persistent producer blocks. The warning signs were visible for weeks.

ActiveMQ StorePercentUsage vs actual disk space: the limit that fires after the disk is already full

Mon, 01 Jan 0001 00:00:00 +0000

Your broker just stopped accepting persistent messages. Producers are blocked or erroring, the KahaDB partition is at 100% disk usage, and yet your dashboard shows StorePercentUsage at 62%. The alert you set on store usage never fired. This is one of the most common ActiveMQ monitoring traps: StorePercentUsage and actual disk usage are two different counters measured against two different denominators, and if you have not reconciled them, one of them is lying to you.

ActiveMQ temp store full: non-persistent overflow and silent message drops

Mon, 01 Jan 0001 00:00:00 +0000

Non-persistent messages in ActiveMQ Classic live in broker memory. When memory fills, the broker spills them to an on-disk overflow area called the temp store (default data/tmp_storage). When the temp store fills, non-persistent messaging breaks. Depending on your destination policies, it can break in the worst possible way: messages are silently discarded while producers see no error and consumers just see less traffic.

This is one of the least-monitored resources in ActiveMQ. Many teams do not know the temp store exists until it fills. The signal you need is TempPercentUsage on the Broker MBean. At 50% you have a performance problem. At 100% you have a correctness problem.

ActiveMQ temporary destination leak: request-reply queues that never close

Mon, 01 Jan 0001 00:00:00 +0000

You open the broker’s JMX view and the TemporaryQueues attribute lists hundreds or thousands of entries. The count only goes up. Nothing in the application logs looks wrong, request-reply calls mostly work, but the destination count climbs with traffic and never comes back down. That is a temporary destination leak.

Temporary destinations back the classic JMS request-reply pattern: a client creates a TemporaryQueue, sends a request with the temp queue as the reply-to, and waits for the response. In a healthy system these destinations cycle constantly: created, used, deleted. When the count grows monotonically with request volume, something in that lifecycle is broken, and the broker is accumulating MBeans, metadata, and advisory traffic for destinations that will never be used again.

ActiveMQ Too many open files: file descriptor exhaustion and refused connections

Mon, 01 Jan 0001 00:00:00 +0000

The broker log shows Too many open files on the accept path, new clients cannot connect, and existing clients may start timing out. In the worst case the broker also fails to open a new KahaDB journal file during rotation, and now you have a store integrity problem that looks, at first glance, like a disk failure. It is not. The broker has run out of file descriptors.

The root cause is usually boring: the broker is running with the default Linux per-process limit of 1024 open files. Every client connection, every KahaDB journal file, every temp store file, every log file, and every network bridge connection consumes one descriptor. A modest production broker blows through 1024 without trying.

ActiveMQ transport connector not accepting: a live broker that refuses one protocol

Mon, 01 Jan 0001 00:00:00 +0000

The broker process is running. The JVM is up, OpenWire clients on 61616 are producing and consuming, and your process-level health check is green. But the AMQP clients on 5672 cannot connect, and their reconnect loops are filling your application logs.

This partial failure is worse than a full outage in one specific way: most monitoring does not see it. A TCP check against one port, or a process-alive check, tells you nothing about the other four connectors. ActiveMQ Classic runs each transport connector (OpenWire 61616, AMQP 5672, STOMP 61613, MQTT 1883, WebSocket 61614) as its own accept path, and each one can fail independently while the rest of the broker looks healthy.

ActiveMQ.DLQ growing: dead letter queue accumulation and poison messages

Mon, 01 Jan 0001 00:00:00 +0000

The queue named ActiveMQ.DLQ is growing and nobody is consuming from it. That is the default dead letter queue: it has no consumers, no TTL, and no automatic cleanup, so every message that lands there stays there forever.

Every message in the DLQ is two problems at once. It is a failed business transaction that some consumer gave up on after exhausting redeliveries. And it is a storage leak: DLQ messages are never acknowledged, and in KahaDB a single unacknowledged message pins its entire journal file. A DLQ that grows quietly for weeks is a common root cause behind store exhaustion, which eventually halts persistent messaging on the whole broker.

How ActiveMQ Classic actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

Most ActiveMQ incidents are misdiagnosed in the first hour because the operator is looking at the wrong subsystem. A “hung producer” is flow control doing its job. A “slow broker” is fsync latency on the KahaDB device. A “healthy” broker that is losing messages is a full temp store with flow control disabled. Each of these looks like a different problem until you know which subsystem is actually involved.

This article builds the mental model you need before any runbook makes sense: what the broker is doing at any moment, the path a message takes from producer socket to consumer ack, and where each subsystem fails. It covers ActiveMQ Classic 5.x. Artemis has fundamentally different internals (journal-based store, paging instead of flow control, a different MBean tree), and almost nothing here transfers.