$ guides / activemq ▌

ACTIVEMQ · OPERATIONS PLAYBOOK

ActiveMQ's two silent cliffs: a memory limit that blocks producers without a word, and a store that fills until nothing persists

A JVM message broker where a single memory or store limit reaching 100% stops reading from producer sockets and hangs every send() with no exception, and where KahaDB journal files are pinned by one unacked message until the disk fills. We trace how that design behaves under load, where throttling turns into a hard stop, and what to do when it does.

> Start with the monitoring checklist → # Jump to the full guide list

ActiveMQ's defaults get you to production quickly, then hand you a set of cliff-edges that most teams only discover during an incident.

The defaults work. Until a consumer slows, its prefetch fills, pending messages pile up in broker memory, and MemoryPercentUsage hits 100% — at which point producer flow control engages and every send() blocks silently, with no exception and no log, while upstream services hang. Until the DLQ nobody watches pins journal files, StorePercentUsage reaches 100%, and persistent messaging halts. Until a GC pause longer than wireFormat.maxInactivityDuration makes clients throw InactivityIOException: Channel was inactive for too long and reconnect in a storm. Until the default ulimit of 1024 file descriptors runs out and the broker refuses connections and cannot open journal files.

These guides are written for engineers who already run ActiveMQ Classic 5.x, not for people learning what a queue is. The goal is the mental model of how the broker actually behaves under load, the failure patterns that keep recurring, the monitoring story that catches them before they page anyone, and the runbooks you wish someone had handed you before your last incident.

How ActiveMQ Classic actually runs in production

ActiveMQ is not just a queue. It is a JVM broker where every connection is a transport thread, messages are charged against a memory budget that is separate from the JVM heap, persistence goes through a single KahaDB write-ahead log, and dispatch pushes into consumer prefetch buffers. Most production failures live between these layers, not inside any one of them.

clients + transport connectors

Each client connects over a transport connector — OpenWire (61616), AMQP (5672), STOMP (61613), MQTT (1883), WebSocket (61614). With the default blocking TCP transport, every connection is its own thread and file descriptor; NIO shares a pool. The web console runs embedded Jetty on 8161, bound to localhost by default since 5.16+.

CLIENT

destinations (queues + topics)

The broker holds a registry of destinations. Queues use competing-consumer dispatch — one consumer gets each message. Topics fan out to every active subscriber plus offline durable subscribers' backing queues. Dynamic destination creation without cleanup is how a broker ends up with thousands of destinations and MBean bloat.

ROUTE

memory accounting + flow control

Every pending message is charged against its destination's memory limit and the broker's system memory limit. This <code>memoryUsage</code> accounting is separate from JVM heap. When either limit hits 100%, producer flow control stops reading the producer's socket — the cliff-edge that hangs <code>send()</code> across the broker.

MEMORY

cursors + dispatch

A pending message cursor holds what is waiting to be delivered — store-based for queues, VM for non-durable topics. A dispatch thread pushes messages to each consumer up to its prefetch window (default 1000 for queues, 32767 for topics). High prefetch hides depth in the client, not the broker.

DISPATCH

KahaDB store (journal + index)

Persistent messages are written to a KahaDB journal (<code>db-*.log</code>, 32MB each) and fsync'd before the producer is acked; <code>db.data</code> is the B-tree index. A journal file is reclaimed only when every message in it is acknowledged — one stuck message pins the whole 32MB file, and enough pinned files fill the disk.

STORE

consumers + acks

Consumers process messages and acknowledge them; the ack triggers store cleanup and journal GC. A message dispatched but not yet acked is <code>InFlightCount</code>. Connected does not mean processing — a consumer can hold a full prefetch, ack nothing, and leave the queue draining from the broker's view but stalled in reality.

CONSUME

JVM runtime

Everything runs on one JVM. Heap holds message bodies in cursors, MBeans, and connection state; a GC pause freezes all threads and, past <code>maxInactivityDuration</code>, disconnects clients. Threads scale with connections on blocking TCP; file descriptors cover connections, journal files, and logs. The default 1024 FD limit is the classic footgun.

JVM

topology (Network of Brokers, HA)

A Network of Brokers forwards demand between brokers over directional bridges; shared-storage HA elects one active broker by a store lock while the standby waits. A broken bridge or a split store lock produces symptoms that look local but are not — messages stranded on one broker, or two brokers writing the same store.

TOPOLOGY

Why this matters: 'ActiveMQ is slow' or 'our producers are stuck' can come from broker-wide flow control at the memory limit, a single noisy destination filling the shared pool, a full KahaDB store, a slow-fsync disk, a GC pause disconnecting clients, FD exhaustion, or a Network of Brokers bridge that stopped forwarding. The symptom rhymes but each layer has a different signal — and a different fix.

The failures you'll actually see

Most ActiveMQ incidents fall into a small set of recurring patterns. Recognise the shape, and triage gets dramatically faster.

CRITICAL

The memory-pressure cascade

A consumer slows, its prefetch fills, and pending messages accumulate in broker memory. A destination's memory limit — or the broker's system limit — reaches 100%, and producer flow control engages. The broker stops reading producer sockets and every send() blocks silently, with no exception and no log by default. Upstream services hang on send; retry loops amplify it. This is the number-one ActiveMQ failure pattern.

MemoryPercentUsage at or near 100% (broker or a specific destination)
Producers hung on send() with no error; enqueue rate collapses
ActiveMQ.Advisory.FULL.Queue.* messages being published
Dequeue rate down and InFlightCount pinned on one destination

Investigate →

CRITICAL

The store exhaustion spiral

Consumption lags production — often because an unwatched DLQ is accumulating poison messages. KahaDB journal files pile up because a single unacked message pins each 32MB file, StorePercentUsage creeps up over days or weeks, and at 100% the broker cannot persist and all persistent producers block. A slow burn with a sudden, total ending.

StorePercentUsage climbing steadily toward 100%
KahaDB db-*.log journal file count growing monotonically
DLQ depth rising in step with the journal file count
Disk free on the KahaDB partition declining

Investigate →

CRITICAL

The GC pause death spiral

JVM heap pressure produces a long GC pause that freezes every thread. When it exceeds wireFormat.maxInactivityDuration (default 30s), clients declare the broker dead, throw InactivityIOException: Channel was inactive for too long, and disconnect en masse. The reconnection storm adds heap pressure, which lengthens the next pause. Connection count shows a sawtooth.

JVM heap > 90% after major GC, full-GC duration rising
Clients logging Channel was inactive for too long
Connection count sawtooth: mass drops then reconnect spikes
Transport accept rate spiking during the reconnect storm

Investigate →

ACTIVE

The poison-message / DLQ storm

A message that cannot be processed is rolled back and redelivered up to maximumRedeliveries (default 6), then lands in ActiveMQ.DLQ. Redelivery rate rises, the DLQ grows, and dequeue looks busy while useful completion stalls and message age climbs. The DLQ has no TTL by default, so it becomes a silent storage leak that feeds the store spiral.

Redelivery rate climbing on specific queues
ActiveMQ.DLQ depth growing (each message a failed transaction)
Dequeue rate looks healthy but is partly DLQ transfers
Oldest-message age rising while the queue never drains

Investigate →

IMMINENT

File descriptor exhaustion

Each connection, each KahaDB journal file, and each log file consumes a file descriptor. Left at the default Linux ulimit of 1024 — the single most common production misconfiguration — the broker runs out, refuses new connections, and cannot open journal files or page to disk. The store-can't-open-files symptom is routinely misdiagnosed as a disk problem.

OpenFileDescriptorCount approaching MaxFileDescriptorCount
Too many open files in the broker log
New connections refused while existing ones work
FD count growing without a matching connection increase (leak)

Investigate →

CRITICAL

KahaDB corruption on restart

An unclean shutdown — a kill -9, an OOM kill, or power loss — with writes in flight corrupts the db.data index or a journal file. The broker then refuses to start, or starts only with ignoreMissingJournalfiles and checkForCorruptJournalFiles at the cost of losing messages. On a large store, recovery replay can run for many minutes with the port open but not serving clients.

Broker fails to start after an unclean shutdown
Journal or index corruption errors in the startup log
Port open during recovery but not accepting client connections
Long index rebuild / journal replay on a multi-GB store

Investigate →

Choosing a tool

Best ActiveMQ Monitoring Tools: 10 Ranked for 2026

A ranked review of the tools teams actually shortlist here, what each one is genuinely good at, and how the pricing behaves as you scale.

Read the buyer's guide →

ActiveMQ monitoring maturity levels

ActiveMQ observability works in four practical levels. Each is a complete operation, not a stepping stone. Pick the level that matches how much your broker matters. Most production brokers should land at the second level.

Level 1: Survival

Know that something is wrong

Survival monitoring is the floor. With these signals you can answer one question: is the broker alive and are the two cliff-edges — memory and store — still clear? You will not learn what broke, but you will learn that something broke before users do. Survival is enough for dev brokers and non-critical pipelines.

Broker process up (TCP 61616) Is the broker listening, not just the host up?
Broker MemoryPercentUsage 100% means producer flow control — send() hangs everywhere.
StorePercentUsage 100% means persistent messaging has halted.
JVM heap usage after GC > 95% after GC is an imminent OOM kill.
Disk free on the KahaDB partition A full disk fails journal writes and risks corruption.
Consumer count on critical queues Zero consumers means nobody is processing.
DLQ depth Growing means processing failures are piling up.

Level 2: Operational

Diagnose most incidents on your own

Operational monitoring is what most production brokers should target. Survival tells you something is wrong; operational tells you what. With this coverage your team can usually diagnose an incident on its own: slow consumers, backlogs, flow control, GC disconnects, and FD pressure.

Per-destination QueueSize The canonical are-consumers-keeping-up signal per queue.
Enqueue and dequeue rates The imbalance predicts backlog before depth does.
Per-destination consumer count A live queue with zero consumers is an investigation.
GC pause duration and frequency A pause past maxInactivityDuration disconnects clients.
Connection count Baseline plus anomaly: storms spike, partitions drop.
InFlightCount per critical destination High inflight with low dequeue is a zombie consumer.
TempPercentUsage Non-persistent overflow; a full temp store drops messages.
File descriptor usage Cliff-edge: at the limit, connections and journal writes fail.
KahaDB journal file count Growing count means messages are not fully consumed.
Expired message count Silent correctness loss when TTL beats the consumer.

Level 3: Mature

Catch problems before they become incidents

Mature monitoring catches problems before they wake anyone up. A single noisy destination consuming the shared memory pool, message age creeping past an SLA, an orphaned durable subscriber leaking storage, a Network of Brokers bridge that stopped forwarding. None of these page you on day one — they become incidents on day thirty.

Per-destination MemoryPercentUsage Isolate which destination is filling the shared pool.
Oldest-message age on critical queues Latency the depth number alone cannot show.
Per-consumer inflight and dispatch Which specific consumer is slow or stuck?
Durable subscriber pending count Offline subscribers accumulate messages forever.
Network bridge status and throughput Connected but not forwarding strands messages.
Total and temporary destination count Explosion and request-reply leak detection.
KahaDB store write latency Journal fsync is the persistent-throughput ceiling.
KahaDB db.data index size A large index slows lookups and startup recovery.

Level 4: Expert

Reactive instrumentation after real incidents

Expert signals enter your stack the day after a specific incident proved you needed them. A canary round-trip that proves the broker can actually accept and deliver, a stuck message group invisible in aggregate depth, advisory-topic overhead, prefetch saturation, JMX query latency. Most teams never need every signal here — add the ones your incident history says you do.

Canary message round-trip latency Proves end-to-end health a green metric can hide.
Per-message-group depth A stuck group is invisible in aggregate QueueSize.
Advisory topic resource consumption Each advisory is a real destination with MBeans.
Prefetch buffer saturation per consumer InFlightCount / prefetch approaching 1.0.
Connection create/destroy rate Churn, separate from steady connection count.
MBean count and JMX query latency JMX itself becomes a bottleneck at scale.
Scheduled / delayed message count Scheduler-store messages hide from queue depth.
HA role / store lock state Shared-storage HA: active, standby, or split-brain.

Operating mistakes worth avoiding

The traps ActiveMQ teams keep falling into. Each has a clear, well-known fix. Most teams only learn it after an incident.

Treating the DLQ as a black hole

<code>ActiveMQ.DLQ</code> is the most under-monitored resource. It grows silently, pins journal files, consumes store space, and hides processing failures — and it has no TTL by default, so it accumulates forever. Every DLQ message is a bug report. Alert on any non-zero depth, configure per-destination DLQs so failures are attributable, and set an expiration so it stops leaking storage.

Not knowing producer flow control is silent

When flow control activates, a producer's <code>send()</code> blocks with no exception, no log, and no timeout by default — the upstream service simply hangs. Teams burn hours debugging 'why is our service slow' without realising the broker is blocking producers. Monitor <code>MemoryPercentUsage</code> with a tight threshold and configure <code>sendFailIfNoSpaceAfterTimeout</code> so producers get an exception instead of a silent hang.

Confusing ActiveMQ memoryUsage with JVM heap

They are not the same. ActiveMQ's <code>memoryUsage</code> is internal accounting on top of the heap. The JVM heap can be exhausted (MBeans, index, connection state) while <code>MemoryPercentUsage</code> shows headroom, and vice versa. Never set <code>memoryUsage</code> equal to max heap — keep it around 60-70% — and monitor both independently.

Watching queue depth without message age

Depth is ambiguous: a shallow queue full of hours-old messages is worse than a deep queue of fresh ones. <code>QueueSize</code> also includes inflight messages, so it can look busy while nothing moves. Track oldest-message age (or time-to-clear = depth / dequeue rate) as the SLA signal, not raw count.

Ignoring InFlightCount

Teams watch <code>QueueSize</code> and see an empty queue and assume all is well. But a high <code>InFlightCount</code> means consumers received messages into prefetch and are not acking — a zombie consumer. <code>QueueSize</code> can even read zero while messages are stuck in consumer limbo. Monitor <code>InFlightCount</code> alongside depth and dequeue rate.

Leaving the file-descriptor limit at 1024

The default Linux <code>ulimit</code> of 1024 is the single most common ActiveMQ misconfiguration. Each connection, journal file, and log file costs an FD; at the limit the broker refuses connections and cannot open journal files — often misread as a disk fault. Raise the OS <code>ulimit</code> and the systemd <code>LimitNOFILE</code> to at least 65536, and alert at 80%.

Confusing store limit with disk space

<code>StorePercentUsage</code> tracks the configured <code>storeUsage</code> limit, not the physical disk. Set the limit higher than the disk and the OS fills first — the broker never reaches 100% before writes fail. Set it far lower and the broker blocks with disk to spare. Monitor <code>StorePercentUsage</code> AND actual disk free independently, and reconcile the limit against real capacity.

Never cleaning up durable subscriptions

Offline durable topic subscribers accumulate messages indefinitely. A dev or test subscription that was never unsubscribed becomes a permanent, growing storage leak that pins journal files. There is no automatic expiration by default. Enumerate subscriptions regularly and alert on offline subscribers with a growing pending count.

ActiveMQ runbooks in this section

Each guide is a focused runbook for one symptom or topic. Pick one when you have an incident, or use the categories to learn the area.

▸

Start here

▸

Broker memory and producer flow control

▸

KahaDB store, disk, and journal reclamation

▸

KahaDB corruption, index, and I/O

▸

Queue depth, message age, and throughput

▸

Consumer health, prefetch, and expiry

▸

Dead letter queues, poison messages, and redelivery

▸

JVM heap, GC, threads, and file descriptors

▸

Broker availability, connections, and transports

▸

Network of Brokers and shared-storage HA

▸

Destination and temp-destination management

▸

Authentication, authorization, and exposure

WHERE TO GO NEXT

Setting up ActiveMQ monitoring, or putting out a fire?

If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, jump straight to the symptom that matches what you're seeing.

> Start with the checklist > Back to Operations Guides