ActiveMQ monitoring maturity model: from survival to expert

Most ActiveMQ outages are not caused by exotic failure modes. They are caused by signals nobody was watching: a DLQ that grew for three weeks, a memory limit that hit 100% and silently blocked every producer, an offline durable subscription that pinned journal files until the disk filled. The signals were available the whole time. The team had not instrumented them yet.

This article lays out a four-level maturity model for monitoring ActiveMQ Classic 5.x. It is a map of which signals to add, in which order, so that each level closes the blind spots the previous level leaves open. Use it to audit your current coverage and to prioritize instrumentation work after your next incident review.

The levels are cumulative. Level 2 assumes everything in Level 1 is in place, and so on. Jumping to Level 4 signals without Level 1 coverage is a common mistake: canary latency means little if you are not also watching store usage.

flowchart TD
  L4["Level 4: Expert - canary, prefetch saturation, churn, JMX latency"]
  L3["Level 3: Mature - message age, per-consumer dispatch, bridge health, store latency"]
  L2["Level 2: Operational - per-queue depth, rates, GC, inflight, FDs"]
  L1["Level 1: Survival - process, memory, store, heap, disk, consumers, DLQ"]
  L4 --> L3 --> L2 --> L1

How to use this model

Read the levels bottom-up. For each signal, ask two questions: do we collect it, and does anything page or ticket on it. Collection without an alert path is a dashboard, not monitoring.

Rules that apply across all levels:

Thresholds are workload-dependent. Queue depth, connection counts, and destination counts have no universal safe values. Establish baselines first, then alert on deviation.
ActiveMQ memory usage is not JVM heap. MemoryPercentUsage is the broker’s internal accounting, a subset of heap. Both must be monitored independently; either one can be exhausted while the other shows headroom.
Store usage is not disk usage. StorePercentUsage tracks against the configured store limit in activemq.xml. If the limit is larger than the physical partition, the OS fills first. Monitor both.
Artemis is different. This model is Classic-specific. Artemis has a different store, memory model, and MBean tree; do not carry these thresholds across.

Level 1: Survival

The minimum: you know the broker is alive, not about to run out of a hard resource, and someone is processing messages. Every signal here maps to a failure that takes the broker down or halts message flow.

Broker process reachability. Check the process and a TCP connect to the OpenWire port (default 61616). Caveat: a listening port does not prove useful service, since the JVM may be stuck in GC or store recovery. In shared-storage HA, do not alert on the standby, whose connectors are intentionally not started.
Broker memory usage percent. At 100%, producer flow control activates and every producer’s send() blocks silently: no exception, no log on the producer side by default. This is the most common ActiveMQ failure pattern, and this gauge is the only reliable warning.
Store usage percent. At 100%, persistent messaging halts. Alert at 80% so you have runway; this resource usually degrades over days, so you get warned if you are watching.
JVM heap usage after GC. The pre-OOM signal. Instantaneous heap fluctuates; heap after a major GC that trends upward is a leak. Above roughly 95% after GC, you are one allocation spike from an OOM kill.
Disk free space on the KahaDB partition. Independent of store usage percent, and it fires first if the store limit exceeds the physical disk. A full partition halts journal writes and risks store corruption.
Consumer count on critical queues. Zero consumers on a queue with traffic means nobody is draining it. Define an expected minimum per critical queue and alert below it.
DLQ depth. Every message in ActiveMQ.DLQ is a failed business transaction, and DLQ messages have no TTL by default, so they accumulate forever. Alert on any unexpected growth, not just large absolute depth.

What Level 1 still misses: which destination is filling memory, whether consumers are processing or zombie-connected, why the store is growing, and anything about latency. A broker can pass all seven checks while a critical queue’s oldest message is an hour old.

Level 2: Operational

Level 2 adds the per-destination and JVM-internal signals that let you answer “why” during an incident instead of just “that”. This is the baseline for a team running ActiveMQ in production with real SLAs.

Per-destination queue depth. Pending messages, including inflight ones. Growth rate matters more than absolute depth; time-to-clear (depth divided by dequeue rate) is a better threshold than a raw count.
Enqueue and dequeue rates. Derived from the cumulative TotalEnqueueCount / TotalDequeueCount counters (and per-destination EnqueueCount / DequeueCount). A sustained positive enqueue-minus-dequeue delta is backlog growth. Watch for the trap: messages moved to the DLQ count as dequeued, so a healthy-looking dequeue rate can hide a DLQ storm.
Per-destination consumer count. Same logic as the Level 1 global check, applied per queue. In a Network of Brokers, demand-forwarding creates virtual consumers that inflate this number.
GC pause duration and frequency. Long pauses freeze every transport thread. The default OpenWire wireFormat.maxInactivityDuration is 30 seconds; a GC pause longer than that disconnects clients, and the reconnection storm that follows makes the next pause worse. Watch for that sawtooth in connection count.
Connection count. Sudden drops indicate partition or GC-driven disconnection; sudden spikes indicate reconnection storms; slow monotonic growth indicates a leak. Baseline it and alert on deviation.
Inflight message count per critical destination. Messages dispatched but not acked. Inflight pinned at consumer_count x prefetch_size with a collapsed dequeue rate means consumers received messages and are not processing them. The zombie-consumer pattern: QueueSize can be zero while InFlightCount is high.
Temp store usage percent. Non-persistent overflow lives here. Many teams do not know the temp store exists until it fills. Sustained non-zero usage already warrants investigation; at 100%, non-persistent messaging is broken.
File descriptor usage. Each connection and each journal file costs an FD. The classic misconfiguration is the default 1024 ulimit; production brokers should run with 65536 or more. Growth without matching connection growth is a leak.
KahaDB journal file count. No JMX metric for this; count db-*.log files on the filesystem. Files are only reclaimed when every message in them is acked, so one unacked message pins a whole 32MB file. Steady growth means consumption lag or pinning.
Expired message count. Each expired message is a lost business event, and by default expired messages go to the DLQ, feeding its growth silently. Ideally zero on critical queues; alert on the expired-to-enqueued ratio.
Redelivery rate. The leading indicator for poison messages and DLQ growth. Redelivery happens before dead-lettering, so this is where you catch a poison-message incident early.

Level 3: Mature

Level 3 is about isolation and leading indicators: narrowing a symptom to the exact destination, consumer, or subscription responsible, and seeing saturation days before it pages.

Per-destination memory usage. On a multi-tenant broker, one noisy destination can consume the shared broker memory pool and trigger flow control that blocks every producer. Per-destination MemoryPercentUsage tells you which one. If you have not set per-destination limits via <policyEntry>, this signal is also your argument for doing so.
Message age on critical queues. The most business-relevant latency signal: a shallow queue with old messages is worse than a deep queue with fresh ones. There is no direct JMX attribute; you derive it by browsing the queue and reading the first message’s JMSTimestamp. Browsing is expensive on deep queues, so sample carefully, and remember the timestamp comes from the producer’s clock.
Per-consumer dispatch metrics. DispatchedQueueSize and MessageCountAwaitingAcknowledge on subscription MBeans identify the specific slow consumer, not just “some consumer is slow”. During an incident this is the difference between minutes and hours.
Durable subscriber pending count. An offline durable subscription accumulates every message published to its topic, forever. Decommissioned dev and test subscribers are a classic silent storage leak. Alert on any offline subscriber with pending messages growing for more than an hour.
Network bridge status and throughput. For NoB deployments: enumerate the per-bridge MBeans (there is no single bridgeCount attribute) and track bridge enqueue/dequeue counters. A bridge can be connected with demand-forwarding broken, leaving messages stranded on one broker and consumers idle on another. Both brokers look healthy in isolation.
Total and temporary destination counts. Each destination creates at least four MBeans. Unbounded growth means dynamic creation without cleanup, and it ends in heap pressure and sluggish JMX. Temp destinations should cycle; a monotonically rising count is a leak, often caused by connection pooling keeping temp destinations alive.
Store write latency. Journal fsync latency is the throughput ceiling for persistent messaging. No broker metric exists for it; measure block-device latency with iostat -x on the KahaDB device. Under 2ms is healthy on SSD, 10-50ms is degraded, above 50ms is critical.
KahaDB index file size. db.data grows with pending messages. Past about 1GB, lookups slow and crash recovery stretches past 30 minutes. Trend it.
Thread count. Roughly 50 plus one to two per connection with the default TCP transport; NIO decouples this. Growth without connection growth is a thread leak.
Authentication failure rate. Sporadic failures are usually a misconfigured client after credential rotation; broad multi-source failure after a change is urgent.
HA role and lock state. For shared-storage HA: alert on split-brain (both brokers active) and on a standby unable to acquire the lock after a failure. Never alert on the standby’s connectors being down.

Level 4: Expert

Level 4 signals are the ones teams add after their third or fourth major incident, when they realize metric-based monitoring has a structural blind spot: a broker can be green on every gauge and still be functionally broken.

Canary message round-trip latency. Continuously produce a test message to a dedicated queue, consume it back, and measure end-to-end latency and success rate. This is the single best broker health indicator because it exercises the entire critical path: transport accept, persistence, dispatch, ack, store cleanup. It catches dispatch bugs, selector misconfigurations, and authorization changes that no gauge will ever show.
Per-message-group depth. With JMS message groups, all messages for a group are pinned to one consumer. A stuck group is invisible in aggregate queue depth because other groups keep draining.
Advisory topic resource consumption. Advisories are real destinations with real MBeans. With thousands of application destinations, advisories can double your destination count and inflate TotalEnqueueCount, distorting your throughput baselines. Measure the overhead; disable advisorySupport where you do not consume advisories.
Prefetch buffer saturation. The inflight-to-prefetch ratio per consumer, as a continuous signal rather than an incident-time check. Sustained saturation near 1.0 predicts the slow-consumer cascade before memory starts climbing.
Connection create/destroy rate. Churn is separate from count. A stable count of 500 connections can hide 100 clients reconnecting every second, burning threads, FDs, and GC. Older client libraries with aggressive no-backoff reconnects are a common source.
Producer count per destination. Anomalous producer activity (a rogue service, a retry loop) shows here before it shows in enqueue rate on busy destinations.
Scheduled and delayed message count. Messages in the scheduler store are invisible to normal queue depth. If you use scheduled delivery, this is a blind spot unless you track it.
MBean count and JMX query latency. JMX is not free. With many destinations, MBean serialization makes queries take seconds, and high-frequency polling itself degrades the broker. Track both so you know when your monitoring is becoming part of the problem.

Moving between levels

The trigger for leveling up is usually an incident the current level could not see. Common patterns map cleanly onto the gaps:

Paged by upstream services hanging with no broker alert: flow control without memory thresholds, a Level 1 gap.
Spent an incident finding which consumer was stuck: you needed per-consumer dispatch metrics, Level 3.
Store filled over weeks and nobody noticed: missing DLQ and journal file trending, Level 1 and 2.
All metrics green but messages not flowing: you needed canary checks, Level 4.

Do not skip ahead. Expert signals assume you trust your basics.

How Netdata helps

The maturity levels above are a collection and correlation problem as much as a signal-list problem:

Netdata’s ActiveMQ collector pulls broker, destination, and JVM metrics over JMX, covering most of Level 1 and 2 out of the box: memory/store/temp usage percent, queue depth, enqueue/dequeue, consumer counts, heap, GC, threads, and FDs.
Per-second collection makes the GC-pause-to-connection-drop sawtooth and reconnection storms visible, where 60-second polling averages them away.
Per-destination charts let you line up depth, inflight, memory, and consumer count for one queue on a single screen, which is exactly the correlation the slow-consumer cascade requires.
Derived-metric alarms support composite conditions, so you can alert on enqueue-minus-dequeue deltas or inflight-pinned-at-prefetch bundles instead of noisy single-gauge thresholds.
Anomaly detection on connection count and destination count catches slow leaks (connection growth, destination explosion) that fixed thresholds miss.