ActiveMQ monitoring checklist: the signals every production broker needs

Most ActiveMQ monitoring setups fail in one of two ways. Either they only watch process liveness and queue depth, and discover producer flow control from angry users. Or they export every JMX attribute into a dashboard nobody reads, and the one signal that mattered is buried on page four.

This checklist organizes the signals that matter into four maturity levels, from “is the broker alive” to “can you prove end-to-end message flow works.” Each level lists the signal, where it comes from, and the threshold that makes it actionable. Scope is ActiveMQ Classic (5.x and 6.x Classic stream). Artemis has a different store, memory model, and MBean tree; do not apply these thresholds to it.

Use it two ways: as an audit of what you have today, and as a build order for what to add next.

Why queue depth alone is not enough

The four signals teams most often monitor are process up/down, total connections, queue depth, and CPU. None of them catch ActiveMQ’s two most common production failures: producer flow control and store exhaustion.

Flow control is silent by design. When a destination or broker memory limit is reached, the broker stops reading from the producer’s socket. The producer’s send() blocks with no exception, no log line, no timeout. Upstream services hang and operators see a “slow application,” not a full broker. Store exhaustion is the mirror image: KahaDB journal files accumulate for weeks because one unacknowledged message pins an entire 32MB journal file, until persistent messaging halts.

Both failures are visible days in advance if you watch the right counters. The checklist below is built around catching them early.

flowchart TD
  L1[Level 1: Survival - is the broker alive and accepting work]
  L2[Level 2: Operational - throughput, backlog, consumer health]
  L3[Level 3: Mature - per-destination detail, latency, topology]
  L4[Level 4: Expert - synthetic checks, saturation ratios, internals]
  L1 --> L2 --> L3 --> L4

Level 1: survival

The absolute minimum. If you monitor nothing else, monitor these. Each one maps directly to a total-service-loss failure mode.

Two of these need extra care:

Broker memory is not JVM heap. MemoryPercentUsage is ActiveMQ’s internal accounting against its configured memoryUsage limit. JVM heap can be exhausted by MBeans, connection state, and cursor metadata while broker memory shows headroom. Monitor both, and keep the ActiveMQ memory limit at roughly 60-70% of JVM max heap.

Store usage is not disk usage. StorePercentUsage measures against the limit configured in activemq.xml. If that limit is set higher than the physical partition, the OS fills the disk before ActiveMQ’s counter reaches 100%. Watch both independently. This is one of the most common ways teams get surprised.

HA caveat: in a shared-storage pair, the standby broker’s process runs but its transport connectors are not started. Do not alert on the standby’s closed ports.

Level 2: operational

Everything in Level 1, plus the signals that tell you whether the broker is keeping up and where pressure is building. This is the level a competent production team should reach.

Three interpretive rules prevent the most common misreads at this level:

1. QueueSize includes inflight messages. A queue showing QueueSize=1000 with InFlightCount=1000 is drained from the broker’s perspective; all the messages are sitting in consumer prefetch buffers. The inverse is the zombie consumer pattern: QueueSize=0 with high InFlightCount and no dequeue rate means consumers received messages and never acked. Always read depth and inflight together.

2. Dequeue counts DLQ transfers. Messages moved to the dead letter queue increment the source queue’s dequeue counter. A “healthy” dequeue rate can actually be the broker discarding poison messages. Cross-check dequeue against DLQ enqueue rate before trusting it.

3. GC pauses cascade into connection storms. The default OpenWire wireFormat.maxInactivityDuration is 30000ms. A GC pause longer than that disconnects every client, which then reconnects simultaneously, allocating more objects and making the next GC worse. If you see a sawtooth in connection count, look at GC logs first.

On file descriptors: the default Linux ulimit of 1024 is the single most common production misconfiguration. Each client connection, journal file, and log file consumes one. Raise the limit to at least 65536 and alert on open FDs above 70% of it.

Level 3: mature

Everything above, plus per-destination isolation, latency, and topology signals. This level is where you stop reacting to saturation and start seeing it days out.

• Per-destination memory usage. Isolates the noisy destination before it trips broker-wide flow control. Without per-destination limits set via <policyEntry>, one runaway queue can starve every other producer.
• Message age on critical queues. The most business-relevant latency signal: a shallow queue with a two-hour-old oldest message is worse than a deep queue of fresh messages. There is no direct JMX attribute; derive it by browsing the first message and reading JMSTimestamp. Queue browses are expensive on deep queues, so sample, do not poll aggressively. The timestamp comes from the producer’s clock, so NTP skew distorts it.
• Per-consumer dispatch metrics. DispatchedQueueSize and MessageCountAwaitingAcknowledge on subscription MBeans identify which specific consumer is stuck, not just that a consumer is stuck.
• Durable subscriber pending count. Offline durable subscriptions accumulate messages forever. A decommissioned test subscriber that was never unsubscribed is a permanent storage leak. Alert on any offline subscriber with growing pending messages.
• Network bridge status and throughput (Network of Brokers only). Bridge down is a partition; bridge connected with zero throughput is demand-forwarding broken. Both brokers look healthy in isolation while messages pile up on one and consumers idle on the other. Watch store-and-forward replay on reconnect; the burst can push the receiving broker into flow control.
• Total destination count. Each destination creates at least four MBeans. Unbounded growth (usually dynamic destination creation without cleanup) produces JMX sluggishness, then GC pressure, then outage. Alert above 2x expected count. Advisory topics inflate this silently; set advisorySupport="false" where you do not need advisories.
• Temporary destination count. Temp destinations should cycle with request-reply traffic. Monotonic growth is a leak, usually connection pooling keeping connections (and their temp destinations) alive.
• Store write latency. iostat -x on the KahaDB device. Journal fsync latency is the hard ceiling on persistent throughput. Under 2ms is healthy on SSD; sustained writes above 10ms mean investigate.
• KahaDB index file size. Watch db.data. Under 100MB is healthy; above 1GB degrades lookup performance and stretches crash-recovery startup to tens of minutes.
• Thread count. Roughly 50 plus 1-2 per connection with the default TCP transport. Growth without connection growth is a leak. NIO transport decouples threads from connections, so this signal is less useful there.
• Authentication failure rate. From broker logs. Sporadic failures are usually a misconfigured client; broad multi-source failures after a credential change are urgent.
• HA role and lock state (shared-storage HA only). Split-brain, both brokers active, risks store corruption and is a page. A standby that cannot acquire the lock after the active dies means failover is broken.

Level 4: expert

Signals teams add after the third or fourth major incident, when the aggregate metrics looked green but the system was still wrong.

• Canary message round-trip. Continuously send a test message to a canary queue and consume it back, measuring latency and success. This is the single best health signal because a broker can be metric-green and functionally dead (dispatch bug, selector mismatch, authorization change blocking consumers). A TCP connect proves a listener exists; a canary proves the broker works.
• Prefetch buffer saturation. The inflight/prefetch ratio per consumer, approaching 1.0. This is the earliest form of the slow-consumer signal, before memory usage moves.
• Connection create/destroy rate. Churn is invisible in the connection count. A broker can hold a steady 500 connections while 100 clients reconnect every second, burning threads and GC.
• Per-message-group depth. With JMS message groups, one stuck group owner is invisible in aggregate depth.
• Advisory topic resource consumption. Monitoring the monitoring overhead.
• Producer count per destination. Catches anomalous producers (a misconfigured retry loop) before their enqueue rate shows up in aggregates.
• Scheduled message count. Messages in the scheduler store are invisible to normal queue depth.
• MBean count and JMX query latency. JMX polling itself becomes a bottleneck at high destination counts; bulk queries and longer scrape intervals matter.

The thresholds worth committing to memory

These come up in nearly every ActiveMQ incident review:

• MemoryPercentUsage 100% equals producers silently blocked. Page when producers are active.
• StorePercentUsage 100% equals persistent messaging halted. Page on the active role.
• Disk above 90% on the KahaDB partition equals imminent failure, and may fire before StorePercentUsage does.
• Heap above 95% after major GC, repeated, equals OOM imminent.
• GC pause above 30s equals every OpenWire client disconnects (default inactivity timeout).
• Zero consumers on a critical queue with pending messages equals page.
• Any DLQ growth equals a processing failure per message. Ticket, always.
• FDs above 90% of limit with accept failures equals page.

How Netdata helps

The value here is correlation, not collection. The signals above only work in combination:

• Netdata charts JVM heap, GC pause time, and thread count alongside per-broker and per-destination JMX metrics on the same node, so the GC-to-connection-storm cascade shows up as one timeline instead of three tools.
• Broker memory usage, store usage, and temp store usage are collected as distinct signals, which keeps the “ActiveMQ memory is not JVM heap” and “store limit is not disk space” distinctions visible instead of averaged away.
• Per-second collection catches the sawtooth connection pattern and inflight spikes that minute-interval scrapes flatten into normal-looking averages.
• Disk space and block device latency on the KahaDB partition sit on the same dashboard as StorePercentUsage, so you can see the store limit and the physical disk racing each other before either hits the wall.
• Alerting on ConsumerCount per destination, DLQ depth, and ExpiredCount catches the silent-correctness-loss patterns that throughput dashboards never surface.

Related guides

• How ActiveMQ Classic actually works in production: a mental model for operators
• ActiveMQ monitoring maturity model: from survival to expert
• ActiveMQ producer flow control: why send() hangs and producers block silently
• ActiveMQ memory limit reached: MemoryPercentUsage at 100% and the flow-control cliff
• ActiveMQ MemoryPercentUsage climbing: reading the flow-control leading indicator
• ActiveMQ per-destination memory usage: one noisy queue blocking every producer
• ActiveMQ memoryUsage vs JVM heap: the two memory budgets teams confuse
• ActiveMQ store is full: StorePercentUsage at 100% and persistent messaging halted
• ActiveMQ store usage climbing: the store exhaustion spiral
• ActiveMQ KahaDB journal files not deleted: one unacked message pinning a 32MB log
• ActiveMQ disk full on the KahaDB partition: write failures and store corruption risk
• ActiveMQ StorePercentUsage vs actual disk space: the limit that fires after the disk is already full