How ActiveMQ Classic actually works in production: a mental model for operators

Most ActiveMQ incidents are misdiagnosed in the first hour because the operator is looking at the wrong subsystem. A “hung producer” is flow control doing its job. A “slow broker” is fsync latency on the KahaDB device. A “healthy” broker that is losing messages is a full temp store with flow control disabled. Each of these looks like a different problem until you know which subsystem is actually involved.

This article builds the mental model you need before any runbook makes sense: what the broker is doing at any moment, the path a message takes from producer socket to consumer ack, and where each subsystem fails. It covers ActiveMQ Classic 5.x. Artemis has fundamentally different internals (journal-based store, paging instead of flow control, a different MBean tree), and almost nothing here transfers.

What it is and why it matters

ActiveMQ Classic is a JVM-based message broker implementing JMS 1.1 plus OpenWire, AMQP, STOMP, and MQTT transports. At any moment it is doing four interdependent things:

1. Transport: accepting and managing client connections.
2. Destination management: routing messages through queues and topics.
3. Message flow: persisting, accounting, cursoring, and dispatching messages. This is the critical path.
4. Storage: writing and garbage-collecting the KahaDB store.

This model matters because the broker’s worst failure modes are cross-subsystem. Slow consumers fill broker memory, memory pressure blocks producers, blocked producers hang upstream services. That cascade is the #1 operational failure pattern, and you cannot interrupt it if you do not know which stage you are looking at.

The four subsystems

Transport

Listener threads accept TCP connections on the configured transport connectors. Defaults: OpenWire on 61616, AMQP on 5672, STOMP on 61613, MQTT on 1883, WebSocket on 61614. Each accepted connection on the default TCP transport gets a dedicated transport thread that reads frames off the wire; the NIO transport (nio://) uses a shared pool instead, which decouples thread count from connection count. The web console runs an embedded Jetty on 8161, bound to 127.0.0.1 by default since 5.16.

Operational consequence: with the default transport, thread count scales roughly linearly with connection count, and each connection also costs a file descriptor. Connection leaks and reconnection storms show up here first, as thread growth and FD pressure, before they show up as messaging problems. Check FD headroom with cat /proc/<broker-pid>/limits against lsof -p <broker-pid> | wc -l.

Destinations

The broker maintains a registry of queues and topics. Queues use competing-consumer dispatch: each message goes to exactly one consumer. Topics fan out to all active subscribers plus the offline queues of durable subscribers. Virtual destinations, composite destinations, and wildcard subscriptions add routing on top.

Every destination is also a resource cost: at least four JMX MBeans each, plus in-memory data structures and store index entries. Dynamic destination creation without cleanup (the per-request or per-user destination antipattern) is a slow-motion failure: thousands of destinations produce MBean explosion, JMX sluggishness, and GC pressure. Advisory topics, enabled by default, are real destinations too and roughly double the count.

Message flow

This is the subsystem that matters most. It gets its own section below.

KahaDB

KahaDB is the default persistent store. Three things live in its directory:

• db.data: the B-tree index mapping message IDs to journal locations (page-file I/O, not memory-mapped).
• db-*.log: sequential journal files, 32 MB each by default.
• lock: the lock file used for shared-storage HA.

The journal is a write-ahead log. Messages hit the journal first; index updates follow. The rule that drives most storage incidents: a journal file is only reclaimable when every message in it has been acknowledged. One unacknowledged message pins an entire 32 MB file. This is why a “stable” DLQ of a few hundred messages can pin hundreds of megabytes of journal files, and why disk reclamation lags behind backlog drains: checkpoint and cleanup run periodically (defaults: checkpoint every 5 s, cleanup every 30 s), not immediately.

A fourth, often forgotten piece: the temp store (data/tmp_storage). Non-persistent messages that overflow memory spill here, under their own configurable limit. When it fills, non-persistent messaging has nowhere to go. Teams that chose non-persistent delivery specifically to avoid disk I/O discover the temp store exists only when it is full.

How it works: the message path

Here is the critical path for a persistent message, end to end:

flowchart LR
  P[Producer] -->|send| T[Transport thread reads off wire]
  T --> J[KahaDB journal write and fsync]
  J -->|ack after fsync| P
  T --> M[Memory accounting vs destination and broker limits]
  M -->|limit reached| FC[Producer flow control: broker stops reading socket]
  M --> C[Pending message cursor]
  C --> D[Dispatch to consumer prefetch]
  D --> A[Consumer ack]
  A --> GC[Journal file GC]

1. Producer sends. The transport thread reads the message off the wire.
2. Persistence. For persistent messages, the broker writes to the KahaDB journal and acknowledges the producer only after the fsync completes. Consequence: journal write latency is the single biggest throughput bottleneck for persistent messaging. Every persistent send blocks on disk, so w_await on the KahaDB device (watch it with iostat -x 1) sets your throughput ceiling.
3. Memory accounting. The message footprint is charged against the destination’s memory limit and the broker’s system memory limit (<systemUsage><memoryUsage> in activemq.xml). If either limit is reached, producer flow control activates: the broker stops reading from that producer’s socket, creating TCP backpressure. This is the single most important operational behavior in ActiveMQ Classic, and the next section is about why.
4. Cursor selection. The message lands on a pending message cursor. Queues default to store-based cursors (messages paged from disk, only a batch held in memory); non-durable topics use VM cursors (messages held in heap). Cursor type determines whether a slow consumer becomes a disk problem or a heap problem. Non-persistent messages on VM cursors bypass the store entirely and are the most common path to memory exhaustion.
5. Dispatch. The dispatch thread pushes messages to consumers up to their prefetch window (default 1000 for queues, 32767 for non-durable topics, 100 for durable topic subscribers). Messages sitting in a consumer’s prefetch buffer are counted in the broker’s QueueSize as inflight: dispatched but not acknowledged.
6. Acknowledgment. Consumer acks flow back, freeing prefetch slots and triggering store cleanup. The KahaDB GC deletes journal files where all messages have been acknowledged, subject to the pinning rule above.

Producer flow control: the behavior that confuses everyone

Flow control deserves its own section because it is the most misread behavior in the broker.

When memory accounting hits a limit, the broker does not reject the message, throw an exception, or log anything on the producer side by default. It stops reading from the producer’s socket. The producer’s send() call blocks, silently, potentially forever. The official documentation warns that this blocking “is sometimes misinterpreted as a ‘hung producer’, when in fact the producer is simply diligently waiting until space is available.” The upstream service hangs, its timeouts and retry loops fire, and the cascade begins. From the application’s side it looks like an application problem. From the broker’s side it looks like success at 100% memory.

Two configuration axes change this behavior, and you need to know which mode your destinations are in:

• sendFailIfNoSpace / sendFailIfNoSpaceAfterTimeout: instead of blocking, the producer gets a ResourceAllocationException. This converts a silent hang into a visible error and is almost always what you want in production.
• producerFlowControl="false": the broker stops blocking producers. Depending on other settings, overflow messages may spool to the temp store or be silently dropped. The failure mode changes from “producers block” to “messages lost without error,” which is worse unless you have made a deliberate choice.

Broker-level memory exhaustion blocks all producers on all destinations; a per-destination limit (via <policyEntry>) contains the block to one destination. Without per-destination limits, one runaway queue can starve every other producer on the broker.

Where it shows up in production

The mental model pays off when you can map a symptom to a subsystem in one step. The characteristic failure archetypes:

1. Memory pressure cascade. Slow consumer, prefetch fills, destination memory limit reached, flow control, blocked producers, hanging upstream services. The #1 pattern.
2. Store exhaustion spiral. Consumption lags production (often DLQ accumulation), journal files grow for days or weeks, store hits 100%, persistent messaging halts.
3. GC pause death spiral. Heap pressure, long pauses, client heartbeat timeouts (default wireFormat.maxInactivityDuration is 30 s), mass disconnection, reconnection storm, more heap pressure.
4. Poison message / DLQ accumulation. Redelivery up to the limit, then DLQ, which has no TTL by default and pins journal files forever.
5. Connection/thread exhaustion. Leaked or churning connections exhaust threads or FDs; the broker stops accepting everyone, including healthy clients.
6. KahaDB corruption. Unclean shutdown with writes in flight; the broker refuses to start or loses messages, and recovery time scales with index and journal size.
7. Network of Brokers partition and replay storm. Bridge drops, messages accumulate locally, bridge reconnects, and the replay burst can push the receiving broker into flow control.
8. Destination explosion. Uncleaned dynamic destinations, MBean explosion, GC pressure, slow degradation.

Tradeoffs and common misuses

ActiveMQ memory usage is not JVM heap. MemoryPercentUsage is the broker’s internal accounting, a subset of heap. You can be at 50% broker memory and 95% heap (MBeans, connection state, non-message objects), or at 100% broker memory with heap headroom (limit misconfigured). Monitor both independently; a typical ratio is the ActiveMQ memory limit at 60-70% of JVM max heap.

Store usage is not disk usage. StorePercentUsage tracks a configured limit in activemq.xml, not physical capacity. If the limit exceeds the disk, the OS fills first. If it is smaller, you hit the limit with free disk. Monitor both.

QueueSize is not backlog. QueueSize includes inflight messages. QueueSize=1000 with InFlightCount=1000 means the broker has dispatched everything and consumers are sitting on unacked prefetches: the queue is “empty” but nothing is being processed. This is the zombie-consumer pattern, invisible if you only watch depth.

Prefetch is a tradeoff, not a tuning knob to max out. Large prefetch improves throughput per consumer but concentrates unacked messages in client memory, inflates broker memory accounting, and worsens the zombie-consumer blast radius when a consumer stalls.

Cursor choice decides your failure mode. Store-based cursors turn consumer lag into journal growth and eventual store exhaustion. VM cursors turn it into heap pressure and flow control. Neither is wrong; they fail differently.

Signals to watch in production

How Netdata helps

• Netdata’s ActiveMQ collector surfaces the broker signals that map directly onto this model: MemoryPercentUsage, StorePercentUsage, TempPercentUsage, per-destination QueueSize, InFlightCount, ConsumerCount, and EnqueueCount/DequeueCount as rates, so the flow-control trigger and its cause appear on one dashboard.
• Correlating enqueue rate against write latency on the KahaDB block device distinguishes “broker is slow” from “disk is slow” immediately, which is the correct first fork for any persistent-throughput complaint.
• Watching InFlightCount next to prefetch configuration and dequeue rate surfaces zombie consumers, which queue depth alone hides.
• Per-second JVM metrics (heap after GC, GC pause duration) alongside connection count reveal the GC-pause/reconnect-storm sawtooth pattern that minute-resolution monitoring misses entirely.
• Disk free on the KahaDB partition tracked independently of StorePercentUsage catches the limit-vs-physical-disk mismatch before the OS fills.

Related guides

• ActiveMQ monitoring checklist: the signals every production broker needs
• ActiveMQ monitoring maturity model: from survival to expert
• ActiveMQ producer flow control: why send() hangs and producers block silently
• ActiveMQ memory limit reached: MemoryPercentUsage at 100% and the flow-control cliff
• ActiveMQ MemoryPercentUsage climbing: reading the flow-control leading indicator
• ActiveMQ per-destination memory usage: one noisy queue blocking every producer
• ActiveMQ memoryUsage vs JVM heap: the two memory budgets teams confuse
• ActiveMQ store is full: StorePercentUsage at 100% and persistent messaging halted
• ActiveMQ store usage climbing: the store exhaustion spiral
• ActiveMQ KahaDB journal files not deleted: one unacked message pinning a 32MB log
• ActiveMQ disk full on the KahaDB partition: write failures and store corruption risk
• ActiveMQ StorePercentUsage vs actual disk space: the limit that fires after the disk is already full