ActiveMQ temp store full: non-persistent overflow and silent message drops

Non-persistent messages in ActiveMQ Classic live in broker memory. When memory fills, the broker spills them to an on-disk overflow area called the temp store (default data/tmp_storage). When the temp store fills, non-persistent messaging breaks. Depending on your destination policies, it can break in the worst possible way: messages are silently discarded while producers see no error and consumers just see less traffic.

This is one of the least-monitored resources in ActiveMQ. Many teams do not know the temp store exists until it fills. The signal you need is TempPercentUsage on the Broker MBean. At 50% you have a performance problem. At 100% you have a correctness problem.

If you chose non-persistent delivery to avoid disk I/O latency, a broker spilling to the temp store has already taken that benefit away, quietly, before anything actually failed.

What this means

The non-persistent message path has three capacity stages:

1. Memory. Non-persistent messages are charged against the destination memory limit and the broker system memory limit, same as persistent ones.
2. Temp store. When memory is exhausted, the broker swaps non-persistent messages to the temp store under data/tmp_storage. This store has its own configurable limit (<tempUsage>), often small or never reviewed.
3. Full. When the temp store limit is reached, the broker has nowhere to put non-persistent messages. What happens next depends entirely on your flow control configuration:
   • With producerFlowControl="true" (the default), producers block. Painful but safe: no data loss, just hangs.
   • With producerFlowControl="false" and no sendFailIfNoSpace, messages are dropped. Producers are not notified. This is the silent-loss path.

flowchart TD
  A[Non-persistent messages arrive] --> B{Memory limit reached?}
  B -->|No| C[Held in memory, dispatched normally]
  B -->|Yes| D[Spill to temp store data/tmp_storage]
  D --> E{Temp store limit reached?}
  E -->|No| F[Paged back to memory for dispatch]
  E -->|Yes| G{producerFlowControl?}
  G -->|true default| H[Producers block on send - no data loss]
  G -->|"false, no sendFailIfNoSpace"| I[Messages silently dropped]

What makes this failure mode nasty:

• The temp store limit is independent of memory and store limits. Raising memoryUsage does not help once spill is happening; it can make things worse if the temp store is smaller than the memory that feeds it.
• The temp store usually shares a partition with KahaDB. Temp store growth competes with journal files for the same disk. A temp store runaway can become a disk-full incident for persistent messaging too.
• Temp store usage can stay above 100% even after traffic stops. If tempUsage is set below the size of a single journal file the temp store creates (32 MB by default), usage can sit above 100% permanently and producers stay flow-controlled forever. Never configure tempUsage below the journal file size.
• A startup warning can pre-announce the problem. If the broker logs Temporary Store limit is X mb, whilst the temporary data directory only has Y mb of usable space, the broker has clamped the temp store limit to actual free disk, which can be far below what you configured.

Common causes

Quick checks

All read-only. Adjust brokerName, credentials, and paths to your deployment.

# Temp store usage: the headline number
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/TempPercentUsage'

# Broker memory usage: temp store spill starts when this is at or near the limit
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/MemoryPercentUsage'

# How big is the temp store on disk right now
du -sh /opt/activemq/data/tmp_storage/

# Free space on the partition the temp store shares with KahaDB
df -h /opt/activemq/data/

# Broker log: flow control and temp store events
grep -i "temp\|memory limit\|usage manager" /opt/activemq/data/activemq.log | tail -30

Then, for the destinations you suspect, check consumption:

# Consumer count on a suspect queue
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/ConsumerCount'

# Queue depth, inflight, and counters on the same queue
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/QueueSize,InFlightCount,DequeueCount,EnqueueCount'

How to diagnose it

1. Confirm the state. Read TempPercentUsage and MemoryPercentUsage together. Temp at 100% with memory at 100% confirms the spill-and-fill sequence. Temp high with memory low is unusual and points at cursor or limit misconfiguration.

2. Determine the failure mode you are in. Check the destination policy for the affected destinations. If producerFlowControl is not explicitly false, producers are blocking, not dropping: expect upstream send() calls to hang. If it is false, check for sendFailIfNoSpace and sendFailIfNoSpaceAfterTimeout. If none are set, assume messages are being discarded and treat it as confirmed data loss on the non-persistent path.

3. Find which destinations are filling memory. Walk per-destination MemoryPercentUsage, QueueSize, ConsumerCount, and DequeueCount. The culprit is usually one destination with pending messages and slow or zero consumers. For topics with non-durable subscribers, look for one lagging subscriber; topic fan-out means the slowest subscriber dictates accumulation.

4. Check the temp store sizing. Read the <tempUsage> limit in activemq.xml and compare it to (a) the broker memoryUsage limit and (b) actual free disk on the partition. Three classic misconfigurations: tempUsage smaller than memoryUsage (memory can hold more than the temp store has room for), tempUsage below the 32 MB journal file size (usage can pin above 100%), and tempUsage larger than free disk (broker clamps it at startup; check the log warning).

5. Check the disk partition. df on the data directory. If the temp store and KahaDB share a partition, temp store growth may be pushing you toward a disk-full incident that would also halt persistent messaging. That changes your urgency.

6. Quantify the loss window, if dropping. Compare producer-side send counts (application logs) against broker EnqueueCount deltas for the affected destinations during the incident window. The gap is your drop estimate. There is no broker-side counter that cleanly reports “messages discarded due to full temp store,” which is exactly why this failure mode is silent.

Metrics and signals to monitor

Fixes

Restore consumption first

If consumers are down or stuck, that is the root cause and the fix. Restart the consumer application, scale it out, or disconnect a wedged consumer so messages rebalance. Until dequeue resumes, every other fix just buys capacity.

If producers are blocking and you need immediate relief

Raising the tempUsage limit in activemq.xml gives the broker somewhere to put the overflow, but it requires a broker restart to take effect, so it is a planned fix, not a mid-incident one. In the short term, reducing producer rate or pausing non-critical producers is the lever that does not require a restart. Do not restart the broker as a first move: with a large temp store, restart adds recovery time, and it does nothing about the consumer problem that caused the backlog.

If messages are being dropped

Stop the bleeding by making loss loud:

<!-- Per-destination policy: fail sends instead of silently discarding -->
<policyEntry queue=">" sendFailIfNoSpace="true"/>

With sendFailIfNoSpace="true", producers get an exception they can log, alert on, and retry from, instead of a silent discard. If you cannot tolerate blocked producers, sendFailIfNoSpaceAfterTimeout is the middle ground: block briefly, then fail. Either way, the failure becomes visible at the producer, which is the minimum acceptable posture for a full-store condition.

Note the async-send caveat: non-persistent messages are sent asynchronously by default, and async sends do not wait for broker acknowledgment. If you rely on producers noticing broker-side rejections, verify your client actually surfaces them; in many setups the producer never learns anything went wrong.

Fix the sizing

• Set tempUsage deliberately. It should be larger than the amount of non-persistent data you expect to overflow during your worst plausible consumer outage, and never smaller than the 32 MB journal file granularity.
• Keep memoryUsage and tempUsage coherent. If memoryUsage exceeds tempUsage, memory can hold more pending data than the temp store can absorb, and TempPercentUsage can exceed 100% the moment spill starts.
• Keep tempUsage within real disk. If the configured limit exceeds free space on the partition, the broker clamps it at startup and logs the warning. Monitor the partition, not just the configured limit.

Reconsider delivery mode

If the messages matter, the honest fix is often to make them persistent. Persistent messages go to KahaDB, which is sized, monitored, and journaled for exactly this. Non-persistent delivery with a tiny temp store is an implicit bet that consumers never fall behind. If that bet has already failed once, stop taking it for that destination.

Prevention

• Monitor TempPercentUsage as a first-class signal. Ticket above 50%, page at 100% on brokers carrying non-persistent traffic. Any sustained non-zero value on a latency-sensitive workload means you are already spilling.
• Alert on the spill itself, not just the full state. TempPercentUsage going non-zero means memory filled. That is your early warning, with time to act.
• Never ship producerFlowControl=“false” without sendFailIfNoSpace or sendFailIfNoSpaceAfterTimeout. That combination is the silent-drop configuration. Audit existing policyEntries for it.
• Size tempUsage against the 32 MB floor and against real free disk. Re-check after any partition resize or memory limit change.
• Watch slow non-durable topic subscribers. They are the most common trigger: one lagging subscriber forces the whole topic backlog into memory and then into the temp store.
• Put temp store and KahaDB disk on your capacity dashboards together. They share the partition; trend both against free space.

How Netdata helps

• Netdata surfaces TempPercentUsage, MemoryPercentUsage, and StorePercentUsage from the ActiveMQ JMX tree on the same broker dashboard, so you can see the memory-to-temp spill sequence as it happens instead of discovering it from producer hangs.
• Per-destination queue depth, enqueue/dequeue rates, consumer counts, and inflight counts let you jump from “temp store filling” to the specific destination and consumer causing it.
• Disk metrics for the data partition are collected alongside broker metrics, which matters because the temp store and KahaDB compete for the same filesystem.
• ML-based anomaly detection on TempPercentUsage catches the slow-burn case: gradual temp store growth over hours that stays under static thresholds but is clearly abnormal for your workload.
• Alerting on the correlated bundle (memory near limit, temp usage rising, dequeue rate falling) shortens the path from symptom to cause.

Related guides

• ActiveMQ store is full: StorePercentUsage at 100% and persistent messaging halted
• ActiveMQ memory limit reached: MemoryPercentUsage at 100% and the flow-control cliff
• ActiveMQ MemoryPercentUsage climbing: reading the flow-control leading indicator
• ActiveMQ producer flow control: why send() hangs and producers block silently
• ActiveMQ disk full on the KahaDB partition: write failures and store corruption risk
• ActiveMQ per-destination memory usage: one noisy queue blocking every producer
• How ActiveMQ Classic actually works in production: a mental model for operators
• ActiveMQ monitoring checklist: the signals every production broker needs