ActiveMQ expired message count climbing: TTL expiry and silent correctness loss

A queue looks fine. QueueSize is stable, dequeue rate is non-zero, no alerts on memory or store. Then someone asks why a customer order never processed, and you find ExpiredCount on the destination has been climbing for days. Every tick of that counter is a message the broker threw away or shuffled into the DLQ because its TTL ran out before a consumer got to it. Nothing crashed. Nothing paged. The data is just gone.

Expired messages are one of the quietest failure modes in ActiveMQ Classic. The broker stays green on every saturation signal while correctness erodes underneath. Worse, expiry actively hides backlog: if messages expire as fast as they arrive, QueueSize looks stable and the queue appears healthy while real work is being dropped.

This guide covers how to read ExpiredCount, how to tell whether expiry is intentional load-shedding or silent data loss, and how to fix the root cause instead of papering over it.

What this means

Every JMS message can carry a time-to-live. If a message sits in the broker past its TTL without being consumed, the broker expires it. The per-destination ExpiredCount JMX attribute is a cumulative counter of how many messages this has happened to. It only ever goes up; you need two readings to get a rate.

What happens to an expired message depends on the destination’s dead letter strategy:

processExpired="true" (default): expired persistent messages are moved to the DLQ. They stop counting against the source queue, but they land in ActiveMQ.DLQ (or a per-destination DLQ) where they accumulate with no TTL by default. Expiry silently feeds DLQ growth.
processExpired="false": expired messages are simply discarded. No DLQ entry, no advisory event, no trace. This is the purest form of silent correctness loss.

Either way, the message never reached a consumer. Whether that is a disaster or a design choice depends entirely on the queue. A telemetry stream with a 30-second TTL is supposed to shed stale data. An order queue where messages expire after five minutes of consumer lag is losing money one message at a time.

One more subtlety: expiry is not instantaneous. The broker checks for expired messages periodically (during dispatch, prefetch fill, and a background expiration pass), so messages can live past their TTL before the broker reclaims them. QueueSize can briefly include already-dead messages.

flowchart TD
  P[Producer sends with TTL] --> Q[Queue pending]
  Q --> C{Consumed before TTL?}
  C -- yes --> OK[Dequeued and acked]
  C -- no --> E[Message expires, ExpiredCount +1]
  E --> D{processExpired?}
  D -- "true (default)" --> DLQ[Moved to DLQ, no TTL, accumulates]
  D -- false --> GONE[Discarded, no advisory, no trace]
  DLQ --> RISK[Store growth and buried poison messages]
  GONE --> LOSS[Silent correctness loss]

Common causes

Cause	What it looks like	First thing to check
Consumer lag exceeds TTL	ExpiredCount climbs alongside rising message age; dequeue rate too low for enqueue rate	Oldest message age vs configured TTL
No consumers on the destination	ConsumerCount = 0, enqueue continues, everything eventually expires	`ConsumerCount` on the destination
TTL set too aggressively	Expiry rate tracks enqueue rate proportionally even with healthy consumers	Producer-side TTL setting vs real end-to-end latency
Consumers connected but stuck	InFlightCount pinned at prefetch, dequeue near zero, expiry climbing	InFlightCount vs prefetch size
Wrong destination or selector mismatch	Producers send, no consumer’s selector matches, messages sit until TTL	Consumer selectors vs message properties
Intentional expiry misread as failure	ExpiredCount non-zero on a queue designed to shed stale data	Whether the queue’s TTL policy is deliberate

Quick checks

All of these are read-only. They assume the Jolokia HTTP JMX bridge on the web console port (8161); adjust the broker name and credentials to your deployment.

# Expired message count for one queue (cumulative; take two readings for a rate)
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/ExpiredCount'

# Expired count across all queues at once
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=*/ExpiredCount'

# Queue depth, consumer count, inflight on the same destination
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/QueueSize,ConsumerCount,InFlightCount'

# Enqueue and dequeue counters (compute rates from deltas)
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/EnqueueCount,DequeueCount'

# DLQ depth: is expiry feeding the dead letter queue?
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ/QueueSize'

Two readings of ExpiredCount separated by a known interval give you the expiry rate. Divide by the enqueue rate over the same window: that ratio is the single most useful number in this investigation.

How to diagnose it

Quantify the loss. Read ExpiredCount twice, 60 seconds apart, on the affected destination. Do the same for EnqueueCount. Compute expired_rate / enqueue_rate. A ratio of 0.02 means one message in fifty is dying before consumption. On a critical queue, anything above zero deserves an explanation.
Check whether expiry is masking backlog. A stable QueueSize with a non-zero expiry rate means a real backlog can be hidden: messages leave the queue by dying instead of by being consumed. If QueueSize is flat but EnqueueCount is climbing and dequeue is low, expiry is your invisible drain.
Verify consumers exist and are working. ConsumerCount of zero on a production queue means every message lives out its TTL and dies. ConsumerCount greater than zero with InFlightCount pinned at total prefetch means consumers are connected but not acking, which produces the same expiry outcome with a different root cause.
Check message age. Browse the queue and read the JMSTimestamp of the oldest pending message. If the oldest message is already near the TTL, pipeline latency has grown to the edge of the TTL and any further slowdown becomes expiry. Browsing is expensive on deep queues, so use it once for diagnosis, not as a polling loop.
Find where expired messages go. Read the destination policy in activemq.xml. If processExpired is unset, it defaults to true: check DLQ depth and confirm the growth rate matches the expiry rate. If processExpired="false", the messages are gone and the only record is the counter itself.
Inspect DLQ contents. If expired messages are landing in the DLQ, browse a few and look at JMSDestination and the message properties. This tells you which source queues are bleeding and sometimes why (for example, a producer that started stamping short TTLs after a deploy).
Confirm intent. Ask the owning team: is TTL on this queue a deliberate load-shedding policy, or an inherited default? A queue where expiry is by design needs a monitored threshold, not a fix. A queue where nobody knew TTL was set needs a producer change.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`ExpiredCount` rate per destination	Each tick is a lost business event	Any sustained non-zero rate on a queue designed for zero expiry
Expired / enqueued ratio	Normalizes loss to traffic; the best alert threshold	Above zero on critical queues, sustained >5 minutes
`ConsumerCount`	Zero consumers means everything enqueued will expire	Zero on a queue with active producers
`InFlightCount` vs prefetch	Consumers connected but not acking leads to TTL death in prefetch limbo	Inflight pinned at prefetch with collapsed dequeue
Oldest message age	Shows how close the pipeline is running to the TTL	Age approaching TTL
DLQ `QueueSize` and growth rate	With default `processExpired`, expiry feeds DLQ growth silently	DLQ growth rate matching expiry rate
`StorePercentUsage`	Expired messages parked in the DLQ have no TTL and pin store space	Slow climb correlating with DLQ depth
`DequeueCount` rate	Messages moved to DLQ count as dequeued from the source queue, so a “healthy” dequeue rate can actually be DLQ transfers	Dequeue non-zero while consumers report no messages processed

Fixes

Consumers are too slow or absent

The most common root cause: messages outlive their TTL because consumption cannot keep up.

Restore or scale consumers. If ConsumerCount is zero, that is the fix. If consumers are up but slow, profile the processing path (downstream database, external API, thread pool) before touching broker settings.
Unstick saturated consumers. InFlightCount pinned at prefetch with no acks means messages are dying in consumer buffers. Reduce prefetch so fewer messages are held hostage per consumer, and fix whatever is blocking acks. See the inflight guide linked below for that pattern.
Do not just raise the TTL. Extending TTL without fixing consumption converts expiry into backlog. The messages survive longer, memory and store grow, and you trade silent loss for a flow-control incident later.

TTL is set wrong

Align TTL with real pipeline latency. Measure actual enqueue-to-consume latency at peak, then set TTL with headroom above it. If producers copied a TTL from a tutorial or a different queue, that number is probably meaningless for this flow.
Remove TTL where loss is unacceptable. If the business event must not be dropped, TTL should be absent and backpressure should come from flow control, not expiry. That is a producer-side change.

Expired messages are polluting the DLQ

With the default processExpired="true", expired messages mix with genuine poison messages in the DLQ, burying the ones that actually need investigation and consuming store with no TTL.

Set processExpired="false" on queues where expiry is expected (telemetry, cache-invalidation, presence updates). This stops DLQ pollution. The tradeoff is real: discarded messages leave no DLQ record, so you are choosing the counter as your only evidence. Only do this where the loss is genuinely tolerable.
Use per-destination DLQs (IndividualDeadLetterStrategy) so expired messages from one noisy queue do not bury poison messages from critical queues.
Put an expiration on DLQ messages. DLQ messages have no TTL by default and accumulate forever, pinning store. The dead letter strategy supports an expiration attribute in milliseconds. Do not apply it via a wildcard policy that matches the DLQ destinations themselves, or an expiring DLQ entry can be forwarded into another DLQ and loop.
Do not disable the expiry scan to hide the symptom. expireMessagesPeriod controls how often the broker sweeps for expired messages (default 30000 ms; setting it to 0 disables expiry checking entirely). Setting it to 0 stops the counter from moving but leaves dead messages occupying memory and store. That trades visible loss for invisible resource exhaustion.

Prevention

Alert on the ratio, not the counter. expired_rate / enqueue_rate with a threshold of zero on critical queues, sustained over a few minutes, is the alert that catches this. Absolute counts false-positive on low-traffic queues.
Alert on DLQ depth and growth separately. Any non-zero DLQ depth is a ticket; growth rate relative to enqueue rate is the severity signal. Expiry and poison messages both land here by default.
Correlate dequeue with actual processing. Because DLQ transfers count as dequeues, a healthy-looking dequeue rate is not proof of consumption. Pair dequeue rate with consumer-side processing metrics or with DLQ enqueue rate.
Document TTL intent per queue. For every destination with a TTL, record whether expiry is load-shedding (expected, monitor the ratio) or a bug risk (alert at zero tolerance). This one decision turns a 3 a.m. mystery into a known behavior.
Watch message age as the leading indicator. Age climbing toward TTL is your early warning. ExpiredCount is the lagging one.
Mind clock skew. JMSTimestamp and TTL evaluation depend on clocks. If producer, broker, and consumer clocks are not NTP-synced, messages can be born nearly expired, and age-based diagnosis becomes unreliable.

How Netdata helps

Per-destination ExpiredCount as a rate. Netdata collects ActiveMQ destination metrics continuously, so the cumulative ExpiredCount becomes a per-second rate you can threshold directly, with no manual delta math.
Ratio and correlation in one view. Plotting expiry rate next to enqueue rate, dequeue rate, QueueSize, ConsumerCount, and InFlightCount on the same dashboard is what exposes the “stable queue, silent loss” pattern. Each signal alone looks innocent.
DLQ correlation. With default processExpired, expiry shows up as DLQ growth. Seeing expired rate and DLQ depth rise in lockstep confirms the routing path without opening activemq.xml.
Age and backlog context. Combining expiry signals with queue depth trend and consumer count lets you distinguish “consumers gone” from “consumers slow” from “TTL too short” in minutes instead of a browse-and-guess session.
Anomaly detection on low-traffic queues. On queues where any expiry is abnormal, ML-based anomaly flagging on the expired counter catches the first deviation without needing a tuned static threshold per queue.