$ guides / activemq / activemq-durable-subscriber-pending ▌

Operations Guides

ActiveMQ offline durable subscriber pending messages: the silent storage leak

StorePercentUsage has been climbing for three weeks. Your queues are draining, consumer counts look right, dequeue rates track enqueue rates, and yet the KahaDB directory keeps growing and journal files keep stacking up. Nothing in the queue-depth metrics explains it.

The usual suspect is not a queue at all. It is a durable topic subscription whose subscriber went away and never came back. Every persistent message published to that topic is still being written to the store on behalf of that subscription, and it will keep happening forever: ActiveMQ Classic has no automatic expiration for offline durable subscribers by default.

This is one of the top operational gotchas in ActiveMQ Classic, and it is usually created by accident: a developer stands up a test consumer with a durable subscription against a shared broker, the app gets deleted, and the subscription stays behind, accumulating every message published to the topic from that point on.

What this means

A durable topic subscription is identified by the client’s clientId plus the subscription name. The broker keeps a per-subscription store of messages so a subscriber can disconnect, reconnect later, and receive everything it missed. That is the intended behavior, and it works exactly as designed.

The failure mode is when “later” never comes. The broker cannot distinguish “subscriber temporarily offline” from “subscriber decommissioned six months ago”. There is no TTL on the subscription itself and no automatic removal by default. Worse, the subscription’s pending messages pin KahaDB journal files: a journal file is only reclaimable when every message reference in it has been acknowledged, so a single pending message for one orphaned subscription can pin an entire 32MB journal file. Spread across many files, one orphan blocks reclamation broadly, and the store grows far beyond the logical size of the retained messages.

flowchart LR
  P[Producer publishes to topic] --> B[Broker]
  B --> A[Active subscriber: dispatched and acked]
  B --> O[Offline durable subscription: stored per message]
  O --> J[Pending messages pin KahaDB journal files]
  J --> S[StorePercentUsage climbs for weeks]
  S --> F[Store hits 100%: persistent messaging halts]

The end state is store exhaustion: StorePercentUsage or the physical disk hits its limit and all persistent producers block. This article covers the durable-subscription root cause; the broader store-exhaustion pattern is covered in the related guides below.

Common causes

Cause	What it looks like	First thing to check
Decommissioned app, subscription never unsubscribed	Offline subscription with a large, steadily growing pending count	Enumerate durable subscribers; look for clientIds of apps that no longer exist
Dev/test durable subscription on a shared broker	Unfamiliar or throwaway subscription name (“test”, laptop hostname) with nonzero pending	Match subscription names and clientIds to known applications
Subscriber reconnecting with a different clientId	Two subscriptions for the same logical consumer, one online and one orphaned and growing	Compare current subscriber clientIds against the offline subscription list
Subscriber that should be online but is not	Growing pending count on a subscription whose app is supposed to be running	Check the consumer application itself before assuming the subscription is orphaned
offlineDurableSubscriberTimeout configured but not removing subscribers	Offline subscribers persist past the configured timeout	Broker log for durable subscriber removal errors

Quick checks

All of these are read-only.

# List pending counts for durable topic subscriptions via Jolokia
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Topic,destinationName=*/PendingQueueSize'

# Broker-level store pressure
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/StorePercentUsage'

# How many journal files exist and how big is the store
ls /opt/activemq/data/kahadb/db-*.log | wc -l
du -sh /opt/activemq/data/kahadb/
df -h /opt/activemq/data/

# Check the broker log for durable subscriber removal problems
grep -i "durable" /opt/activemq/data/activemq.log | tail -50

Two things to note when reading the subscription list. First, a nonzero pending count alone does not mean the subscription is offline: an active subscriber that is simply behind also shows pending messages. Pair the pending count with whether the subscription currently has an active consumer. Second, with many topics and subscriptions, broad JMX queries can get slow, so keep polling frequency reasonable.

How to diagnose it

Confirm the store is actually growing. Check StorePercentUsage and the KahaDB directory size at two points in time. If both climb while application queue depths are flat, messages are being retained somewhere outside queue metrics. Durable subscriptions are the prime suspect alongside DLQ accumulation.
Enumerate every durable subscription. Pull the per-subscription pending list and record clientId, subscription name, topic, and pending count for each. Do this on every broker if you run a network of brokers: removing a durable subscription on one broker does not automatically clean up demand on the others.
Classify each subscription as active, offline-but-expected, or orphaned. Active consumer plus modest pending: normal. No consumer plus a pending count that grows between two readings: either a broken subscriber or an orphan. No consumer plus a clientId you cannot map to any deployed application: an orphan.
Verify the orphan is what pins the journal files. If the journal file count grows alongside the orphaned subscription’s pending count, the correlation is strong. For a definitive view of which destinations pin which journal files, the documented approach is to enable TRACE logging on org.apache.activemq.store.kahadb.MessageDatabase and watch the GC candidate output in the broker log. Do this temporarily; TRACE on the store is very verbose on a busy broker.
Decide: fix the subscriber or kill the subscription. If the subscriber is supposed to exist, the problem is on the consumer side (down, misconfigured clientId, auth failure) and the pending messages are real work waiting to be processed. If the subscriber is gone for good, the subscription and its backlog are pure waste.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Per-subscription PendingQueueSize	The direct measure of the leak	Offline subscription with pending count growing between readings
Offline durable subscriber count	Tracks how many subscriptions have no active consumer	Any offline subscriber with pending > 0 for more than an hour
StorePercentUsage	The damage meter; at 100% persistent messaging halts	Steady climb over days with flat queue depths
KahaDB journal file count	Shows the physical pinning effect	Count growing while queue depths stay low
Disk free on the KahaDB partition	Independent of the configured store limit; whichever fills first wins	Declining free space not explained by queue backlog
Topic enqueue rate	Sizes the leak: pending growth rate equals the topic’s persistent publish rate for each orphaned subscription	High-rate topic with orphaned subscriptions is a fast-moving leak

Page on the store and disk signals, not on the subscription itself. An offline durable subscriber with a growing pending count is a ticket: investigate within hours. StorePercentUsage at 100% on the active broker is the page.

Fixes

Unsubscribe the orphaned subscription

The permanent fix for a genuinely orphaned subscription is to remove it, via JMX or the web console; there is no broker CLI flag for this. Removal drops the pending backlog, which releases the acknowledgments that were pinning journal files. Journal file deletion is not immediate: KahaDB cleanup runs periodically, so disk reclamation lags the logical removal.

Safety notes:

Confirm the subscription is truly orphaned before removing it. Removal discards all pending messages for that subscription, permanently. If the subscriber ever reconnects, it starts fresh from that point. If there is any chance it is a production consumer that is merely down, get confirmation from the owning team first.
In a network of brokers, repeat the check and cleanup on every broker.

Fix the clientId drift

If the orphan was created because a real consumer reconnected with a different clientId, removing the orphan only fixes the symptom. The consumer will orphan its new subscription the next time its clientId changes again. Make the clientId stable in the consumer configuration so reconnects resume the existing subscription.

Configure automatic removal, with eyes open

Since ActiveMQ 5.6, the broker supports offlineDurableSubscriberTimeout and offlineDurableSubscriberTaskSchedule to automatically remove durable subscribers that have been offline longer than the timeout. The default is never remove.

Operators have reported cases where the timeout fails to actually drop offline subscribers, with removal errors in the broker log. If you rely on it, verify it in a test broker: set a short timeout, take a subscriber offline, and confirm the subscription disappears. Keep enumeration and alerting in place regardless as the safety net.

Bound the backlog with message expiry

If the messages themselves have a TTL, the broker’s periodic expiry check for offline durable subscribers will eventually clear the backlog. Expiry checking for offline durable subscribers runs on a configurable period (expireMessagesPeriod on the destination policy, default 30 seconds). This bounds disk growth to roughly topic rate x TTL per orphaned subscription instead of unbounded growth, but it does not remove the subscription itself.

Emergency: store already at 100%

If persistent messaging has already halted, removing orphaned subscriptions is still the right first move: it releases journal files and brings StorePercentUsage back below the limit without touching live queues. Do not delete KahaDB journal files by hand; that path leads to store corruption. See the disk-full and journal-pinning guides linked below for the full store-recovery procedure.

Prevention

Regular enumeration. List all durable subscriptions on a schedule (daily or weekly) and reconcile them against deployed applications. Anything unmapped is a removal candidate.
Alert on offline subscribers with growing pending counts. Zero active consumers plus a pending count that increases between readings is the exact signature of this leak, and it is cheap to detect.
Ban durable subscriptions from shared dev/test brokers, or auto-expire them. The top source of this leak is throwaway test consumers. If developers need durable semantics for testing, point them at ephemeral brokers or enforce a short offline timeout on the shared one.
Set message TTLs where the business allows. A TTL turns an unbounded leak into a bounded one.
Document subscription ownership. Record the owning application, the expected clientId, and whether the subscription is still needed at every decommission. A durable subscription with no known owner will eventually be an incident.

How Netdata helps

Per-subscription pending counts over time. Tracking PendingQueueSize per durable subscription turns “offline subscriber with growing backlog” from a weekly audit into a continuous signal, with the growth rate visible at a glance.
Correlation with store pressure. Plotting offline subscription pending counts next to StorePercentUsage and journal file count shows the causal link directly: when the subscription count rises and store usage follows, you have your answer without a log dig.
Disk versus store limit. Monitoring physical disk usage on the KahaDB partition alongside the broker’s StorePercentUsage catches the case where the configured store limit exceeds actual disk and the OS fills first.
Alerting on the leak signature. An alert on “offline durable subscriber with pending > 0 sustained” fires while you still have weeks of runway, instead of the page you get when the store hits 100%.