$ guides / apache-pulsar / apache-pulsar-subscription-cursor-leak ▌

Operations Guides

Apache Pulsar abandoned subscriptions: cursor leaks that pin storage forever

Bookie disk usage climbs steadily across the cluster, but publish rates are flat and there is no traffic spike. Per-subscription backlogs look fine for the subscriptions you know about. When you try to delete an old topic to reclaim space, the operation fails with a message about active subscriptions. The topic has not had a consumer in weeks.

This is the signature of a silent subscription cursor leak. Applications create dynamically-named durable subscriptions and then disconnect without unsubscribing. Each abandoned subscription leaves behind a cursor pinned at its last acknowledged position. Pulsar cannot delete any message after that cursor’s position until the subscription acknowledges past it, which never happens because no consumer is connected. Storage grows monotonically, and the growth is invisible in per-subscription backlog dashboards because those dashboards only track subscriptions the team knows about.

By the time a bookie approaches its read-only threshold, dozens or hundreds of abandoned cursors may be pinning terabytes of data that cannot be garbage collected.

What this means

A Pulsar subscription is a durable cursor into a topic’s managed ledger. When a consumer creates a subscription, the broker records a mark-delete position and begins tracking acknowledged messages. Even after the consumer disconnects, the cursor persists. Pulsar’s design guarantee is that messages are retained until all subscriptions have acknowledged them.

This guarantee becomes a liability when subscriptions are abandoned. The managed ledger’s garbage collection cannot reclaim ledger entries that sit after the oldest cursor position. If a cursor was last advanced three months ago, every message published since then is pinned in storage, regardless of whether every other subscription on the topic has long since acknowledged those same messages.

The cascade looks like this:

flowchart TD
    A[App creates dynamic subscription] --> B[Consumer disconnects without unsubscribe]
    B --> C[Cursor frozen at old mark-delete position]
    C --> D[Managed ledger GC cannot reclaim entries after cursor]
    D --> E[Storage grows despite flat traffic]
    E --> F[Bookie disk usage climbs toward threshold]
    F --> G[Bookie hits diskUsageThreshold and goes read-only]
    G --> H[New writes fail if enough bookies are read-only]

The storage cost scales with publish rate multiplied by time since abandonment for each abandoned cursor. A topic producing 1,000 messages per second with an abandoned cursor from 90 days ago has roughly 7.8 billion messages pinned.

Common causes

Cause	What it looks like	First thing to check
Microservice per-instance subscriptions	Subscription names contain hostnames, pod names, or instance IDs; consumer count is zero after deployment rollout	Topic stats, look for subscriptions matching pod or host naming patterns
Batch processing framework leftovers	Frameworks like Spark or Flink create durable subscriptions and do not clean them up on job termination	Subscription names matching job or application names with zero consumers
Test or dev subscriptions left behind	Subscriptions with names like “test”, “debug”, or developer usernames; zero consumers on production topics	Subscription list on production topics for non-production naming
Failed deployment cleanup	Application creates subscriptions on startup but crash or shutdown path does not call unsubscribe	Compare subscription creation timestamps against deployment events

Quick checks

Run these read-only commands to identify abandoned subscriptions and assess storage impact.

# List all topics in a namespace
curl -s http://<broker-host>:8080/admin/v2/persistent/tenant/namespace | jq -r '.[]'

# Get subscription details for a topic: consumer count and backlog
curl -s http://<broker-host>:8080/admin/v2/persistent/tenant/namespace/topic/stats \
  | jq '.subscriptions | to_entries[] | {sub: .key, consumers: (.value.consumers | length), backlog: .value.msgBacklog}'

# Find subscriptions with zero consumers across all topics in a namespace
# NOTE: topic names from the list endpoint are local names; URL-encode if needed
curl -s http://<broker-host>:8080/admin/v2/persistent/tenant/namespace | jq -r '.[]' | while read topic; do
  curl -s "http://<broker-host>:8080/admin/v2/persistent/tenant/namespace/${topic}/stats" \
    | jq -r --arg t "$topic" \
      '.subscriptions | to_entries[] | select(.value.consumers | length == 0) | "\($t)\t\(.key)\t\(.value.msgBacklog)"'
done

# Check cursor positions relative to last confirmed entry
curl -s http://<broker-host>:8080/admin/v2/persistent/tenant/namespace/topic/internalStats \
  | jq '{lastConfirmedEntry: .lastConfirmedEntry, cursors: [.cursors | to_entries[] | {name: .key, markDeletePosition: .value.markDeletePosition}]}'

# Check bookie disk usage
curl -s http://<bookie-host>:8000/metrics | grep bookie_ledger_dir

# Check per-subscription backlog from Prometheus metrics
curl -s http://<broker-host>:8080/metrics | grep pulsar_subscription_back_log

# Count subscriptions on a topic
curl -s http://<broker-host>:8080/admin/v2/persistent/tenant/namespace/topic/stats \
  | jq '.subscriptions | length'

How to diagnose it

Identify topics with growing storage but flat traffic. Pull bookie disk usage trends over weeks using the bookie_ledger_dir_{path}_usage metric. Cross-reference with publish rate (pulsar_rate_in). If disk grows while publish rate is flat, something is preventing garbage collection.
List all subscriptions on affected topics. Use the stats endpoint to enumerate every subscription. Look at the consumer count for each one. Subscriptions with zero connected consumers are candidates for abandonment.
Verify abandonment. Before deleting anything, confirm with application teams that the subscription is truly unused. Some subscriptions may have consumers that reconnect intermittently, such as batch consumers that connect once per day. Query internal stats for markDeletePosition and compare it across checks spaced hours or days apart. A cursor whose position does not advance is abandoned.
Check backlog age, not just size. A subscription with a small backlog might still be abandoned if that backlog has not advanced in weeks. Compare cursor mark-delete positions over time. The internal stats endpoint shows markDeletePosition per cursor, which tells you where each cursor sits relative to lastConfirmedEntry.
Assess storage impact. For each abandoned subscription, the pinned storage is approximately the total data written to the topic since the cursor’s last advancement. Prioritize cleanup of subscriptions with the oldest cursors on the highest-traffic topics.
Confirm the pattern at scale. If one topic has abandoned subscriptions, check the entire namespace or tenant. The application behavior that caused the leak likely affected multiple topics. Run the namespace-wide scan from the quick checks section to enumerate all zero-consumer subscriptions.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Subscription count per topic	Growing count without corresponding consumer growth indicates leaks	Count increasing over days or weeks without deployment changes
Subscriptions with zero consumers	Direct indicator of abandoned cursors	Any sustained zero-consumer subscription on an active topic
Bookie disk usage (`bookie_ledger_dir_{path}_usage`)	Pinned storage eventually fills disks	Sustained growth despite flat publish rate
Subscription backlog (`pulsar_subscription_back_log`)	Abandoned subscriptions may show large or frozen backlogs	Non-zero backlog on a subscription with zero consumers
`bookie_SERVER_STATUS`	Terminal state if disk fills to threshold	Value drops to 0, meaning read-only
Cursor mark-delete position (internal stats)	Shows exactly where each cursor is frozen in the ledger	Position unchanged over days or weeks
Topic deletion failures	Attempting cleanup surfaces the problem	Error: “Topic has active subscriptions or producers”

Fixes

Unsubscribe confirmed abandoned subscriptions

Once you have confirmed a subscription is truly abandoned, remove it:

# Unsubscribe a single abandoned subscription
pulsar-admin topics unsubscribe \
  --subscription <subscription-name> \
  persistent://tenant/namespace/topic

The cursor is removed, and the managed ledger can immediately begin garbage collecting entries that were pinned by that cursor, subject to retention policy and other cursors on the same topic.

After unsubscribing, monitor bookie disk usage. BookKeeper’s entry log garbage collection runs asynchronously, so reclaimed space may not appear immediately. The bookie_ACTIVE_ENTRY_LOG_SPACE_BYTES metric should begin trending down as GC reclaims entry logs that contained the pinned data.

Batch unsubscribe for multiple abandoned subscriptions

There is no built-in batch unsubscribe command. Script the operation carefully:

# WARNING: Destructive. Verify each subscription is truly abandoned before running.
# Review the echo output first by removing the pulsar-admin line.
curl -s http://<broker-host>:8080/admin/v2/persistent/tenant/namespace | jq -r '.[]' | while read topic; do
  for sub in $(curl -s "http://<broker-host>:8080/admin/v2/persistent/tenant/namespace/${topic}/stats" \
    | jq -r '.subscriptions | to_entries[] | select(.value.consumers | length == 0) | .key'); do
    echo "Unsubscribing: persistent://tenant/namespace/${topic} / ${sub}"
    pulsar-admin topics unsubscribe \
      --subscription "$sub" \
      "persistent://tenant/namespace/${topic}"
  done
done

Always verify the subscription list with application owners before running this against production.

Truncate as a last resort

If a topic has so many abandoned subscriptions that manual cleanup is impractical, pulsar-admin topics truncate moves all cursors to the end of the topic and deletes all messages up to that point. This is destructive: it acknowledges all existing messages for every subscription, including active ones. Use this only when you have confirmed no active consumer needs the historical data.

# DESTRUCTIVE: Moves all cursors to end, deletes all existing messages
pulsar-admin topics truncate persistent://tenant/namespace/topic

Prevention

Set subscriptionExpirationTimeMinutes at the namespace level. This tells the broker to automatically delete subscriptions that have had no connected consumers for the configured duration. Set it per namespace rather than relying on the global broker.conf default, since different applications have different lifecycle needs.

# Set subscription expiration to 7 days (10080 minutes) for a namespace
pulsar-admin namespaces set-subscription-expiration-time \
  -t 10080 \
  tenant/namespace

The default value is 0, meaning inactive subscriptions are never deleted automatically. The broker checks for expired subscriptions periodically based on subscriptionExpiryCheckIntervalInMinutes, which defaults to 5 minutes.

Caveat: expiration may not work on completely inactive topics. If a topic has no connected producers and no consumers on any subscription, the broker may not evaluate subscription expiration for it. This is a known limitation. For topics that are purely accumulating data with no active traffic, you may need to manually unsubscribe abandoned subscriptions.

Caveat for versions before Pulsar 2.8.1 and 2.9.0. In these older versions, setting subscriptionExpirationTimeMinutes at the namespace level stored 0 instead of null for existing namespaces, which prevented auto-deletion from working. The fix is to run pulsar-admin namespaces remove-subscription-expiration-time tenant/namespace first to clear the stored value, then set the desired expiration time.

Ensure applications call unsubscribe on shutdown. The root cause is almost always application-side. Microservices that create per-instance subscriptions must unsubscribe during graceful shutdown. Batch processing frameworks that create durable subscriptions must clean them up when jobs terminate. Review the application lifecycle code for the topics where you found leaks.

Monitor subscription count as a time series. Track the number of subscriptions per topic over weeks, not just the current value. Any monotonically increasing trend without corresponding consumer growth is a leak. Alert on sustained subscription count growth.

Track backlog age, not just backlog size. A subscription with a backlog of 100 messages that has not advanced in 30 days is a leak. A subscription with a backlog of 100,000 messages that is actively draining is healthy. Age reveals abandonment; size alone does not. Compare cursor mark-delete positions to the managed ledger’s lastConfirmedEntry to compute how far behind each cursor is.

How Netdata helps

Netdata’s per-second metrics collection surfaces several signals relevant to cursor leak diagnosis:

Bookie disk usage trends at per-second resolution make slow monotonic growth from pinned storage visible long before disk thresholds trigger. Anomaly detection flags unusual growth patterns that deviate from the established baseline.
Subscription backlog metrics (pulsar_subscription_back_log) collected per-topic and per-subscription let you correlate specific subscriptions with storage growth, even when those subscriptions are not in your actively monitored set.
Bookie server status (bookie_SERVER_STATUS) transitions to read-only are immediately visible, giving you the terminal-state alert if cursor leaks have already pushed a bookie past its disk threshold.
Publish and dispatch rates alongside disk usage trends let you confirm the “flat traffic but growing storage” signature that distinguishes cursor leaks from legitimate backlog accumulation.
Active connection counts per broker, correlated with subscription data from the stats API, help identify subscriptions that have zero connected consumers.