$ guides / apache-pulsar / apache-pulsar-subscription-backlog-growing ▌

Operations Guides

Apache Pulsar subscription backlog growing: consumers falling behind producers

A growing subscription backlog means producers are publishing faster than consumers can acknowledge. The absolute backlog size matters less than its trajectory. A high but stable backlog is normal for lagged or replay consumers. A monotonically increasing backlog is a problem regardless of size: it consumes bookie disk until the bookie goes read-only, the backlog quota trips and throttles producers, or retention and TTL silently delete messages the consumer never saw.

The most insidious variant is the Silent Consumer Stall. Producers continue writing normally (pulsar_rate_in looks fine), but dispatch to consumers has stopped or flatlined (pulsar_rate_out is zero or well below rate_in). Consumers may even appear connected. The backlog grows linearly while the broker, bookies, and network all report healthy. Operators who only monitor rates, not backlog directly, miss this until it cascades.

What this means

Subscription backlog is the count of entries published but not yet acknowledged by a specific subscription. Pulsar tracks this per subscription, not per topic. A topic with five subscriptions has five independent backlogs, each of which can grow or drain independently.

The relevant Prometheus metrics are:

Metric	Scope	What it counts
`pulsar_subscription_back_log`	Per subscription	Entries (not individual messages)
`pulsar_subscription_back_log_no_delayed`	Per subscription	Entries excluding delayed messages
`pulsar_msg_backlog`	Per topic	Aggregate across subscriptions on that topic
`pulsar_broker_msg_backlog`	Per broker	Aggregate across all topics on that broker

All four are gauges with labels including cluster, namespace, topic, and subscription. The per-subscription metric is the one to alert on.

flowchart TD
    A[Backlog growing] --> B{rate_out near zero?}
    B -->|Yes| C{Consumers connected?}
    C -->|No| D[Consumer down or disconnected]
    C -->|Yes| E{unackedMessages high?}
    E -->|Yes| F[Unacked dispatch freeze]
    E -->|No| G[Dispatcher blocked or slow]
    B -->|No| H{rate_out < rate_in?}
    H -->|Yes| I{Redelivery high?}
    I -->|Yes| J[Poison message loop]
    I -->|No| K[Consumer too slow]

One important note: pulsar_subscription_back_log counts entries, not individual messages. With batch messages enabled, one entry can contain many messages. The msgBacklog field in topic stats also counts entries. For exact message-level counts, the analyzeBacklog admin API exists but is expensive and reads from storage.

A newly created subscription with SubscriptionInitialPosition.Earliest will show all historical messages as backlog. This is expected behavior, not a consumer problem. Alert on rate-of-change, not absolute size.

Common causes

Cause	What it looks like	First thing to check
Consumer is down or disconnected	`rate_out` is zero, `pulsar_subscription_back_log` grows linearly, no consumer connections for the subscription	Consumer application health and deployment status
Dispatch frozen on unacked limit	`unackedMessages` at or near `maxUnackedMessagesPerSubscription` (default 200,000), `blockedSubscriptionOnUnackedMsgs` is true in topic stats, `rate_out` drops to zero	`pulsar_subscription_unacked_messages` and the `blockedSubscriptionOnUnackedMsgs` flag
Consumer processing too slow	`rate_out` is non-zero but consistently below `rate_in`, backlog grows steadily, `unackedMessages` moderate but not at limit, redelivery rate low	Compare `pulsar_rate_in` vs `pulsar_rate_out` per topic
Poison message or processing failure loop	High `pulsar_subscription_msg_rate_redeliver`, backlog grows, same messages redelivered repeatedly, `rate_out` may look normal but no forward progress	Redelivery rate relative to dispatch rate
TTL deleting messages before consumers read them	Non-zero `pulsar_subscription_msg_rate_expired`, backlog may stabilize or shrink but consumers are silently losing data	TTL configuration and expiration rate

Quick checks

# Per-subscription backlog (filter by namespace in production to avoid huge output)
curl -s http://<broker-host>:8080/metrics | grep pulsar_subscription_back_log | grep <namespace>

# Publish and dispatch rates for a specific topic
curl -s http://<broker-host>:8080/metrics | grep -E "pulsar_rate_(in|out)" | grep <topic>

# Unacked messages per subscription
curl -s http://<broker-host>:8080/metrics | grep pulsar_subscription_unacked_messages | grep <namespace>

# Redelivery rate
curl -s http://<broker-host>:8080/metrics | grep pulsar_subscription_msg_rate_redeliver | grep <namespace>

# Message expiration rate (TTL-driven loss)
curl -s http://<broker-host>:8080/metrics | grep pulsar_subscription_msg_rate_expired | grep <namespace>

# Admin API: subscription-level detail for a specific topic
pulsar-admin topics stats persistent://tenant/namespace/topic
# Look for: subscriptions.<name>.msgBacklog
# Look for: subscriptions.<name>.unackedMessages
# Look for: subscriptions.<name>.blockedSubscriptionOnUnackedMsgs
# Look for: subscriptions.<name>.consumers (array length)

# Bookie disk usage
# TODO: verify HTTP metrics endpoint. Many deployments expose bookie metrics via JMX, not HTTP.
curl -s http://<bookie-host>:8000/metrics | grep bookie_ledger_dir_.*_usage

How to diagnose it

Identify which subscription is growing. Filter pulsar_subscription_back_log by namespace, topic, and subscription. The backlog is per-subscription, so isolate the specific subscription before investigating causes.
Compare rate_in to rate_out for the affected topic. If rate_out is zero or flat while rate_in is normal, dispatch has stopped. If rate_out is non-zero but consistently below rate_in, the consumer is simply too slow.
Check whether the dispatch freeze is caused by unacked messages. Look at pulsar_subscription_unacked_messages for the affected subscription. If it is at or near the configured maxUnackedMessagesPerSubscription (default 200,000), the broker has silently stopped dispatching. Confirm with the blockedSubscriptionOnUnackedMsgs flag in topic stats. Note: there is also a per-consumer limit (maxUnackedMessagesPerConsumer, default 50,000) that can freeze a single consumer within a shared subscription.
Check for redelivery storms. A high pulsar_subscription_msg_rate_redeliver relative to dispatch rate means consumers are receiving messages but failing to process them. Every redelivered message occupies dispatch and broker resources without making forward progress.
Verify consumers are actually connected. In the Admin API topic stats, check the consumers array for each subscription. An empty array means no consumers are connected. A non-empty array with growing backlog means connected consumers are not acking.
Check for TTL-driven silent loss. A non-zero pulsar_subscription_msg_rate_expired means messages are being deleted by TTL before the consumer reads them. The backlog may stabilize or shrink, but the consumer is losing data.
Assess bookie disk impact. Growing backlog consumes bookie disk. Check disk usage across bookies. If disk approaches the configured threshold (default 95% via diskUsageThreshold), the bookie will go read-only and stop accepting writes.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`pulsar_subscription_back_log`	Primary backlog metric per subscription	Monotonic growth sustained for more than 15 minutes with active consumers
`pulsar_rate_in` vs `pulsar_rate_out`	Rate divergence directly causes backlog growth	`rate_out` consistently below `rate_in`
`pulsar_subscription_unacked_messages`	Dispatch freeze when limit is hit	Sustained above 50% of `maxUnackedMessagesPerSubscription`
`pulsar_subscription_msg_rate_redeliver`	Messages received but not processed	Redelivery rate above 10% of dispatch rate
`pulsar_subscription_msg_rate_expired`	TTL deleting messages before consumption	Any non-zero rate on topics where data loss is unacceptable
Bookie disk usage	Backlog growth fills bookie disk	Growth trend approaching 85-90%
`blockedSubscriptionOnUnackedMsgs`	Definitive indicator of dispatch freeze	Value is `true` in topic stats

Fixes

Consumer is down or disconnected

Restart the consumer or investigate why it crashed. If the consumer was undeployed intentionally, verify whether the subscription should still exist. Abandoned subscriptions with no consumers hold backlog and prevent data deletion.

Dispatch frozen on unacked limit

The broker stopped dispatching because unackedMessages hit maxUnackedMessagesPerSubscription. This is a consumer-side problem: the consumer received messages but is not acknowledging them. Investigate:

Consumer processing logic (is it blocked on a downstream dependency?).
Ack path (are acks being sent?).
ackTimeout configuration (is it too generous, allowing messages to stay unacked indefinitely?).

Do not raise the unacked limit as a first response. It only delays the dispatch freeze and increases memory pressure. Fix the consumer’s ack behavior.

If the consumer is genuinely stuck and cannot recover, consider skipping the subscription cursor forward to unblock dispatch.

Warning: Skipping the cursor acknowledges messages the consumer never processed. This is destructive data loss. Coordinate with the application team before doing this.

Consumer processing too slow

The consumer is acking but cannot keep up with the publish rate. Options, in order of preference:

Scale out consumers. For shared or failover subscriptions, adding consumers increases parallel dispatch capacity. For exclusive subscriptions, this requires a subscription type change.
Reduce producer rate. If the consumer legitimately cannot process faster, throttle producers to match. This is preferable to unbounded backlog growth.
Increase consumer processing parallelism. Application-level optimization: batch processing, async acks, or reducing per-message work.

Poison message or processing failure loop

A high redelivery rate with no forward progress indicates messages that always fail processing. Inspect the message content causing failures. Options:

Route the poison message to a dead letter topic if DLQ is configured. After maxRedeliveryCount redeliveries, Pulsar moves the message to {topic}-{subscription}-DLQ.
Skip the message by acknowledging it manually (if the application supports it).
Fix the consumer’s error handling so it does not infinite-loop on bad messages.

TTL deleting messages before consumers read them

If pulsar_subscription_msg_rate_expired is non-zero and unexpected, the consumer is falling behind its TTL window. Either:

Increase TTL for the namespace or topic.
Fix the consumer so it reads within the TTL window.
Accept the data loss if TTL is intentionally aggressive and the topic is a real-time-only feed.

Prevention

Alert on backlog rate-of-change, not absolute size. A sustained growth rate over 15 minutes with active consumers is the actionable signal. Absolute thresholds produce false positives for legitimately lagged consumers and false negatives for slowly growing backlogs.
Monitor per-subscription, not just per-topic. A topic with multiple subscriptions can have one healthy subscription and one stuck one. Topic-level aggregates hide the problem.
Track unacked message count. The dispatch freeze at maxUnackedMessagesPerSubscription is silent. Alert when unacked messages exceed 50% of the configured limit.
Include system namespaces in backlog monitoring. Pulsar uses internal topics in the pulsar/system namespace for cluster coordination. Backlogs on system topics cause strange cluster behavior but are often excluded from monitoring.
Understand your backlog quota policy. producer_request_hold and producer_exception throttle producers when quota is exceeded. consumer_backlog_eviction silently discards the oldest unacked messages from the slowest subscriber. Know which policy is active on each namespace.
Check for abandoned subscriptions. Subscriptions created dynamically (per microservice instance, per test run) and never cleaned up hold cursors that prevent data deletion. Monitor subscription count over time. Consider subscriptionExpirationTimeMinutes to auto-expire abandoned subscriptions.

How Netdata helps

Per-second backlog metrics let you see the rate-of-change immediately, not minutes after the growth starts. The difference between a burst and a sustained trend is visible within the first few data points.
Correlate pulsar_rate_in with pulsar_rate_out on the same dashboard to instantly spot the Silent Consumer Stall: normal publish rate with flatlined dispatch.
ML anomaly detection on backlog trends flags monotonic growth even when the absolute value is still low, catching the problem before it triggers quota enforcement or disk pressure.
Cross-layer correlation connects subscription backlog growth to bookie disk usage and bookie server status, so you can see the downstream impact of a slow consumer in real time.
Unacked message tracking surfaces the dispatch-freeze pattern before blockedSubscriptionOnUnackedMsgs flips to true, giving you time to fix the consumer before dispatch stops entirely.