$ guides / apache-pulsar / apache-pulsar-backlog-quota-exceeded ▌

Operations Guides

Apache Pulsar backlog quota exceeded: producers held or rejected when consumers stall

Producers are timing out or receiving errors, but the root cause is not on the producer side. A subscription backlog has crossed a configured quota, and the backlog policy has kicked in. Depending on the policy, producers are now blocked, rejected, or the oldest messages are being silently deleted. The symptom is a producer outage or data loss, but the root cause is a consumer that stopped keeping up.

The blast radius depends entirely on a namespace-level policy that many teams set once and forget. The same growing backlog can produce three different outcomes: producer timeouts, producer exceptions, or silent message eviction. Knowing which policy is active on the affected namespace is the first thing you need before touching anything.

The backlog quota check runs periodically (default every 60 seconds via backlogQuotaCheckIntervalInSeconds). Between checks, backlog can overshoot the quota. After the check fires, the configured policy determines what happens to producers writing to any topic in that namespace.

What this means

A backlog quota is a per-namespace limit on how much unacknowledged data (or how old the oldest unacknowledged message) can accumulate for any subscription. When a subscription’s backlog crosses the quota, the broker enforces one of three policies:

producer_request_hold (default): The broker holds producer writes instead of acknowledging them. Producers block on their send() call until their client-side sendTimeoutMs expires, then they receive a timeout error. From the producer’s perspective, this looks like the broker hung or the network failed.

producer_exception: Producers receive immediate exceptions on write attempts. This is a fast-fail policy. Producers that handle exceptions with retries will retry indefinitely against a quota that is still exceeded, generating load without progress.
consumer_backlog_eviction: The broker silently acknowledges (discards) the oldest unacknowledged messages in the subscription’s backlog, bringing it back under quota. Producers continue writing normally. There are no producer-side errors or timeouts. This is data loss dressed as normal operation. The subscription’s consumers will never see the evicted messages.

The same root cause (a stalled consumer) produces opposite symptoms depending on the policy. With producer_request_hold or producer_exception, you get a noisy producer outage that is easy to detect. With consumer_backlog_eviction, you get silent data loss that may not be detected until downstream consumers notice missing data.

flowchart TD
    A[Consumer stalls or crashes] --> B[Subscription backlog grows]
    B --> C{Backlog crosses quota?}
    C -->|No| D[Producers write normally]
    C -->|Yes| E{Backlog quota policy}
    E -->|producer_request_hold| F[Producers block until sendTimeoutMs]
    E -->|producer_exception| G[Producers receive write exceptions]
    E -->|consumer_backlog_eviction| H[Oldest messages silently evicted]
    F --> I[Producer timeout outage]
    G --> I
    H --> J[Silent data loss, producers unaffected]

Common causes

Cause	What it looks like	First thing to check
Consumer application crash or deployment failure	`pulsar_rate_out` drops to zero for a subscription while `pulsar_rate_in` continues normally	Consumer process health and connection count
Slow consumer in a Shared subscription	One consumer in the subscription lags, backlog grows for the entire subscription	Per-consumer stats in `pulsar-admin topics stats`
Stuck consumer (poison message or downstream failure)	High redelivery rate (`pulsar_subscription_msg_rate_redeliver`), backlog grows, consumers connected but not acking	Redelivery rate and unacked message count
Unacked message saturation	`pulsar_subscription_unacked_messages` at `maxUnackedMessagesPerSubscription` limit, dispatch frozen silently	Unacked count vs configured limit
Abandoned subscription cursor	Subscription with zero connected consumers and large, growing backlog	Subscription list vs connected consumer count
No backlog quota configured	`backlogQuotaDefaultLimitBytes=-1`, backlog grows unbounded until bookie disk fills	Namespace backlog quota configuration

Quick checks

# Identify which subscriptions have the largest backlog on a topic
pulsar-admin topics stats persistent://tenant/namespace/topic

# List all subscriptions on the topic
pulsar-admin topics subscriptions persistent://tenant/namespace/topic

# Check the configured backlog quota policy and limit for a namespace
pulsar-admin namespaces get-backlog-quota tenant/namespace

# Check rate_in vs rate_out for the affected topic from Prometheus metrics
curl -s http://<broker-host>:8080/metrics | grep -E "pulsar_(rate|throughput)_(in|out).*topic"

# Check per-subscription backlog from metrics
curl -s http://<broker-host>:8080/metrics | grep pulsar_subscription_back_log

# Check unacked messages per subscription
curl -s http://<broker-host>:8080/metrics | grep pulsar_subscription_unacked_messages

# Check redelivery rate per subscription
curl -s http://<broker-host>:8080/metrics | grep msg_rate_redeliver

# Check bookie disk usage (backlog fills bookie disk)
curl -s http://<bookie-host>:8000/metrics | grep bookie_ledger_dir

How to diagnose it

Confirm the backlog quota is the cause. Check the namespace backlog quota configuration to see the configured limit and policy. If the policy is producer_request_hold or producer_exception, look for ProducerBlockedQuotaExceededError in broker logs. If the policy is consumer_backlog_eviction, there may be no explicit error at all.
Identify the offending subscription. Run pulsar-admin topics stats on the affected topic and look at subscriptions.<name>.msgBacklog for each subscription. The one with the largest backlog, or the one growing fastest, is the trigger. A topic with multiple subscriptions can have one stalled subscription that breaches the quota for all producers on the topic.
Determine why the subscription is stalled. Check whether the subscription has connected consumers:
- Consumer count is zero: the consumer application has crashed or been undeployed. Backlog grows linearly.
- Consumers are connected but backlog is growing: check unackedMessages in subscription stats. If near maxUnackedMessagesPerSubscription, the broker has frozen dispatch silently. Check msgRateRedeliver: high redelivery means consumers are receiving messages but failing to process them (poison message, downstream dependency timeout). Compare msgRateOut to msgRateIn: if msgRateOut is non-zero but lower, consumers are alive but too slow.
Check the consumer application. Review consumer logs for processing errors, downstream connection failures, or unhandled exceptions. For poison messages, inspect the message at the head of the backlog.
Assess disk impact. A growing backlog consumes bookie disk space. Check bookie_ledger_dir_{path}_usage on bookies. If disk usage is above 85%, the bookie is approaching read-only transition, which would turn a backlog problem into a cluster-wide write failure.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`pulsar_subscription_back_log`	The direct backlog measurement per subscription	Sustained growth, especially approaching the quota limit
`pulsar_rate_in` vs `pulsar_rate_out`	Divergence between publish and dispatch rate equals backlog growth rate	`rate_in` exceeds `rate_out` for a sustained period
`pulsar_subscription_unacked_messages`	Messages dispatched but not acknowledged; when this hits the limit, dispatch freezes silently	Approaching `maxUnackedMessagesPerSubscription`
`pulsar_subscription_msg_rate_redeliver`	Consumers receiving but failing to process messages	Redelivery rate above 10% of dispatch rate
`pulsar_subscription_msg_rate_expired`	Messages deleted by TTL before consumers read them (related but distinct from backlog eviction)	Non-zero rate on topics where message loss is unacceptable
`bookie_ledger_dir_{path}_usage`	Backlog consumes disk; a bookie going read-only cascades into cluster-wide write failure	Usage above 85% with growing backlog

Fixes

Immediate recovery: clear the backlog

Warning: this is destructive. Clearing a backlog discards all unacknowledged messages for that subscription. Coordinate with the application team before running this.

# Clear backlog for a specific subscription on a topic
pulsar-admin topics clear-backlog persistent://tenant/namespace/topic -s <subscription-name>

This is the fastest way to unblock producers when the policy is producer_request_hold or producer_exception. The backlog drops to zero, the quota is no longer exceeded, and producers resume. You lose all unacknowledged messages for that subscription.

Fix the consumer

If the consumer has crashed, restart it. If the consumer is stuck on a poison message, consider skipping that message or routing it to a dead letter topic. If the consumer is too slow for the publish rate, add consumer capacity or investigate the processing bottleneck (downstream database, API timeouts, deserialization cost).

For Shared subscriptions where one slow consumer drags the entire subscription, configure ackTimeout on the consumer client so that unprocessed messages are redelivered to faster consumers instead of blocking on the slow one.

Adjust the backlog quota policy

If the current policy is causing the wrong kind of failure, change it. Understand the tradeoff:

# Set backlog quota with a specific policy
pulsar-admin namespaces set-backlog-quota tenant/namespace --limit 2G --policy producer_request_hold

producer_request_hold (default): Producers block. Safest policy for data durability because no messages are lost. But it creates a producer outage that can cascade upstream if producers cannot handle sustained blocking.
producer_exception: Producers get fast failures. Useful when upstream systems can handle errors with backoff. Dangerous if producers retry aggressively, generating load against a still-exceeded quota.
consumer_backlog_eviction: Producers are never affected, but messages are silently lost. Only appropriate for topics where data loss is acceptable (for example, real-time telemetry feeds where stale data has no value).

Increase the quota

If consumers are temporarily slow but will catch up (planned maintenance, batch processing window), increasing the quota buys time. But increasing the quota also increases disk consumption. Verify that bookie disk has headroom before raising the limit. Retention must be greater than backlog quota; Pulsar throws an error if retention is set smaller than the backlog quota limit.

Address unacked message saturation separately

If the root cause is unacked message saturation (dispatch frozen at maxUnackedMessagesPerSubscription), the fix is different from backlog quota enforcement. Increase the limit, fix the consumer’s processing speed, or configure ackTimeout so messages are redelivered instead of sitting unacked forever. This is a separate mechanism from backlog quotas but produces a similar symptom: backlog growth due to stalled dispatch.

Prevention

Audit backlog quota policies on every namespace. The default is producer_request_hold, but teams that changed it to consumer_backlog_eviction and forgot are the most dangerous case. They have silent data loss with no producer-side errors to trigger alerts.
Set quotas deliberately. The broker default backlogQuotaDefaultLimitBytes=-1 means no limit. Without a quota, backlog grows until bookie disk fills, which cascades into a cluster-wide write failure. Set explicit per-namespace quotas that make sense for the topic’s data criticality and consumer SLA.
Alert on backlog growth rate, not just absolute size. A high but stable backlog is fine if consumers are keeping up at a lagged pace. A monotonically growing backlog is a problem regardless of absolute size. Alert when backlog growth is sustained for more than 15 minutes with active consumer connections.
Monitor per-subscription backlog, not just topic-level aggregates. A topic with five subscriptions has five independent backlogs. The slowest subscription triggers the quota for all producers on the topic. Per-subscription granularity is essential.
Set consumer ackTimeout for Shared subscriptions. Without ackTimeout, a slow consumer in a Shared subscription can block the entire subscription indefinitely. A reasonable ackTimeout matching the expected processing time allows the broker to redeliver messages to faster consumers.

How Netdata helps

Per-second subscription backlog tracking: Netdata collects pulsar_subscription_back_log at per-second resolution, letting you see backlog growth the moment it starts rather than waiting for the 60-second quota check interval.
Rate divergence correlation: Correlating pulsar_rate_in with pulsar_rate_out on the same chart visually identifies the moment dispatch falls behind publish, which is the leading indicator that a backlog quota breach is imminent.
Unacked message saturation detection: pulsar_subscription_unacked_messages alongside the configured maxUnackedMessagesPerSubscription limit reveals dispatch freezes before they trigger backlog growth.
Redelivery storm visibility: pulsar_subscription_msg_rate_redeliver correlated with backlog growth distinguishes “consumer is down” (zero redelivery) from “consumer is stuck on a poison message” (high redelivery).
Bookie disk pressure correlation: Correlating backlog growth with bookie_ledger_dir_{path}_usage shows whether the backlog is about to cause a bookie to go read-only, which would turn a single-topic problem into a cluster-wide write failure.