Apache Pulsar unacked messages at the limit: the silent dispatch freeze

Consumers are connected. Producers are publishing. Backlog looks flat or zero. But no messages are being processed.

Pulsar brokers enforce a per-subscription limit on unacknowledged messages (maxUnackedMessagesPerSubscription, default 200,000) and a per-consumer limit (maxUnackedMessagesPerConsumer, default 50,000). When unacked messages hit either ceiling, the broker stops dispatching new messages to that subscription or consumer. No error is returned to the client. No exception is thrown. Dispatch pauses silently.

Most monitoring focuses on backlog and connection count. Both can look healthy during a freeze. Messages were already dispatched to consumers, so backlog may read zero. Consumers remain connected at the TCP level. The definitive indicators are the unacked message count itself and the blockedSubscriptionOnUnackedMsgs flag in topic stats.

Mechanism

When a consumer receives a message, the broker increments the unacked counter for that subscription. When the consumer acknowledges, the counter decrements. If consumers process slower than the dispatch rate, or stop acknowledging entirely, the counter climbs. At the limit, the broker sets blockedSubscriptionOnUnackedMsgs to true and halts further dispatch.

The freeze persists until the unacked count drops below the limit. This happens when consumers acknowledge messages, when ack timeouts trigger redelivery (if configured), or when messages are otherwise cleared from the unacked set. If consumers are genuinely stuck and no ack timeout is configured, the freeze persists indefinitely.

flowchart TD
    A[Consumer connected, receiving messages] --> B[Consumer stops or slows acknowledging]
    B --> C[Unacked count rises toward limit]
    C --> D{Unacked at limit?}
    D -->|No| C
    D -->|Yes| E[Broker sets blockedSubscriptionOnUnackedMsgs = true]
    E --> F[Dispatch stops for this subscription]
    F --> G[Consumers stay connected, no errors]
    G --> H[Backlog may read stable or zero]
    H --> I[Silent freeze: no forward progress]

There is also a per-consumer limit. When maxUnackedMessagesPerConsumer (default 50,000) is exceeded for an individual consumer, the broker sets blockedConsumerOnUnackedMsgs to true for that consumer. In a shared subscription with multiple consumers, this can affect individual consumers independently.

Version-specific gotchas

Two upstream bugs complicate diagnosis on certain Pulsar versions:

• The blocked metric was unreliable in older 2.x builds. The pulsar_subscription_blocked_on_unacked_messages Prometheus metric could report 0 even when the subscription was actively blocked. The fix (PR #18621, merged Nov 2022) is in 2.10.x/2.11.x patch releases and 3.0.0. On earlier 2.x builds, do not rely on this metric alone; use the Admin API field blockedSubscriptionOnUnackedMsgs instead.
• A regression caused permanent consumer freeze on some 3.0.x through 4.0.x versions. Consumers could stop receiving messages even when pulsar_subscription_unacked_messages was 0 and pulsar_subscription_blocked_on_unacked_messages was 0, because the per-consumer blockedConsumerOnUnackedMsgs flag got stuck at true after acknowledgements completed (apache/pulsar#22657; regression introduced by PR #20990). If you see freeze symptoms but the subscription-level metrics look clean, check the per-consumer blocked flag in topic stats.

There is also a long-standing issue where pulsar_subscription_unacked_messages can report negative values due to batch acknowledgement counting problems. Monitor the absolute value or cross-check with the Admin API for authoritative counts.

Common causes

Quick checks

Run these read-only checks on the affected topic and subscription. None of them modify state.

# Check unacked message count per subscription from Prometheus metrics
curl -s http://<broker-host>:8080/metrics | grep pulsar_subscription_unacked_messages

# Check whether the subscription is flagged as blocked
curl -s http://<broker-host>:8080/metrics | grep pulsar_subscription_blocked_on_unacked_messages

# Get detailed subscription stats via Admin API
pulsar-admin topics stats persistent://tenant/namespace/topic
# Look for:
#   subscriptions.<name>.unackedMessages
#   subscriptions.<name>.blockedSubscriptionOnUnackedMsgs
#   subscriptions.<name>.consumers[].blockedConsumerOnUnackedMsgs
#   subscriptions.<name>.consumers[].availablePermits

# Check dispatch rate vs publish rate for the affected topic
curl -s http://<broker-host>:8080/metrics | grep -E "pulsar_rate_(in|out)"

# Check redelivery rate for the subscription
curl -s http://<broker-host>:8080/metrics | grep pulsar_subscription_msg_rate_redeliver

# Verify consumer connections are still active
curl -s http://<broker-host>:8080/admin/v2/persistent/tenant/namespace/topic/stats | \
  jq '.subscriptions | to_entries[] | {sub: .key, consumers: (.value.consumers | length)}'

The availablePermits field in consumer stats deserves attention. A value of 0 means the client library’s internal receive queue is full and receive() is not being called. There is no documented -1 sentinel, but negative availablePermits values have been observed on stuck consumers (apache/pulsar#24418), so treat 0 or negative as stalled. This is a separate mechanism from the unacked limit, but it produces the same symptom: the dispatcher stops sending messages. If availablePermits is 0 and blockedSubscriptionOnUnackedMsgs is false, the bottleneck is client-side, not broker-side.

How to diagnose it

1. Identify the affected subscription. Compare pulsar_rate_in and pulsar_rate_out per topic. A topic with non-zero publish rate but zero dispatch rate is your target. If multiple subscriptions exist on the topic, check each independently.

2. Check the unacked count. Pull pulsar_subscription_unacked_messages for the subscription. Compare against your configured maxUnackedMessagesPerSubscription. If you have not changed the default, the limit is 200,000. Alerting at 50% of the limit (100,000 for the default) gives a leading indicator before the freeze triggers.

3. Check the blocked flag. Query the Admin API for blockedSubscriptionOnUnackedMsgs on the subscription and blockedConsumerOnUnackedMsgs on each consumer. The Admin API is more authoritative than the Prometheus metric, especially on versions before 3.0.0.

4. Check availablePermits. If the subscription is not blocked but dispatch has stopped, check availablePermits on each consumer. A value of 0 means the client is not requesting more messages. This points to a client-side issue, not a broker-side block.

5. Check redelivery rate. High redelivery (pulsar_subscription_msg_rate_redeliver) relative to dispatch rate indicates poison messages or processing failures. If every message is being redelivered, the consumer is receiving but failing to process, which means it will never ack and the unacked count will stay elevated.

6. Check the backlog. If pulsar_subscription_back_log is zero or stable while pulsar_rate_out is zero, messages were dispatched but not acknowledged. This is the signature of the unacked freeze. A growing backlog with zero dispatch rate means no consumers are connected at all, which is a different problem.

7. Rule out the version-specific freeze bug. If unacked is 0, the subscription is not blocked, dispatch is zero, and consumers are connected, check blockedConsumerOnUnackedMsgs per consumer. If it is stuck at true, you may be hitting the stuck-blocked-flag regression (apache/pulsar#22657, present in 3.0.2+ until fixed). The temporary workaround is to restart the affected consumer.

Metrics and signals to monitor

Fixes

Consumer is processing too slowly

The consumer receives messages but its processing pipeline cannot keep up with the dispatch rate.

• Scale out consumers. If the subscription type is Shared or Failover, adding consumer instances increases parallelism. For Key_Shared, ensure keys are well-distributed across consumers.
• Increase the unacked limit. Raising maxUnackedMessagesPerSubscription (broker-level config) gives consumers more headroom. This is a bandage, not a fix. If consumers cannot keep up, a higher limit just delays the freeze.
• Tune consumer receiver queue size. A larger receiver queue means more messages buffered client-side, but it also means more memory per consumer and more unacked messages in flight.

Consumer is not acknowledging at all

The consumer receives messages but never calls acknowledge(). This is a code defect.

• Review the consumer’s processing pipeline. Ensure acknowledge() is called on every successfully processed message, including in error paths that handle and recover from failures.
• Check for swallowed exceptions. If the consumer catches an exception, logs it, but does not nack or ack the message, the message stays unacked indefinitely (or until ack timeout triggers redelivery).

Downstream dependency failure

The consumer is blocked waiting on a slow database, API, or external service. Messages pile up in the unacked set because processing cannot complete.

• Fix the downstream dependency. This is the root cause.
• Set appropriate timeouts. If the consumer is stuck on a downstream call, it should time out and either nack the message or move on. Without a timeout, one hung downstream call can freeze the entire subscription.
• Configure ackTimeout on the consumer. This causes the broker to redeliver messages that have been unacked for too long. Without ackTimeout configured (the default is no timeout), unacked messages stay unacked indefinitely.

Poison message

A specific message always fails processing. The consumer receives it, fails, and the message gets redelivered. Redelivery rate spikes.

• Configure a dead letter topic. After maxRedeliveryCount redeliveries, the message routes to a DLQ and stops clogging the subscription.
• Skip the message manually. If you identify the poison message by ID, you can acknowledge it explicitly to clear it from the unacked set. This loses the message, so coordinate with the application team.
• Fix the consumer. If the consumer should handle this message type, fix the deserialization or processing logic.

Stuck blocked flag (version bug)

On affected Pulsar versions, blockedConsumerOnUnackedMsgs can remain true after all messages are acknowledged. The consumer stops receiving permanently.

• Restart the affected consumer. This clears the stuck flag temporarily. The bug will recur.
• Upgrade. Upgrade to a version containing PR #23796 (merged Feb 2025) to resolve permanently. Do not rely on a specific patch number without checking your release’s changelog for that PR.

Prevention

• Alert on unacked count at 50% of the limit. By the time you hit 100%, the freeze is already active and consumers are stuck. Alert when pulsar_subscription_unacked_messages exceeds 50% of maxUnackedMessagesPerSubscription for more than 10 minutes.
• Configure ackTimeout on consumers. Without it, unacked messages never expire. A reasonable ackTimeout (matching your expected processing time) ensures stuck messages eventually get redelivered rather than sitting in the unacked set forever.
• Use dead letter topics. Poison messages cause repeated redelivery, consuming broker resources and keeping unacked counts elevated. A DLQ with an appropriate maxRedeliveryCount prevents infinite retry loops.
• Monitor the blocked flag directly. Even with unacked alerting, a separate alert on pulsar_subscription_blocked_on_unacked_messages == 1 (or the Admin API equivalent) provides a definitive “this is frozen right now” signal.
• Track redelivery rate. High redelivery relative to dispatch rate is a leading indicator of poison messages and processing failures that will eventually push unacked counts up.
• Upgrade past known bugs. If you are on an affected version range, the stuck-blocked-flag regression is a live risk. Plan an upgrade to a patched release.

How Netdata helps

• Per-second resolution on pulsar_subscription_unacked_messages. The difference between a consumer slowly falling behind and one that has hit the wall is visible in the slope of the unacked count. Per-second granularity makes that slope legible before the freeze triggers.
• Correlate unacked count with dispatch rate. When pulsar_rate_out drops to zero while pulsar_subscription_unacked_messages is at the limit, the diagnosis is immediate. Netdata’s correlated dashboards put these signals on the same timeline without manual cross-referencing.
• ML anomaly detection on unacked trends. A slowly climbing unacked count that has not yet crossed the 50% threshold is easy to miss in threshold-based alerting. Anomaly detection flags the deviation from baseline earlier.
• Alert at 50% of the limit. Configurable alerts on the unacked-to-limit ratio give you the leading indicator before the freeze happens, not after.
• Track the blocked flag as a dedicated signal. The pulsar_subscription_blocked_on_unacked_messages metric is surfaced directly as a binary state change rather than requiring inference from rate drops.

Related guides

• Apache Pulsar active connections climbing: connection leaks and file descriptor exhaustion
• Apache Pulsar bookie add-entry queue not draining: writes arriving faster than the disk can commit
• Apache Pulsar AutoRecovery stalled: under-replicated ledgers that never heal
• Apache Pulsar bookie disk filling: runway to read-only and how to reclaim space
• Apache Pulsar bookie failure cascade: recovery I/O that topples surviving bookies
• Apache Pulsar bookie read latency high: catch-up reads competing with the write path
• Apache Pulsar bookie read-only: disk full and bookie_SERVER_STATUS at zero
• Apache Pulsar broker down: telling a dead broker from a fenced one
• Apache Pulsar broker GC death spiral: heap pressure, stop-the-world pauses, and lost topic ownership
• Apache Pulsar broker lookup failures: new clients cannot find their topic
• Apache Pulsar entry log GC falling behind: reclaimed space that never comes back
• How Apache Pulsar actually works in production: a mental model for operators