Apache Pulsar message redelivery storm: poison messages and consumers making no progress

Subscription redelivery rate is climbing. Consumers are connected, the dispatch rate (pulsar_rate_out) looks healthy, but messages keep cycling back without being acknowledged. The backlog may even appear stable or zero, because pulsar_subscription_back_log measures messages not yet dispatched, not messages dispatched but unacked. The definitive signal is pulsar_subscription_msg_rate_redeliver trending upward relative to dispatch. When redelivery exceeds 10% of your dispatch rate, something is wrong. When it approaches 100%, you have zero forward progress: the broker burns memory and network re-sending messages that will never succeed.

This is a redelivery storm. The root cause is always the same loop: a consumer receives a message, fails to process it, and the broker redelivers it. Pulsar has three mechanisms that trigger redelivery (negative acknowledgment, ack timeout, and retry letter topic), and a dead letter topic (DLQ) is supposed to be the escape valve after maxRedeliveryCount attempts. But several well-known behaviors and bugs can prevent that valve from opening, leaving poison messages stuck in an infinite loop.

Redelivery mechanisms

Three mechanisms cause messages to be redelivered in Pulsar:

1. Negative acknowledgment (nack): The consumer explicitly calls negativeAcknowledge(). The broker redelivers the message after negativeAckRedeliveryDelay (default 1 minute).
2. Ack timeout: If the consumer does not acknowledge within ackTimeout, the broker automatically redelivers. By default, ackTimeout is 0 (disabled). With no timeout configured, messages can stay in flight indefinitely.
3. Retry letter topic: The consumer calls reconsumeLater() with enableRetry(true). Messages go to a retry topic and are redelivered after a delay. The retry count is persisted as a message property.

After maxRedeliveryCount redeliveries, messages are supposed to route to the DLQ. The default DLQ topic format is <topicname>-<subscriptionname>-DLQ. DLQ is supported in Shared and Key_Shared subscription types only.

The redelivery counter for nack and ack timeout is tracked on the broker, in memory, at the subscription level: the broker increments it on explicit redelivery commands (nack or ack timeout) and it resets on broker restart, bundle unload, topic unload, and consumer disconnect.

The critical problem is that the redelivery counter for nack and ack timeout is kept in memory. It resets on broker restart, bundle unload, topic unload, and consumer disconnect. This means maxRedeliveryCount may never be reached, and failing messages can be redelivered indefinitely without reaching the DLQ. Only reconsumeLater() with enableRetry(true) persists the retry count as a message property, surviving restarts and disconnects.

flowchart TD
    A["Broker dispatches message"] --> B["Consumer receives and processes"]
    B --> C{"Outcome"}
    C -->|Ack| D["Cursor advances: forward progress"]
    C -->|"Nack, ackTimeout, or crash"| E["Message scheduled for redelivery"]
    E --> F{"redeliveryCount check"}
    F -->|"Below maxRedeliveryCount"| A
    F -->|"Reaches maxRedeliveryCount"| G["Route to dead letter topic"]
    F -->|"Counter reset: restart, disconnect, bundle unload"| A

When the counter resets, the message re-enters the dispatch cycle instead of reaching the DLQ. A poison message is any message that deterministically fails processing: malformed payload, schema mismatch, oversized message triggering OOM, or a downstream dependency that times out. The consumer receives it, fails, and the broker sends it again. If the consumer crashes before acknowledging (OOM, unhandled exception), the in-memory counter does not increment. The message returns with redeliveryCount = 0 and kills the next consumer too.

Common causes

Quick checks

# Redelivery rate per subscription (Prometheus metrics)
curl -s http://<broker-host>:8080/metrics | grep msg_rate_redeliver

# Full subscription stats from the Admin API
pulsar-admin topics stats persistent://tenant/namespace/topic
# Key fields to inspect:
#   subscriptions.<name>.msgRateRedeliver             -> redelivery rate
#   subscriptions.<name>.msgRateOut                   -> dispatch rate
#   subscriptions.<name>.unackedMessages              -> dispatched but not acked
#   subscriptions.<name>.blockedSubscriptionOnUnackedMsgs -> dispatch freeze flag
#   subscriptions.<name>.consumers[]                  -> per-consumer stats, connected count

# Whether the DLQ topic exists and has received messages
pulsar-admin topics stats persistent://tenant/namespace/topic-subscription-DLQ

# Consumer-level redelivery (requires exposeConsumerLevelMetricsInPrometheus=true)
curl -s http://<broker-host>:8080/metrics | grep pulsar_consumer_msg_rate_redeliver

How to diagnose it

1. Confirm the storm. Check pulsar_subscription_msg_rate_redeliver for the affected topic. Compare it to pulsar_rate_out for the same topic and subscription. Redelivery exceeding 10% of dispatch sustained for more than 5 minutes indicates a problem. At 100%, no message is being successfully processed.

2. Isolate the subscription. The storm is per-subscription. If only one subscription on one topic is affected, suspect a poison message or consumer bug. If multiple subscriptions across topics are affected simultaneously, suspect a broker-wide event (restart, bundle unload) that reset in-memory counters.

3. Identify the redelivery mechanism. Check consumer configuration and application logs:

   • Nack-based redelivery: the consumer calls negativeAcknowledge() on failure. Messages return after negativeAckRedeliveryDelay (default 1 minute).
   • Ack timeout: ackTimeout is set and processing takes longer. Compare the configured value against actual processing latency.
   • Crash before ack: the consumer crashes (OOM, exception) before acking. Check restart count and crash logs.

4. Check whether the DLQ escape valve works. If DeadLetterPolicy with maxRedeliveryCount is configured, inspect the DLQ topic for message arrival. An empty DLQ during a redelivery storm means the counter is resetting before reaching the threshold.

5. Check for counter reset conditions. The in-memory redelivery counter resets on broker restart, bundle unload, topic unload, and consumer disconnect. Review broker uptime, recent unloads (pulsar_lb_unload_bundle_total), and consumer connection stability. Frequent resets make maxRedeliveryCount unreachable.

6. Verify subscription type. DLQ is supported in Shared and Key_Shared only. If the subscription is Exclusive or Failover, DeadLetterPolicy will never route messages to a DLQ.

7. Inspect the failing message. Capture the message content from consumer logs. A poison message causes a deterministic failure: parse error, schema mismatch, null field access, oversized payload. The failure reproduces on every delivery attempt.

8. Check for the ackTimeout implicit default. In Pulsar 2.3.x through 2.11.x, setting DeadLetterPolicy on the Java consumer implicitly set a 30-second ackTimeout if no timeout was explicitly provided. If processing takes longer than 30 seconds, every message triggers a redelivery. This default was removed in Pulsar 3.0.0. The implicit set also fired when ackTimeout(0) was called before deadLetterPolicy(), so an explicit 0 did not protect you on those versions; the block was removed in 3.0.0.

Metrics and signals to monitor

Fixes

Poison message that always fails processing

Short-term: break the loop by resetting the subscription cursor past the poison message using pulsar-admin topics reset-cursor.

WARNING: this is destructive. The skipped message is permanently lost. Coordinate with the application team before doing this.

Medium-term: fix the consumer to handle the message without crashing or looping. Catch the exception, log the message content, and acknowledge it explicitly. A single unprocessable message should never block all forward progress.

Long-term: switch from negativeAcknowledge() to reconsumeLater() with enableRetry(true). The retry count is persisted as a message property and survives broker restarts, bundle unloads, and consumer disconnects. This is the only reliable mechanism for enforcing maxRedeliveryCount in the presence of failures.

ackTimeout shorter than processing time

If ackTimeout is shorter than your P99 processing latency, slow messages trigger redelivery even though they would eventually succeed. This creates cascading redeliveries that look like a poison message but are not.

Fix: set ackTimeout to 0 (disabled) if you do not need a timeout safety net, or set it to at least 3-5x your P99 processing time. On Pulsar 2.3.x through 2.11.x, verify that setting DeadLetterPolicy did not silently enable a 30-second ack timeout.

Consumer crash before ack

When a consumer crashes before acknowledging, the broker redelivers the message, but the in-memory redelivery counter does not increment on crash or disconnect. The message returns with redeliveryCount = 0. If the same message always causes the crash, you get infinite redelivery without ever approaching maxRedeliveryCount.

This is a known open issue (GitHub issue #18239); it remains open as of this review. message.getRedeliveryCount() returns 0 for redelivered messages after consumer reconnect because the broker only increments the counter when the client explicitly calls redeliver (nack or ack timeout), not on crash.

Fix: identify the crash cause. For OOM from large payloads, add message size limits or increase consumer heap. For unhandled exceptions, add try-catch around message processing. Use reconsumeLater() with enableRetry(true) so the retry count survives consumer restarts.

nack and ackTimeout interaction

If both nack and ack timeout are active on the same consumer, a race can prevent the redelivery count from incrementing correctly. Reported on Pulsar 2.5.0 (GitHub issue #6451): when processing takes longer than ackTimeout and the consumer then sends a negative acknowledgment, the message gets stuck in a loop and never reaches the DLQ. The issue was closed as not-planned and no fix exists in Pulsar 3.x or 4.x.

Fix: avoid using both mechanisms on the same consumer. If you need nack-based redelivery, set ackTimeout to 0.

DLQ not configured or on wrong subscription type

If no DeadLetterPolicy is set, redeliveries are infinite by default. If DeadLetterPolicy is set on an Exclusive or Failover subscription, the DLQ is never created because DLQ requires Shared or Key_Shared.

Fix: configure DeadLetterPolicy with a realistic maxRedeliveryCount (3-5 is typical). Ensure the subscription type is Shared or Key_Shared. Verify the DLQ topic name matches the expected format (<topicname>-<subscriptionname>-DLQ on Pulsar 2.8.x+).

Prevention

• Use reconsumeLater with enableRetry(true) for reliable retry limits. The retry count persists as a message property, surviving broker restarts, bundle unloads, and consumer disconnects. This is the only mechanism that reliably enforces maxRedeliveryCount.
• Set ackTimeout above P99 processing time or disable it. If you enable it, set it to at least 3-5x expected processing latency.
• Do not use both nack and ackTimeout on the same consumer. Known interaction bugs can break the redelivery count.
• Catch exceptions in consumer code, log the message, and acknowledge. Never let a single message block the subscription indefinitely.
• Verify subscription type supports DLQ before relying on it. Shared and Key_Shared only.
• Alert when pulsar_subscription_msg_rate_redeliver exceeds 10% of pulsar_rate_out for more than 5 minutes.
• Monitor DLQ topic arrival rate. Messages reaching the DLQ means the escape valve works. An empty DLQ during a redelivery storm means the counter is resetting.
• Track consumer restart count. Frequent restarts reset the in-memory counter and prevent DLQ routing.

How Netdata helps

• Per-second redelivery metrics. Netdata collects pulsar_subscription_msg_rate_redeliver and pulsar_consumer_msg_rate_redeliver at 1-second resolution, letting you pinpoint the exact moment a redelivery storm begins and correlate it with consumer restarts or bundle unloads.
• Redelivery-to-dispatch ratio. Correlating pulsar_subscription_msg_rate_redeliver with pulsar_rate_out on the same dashboard shows immediately whether redelivery is a minor fraction or approaching 100% (zero forward progress).
• Unacked message saturation. Netdata surfaces pulsar_subscription_unacked_messages alongside the configured limits. When unacked messages approach maxUnackedMessagesPerSubscription (default 200,000), you can see the dispatch freeze forming before it happens.
• Bundle unload correlation. Tracking pulsar_lb_unload_bundle_total alongside redelivery rate reveals whether frequent unloads are resetting the in-memory counter and preventing DLQ routing.
• Anomaly detection. Netdata’s anomaly detection flags unusual spikes in redelivery rate before a static threshold is crossed, which matters because the normal redelivery rate for most subscriptions is near zero.

Related guides

• Apache Pulsar active connections climbing: connection leaks and file descriptor exhaustion
• Apache Pulsar bookie add-entry queue not draining: writes arriving faster than the disk can commit
• Apache Pulsar AutoRecovery stalled: under-replicated ledgers that never heal
• Apache Pulsar bookie disk filling: runway to read-only and how to reclaim space
• Apache Pulsar bookie failure cascade: recovery I/O that topples surviving bookies
• Apache Pulsar bookie read latency high: catch-up reads competing with the write path
• Apache Pulsar bookie read-only: disk full and bookie_SERVER_STATUS at zero
• Apache Pulsar broker down: telling a dead broker from a fenced one
• Apache Pulsar broker GC death spiral: heap pressure, stop-the-world pauses, and lost topic ownership
• Apache Pulsar broker lookup failures: new clients cannot find their topic
• Apache Pulsar entry log GC falling behind: reclaimed space that never comes back
• How Apache Pulsar actually works in production: a mental model for operators