Apache Pulsar dead letter queue filling: maxRedeliveryCount and the DLQ topic

The dead letter topic is where Pulsar parks messages that exhausted their retry budget. When you set DeadLetterPolicy.builder().maxRedeliverCount(N).build() on a Shared or Key_Shared subscription, the consumer stops redelivering a message after N failed attempts, sends it to {topic}-{subscription}-DLQ, and acknowledges it on the origin subscription so backlog clears.

A growing DLQ arrival rate is a ledger of application processing failures. A few poison messages parked for inspection is the DLQ working as designed. A steady stream of thousands of messages per second is a systemic failure. Unmonitored, the DLQ topic itself becomes the incident: messages accumulate, storage grows, and nobody notices until disk fills or a downstream team asks why their data is missing.

Pulsar exposes no dedicated DLQ metric. There is no pulsar_subscription_dlq_rate in Prometheus. You infer DLQ pressure from the redelivery rate on the origin subscription and from backlog stats on the DLQ topic itself.

What this means

When maxRedeliverCount is exceeded, the message is gone from the origin subscription’s perspective. It will not be redelivered. This prevents poison messages from blocking the subscription indefinitely. But the DLQ topic is just another Pulsar topic. If nobody is consuming it, messages accumulate subject to retention policy. Without initialSubscriptionName configured in the DeadLetterPolicy (available since Pulsar 2.10 via PIP-124), messages sent to a DLQ topic with no subscription may be auto-deleted based on retention settings: silent data loss of your failed messages.

The critical diagnostic question is always: are these failures concentrated or broad? A few messages failing repeatedly points to poison messages, schema mismatches, or deserialization errors. Many different messages failing at once points to a downstream outage, a bad deploy, or a consumer bug affecting all traffic. The answer determines whether you inspect message content or check downstream dependencies.

flowchart TD
    A[Producer publishes message] --> B[Broker dispatches to consumer]
    B --> C{Consumer processes}
    C -->|Success| D[Acknowledge: cursor advances]
    C -->|Failure: nack or ack timeout| E[Broker redelivers]
    E --> F{redeliveryCount reached maxRedeliverCount?}
    F -->|No: counter increments| B
    F -->|Yes| G[Route to DLQ topic]
    G --> H[Auto-ack in origin subscription]
    H --> I[Backlog clears on origin topic]
    G --> J{Subscription on DLQ topic?}
    J -->|No| K[Messages accumulate or deleted by retention]
    J -->|Yes| L[Consumer inspects or replays]

One subtlety: the redelivery counter tracked by negative acknowledgment is in-memory only. It resets on broker restart, bundle unload, topic unload, or consumer disconnect. maxRedeliverCount may never be reached with negative acks alone if the broker restarts or the consumer reconnects between retries. Failing messages can be redelivered indefinitely without ever reaching the DLQ. To reliably honor maxRedeliverCount, use reconsumeLater with enableRetry(true), which persists the retry count as a message property on the retry letter topic ({topic}-{subscription}-RETRY).

Common causes

Quick checks

Safe, read-only operations.

# Check redelivery rate on the origin subscription
curl -s http://<broker-host>:8080/metrics | grep msg_rate_redeliver

# Check origin subscription stats: backlog, unacked, redelivery
pulsar-admin topics stats persistent://tenant/namespace/topic
# Look for: subscriptions.<name>.msgRateRedeliver
# Look for: subscriptions.<name>.unackedMessages
# Look for: subscriptions.<name>.msgBacklog

# Check DLQ topic backlog and whether a subscription exists
pulsar-admin topics stats persistent://tenant/namespace/topic-subscription-DLQ
# Empty subscriptions object means no consumer

# Check subscription unacked saturation
curl -s http://<broker-host>:8080/metrics | grep pulsar_subscription_unacked_messages

The DLQ topic naming format changed in Pulsar 2.8.0 (see Issue 9327). In 2.6.x and 2.7.x, the default was <subscription>-DLQ under the topic’s namespace (no topic name prefix). In 2.8.0 and later, it is <topic>-<subscription>-DLQ. If you upgraded from 2.7.x to 2.8.x without cleaning up old DLQ topics, Pulsar may continue using the old naming format.

How to diagnose

1. Confirm the DLQ is actually growing. Use pulsar-admin topics stats on the DLQ topic. Check msgBacklog and whether it is increasing. If there is no subscription on the DLQ topic, msgBacklog may report zero even though messages are arriving and being deleted by retention. Check message expiration rate on the DLQ topic if retention is configured.

2. Measure redelivery rate on the origin subscription. Pull pulsar_subscription_msg_rate_redeliver for the affected subscription. Compare it to the dispatch rate for the same subscription (pulsar_subscription_msg_rate_out). If redelivery is more than 10% of dispatch rate, consumers are failing to process a significant fraction of messages. At 100%, no forward progress is being made.

3. Determine whether failures are concentrated or broad. If only a few messages are cycling through the DLQ, inspect their content. If the DLQ arrival rate correlates with a downstream service degradation window, the root cause is external. Check consumer application logs for the time window when DLQ arrivals started.

4. Check the redelivery mechanism in use. If the consumer uses negativeAcknowledge, the redelivery counter is in-memory and resets on broker restart, bundle unload, or consumer disconnect. Messages may never reach maxRedeliverCount. If the consumer uses reconsumeLater with enableRetry(true), the counter is persisted on the retry letter topic and survives restarts.

5. Verify the DLQ topic has a consumer or initial subscription. Without initialSubscriptionName in the DeadLetterPolicy, messages arrive at the DLQ topic with no subscription. Depending on retention and TTL settings, they may be silently deleted. Check pulsar-admin topics stats on the DLQ topic for active subscriptions.

6. Check for version-specific bugs. From Pulsar 2.5.0 to 2.10.x, a known issue (introduced by PR #5895, reverted by PR #17060 in Pulsar 3.0.0) caused redeliveryCount to increment eagerly each time the broker re-dispatched a message, including fills to the consumer receive queue, not when the consumer actually received it. Messages hit maxRedeliverCount prematurely with fewer retries than expected. On versions before 4.1.0, schema validation can prevent DLQ delivery entirely (issue #10377, fixed by PR #24663). Check your Pulsar version against known fixes.

Metrics and signals to monitor

Fixes

Poison messages: inspect and fix the consumer

If a small number of messages are cycling through the DLQ, inspect their content. Identify the deserialization or processing error. Fix the consumer code to handle the malformed input, then replay the DLQ messages through a corrected consumer or discard them.

# Read messages from the DLQ topic for inspection
# WARNING: creates a new subscription; clean it up after inspection
pulsar-client consume persistent://tenant/namespace/topic-subscription-DLQ -s dlq-inspection -n 10

A few poison messages parked for inspection is the DLQ working as designed.

Downstream outage: fix the dependency, then replay

If the DLQ filled because a downstream dependency was down, the messages themselves are valid. After restoring the downstream service, replay the DLQ messages through a consumer that can process them. If the volume is too large to replay, coordinate with the application team on which messages can be skipped.

Do not delete the DLQ topic until you have confirmed the messages are either replayed or intentionally discarded.

ackTimeout too short: increase the timeout or switch to reconsumeLater

If consumer processing time P99 exceeds the configured ackTimeout, messages are redelivered because the timeout fired, not because processing failed. Increase ackTimeout to accommodate processing time, or switch to explicit reconsumeLater with enableRetry(true).

Since Pulsar 3.0.0, the Java SDK no longer sets a default 30-second ackTimeout when a deadLetterPolicy is configured. If you relied on this implicit default in Pulsar 2.x, messages may stop reaching the DLQ after upgrading to 3.0 or later unless you set an explicit ackTimeout or use negative acks.

Redelivery counter resets: switch to reconsumeLater

If you are using negativeAcknowledge and the redelivery counter resets too frequently for maxRedeliverCount to trigger, switch to reconsumeLater with enableRetry(true). This persists the retry count as a message property on the retry letter topic ({topic}-{subscription}-RETRY), so it survives broker restarts, bundle unloads, and consumer disconnects.

Version bugs: upgrade

If you are on Pulsar 2.5.0 to 2.10.x and messages reach the DLQ with fewer retries than configured, upgrade to Pulsar 3.0.0 or later. PR #17060 reverted the eager redeliveryCount incrementing behavior. If schema validation is blocking DLQ delivery on versions before 4.1.0, upgrade or remove schema validation for the affected topics.

Prevention

• Monitor redelivery rate as a first-class signal. Pulsar does not expose a dedicated DLQ metric in Prometheus. Track pulsar_subscription_msg_rate_redeliver on the origin subscription as the leading indicator, and periodically check DLQ topic backlog via Admin API. Alert on sustained redelivery rate above 10% of dispatch rate.

• Configure initialSubscriptionName in your DeadLetterPolicy. Without it, messages on the DLQ topic with no subscription are subject to retention-based deletion. Set an initial subscription so messages are preserved for inspection.

• Prefer reconsumeLater over negative ack for retry semantics. The in-memory redelivery counter from negative ack is fragile. reconsumeLater with enableRetry(true) persists the retry count, making DLQ behavior predictable across restarts and reconnections.

• Distinguish intentional DLQ from systemic failure in alerting. A few messages per minute in the DLQ is normal. A sudden spike that correlates with a downstream event is an incident. Rate-of-change alerting on DLQ topic backlog is more useful than absolute thresholds.

• Verify DLQ behavior after version upgrades. The 2.8.x naming format change, the 3.0.x ackTimeout default removal, and the 4.1.0 schema validation fix all change DLQ behavior. Test consumer retry logic against the DLQ after any Pulsar upgrade.

How Netdata helps

• Per-second redelivery rate visibility. Netdata collects pulsar_subscription_msg_rate_redeliver at one-second resolution. A redelivery spike that lasts 30 seconds and triggers maxRedeliverCount is visible before a 60-second scrape interval would catch it.

• Correlating redelivery with consumer health signals. Overlay pulsar_subscription_msg_rate_redeliver, unacked message count, and backlog on the same timeline. When unacked count rises and then redelivery spikes, the causal chain is visible: consumer stalls, messages time out, redelivery accelerates, DLQ fills.

• Anomaly detection on redelivery rate. A poison message scenario produces low baseline redelivery with sudden spikes. A downstream outage produces sustained elevated redelivery across multiple subscriptions. Both patterns deviate from learned baselines.

• Separating origin topic health from DLQ accumulation. Per-topic metrics let you monitor the origin subscription and the DLQ topic independently. If the origin subscription is healthy (backlog draining, low unacked) while the DLQ accumulates, the failures are recent and ongoing.

• Backlog quota and storage pressure correlation. Netdata correlates bookie disk usage with topic-level backlog, so you can see whether DLQ growth is pushing bookies toward their read-only threshold.

Related guides

• Apache Pulsar active connections climbing: connection leaks and file descriptor exhaustion
• Apache Pulsar bookie add-entry queue not draining: writes arriving faster than the disk can commit
• Apache Pulsar AutoRecovery stalled: under-replicated ledgers that never heal
• Apache Pulsar backlog age vs size: the latency depth alone cannot show
• Apache Pulsar backlog quota exceeded: producers held or rejected when consumers stall
• Apache Pulsar bookie disk filling: runway to read-only and how to reclaim space
• Apache Pulsar bookie failure cascade: recovery I/O that topples surviving bookies
• Apache Pulsar bookie read latency high: catch-up reads competing with the write path
• Apache Pulsar bookie read-only: disk full and bookie_SERVER_STATUS at zero
• Apache Pulsar broker down: telling a dead broker from a fenced one
• Apache Pulsar broker GC death spiral: heap pressure, stop-the-world pauses, and lost topic ownership
• Apache Pulsar broker lookup failures: new clients cannot find their topic