The dead letter topic is where Pulsar parks messages that exhausted their retry budget. When you set DeadLetterPolicy.builder().maxRedeliverCount(N).build() on a Shared or Key_Shared subscription, the broker stops redelivering a message after N failed attempts, routes it to {topic}-{subscription}-DLQ, and auto-acks it in the origin subscription so backlog clears.

A growing DLQ arrival rate is a ledger of application processing failures. A few poison messages parked for inspection is the DLQ working as designed. A steady stream of thousands of messages per second is a systemic failure. Unmonitored, the DLQ topic itself becomes the incident: messages accumulate, storage grows, and nobody notices until disk fills or a downstream team asks why their data is missing.

Pulsar exposes no dedicated DLQ metric. There is no pulsar_subscription_dlq_rate in Prometheus. You infer DLQ pressure from the redelivery rate on the origin subscription and from backlog stats on the DLQ topic itself.

What this means

When maxRedeliveryCount is exceeded, the message is gone from the origin subscription’s perspective. It will not be redelivered. This prevents poison messages from blocking the subscription indefinitely. But the DLQ topic is just another Pulsar topic. If nobody is consuming it, messages accumulate subject to retention policy. Without initialSubscriptionName configured in the DeadLetterPolicy (available since Pulsar 2.10 via PIP-124), messages sent to a DLQ topic with no subscription may be auto-deleted based on retention settings: silent data loss of your failed messages.

The critical diagnostic question is always: are these failures concentrated or broad? A few messages failing repeatedly points to poison messages, schema mismatches, or deserialization errors. Many different messages failing at once points to a downstream outage, a bad deploy, or a consumer bug affecting all traffic. The answer determines whether you inspect message content or check downstream dependencies.

flowchart TD
    A[Producer publishes message] --> B[Broker dispatches to consumer]
    B --> C{Consumer processes}
    C -->|Success| D[Acknowledge: cursor advances]
    C -->|Failure: nack or ack timeout| E[Broker redelivers]
    E --> F{redeliveryCount reached maxRedeliveryCount?}
    F -->|No: counter increments| B
    F -->|Yes| G[Route to DLQ topic]
    G --> H[Auto-ack in origin subscription]
    H --> I[Backlog clears on origin topic]
    G --> J{Subscription on DLQ topic?}
    J -->|No| K[Messages accumulate or deleted by retention]
    J -->|Yes| L[Consumer inspects or replays]

One subtlety: the redelivery counter tracked by negative acknowledgment is in-memory only. It resets on broker restart, bundle unload, topic unload, or consumer disconnect. maxRedeliveryCount may never be reached with negative acks alone if the broker restarts or the consumer reconnects between retries. Failing messages can be redelivered indefinitely without ever reaching the DLQ. To reliably honor maxRedeliveryCount, use reconsumeLater with enableRetry(true), which persists the retry count as a message property on the retry letter topic ({topic}-{subscription}-RETRY).

Common causes

CauseWhat it looks likeFirst thing to check
Poison messagesSmall number of distinct messages repeatedly hitting DLQ. Redelivery rate concentrated on a few message IDs or keys.Inspect DLQ payloads for malformed data, schema violations, or unparseable content.
Downstream outageSudden, sustained DLQ arrival rate across many subscriptions or topics. Redelivery rate spikes broadly.Check downstream dependency health. Correlate DLQ growth start time with the downstream incident timeline.
Consumer processing bugDLQ fills after a consumer deploy. All messages for affected subscriptions fail. Error rate in consumer logs.Check consumer application logs for exceptions during message processing. Review recent code changes.
ackTimeout too shortMessages redelivered because processing exceeds ackTimeout, not because processing failed. DLQ fills with messages that would have succeeded given more time.Compare consumer processing time P99 against configured ackTimeout. If processing takes 25s and timeout is 20s, every slow message is nacked.
Redelivery counter resetMessages that should reach DLQ never do, or reach it unpredictably. Counter resets on broker restart or consumer reconnect.Check whether you are using negative ack (in-memory counter) vs reconsumeLater (persisted counter). Check broker restart history.
Schema validation blocking DLQ deliveryMessages fail processing but never appear in DLQ. The DLQ producer itself fails because schema validation rejects the message.Check Pulsar version. Schema validation for DLQ messages is skipped starting in Pulsar 4.1.0. On earlier versions, messages with schemas may fail to publish to the DLQ.

Quick checks

Safe, read-only operations.

# Check redelivery rate on the origin subscription
curl -s http://<broker-host>:8080/metrics | grep msg_rate_redeliver

# Check origin subscription stats: backlog, unacked, redelivery
pulsar-admin topics stats persistent://tenant/namespace/topic
# Look for: subscriptions.<name>.msgRateRedeliver
# Look for: subscriptions.<name>.unackedMessages
# Look for: subscriptions.<name>.msgBacklog

# Check DLQ topic backlog and whether a subscription exists
pulsar-admin topics stats persistent://tenant/namespace/topic-subscription-DLQ
# Empty subscriptions object means no consumer

# Check subscription unacked saturation
curl -s http://<broker-host>:8080/metrics | grep pulsar_subscription_unacked_messages

The DLQ topic naming format changed in Pulsar 2.8.x. In 2.6.x and 2.7.x, the default was <subscription>-DLQ (no topic name prefix). In 2.8.x and later, it is <topic>-<subscription>-DLQ. If you upgraded from 2.7.x to 2.8.x without cleaning up old DLQ topics, Pulsar may continue using the old naming format.

How to diagnose

  1. Confirm the DLQ is actually growing. Use pulsar-admin topics stats on the DLQ topic. Check msgBacklog and whether it is increasing. If there is no subscription on the DLQ topic, msgBacklog may report zero even though messages are arriving and being deleted by retention. Check message expiration rate on the DLQ topic if retention is configured.

  2. Measure redelivery rate on the origin subscription. Pull pulsar_subscription_msg_rate_redeliver for the affected subscription. Compare it to the dispatch rate for the same subscription. If redelivery is more than 10% of dispatch rate, consumers are failing to process a significant fraction of messages. At 100%, no forward progress is being made.

  3. Determine whether failures are concentrated or broad. If only a few messages are cycling through the DLQ, inspect their content. If the DLQ arrival rate correlates with a downstream service degradation window, the root cause is external. Check consumer application logs for the time window when DLQ arrivals started.

  4. Check the redelivery mechanism in use. If the consumer uses negativeAcknowledge, the redelivery counter is in-memory and resets on broker restart, bundle unload, or consumer disconnect. Messages may never reach maxRedeliveryCount. If the consumer uses reconsumeLater with enableRetry(true), the counter is persisted on the retry letter topic and survives restarts.

  5. Verify the DLQ topic has a consumer or initial subscription. Without initialSubscriptionName in the DeadLetterPolicy, messages arrive at the DLQ topic with no subscription. Depending on retention and TTL settings, they may be silently deleted. Check pulsar-admin topics stats on the DLQ topic for active subscriptions.

  6. Check for version-specific bugs. On Pulsar 2.8.x to 2.9.x, a known issue (PR #17060) caused redeliveryCount to increment eagerly each time the broker delivered a message to the consumer receive queue, not when the consumer actually received it. Messages hit maxRedeliveryCount prematurely with fewer retries than expected. On versions before 4.1.0, schema validation can prevent DLQ delivery entirely (issue #10377 ). Check your Pulsar version against known fixes.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
pulsar_subscription_msg_rate_redeliverPrimary proxy for DLQ pressure since no dedicated DLQ rate metric exists.Sustained rate above 10% of dispatch rate. 100% means zero forward progress.
DLQ topic backlog (pulsar-admin topics stats)Direct measure of DLQ accumulation. Must be checked via Admin API, not Prometheus.Growing backlog with no consumer draining it.
pulsar_subscription_unacked_messagesMessages dispatched but not acknowledged. When this hits maxUnackedMessagesPerSubscription (default 200,000), dispatch freezes silently.Sustained count above 50% of configured limit.
Subscription dispatch rateCompare against redelivery rate to compute the failure ratio.Dispatch rate stable but redelivery rate climbing: a growing fraction of messages are failing.
pulsar_subscription_msg_rate_expiredMessages deleted by TTL before consumption. If DLQ messages are also expiring on the DLQ topic, you are losing failed messages silently.Non-zero expiration rate on topics where message loss is unacceptable.
Consumer connection countConsumers that disconnect trigger redelivery counter resets (with negative ack). Frequent reconnects destabilize the retry counter.Connection churn on the origin subscription’s consumers.

Fixes

Poison messages: inspect and fix the consumer

If a small number of messages are cycling through the DLQ, inspect their content. Identify the deserialization or processing error. Fix the consumer code to handle the malformed input, then replay the DLQ messages through a corrected consumer or discard them.

# Read messages from the DLQ topic for inspection
# WARNING: creates a new subscription; clean it up after inspection
pulsar-client consume persistent://tenant/namespace/topic-subscription-DLQ -s dlq-inspection -n 10

A few poison messages parked for inspection is the DLQ working as designed.

Downstream outage: fix the dependency, then replay

If the DLQ filled because a downstream dependency was down, the messages themselves are valid. After restoring the downstream service, replay the DLQ messages through a consumer that can process them. If the volume is too large to replay, coordinate with the application team on which messages can be skipped.

Do not delete the DLQ topic until you have confirmed the messages are either replayed or intentionally discarded.

ackTimeout too short: increase the timeout or switch to reconsumeLater

If consumer processing time P99 exceeds the configured ackTimeout, messages are redelivered because the timeout fired, not because processing failed. Increase ackTimeout to accommodate processing time, or switch to explicit reconsumeLater with enableRetry(true).

Since Pulsar 3.0.x, the Java SDK no longer sets a default 30-second ackTimeout when a deadLetterPolicy is configured. If you relied on this implicit default in Pulsar 2.x, messages may stop reaching the DLQ after upgrading to 3.x unless you set an explicit ackTimeout or use negative acks.

Redelivery counter resets: switch to reconsumeLater

If you are using negativeAcknowledge and the redelivery counter resets too frequently for maxRedeliveryCount to trigger, switch to reconsumeLater with enableRetry(true). This persists the retry count as a message property on the retry letter topic ({topic}-{subscription}-RETRY), so it survives broker restarts, bundle unloads, and consumer disconnects.

Version bugs: upgrade

If you are on Pulsar 2.8.x to 2.9.x and messages reach the DLQ with fewer retries than configured, upgrade. PR #17060 reverted the eager redeliveryCount incrementing behavior. If schema validation is blocking DLQ delivery on versions before 4.1.0, upgrade or remove schema validation for the affected topics.

Prevention

  • Monitor redelivery rate as a first-class signal. Pulsar does not expose a dedicated DLQ metric in Prometheus. Track pulsar_subscription_msg_rate_redeliver on the origin subscription as the leading indicator, and periodically check DLQ topic backlog via Admin API. Alert on sustained redelivery rate above 10% of dispatch rate.

  • Configure initialSubscriptionName in your DeadLetterPolicy. Without it, messages on the DLQ topic with no subscription are subject to retention-based deletion. Set an initial subscription so messages are preserved for inspection.

  • Prefer reconsumeLater over negative ack for retry semantics. The in-memory redelivery counter from negative ack is fragile. reconsumeLater with enableRetry(true) persists the retry count, making DLQ behavior predictable across restarts and reconnections.

  • Distinguish intentional DLQ from systemic failure in alerting. A few messages per minute in the DLQ is normal. A sudden spike that correlates with a downstream event is an incident. Rate-of-change alerting on DLQ topic backlog is more useful than absolute thresholds.

  • Verify DLQ behavior after version upgrades. The 2.8.x naming format change, the 3.0.x ackTimeout default removal, and the 4.1.0 schema validation fix all change DLQ behavior. Test consumer retry logic against the DLQ after any Pulsar upgrade.

How Netdata helps

  • Per-second redelivery rate visibility. Netdata collects pulsar_subscription_msg_rate_redeliver at one-second resolution. A redelivery spike that lasts 30 seconds and triggers maxRedeliveryCount is visible before a 60-second scrape interval would catch it.

  • Correlating redelivery with consumer health signals. Overlay pulsar_subscription_msg_rate_redeliver, unacked message count, and backlog on the same timeline. When unacked count rises and then redelivery spikes, the causal chain is visible: consumer stalls, messages time out, redelivery accelerates, DLQ fills.

  • Anomaly detection on redelivery rate. A poison message scenario produces low baseline redelivery with sudden spikes. A downstream outage produces sustained elevated redelivery across multiple subscriptions. Both patterns deviate from learned baselines.

  • Separating origin topic health from DLQ accumulation. Per-topic metrics let you monitor the origin subscription and the DLQ topic independently. If the origin subscription is healthy (backlog draining, low unacked) while the DLQ accumulates, the failures are recent and ongoing.

  • Backlog quota and storage pressure correlation. Netdata correlates bookie disk usage with topic-level backlog, so you can see whether DLQ growth is pushing bookies toward their read-only threshold.