$ guides / apache-pulsar / apache-pulsar-consumer-stalled-no-acks ▌

Operations Guides

Apache Pulsar consumers connected but not acknowledging: the silent consumer stall

Your Pulsar cluster looks healthy. Brokers are up, bookies are writable, producers are publishing at normal rates. But for one topic or subscription, pulsar_rate_out has gone flat. Messages are flowing in but nothing is coming out. The backlog is climbing. Consumers show as connected, but the acknowledgment rate is zero.

No error surfaces in broker logs. The broker serves connections and bookie writes are fast. The problem lives in the consumer application or in the interaction between consumer configuration and broker dispatch policy.

Two variants produce this symptom and require different responses:

Variant 1: no consumers. The consumer application crashed, was undeployed, or never started. No consumer connections exist for the subscription. Backlog grows linearly with publish rate. Redelivery rate is zero because there is nobody to deliver to.
Variant 2: stuck consumers. Consumers are connected to the broker. They may even have messages dispatched to them (visible as non-zero unackedMessages in topic stats). But they are not calling acknowledge(). Redelivery behavior depends entirely on the ackTimeout setting. If ackTimeout is 0 (the default), messages are never redelivered, redelivery rate stays at zero, and the stall is truly silent. If ackTimeout is set, you will see high redelivery as the broker keeps pushing the same unacked messages back.

What this means

The core signal is an imbalance between pulsar_rate_in and pulsar_rate_out for a specific topic or subscription. Producers are writing normally, but the dispatch rate to consumers is flat or zero.

The critical distinction from a slow-but-progressing consumer is the acknowledgment rate. A slow consumer still acks messages, just slower than producers publish. The stalled consumer acks nothing. The dispatch rate is flat, not merely reduced.

flowchart TD
    A["rate_in normal, rate_out flat"] --> B{Consumers connected?}
    B -->|No| C["No consumers: crashed or undeployed"]
    C --> C1["Backlog grows linearly, zero redelivery"]
    B -->|Yes| D{Redelivery rate?}
    D -->|Zero| E["Stuck: deadlock or ackTimeout=0"]
    E --> E1["unackedMessages growing, availablePermits=0"]
    D -->|High| F["Poison message loop"]
    F --> F1["Same messages cycling, DLQ may fill"]

A third trap: pulsar_subscription_back_log measures entries not yet dispatched, not entries dispatched but unacknowledged. If the broker has dispatched messages to consumers and those messages are unacked, the backlog metric may appear stable or even zero while no forward progress is made. In that case, pulsar_subscription_unacked_messages and pulsar_subscription_msg_rate_redeliver are the signals that reveal the stall.

Common causes

Cause	What it looks like	First thing to check
Consumer crashed or undeployed	Zero consumer connections, backlog grows linearly, zero redelivery	Pod or process status, deployment logs
Consumer deadlock or thread starvation	Consumers connected, `availablePermits` at 0, flat ack rate, zero redelivery if ackTimeout=0	Consumer thread dump, application logs
Slow downstream dependency (DB, API)	Consumers connected, `unackedMessages` growing toward limit, eventual dispatch freeze	Downstream system latency, connection pool stats
Poison message causing repeated failures	High `msgRateRedeliver`, same messages cycling, consumers may crash and restart	Message content at head of backlog, consumer error logs
ackTimeout not configured (default 0)	Messages dispatched but never redelivered on stall, truly silent failure	Consumer builder configuration for `ackTimeout`

Quick checks

# Check topic stats: msgBacklog, unackedMessages, msgRateRedeliver, availablePermits
pulsar-admin topics stats persistent://tenant/namespace/topic

# Check subscription backlog from Prometheus metrics
curl -s http://<broker-host>:8080/metrics | grep pulsar_subscription_back_log

# Check unacked messages per subscription
curl -s http://<broker-host>:8080/metrics | grep pulsar_subscription_unacked_messages

# Check redelivery rate per subscription
curl -s http://<broker-host>:8080/metrics | grep msg_rate_redeliver

# Check rate in vs rate out for the affected topic
curl -s http://<broker-host>:8080/metrics | grep -E "pulsar_(rate|throughput)_(in|out)"

# Check if subscription is blocked on unacked messages
<!-- Verified: pulsar_subscription_blocked_on_unacked_messages (0/1) exists as a Prometheus metric; per-consumer variant pulsar_consumer_blocked_on_unacked_messages also exists. -->
curl -s http://<broker-host>:8080/metrics | grep pulsar_subscription_blocked_on_unacked_messages

# Confirm broker health is not the root cause
curl -sf http://<broker-host>:8080/admin/v2/brokers/health

# Confirm bookie health is not the root cause
curl -s http://<bookie-host>:8000/metrics | grep bookie_SERVER_STATUS

How to diagnose it

Confirm the scope. Is the stall affecting one topic, one subscription, or the entire cluster? If only specific topics are affected while others are healthy, the problem is consumer-side, not infrastructure. Run pulsar-admin topics stats on the affected topic and compare rate_in against rate_out in the subscription stats.
Identify the variant. Look at the consumers array in the topic stats output. If it is empty, you have Variant 1 (no consumers). If consumers are listed, check availablePermits and unackedMessages per consumer. An availablePermits value of 0 means the client library’s internal queue is full and receive() is not being called: the application is not picking up messages.
Check the acknowledgment rate. In the consumer stats, look at lastAckedTimestamp (the lastAckedTime field does not exist). If this timestamp is stale relative to the current time, the consumer stopped acking. Compare it to lastConsumedTimestamp to see whether the consumer is at least receiving messages.
Check redelivery behavior. Look at msgRateRedeliver in the subscription stats. If it is zero and ackTimeout is not configured, the stall is truly silent: the broker will never redeliver the messages. If it is high, you likely have a poison message or a processing failure loop.
Check if dispatch is frozen. Look for blockedSubscriptionOnUnackedMsgs in the topic stats. If this is true, the unacked message count has hit the configured limit (maxUnackedMessagesPerSubscription, default 200,000, or maxUnackedMessagesPerConsumer, default 50,000) and the broker has stopped dispatching entirely. There is no error in the logs, just silence.
Verify broker and bookie health. Confirm that broker publish latency is normal, bookie journal sync latency is within baseline, and bookie disk usage is not critical. If these are degraded, the stall may be a downstream symptom of an infrastructure problem rather than a consumer problem.
Inspect consumer application logs and thread dumps. Look for exceptions, timeouts, or deadlock indicators in the consumer process. Thread dumps are essential if you suspect a deadlock or thread pool starvation.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`pulsar_rate_out` per topic	Dispatch rate to consumers	Flat or zero while `pulsar_rate_in` is normal
`pulsar_subscription_back_log`	Accumulation of undispatched messages	Sustained growth for specific subscriptions
`pulsar_subscription_unacked_messages`	Messages dispatched but not acknowledged	Growing toward `maxUnackedMessagesPerSubscription`. Note: this metric can occasionally report negative values; use the Admin API for authoritative counts.
`pulsar_subscription_msg_rate_redeliver`	Messages re-sent after ack timeout or nack	High rate relative to dispatch rate indicates poison messages or processing failures
`blockedSubscriptionOnUnackedMsgs`	Dispatch frozen by unacked limit	Value is `true`
`availablePermits` per consumer	Client library queue capacity	Zero means consumer is not calling `receive()`
`lastAckedTimestamp` per consumer	Most recent acknowledgment timestamp	Stale relative to current time

Fixes

No consumers (Variant 1)

If the consumers array is empty, the consumer application is not running. Check pod or process status in your orchestration system. Check deployment logs for crash loops or startup failures. Verify that the subscription was not accidentally deleted or the consumer group renamed.

Restart or redeploy the consumer application. The backlog will begin draining once consumers reconnect and start processing.

Stuck consumers: deadlock or thread starvation

If consumers are connected but availablePermits is 0 and unackedMessages is flat or growing:

Take a thread dump of the consumer process. Look for blocked threads, especially in the message processing path.
Check if the consumer’s executor is saturated. If using a shared thread pool, a slow task can starve the acknowledgment path.
If the consumer is deadlocked, restarting the process will clear it. In Shared subscription mode, unacked messages are redelivered to another consumer when the stalled consumer disconnects.

Stuck consumers: slow downstream dependency

If unackedMessages is growing steadily and availablePermits eventually drops to 0:

Check the downstream system (database, API, cache) that the consumer writes to. Latency spikes there translate directly to slow acks here.
Check connection pool sizes. If the consumer is blocked waiting for a database connection, it cannot process or acknowledge messages.
Temporarily increasing maxUnackedMessagesPerSubscription can buy headroom, but it does not fix the root cause. This is a broker-level setting that requires a config update or restart, so plan accordingly. The underlying dependency must be addressed.

Poison messages

If msgRateRedeliver is high relative to pulsar_rate_out:

Inspect the message at the head of the backlog. It may be malformed, too large, or trigger an exception in consumer processing logic.
If a dead letter topic is configured, verify it is receiving messages. Messages should route to the DLQ topic ({topic}-{subscription}-DLQ) after maxRedeliveryCount attempts.
If no DLQ is configured, consider skipping the poison message by acknowledging it manually or using pulsar-admin to reset the subscription cursor forward. This is destructive: skipped messages are lost. Coordinate with the application team before doing this.

ackTimeout not configured

The default ackTimeout is 0 (disabled), meaning the broker will never redeliver unacked messages unless the consumer crashes or disconnects. This is the root cause of many silent stalls: the consumer appears connected, messages appear dispatched, but nothing is acknowledged and nothing is redelivered.

Set an explicit ackTimeout in the consumer builder (30 seconds is a common starting point). This causes the broker to redeliver messages that are not acknowledged within the timeout window.
Be aware that ackTimeout-based redelivery breaks message ordering in Exclusive, Failover, and Key_Shared subscriptions. For ordered subscriptions, implement explicit negative acknowledgment (nack) in consumer code rather than relying on timeout-based redelivery. Note: Pulsar 2.x Java clients automatically applied a 30s ackTimeout when a DeadLetterPolicy was configured without an explicit ackTimeout (issue #2987); client builders since Pulsar 3.0 no longer do this, so set ackTimeout explicitly.

Prevention

Set ackTimeout explicitly. The default of 0 is a trap. Every consumer should have a sensible ackTimeout (30 seconds is a common starting point) so that stalled processing triggers redelivery rather than silent accumulation. For ordered subscriptions where timeout-based redelivery breaks ordering, implement explicit nack handling in consumer code.

Configure a dead letter topic. Without a DLQ, poison messages cycle forever. Set maxRedeliveryCount and a DLQ topic so poison messages are quarantined after a bounded number of attempts.

Monitor the unacked message ratio. Alert when pulsar_subscription_unacked_messages exceeds 50% of the configured maxUnackedMessagesPerSubscription for more than 10 minutes. At 100%, dispatch freezes silently with no error in broker logs.

Monitor redelivery rate. Alert when pulsar_subscription_msg_rate_redeliver exceeds 10% of the dispatch rate. Sustained high redelivery means consumers are receiving but failing to process.

Track per-consumer stats. The availablePermits field per consumer reveals whether the client library is keeping up. A consumer with availablePermits: 0 for an extended period is stalled, even if its connection looks healthy.

Know your backlog quota policy. When backlog grows past the configured quota, the policy (producer_request_hold, producer_exception, or consumer_backlog_eviction) determines whether producers are held, errored, or oldest messages are silently dropped. Understand which policy is in effect so a consumer stall does not become a producer outage or data loss event.

How Netdata helps

Per-second rate correlation. Netdata collects pulsar_rate_in and pulsar_rate_out per second, making the divergence between publish and dispatch visible immediately, often before the backlog has grown enough to trigger threshold-based alerts.
Unacked message tracking. The pulsar_subscription_unacked_messages metric, collected per subscription, reveals when consumers stop acknowledging before the broker freezes dispatch. Correlating unacked count growth with downstream dependency metrics (database latency, API response times) pinpoints the root cause without switching tools.
Redelivery rate as an anomaly signal. ML-based anomaly detection on pulsar_subscription_msg_rate_redeliver surfaces poison message loops early, before they exhaust consumer resources or fill the dead letter topic.
Backlog growth rate, not just size. Rather than alerting on absolute backlog size (which varies by workload), anomaly detection on backlog growth rate identifies when a specific subscription deviates from its established baseline.
Infrastructure exclusion in one view. Correlating consumer stall symptoms with broker publish latency, bookie journal sync latency, and bookie server status in a single dashboard lets you quickly confirm that the infrastructure is healthy and the problem is consumer-side.