$ guides / apache-pulsar / apache-pulsar-message-ttl-expiry ▌

Operations Guides

Apache Pulsar messages expiring before consumers read them: TTL and silent loss

When pulsar_subscription_msg_rate_expired is non-zero on a topic where every message matters, messages are being silently deleted before consumers can read them. No errors appear in consumer logs. No producer failures occur. The broker’s TTL mechanism acknowledges messages on behalf of the subscription without ever delivering them to the consumer application.

The backlog can hide the problem. TTL expiry trims unacked messages from subscription cursors, so backlog can appear stable or declining while real consumer lag grows underneath. Operators monitoring backlog size alone see a healthy-looking number. The data loss is invisible unless you specifically monitor the expiry rate.

TTL is per-subscription and applied asynchronously. Each subscription on a topic has its own backlog and its own expiry behavior. One slow subscription can lose messages to TTL while a fast subscription on the same topic retains everything. This is distinct from retention policy, which governs how long acknowledged messages persist after consumers have processed them.

What this means

TTL causes the broker to automatically acknowledge and delete unacked messages older than the configured window. Once a message passes the TTL threshold without being consumed by a given subscription, the broker marks it as acknowledged on behalf of that subscription, making it eligible for deletion. The consumer never sees the message.

This is designed for use cases where data has temporal value: real-time telemetry, live pricing feeds, ephemeral notifications. For those workloads, delivering stale data can be worse than delivering no data, and TTL prevents stale messages from accumulating indefinitely.

The problem arises when TTL is active on topics where every message must be consumed. Common scenarios:

TTL set at the namespace level, inherited by topics that require full delivery
Consumer outage lasting longer than the TTL window
Consumer processing too slow to drain messages within the TTL window
TTL configured alongside delayed message delivery, where TTL does not respect the scheduled delay and can silently expire messages before their delivery time

The key metric is pulsar_subscription_msg_rate_expired (also visible as msgRateExpired in Admin API subscription stats). On topics where data loss is unacceptable, this value should always be zero. Any non-zero rate means the system is actively discarding messages that consumers never processed.

Operational details that make this harder to catch:

TTL is applied per-subscription, not per-topic. Each subscription has its own cursor and its own expiry behavior. A shared subscription with one slow consumer can lose messages while other subscriptions on the same topic are unaffected.
Expiry runs asynchronously. The broker checks for expired messages on an interval controlled by messageExpiryCheckIntervalInMinutes in broker.conf (default: 5 minutes). There is a lag between when a message crosses the TTL boundary and when it is actually deleted.
Backlog can look healthy while expiry masks the problem. If expiry trims messages at the same rate consumers fall behind, the backlog appears stable. Without monitoring the expiry rate directly, the data loss is invisible.
A hardcoded 1.5x threshold delays actual expiry. The PersistentTopic.isOldestMessageExpired() method multiplies the configured TTL by a hardcoded factor of 1.5. A 24-hour TTL may not actually expire messages until approximately 36 hours have elapsed. This factor is not configurable.

Common causes

Cause	What it looks like	First thing to check
Consumer outage exceeding TTL	`msgRateExpired` spikes, consumer connection count drops to zero	Consumer process health and deployment status
Consumer too slow for TTL window	`msgRateExpired` steady non-zero, backlog flat or slowly growing while `rate_in` continues	Consumer processing latency vs configured TTL value
TTL misconfigured (too aggressive)	`msgRateExpired` non-zero across all subscriptions on the namespace, consumers appear healthy	Namespace TTL setting vs message processing SLA
TTL inherited from broker default	Unexpected expiry on topics where TTL was never explicitly configured	`ttlDurationDefaultInSeconds` in broker.conf
Backlog quota eviction (not TTL)	Messages disappearing but `msgRateExpired` is zero	Backlog quota policy on the namespace

Quick checks

# Check message expiry rate from broker Prometheus metrics
curl -s http://<broker-host>:8080/metrics | grep pulsar_subscription_msg_rate_expired

# Check expiry rate and cumulative count via Admin API
pulsar-admin topics stats persistent://tenant/namespace/topic
# Look for subscriptions.<name>.msgRateExpired and totalMsgExpired

# Check namespace TTL configuration
pulsar-admin namespaces get-message-ttl tenant/namespace

# Check broker-level TTL default
grep ttlDurationDefaultInSeconds conf/broker.conf

# Check expiry check interval
grep messageExpiryCheckIntervalInMinutes conf/broker.conf

# Check backlog quota policy (separate mechanism from TTL)
pulsar-admin namespaces get-backlog-quotas tenant/namespace

# Check consumer connections on affected subscription
pulsar-admin topics stats persistent://tenant/namespace/topic
# Look for subscriptions.<name>.consumers - empty array means no active consumers

# Correlate backlog with expiry rate
curl -s http://<broker-host>:8080/metrics | grep -E "pulsar_subscription_(back_log|msg_rate_expired)"

How to diagnose it

flowchart TD
    A["Non-zero msgRateExpired"] --> B{"TTL intended
on this topic?"}
    B -->|No| C{"Consumers
connected?"}
    B -->|Yes| D["Expected for
time-sensitive data"]
    C -->|No| E["Consumer outage
exceeds TTL window"]
    C -->|Yes| F{"Consumer keeping
up with rate_in?"}
    F -->|No| G["Consumer too slow
for TTL window"]
    F -->|Yes| H{"TTL explicitly set
on namespace?"}
    H -->|No| I["Inherited from broker
default, check broker.conf"]
    H -->|Yes| J["TTL value too low
for processing SLA"]

Confirm expiry is happening. Check pulsar_subscription_msg_rate_expired for the affected topic and subscription. A non-zero value means messages are being deleted due to TTL. Cross-reference with the Admin API: pulsar-admin topics stats persistent://tenant/namespace/topic and look for msgRateExpired in the subscription stats section.
Determine whether the TTL is intentional. Check the namespace policy: pulsar-admin namespaces get-message-ttl tenant/namespace. If TTL is set and the topic genuinely handles time-sensitive data, expiry may be expected. If the topic requires full delivery, any non-zero expiry rate is a problem.
Check whether TTL is inherited from a broker default. If namespace TTL shows as configured but no one on your team set it, check ttlDurationDefaultInSeconds in broker.conf. A non-zero broker-level default applies to all namespaces that do not explicitly override it. Note: setting --messageTTL 0 on a namespace may not override a non-zero broker default in some Pulsar versions. If you encounter this, set TTL to a very high value as a workaround.
Identify which subscriptions are affected. Because TTL is per-subscription, pull topic stats and compare msgRateExpired across all subscriptions. One slow subscription losing messages while others are unaffected points to a consumer-specific problem, not a TTL misconfiguration.
Check consumer health for affected subscriptions. In the topic stats output, look at the consumers array for each affected subscription. An empty array means the consumer is disconnected. A non-empty array with growing backlog means the consumer is connected but not processing fast enough.
Correlate expiry rate with consumer outage duration. If the consumer was down for longer than the TTL window, all messages published during the outage that exceeded the TTL threshold will be lost. The expiry rate will spike when the broker’s expiry check runs after the consumer reconnects.
Verify the actual TTL window. Remember the 1.5x threshold multiplier. If TTL is configured at 1 hour, messages may not actually expire until 1.5 hours. Factor this into your timeline analysis when correlating consumer outages with expiry spikes.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`pulsar_subscription_msg_rate_expired`	Direct measure of TTL-driven data loss	Any non-zero value on topics requiring full consumption
`pulsar_subscription_back_log`	Should grow if consumers fall behind; flat backlog with expiry suggests masking	Backlog flat or declining while `rate_in` is non-zero and expiry is non-zero
`pulsar_rate_in`	Producer throughput into the topic	Normal or high `rate_in` with non-zero expiry means consumers cannot keep up
`pulsar_rate_out`	Consumer dispatch rate from the topic	`rate_out` consistently below `rate_in` with non-zero expiry indicates consumer capacity problem
Consumer connection count	Whether consumers are connected to the subscription	Zero connections indicates consumer outage
`totalMsgExpired`	Cumulative count of expired messages per subscription	Growing counter means ongoing silent data loss

Fixes

Consumer outage exceeding TTL

The consumer was down longer than the TTL window. Messages published during the outage that crossed the TTL threshold are gone. There is no recovery for already-expired messages.

Short-term: restart the consumer immediately. Every additional minute of downtime with a non-zero TTL produces more data loss.

Long-term: evaluate whether the TTL window is appropriate for your consumer availability SLA. If consumers can be down for hours, the TTL must be longer than the maximum acceptable outage. Consider removing TTL entirely on topics where consumers may experience planned or unplanned downtime.

Consumer too slow for TTL window

The consumer is connected and processing, but throughput cannot keep up with the publish rate within the TTL window.

Options, in order of preference:

Scale consumers. Add more consumer instances to the subscription (shared or failover mode). More parallelism directly increases dispatch and acknowledgment throughput.
Increase the TTL. Give consumers more time to process. This trades disk usage for data retention as backlog grows.
Optimize consumer processing. Profile the consumer application. Downstream dependency latency (database writes, API calls) is the most common bottleneck.
Reduce publish rate. If consumers genuinely cannot keep up, throttle producers or redistribute load across topics.

TTL misconfiguration

TTL is set too low for the workload’s processing requirements, or TTL is inherited from a namespace or broker-level default that was not intended for this specific topic.

# Set TTL (in seconds)
pulsar-admin namespaces set-message-ttl tenant/namespace --messageTTL 3600

# Remove TTL (set to 0)
pulsar-admin namespaces set-message-ttl tenant/namespace --messageTTL 0

# Verify the setting took effect
pulsar-admin namespaces get-message-ttl tenant/namespace

To check the broker-level default:

grep ttlDurationDefaultInSeconds conf/broker.conf

If ttlDurationDefaultInSeconds is non-zero, it applies to all namespaces without an explicit override. To change it, update broker.conf and restart brokers.

Cannot disable TTL at namespace level

In some Pulsar versions, setting `--messageTTL 0` on a namespace does not override a non-zero `ttlDurationDefaultInSeconds` at the broker level. The namespace continues to inherit the cluster default. As a workaround, set TTL to a very large value (for example, `--messageTTL 999999999`) instead of zero.

Backlog quota eviction masquerading as TTL

If msgRateExpired is zero but messages are still disappearing, the backlog quota policy may be the cause. The consumer_backlog_eviction policy discards the oldest unacked messages when backlog exceeds the configured size or time limit. This is a separate mechanism from TTL but produces the same symptom: silent data loss.

# Check backlog quota policy
pulsar-admin namespaces get-backlog-quotas tenant/namespace

If the policy is consumer_backlog_eviction and messages are being lost, either increase the quota or change the policy to producer_request_hold or producer_exception to make the problem visible at the producer side rather than silently dropping data.

Prevention

Alert on non-zero msgRateExpired for critical topics. This should be a high-severity alert on any topic where data loss is unacceptable. The threshold is zero. Any deviation requires immediate investigation.
Set TTL to zero explicitly on topics requiring full delivery. Do not rely on the absence of configuration, especially if the broker has a non-zero ttlDurationDefaultInSeconds.
Monitor backlog alongside expiry rate. A flat backlog with non-zero expiry is a red flag. The backlog is being trimmed by expiry, not by consumption.
Document TTL intent per namespace. Track which namespaces have TTL, why it was set, and what the expected consumer processing window is. This prevents confusion when new topics inherit the namespace policy.
Size TTL against consumer availability SLAs. If consumers can be down for N hours during deployments, incidents, or dependency failures, TTL must exceed N hours or data will be lost.
Audit broker-level defaults. A non-zero ttlDurationDefaultInSeconds silently applies TTL to every namespace that does not override it. Verify this value during cluster setup and after configuration changes.
Test TTL behavior before relying on it. Publish messages and verify the actual expiry time against the configured value. Factor in the 1.5x threshold multiplier and the 5-minute check interval when validating.

How Netdata helps

Per-second granularity on pulsar_subscription_msg_rate_expired. TTL expiry happens in bursts when the 5-minute check interval fires. Per-second collection captures these bursts that coarser scrap intervals miss entirely.
Correlation of expiry rate with backlog and dispatch rate. Seeing msgRateExpired, pulsar_subscription_back_log, pulsar_rate_in, and pulsar_rate_out on the same dashboard makes the masking effect immediately visible. Flat backlog with non-zero expiry is obvious when all four signals are side by side.
Consumer connection tracking. A drop in consumer connections preceding an expiry spike confirms a consumer outage as the root cause.
Anomaly detection on expiry rate. ML-based anomaly detection flags unexpected changes in expiry rate even when no explicit threshold alert is configured, catching slow consumer drift before it becomes a data loss incident.
Namespace-level filtering. Per-namespace and per-subscription labels allow isolating which subscriptions are losing messages, critical for diagnosing per-subscription TTL behavior on shared topics.