Apache Pulsar backlog age vs size: the latency depth alone cannot show

Backlog size tells you how much data is waiting. Backlog age tells you how long it has been waiting. Monitoring only one leaves you blind to entire classes of consumer health failures.

A stable backlog of 1,000 entries is unremarkable if those entries are 5 seconds old. The same 1,000 entries from 5 hours ago means a cursor is stuck, a consumer is down, or an application has silently stopped acknowledging. Size alone cannot distinguish these scenarios. Age can.

What it is and why it matters

Backlog in Pulsar is the gap between the latest published message position and the furthest-behind subscription cursor’s mark-delete position. Every persistent subscription has its own backlog. A topic with five subscriptions has five independent backlogs.

Two metrics measure this gap from different angles:

• Backlog size (pulsar_subscription_back_log, entries): How many entries sit between the current write position and the cursor’s mark-delete position. This is a volume metric.
• Backlog age (pulsar_storage_backlog_age_seconds, seconds): How old the oldest unacknowledged message is. This is a latency metric.

Size tells you magnitude. Age tells you severity.

A backlog of 50,000 entries that is 2 seconds old is healthy throughput lag. The same 50,000 entries at 3 hours old is a stuck cursor. A backlog of 200 entries at 4 hours old is small in volume but ancient in age, typically a cursor leak or an abandoned subscription holding a position near the tip.

Patterns age reveals that size hides:

• Cursor leaks: Abandoned subscriptions hold cursors that never advance. Backlog size may be small if the subscription was created near the tip, but age will be very old for whatever backlog does exist.
• Abandoned subscriptions: A subscription with zero consumers but non-zero backlog. Size might be stable and small. Age grows monotonically.
• Slow-consumer SLA breaches: A consumer processing at 95% of publish rate shows stable backlog size but continuously growing age. Size looks fine. Age tells the truth.

How it works

Pulsar’s managed ledger model creates one cursor per subscription. The cursor tracks a mark-delete position that advances as messages are acknowledged. The backlog is the set of entries between the mark-delete position and the current write position.

flowchart LR
    A[Producer writes entry] --> B[Backlog: not yet acknowledged]
    B --> C[Dispatched to consumer]
    C --> D{Consumer acks?}
    D -->|Yes| E[Cursor mark-delete advances]
    D -->|No / timeout| F[Redelivery loop]
    B -.->|Size metric| G["pulsar_subscription_back_log
(entries, not messages)"]
    B -.->|Age metric| H["pulsar_storage_backlog_age_seconds
(seconds, oldest unacked)"]
    C -.->|In-flight metric| I["pulsar_subscription_unacked_messages
(dispatched, not acked)"]
    F -.->|Retry metric| J["pulsar_subscription_msg_rate_redeliver"]

Backlog size: estimated, not exact

pulsar_subscription_back_log counts entries, not individual messages. If a producer batches 1,000 messages into a single entry, that batch counts as 1 entry in the backlog. A backlog of 1,000 entries might represent 1,000 messages or 1,000,000 messages depending on batch size. For message-level granularity, the admin API’s analyzeSubscriptionBacklog endpoint (PIP-187) inspects individual messages rather than entries.

For byte-level backlog at the subscription level, there is no dedicated pulsar_subscription_back_log_size Prometheus metric. The exposeSubscriptionBacklogSizeInPrometheus setting (default false) controls whether the broker computes each subscription’s backlog size in bytes when building stats; that size is surfaced per subscription through the admin API (pulsar-admin topics stats -sbs / --get-subscription-backlog-size), while Prometheus continues to expose only the entry counts (pulsar_subscription_back_log) and the topic-level byte metric pulsar_storage_backlog_size. Without it, you only have entry counts.

At the topic level, pulsar_storage_backlog_size reports backlog size in bytes, but it aggregates across all subscriptions on the topic. A single slow subscription can hide behind fast ones in the aggregate.

Backlog size is also estimated, not exact. It is calculated by summarizing ledger sizes from the current active ledger up to the ledger containing the oldest unacknowledged message. For partitioned topics, updates are asynchronous.

Backlog age: cached and update-gated

pulsar_storage_backlog_age_seconds was introduced in PIP-323, first available in Pulsar 3.2.0. It reports the age of the oldest unacknowledged message at the topic level. Two admin API fields accompany it in topic stats: oldestBacklogMessageAgeSeconds and oldestBacklogMessageSubscriptionName, which identify the age and the responsible subscription.

The metric is cached. It updates only every backlogQuotaCheckIntervalInSeconds (default 60 seconds). Between checks, the value reflects the last check, not the current state. A 60-second staleness window is built in.

A related subscription-level field, earliestMsgPublishTimeInBacklogs (in milliseconds), persists across broker restarts. The topic-level age metric does not persist and resets on broker unload or restart.

For per-subscription age detail, inspect earliestMsgPublishTimeInBacklogs in each subscription’s stats via the admin API. The Prometheus metric is topic-level only and cannot provide per-subscription granularity.

Version bugs that break the age metric

Two bugs in the PIP-323 implementation affected pulsar_storage_backlog_age_seconds:

1. The metric only updated when a message_age backlog quota policy was set on the topic. Topics without that policy showed stale or zero age regardless of actual backlog age. Fixed in Pulsar 3.0.8, 3.3.3, and 4.0.1 (PR #23619). On affected versions, the age metric is unreliable unless you configure time-based backlog quotas on every topic you want to monitor.

2. oldestBacklogMessageAgeSeconds kept increasing on open ledgers even when the subscription was caught up. When the cursor’s mark-delete position pointed to an open ledger, the age would not reset. Fixed in Pulsar 3.0.15, 3.3.10, 4.0.8, 4.1.2, and 4.2.0 (PR #24915).

If the age metric seems stuck or nonsensical on your deployment, check whether these bugs apply to your version.

The precise time check tradeoff

By default, preciseTimeBasedBacklogQuotaCheck=false. When false, the broker uses the ledger’s close time (in-memory) to estimate message age. When true, the broker reads the oldest message from BookKeeper to obtain its exact publish timestamp. The precise check is accurate but I/O expensive because it forces a bookie read on every check interval. For most deployments, the in-memory estimate is sufficient for alerting. Reserve the precise check for forensic investigation.

Where it shows up in production

Stable size, growing age: the silent SLA breach

A consumer processing at 95% of publish rate maintains a stable backlog size. Messages accumulate at 5% but the backlog also drains. Size plateaus. Without an age metric, the dashboard looks healthy.

Age tells a different story. The oldest message gets progressively older. If your SLA requires consumption within 60 seconds, age climbing past that threshold is the breach, regardless of what size says.

Zero backlog, consumers not processing

A near-zero pulsar_subscription_back_log does not guarantee consumers are making forward progress. Per the official metric definition it counts entries in the unacknowledged state, which includes entries already dispatched and held by a consumer; a consumer holding many unacked messages therefore does not register as backlog. Pair backlog with metrics that capture different failure modes.

pulsar_subscription_unacked_messages tracks messages dispatched to consumers but not yet acknowledged. When this count hits maxUnackedMessagesPerSubscription (default 200,000), the broker stops dispatching. No error surfaces. The consumer stays connected, the backlog stays flat, and no forward progress is made. The blockedSubscriptionOnUnackedMsgs flag in topic stats turns true when this limit is hit, which is the definitive dispatch-freeze indicator.

Pair backlog metrics with:

• pulsar_subscription_unacked_messages to detect dispatched-but-unacked pressure
• pulsar_subscription_msg_rate_redeliver to detect consumers that receive but cannot process (poison messages, downstream failures)

Small backlog, very old age: cursor leak

An abandoned subscription created near the topic tip has a small backlog but very old backlog age. No consumer is connected, so the cursor never advances. Every check interval, age increases. Size stays low because the subscription started late.

This pattern is invisible with size-only monitoring. Over weeks, multiple abandoned subscriptions accumulate, each holding back garbage collection. The oldestBacklogMessageSubscriptionName field in topic stats identifies which subscription is the oldest, making it easier to find the culprit.

Stable size, reasonable age, high redelivery: poison message

A consumer stuck on a single message that always fails processing shows a small backlog (possibly 1 entry), a reasonable age, but high redelivery rate. The message is dispatched, fails, gets nack’d or times out, and is redelivered. No forward progress, but the backlog metrics look fine.

This is why pulsar_subscription_msg_rate_redeliver must be paired with backlog metrics. Redelivery rate above approximately 10% of dispatch rate warrants investigation. At 100% redelivery, zero forward progress is being made.

What backlog age alone cannot show

Age tells you the oldest unacked message is old. It does not tell you why. The same age reading could mean:

• A single abandoned subscription on a topic with otherwise healthy consumers
• A poison message blocking one consumer in a shared subscription
• A consumer that crashed hours ago with no auto-restart
• A downstream dependency (database, API) timing out on every message

Age also does not distinguish between many slightly-old messages and one very-old message. A backlog of 10,000 entries at 5 minutes average age might contain one entry that is 4 hours old, buried behind 9,999 entries that are seconds old. The topic-level metric reports the oldest, which is useful for detecting the worst case, but it does not give you the distribution.

For per-subscription detail, the admin API is required. pulsar_storage_backlog_age_seconds is topic-level only. It is not exposed when exposeTopicLevelMetricsInPrometheus=false because it cannot be meaningfully aggregated to namespace level.

The full signal set

No single metric covers all failure modes. Size plus age plus unacked plus redelivery gives you the full picture.

Tradeoffs and when to use each

Backlog size alone is sufficient for tailing workloads where consumers must keep up with the tip. If the backlog is zero or near-zero and stable, the consumer is keeping up. Growth in size is the primary detection signal.

Backlog age is essential when you have SLA requirements on message processing latency, subscriptions that might be abandoned (dynamic subscription names, ephemeral microservice deployments), slow consumers that process most messages but occasionally stall, or a need to distinguish “caught up with a small lag” from “stuck at the tip.”

Both are insufficient for determining whether consumers are actually processing successfully. Both measure the broker’s view of cursor positions, not consumer application health. pulsar_subscription_unacked_messages and pulsar_subscription_msg_rate_redeliver are required for that layer of visibility.

Retention and backlog quota interaction. Retention settings must exceed backlog quota settings. If retention is configured too aggressively relative to backlog quota, operators hit “Please increase retention quota and retry” errors. Additionally, consumer_backlog_eviction quota policy silently drops the oldest unacknowledged messages when backlog exceeds the quota, which is data loss presented as capacity management. If this policy is active, backlog size will appear stable while messages are being silently deleted. Check pulsar_subscription_msg_rate_expired to detect this.

How Netdata helps

• Per-second collection of pulsar_subscription_back_log captures backlog growth rates at finer granularity than the 60-second cached age metric.
• Correlating pulsar_subscription_back_log with pulsar_subscription_unacked_messages on a single dashboard surfaces the “zero backlog but not processing” pattern.
• Anomaly detection on backlog age flags monotonic growth on stable-size backlogs, indicating cursor leaks or abandoned subscriptions.
• Pairing pulsar_subscription_msg_rate_redeliver with backlog metrics provides the “received but not processed” signal that neither size nor age surfaces.
• Per-broker collection across the full cluster provides per-subscription backlog trends without manual aggregation across broker boundaries.

Related guides

• Apache Pulsar active connections climbing: connection leaks and file descriptor exhaustion
• Apache Pulsar bookie add-entry queue not draining: writes arriving faster than the disk can commit
• Apache Pulsar AutoRecovery stalled: under-replicated ledgers that never heal
• Apache Pulsar backlog quota exceeded: producers held or rejected when consumers stall
• Apache Pulsar bookie disk filling: runway to read-only and how to reclaim space
• Apache Pulsar bookie failure cascade: recovery I/O that topples surviving bookies
• Apache Pulsar bookie read latency high: catch-up reads competing with the write path
• Apache Pulsar bookie read-only: disk full and bookie_SERVER_STATUS at zero
• Apache Pulsar broker down: telling a dead broker from a fenced one
• Apache Pulsar broker GC death spiral: heap pressure, stop-the-world pauses, and lost topic ownership
• Apache Pulsar broker lookup failures: new clients cannot find their topic
• Apache Pulsar entry log GC falling behind: reclaimed space that never comes back