Apache Pulsar geo-replication backlog: replication lag and your real RPO window

Geo-replication in Pulsar is asynchronous by design. Messages are persisted locally and acknowledged to producers before they reach the remote cluster. Every message sitting in the replication backlog is a message that would be lost if you failed over right now. The replication backlog is your recovery point objective (RPO) window, measured in real time.

The common operational mistake is checking whether replication is “working” (connected, producing to the remote cluster) without checking how far behind it is. A replicator can be fully connected and actively shipping messages while being hours behind. The dashboard shows green. The failover plan assumes near-zero data loss. The reality is data loss measured in hours.

What it is and why it matters

Pulsar geo-replication uses internal replication producers per topic per remote cluster. Each replicated topic maintains a replicator cursor that tracks the local position up to which messages have been successfully sent to the remote cluster. The gap between the topic’s current write position and the replicator cursor position is the replication backlog.

Two metrics capture this gap:

• pulsar_replication_backlog: message count pending replication to each remote cluster. Labels include cluster, namespace, topic, and remoteCluster.
• pulsar_replication_delay_in_seconds: time elapsed since the oldest unreplicated message was published. This is the direct measure of replication lag in time units.

Message count tells you volume, not urgency. A backlog of 50,000 messages at 10,000 messages/second is 5 seconds of RPO exposure. The same 50,000 messages at 1 message/second is nearly 14 hours. Always convert backlog to time using pulsar_replication_delay_in_seconds, or estimate it from the backlog size and replication rate.

The lag is not a bug or a degradation signal by itself. It is an inherent property of asynchronous replication. Your job as an operator is to know whether the current lag is within your declared RPO, not whether it is zero.

How it works

The replication data path

When a producer publishes a message to a replicated topic, the local broker:

1. Appends the message to the local managed ledger.
2. Writes to the local BookKeeper ensemble and waits for ack quorum (Qa) acknowledgments.
3. Acknowledges the producer. The message is now durably persisted locally.
4. The replicator reads the message from the managed ledger and sends it to the remote cluster via an internal producer over the WAN.
5. The remote broker receives the message, persists it to its own BookKeeper ensemble, and acknowledges.
6. The replicator cursor advances only after the remote broker acknowledges.

Steps 4 through 6 happen asynchronously. The producer does not wait for them. This is the window of potential data loss during failover.

flowchart LR
    P[Producer] -->|1 publish| LB[Local Broker]
    LB -->|2 write + Qa ack| LDB[Local Bookies]
    LB -->|3 producer ack| P
    LB -->|4 async read| REP[Replicator]
    REP -->|5 WAN produce| RB[Remote Broker]
    RB -->|6 write + Qa ack| RDB[Remote Bookies]
    RB -->|ack| REP
    REP -->|cursor advance| LB

The replicator cursor advances only on step 6. Until then, every message between the cursor and the write head is in the replication backlog and is at risk during failover.

What limits replication throughput

Cross-region WAN bandwidth and latency are the fundamental constraints. TCP windowing means throughput is bounded by window size divided by round-trip time. High inter-region latency directly caps the maximum replication rate regardless of available bandwidth. If local publish rate exceeds this ceiling, backlog grows continuously.

When pulsar_replication_connected_count > 0 and backlog is growing continuously, the replicator is hitting this cross-region throughput limit. The connection is healthy, the producer is active, but the pipe is too narrow. This is a capacity problem, not a failure.

Where it shows up in production

Normal: transient backlog during spikes

Non-zero replication backlog is expected during traffic bursts. If the local publish rate temporarily exceeds replication throughput, backlog accumulates. The signal that this is benign: the backlog recovers to near-zero after the spike passes, and pulsar_replication_delay_in_seconds returns to baseline.

Problem: continuous growth with connected replicator

When backlog grows continuously for more than 30 minutes and pulsar_replication_connected_count > 0, the replicator is connected but cannot keep up. This is the most commonly misdiagnosed pattern. Teams see “connected” and assume replication is healthy, missing that the RPO window is growing unbounded.

Check the replication rate metrics. If pulsar_replication_rate_out and pulsar_replication_throughput_out are flat while backlog grows, the replicator has hit the WAN throughput ceiling. Remediation options are limited: increase cross-region bandwidth, reduce publish rate, or add more replication producers by partitioning.

Problem: silently stopped replication

A more insidious pattern: replication for some topics randomly stops during normal operation, often triggered by bursts of high publish rate or BookKeeper iowait. The replicator shows connected: true with msgRateOut: 0.0 and a growing backlog. No error messages appear in logs.

If you see this signature (connected replicator, zero output rate, growing backlog on specific topics), check per-topic replication stats from pulsar-admin topics stats, since aggregate metrics can mask individual topic stalls.

The reported recovery is disabling and re-enabling replication for the affected namespace. This is disruptive: it interrupts replication for all topics in the namespace. Apply it during a maintenance window or only when the RPO exposure from continued stall exceeds the disruption cost.

Problem: backlog quota stalling replication

Geo-replication producers on the remote cluster are subject to the same backlog quotas as regular producers. If the remote cluster has a producer_request_hold or producer_exception backlog quota policy and the remote-side subscription falls behind, the replication producer can be blocked. The replicator cursor may enter a “NoLedger” state with active: false. Backlog grows on the local side even though the network and the remote cluster are otherwise healthy.

Check the remote cluster’s backlog quota configuration with pulsar-admin namespaces get-backlog-quotas <namespace> if you suspect this.

Partial replication hides aggregate problems

Aggregate replication metrics can hide topic-level failures. If 99 of 100 topics are replicating fine and one is stuck, the aggregate backlog may look stable. Always check per-topic replication stats for critical topics, not just namespace or cluster aggregates.

Common misuses and risks

Treating “replication working” as “DR-ready”

Teams verify that replication is connected and producing, then assume they can fail over with near-zero data loss. Without checking pulsar_replication_delay_in_seconds against the declared RPO, the actual data loss on failover could be hours.

Your failover runbook must include a pre-failover gate: check pulsar_replication_delay_in_seconds for every remote cluster. If the delay exceeds the declared RPO, you either wait for replication to catch up or accept the data loss explicitly.

Modifying cluster configuration triggers cascading deletions

Modifying the clusters list at the namespace or topic policy level can trigger automatic topic deletions on excluded clusters. This is a data-loss risk. Always verify the impact of cluster configuration changes before applying them, and maintain independent backups.

Replicated subscriptions have limitations

Replicated subscriptions do not replicate individual acknowledgments. Only the mark-delete position (baseline cursor) is replicated via periodic snapshots (default every 1000ms). Messages acknowledged out of order may be redelivered after failover. For deployments with more than 2 clusters, the snapshot timeout (default 30 seconds) may need to be increased to 60 seconds to avoid replication stalls.

Signals to watch in production

How Netdata helps

• Per-second collection of pulsar_replication_backlog and pulsar_replication_delay_in_seconds with the remoteCluster label lets you see the RPO window for each remote cluster individually, not just in aggregate.
• Correlating pulsar_replication_delay_in_seconds against local pulsar_rate_in in the same dashboard reveals whether growing lag is caused by a publish spike (transient) or a structural throughput mismatch (continuous growth).
• ML anomaly detection on pulsar_replication_backlog distinguishes normal burst-driven accumulation from unusual sustained growth that may indicate a stuck replicator or WAN degradation.
• Alerting on pulsar_replication_delay_in_seconds with per-remoteCluster granularity lets you set RPO-based thresholds per DR target rather than relying on generic message-count thresholds.
• Correlating replication metrics with cross-region network bandwidth and remote broker health in a single view shortens root-cause isolation.

Related guides

• Apache Pulsar active connections climbing: connection leaks and file descriptor exhaustion
• Apache Pulsar bookie add-entry queue not draining: writes arriving faster than the disk can commit
• Apache Pulsar AutoRecovery stalled: under-replicated ledgers that never heal
• Apache Pulsar backlog age vs size: the latency depth alone cannot show
• Apache Pulsar backlog quota exceeded: producers held or rejected when consumers stall
• Apache Pulsar bookie disk filling: runway to read-only and how to reclaim space
• Apache Pulsar bookie failure cascade: recovery I/O that topples surviving bookies
• Apache Pulsar bookie read latency high: catch-up reads competing with the write path
• Apache Pulsar bookie read-only: disk full and bookie_SERVER_STATUS at zero
• Apache Pulsar broker down: telling a dead broker from a fenced one
• Apache Pulsar broker GC death spiral: heap pressure, stop-the-world pauses, and lost topic ownership
• Apache Pulsar broker lookup failures: new clients cannot find their topic