Apache Pulsar geo-replication disconnected: a remote cluster that stopped receiving

The symptom is unmistakable: pulsar_replication_disconnected_count is non-zero for all replicators to a specific remote cluster, pulsar_replication_connected_count has dropped to zero, and pulsar_replication_backlog is growing at the rate of local publish throughput. The remote cluster has stopped receiving messages. Local producers and consumers continue working normally because Pulsar geo-replication is asynchronous. Messages persist locally first, then replicate. The damage is silent and cumulative.

Every unreplicated message widens the recovery point objective (RPO) gap. If you fail over to the remote cluster right now, those messages are gone. The backlog is your exposure, measured in entries and in seconds via pulsar_replication_delay_in_seconds.

This article covers diagnosis and resolution for a full replication disconnection to a single remote cluster, where the backlog is growing and the RPO clock is ticking.

What this means

Pulsar geo-replication uses internal replication producers. Each replicated topic maintains one replicator per remote cluster, backed by a replicator cursor that tracks the last successfully replicated position in the local managed ledger. When the replication producer cannot establish or maintain a connection to the remote broker, it disconnects and the cursor stops advancing.

The replication backlog is the durable cursor backlog of the replicator. There is no maxReplicationBacklog configuration cap. If the remote cluster stays unreachable, the backlog grows without limit, consuming local bookie storage even when local consumers have already processed and acknowledged all messages. The only mechanisms that eventually bound this growth are backlog quota policies and namespace TTL, but TTL expiry is not immediate. It depends on the broker’s periodic expiry check, and expired messages represent permanent data loss from the replication perspective.

The severity depends entirely on your RPO requirements. For strict-RPO deployments, the page condition should fire when pulsar_replication_delay_in_seconds exceeds the declared RPO, sustained for more than 10 minutes, with non-zero local publish demand, and not during initial bootstrap or warmup. A generic playbook cannot define this threshold for you. It requires deployment-specific RPO knowledge and business-criticality assessment.

Common causes

Quick checks

# Check replication connected/disconnected count per remote cluster
curl -s http://<broker-host>:8080/metrics | grep -E "pulsar_replication_(connected|disconnected)_count"

# Check replication backlog per remote cluster
curl -s http://<broker-host>:8080/metrics | grep pulsar_replication_backlog

# Check replication delay in seconds (real-time RPO)
curl -s http://<broker-host>:8080/metrics | grep pulsar_replication_delay_in_seconds

# Check replication rate in/out (should be zero if fully disconnected)
curl -s http://<broker-host>:8080/metrics | grep -E "pulsar_replication_rate_(in|out)"

# Check authentication failures on the local broker
curl -s http://<broker-host>:8080/metrics | grep pulsar_authentication_failures_total

# Check remote broker health
curl -sf http://<remote-broker>:8080/admin/v2/brokers/health

# Check remote bookie server status (1 = writable, 0 = read-only, -1 = unregistered)
curl -s http://<remote-bookie>:8000/metrics | grep bookie_SERVER_STATUS

# Check remote bookie disk usage percentage
curl -s http://<remote-bookie>:8000/metrics | grep bookie_ledger_dir

# Inspect replication stats for a specific topic via Admin API
pulsar-admin topics stats persistent://tenant/namespace/topic
# Look for replication.<remoteCluster>.replicationBacklog and .connected

How to diagnose it

flowchart TD
    A["All replicators to remote cluster
disconnected, backlog growing"] --> B{"Remote broker
health endpoint
reachable?"}
    B -- No --> C["Remote cluster down or
network partition"]
    B -- Yes --> D{"Auth failures rising
on remote broker?"}
    D -- Yes --> E["Replication producer
auth mismatch"]
    D -- No --> F{"Remote bookies
read-only?"}
    F -- Yes --> G["Remote storage full"]
    F -- No --> H{"Namespace assigned
to remote cluster?"}
    H -- No --> I["Replication policy
misconfiguration"]
    H -- Yes --> J["Check PIP-370
topic creation issue"]

1. Confirm the disconnection is real and cluster-wide. Pull pulsar_replication_disconnected_count and pulsar_replication_connected_count, filtered by the remote cluster label. If connected_count is zero across all brokers and topics for that remote cluster while disconnected_count is non-zero, replication is fully broken. A partial disconnect, where some topics remain connected, points to a per-topic issue such as a missing remote topic rather than a cluster-wide outage.

2. Verify remote cluster reachability. From a local broker host, attempt to reach the remote broker’s admin endpoint and Pulsar service port. If the admin endpoint times out, the remote cluster is either down or there is a network partition. Check whether other remote clusters (if configured) remain connected. If only one remote cluster is affected, the problem is specific to that link. If all remote clusters are disconnected, the local broker may have a network issue.

3. Check for authentication failures. The replication producer uses credentials configured on the cluster definition via --auth-plugin and --auth-parameters on the pulsar-admin clusters create or update command. These override the broker-level brokerClientAuthenticationPlugin and brokerClientAuthenticationParameters settings. If these credentials are expired, rotated, or mismatched, the replicator connects to the remote broker but immediately fails authentication. Check pulsar_authentication_failures_total on the remote broker and look for authentication errors in the remote broker logs.

4. Check remote cluster storage. If remote bookies have transitioned to read-only (bookie_SERVER_STATUS at 0), the replication producer may connect but fail to write because the remote cluster cannot persist new messages. Check disk usage on remote bookies with bookie_ledger_dir_*_usage. This cascades predictably: remote storage fills, bookies go read-only, replication producers cannot write, backlog accumulates locally. See bookie read-only and bookie disk filling for detailed procedures.

5. Verify replication configuration. Confirm the namespace is assigned to the remote cluster in its replication policy. Use pulsar-admin namespaces get-clusters <tenant>/<namespace> to list the configured replication clusters. If the remote cluster was accidentally removed from the policy, replication silently stops. Also verify the cluster definition itself with pulsar-admin clusters get <remote-cluster-name> to check the broker URL, TLS settings, and authentication parameters.

6. Check version-specific topic creation issues. On Pulsar 4.0.x, verify the createTopicToRemoteClusterForReplication setting. PIP-370, introduced in Pulsar 4.0.0, controls whether the broker auto-creates topics on remote clusters during geo-replication. The default is true. If this is set to false, or if a regression prevents topic creation, replication producers cannot publish to topics that do not exist on the remote side.

Metrics and signals to monitor

Fixes

Remote cluster brokers down

If the remote cluster is down or degraded, replication cannot resume until the remote brokers are healthy. Focus on restoring the remote cluster independently. Check remote broker process health, ZooKeeper connectivity, and bookie availability. See broker down for distinguishing a dead broker from a fenced one.

Once the remote cluster is back, replicators reconnect automatically and begin draining the backlog. Verify recovery by watching pulsar_replication_rate_out return to non-zero and pulsar_replication_backlog decrease toward zero.

If the remote cluster will be down for an extended period, assess local bookie storage carefully. The replication backlog consumes local disk with no cap. If storage pressure is severe, consider whether TTL or backlog quota policies can help. Understand the tradeoff: TTL expiry on unreplicated messages means permanent data loss from the replication perspective. See backlog quota exceeded for quota policy behavior.

Replication producer authentication failure

Review the current cluster configuration and update credentials:

# Review current cluster config
pulsar-admin clusters get <remote-cluster-name>

# Update with correct auth parameters
pulsar-admin clusters update <remote-cluster-name> \
  --auth-plugin <plugin> \
  --auth-parameters <parameters>

After updating, replicators reconnect on the next retry cycle. Verify by checking that pulsar_replication_connected_count recovers and that pulsar_authentication_failures_total stops rising on the remote broker.

If authentication uses TLS certificates, verify that the certificates used for inter-cluster communication have not expired. An expired certificate causes immediate TLS handshake failures. Track certificate expiration proactively.

Remote cluster storage full

Clear space on remote bookies or add capacity. This may involve running BookKeeper garbage collection and compaction to reclaim space from deleted ledgers, expanding disk capacity, adding new bookies, or reviewing retention policies on the remote cluster. See bookie disk filling for runway estimation and reclamation procedures.

Once remote bookies return to writable status (bookie_SERVER_STATUS returns to 1), replication resumes automatically. Confirm by watching pulsar_replication_rate_out.

Cross-region network partition

This is an infrastructure issue requiring coordination with the network team. While the partition persists:

• The local cluster continues to accept and persist messages normally.
• The replication backlog grows linearly with publish rate, consuming local storage.
• Monitor local bookie disk usage to ensure the accumulated backlog does not trigger a local storage crisis.

If the partition is prolonged and local storage is at risk, consider whether TTL or backlog quota policies are appropriate. This is a business decision: expiring unreplicated messages accepts data loss in exchange for local cluster stability.

Replication configuration error

If the namespace was accidentally removed from the remote cluster’s replication policy:

# Check current replication clusters for the namespace
pulsar-admin namespaces get-clusters <tenant>/<namespace>

# Re-add the remote cluster
pulsar-admin namespaces set-clusters <tenant>/<namespace> \
  --clusters <local-cluster>,<remote-cluster>

Warning: modifying the clusters list at the namespace or topic policy level can trigger cascading topic deletions on excluded clusters. Always back up topic metadata before changing replication cluster assignments.

Remote topic creation failure (PIP-370)

On Pulsar 4.0.x with createTopicToRemoteClusterForReplication set to false, explicitly create topics on the remote cluster:

# Create the topic on the remote cluster
pulsar-admin --admin-url <remote-admin-url> topics create \
  persistent://tenant/namespace/topic

If you are hitting the reported 4.0.1 regression where topic creation fails even with default settings, pre-create all replicated topics on all clusters as a workaround. Also consider upgrading to at least 4.0.9 or 4.1.3, which fixed a replicated subscription snapshot cache eviction bug that could cause replication stalls.

Prevention

Monitor replication delay against RPO, not just disconnection count. Alert on pulsar_replication_delay_in_seconds exceeding your declared RPO. A connected but slow replicator can also miss RPO targets without triggering a disconnection alert.

Gate alerts on real traffic and exclude bootstrap. Only page when local publish demand is non-zero and the condition persists beyond warmup. Idle clusters and freshly initialized replication produce false positives.

Automate credential rotation. Replication producer authentication failures often follow certificate or token expiry. Track TLS certificate expiration for inter-cluster communication and rotate before expiry. See the playbook’s TLS certificate expiration signal for monitoring guidance.

Pre-create topics on all clusters. If you run Pulsar 4.0.x, explicitly create replicated topics on all clusters rather than relying on auto-creation. This avoids PIP-370-related failures and makes topic lifecycle explicit.

Watch local storage during remote outages. The replication backlog grows without limit during a remote cluster outage. Model the growth rate against available local bookie capacity and plan for the worst-case outage duration. A prolonged remote outage can topple the local cluster through storage exhaustion even when local consumers are healthy.

How Netdata helps

• Per-second replication metrics: pulsar_replication_disconnected_count, pulsar_replication_connected_count, and pulsar_replication_backlog are collected every second, so you see the exact moment replication breaks rather than discovering it minutes later through a slower scrape interval.
• RPO as a live signal: pulsar_replication_delay_in_seconds is tracked alongside the backlog, giving you a direct read on real-time RPO without manual conversion from message counts to time.
• Flow confirmation: pulsar_replication_rate_in and pulsar_replication_rate_out let you verify in real time that replication has resumed after a fix, not just that the replicator reconnected.
• Correlation with auth failures: pulsar_authentication_failures_total is collected alongside replication metrics, so a credential expiry causing a replication disconnect appears as a correlated spike rather than a mystery.
• Remote bookie health in context: Bookie server status, disk usage, and journal latency from the remote cluster appear alongside local metrics, letting you confirm remote storage health without switching tools.

Related guides

• Apache Pulsar broker down: telling a dead broker from a fenced one
• Apache Pulsar bookie read-only: disk full and bookie_SERVER_STATUS at zero
• Apache Pulsar bookie disk filling: runway to read-only and how to reclaim space
• Apache Pulsar backlog quota exceeded: producers held or rejected when consumers stall
• Apache Pulsar bookie failure cascade: recovery I/O that topples surviving bookies
• Apache Pulsar backlog age vs size: the latency depth alone cannot show
• Apache Pulsar broker GC death spiral: heap pressure, stop-the-world pauses, and lost topic ownership
• Apache Pulsar active connections climbing: connection leaks and file descriptor exhaustion
• Apache Pulsar AutoRecovery stalled: under-replicated ledgers that never heal
• Apache Pulsar broker lookup failures: new clients cannot find their topic
• Apache Pulsar bookie add-entry queue not draining: writes arriving faster than the disk can commit
• Apache Pulsar bookie read latency high: catch-up reads competing with the write path