$ guides / consul / consul-catalog-staleness-anti-entropy ▌

Operations Guides

Consul anti-entropy not syncing: local agent state and the catalog drifting apart

A service is registered on the local agent. The health check is passing locally. But dig myservice.service.consul returns nothing, the load balancer has no targets, and the catalog API shows no instances. The servers have a leader, gossip is healthy, Raft metrics look normal. The problem is between the agent and the catalog.

Consul’s catalog is authoritative on the servers. Each agent maintains its own local state (registered services, health checks, node metadata) and periodically reconciles that state with the server catalog through a background process called anti-entropy sync. The agent treats its local view as authoritative and pushes changes to the catalog. When this sync fails, the catalog retains the last-known state it received from that agent. A service registered locally never appears cluster-wide. A service deregistered locally continues to show up in DNS and API queries.

The failure is silent by design. Anti-entropy is best-effort: if a sync fails, the agent logs the error and retries on the next cycle. There is no page, no user-visible error from Consul itself. The staleness only becomes apparent when a consumer gets the wrong answer from service discovery.

What this means

Anti-entropy sync runs on every Consul agent as a background goroutine. The agent collects its local state (services, checks, node metadata) and makes an RPC to a server to reconcile differences. The server updates the catalog accordingly. On a healthy cluster, this cycle completes within seconds and repeats at the sync interval.

The sync interval scales with cluster size to avoid a thundering herd. Small clusters (1-128 nodes) sync approximately every minute, with the interval increasing for larger clusters. Each agent picks a random staggered start time within the window to spread the load. Even in a healthy cluster, there is an inherent eventual-consistency window: a service registered on an agent may take up to one sync interval before it is visible cluster-wide. That delay is normal.

The problem this article covers is when the sync never completes, or completes with errors, and the staleness persists far beyond the expected window.

flowchart TD
    A[Client agent local state] -->|anti-entropy sync| B[RPC to server port 8300]
    B -->|sync succeeds| C[Catalog matches local state]
    B -->|sync fails| D[Catalog retains stale state]
    D --> E[Service missing from discovery or dead instance still listed]

Anti-entropy is a client-to-server RPC operation. It fails for the same reasons any client-to-server RPC fails: network connectivity loss, server overload, ACL denials, or the absence of a cluster leader. But unlike a user-facing API call that returns an error to the caller, anti-entropy failures are logged and retried silently. The operator discovers the problem only when the catalog is wrong.

A related trap: the agent treats its local state as authoritative. If someone registered a service directly through the catalog API (/v1/catalog/register) rather than through the agent (/v1/agent/service/register), the agent has no local record of that service. On the next anti-entropy cycle, the agent pushes its local state to the server, and the catalog-only registration is removed. This looks like anti-entropy is deleting services, but it is working as designed.

Common causes

Cause	What it looks like	First thing to check
RPC port 8300 blocked	Gossip healthy, agent alive, but sync never completes	Firewall or security group rules between agent and server subnets
No cluster leader	All writes failing, not just anti-entropy	`/v1/status/leader` returns empty string
ACL token lacks `node:write`	“Permission denied” or “ACL not found” in agent sync logs	Agent token permissions
Server FD exhaustion	Intermittent RPC failures across many agents	FD count on servers (see quick checks below)
Service registered via catalog API	Service disappears after approximately one sync cycle	Which registration endpoint was used
TLS certificate mismatch	RPC handshake failures after cert renewal on servers	Agent and server certificate expiry dates

Quick checks

Run these on the affected client agent (or the server, where noted). All are read-only.

# Agent's known servers (0 = agent cannot find any server)
curl -s http://localhost:8500/v1/agent/self | jq '.Stats.consul.known_servers'

# Leader existence (empty string = no leader, anti-entropy cannot sync)
curl -s http://localhost:8500/v1/status/leader

# Anti-entropy sync metrics on this agent
# TODO: verify exact metric names for anti-entropy sync
curl -s http://localhost:8500/v1/agent/metrics | grep -E "anti.entropy|sync"

# Client RPC failure counter
# TODO: verify exact metric name for RPC failure counter
curl -s http://localhost:8500/v1/agent/metrics | grep "client.rpc"

# Local agent services vs catalog
curl -s http://localhost:8500/v1/agent/services | jq 'keys'
curl -s http://localhost:8500/v1/catalog/service/<service-name> | jq length

# RPC port connectivity from client to server
nc -zv <server-ip> 8300

# Server file descriptor usage
ls /proc/$(pgrep -x consul)/fd | wc -l

# Recent anti-entropy errors in agent logs
# Adjust service name and log source for non-systemd deployments
journalctl -u consul --since "30 min ago" | grep -iE "anti.entropy|failed to sync"

How to diagnose it

Confirm the drift. Compare what the agent has locally against what the catalog shows. Register a test service on the agent and check whether it appears in the catalog within two sync intervals (approximately two minutes for a small cluster). If it does not appear, sync is failing.
Check agent-to-server connectivity. Run consul info on the affected agent and look at the known_servers count. If it shows 0, the agent has lost all server references. This can happen after server replacements where IPs change, because agents cache server addresses discovered through gossip. Test RPC port reachability directly with nc -zv <server-ip> 8300.
Verify leader existence. Query /v1/status/leader. An empty response means no leader, which blocks all writes including anti-entropy. If there is no leader, the problem is upstream. See Consul “No cluster leader”: every write is failing.
Check anti-entropy and RPC metrics. Look for RPC failure counters on the affected agent. Any sustained non-zero rate means the agent cannot complete RPCs to servers. Check anti-entropy metrics for sync failure counts. A consistently failing sync shows up as an error counter that increments at the sync interval.
Review agent logs for sync errors. Search for “anti-entropy” or “failed to sync remote state” in the agent logs. The error suffix identifies the cause: “no cluster leader,” “ACL not found,” “connection refused,” or an RPC timeout.
Check server-side capacity. If multiple agents fail simultaneously, the problem is likely server-side. Check FD usage on servers: approaching the FD limit prevents new RPC connections. Check Raft commit time for server overload. Check server logs for RPC connection rejections or handler errors.
Verify the registration method. If a specific service keeps disappearing despite being “registered,” check whether it was registered via /v1/catalog/register (catalog-direct) or /v1/agent/service/register (agent-local). Catalog-direct registrations are removed by anti-entropy because the agent has no local record.
Check ACL token permissions. If ACLs are enabled, verify the agent token includes node:write. The anti-entropy sync uses Catalog.Register, which includes node info updates alongside service registrations. A token with only service:write will fail. Look for “Permission denied” in the agent logs.
Check for script check rejections. If enable_local_script_checks is disabled (the secure default), script-based health checks defined in local config files will not be registered by the agent. Since they are not in local state, they will not appear in the catalog. This looks like a sync failure but is a registration rejection.

Metrics and signals to monitor

These signals catch silent catalog staleness before consumers notice. Prioritize client-agent signals over server signals for this failure mode.

Signal	Why it matters	Warning sign
Client RPC failure counter	Primary indicator of agent-to-server RPC pipeline health	Any sustained non-zero rate on a client agent
Anti-entropy sync metrics	Direct measure of sync completion	Failure counter incrementing at the sync interval
`consul.raft.commitTime`	Server-side write pipeline capacity	Sustained above 100ms indicates server strain
Agent `known_servers` count	Agent’s ability to reach any server	Drops to 0, or sudden drop from expected count
Server FD utilization	Capacity to accept new RPC connections	Above 70% of configured ulimit
Catalog registration rate	Catalog churn from anti-entropy reconciliation	Spike without corresponding deployments may indicate sync fighting against external changes

Fixes

RPC port blocked

Open port 8300 (the RPC port) in firewall or security group rules between agent and server subnets. Gossip uses port 8301 (LAN), so an agent can be alive in gossip while unable to make RPC calls. If only port 8301 is open, gossip works but anti-entropy does not. Verify bidirectional connectivity on 8300 after the rule change.

No cluster leader

Anti-entropy cannot sync without a Raft leader, because the sync writes to the catalog. Restore quorum first. See Consul “No cluster leader”: every write is failing.

ACL token lacks permissions

Update the agent’s ACL token to include node:write. The anti-entropy sync calls Catalog.Register, which updates node metadata alongside service registrations. A token with only service:write will be rejected. After updating the token, trigger a sync by reloading the agent configuration with consul reload, or wait for the next sync cycle.

Server FD exhaustion

Increase the file descriptor limit on servers. Consul recommends a minimum ulimit of 65536 for servers. Check both the shell ulimit and any systemd LimitNOFILE override, since systemd may impose a lower limit than the shell. Increasing the limit requires a process restart. Investigate the root cause of FD growth (connection leaks, excessive blocking queries, watch accumulation) separately.

Service registered via catalog API

Re-register through the agent endpoint: /v1/agent/service/register. The agent stores the registration locally, and anti-entropy pushes it to the catalog on the next cycle. Update any automation or scripts that use /v1/catalog/register. That endpoint is intended for out-of-band registration of external services without a local agent, not for normal service registration.

If external agents modify service tags via the catalog API, set enable_tag_override: true on the service definition. Without it, anti-entropy reverts tag changes on the next cycle because the agent’s local tags differ from the catalog.

TLS certificate mismatch

Distribute renewed server certificates to all agents. If verify_outgoing is enabled on agents (recommended for production), they will refuse RPC connections to servers presenting certificates they do not trust. After distributing the updated certificates, verify that the agent can reach the server RPC port. Plan certificate renewal to hit servers and agents in the same maintenance window.

Prevention

Monitor RPC failure rate on every client agent. This is the single most important signal for catching silent catalog staleness. Alert on any sustained non-zero rate. Server-side monitoring alone will not detect this failure.
Track anti-entropy sync success rate. Any sync failure is abnormal and warrants investigation. Sync interval jitter (randomized staggering) is normal; alert on sustained failure, not on timing variation.
Always register services through the agent API. Never use /v1/catalog/register for services that have a local agent. Document this in registration runbooks and review automation scripts.
Ensure ACL tokens include node:write. Audit agent tokens when ACL policies change. A token that worked before a policy tightening will silently break anti-entropy.
Keep FD limits high on servers. Set the ulimit to at least 65536 and monitor utilization proactively. FD exhaustion is a cliff-edge failure: connections work fine until the limit is hit, then all new RPCs fail.
Coordinate TLS certificate renewal. Renew agent and server certificates in the same maintenance window to eliminate mismatch periods.

How Netdata helps

Per-second collection of client RPC failure metrics on every agent catches the sustained non-zero rate that signals a broken agent-to-server pipeline before consumers notice stale discovery results.
Anti-entropy sync metrics are collected alongside server-side Raft, gossip, and RPC metrics, so you can correlate agent-side sync failures with server-side events like leader changes or FD exhaustion in a single view.
ML anomaly detection on sync interval deviations and catalog registration rates surfaces slow drift that threshold-based alerts miss, particularly when sync failures are intermittent.
File descriptor utilization tracking on servers provides early warning before FD exhaustion blocks new RPC connections from agents.
Correlation across the agent-to-server path shows whether a sync failure on one agent is isolated (network issue on that node) or correlated across many agents (server-side problem), narrowing the search from “which of 500 agents is broken” to “the server is rejecting connections.”