Consul client rpc failed: agents alive but the catalog is going stale

A client agent’s consul.client.rpc.failed counter ticks up. The agent process is running, gossip reports the node as alive, local health checks execute on schedule, and consul members lists the node as healthy. The server cluster looks fine: leadership is stable, Raft commit times are normal, and there is no election noise. Nothing on the standard dashboard is red.

The catalog is going stale anyway.

Consul runs two independent network paths between an agent and the server cluster. Gossip membership flows over LAN Serf (TCP and UDP port 8301). State updates flow over the server RPC pipeline (TCP port 8300). The first can be perfectly healthy while the second is broken, and most dashboards only watch the first. Anti-entropy sync runs on an adaptive interval (about one minute for clusters of 1–128 agents, increasing with cluster size) and pushes local agent state to the server catalog over RPC. When RPC fails, the catalog stops receiving updates from that agent but keeps serving whatever it last knew. Consumers (DNS, HTTP API, load balancer integrations, service mesh sidecars) keep getting answers, just progressively wrong ones.

For the broader architecture, see the mental model. For the full signal list, see the monitoring checklist. This article is the narrow playbook for elevated consul.client.rpc.failed.

What this means

consul.client.rpc.failed is a counter that increments whenever a client agent attempts an RPC against a server and the call fails. The companion counter consul.client.rpc counts every attempt. A healthy agent shows consul.client.rpc.failed flat or near zero, with consul.client.rpc ticking at the anti-entropy cadence plus any watches or DNS forwarding.

When the failed counter climbs, the agent cannot push the following to the catalog:

• New service registrations and deregistrations
• Health check transitions (passing to critical, or the reverse)
• Coordinate updates used by network coordinates
• KV writes routed through the agent

The agent keeps running checks locally, so a process that crashed on the host is correctly marked critical in the agent’s local state, but that critical state never reaches the catalog. From the catalog’s perspective the instance still looks healthy. Load balancers and service mesh sidecars that read from the catalog keep sending traffic to a dead endpoint.

flowchart LR
  A[Client agent
checks run locally] -->|gossip 8301 OK| B[Server cluster
alive in members]
  A -->|RPC 8300 FAIL| C[Catalog
stale state]
  A -->|anti-entropy| C
  C -->|serves stale| D[DNS / API / LB
route to dead instance]

The asymmetry is the trap. Gossip works, so the node appears alive. The leader is stable, so server dashboards are green. The failure is silent and visible only in two places: the consul.client.rpc.failed counter on the affected agent, and drift between what the agent believes locally and what the catalog exposes.

Common causes

A subtle variant: if the agent’s own rate limiter (the limits block) is configured too aggressively, the counter that climbs is consul.client.rpc.exceeded, not consul.client.rpc.failed. The RPC is rejected locally before it leaves the agent, so there is no connection error in the logs. Treat sustained non-zero consul.client.rpc.exceeded as the same class of problem: the agent is not getting state to servers.

Quick checks

Run these on the affected client agent first, then on a server. All are read-only and safe during incidents.

# How many servers does this agent know about?
consul info | grep -A2 "known_servers"

# Is the agent alive in gossip from the server side?
consul members

# Is the RPC port actually reachable from this agent?
nc -zv <server-ip> 8300

# Pull the RPC counters from the agent's telemetry
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -E "consul.client.rpc"

# Agent self view: known servers and config
curl -s http://127.0.0.1:8500/v1/agent/self | jq '.Stats.consul | {known_servers, server}'

# Check recent anti-entropy outcomes in the agent log
journalctl -u consul --since "10 min ago" | grep -i anti_entropy

# Tail agent logs for RPC, TLS, or ACL errors
journalctl -u consul -f --since "10 min ago" | grep -iE "rpc|tls|x509|permission|anti_entropy"

On a server, run:

# Leader identity and Raft peer count
consul operator raft list-peers

# Server FD consumption
ls /proc/$(pgrep -x consul)/fd | wc -l
cat /proc/$(pgrep -x consul)/limits | grep "Max open files"

# How many client RPC connections is this server holding?
ss -tnp '( sport = :8300 )' | wc -l

How to diagnose it

1. Confirm the failure is RPC, not gossip. From the affected agent, consul members must show the server nodes as alive. If gossip itself is partitioned, you are looking at a different problem. See Consul gossip flapping or Consul serf queue backlog.
2. Localize the scope. One agent points at host-level network, cert, or ACL. Many agents in the same subnet point at a firewall or route change. Many agents across subnets point at the server side.
3. Verify the RPC port end to end. From the agent, nc -zv <server-ip> 8300. A timeout or refused connection narrows the cause to network or server handler.
4. Inspect the agent’s known server list. consul info shows known_servers. Zero known servers means the agent has lost its server discovery path. This happens when servers are replaced simultaneously with new IPs and the agent’s cached addresses are stale.
5. Read the agent logs for the failure class. The error string tells you which cause you are dealing with:
   • i/o deadline reached or connection refused: network or server handler.
   • x509: certificate is valid for ...: TLS hostname mismatch.
   • Permission denied: ACL.
   • No known Consul servers: empty server list.
6. Check server-side capacity. If many agents fail at once, the servers are the bottleneck. Check FD usage, goroutine count, and open FD count against the configured process limit. Check whether limits.rpc_max_conns_per_client is configured (default 100; introduced in Consul 1.6.3) and whether per-source-IP connection caps are being hit.
7. Check anti-entropy success and drift. Look for sustained anti_entropy errors in the agent log, then compare /v1/agent/services locally with /v1/catalog/service/<name> in the catalog; persistent divergence confirms that the RPC break is causing catalog drift.

Metrics and signals to monitor

For the fuller signal list, see Consul monitoring maturity model.

Fixes

Firewall asymmetry (8301 open, 8300 blocked)

The most common cause. A security-group, host firewall, or network policy change keeps 8301 open for gossip but drops 8300 for RPC.

• Confirm with nc -zv <server-ip> 8300 from the agent host.
• Compare against the working baseline. If only the agent’s subnet is affected, the change is scoped to that path.
• Open TCP 8300 from client agents to server nodes in the direction the connection is initiated.
• Document the ports together so the next change does not split them. Consul requires 8300 for server RPC and 8301 (TCP and UDP) for LAN gossip as a pair.

Server RPC handler exhaustion

When servers run low on file descriptors, new RPC connections are refused. The client sees i/o deadline reached or connection reset by peer.

• Check the Consul process open-FD count against its configured limit (/proc/$PID/fd versus /proc/$PID/limits). For write-heavy clusters, HashiCorp says the default ulimit of 1024 must be increased; no universal floor is published.
• Check for connection leaks: ss -tnp '( sport = :8300 )' connection count vs agent count. A server holding tens of thousands of connections from a handful of agents indicates a leak in those agents.
• Raise LimitNOFILE (systemd) or ulimit -n and restart the server during a maintenance window. This is disruptive and drops in-flight RPC connections.
• Investigate the leak separately. Common sources are consul-template blocking queries, leaked watches, and service mesh xDS streams.

TLS certificate mismatch

With tls.internal_rpc.verify_server_hostname = true (the nested form added in Consul 1.12; verify_server_hostname is the deprecated pre-1.12 top-level equivalent), the agent validates that the server certificate is valid for server.<datacenter>.consul.<domain>. A cert valid for a different name produces x509: certificate is valid for X, not server.dc1.consul.example.com.

• Pull the cert the agent is presenting or trusting and inspect SANs: openssl x509 -in <cert> -noout -text | grep -A1 "Subject Alternative Name".
• Confirm the cert was issued with the Consul-internal RPC naming convention, not a generic service DNS name.
• Re-issue and redistribute the cert. Rotating certs on servers without also rotating on agents (or vice versa) is the typical trigger.
• Verify verify_incoming and verify_outgoing are consistent across the cluster. Mixed modes produce confusing partial failures.

ACL permission denied

The agent has a token but the token lacks permission for a specific RPC method. The classic signature is Coordinate.Update failing repeatedly because the token cannot write coordinates.

• Pull the agent’s token and check the attached policies.
• Confirm the token grants service:write (or the equivalent for what the agent registers), node:write, and coordinate-write permissions.
• If the failure started after an ACL policy change, roll back the policy first, then tighten deliberately.
• Distinguish from token replication lag in federated DCs. In a secondary DC, a recently created token may not have replicated yet.

No known servers

The agent has nobody to dial. Logs show No known Consul servers and consul info reports known_servers: 0.

• Common cause: all servers were replaced with new IPs (new subnets, ASG replacement, redeploy) and the agent’s cached server list is stale.
• Trigger a re-join: add retry_join pointing at the new server addresses, or restart the agent so it re-discovers via gossip.
• Brief blips during rolling server restarts are normal. Persistent zero is not.

Agent-side rate limiter

If the climbing counter is consul.client.rpc.exceeded rather than consul.client.rpc.failed, the agent’s limits configuration is dropping RPCs locally.

• Inspect the limits block on the affected agent.
• Compare the configured RPC rate and burst against the agent’s actual workload (number of services, checks, watches).
• Tune upward or remove the limit if it was set defensively without considering anti-entropy spikes after recovery.

Prevention

• Alert on sustained non-zero consul.client.rpc.failed on every agent, not just servers. Server-side dashboards do not show this signal. It only exists on the agent.
• Alert on consul.client.rpc.exceeded separately. Different cause, different fix, same user-visible symptom.
• Treat ports 8300 and 8301 as a pair in firewall policy. Any change to one must review the other.
• Monitor server FD usage with low thresholds. Page at 80% of ulimit -n, plan at 60%. FD exhaustion is cliff-edge and cascades into RPC refusal across the fleet.
• Run cert rotation as a coordinated procedure, not a server-only task. Test that agent-trusted certs validate server.<dc>.consul.<domain> before rollout.
• Periodically compare local agent state with catalog state. For a sample service, query both /v1/agent/services on the agent and /v1/catalog/service/<name> on a server. Persistent divergence indicates the RPC pipeline is not keeping up even if consul.client.rpc.failed is quiet.

How Netdata helps

• Per-second collection of consul.client.rpc.failed, consul.client.rpc, and consul.client.rpc.exceeded on every agent lets you see the failed counter climb before anti-entropy lag becomes user-visible.
• Correlate agent RPC failures with server-side open-FD count and goroutine count, and Raft commit time on the same timeline to localize the cause in seconds.
• Composite alerts pair consul.client.rpc.failed > 0 with healthy consul.serf.lan.members to surface the silent catalog staleness signature directly, rather than waiting for downstream consumer complaints.
• Per-agent dashboards make it cheap to spot the difference between one host with a bad cert and a whole subnet behind a bad firewall rule.

Related guides

• Consul gossip encryption key mismatch
• Consul gossip flapping
• Consul serf queue backlog
• Consul gossip storm after mass recovery
• How Consul actually works in production
• Consul leader election storm
• Consul monitoring checklist
• Consul monitoring maturity model
• Consul “No cluster leader”
• Consul raft commitTime high
• Consul Raft data directory full
• Consul raft lastContact rising