Consul Operations Guides on Netdata

Consul "ACL not found": requests rejected after a token or policy change

Mon, 01 Jan 0001 00:00:00 +0000

“ACL not found” appears as HTTP 403 responses carrying “ACL not found” or “token does not exist: ACL not found”. It surfaces in agent and server logs, sidecar injector output, and as failed service registrations, health check updates, or KV writes. The blast radius depends on which token is missing: a single application token breaks one service; a replication or agent token can break an entire datacenter’s authorization pipeline.

The message is frequently misread. “ACL not found” does not mean the token has the wrong permissions. It means the Consul server receiving the request has no record of the token’s SecretID. The token was deleted, never created in this datacenter, or has not yet arrived via ACL replication. The fix path is completely different from “Permission denied”, which indicates the token is known but lacks a specific right.

Consul "No cluster leader": every write is failing

Mon, 01 Jan 0001 00:00:00 +0000

When Consul cannot elect a Raft leader, every write fails. The error surfaces to clients as rpc error making call: No cluster leader, sometimes as no known leader. Service registrations hang, KV writes return errors, sessions and ACL tokens cannot be created, and health check state stops reaching the catalog.

Only stale reads continue to work: stale-mode API queries and DNS lookups (which default to allow_stale) return whatever the responding server last committed. Default and consistent reads fail because they require a leader, which can surface differently to callers depending on their read mode.

Consul "too many open files": file descriptor exhaustion on servers

Mon, 01 Jan 0001 00:00:00 +0000

When a Consul server hits its file descriptor ceiling, every consumer that needs a new FD fails immediately: RPC connections are refused, DNS listeners stop accepting queries, xDS streams to Envoy sidecars cannot open, and outbound health-check connections fail. Gossip probes time out and the failure detection protocol marks otherwise healthy peers as suspect or failed. A healthy-looking cluster a minute ago is now generating cascading pages.

File descriptor exhaustion is not graceful degradation. Each socket, pipe, and file handle counts against a hard kernel limit. Once that limit is reached, every syscall that creates a new FD (accept, socket, open, pipe, epoll_create) returns EMFILE, regardless of available CPU, memory, or network capacity. A consul reload either fails outright (it needs FDs to read config and re-open listeners) or succeeds but leaves the server half-functional. Recovery requires raising the limit and restarting the process.

Consul ACL resolution latency: token cache thrashing on every request

Mon, 01 Jan 0001 00:00:00 +0000

Every authenticated Consul request pays an ACL resolution tax. When that tax is sub-millisecond, nobody notices. When the token cache cannot hold the working set, every request becomes a miss and that tax multiplies across DNS lookups, HTTP API calls, RPC forwarding, and Connect intention evaluation. The cluster keeps answering, but slowly, and the slowness is uniform across every authenticated path.

The primary signal is consul.acl.ResolveToken (a summary timer, milliseconds). Healthy cached resolution is sub-millisecond; a cold miss against the authoritative datacenter costs low single-digit milliseconds. Sustained values above ~10ms are TICKET-worthy: ACL resolution sits on the hot path of every authenticated operation. It surfaces as elevated DNS latency, slow HTTP API responses, and stretched Connect sidecar handshakes long before any single subsystem fails.

Consul anti-entropy not syncing: local agent state and the catalog drifting apart

Mon, 01 Jan 0001 00:00:00 +0000

A service is registered on the local agent. The health check is passing locally. But dig myservice.service.consul returns nothing, the load balancer has no targets, and the catalog API shows no instances. The servers have a leader, gossip is healthy, Raft metrics look normal. The problem is between the agent and the catalog.

Consul’s catalog is authoritative on the servers. Each agent maintains its own local state (registered services, health checks, node metadata) and periodically reconciles that state with the server catalog through a background process called anti-entropy sync. The agent treats its local view as authoritative and pushes changes to the catalog. When this sync fails, the catalog retains the last-known state it received from that agent. A service registered locally never appears cluster-wide. A service deregistered locally continues to show up in DNS and API queries.

Consul blocking query accumulation: leaked watches that pile up goroutines

Mon, 01 Jan 0001 00:00:00 +0000

Goroutine count on your Consul servers is creeping upward. File descriptor usage follows the same trajectory. Neither reverses during quiet periods. Eventually the server hits its FD limit or goroutine scheduling overhead degrades everything: new connections are refused, health checks stop reaching the catalog, anti-entropy sync stalls, and the cluster cascades.

The culprit is leaked blocking queries. Consul’s watch mechanism and any client using the HTTP long-polling API (?index= and ?wait= parameters) holds a goroutine and an FD on the server for up to the wait timeout (5 minutes default, 10 minutes maximum). A fleet of consul-template instances, application-level watchers, or service mesh control planes can open thousands of concurrent blocking queries. This is fine when queries are properly opened and closed. The leak starts when they are not: a client crashes without cleanly closing its connection, a consul-template bug prevents goroutine cleanup, a watch configuration exceeds the server’s tracking capacity, or a go-memdb WatchSet bug causes goroutine proliferation.

Consul catalog bloat: too many services and checks slowing everything down

Mon, 01 Jan 0001 00:00:00 +0000

DNS latency is creeping up. API calls to /v1/catalog/services take longer than they used to. Server restarts that used to take 20 seconds now take minutes. The Raft snapshot on disk keeps growing, and so does server RSS. No single failure point, no stack trace, no page. Just a slow, compounding drag on everything Consul does.

This is catalog bloat. Registered service instances and health checks grow the in-memory state store, the on-disk Raft snapshot, anti-entropy reconciliation work, and catalog-scan latency for every DNS and API query. The cost compounds week over week until a snapshot creation tips a leader election, a restart takes long enough to miss a deploy window, or DNS p99 crosses the threshold where applications time out on service discovery.

Consul catalog poisoning: services registered from unknown nodes

Mon, 01 Jan 0001 00:00:00 +0000

A service instance appears in your catalog on a node you have never deployed. Its ServiceAddress points somewhere you do not control. That is catalog poisoning: a registration that did not come from your infrastructure.

Without strict ACLs, any process that can reach the Consul HTTP API can call the registration endpoints and add an instance for any service name. Consumers resolving discovery through Consul DNS or the catalog and health APIs will route a fraction of their traffic to that address. In a non-Connect deployment there is nothing in the data path that verifies the instance is what it claims to be.

Consul client rpc failed: agents alive but the catalog is going stale

Mon, 01 Jan 0001 00:00:00 +0000

A client agent’s consul.client.rpc.failed counter ticks up. The agent process is running, gossip reports the node as alive, local health checks execute on schedule, and consul members lists the node as healthy. The server cluster looks fine: leadership is stable, Raft commit times are normal, and there is no election noise. Nothing on the standard dashboard is red.

The catalog is going stale anyway.

Consul runs two independent network paths between an agent and the server cluster. Gossip membership flows over LAN Serf (TCP and UDP port 8301). State updates flow over the server RPC pipeline (TCP port 8300). The first can be perfectly healthy while the second is broken, and most dashboards only watch the first. Anti-entropy sync runs on a fixed interval and pushes local agent state to the server catalog over RPC. When RPC fails, the catalog stops receiving updates from that agent but keeps serving whatever it last knew. Consumers (DNS, HTTP API, load balancer integrations, service mesh sidecars) keep getting answers, just progressively wrong ones.

Consul Connect "no healthy upstream": Envoy has no endpoints to route to

Mon, 01 Jan 0001 00:00:00 +0000

When an application in a Consul Connect mesh receives HTTP 503 with response_flags=UH in the Envoy access log, Envoy’s router rejected the request because it had zero eligible endpoints for the upstream cluster. The downstream service is fine, the network is fine, and the request never left the sidecar. Somewhere between the Envoy admin port, the local Consul agent, the Consul servers, and the upstream instances, the endpoint set has gone empty, all-unhealthy, or silently ejected.

Consul Connect CA rotation failure: a root roll that never finished

Mon, 01 Jan 0001 00:00:00 +0000

Consul Connect CA root rotation promotes a new root, cross-signs the new intermediate against the old root, lets both roots coexist while leaf certificates roll over, and retires the old root only after the last leaf signed by it expires. When that rollout never completes, the first thing you notice is intermittent mTLS failures between specific service pairs, hours after the rotation was triggered, with no obvious network or config change to blame.

Consul Connect certificate expired: mTLS handshakes failing across the mesh

Mon, 01 Jan 0001 00:00:00 +0000

Consul Connect relies on short-lived leaf certificates for service-to-service mTLS. When those certificates expire without rotating, the failure is cliff-edged: handshakes that worked seconds ago start failing, Envoy logs fill with TLS errors, and dependent services stop communicating. The blast radius grows over minutes to hours as each leaf certificate reaches its own expiry independently.

The first symptom is usually intermittent: a specific upstream starts returning connection resets or 5xx errors. As more leaves expire, failures fan out across service pairs until the mesh is broadly broken. Expiry alone is a countdown with no early warning. The signal that matters earlier is renewal success: are leaves actually being re-issued, or is the rotation pipeline silently broken while the clock runs out?

Consul critical health checks spiking: real outage or broken checks?

Mon, 01 Jan 0001 00:00:00 +0000

The alert fires when /v1/health/state/critical jumps from single digits to dozens or hundreds of checks in minutes. The page says “services are unhealthy,” but that is ambiguous: either real services are down, or the checks themselves are broken. Pick the wrong fork and you burn an hour on the wrong dependency.

The first decision is not “what failed.” It is “is this a service-level event, a node-level event, or a monitoring-level event?” The cheapest signal is the check Output text field, which carries the raw error from check execution. It holds strings like connection refused, i/o timeout, certificate has expired. That field usually resolves the diagnosis in seconds, and most teams ignore it until their second or third incident.

Consul cross-datacenter query failure: prepared-query failover masking a DC outage

Mon, 01 Jan 0001 00:00:00 +0000

A Consul prepared query with cross-DC failover does exactly what it was configured to do: it silently forwards lookups to a remote DC when local results are empty. Consumers receive valid service discovery results, applications keep connecting, and no error fires. The local DC may have zero healthy instances, and nobody on the consumer side knows.

The DNS response is identical whether nodes came from the local DC or a remote one. There is no EDNS flag, no source record, no indicator in the answer. Receiving results does not mean local health.

Consul DeregisterCriticalServiceAfter: instances vanishing from the catalog

Mon, 01 Jan 0001 00:00:00 +0000

A service that was merely slow to recover has disappeared from the Consul catalog. Healthy-instance count dropped to zero. DNS returns empty results. Load balancers have no targets. Yet the service process is running, the host is up, and the Consul agent on that host is healthy and gossiping. DCSA did exactly what you configured.

DeregisterCriticalServiceAfter (DCSA) is an agent-side reaper that removes a service instance and its checks from the catalog once the check has been critical for a configured duration. It is deliberate garbage collection for stale entries. Leave it unset and critical checks plus their output strings accumulate unbounded in the state store, growing memory and slowing snapshots.

Consul DNS latency high: slow lookups stalling connections and failovers

Mon, 01 Jan 0001 00:00:00 +0000

Slow Consul DNS lookups rarely look like a DNS problem at first. Applications see connection timeouts, retries, and slow startup. Load balancers see health check flapping. Service mesh sidecars see upstream resolution failures. The DNS layer is invisible to most teams until consul.dns.domain_query crosses the page threshold.

The first decision in a Consul DNS latency incident is structural: is the slowness in the server cluster (Raft, disk, catalog size) or in the DNS path itself (TTL, agent CPU, query complexity, downstream resolvers)? The signals are all in agent telemetry, but you have to know which ones to correlate.

Consul DNS SERVFAIL: service discovery is broken for your applications

Mon, 01 Jan 0001 00:00:00 +0000

SERVFAIL from Consul’s DNS interface on port 8600 means service discovery is broken for every application resolving names through it. Applications see connection refused, timeouts, and cascading failovers. The symptom is broad, but the root cause lives along one of four layers: the downstream resolver (if present), the Consul agent’s built-in DNS server, the agent-to-server RPC channel, or the server’s catalog state.

NXDOMAIN is a related but distinct signal. SERVFAIL means the resolver could not complete the query: server overload, timeout, unreachable upstream, or internal error. NXDOMAIN means the query completed but the name does not exist in the catalog. Both break discovery, but they point to different layers. Conflating them during an incident wastes time.

Consul Go GC pauses: stop-the-world stalls that disturb Raft timing

Mon, 01 Jan 0001 00:00:00 +0000

Consul servers are Go binaries, and Go’s garbage collector still has stop-the-world phases despite doing most of its work concurrently. On a server with a large live heap and a high allocation rate, those pauses can stretch into hundreds of milliseconds. The Raft leader sends heartbeats from a goroutine in the same process. When a pause runs long enough, the leader stops heartbeating; followers see consul.raft.leader.lastContact climbing toward the election timeout and start a new election. Writes block for the duration.

Consul goroutine count climbing: the leak behind slow resource exhaustion

Mon, 01 Jan 0001 00:00:00 +0000

consul.runtime.num_goroutines is a gauge of how many concurrent operations the process is juggling: every blocking query, every gRPC or xDS stream, every health check handler runs inside a goroutine. In a healthy cluster the number is baseline-dependent but flat. When it creeps upward hour over hour, something is spawning goroutines and not cleaning them up.

The absolute number is a distraction. A medium cluster idles between a few hundred and 20,000 goroutines; a large Connect deployment legitimately runs higher. The trend is the signal. Monotonic growth without a matching increase in services, watchers, or sidecar count is a leak, regardless of where the number sits.

Consul gossip encryption key mismatch: a botched keyring rotation splits the pool

Mon, 01 Jan 0001 00:00:00 +0000

You rotated the gossip encryption key. Nodes are dropping from the member list, showing as “failed” or “suspect” to peers, and service discovery is degrading. The network is fine. The agents are running. Raft may even be stable. But the gossip pool has split.

The root cause is at the protocol layer. Consul’s Serf gossip protocol encrypts every message with a symmetric key. When nodes hold different keys, they cannot decrypt each other’s gossip messages. Each side sees the other as unresponsive, and failure detection kicks in. This looks exactly like a network partition, but pings between the affected hosts succeed.

Consul gossip flapping: nodes oscillating between alive, suspect, and failed

Mon, 01 Jan 0001 00:00:00 +0000

When Consul nodes rapidly flip between alive, suspect, and failed in the gossip pool, every state transition fires a membership event that downstream consumers react to. Load balancers pull endpoints in and out, consul-template reloads configurations, and Envoy sidecars receive new xDS endpoint pushes. A flapping node generates more operational noise than a cleanly failed one.

The Serf gossip protocol converges under stable network conditions. Under packet loss, CPU starvation, or partial partitions, it oscillates instead. The symptom appears in consul members as nodes cycling between alive, suspect, and failed; in Serf metrics as spikes in consul.serf.lan.member.flaps; and in downstream systems as configuration reload storms that track the flap cadence.

Consul gossip storm after mass recovery: rejoin floods and anti-entropy spikes

Mon, 01 Jan 0001 00:00:00 +0000

After a mass recovery event (AZ healing, partition recovery, fleet-wide rolling restart), dozens or hundreds of Consul agents rejoin the LAN gossip pool in a compressed window. Each returning agent generates member-join events that every other member processes, and triggers anti-entropy sync to reconcile its local service registrations against the server catalog. The coordinated spike hits Serf message queues, catalog registration writes, and Raft commits simultaneously.

Most clusters absorb the spike and recover within minutes. The risk is write volume pushing Raft commitTime toward the election timeout. Once followers cannot reach the leader within the election window, they start new elections, cascading a transient burst into leader thrashing and repeated write outages.

Consul health check flapping: the passing/critical oscillation that churns the catalog

Mon, 01 Jan 0001 00:00:00 +0000

Every health check state transition is a catalog write. The client agent runs the check locally, detects a status change, and pushes the update to the leader. The leader commits a Raft log entry, the FSM applies it, per-service caches invalidate, blocking queries watching that service return, and every downstream consumer (load balancer, Envoy control plane, consul-template, DNS resolver) re-evaluates. One flapping check is an annoyance. A dozen flapping in unison saturates the Raft write pipeline and looks indistinguishable from a registration storm.

Consul intention denied: service-to-service traffic blocked by policy

Mon, 01 Jan 0001 00:00:00 +0000

Service A can no longer reach service B. Envoy sidecars log 403 responses or connection resets. The Consul server cluster looks healthy: leader stable, Raft committing, gossip intact, certificates within their lifetime. Traffic that worked an hour ago is now blocked.

This is almost always policy, not infrastructure. Consul Connect intentions are the mesh authorization layer, evaluated at connection establishment and enforced by Envoy RBAC filters. When an intention denies a connection, the data plane is doing exactly what it was configured to do. The diagnostic job is to determine whether the denial is correct (policy working as designed) or a misconfiguration: wrong identity, missing allow, precedence mistake, or stale cache.

Consul KV store saturation: using the KV store as a database

Mon, 01 Jan 0001 00:00:00 +0000

Consul KV writes are slow. Raft commit times are climbing. Service registration and health check updates lag. DNS queries for service discovery take longer than usual. This combination often points to an application treating Consul’s KV store as a general-purpose database: high-frequency writes, large values, or deep key trees.

The problem is architectural, not a tuning issue. The KV store is part of Consul’s Raft finite state machine. Every KV write is a Raft log entry that must be committed through the leader, replicated to a quorum of servers, and applied to the in-memory state store on every server. Every KV value is included in every Raft snapshot. When an application treats the KV store as a database, the write load backs up the entire Raft pipeline, and every Consul subsystem that depends on Raft degrades with it.

Consul KV value too large: the 512KB limit and why approaching it hurts

Mon, 01 Jan 0001 00:00:00 +0000

You pushed a value into Consul KV and got HTTP 413, or your server logs are filling with lines like Request body(524401 bytes) too large, max size: 524288 bytes. That is the obvious failure: a single value crossed the default 512KB ceiling. The less obvious failure is that values well under the limit are still expensive, because every KV byte is replicated through Raft to every server and re-emitted in every snapshot.

Consul KV write latency high: every write is a Raft commit

Mon, 01 Jan 0001 00:00:00 +0000

A KV PUT that used to return in 20ms is now taking 500ms, 2s, or timing out entirely. Applications stall on locks, configuration writes queue, and leader election alerts may be firing. Consul’s KV is not a standalone subsystem. Every KV write is a Raft log entry that must be replicated to a quorum of servers, fsynced to disk on each, and applied to the in-memory state machine before the API call returns. KV write latency is a direct reflection of Raft commit health.

Consul leader election storm: repeated elections and rolling write outages

Mon, 01 Jan 0001 00:00:00 +0000

Your Consul cluster has a leader. Writes are still failing. The leader keeps changing. Applications see intermittent “no cluster leader” errors, watches reconnect, DNS returns stale results, and every few seconds a different server wins an election only to lose it again.

This is a leader election storm. Unlike a clean one-time failover, the cluster never stabilizes. Each election blocks all writes for one to several seconds. When elections recur faster than the recovery window, the cluster is effectively write-unavailable while technically always having “a leader.” A naive alert on “no leader” stays silent. The real signal is recurrence: leadership transitions accumulating over time, each one a brief but real outage.

Consul leader stable but commits stalled: writes silently failing

Mon, 01 Jan 0001 00:00:00 +0000

You are paged for “Consul writes failing.” You check /v1/status/leader: it returns an address. You check consul members: all servers alive. Leadership has been stable for hours. Your “is there a leader?” alert is silent. Yet KV writes time out, service registrations hang, and sessions cannot be created.

The leader is alive, holds leadership, and answers health probes. But its commit index is not advancing. Somewhere between the leader receiving a write and that write becoming visible in the FSM, the pipeline is stalled. The cluster is write-dead.

Consul lost quorum: Raft peers below the majority needed to elect a leader

Mon, 01 Jan 0001 00:00:00 +0000

A Consul cluster has lost quorum when the number of voting Raft peers drops below the majority required to elect a leader. For a 3-server cluster that means fewer than 2 voters. For a 5-server cluster, fewer than 3. With no leader, every write fails: service registrations, health-check state updates, KV writes, session creation, ACL token creation. Reads served in stale mode still return data, but that data is frozen at the moment quorum was lost.

Consul monitoring checklist: the signals every production cluster needs

Mon, 01 Jan 0001 00:00:00 +0000

Consul runs three subsystems concurrently: Raft consensus, Serf gossip, and a catalog state machine backed by BoltDB. Each fails independently. A monitoring setup that checks only “is there a leader” and “are all members alive” will miss the most common production incidents: slow disk causing leader churn, client agents losing RPC connectivity while gossip stays green, blocking query leaks, and Connect certificate rotation failures.

This checklist organizes signals into four maturity levels. Each builds on the previous one and maps to specific failure patterns: leader thrashing, split brain after partition, gossip storms during mass recovery, silent catalog staleness, and KV store saturation. Use it as an audit tool. Identify which level your current monitoring reaches, then close gaps at that level before chasing the next tier.

Consul monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Consul runs three independent subsystems that must all be healthy for the cluster to function: Raft consensus, Serf gossip, and the catalog state machine with its anti-entropy sync pipeline. Most teams monitor one or two well and discover the third during an incident.

The model is cumulative. Each level adds signals to the previous one; you cannot skip to Mature without the Operational baseline in place. A team with composite-pattern detection at Expert but no client-agent RPC monitoring at Operational is blind to the most common silent failure mode: the catalog drifting from reality while every server-side metric looks healthy.

Consul on EBS: burst-credit exhaustion and the sudden latency cliff

Mon, 01 Jan 0001 00:00:00 +0000

Consul is stable. Raft commit times under 10ms, leadership steady, discovery in single-digit milliseconds. Then, with no deploy, no config change, and no traffic spike, writes start timing out. consul.raft.commitTime jumps from 5ms to 500ms. Followers report lastContact climbing toward the election timeout. Within seconds, the cluster elects a new leader, then another, then another. Every write fails with “no cluster leader.”

The root cause is under Consul, in the block storage layer. AWS gp2 EBS volumes accumulate I/O burst credits when idle and spend them to sustain burst performance up to 3000 IOPS. When credits deplete, the volume drops to its baseline rate of 3 IOPS per GiB of volume size. A 100 GB gp2 volume falls from 3000 IOPS to 300 IOPS. Consul has no metric for burst credit depletion. The disk simply gets slow, and Raft, which fsyncs every log entry to disk before acknowledging it, hits a wall.

Consul Permission denied (403): authorized token, missing permission

Mon, 01 Jan 0001 00:00:00 +0000

A 403 “Permission denied” from the Consul HTTP API means the request reached the server with a token it recognizes, but policy evaluation denied the operation. The token exists, resolves, and is not expired. It lacks the rule covering the resource being touched: a service, a KV prefix, a node, an operator endpoint. This differs from “ACL not found”, where the SecretID is unknown to the server, and the two need different responses.

Consul raft commitTime high: the write pipeline is slowing down

Mon, 01 Jan 0001 00:00:00 +0000

consul.raft.commitTime is the single best indicator of Raft health on a Consul server. It is a leader-only timer that folds disk write latency, replication latency, and FSM apply time into one number. When it climbs, every Consul write slows: service registrations queue, KV writes block, health check updates lag, and consumers downstream of the catalog start timing out. The disk-bound leader cannot get log entries committed fast enough, followers lose touch, and the next step is leader elections, during which the cluster cannot commit writes at all.

Consul Raft data directory full: the server that can no longer write

Mon, 01 Jan 0001 00:00:00 +0000

The server’s data_dir volume hits 100%. Raft cannot append to raft.db and cannot fsync snapshot files. The server either refuses writes or crashes. If the affected server is the leader, the commit index stalls and followers drift toward an election.

The most common cause is not disk hardware failure. It is a snapshot that keeps failing. Raft truncates its log only after a successful snapshot, so each failed attempt leaves raft.db larger than it should be. Eventually the log fills the volume.

Consul raft lastContact rising: followers drifting toward an election

Mon, 01 Jan 0001 00:00:00 +0000

consul.raft.leader.lastContact is the predictive signal before a Raft election fires. It measures the elapsed time since the leader last successfully contacted each follower server. When this value rises, a follower is drifting toward the election timeout. Cross that threshold and the follower starts a new election: writes stall, clients see “no cluster leader” errors, and downstream consumers retry in a thundering herd.

Most Consul metrics tell you something already broke. lastContact tells you something is about to break. A healthy cluster holds this value in the low tens of milliseconds. Trending above 200ms warrants investigation. Sustaining above 500ms means an election is imminent. The election timeout sits at approximately 1000ms with raft_multiplier set to 1 (the production recommendation). Higher multipliers raise the timeout proportionally.

Consul Raft log divergence: catching a corrupt follower before it wins an election

Mon, 01 Jan 0001 00:00:00 +0000

A divergent Consul follower can gossip normally, answer Serf probes, and appear in consul members as alive while its persisted Raft state is corrupt or inconsistent. The leader replicates to the healthy majority and clients see correct data. The divergence becomes catastrophic only when that follower wins an election and serves state the cluster never agreed on.

Two failure modes look identical from gossip. Replication lag: a follower falls behind on commit index due to slow disk or degraded network. Raft’s Leader Completeness Property prevents a lagging follower from winning, so this is self-correcting once the bottleneck clears. Silent corruption: the follower’s log index and term match the leader’s, but the entries it applied are wrong. The up-to-date check passes, the follower can win, and the divergence surfaces as incorrect registrations, stale KV, or broken ACL state.

Consul registration storm: catalog churn overwhelming Raft

Mon, 01 Jan 0001 00:00:00 +0000

The complaint comes in as “Consul is slow.” DNS lookups lag, service discovery returns stale results, HTTP API writes take longer than usual. The cluster has a leader, gossip is healthy, no node is down. The Raft commit time metric is climbing. The root cause is often hiding in the catalog registration rate.

Every service registration and deregistration is a Raft write. The leader appends the entry to its log, persists it (fsync), replicates to followers (who also persist), waits for quorum, and applies the entry to the in-memory state store. This is the same pipeline that handles KV writes, health-check state updates, session creation, and ACL operations. When the registration rate is high enough, it saturates the pipeline and every consumer of the consensus layer pays the cost.

Consul serf queue backlog: an agent falling behind on gossip

Mon, 01 Jan 0001 00:00:00 +0000

A Consul agent whose consul.serf.queue.Event, consul.serf.queue.Intent, or consul.serf.queue.Query metric stays above zero is no longer keeping up with gossip. In steady state all three should read zero. Transient spikes during bulk joins, leaves, and rolling restarts are expected and self-drain within a few gossip intervals. The problem begins when the spike does not drain.

Serf’s queue depth is the observable proxy for the node’s internal health score. A sustained non-zero value means the agent is receiving gossip faster than its event loop can process. Downstream effects start subtle: failure detection latency rises, join and leave intents propagate slowly, and the node’s own probe replies arrive late at peers. Then the feedback loop engages.

Consul serfHealth check failing: the node-level check behind mass deregistration

Mon, 01 Jan 0001 00:00:00 +0000

Every Consul node carries an automatic health check called serfHealth. It is not user-defined, it cannot be removed, and it reflects the node’s membership in the Serf LAN gossip pool. When this check goes critical, Consul treats the entire node and every service registered on it as unhealthy, regardless of what individual service checks report.

This is the most commonly misdiagnosed failure in Consul. Operators see a service “go down” across all instances on a host, investigate the service checks, and find them still passing. The services are fine. The node’s gossip membership broke, and serfHealth cascaded that failure into every service on the node.

Consul server in failed state: reading consul members during an incident

Mon, 01 Jan 0001 00:00:00 +0000

You ran consul members during an incident and one of your servers shows failed. Before fixing anything, understand what the status means. Gossip failure and Raft failure are separate subsystems. A server in failed state is not answering gossip probes, but it may still be a voting Raft peer, or its agent process may still be running but starved of resources.

The distinction that matters during an incident: gossip health is not Raft health and is not agent health. A node can be alive in gossip yet useless for consensus. A node can show failed in gossip while its process is still running. These distinctions are the difference between a two-minute diagnosis and a misdiagnosis that makes the incident worse.

Consul server memory climbing toward OOM: heap, catalog, and snapshots

Mon, 01 Jan 0001 00:00:00 +0000

A Consul server’s RSS is climbing and OOMKill is close. The cause could be legitimate catalog growth, a goroutine leak, snapshot amplification, or a runtime pathology. Memory growth in Consul usually mixes real state growth, Go runtime behavior, and at least one subsystem leaking.

Go’s garbage collector does not return memory to the OS immediately. RSS of roughly 2x the live heap is normal and is not a leak. What you should investigate is monotonic RSS growth that survives a GC, or live heap (consul.runtime.alloc_bytes) that does not return to a baseline after a transient event.

Consul service has zero healthy instances: discovery returns nothing

Mon, 01 Jan 0001 00:00:00 +0000

A consumer calls /v1/health/service/<name>?passing=true and gets []. DNS lookups for <name>.service.consul return nothing. Every downstream consumer treats the service as gone. This is total unavailability for that one service, even if the rest of the Consul cluster is healthy.

Two consumer surfaces are affected at once. API consumers see the empty array directly. DNS consumers (port 8600) get a negative response, and clients may cache it. Recovery from zero means both fixing the checks and flushing the negative caches downstream.

Consul session invalidation: distributed locks releasing across the cluster

Mon, 01 Jan 0001 00:00:00 +0000

Sessions are Consul’s mechanism for binding distributed locks to node and application health. When a session is invalidated, every KV key it holds is acted on according to its Behavior. With the default release, the lock holder is cleared and the key’s ModifyIndex increments. With delete, the key is removed entirely. Every consumer watching that key reacts.

A spike in session invalidation surfaces as a cluster-wide release of distributed locks. Leader elections fire, caches flush, configuration reloads trigger, and any logic keyed off ModifyIndex change wakes up. The operator’s job is to identify which sessions invalidated, why, and whether the root cause is a single node, a renewal-path failure, or a gossip flap.

Consul slow disk causing Raft timeouts: the number-one cause of leader instability

Mon, 01 Jan 0001 00:00:00 +0000

Your Consul cluster is cycling through leaders. Writes fail with “no cluster leader” errors. The cluster technically has a leader at any given moment, but constant re-elections make writes effectively unavailable. The logs show [WARN] raft: heartbeat timeout reached, starting election repeating across all servers.

The root cause is almost always the same: slow disk I/O on the volume backing the Raft data directory. Consul’s Raft log store performs an fsync on every appended entry. When the disk cannot complete those syncs quickly enough, heartbeats and commits slow down. Followers miss their heartbeat window and start new elections. The new leader lands on the same slow storage. The cycle repeats.

Consul snapshot size growing: state bloat, slow restores, and commit spikes

Mon, 01 Jan 0001 00:00:00 +0000

Consul snapshots grow with the size of the FSM state: every registered service instance, every health check, every KV entry, every ACL token, every session. None of those individual writes looks expensive, so a team that registers a few new services per week will not see a spike in any single metric. What they get instead is a slow, monotonic climb in snapshot size that compounds across months until one day a snapshot save takes long enough to push Raft commit times past the heartbeat timeout, or a rejoined follower sits in snapshot restore long enough to fall out of the replication loop entirely.

Consul stale DNS queries: the agent is answering from cache

Mon, 01 Jan 0001 00:00:00 +0000

When consul.dns.stale_queries climbs, the Consul agent is doing what it was designed to do: keep answering DNS from local state when it cannot get a fresh answer from a server. That behavior has been the DNS default since Consul 0.7. But “stale” means two very different things depending on how stale, and why.

A stale answer two seconds old during a leader handoff is harmless availability. A stale answer five minutes old, still pointing at instances that crashed four minutes ago, is silently routing traffic to dead services. The counter alone cannot tell you which case you are in. Correlate it with agent-to-server RPC health, server health, and your consistency configuration.

Consul stale Raft peer: removing a failed server from the configuration

Mon, 01 Jan 0001 00:00:00 +0000

A Consul server has crashed, been decommissioned, or been replaced, but its entry still sits in the Raft configuration as a voting peer. Serf gossip already marks the node failed or left, yet GET /v1/operator/raft/configuration still lists it as a voter. The dead peer keeps counting toward quorum even though it will never cast another vote.

This is not always an emergency. A healthy cluster with autopilot enabled usually self-heals within tens of seconds. It becomes an emergency when autopilot is disabled, when the cleanup window has not elapsed, or when a second failure tips the cluster below the now-inflated quorum threshold.

Consul TLS not fully enforced: verify_incoming, verify_outgoing, and server hostname

Mon, 01 Jan 0001 00:00:00 +0000

You have CA, certificate, and key files configured for Consul. The agent starts without errors, health checks pass, services register, DNS resolves. But the cluster may still accept plaintext or unauthenticated connections because verify_incoming, verify_outgoing, or verify_server_hostname are missing, set to false, or placed in the wrong configuration stanza.

Certificate files (ca_file, cert_file, key_file) do not enforce TLS on their own. Consul requires explicit verification flags to reject plaintext connections and require client certificates. Without them, any network observer between agents and servers can read the catalog, KV values, and health check results, and any process that can reach the RPC port can impersonate a Consul agent.

Consul WAN federation down: a datacenter missing from the WAN pool

Mon, 01 Jan 0001 00:00:00 +0000

When an entire remote datacenter’s servers vanish from the WAN gossip pool, the symptom is usually downstream: cross-DC service lookups return empty results, prepared queries that failover across DCs stop working, and ACL or Connect CA replication between datacenters silently stalls. The Consul UI on the local DC may still look healthy because the local cluster is fine. The damage is in the federation layer.

The canonical diagnostic signal is consul members -wan. On any server in any federated DC, this should list every server in every federated DC. When a whole DC’s server rows are missing or stuck in failed state, WAN federation for that DC is broken. The equivalent API call is /v1/agent/members?wan=true.

Consul WAN link saturation: cross-DC latency triggering gossip suspicion

Mon, 01 Jan 0001 00:00:00 +0000

Remote datacenter servers start oscillating between alive and suspect in the WAN gossip pool. Cross-DC RPCs fail with “no path to datacenter” errors. ACL replication lag in secondary DCs climbs into seconds. Cross-DC prepared queries time out. LAN Consul signals look fine, but the federation is broken.

The usual cause is WAN link saturation. Consul’s WAN gossip pool uses a separate, more conservative set of timing defaults than LAN, but it is still vulnerable to high or spiking RTT. When the inter-DC link saturates, gossip probes time out, remote servers are marked suspect then failed, and WAN membership flaps. Each flap disrupts cross-DC RPC routing, which compounds the problem because RPC retries add traffic to the already-saturated link.

Consul xDS stream churn: Envoy sidecars running on stale configuration

Mon, 01 Jan 0001 00:00:00 +0000

Each Envoy sidecar in a Consul Connect mesh holds a long-lived gRPC xDS stream to a Consul server. That stream is the control plane: endpoint lists, intentions, and mTLS certificate rotations all flow across it. When the stream is healthy, the data plane converges within seconds of a catalog change. When it breaks or churns, the sidecar falls back to whatever Envoy cached last.

The dangerous part is not the disconnect. Envoy keeps its last-known-good configuration when the management server disappears. Traffic flows, health checks pass, and nothing in the data plane alarms. What you get is silent staleness: routes to endpoints removed minutes ago, certificates that have already rotated on the server, intentions that no longer match policy. The first visible symptom is usually a customer-facing incident, not a Consul alert.

How Consul actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

Most Consul incidents are debugged with the wrong mental model. Operators check consul members, see all nodes alive, conclude the cluster is healthy, and miss that Raft has no leader or that the catalog is stale by minutes. Consul runs three concurrent subsystems with different failure modes, signals, and consistency guarantees. Most monitoring collapses them into one “is Consul up?” check.

The three subsystems are Raft consensus (the brain), Serf gossip (the nervous system), and the catalog plus anti-entropy (the truth). With Connect enabled, a fourth layer adds a certificate authority, xDS configuration distribution, and intention enforcement. Each operates independently, and understanding that independence is the single most important thing an operator can learn.