Consul serf queue backlog: an agent falling behind on gossip

A Consul agent whose consul.serf.queue.Event, consul.serf.queue.Intent, or consul.serf.queue.Query metric stays above zero is no longer keeping up with gossip. In steady state all three should read zero. Transient spikes during bulk joins, leaves, and rolling restarts are expected and self-drain within a few gossip intervals. The problem begins when the spike does not drain.

Serf’s queue depth is the observable proxy for the node’s internal health score. A sustained non-zero value means the agent is receiving gossip faster than its event loop can process. Downstream effects start subtle: failure detection latency rises, join and leave intents propagate slowly, and the node’s own probe replies arrive late at peers. Then the feedback loop engages.

The slow node gets marked suspect and then failed by peers that no longer hear from it within the probe window. Each state transition generates fresh gossip: suspicion, failure, and later refute messages when the node recovers and re-announces itself. That additional gossip lands back on the overloaded node’s queue, deepening the backlog. If the affected agent is a server, the cost compounds: CPU cycles spent draining gossip are cycles not available for Raft heartbeats and FSM applies.

What this means

Consul’s Serf layer maintains three named queues that buffer inbound gossip before the agent’s event loop processes it: Event for user events and member transitions, Intent for join and leave intents, and Query for serf queries. Beneath those, memberlist maintains its own broadcast queue exposed as consul.memberlist.queue.broadcasts. A backlog in any layer delays everything that depends on timely membership convergence.

The healthy baseline is zero across all queues, all the time. When you see non-zero values, ask two questions: is it transient, and is it growing? A spike that returns to zero within 30 to 60 seconds during a known event is the protocol working as designed. A value that plateaus above zero or climbs monotonically is the protocol losing ground.

Sustained backlog is self-reinforcing. A node that falls behind cannot answer probes within the configured timeout. Peers suspect it, and suspicion and failure transitions are themselves gossip messages distributed to every member. Lifeguard (the Serf health-score mechanism) dampens this by letting an overloaded node advertise a degraded health score so peers back off, but it mitigates rather than eliminates the loop. Encryption amplifies the cost: every gossip message is encrypted on send and decrypted on receive, so a CPU-starved node pays the crypto tax on backlog it cannot drain.

flowchart TD
    A[Agent receives gossip faster than it processes] --> B[serf.queue.Event / .Intent / .Query above 0]
    B --> C[Probe replies arrive late or miss timeout]
    C --> D[Peers mark node suspect then failed]
    D --> E[Transitions generate more gossip]
    E --> A
    B --> F[Node serfHealth check goes critical]
    F --> E

Common causes

Quick checks

Read-only and safe to run on any agent.

# Current serf queue depths (look for sustained non-zero)
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -E 'serf.queue'

# Memberlist broadcast queue (the layer beneath serf)
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -E 'memberlist.queue'

# Member status from this node's gossip view
curl -s http://127.0.0.1:8500/v1/agent/members | python3 -c "
import sys,json
for m in json.load(sys.stdin):
    print(m['Name'], m['Status'])
"

# Agent CPU pressure proxies
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -E 'runtime.gc_pause|runtime.num_goroutines'

# Serf debug block from consul info
consul info | grep -A 20 serf_lan

# Gossip encryption state (all nodes should report the same key)
consul keyring -list

# UDP receive errors on the gossip interface
netstat -s | grep -iE 'receive.*error|packet receive'

Some telemetry exporters (Prometheus format) replace dots with underscores in metric names, for example consul_serf_queue_Event. Both forms refer to the same underlying value.

How to diagnose it

1. Confirm the backlog is sustained, not transient. Sample the queue metrics every few seconds for at least a minute. If values return to zero between samples, the cause was a bounded event. Move to prevention rather than remediation.

2. Determine scope. Is this one node, one rack, one datacenter, or every agent? Query the metrics endpoint across a representative sample. A single affected node points to local CPU, network, or disk. A cluster-wide pattern points to gossip parameter tuning or cluster size.

3. Correlate with CPU. Pull consul.runtime.gc_pause_ns and OS-level CPU for the consul process on affected nodes. Gossip processing is CPU-bound. If the node is at or near its CPU limit, the backlog is a symptom of starvation, not a gossip problem.

4. Check member status divergence. Compare consul members output from several nodes. If the affected node appears failed or suspect on peers while reporting itself alive, the feedback loop is active. The backlog is now generating its own load.

5. Verify encryption state. Run consul keyring -list. If nodes report different key counts or a rotation is mid-flight, some nodes are encrypting or decrypting against keys others have discarded, adding dropped-message overhead.

6. Compare agent count to the gossip pool guidance. HashiCorp’s scale documentation recommends a maximum of 5,000 client agents per gossip pool. Near or above that ceiling with default parameters, the backlog reflects protocol saturation rather than any single node’s health.

7. Inspect gossip parameters. Defaults for LAN are gossip_interval 200ms, probe_interval 1s, retransmit_mult 4, suspicion_mult 4. If your cluster has grown without revisiting these values, the parameter set may no longer match the workload.

Metrics and signals to monitor

There is no officially documented numeric threshold for any serf queue metric. HashiCorp’s guidance is qualitative: consistently high values indicate the gossip pool cannot keep up with churn. Treat any sustained non-zero value as the alert condition and let duration and trend determine severity.

Fixes

The right fix depends on whether the backlog is local to one node or systemic. Do not restart the consul process as a first response. A restart clears the queue but does not address the cause, and the backlog returns as soon as the node rejoins and receives catch-up gossip.

CPU starvation on a single node

If the backlog correlates with high CPU on one agent, the gossip layer is competing with something else on the host. Common offenders: script-based health checks spawning subprocesses, GC pauses on a large heap, or CPU limits in a container scheduler.

• Raise the CPU limit or move the agent to a less contended host.
• Audit health check scripts for subprocess cost. A check that shells out to curl or jq on a tight interval multiplies fast.
• If GC pauses are the driver (consul.runtime.gc_pause_ns spiking), raise GOGC to reduce GC frequency at the cost of higher peak heap, or reduce heap pressure by trimming catalog bloat.

Cluster too large for current gossip parameters

If every agent shows elevated queues during normal operation, the protocol is saturated. The 5,000-agent-per-pool ceiling is the documented soft limit for default parameters.

• Increase gossip_interval to spread message emission over more time. This trades convergence speed for processing headroom.
• Increase gossip_nodes to raise the fan-out per round, reducing total rounds needed at the cost of larger packets.
• Adjust suspicion_mult to give slow nodes more time before peers declare them failed, dampening the feedback loop.
• For clusters well above 5,000 agents, consider network segments or Consul on Kubernetes, which reduces the need for a client agent on every node and shrinks the gossip pool.

Test any gossip parameter change on a staging cluster first. These values affect failure detection latency cluster-wide.

Encryption overhead

Gossip encryption is enabled with the encrypt config field. Every message is encrypted on send and decrypted on receive. On CPU-constrained nodes, the crypto cost competes directly with gossip processing.

• Ensure encrypted nodes have CPU headroom. A node sized for plaintext gossip may need more CPU once encryption is on.
• Verify all nodes share the same key with consul keyring -list. A mid-rotation state where some nodes hold old and new keys adds processing overhead on every message.
• Complete key rotations promptly. Lingering dual-key states increase per-message work.

Mass recovery storms

If the backlog follows an AZ recovery, partition heal, or mass rolling restart, the cause is legitimate catch-up load. The cluster is processing a bounded burst of join events and anti-entropy syncs.

• Monitor Raft commit time during the storm. If commit time stays below the election timeout, the cluster will self-recover. Intervention risks making it worse.
• If commit time approaches the election timeout, reduce write load temporarily: disable non-critical health checks or stagger the remaining rejoining nodes.
• Do not tune gossip parameters reactively during a storm. Changes made under load are hard to validate and may persist after the storm drains.

Prevention

• Alert on sustained non-zero serf queue depth. A 60-second window above zero outside known maintenance windows is a reliable early signal.
• Track agent count against the 5,000-per-pool guidance as a capacity-planning metric, not just an incident trigger.
• Include gossip parameters in your cluster documentation. If you have grown past the size where defaults were chosen, schedule a tuning review.
• Monitor consul.runtime.gc_pause_ns and CPU on every agent, not just servers. Client agents run gossip too.
• Periodically verify gossip key state with consul keyring -list, especially after any rotation.
• Treat the feedback loop as a design constraint. If your failure detection is tight enough that a slow node gets marked failed before it can recover, your suspicion_mult may be too aggressive for your infrastructure’s worst-case CPU latency.

How Netdata helps

• Per-second collection of consul.serf.queue.Event, .Intent, and .Query exposes sustained backlog that minute-granular scraping misses. A queue that spikes and drains between samples is invisible at lower resolution.
• ML anomaly detection flags the transition from transient spike to sustained elevation without requiring a hand-tuned threshold, which matters because HashiCorp publishes no official numeric threshold for these metrics.
• Correlating serf queue depth with OS CPU, GC pause duration, and goroutine count on the same timeline isolates CPU starvation from protocol saturation in a single view.
• Member status and serfHealth signals sit alongside the queue metrics, so the feedback loop (queue grows, node marked suspect, more gossip generated) is visible as a coordinated pattern rather than isolated alerts.
• The memberlist broadcast queue metric, when exposed, appears in the same dashboard, letting you see whether the backlog originates at the serf layer or beneath it.

Related guides

• How Consul actually works in production: a mental model for operators
• Consul leader election storm: repeated elections and rolling write outages
• Consul monitoring checklist: the signals every production cluster needs
• Consul monitoring maturity model: from survival to expert
• Consul “No cluster leader”: every write is failing
• Consul raft commitTime high: the write pipeline is slowing down
• Consul raft lastContact rising: followers drifting toward an election
• Consul Raft log divergence: catching a corrupt follower before it wins an election
• Consul lost quorum: Raft peers below the majority needed to elect a leader
• Consul stale Raft peer: removing a failed server from the configuration
• Consul leader stable but commits stalled: writes silently failing