Consul catalog poisoning: services registered from unknown nodes

A service instance appears in your catalog on a node you have never deployed. Its ServiceAddress points somewhere you do not control. That is catalog poisoning: a registration that did not come from your infrastructure.

Without strict ACLs, any process that can reach the Consul HTTP API can call the registration endpoints and add an instance for any service name. Consumers resolving discovery through Consul DNS or the catalog and health APIs will route a fraction of their traffic to that address. In a non-Connect deployment there is nothing in the data path that verifies the instance is what it claims to be.

The durable fix is service-registration ACLs. Everything else is detection, scoping, and cleanup.

What this means

Two registration paths matter, with different ACL requirements and persistence characteristics.

The catalog API, PUT /v1/catalog/register, writes directly to the authoritative catalog held in Raft. Per the Consul API docs it requires both node:write and service:write on the presenting token. Because it bypasses the agent, a registration made this way has no corresponding local state on any agent. The anti-entropy sync on that node’s agent may later remove it, because the agent has no record of the service. That makes some catalog-API poisoning self-healing when the named node has a healthy agent actively syncing. A registration pinned to a fabricated node name can persist indefinitely.

The agent API, PUT /v1/agent/service/register, registers through the local agent and requires only service:write. The agent then propagates the registration to the catalog via anti-entropy. This is the path legitimate workloads use. A poisoned registration via this path means an attacker ran a process on a node that already held, or could obtain, a token with service:write.

Blast radius depends on how consumers resolve the service:

• Consul DNS (<service>.service.consul) returns whatever the catalog says, including poisoned addresses, with no identity check.
• HTTP catalog and health queries (/v1/catalog/service/<name>, /v1/health/service/<name>) return the same.
• Consul Connect, the service mesh, is different. Connect issues SPIFFE-based mTLS leaf certificates and verifies service identity at handshake. A poisoned ServiceAddress that an attacker controls cannot present a valid leaf for the claimed identity, so mTLS handshakes to it fail. Plain TCP/HTTP discovery has no such backstop and is fully exposed.

Common causes

Quick checks

All read-only. Replace <name> with the suspect service and <management-token> with your bootstrap or management token.

# List every instance; inspect Node, Address, ServiceAddress, ServicePort
curl -s http://127.0.0.1:8500/v1/catalog/service/<name> | \
  python3 -c "import sys,json
for s in json.load(sys.stdin):
    print(s['Node'], s['Address'], s.get('ServiceAddress',''), s.get('ServicePort',''))"

# Cross-check: every catalog node should be a known gossip member
curl -s http://127.0.0.1:8500/v1/catalog/nodes | python3 -c "import sys,json
[print(n['Node'], n['Address']) for n in json.load(sys.stdin)]"
curl -s http://127.0.0.1:8500/v1/agent/members | python3 -c "import sys,json
[print(m['Name'], m['Addr']) for m in json.load(sys.stdin)]"

# Is ACL even enabled?
consul info | grep -i acl

# Read the anonymous token policy (reserved accessor ID)
curl -s -H "X-Consul-Token: <management-token>" \
  http://127.0.0.1:8500/v1/acl/token/00000000-0000-0000-0000-000000000002

# Confirm whether ACL filtering hid results from you
curl -s -D - http://127.0.0.1:8500/v1/catalog/service/<name> -o /dev/null | \
  grep -i 'X-Consul-Results-Filtered-By-ACLs'

# Enterprise audit log: who registered what (path varies by audit.sink.file.path)
grep -i catalog /var/log/consul/audit.log | tail -50

How to diagnose it

1. Confirm the instance is truly unknown. Compare its Node against your infrastructure inventory and gossip membership. A node in the catalog but missing from /v1/agent/members signals a catalog-API write to a fabricated node.
2. Determine the registration path. If the node exists and runs a healthy agent but has no local record of the service, the registration was a direct catalog write that anti-entropy has not yet reconciled. Query the agent directly: curl http://<node>:8500/v1/agent/services.
3. Identify the token. With audit logging (Enterprise), search the registration event, extract the token AccessorID, then resolve that token’s policies. Without audit logs, infer from network reach: which hosts can reach port 8500 and hold a token carrying service:write or node:write.
4. Check the anonymous token. If ACLs are enabled but the anonymous token carries service:write or node:write, any unauthenticated process on the network can register services. This is the most common root cause.
5. Verify persistence. Re-query the service after two or three anti-entropy intervals. If the poisoned instance disappears, it was a catalog-only write that anti-entropy cleaned up. If it persists, it is backed by an agent process or pinned to a non-syncing node.
6. Assess blast radius. Resolve the service through DNS and through the health API from a consumer’s perspective. Identify which consumers are non-Connect and therefore exposed, versus Connect consumers protected by mTLS identity checks.

flowchart TD
  A["Unknown instance in catalog"] --> B{"Node in gossip?"}
  B -- No --> C["Catalog-API write to fabricated node"]
  B -- Yes --> D{"Agent has local service?"}
  D -- No --> E["Catalog-API write, anti-entropy may remove"]
  D -- Yes --> F["Agent-backed registration on live host"]
  C --> G["Check anonymous token policy"]
  E --> G
  F --> H["Identify token on that host"]
  G --> I["Deregister + lock down ACLs"]
  H --> I

Metrics and signals to monitor

Fixes

Immediate containment: remove the poisoned instance

Warning: this removes a service instance from the catalog and disrupts traffic routing. Only run this after confirming the instance is poisoned, not merely unexpected.

# Remove a specific service instance from the catalog
curl -s -X PUT -H "X-Consul-Token: <management-token>" \
  -d '{"Datacenter":"dc1","Node":"unknown-node","ServiceID":"poisoned-svc"}' \
  http://127.0.0.1:8500/v1/catalog/deregister

If the registration was agent-backed on a live host, you must also stop the registering process or it will re-register on the next anti-entropy sync. If it was a catalog-only write, deregister is durable. Either way, monitor the service for reappearance after two or three sync intervals.

Revoke or rotate the compromised token

Once you identify the token, revoke it. Revoking a token in active use by legitimate services causes an outage. If you are unsure which workloads depend on it, rotate to a new scoped token first, update the consumers, then revoke the old one.

Lock down the anonymous token

Ensure the anonymous token has no service:write and no node:write. Typically it should carry read access only, and only for the scope you need for UI visibility or unauthenticated DNS. Any write permission on the anonymous token is an open door.

Enforce service-registration ACLs (the durable fix)

• Set acl.default_policy = "deny" so unauthenticated requests get nothing by default.
• Issue per-service tokens with service:write scoped to the specific service name only, not *.
• Scope the agent token’s node:write to the agent’s own node name so an agent cannot register services on behalf of arbitrary nodes.

Restrict API exposure

• Enforce TLS with verify_incoming and verify_outgoing so a stolen plaintext token off the wire is not trivially possible.
• Firewall ports 8500 (HTTP API) and 8300 (server RPC) to known agents and operators only. Network reach is the first ACL.

Patch known bypasses

CVE-2021-38698 allowed a service holding service:write on any service to register proxies for other services via the transaction endpoint, enabling traffic interception. It was fixed in 1.8.15, 1.9.9, and 1.10.2. If you run an older release, treat an upgrade as part of the fix, not optional.

Prevention

• Prefer agent-based registration over direct catalog writes. The agent path requires only service:write and leaves a local record that makes auditing and cleanup easier.
• Scope every token to the minimum service or node name. Avoid service:write on * outside of bootstrap.
• Enable audit logging where available and alert on catalog write events whose source node is not in your expected fleet.
• Track consul.catalog.register rate and per-service instance counts as baseline signals, so a single unexpected registration is visible against normal churn.
• Keep Consul patched. Catalog and ACL semantics change between versions; running an old release leaves known bypasses open.

How Netdata helps

• Per-second consul.catalog.register rates expose a poisoning burst against normal registration baseline without waiting for minute-level rollup.
• ML anomaly detection on per-service instance counts flags a new instance on an unrecognized node faster than static thresholds.
• Correlating registration spikes with HTTP 403 and ACL resolution error rates helps distinguish an active attack from a legitimate deploy.
• Catalog nodes vs gossip membership divergence provides the cross-check needed to confirm a fabricated node rather than a slow inventory sync.

Related guides

• Consul “ACL not found”: requests rejected after a token or policy change
• Consul ACL resolution latency: token cache thrashing on every request
• Consul blocking query accumulation: leaked watches that pile up goroutines
• Consul catalog bloat: too many services and checks slowing everything down
• Consul registration storm: catalog churn overwhelming Raft
• Consul anti-entropy not syncing: local agent state and the catalog drifting apart
• Consul client rpc failed: agents alive but the catalog is going stale
• Consul Connect CA rotation failure: a root roll that never finished
• Consul Connect certificate expired: mTLS handshakes failing across the mesh
• Consul cross-datacenter query failure: prepared-query failover masking a DC outage
• Consul DeregisterCriticalServiceAfter: instances vanishing from the catalog
• Consul DNS latency high: slow lookups stalling connections and failovers