Consul "ACL not found": requests rejected after a token or policy change

“ACL not found” appears as HTTP 403 responses carrying “ACL not found” or “token does not exist: ACL not found”. It surfaces in agent and server logs, sidecar injector output, and as failed service registrations, health check updates, or KV writes. The blast radius depends on which token is missing: a single application token breaks one service; a replication or agent token can break an entire datacenter’s authorization pipeline.

The message is frequently misread. “ACL not found” does not mean the token has the wrong permissions. It means the Consul server receiving the request has no record of the token’s SecretID. The token was deleted, never created in this datacenter, or has not yet arrived via ACL replication. The fix path is completely different from “Permission denied”, which indicates the token is known but lacks a specific right.

The most common trigger is a credential rotation, a policy migration, or a failover in a federated setup where ACL replication is asynchronous and rate-limited.

What this means

Consul resolves every authenticated API request by looking up the presented token in the ACL subsystem. When the local server has no record of the SecretID, the request is rejected before policy evaluation runs. A token can be valid in the primary datacenter and still produce “ACL not found” in a secondary that has not yet replicated it.

The authoritative distinction:

• “ACL not found” - the token does not exist in this server’s view of the ACL store. The SecretID is unknown.
• “Permission denied” - the token exists, but its attached policies, roles, or service identities do not grant the requested operation.

Mixing these wastes time. Re-issuing policies or expanding token scope does nothing when the token was destroyed, never replicated, or replaced during a rotation and some client still holds the old SecretID.

The error string is identical whether the ACL system is healthy and the token is genuinely gone, or whether the ACL system was never bootstrapped. A cluster where ACLs were never initialized will reject every token-bearing request with “ACL not found” because there is no token store to consult.

flowchart TD
    A[Request with X-Consul-Token] --> B{Token in local ACL store?}
    B -- no --> C{ACL system bootstrapped?}
    C -- no --> D["ACL not found: ACL system not initialized"]
    C -- yes --> E{Replication lag?}
    E -- lagging --> F["ACL not found: token not yet replicated"]
    E -- up to date --> G["ACL not found: token deleted or never created"]
    B -- yes --> H{Policies grant operation?}
    H -- no --> I["Permission denied"]
    H -- yes --> J[Request allowed]

Common causes

Quick checks

Run these read-only checks from a Consul server with a privileged token. None mutate state.

# Confirm a leader exists; ACL writes need a leader
curl -s http://127.0.0.1:8500/v1/status/leader

# Replication status in a secondary DC
# Key fields: Enabled, ReplicatedIndex, ReplicatedTokenIndex, LastSuccess, LastError
curl -s http://127.0.0.1:8500/v1/acl/replication | jq .

# Read a specific token by accessor ID
consul acl token read --accessor-id <accessor-id>

# Search the token list for a known SecretID (requires listing all tokens)
consul acl token list -format json | jq '.[] | select(.SecretID == "<secret-id>")'

# ACL resolution latency and token upsert counters
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -E 'acl.resolveToken|acl.token'

# Count recent ACL-not-found vs permission-denied events
journalctl -u consul --since '1 hour ago' | grep -c 'ACL not found'
journalctl -u consul --since '1 hour ago' | grep -c 'Permission denied'

# Confirm the ACL system is bootstrapped (token count should be non-zero)
consul acl token list -format json | jq 'length'

If /v1/acl/replication returns Enabled: false on a secondary DC that should be replicating, replication is off. It must be explicitly enabled in the config and the replication token must be valid.

How to diagnose it

1. Classify the failure scope. A single token failing points to rotation or deletion. Many distinct tokens failing points to replication or bootstrap. Every token failing points to an ACL subsystem that is down or not bootstrapped.

2. Confirm the token exists in the primary DC. Run consul acl token read --accessor-id <id> from a primary server. If the token is gone, the root cause is upstream: find what deleted it and reissue. If the token is present, the issue is propagation or local resolution.

3. Check replication status in every secondary DC. Compare ReplicatedTokenIndex against the primary’s latest token index. Recommended thresholds: under 1s healthy, 1-5s degraded, over 5s critical, over 30s page-worthy. Replication is rate-limited, so a token created in the primary may not be visible downstream for several seconds under load. LastError populated or LastSuccess stale means the stream is broken, not merely slow.

4. Distinguish rotation aftermath from an attack. Pull access or audit logs and group the failing SecretIDs. A rotation aftermath shows a small set of recently-rotated tokens failing from known service identities. An unauthenticated flood shows many distinct unknown SecretIDs from a small set of source IPs, often hitting endpoints that should require auth.

5. Check the replication token. In a secondary DC, ACL replication runs under a specific token. If that token was deleted, expired, or had its policy narrowed, replication silently stops advancing. Verify the token exists in the primary and still has sufficient ACL scope.

6. Correlate with leader changes. ACL replication depends on a healthy primary. A leader election in the primary during a token creation burst can stall replication. Cross-reference consul.raft.state.leader transitions against the spike in “ACL not found” errors.

7. Check acl_down_policy. With the default extend-cache, secondaries can still resolve cached tokens when the authoritative source is unreachable. If someone changed this under pressure, behavior shifts immediately. Confirm the configured value before assuming replication is the only path.

Metrics and signals to monitor

Fixes

Token deleted or replaced during rotation

Reissue the token and update every client that still holds the old SecretID. Find all the places the old token lives: environment variables, Kubernetes secrets, Vault agent templates, consul-template configs, sidecar injector annotations. Until every consumer is updated, requests with the stale SecretID will keep failing.

If you cannot update consumers immediately, you can re-create a token with the same SecretID via the HTTP API by specifying the SecretID field on creation. This is a stopgap, not a strategy. Rotate properly afterward.

ACL replication lag in a secondary DC

If replication is enabled and the replication token is valid, lag usually resolves on its own as the rate limiter catches up. Do not restart servers to fix lag. Restarts reset caches and can make the problem worse during the warmup window.

If replication is stuck, verify in order: WAN connectivity on the gossip and RPC ports, the replication token exists in the primary with sufficient scope, the primary has a stable leader, and the primary is not saturated by a token creation burst.

Replication token expired or lost permissions

Rotate the replication token in the primary, then update the secondary’s configuration with the new token and reload. Until the new token is in place, the secondary cannot pull ACL changes. Plan this outside an incident if possible; doing it under fire means every new token created in the primary is invisible downstream until you finish.

ACL system not bootstrapped

Run consul acl bootstrap in the affected DC. This produces the initial management token. Until bootstrap completes, every token-bearing request returns “ACL not found” because there is no token store. This is most common immediately after enabling ACLs on an existing cluster or after a disaster recovery restore.

Unauthenticated flood

If the spike is external noise rather than a real rotation, the fix is network-level: restrict API access, rotate any exposed tokens, and verify the anonymous token has only the intended minimal permissions. Do not confuse this with a replication problem, or you will chase replication metrics while a scanning client is the actual source.

Nomad workload identity race

If you are running Consul 1.19.x or later with Nomad workload identities, deregistration may use a token Consul has not yet registered. The fix is on the Nomad side: upgrade to a version where deregistration is owned by the Nomad client rather than the workload.

Prevention

• Treat token rotation as a multi-DC event. Plan for the replication lag window. Stage rotations so the primary creates the new token, replication catches up, and only then do consumers switch.
• Monitor ACL replication lag in every secondary DC. “Works in primary” is not a sufficient health check. Treat lag above 5s as a ticket and above 30s as page-worthy.
• Alert on the 403 split. Track “ACL not found” and “Permission denied” as separate counters. A spike in one and not the other tells you immediately which failure mode you are in.
• Lock down the replication token scope. Give it the minimum it needs. Anything narrower breaks replication silently; anything broader is unnecessary exposure.
• Document acl_down_policy decisions. The default extend-cache buys time during primary unreachability. Changing it under pressure without understanding the tradeoff causes secondary outages.
• Version-pin and test Consul and Nomad together. The workload identity race is a cross-product issue. Catch it in staging before it surfaces as “ACL not found” in production.

How Netdata helps

• Per-second ACL resolution latency exposes cache thrash and policy-complexity spikes as they happen, instead of smoothed-over aggregates that hide the burst.
• Replication lag charts per secondary DC alongside WAN gossip health and Raft leadership let you distinguish replication lag from connectivity loss without switching tools.
• Split 403 counters on a single dashboard separate “ACL not found” from “Permission denied”, which is the single most useful distinction when triaging this error.
• Raft leader transition annotations overlaid on ACL error spikes show whether a primary leadership change caused the replication stall.
• Token upsert rate next to replication lag reveals whether a credential-rotation burst is the upstream cause of downstream authorization failures.

Related guides

• Consul client rpc failed: agents alive but the catalog is going stale
• Consul anti-entropy not syncing: local agent state and the catalog drifting apart
• Consul Connect certificate expired: mTLS handshakes failing across the mesh
• Consul Connect CA rotation failure: a root roll that never finished
• Consul on EBS: burst-credit exhaustion and the sudden latency cliff