Consul Permission denied (403): authorized token, missing permission

A 403 “Permission denied” from the Consul HTTP API means the request reached the server with a token it recognizes, but policy evaluation denied the operation. The token exists, resolves, and is not expired. It lacks the rule covering the resource being touched: a service, a KV prefix, a node, an operator endpoint. This differs from “ACL not found”, where the SecretID is unknown to the server, and the two need different responses.

The error message names the missing permission, for example “anonymous token lacks permission ‘agent:read’ on ‘consul-test-02’”. That string is the fastest path to a fix: it tells you which capability on which resource the policy is missing. A 403 spike that correlates with an ACL policy change is almost always misconfiguration. A spike with no corresponding change points at token revocation, expiry, or external probing.

The most common trap is the anonymous token. When a request arrives without an explicit token and no per-agent default token is configured, Consul falls back to the anonymous token, which by default has no policies attached. With default_policy = "deny", that produces 403s for DNS lookups, health queries, and UI access that operators expected to work without configuration.

What this means

A 403 is the final step of a three-stage authorization pipeline: token resolution, policy lookup, rule evaluation. Each stage has a distinct failure signature:

• Token resolution fails (token unknown): “ACL not found”. The SecretID is not in the local ACL store. Causes: typo, revocation, replication lag in a secondary DC, or a new token that has not propagated yet.
• Token resolves but no linked policy grants the capability: “Permission denied” with the specific missing permission. The fix is in the policy, not the token.
• Token resolves and policy allows: 200 or success.

For a 403, do not rotate the token first. The token is fine. The policy attached to it (or the absence of one) is the problem.

flowchart TD
    A["HTTP 403 Permission denied"] --> B{"Error names the
anonymous token?"}
    B -- Yes --> C["Unauthenticated request
fell through to anonymous"]
    C --> D{"Is acl.tokens.default
set on the agent?"}
    D -- No --> E["Set a default token
with read scope"]
    D -- Yes --> F["Widen anonymous policy
or fix default token scope"]
    B -- No --> G{"Is the SecretID known
on this server?"}
    G -- No --> H["ACL not found:
replication lag or revoked"]
    G -- Yes --> I["Policy lacks the rule
for this capability and resource"]
    I --> J["Add a scoped policy rule"]

Common causes

Quick checks

These are read-only and do not modify ACL state.

# Count recent permission-denied vs ACL-not-found events
journalctl -u consul --since '1 hour ago' | grep -ci 'permission denied'
journalctl -u consul --since '1 hour ago' | grep -ci 'ACL not found'

# Pull the specific missing permission from the most recent 403
journalctl -u consul --since '1 hour ago' | grep -i 'permission denied' | tail -20

# ACL resolution latency and error counters (metric naming varies by version)
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -i 'acl'

# Replication status (run on a secondary DC server)
curl -s http://127.0.0.1:8500/v1/acl/replication | jq .

# Inspect the token an app is using (replace with the real accessor ID)
consul acl token read --accessor-id <accessor-id>

# Inspect a named policy and its rules
consul acl policy read -name <policy-name>

# List policies attached to the anonymous token
consul acl token read --id anonymous

The last command uses the special CLI keyword anonymous as the token ID. The anonymous token’s AccessorID and SecretID are both 00000000-0000-0000-0000-000000000002.

How to diagnose it

1. Extract the missing permission from the log line. The 403 message names the capability and resource. Capture it verbatim before changing anything.
2. Identify which token made the request. If the app logs its token, use it. If not, check the request source IP against known workloads, and check whether the error mentions the anonymous token.
3. Read the token. consul acl token read --accessor-id <id> shows linked policies, roles, and the Local flag. Confirm the token is the one you expect and not a stale or rotated one.
4. Read each linked policy. consul acl policy read -name <name> prints the rules. Verify the rule covers the resource prefix and the required verb (read, write, list, deny).
5. Check the anonymous token path. If the error references the anonymous token, confirm whether acl.tokens.default is set on the agent. With no default token, unauthenticated requests fall through to anonymous.
6. Check replication if in a secondary DC. curl /v1/acl/replication shows enabled status and lag. A 403 that only appears in the secondary DC and clears after a few seconds is replication lag, not a policy bug.
7. Check the Consul version. Behavior changes (see Fixes) can turn a previously-allowed request into a 403 with no policy change.

Metrics and signals to monitor

Metric names vary across Consul versions (dot-separated vs underscore-separated). Grep broadly when first instrumenting.

Fixes

The commands below modify ACL state. Verify policy HCL contents before applying, and test in a non-production environment if possible.

Add the missing policy rule

If the token is correct but the policy is incomplete, add the rule that covers the resource and verb. The error message tells you exactly what is missing. Prefer scoped rules over broad ones to limit blast radius.

# Write the policy HCL to a file, then create or update
consul acl policy create -name billing-write -rules @billing-write.hcl
# Attach to the token
consul acl token update -accessor-id <accessor-id> -policy-name billing-write

A scoped rule looks like service_prefix "billing-" { policy = "write" } rather than the wildcard service_prefix "" { policy = "write" }.

Fix the anonymous token

Two failure modes, opposite fixes:

• Too narrow (breaks unauthenticated DNS/health/UI): Either attach a minimal read policy to the anonymous token, or set acl.tokens.default on the agent to a token with the needed read permissions.
• Too broad (security risk): Remove broad policies from the anonymous token. Any non-empty policy on the anonymous token is worth auditing.

Resolve replication lag in a secondary DC

If 403s appear only in a secondary DC and clear after a few seconds, the cause is ACL replication lag, not a policy bug. Check /v1/acl/replication on the secondary DC. If replication is broken, verify the replication token is valid and has the required ACL read permissions. During a complete replication outage, acl_down_policy controls what happens when the ACL subsystem is unavailable.

Migrate off the token query parameter

The ?token= query parameter is deprecated, with WARN-level logs emitted in recent Consul versions. Move callers to the X-Consul-Token header or the HTTP Bearer scheme. Tokens in URLs leak into access logs, proxy logs, and browser history.

Handle version-specific behavior changes

Consul’s ACL enforcement has changed across versions in ways that produce new 403s after upgrade:

• Health endpoints: /v1/health/connect/ and /v1/health/ingress/ return 403 when the caller lacks service:read where older versions returned an empty list with a success status.
• Legacy ACL system removal: Config fields renamed: master to initial_management, agent_master to agent_recovery.
• Templated policies: During rolling upgrades, nodes that predate templated policy support do not recognize the field, so tokens may lack expected permissions on older nodes until the roll completes.

If a 403 storm starts immediately after an upgrade, check the version-specific upgrade notes before editing policies.

Fix the Terraform remote state lock

Terraform’s Consul backend needs session:write on the lock prefix in addition to key:write. A 403 on consul kv put for the lock key is the classic symptom. Add the session capability to the policy used by the Terraform token.

Prevention

• Treat the error message as the source of truth. It names the missing capability and resource. Build runbooks around parsing it rather than guessing.
• Scope policies narrowly. Prefer prefix-scoped rules over wildcard rules. Broad tokens are both a security risk and a source of confusing 403s when a caller hits an unexpected path.
• Set acl.tokens.default explicitly on every agent. Do not rely on the anonymous token for legitimate unauthenticated traffic.
• Monitor 403 rate alongside policy changes. A spike that correlates with a policy change is misconfiguration. A spike without one is revocation, expiry, or probing.
• Track token inventory growth. Steady growth without corresponding service growth indicates token leak from CI/CD or decommissioned workloads.
• Alert on ACL replication lag in every secondary DC. Authorization in a secondary DC is only as good as the freshest replicated policy.
• Pin your upgrade runbook to version-specific ACL notes. Health endpoint enforcement changes and the token query parameter deprecation have both produced surprise 403 storms.

How Netdata helps

• Correlate 403 rate with deploys and policy edits. A sudden 403 spike next to a deploy or config change pinpoints misconfiguration versus external probing.
• Track ACL resolution latency per server. Resolution is on the hot path of every authenticated request; rising latency indicates cache pressure or an undersized token cache.
• Surface ACL replication lag in secondary DCs. Per-second visibility on replication status separates a transient lag spike from a broken replication stream.
• Flag token creation rate anomalies. Anomaly detection on token upsert counters catches automation drift and unexpected provisioning outside known deploy windows.
• Correlate anonymous token usage with unexpected endpoints. Spikes in anonymous-token traffic to authenticated endpoints indicate a probe or a misconfigured caller.

Related guides

• Consul blocking query accumulation: leaked watches that pile up goroutines
• Consul catalog bloat: too many services and checks slowing everything down
• Consul registration storm: catalog churn overwhelming Raft
• Consul anti-entropy not syncing: local agent state and the catalog drifting apart
• Consul client rpc failed: agents alive but the catalog is going stale
• Consul Connect CA rotation failure: a root roll that never finished
• Consul Connect certificate expired: mTLS handshakes failing across the mesh
• Consul cross-datacenter query failure: prepared-query failover masking a DC outage
• Consul DeregisterCriticalServiceAfter: instances vanishing from the catalog
• Consul DNS latency high: slow lookups stalling connections and failovers
• Consul DNS SERVFAIL: service discovery is broken for your applications
• Consul stale DNS queries: the agent is answering from cache