Consul ACL resolution latency: token cache thrashing on every request

Every authenticated Consul request pays an ACL resolution tax. When that tax is sub-millisecond, nobody notices. When the token cache cannot hold the working set, every request becomes a miss and that tax multiplies across DNS lookups, HTTP API calls, RPC forwarding, and Connect intention evaluation. The cluster keeps answering, but slowly, and the slowness is uniform across every authenticated path.

The primary signal is consul.acl.ResolveToken (a summary timer, milliseconds). Healthy cached resolution is sub-millisecond; a cold miss against the authoritative datacenter costs low single-digit milliseconds. Sustained values above ~10ms are TICKET-worthy: ACL resolution sits on the hot path of every authenticated operation. It surfaces as elevated DNS latency, slow HTTP API responses, and stretched Connect sidecar handshakes long before any single subsystem fails.

Consul keeps resolved ACL tokens in a fixed-size cache keyed on the token itself. When the number of unique active tokens exceeds the cache capacity, or when entries expire before reuse, the cache thrashes and every resolution falls through to the authoritative datacenter or the local server’s state store. The fixes are TTL tuning, down_policy selection, and reducing unique-token cardinality, not throwing more hardware at the servers.

What this means

ACL resolution runs inline on every request that carries a token. That includes:

• HTTP API calls with X-Consul-Token (the token query parameter form is deprecated)
• DNS queries from agents configured with an ACL token
• RPC forwarding from client agents to servers
• Connect intention evaluation, which resolves source and destination identities
• Prepared queries that execute with a token

When consul.acl.ResolveToken is elevated, all of these slow down together. The symptom is rarely “ACL is broken.” It is “everything is a bit slow, and the slowness tracks request volume.”

Two failure-direction settings determine what happens when resolution itself fails (as opposed to being slow):

• fail-closed: acl.down_policy variants that deny on failure reject legitimate traffic when the ACL subsystem cannot reach the authoritative source. This surfaces as 403s and connection resets.
• fail-open / extend-cache: stale cached authorizations continue to be served. This keeps traffic flowing but may allow tokens that have since been revoked.

The default acl.down_policy is extend-cache. For multi-datacenter setups the async-cache value performs asynchronous refreshes when a cached entry’s TTL expires, preventing a thundering herd of blocking RPCs from hitting the primary datacenter simultaneously when TTLs lapse.

flowchart TD
    A[Authenticated request] --> B{Token in cache?}
    B -- yes --> C[Return cached identity
sub-ms]
    B -- no --> D[Resolve against authoritative DC]
    D --> E{Primary DC reachable?}
    E -- yes --> F[RPC round-trip
low single-digit ms]
    E -- degraded WAN --> G[Thundering herd
all agents refresh at once]
    G --> H[Latency multiplies
10ms+ sustained]
    F --> I[Cache result until TTL]
    C --> J[Request proceeds]
    I --> J
    H --> J

Common causes

Quick checks

# Resolution latency (summary timer, milliseconds)
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -i 'acl.*resolve'

# Cache hit and miss counters
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -i 'acl.token.cache'

# ACL replication status (secondary DCs only)
curl -s http://127.0.0.1:8500/v1/acl/replication | jq .

# Total token count (requires management token)
curl -s -H "X-Consul-Token: $CONSUL_HTTP_TOKEN" http://127.0.0.1:8500/v1/acl/tokens | jq length

# Current TTL and down_policy configuration
consul info | grep -iE 'acl|ttl|policy'

# Check for ACL-denied events in the recent logs
journalctl -u consul --since '1 hour ago' | grep -iE 'permission denied|acl not found' | wc -l

Metric names render differently depending on format. The native JSON telemetry from /v1/agent/metrics uses dot notation (consul.acl.ResolveToken); the Prometheus-format endpoint converts dots to underscores. Older Consul versions emitted consul.acl.ResolveTokenToIdentity as a separate timer; that was merged into ResolveToken in Consul 1.12.0 , so dashboards built before that version may silently stop graphing.

How to diagnose it

1. Confirm the latency is ACL-bound. Pull consul.acl.ResolveToken percentiles over the incident window. If the p99 tracks overall HTTP and DNS latency spikes, ACL resolution is on the critical path. If HTTP latency is high but ResolveToken is flat, look elsewhere (Raft, catalog bloat, DNS).

2. Compute the cache hit ratio. Compare consul.acl.token.cache_hit to consul.acl.token.cache_miss over a representative window. A healthy deployment should see hits dominate by at least an order of magnitude. A ratio approaching 1:1 means the cache cannot hold the working set.

3. Check the unique token population. List tokens and compare against your expected service and operator count. If the count is far higher than expected, look for token leakage (automation creating tokens without cleanup) or a known bug.

4. Verify all three TTLs. Operators frequently raise acl.policy_ttl and forget acl.token_ttl (both default to 30s). If only one is raised, the other still expires every 30s and drives the miss pattern. Check all three: acl.policy_ttl, acl.role_ttl, acl.token_ttl.

5. In secondary DCs, check replication lag. GET /v1/acl/replication shows lag and status. Lag above a few seconds means misses in the secondary DC must round-trip to the primary, multiplying latency. Healthy steady-state replication should be sub-second for tokens.

6. For federated deployments, check down_policy and WAN latency. If acl.down_policy is not async-cache, a TTL lapse across many agents produces a synchronized refresh storm. Switching to async-cache serves stale entries while refreshing asynchronously.

7. If login-derived tokens are involved, check for /acl/login contention. The reported workaround is to use pre-generated static or service-identity tokens for Connect injection rather than auth-method login at pod startup.

Metrics and signals to monitor

Fixes

Raise the three TTLs together

The default for acl.policy_ttl, acl.role_ttl, and acl.token_ttl is 30 seconds each. For most production deployments this is conservative. Raising all three to a few minutes reduces the miss rate dramatically with minimal security cost, because revoked tokens are still enforced: a revocation invalidates the cache entry promptly regardless of TTL.

The common mistake is raising only policy_ttl. If token_ttl stays at 30s, the token cache entry still expires every 30s and you see no improvement. Set all three explicitly.

acl {
  policy_ttl = "5m"
  role_ttl   = "5m"
  token_ttl  = "5m"
}

Tradeoff: longer TTLs mean a revoked token stays usable for longer on agents that cannot reach the authoritative source when combined with extend-cache or async-cache. For most workloads this is acceptable. For high-security environments, keep TTLs short and instead attack the miss rate through token cardinality.

Switch to async-cache for multi-DC

In federated deployments, set acl.down_policy = "async-cache". When a cached entry’s TTL expires, Consul serves the stale entry and refreshes asynchronously. This eliminates the thundering-herd refresh storm where every agent simultaneously issues a blocking RPC to the primary DC on TTL lapse.

acl {
  down_policy = "async-cache"
}

This is especially valuable when the WAN link is the bottleneck. Without async-cache, a degraded WAN link turns every TTL boundary into a coordinated latency spike.

Reduce unique token cardinality

The token cache is fixed-size and not user-tunable via the documented agent configuration. If your unique active token population exceeds the cache capacity, no amount of TTL tuning fully eliminates thrashing. Options:

• Reuse tokens across instances of the same service. Per-instance tokens multiply cardinality. A service-identity token shared across instances of the same workload reduces the working set.
• Prefer service-identity tokens with broader policies over many narrowly-scoped tokens, when your security model allows it.
• Audit and clean up leaked tokens. Automation that creates tokens without revoking them grows the population indefinitely and eventually inflates Raft commit latency.

Address replication lag in secondary DCs

If secondary-DC misses are slow because replication lags, the fix is on the replication path, not the cache. Check the replication token validity and permissions, WAN bandwidth, and primary DC ACL load. Replication lag above a few seconds in steady state is abnormal. In a degraded WAN scenario, async-cache keeps secondary-DC traffic moving while replication catches up.

Handle known token-leakage bugs

If you are running a Consul version affected by a token-leakage bug (where login-derived tokens are not properly revoked on agent shutdown), the token population grows without bound and eventually inflates Raft commit latency. The fix is to upgrade to a version with the fix and to clean up accumulated stale tokens.

Prevention

• Monitor the hit ratio, not just latency. Latency is a lagging indicator. A declining cache_hit to cache_miss ratio gives you lead time before resolution latency crosses thresholds.
• Alert on consul.acl.ResolveToken p99 above 10ms sustained. Treat this as a TICKET. Escalate to PAGE if it coincides with elevated 403s or replication-lag alarms.
• Track total token count over time. Steady growth without corresponding service growth indicates a leak.
• Set all three TTLs explicitly in config. Do not rely on the 30s defaults. Document the chosen values so future operators do not assume only policy_ttl matters.
• In federated deployments, default to async-cache. It is the correct down_policy for any topology where the WAN link is not guaranteed to be fast and reliable.
• Version-track the ACL subsystem. Consul 1.12.0 merged ResolveTokenToIdentity into ResolveToken ; older dashboards may break silently. Consul 1.4.0 restructured ACL config into the nested acl {} stanza; pre-1.4 flat keys are deprecated.

How Netdata helps

• Per-second resolution of consul.acl.ResolveToken exposes TTL-boundary miss storms that minute-granularity monitoring misses. The periodic spike pattern is the signature of a too-short TTL.
• Correlate cache hit and miss counters with resolution latency on one timeline. A rising miss rate that precedes the latency spike confirms cache thrashing rather than a server-side bottleneck.
• Cross-signal correlation with Raft commit time and goroutine count distinguishes ACL-bound latency from Raft-bound latency. If commitTime rises in lockstep with token count, token leakage is the likely driver.
• Anomaly detection on the hit ratio flags declining cache effectiveness before latency crosses a static threshold.
• Replication-lag monitoring in secondary DCs surfaces the multi-DC miss-cost amplifier before failover events turn it into an outage.
• HTTP 403 rate alongside ACL metrics clarifies whether elevated latency is also causing fail-closed denials, which changes severity from TICKET to PAGE.

Related guides

• Consul blocking query accumulation: leaked watches that pile up goroutines
• Consul catalog bloat: too many services and checks slowing everything down
• Consul registration storm: catalog churn overwhelming Raft
• Consul anti-entropy not syncing: local agent state and the catalog drifting apart
• Consul client rpc failed: agents alive but the catalog is going stale
• Consul Connect CA rotation failure: a root roll that never finished
• Consul Connect certificate expired: mTLS handshakes failing across the mesh
• Consul cross-datacenter query failure: prepared-query failover masking a DC outage
• Consul DeregisterCriticalServiceAfter: instances vanishing from the catalog
• Consul DNS latency high: slow lookups stalling connections and failovers
• Consul DNS SERVFAIL: service discovery is broken for your applications
• Consul stale DNS queries: the agent is answering from cache