Consul intention denied: service-to-service traffic blocked by policy

Service A can no longer reach service B. Envoy sidecars log 403 responses or connection resets. The Consul server cluster looks healthy: leader stable, Raft committing, gossip intact, certificates within their lifetime. Traffic that worked an hour ago is now blocked.

This is almost always policy, not infrastructure. Consul Connect intentions are the mesh authorization layer, evaluated at connection establishment and enforced by Envoy RBAC filters. When an intention denies a connection, the data plane is doing exactly what it was configured to do. The diagnostic job is to determine whether the denial is correct (policy working as designed) or a misconfiguration: wrong identity, missing allow, precedence mistake, or stale cache.

The classic trigger is switching acl.default_policy from allow to deny without authoring allow intentions for every legitimate service pair first. Every missing intention becomes a block.

What this means

Intentions are authorization policies bound to service identities in the Connect mesh. When a source service opens a connection to a destination, the destination’s Envoy sidecar evaluates the intention set for that (source, destination) pair before the connection completes. L4 (TCP) intentions are evaluated once per connection through Envoy’s network RBAC filter. L7 (HTTP) intentions are evaluated per request through the HTTP RBAC filter.

The acl.default_policy setting controls what happens when no explicit intention matches:

• allow: missing intention means traffic is permitted. Permissive, but no default-deny posture.
• deny: missing intention means traffic is blocked. Zero-trust default, but requires explicit allow intentions for every legitimate pair.

Consul evaluates intentions by specificity: the most specific match wins, and at equal specificity deny takes precedence over allow. A specific deny for (web, billing) overrides a wildcard allow for (*, billing). Wildcards are supported in L4 source and destination names, but not in L7 permission rules.

Intentions are cached locally on the Consul agent and in the Envoy proxy. Updates propagate to proxies via xDS streaming, not instantaneously. If an agent or proxy is disconnected from servers, cached intentions continue to be enforced until connectivity is restored and a new configuration arrives. Envoy’s default initial_fetch_timeout is 15 seconds; this bounds waiting for the first configuration, not the latency of subsequent pushes.

The consul intention check <src> <dst> command evaluates the L4 decision for a pair and returns Allowed or Denied. It requires destination service:read; any intention carrying L7 Permissions is treated as deny by this L4-only check. consul intention match instead requires intentions:read and lists rules without evaluating them.

Common causes

Quick checks

Run these read-only checks before changing anything.

# Check ACL status on the agent
consul info | grep -i acl

# Evaluate the L4 decision for a specific pair
consul intention check <src-service> <dst-service>

# List all intentions matching a destination, ordered by precedence
consul intention match <dst-service>

# Verify CA roots are active and not expired
curl -s http://127.0.0.1:8500/v1/connect/ca/roots | jq '{ActiveRootID, Roots: [.roots[] | select(.Active == true) | {Name, NotAfter}]}'

# Inspect intention enforcement and authorization metrics
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -iE "intention|connect.authorize"

# Confirm the service identity a proxy is actually presenting
curl -s http://127.0.0.1:8500/v1/agent/services | jq '.[].Service'

# Check xDS stream health for the proxy's server
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -i xds

consul info does not reliably expose acl.default_policy; the running agent’s configuration is exposed through /v1/agent/self in its version-dependent DebugConfig, and the configuration file or source of truth should be checked as well.

The consul intention CLI tree is deprecated since 1.9 in favor of service-intentions config entries written via consul config write. The check and match commands still function for diagnosis, but new policies should be authored as config entries.

Envoy access-log formats vary by deployment. For HTTP RBAC denials, Envoy records response_code_details as rbac_access_denied_matched_policy[policy_name]; its RBAC filter also exposes <stat_prefix>.rbac.denied and shadow_denied counters. For TCP RBAC, inspect the Envoy RBAC statistics and connection logs configured by your deployment.

How to diagnose it

Work through the decision tree before touching policy. The goal is to classify the denial as correct, missing, shadowed, or identity-broken.

flowchart TD
    A[Sidecar logs denied or reset] --> B{Server cluster healthy?}
    B -- No --> C[Fix Raft, gossip, or certs first]
    B -- Yes --> D{Many pairs failing at once?}
    D -- Yes --> E[Check acl.default_policy]
    E --> F{Recently switched to deny?}
    F -- Yes --> G[Author allow intentions before switching]
    F -- No --> H[Check CA root and leaf cert expiry]
    D -- No --> I[Run intention check for the pair]
    I --> J{Denied by check?}
    J -- Yes --> K[Review specificity and wildcards]
    J -- No --> L[Verify service Name identity]

1. Confirm the server cluster is healthy. Check leader existence, Raft commit progress, and gossip membership. If the control plane is broken, intention delivery is broken, and denials may be stale-cache artifacts rather than current policy. Fix the cluster first.

2. Scope the blast radius. If many unrelated service pairs fail at once, suspect a global change: default_policy, CA root expiry, or a broad ACL token revocation. If only specific pairs fail, narrow to intention configuration or identity for those pairs.

3. Run the intention check for the failing pair. consul intention check <src> <dst> returns the L4 decision. If it returns Denied, the policy itself blocks the pair. If it returns Allowed but traffic still fails, the problem is identity, certificates, or cache, not the intention set.

4. Inspect the matching intentions for the destination. consul intention match <dst> lists all intentions that apply, ordered by precedence. Look for a specific deny shadowing a wildcard allow. At equal specificity, deny wins.

5. Verify the service identity. Intentions match on the service Name, not the service ID. Check /v1/agent/services on the source agent and confirm the ServiceName matches the intention source. A service redeployed under a different name will never match the old intention. Consul 1.9’s UI had a topology-view defect that could create intentions with the service ID instead of the name (issue #9304); PR #9316 fixed the UI behavior after 1.9.0. Verify the installed UI before relying on topology-driven authoring.

6. Check certificates. If the symptom is mTLS handshake failures rather than clean RBAC denials, inspect CA roots and leaf expiry. Leaf certificates typically have short TTLs (72 hours by default). If rotation has stalled, Envoy holds expired certificates and connections fail. Consul 1.15.0 and 1.15.1 shipped a known leaf-certificate rotation race that could cause mesh communication loss after the 72-hour LeafCertTTL; it was fixed in 1.15.2.

7. Check ACL permissions on the proxy token. The proxy’s token needs service:read on the destination service to evaluate intentions. If the token lacks permission, the proxy cannot read the intention set and may fail closed. Look for “ACL not found” or “permission denied” in server logs around intention endpoints.

8. Rule out cache staleness. If you recently changed an intention and the proxy has not picked it up, check xDS stream health. If the stream is dropped or the proxy is disconnected from servers, it enforces the last-known-good configuration. Reconnecting or restarting the sidecar forces a refresh. Note that restarting the sidecar causes a brief connection drop for all traffic through that proxy.

Metrics and signals to monitor

Envoy and Consul names vary by telemetry format (Prometheus, StatsD, DogStatsD). Confirm rendered names in your deployment; the underlying signals above are Envoy RBAC statistics, consul.intention.apply, and consul.acl.ResolveToken.

Fixes

Group the fix by the root cause you identified. Each has a tradeoff between speed and safety.

default_policy switched to deny

The fastest restore is reverting acl.default_policy to allow. This immediately unblocks all traffic but removes the default-deny posture; treat it as a rollback, not a fix. The safer path is to author allow intentions for every legitimate service pair first, validate each with consul intention check, then switch the default. This is a staged migration, not a flag flip.

Missing or wrong allow intention

Create the intention as a service-intentions config entry via consul config write (preferred post-1.9). For legacy deployments, consul intention create -allow <src> <dst> still works but is deprecated. Verify with consul intention check <src> <dst> after writing. A broad wildcard allow (* to dst) restores traffic fast but weakens least-privilege; prefer specific source matches.

Precedence mistakes (wildcard vs specific)

If a specific deny shadows a wildcard allow, you must either remove the specific deny or add a more specific allow. List the full match set with consul intention match <dst> to see precedence ordering. At equal specificity, deny wins. A deny for (web, billing) beats an allow for (*, billing) because the specific source is more specific than the wildcard.

Service identity mismatch

Update either the service registration (so the Name matches what the intention expects) or the intention source (so it matches the registered Name). The Name field is what matters, not the ID. After fixing, verify with consul intention check using the correct source name.

CA or certificate failure

If the CA root is expired or the signing backend (for example, Vault) is unreachable, no new leaf certificates can be issued. Existing leaves continue to work until they expire, then all mTLS fails. Check /v1/connect/ca/roots for active, non-expired roots. If rotation has stalled and Envoy holds expired leaves, restarting the downstream job or the Consul client forces a certificate refresh. Switching CA providers mid-incident is disruptive and should be a planned operation, not an urgent decision.

ACL permissions blocking intention reads

Grant the proxy’s ACL token service:read on the destination service. Without it, the proxy cannot evaluate intentions and may fail closed. Verify by checking server logs for ACL errors on intention endpoints after the token change propagates.

Stale proxy cache

If the xDS stream is healthy and the intention change is committed, the proxy should refresh on the next configuration push. If it does not, restarting the sidecar forces a full configuration fetch. This drops existing connections through that proxy, so apply it during a maintenance window if possible. Investigate why the stream was not delivering updates rather than relying on restarts as a fix.

Prevention

• Stage default_policy changes. Author all allow intentions first, validate each pair with consul intention check, then switch. Never flip allow to deny cold.
• Monitor denial rate as a baseline. Alert on new source/destination pairs, not just absolute count. A new pair after a deploy is the earliest signal of a missing intention.
• Treat intention changes like firewall changes. Peer review, canary, and rollback plan. A single typo in a source name blocks traffic.
• Track service identity consistency. Validate that the ServiceName in registrations matches what intentions expect, as part of deployment checks.
• Monitor certificate expiry with long runway. Alert on leaf certificates at 24h and CA roots at 7 days. Monitor renewal success rate, not just expiry time.
• Stay current on version behavior. The consul intention CLI is deprecated since 1.9; config entries are the forward path. Know which API surface your automation targets.

How Netdata helps

• Per-second visibility into consul.connect.authorize exposes a denial spike the moment a policy change lands. Use this as the first check when paged on a service-to-service failure.
• Correlate denial rate with xDS stream health and ACL resolution latency on a single timeline to localize whether the cause is policy, propagation, or permissions.
• Certificate expiry dashboards show CA and leaf TTL with enough runway to fix rotation before it becomes mTLS failures.
• Anomaly detection on denial patterns separates expected baseline denials from new pairs introduced by a deploy or config change.
• Envoy sidecar connection metrics confirm whether denials are impacting application traffic or are noise at the mesh boundary.

Related guides

• Consul blocking query accumulation: leaked watches that pile up goroutines
• Consul catalog bloat: too many services and checks slowing everything down
• Consul registration storm: catalog churn overwhelming Raft
• Consul anti-entropy not syncing: local agent state and the catalog drifting apart
• Consul client rpc failed: agents alive but the catalog is going stale
• Consul DeregisterCriticalServiceAfter: instances vanishing from the catalog
• Consul DNS latency high: slow lookups stalling connections and failovers
• Consul DNS SERVFAIL: service discovery is broken for your applications
• Consul stale DNS queries: the agent is answering from cache
• Consul on EBS: burst-credit exhaustion and the sudden latency cliff
• Consul Go GC pauses: stop-the-world stalls that disturb Raft timing
• Consul goroutine count climbing: the leak behind slow resource exhaustion