$ guides / envoy / envoy-rbac-access-denied ▌

Operations Guides

Envoy RBAC: access denied: 403s from the RBAC filter and shadow-mode tuning

A 403 with the body RBAC: access denied is the fingerprint of Envoy’s HTTP RBAC filter blocking a request. It is not the upstream service refusing the request, and it is not the network RBAC filter, which closes the TCP connection instead of returning an HTTP status. When this counter rises you have two questions to answer: which policy matched, and was the match correct.

The filter emits four counters: http.<stat_prefix>.rbac.allowed, http.<stat_prefix>.rbac.denied, http.<stat_prefix>.rbac.shadow_allowed, and http.<stat_prefix>.rbac.shadow_denied. The shadow counters are the safe path for validating new or revised policies against real traffic without enforcing them.

What this means

The HTTP RBAC filter runs in the HTTP connection manager filter chain, before the router filter forwards a request upstream. When the configured policy evaluates to deny, the filter short-circuits the request and emits a 403 with the hardcoded body RBAC: access denied. The body cannot be changed through RBAC configuration; if you need a different body, intercept the local reply (for example via an EnvoyFilter in Istio) or insert a custom filter.

The network RBAC filter, which sits on L4 listener filter chains, behaves differently: on denial it closes the connection without sending an HTTP response. If your client sees an HTTP 403, you are dealing with the HTTP filter.

Envoy does not set a dedicated response flag (such as UO or UF) for RBAC denials. The reliable identification signal is the access log field %RESPONSE_CODE_DETAILS%, which carries rbac_access_denied_matched_policy[policy_name] for an enforced denial, where policy_name is none when no policy matched. This is the single best way to distinguish an RBAC-generated 403 from a 403 returned by the upstream.

Since Envoy 1.31.0, the filter also writes enforced_effective_policy_id and enforced_engine_result to dynamic metadata for non-shadow engines, which is useful when post-processing denials in a logging pipeline.

flowchart TD
  A[Client receives 403] --> B{Body matches RBAC: access denied?}
  B -->|Yes| C[HTTP RBAC filter denied]
  B -->|No| D{RESPONSE_CODE_DETAILS starts with rbac_access_denied?}
  D -->|Yes| C
  D -->|No| E[Upstream 403 or other filter]
  C --> F[Check rbac.denied counter]
  F --> G{Spike right after a config push?}
  G -->|Yes| H[Likely misconfigured policy]
  G -->|No| I{Steady low-rate from unexpected sources?}
  I -->|Yes| J[Possible reconnaissance]
  I -->|No| K[Review matched policy name]

Common causes

Cause	What it looks like	First thing to check
Misconfigured policy after a config push	`rbac.denied` spikes immediately after an xDS update; previously healthy clients start failing	Diff against the prior `shadow_denied` baseline; check `update_rejected` to confirm the new config actually took effect
mTLS / SPIFFE principal matcher failure	Intermittent denials with matched policy `none`, despite valid peer certificate SANs	Check Envoy version and which principal type the policy uses
Upstream 403 confused for RBAC 403	403s with a body that differs from `RBAC: access denied`, or no `rbac_access_denied` details	Inspect `%RESPONSE_CODE_DETAILS%` and the upstream access log
Reconnaissance or attack	Steady, low-rate `rbac.denied` from a small set of unexpected source IPs or user agents	Correlate denied requests with source IP and user-agent distribution
Per-route override gap	Some routes pass, others get denied, in a single listener	Check `RBACPerRoute` overrides on the affected virtual host, route, or weighted cluster

Quick checks

All commands below are read-only.

# RBAC counters (default admin port)
curl -s http://localhost:9901/stats | grep rbac

# Istio sidecar: admin port is typically 15000
curl -s http://localhost:15000/stats | grep rbac

# Enforced vs shadow counters, Prometheus format
curl -s http://localhost:9901/stats/prometheus | grep -E 'rbac_(denied|allowed|shadow_denied|shadow_allowed)'

# Active RBAC filter config (jq path varies across Envoy versions)
curl -s http://localhost:9901/config_dump | \
  jq '.configs[].dynamic_listeners[]?.active_state.listener.filter_chains[].filters[]? |
      select(.name | test("rbac"; "i"))'

# Is the control plane connected? (RBAC policies arrive via xDS / RDS)
curl -s http://localhost:9901/stats | grep control_plane.connected_state

# Was the most recent config push accepted or NACKed?
curl -s http://localhost:9901/stats | grep -E 'update_rejected|listener_create_failure'

# Reproduce a denial and inspect the response body
curl -sv http://localhost:<listener_port>/<path> 2>&1 | grep -i 'rbac'

How to diagnose it

Confirm the 403 originates in RBAC. Look for the body RBAC: access denied and the rbac_access_denied_matched_policy[...] detail string. If neither is present, the 403 is coming from the upstream or another filter, and RBAC is not the culprit.
Identify the matched policy. The policy_name inside the details string tells you which rule fired. none means the request did not match any policy in an ALLOW-action engine, which is itself an important signal.
Correlate with config timing. If rbac.denied rises sharply at the same time as an xDS push, the new policy is the most likely cause. Cross-check update_rejected and listener_create_failure to confirm the new config was actually accepted; Envoy silently keeps the old config on a NACK.
Inspect the matched request. For a single failing request, capture method, path, headers, downstream remote address, and any principal identifiers (mTLS SPIFFE ID, JWT claims). Compare these against the policy’s matchers. A single missing header or a SAN format mismatch is enough to drop a request into none.
If using mTLS principals, check the version. On Envoy 1.36.x in SPIFFE mTLS environments, the envoy.rbac.principals.mtls_authenticated extension can intermittently fail to match, producing matched policy none even though the peer certificate SAN is correctly extracted. If you see this pattern, test the workaround in the next section.
Upstream, this behaviour was reported as Envoy issue #43922 (Envoy 1.36.0, probabilistic denial with mtls_authenticated in SPIFFE mTLS environments). The issue was closed after the reporter observed that session resumption was the likely trigger; disabling TLS session resumption (stateless and stateful) was the suggested workaround upstream.
Validate the fix in shadow mode before enforcing. Convert the suspect policy to shadow rules (or add a parallel shadow matcher) and watch shadow_denied against real traffic. If shadow_denied stays at zero on traffic you expect to allow, the fix is safe to enforce.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`http.<stat_prefix>.rbac.denied` rate	Requests blocked in enforced mode	Spike tightly correlated with an xDS push
`http.<stat_prefix>.rbac.shadow_denied`	Dry-run counter for unenforced policy evaluation	Non-zero and climbing on traffic you expect to allow
`http.<stat_prefix>.rbac.shadow_allowed`	Dry-run counter for requests a shadow policy would permit	Unexpectedly high during a lock-down rollout
`%RESPONSE_CODE_DETAILS%` = `rbac_access_denied_matched_policy[...]`	Names the policy that matched	`none` with mTLS principals and valid peer certs
`control_plane.connected_state`	RBAC rules arrive via xDS or RDS	0 means Envoy is running stale policy
`update_rejected` / `listener_create_failure`	Whether a recent push actually took effect	Non-zero after a “successful” deployment
Source IP and user-agent distribution	Distinguishes misconfiguration from probing	Denied requests concentrated in one IP block

Fixes

Misconfigured policy after a config push

The fastest safe action is to roll back the xDS configuration to the last-known-good policy set. This restores service immediately but discards any other intended changes in the same push, so prefer a targeted revert if your control plane supports it.

For the forward fix, do not enforce blindly. Convert the revised rules to shadow rules and observe shadow_denied and shadow_allowed over a representative traffic window. If shadow_denied falls to zero on legitimate traffic, promote the policy to enforced. The cost is an extra validation cycle; the benefit is not causing a second outage from the same bad assumption.

Also confirm the push was actually accepted. A NACK (update_rejected) leaves Envoy on the old config and can mask the real cause if you assume the new policy is live.

mTLS principal matcher failures

If you are on Envoy 1.36.x in a SPIFFE mTLS environment and see intermittent matched policy none despite valid peer certificate SANs, two workarounds are documented upstream: (a) disable TLS session resumption (stateless and stateful), which was the upstream-maintainer-suggested fix, or (b) replace the envoy.rbac.principals.mtls_authenticated principal with the standard authenticated principal using an explicit principal_name matcher against the SPIFFE ID. The second is more verbose but avoids the extension entirely.

Before upgrading the cluster to a fixed Envoy release, validate the replacement matcher in shadow mode.

Confusing upstream 403s with RBAC denials

Filter your access logs on %RESPONSE_CODE_DETAILS%. RBAC denials always start with rbac_access_denied; upstream-originated 403s carry different detail strings (often the upstream’s response detail or an empty value). The body RBAC: access denied is unique to the filter, so its presence or absence is a quick triage signal.

If the 403s are genuinely upstream, RBAC tuning will not help. Route the investigation to the upstream service team and treat the RBAC counters as a negative control.

Reconnaissance or attack traffic

A steady, low-rate rbac.denied from unexpected sources is often the policy working correctly. Do not “fix” correct denials. Decide whether the probing warrants additional controls: source-based rate limiting, IP blocking at the edge, or alerting on denied-rate anomalies from specific networks.

The tradeoff is between visibility and action. RBAC is already giving you the signal; the decision is whether to escalate enforcement to a rate limiter or network policy layer.

Prevention

Validate new or revised RBAC policies in shadow mode first. Promote to enforced only after shadow_denied stays flat on traffic you expect to allow.
Alert on rbac.denied spikes correlated with xDS pushes. A sudden rise within minutes of a config update is almost always a policy regression.
Monitor update_rejected and listener_create_failure. They tell you whether the policy you deployed is actually live.
Pin the Envoy version and track known RBAC bugs. Principal matcher regressions are version-specific and easy to miss.
Log %RESPONSE_CODE_DETAILS% in your access pipeline. Without it you cannot distinguish RBAC denials from upstream 403s at scale.
Track denied-rate baselines per source network. A shift in the source distribution of denials is an early reconnaissance signal.

How Netdata helps

Per-second RBAC counters. Netdata collects rbac.denied, rbac.allowed, shadow_denied, and shadow_allowed at one-second resolution, which makes the correlation between a denied spike and an xDS push obvious without waiting for a coarse scrape interval.
Anomaly detection on denied rate. Anomaly flags surface the slow reconnaissance pattern: a steady low-rate denied count that never crosses a fixed threshold but represents a meaningful change from baseline.
xDS correlation. Pairing rbac.denied with control_plane.connected_state and update_rejected in a single view tells you whether a denial storm is a policy problem or a stale-config problem.
Shadow-mode validation windows. Watching shadow_denied and shadow_allowed in real time lets you confirm a policy fix is safe before you enforce it, shortening the validation loop from hours to minutes.
Source attribution. Correlating denied counts with downstream connection metrics and, where available, source labels helps separate a misconfiguration from a probing campaign.