$ guides / envoy / envoy-ext-authz-failure-mode-allowed ▌

Operations Guides

Envoy ext_authz failure_mode_allowed: unauthenticated traffic when auth is down

The Envoy ext_authz filter calls an external authorization service on every request that matches the filter chain. That call is on the request critical path: nothing is forwarded upstream until the auth service returns a decision. When the auth service is unreachable, returns an HTTP 5xx, or exceeds its timeout, Envoy either lets the request through without an auth decision or rejects it. That choice is controlled by failure_mode_allow, and every time the fail-open branch fires, Envoy increments http.<stat_prefix>.ext_authz.failure_mode_allowed.

A non-zero rate on failure_mode_allowed means requests are being forwarded upstream without an authorization decision. In any deployment where ext_authz is the security perimeter, that is a live security incident, not a performance problem. The opposite setting (the default) is fail-closed, which rejects every request with status_on_error while the auth service is unavailable. Both modes have valid use cases, but they fail in opposite directions, and the operator response to each is completely different. Know which mode every protected route is running before the auth service has an outage, not during one.

What failure_mode_allow actually does

failure_mode_allow is a boolean field on the v3 ExtAuthz filter config. The default is false (fail-closed).

failure_mode_allow: false (default, fail-closed). When the auth service fails, Envoy rejects the request using the status code configured in status_on_error. The default for status_on_error is 403 Forbidden. The body the auth service might have returned is dropped; the client sees the status_on_error code with an empty body. No traffic reaches the upstream until the auth service is healthy again.
failure_mode_allow: true (fail-open). When the auth service fails, Envoy forwards the request upstream as though it had been allowed. The counter http.<stat_prefix>.ext_authz.failure_mode_allowed increments. The upstream has no way of knowing the request was not authorized unless you also propagate the x-envoy-auth-failure-mode-allowed header.

The word “fails” needs to be precise. It does not mean the same thing as an explicit denial.

What counts as an ext_authz error

failure_mode_allow only triggers on errors, not on explicit denials. This distinction is the source of most operator confusion around the filter.

A 200 OK from the auth service means “allow”. Envoy forwards the request. Neither error nor failure_mode_allowed increments.
A 403 Forbidden (or whatever denied status the auth service returns) means “deny”. Envoy returns the denial to the client and increments ext_authz.denied. The response flag UAEX is set in access logs. This is not an error. failure_mode_allow is irrelevant here.
An HTTP 5xx from the auth service means the auth service could not make a decision. Envoy treats this as an error. Fail-closed returns status_on_error. Fail-open forwards the request and increments failure_mode_allowed.
A network failure (TCP reset, connect timeout, stream idle timeout, deadline exceeded) means Envoy could not reach the auth service. Same handling as a 5xx.

Auth service outcome	`failure_mode_allow: false` (default)	`failure_mode_allow: true`
200 OK (allow)	Forward upstream	Forward upstream
4xx denial (e.g., 403)	Return denial to client, flag `UAEX`, `ext_authz.denied++`	Return denial to client, flag `UAEX`, `ext_authz.denied++`
5xx from auth service	Return `status_on_error` (default 403), `ext_authz.error++`	Forward upstream, `ext_authz.error++` and `ext_authz.failure_mode_allowed++`
Network failure to auth service	Return `status_on_error`, `ext_authz.error++`	Forward upstream, `ext_authz.error++` and `ext_authz.failure_mode_allowed++`

Note the asymmetry in the fail-closed column: a real denial from the auth service and an infrastructure error both surface to the client as a 403 (by default). From the client side they are indistinguishable. From the operator side they are completely different signals. Use ext_authz.denied versus ext_authz.error to separate them, and use the UAEX access log flag to confirm denials.

flowchart TD
    A[Client request] --> B[ext_authz filter]
    B --> C[Call auth service]
    C --> D{Response}
    D -->|200 OK allow| F[Forward upstream
ext_authz.ok++]
    D -->|4xx denial| E[Return denial
flag UAEX
ext_authz.denied++]
    D -->|5xx or network error| G{failure_mode_allow?}
    G -->|false default| H[Return status_on_error
default 403
ext_authz.error++]
    G -->|true| I[Forward upstream
ext_authz.error++
ext_authz.failure_mode_allowed++]

The signals

Every ext_authz-protected HTTP connection manager emits a small, fixed set of counters under http.<stat_prefix>.ext_authz. The stat_prefix comes from the HTTP connection manager config, not the cluster name.

http.<stat_prefix>.ext_authz.ok. Successful allow decisions.
http.<stat_prefix>.ext_authz.denied. Explicit denials returned by the auth service.
http.<stat_prefix>.ext_authz.error. Auth service failures, including both network errors and 5xx responses. In a fail-closed deployment, a non-zero rate here means traffic is being rejected by status_on_error. In a fail-open deployment, every error is also a failure_mode_allowed increment.
http.<stat_prefix>.ext_authz.failure_mode_allowed. The fail-open counter. Counts requests let through without an auth decision because the auth service failed and failure_mode_allow: true. This is the security-hole signal. It should be flat zero in steady state. Any increment is an event.
http.<stat_prefix>.ext_authz.latency. The auth service call latency. This is added directly to the request critical path for every authorized request, and it shows up in downstream_rq_time dollar-for-dollar. A slow auth service is a slow proxy.

The response flag UAEX in the access log marks ext_authz-denied requests. It does not mark fail-open traffic, because fail-open traffic was not denied. If you want fail-open visibility in logs, surface the x-envoy-auth-failure-mode-allowed header upstream and log it there, or build a log pipeline that joins the failure_mode_allowed counter against access log volume.

Where fail-open shows up in production

The auth service does not have to be “down” in the binary sense for fail-open to fire. Any condition that causes Envoy to treat the auth call as an error will trip it:

Auth service deploy or rolling restart. Brief burst of errors as endpoints churn. Generally self-resolves, but in fail-open mode every request during the window is unauthorized.
Auth service CPU saturation or GC pause. Tail latency spikes first, then per-try timeouts start firing, then errors.
Auth service OOM or crash. Sustained error burst, sustained fail-open window.
mTLS certificate rotation between Envoy and the auth service. TLS handshake failures count as ext_authz errors. If SDS is slow to push a new cert, expect a window of fail-open traffic.
Connection pool exhaustion to the auth cluster. The auth service is up but Envoy cannot get a connection. Same fail-open behavior as a hard outage.
Network partition to the auth service zone. Connect timeouts, then errors, then fail-open.
Downstream-client-driven bypass. CVE-2024-23324 describes a case where downstream clients can craft requests that cause Envoy to send invalid gRPC check requests to ext_authz, circumventing the auth check when failure_mode_allow: true. Affected versions include 1.26.x prior to 1.26.7, 1.27.0 through 1.27.2, 1.28.0, and 1.29.0; fixed in 1.26.7, 1.27.3, 1.28.1, and 1.29.1. The upstream advisory lists no known workarounds beyond upgrading. If you run fail-open on an affected version, the fail-open posture is an exploitable bypass, not just an availability tradeoff. Treat fail-open as a setting that needs both monitoring and a current patch level.

The header signal

When failure_mode_allow: true and the fail-open path fires, Envoy adds the header x-envoy-auth-failure-mode-allowed: true to the request headers forwarded upstream — but only when header addition is enabled. In Envoy 1.26.0 through 1.27.x this was automatic (controlled by the runtime feature flag envoy.reloadable_features.http_ext_auth_failure_mode_allow_header_add, default enabled). From 1.28.0 onward, the explicit config field failure_mode_allow_header_add: true must also be set; its default is false. Upstreams can use this header to apply degraded-mode behavior: read-only mode, aggressive rate limiting, feature gating, audit logging at a higher tier.

The header is the only in-band signal the upstream gets that the request was not authorized. Without it, fail-open traffic is indistinguishable from authorized traffic at the application layer.

Quick checks

All read-only. The admin port is 9901 by default and 15000 in Istio sidecar mode. Adjust accordingly.

# Inspect the four core ext_authz counters for a stat prefix
curl -s http://localhost:9901/stats | grep 'ext_authz'

# Confirm which failure_mode_allow is configured on each filter
curl -s http://localhost:9901/config_dump | \
  grep -E 'failure_mode_allow|stat_prefix|status_on_error'

# Spot fail-open traffic in real time (counters are monotonic)
watch -n 1 'curl -s http://localhost:9901/stats | grep ext_authz.failure_mode_allowed'

# Check the auth cluster's health, since auth cluster failure drives fail-open
curl -s http://localhost:9901/stats | grep -E \
  'cluster.<auth_cluster>.(upstream_cx_connect_fail|membership_healthy|upstream_rq_5xx)'

# Verify latency contribution from the auth call
curl -s http://localhost:9901/stats/prometheus | grep 'ext_authz'

If you are running fail-closed and triaging an outage, the same ext_authz.error counter is the one to watch. It tells you Envoy is actively rejecting traffic because the auth service is unavailable.

Fail-open versus fail-closed: the policy decision

Neither mode is universally correct. The choice is a policy decision with security and availability implications that should be made explicitly per route, not inherited from a tutorial config.

Fail-open (failure_mode_allow: true) is appropriate when:

The protected service has layered controls. Authz is one of several gates, not the only one.
The upstream can tolerate unauthenticated traffic for short windows without data integrity risk.
Total outage of the protected service is more costly than partial exposure.
You have aggressive monitoring on failure_mode_allowed and an audited runbook for any non-zero rate.

Fail-open is dangerous when:

ext_authz is the only perimeter. There is no second gate.
The upstream assumes all traffic is authenticated and exposes data or mutations based on that assumption.
Sensitive data, financial operations, or compliance-regulated workloads are involved.
Nobody is watching failure_mode_allowed and it has been incrementing for hours.

Fail-closed (failure_mode_allow: false, default) is appropriate when:

Unauthenticated traffic is unacceptable under any condition.
The service has strict security or compliance requirements.
A full outage of the protected service is preferable to an unauthorized request succeeding.

Fail-closed is dangerous when:

The auth service is a hard dependency with no SLO headroom. An auth service hiccup takes down everything behind Envoy.
The blast radius of total outage is larger than the blast radius of partial exposure.
You are not monitoring ext_authz.error. Fail-closed outages look like a generic 5xx spike on the protected service if you do not separate the cause.

In both modes, the auth service becomes the most critical dependency in the request path. Treat its SLO, capacity, and monitoring with the same rigor as the data plane.

Signals to watch in production

Signal	Why it matters	Warning sign
`http.<stat_prefix>.ext_authz.failure_mode_allowed` rate	Counts requests forwarded without an auth decision in fail-open mode	Any non-zero value. Flat zero is the only healthy state.
`http.<stat_prefix>.ext_authz.error` rate	Auth service is failing (network or 5xx). In fail-closed mode, this is the counter that says traffic is being rejected.	Sustained non-zero
`http.<stat_prefix>.ext_authz.denied` rate	Auth service is explicitly denying requests. Sudden spike may indicate policy change, credential rotation, or attack.	Sudden change from baseline
`http.<stat_prefix>.ext_authz.latency` (P99)	Auth call latency is added to every authorized request’s critical path. A slow auth service is a slow proxy.	Upward trend, especially P99 drifting while P50 is stable
`cluster.<auth_cluster>.upstream_cx_connect_fail`	Auth hosts not accepting connections. Often the leading indicator before errors start.	Non-zero sustained
`cluster.<auth_cluster>.membership_healthy` ratio	Auth cluster host health. A drop here typically precedes an ext_authz error burst.	Ratio below baseline
Response flag `UAEX` in access logs	Confirms explicit ext_authz denials (not errors, not fail-open)	Spike correlated with policy or credential changes

For fail-open deployments, the alerting rule is straightforward: page on any sustained non-zero rate of failure_mode_allowed. For fail-closed deployments, the equivalent rule pages on sustained non-zero ext_authz.error combined with rising downstream_rq_503 (or whatever status_on_error is configured to).

Prevention

Make the mode explicit in config review. failure_mode_allow is a single boolean with outsized impact. Every protected route should have a deliberate choice, documented in the route’s runbook.
Monitor the counter, not just the auth service. A healthy auth service with a misconfigured filter can still produce fail-open traffic if the filter is wired wrong. The counter is ground truth.
Track auth service latency as a first-class SLO. Because ext_authz is on the critical path, auth service P99 directly determines protected service P99. Latency budget overruns on auth should be treated as capacity incidents.
Capacity-plan the auth cluster like a data-plane tier. The auth service is not control-plane infrastructure that can be slow. It is on every request’s hot path.
Re-evaluate fail-open after every CVE. Fail-open magnifies any vulnerability that lets a downstream client influence whether the auth check happens. Keep Envoy current if you run fail-open.
Propagate the header. If you run fail-open, configure upstreams to honor x-envoy-auth-failure-mode-allowed and degrade gracefully. Logging it also gives you an application-layer audit trail of the fail-open window.

How Netdata helps

The failure_mode_allowed counter should be flat zero in steady state. Per-second collection and anomaly detection surface the first increment without waiting for a scrape interval, and correlating auth cluster health metrics alongside the ext_authz counters shortens the diagnosis path.

Per-second collection of ext_authz.ok, ext_authz.denied, ext_authz.error, and ext_authz.failure_mode_allowed lets you see the exact second the auth service started failing and the exact second fail-open traffic started flowing.
ML anomaly detection on failure_mode_allowed flags the first increment in a flat-zero series without needing a fixed threshold.
Correlating ext_authz.error against the auth cluster’s upstream_cx_connect_fail, membership_healthy, and upstream_rq_5xx in a single view shortens the path from “auth is failing” to “auth cluster host 3 is ejecting”.
Tracking ext_authz.latency alongside downstream_rq_time makes the cost of a slow auth service visible as direct proxy overhead rather than a mysterious latency regression.
The same dashboard can surface the auth cluster’s circuit breaker state, so connection pool exhaustion to the auth service (a common root cause of fail-open bursts) is visible alongside the failure_mode_allowed counter that it triggers.