Envoy downstream 4xx spike: 401s, 403s, and 404s from the client side

A spike in http.<stat_prefix>.downstream_rq_4xx is usually a client-side story, not an Envoy story. The proxy is reporting that clients sent bad, unauthorized, or unroutable requests. The single counter lumps 400s, 401s, 403s, and 404s together, and Envoy does not expose per-status-code downstream counters. You will not find downstream_rq_401 or downstream_rq_403 in the stats dump.

The diagnostic path: take the aggregate signal seriously, but break it apart using access logs, the ext_authz and RBAC filter stats, and response flags. A 401/403 spike points at authentication infrastructure, a credential or certificate rotation, or a brute-force attempt. A 404 spike appearing right after an xDS push points at route misconfiguration. These have different owners and different runbooks.

This page covers how to decompose the spike, which Envoy signals separate auth failures from routing failures, and what to fix first.

What this means

downstream_rq_4xx is a monotonic counter on the HTTP connection manager. It increments for every 4xx response Envoy sends to a downstream client, regardless of who produced the response: the upstream app, an Envoy filter (ext_authz, RBAC, rate limit), or Envoy itself (no route, invalid header). It does not tell you which code or which component.

Two instrumentation gaps make this counter noisier than it looks:

• No per-code downstream breakdown. Upstream stats have upstream_rq_401, upstream_rq_403, and so on, but downstream only has the class-level counter. To split 401s from 403s from 404s you need access logs or a log pipeline.
• downstream_rq_4xx does not capture 431 (Request Header Fields Too Large). A header-too-large rejection increments downstream_cx_protocol_error but not the 4xx counter, so a client misconfiguration storm can be invisible here.

Because absolute 4xx volume is dominated by ordinary client behavior (legitimate 404s, scanner noise, expired browser tokens), alert on rate-of-change against a rolling baseline, not on an absolute count. A sudden 5x step on a normally flat counter is the signal that matters.

flowchart TD
    A["downstream_rq_4xx spike"] --> B["Group access logs by\nRESPONSE_CODE + RESPONSE_FLAGS"]
    B --> C{"Dominant code?"}
    C -->|"401 / 403"| D["Check ext_authz + rbac stats"]
    C -->|"404"| E["Check route config + xDS"]
    D --> F{"Flag UAEX or ext_authz.error?"}
    F -->|"UAEX, denied rising"| G["Auth policy regression"]
    F -->|"error rising, fail-closed"| H["Auth backend outage"]
    F -->|"failure_mode_allowed rising"| I["Fail-open active"]
    E --> J{"NR flag + recent push?"}
    J -->|"update_rejected or connected_state=0"| K["Stale or rejected config"]
    J -->|"VHDS convergence window"| L["Transient 404"]

Common causes

Quick checks

Run these read-only. None of them change Envoy state. Adjust the admin port for your deployment: 9901 for standalone Envoy, 15000 for Istio sidecars.

# 1. Confirm the aggregate 4xx rate and ratio (two samples 10s apart)
curl -s http://localhost:9901/stats/prometheus | grep 'downstream_rq_4xx\|downstream_rq_total'

# 2. Pull ext_authz filter stats
curl -s http://localhost:9901/stats | grep 'ext_authz'

# 3. Pull RBAC filter stats
curl -s http://localhost:9901/stats | grep 'rbac'

# 4. Check whether Envoy is rejecting config (silent NACKs)
curl -s http://localhost:9901/stats | grep -E 'update_rejected|listener_create_failure|connected_state'

# 5. Check cert runway
curl -s http://localhost:9901/certs | jq '.certificates[] | {subject: .cert_chain[].subject, days: .days_until_expiration}'

# 6. Confirm 5xx is flat (this is a client-side story, not an upstream outage)
curl -s http://localhost:9901/stats | grep 'downstream_rq_5xx'

# 7. Inspect the current route tables to validate 404s are routing, not missing
curl -s http://localhost:9901/config_dump | jq '.configs[] | select(."@type" | test("RouteConfiguration"))'

How to diagnose it

1. Decompose the aggregate. Grep access logs for the spike window and group by %RESPONSE_CODE% and %RESPONSE_FLAGS%. The flags are access-log only and are not exposed as aggregate stats, so this step is mandatory. Without it, you are guessing at which code dominates.

2. Split 401/403 from 404. They have different root causes and different owners. 401/403 is an auth story. 404 with NR is a routing story. Chasing both at once wastes the first 15 minutes of an incident.

3. For 401/403, identify the denying component. If the response flag is UAEX, the ext_authz filter produced the denial. Cross-check against the filter stats:

   • ext_authz.denied rising with ext_authz.ok flat means the auth service is actively denying more requests. Look for a policy change.
   • ext_authz.error rising with ext_authz.denied roughly flat means the auth service is unreachable or erroring. Envoy is fabricating the 403 via status_on_error (default 403). This is an outage of the auth backend, not a policy problem.
   • ext_authz.failure_mode_allowed rising means Envoy is fail-open. There is no 4xx spike because traffic is passing unauthenticated. Treat any nonzero value here as a security incident.
   • rbac.denied rising without UAEX means the in-proxy RBAC filter is the source, typically after a policy deployment.

4. For 404, correlate with config timing. A 404 spike with the NR flag means Envoy has no route for the Host and path. The two questions are whether a config push just happened and whether Envoy actually accepted it:

   • If control_plane.connected_state is 0, Envoy is on stale config and may be missing routes that the control plane thinks it pushed.
   • If update_rejected is climbing, Envoy is connected but NACKing the new config. The control plane reports a successful deploy; Envoy silently kept the old routes. This is the classic “deployment completed but nothing changed” failure.
   • If you use VHDS, transient 404s can appear when RDS updates land before the corresponding virtual host updates. The window is short but real during convergence.

5. Cross-check upstream. Compare downstream_rq_4xx against the cluster’s upstream_rq_4xx. If the upstream counter is climbing too, the app is genuinely producing these codes and Envoy is just forwarding them. If only the downstream counter moves, the response originated in Envoy or in a filter.

6. For abuse patterns, read the access log distribution, not the counters. A brute-force or credential-stuffing spike is concentrated on a handful of source IPs and a small set of paths, with uniform small bodies. Aggregate counters cannot reveal this shape.

Metrics and signals to monitor

Fixes

Auth policy regression (403 with UAEX)

Roll back the policy change first, then debug the rules. If you use shadow RBAC, check rbac.shadow_denied before enforcing: a policy that denies legitimate traffic shows up there without breaking users. For ext_authz, confirm the auth service is returning the denial you expect by hitting it directly with a representative request. Increasing retry counts or timeouts here does not help; the service is answering, just with deny.

ext_authz outage, fail-closed (403 with ext_authz.error)

Restore the auth service. Envoy is generating the 403s locally via status_on_error because the backend is unreachable or returning errors. Note the stats gap: when the auth service returns 5xx, Envoy treats it as an error and the response code becomes status_on_error regardless of what the auth body said. You cannot distinguish “auth server denied” from “auth server 500” using the 4xx counter alone. While restoring service, do not flip to failure_mode_allow: true as a quick fix unless you intend to run unauthenticated.

ext_authz outage, fail-open (no 4xx spike, failure_mode_allowed climbing)

This is the dangerous variant. Traffic is flowing, the 4xx counter looks fine, and unauthenticated requests are passing. The fix is to restore the auth service, not to celebrate the flat error rate. Any nonzero failure_mode_allowed during an unplanned window is a security incident.

Credential or certificate rotation (401 spike)

Check /certs for expiry and confirm SDS is connected. If ssl.fail_verify_error is climbing on the path to the auth service, the rotation did not propagate. Re-trigger rotation or roll back to the previous credential set. Coordinate with the auth service owner: a 401 spike often means the verifier and the credential issuer disagree on the new key.

Route misconfiguration (404 with NR)

The fix is the config, not Envoy. First confirm whether Envoy accepted the push:

• update_rejected nonzero: fix the rejected config on the control plane side. Envoy is protecting itself by keeping the old routes.
• connected_state = 0: restore control plane connectivity. Envoy picks up the routes on reconnect.
• VHDS ordering: ensure virtual host updates arrive after the RDS updates that reference them.

Roll back the route change if the new config is wrong. Do not paper over NR with catch-all routes; that hides the misconfiguration and breaks routing observability.

Brute force or abuse (401/403 concentrated on few IPs)

This is a rate-limiting or WAF problem, not an Envoy config bug. If you run local or global rate limiting in Envoy, watch ratelimit.over_limit. Otherwise, handle it at the edge. Adding the offending IPs to a deny list via RBAC is a valid short-term mitigation; verify the RBAC rule matches headers correctly given the duplicate-header concatenation behavior described below.

RBAC header bypass (CVE-2026-26308)

If your Envoy is older than the fixed releases (1.37.1, 1.36.5, 1.35.8, 1.34.13) and you rely on RBAC exact-match header rules, treat unexplained denial-rate drops as a possible bypass. Upgrade. The bug is that Envoy concatenates duplicate header values into a comma-separated string before matching, so a request carrying x-role: admin,user can evade a rule keyed on x-role: admin.

Prevention

• Alert on rate-of-change, not absolute count. A flat 4xx baseline with normal scanner noise will trip any reasonable absolute threshold. Alert on deviation from a rolling baseline for the specific stat_prefix.
• Build a 4xx-by-code signal from access logs. Because downstream has no per-code counter, ship access logs to a pipeline that counts by %RESPONSE_CODE% and %RESPONSE_FLAGS%. This is the only way to get 401/403/404 breakdowns in real time.
• Run RBAC in shadow mode before enforcing. Watch rbac.shadow_denied for a full traffic cycle. If it catches legitimate traffic, fix the policy before it becomes denials.
• Monitor failure_mode_allowed as a security signal. Any unplanned nonzero value is a page, not a ticket.
• Track update_rejected alongside connected_state. A connected Envoy that NACKs every push is as stale as a disconnected one, and quieter.
• Keep Envoy current on RBAC CVEs. Header-matching bypasses are silent; the only reliable signal is version hygiene.

How Netdata helps

• Per-second downstream_rq_4xx and downstream_rq_total expose the step change the moment it starts, and ML anomaly detection flags the rate-of-change deviation without hand-tuned absolute thresholds.
• Correlating ext_authz.denied, ext_authz.error, and ext_authz.failure_mode_allowed against the 4xx spike separates “policy is denying more” from “auth backend is down” from “fail-open is active” within a single chart view.
• RBAC filter stats (rbac.denied, rbac.shadow_denied) sit next to the 4xx counter, so a policy rollout that spikes denials reads cleanly against the deploy timeline.
• control_plane.connected_state and update_rejected plotted against a 404 spike make the stale-config or NACK diagnosis immediate, especially when aligned with the config-push window.
• ssl.fail_verify_error alongside 401 bursts points a credential or cert rotation problem at the auth path rather than at the application.

Related guides

• Envoy 502 and upstream resets: rx_reset, tx_reset, and mid-response failures
• Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend
• Envoy 504 upstream timeout: upstream_rq_timeout, per-try timeouts, and the UT flag
• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy clusters stuck warming: warming_clusters non-zero and routes returning 503
• Envoy connection pool exhaustion: a slow upstream that fills the pool
• Envoy control_plane.connected_state = 0: running on stale xDS config
• Envoy downstream_rq_time high: client-observed latency and proxy overhead
• Envoy file descriptor exhaustion: the FD cliff that refuses every new connection
• Envoy health checks vs outlier detection: two systems that eject hosts differently
• How Envoy actually works in production: a mental model for operators
• Envoy listener_create_failure: a listener config Envoy could not apply