Envoy response flags: decoding UO, NR, UF, UT, UC and the rest

A 503 from Envoy can mean a dozen different things. The HTTP status code tells you what the client saw; it does not tell you why Envoy generated that response. The %RESPONSE_FLAGS% access log field separates a circuit breaker trip from a missing route, a dead upstream, or a client that hung up.

Response flags are the most precise debugging signal Envoy emits, and the most operationally misunderstood. They are not aggregate stats; they appear only in access logs. Most teams discover this the first time they try to alert on UO and find no Prometheus counter for it.

This is a reference for the response flags you will see in production: what each means, where in the request lifecycle it fires, what status code it typically pairs with, and what to check next. It covers UF, UO, NR, NC, URX, UT, UC, DC, LR, RL, UAEX, IH, SI, DPE, DI. Newer flags exist in the Envoy source for DNS resolution failure, overload manager actions, and drop overload; if you see a flag not covered here, source/common/stream_info/utility.h in the Envoy repository is the canonical list.

What response flags are (and what they are not)

%RESPONSE_FLAGS% is an access log formatter operator. Envoy sets zero or more flags on each request to record why it handled it the way it did. The flags explain the cause behind a response code, not the response code itself.

Three properties matter operationally.

Access log only. Response flags are not exposed as aggregate stats. cluster.<name>.upstream_rq_503 counts both Envoy-generated 503s and 503s forwarded from the upstream. The response flag is the only way to tell them apart.

Per-request, not per-cluster. Flags attach to individual requests. There is no cluster.<name>.response_flags.UO counter.

Comma-separated when multiple apply. A single request can carry UC,URX: the upstream closed the connection and retries are exhausted. Naive string matching on logs misses these.

Alerting on response flags therefore requires a log pipeline, a sidecar counter, or a Lua/Wasm filter that increments a custom stat. Envoy itself does not aggregate them.

Where to find them

Response flags appear wherever %RESPONSE_FLAGS% is in the access log format string. The default Envoy format includes it. In Istio, the default sidecar format also includes it, but custom configurations sometimes strip it; verify before you rely on it.

A typical line:

[2024-01-15T10:23:45.123Z] "GET /api/v1/users HTTP/2" 503 UO 0 91 3 - "..." "..." "abc123"

The response code is 503; the field immediately after it (UO) is the response flag. When no flag is set, Envoy emits -. A dash means Envoy did not flag this request, not that the request was healthy: a 404 forwarded from an upstream also produces - because Envoy did not intervene.

To extract counts without a full log pipeline:

# Default Envoy format: %RESPONSE_FLAGS% is the 6th whitespace-delimited field
# (the quoted request line splits into three). Adjust if your format is custom.
awk '{print $6}' /var/log/envoy/access.log | sort | uniq -c | sort -rn

The flag reference

The flags group by where in the request lifecycle they fire. Read them in that order when debugging: configuration first, then circuit breaking, then upstream connection, then timeouts, then downstream.

Configuration errors: NR and NC

These are the only flags that almost always indicate a bug in the configuration, not a transient network issue. Any non-zero rate of NR or NC in production is a configuration error.

NR immediately after an xDS push is the textbook signature of a bad config deployment. The control plane reports success, Envoy accepted the config, but the route table no longer matches traffic that worked a minute ago. Check update_rejected (per-resource NACKs) and listener_manager.listener_create_failure; a config that is valid but wrong produces no NACK, so a clean update_success does not rule this out.

Circuit breaker and saturation: UO

UO means Envoy fast-failed a request locally because a per-cluster circuit breaker was open. Envoy generated the 503; the upstream never saw the request.

UO should be zero in steady state. Any sustained UO rate means a breaker is open, which means either the upstream is slow (filling the connection pool) or the limits are set too low for the workload. Increasing the limits is rarely the right fix; the breaker is correctly reporting that the upstream cannot absorb more load. See Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests.

Upstream connection failures: UF and UC

Both indicate a problem reaching the upstream, but at different points in the connection lifecycle.

UF means the SYN never completed (or was RST immediately). UC means the connection was established but the upstream tore it down during the request. The distinction maps to two different upstream failure modes: host not listening versus host crashing mid-response. See Envoy upstream_cx_connect_fail: failed TCP connections to upstream hosts.

Upstream timeouts: UT and URX

These two often co-occur. UT says Envoy gave up waiting. URX says Envoy exhausted its retry budget while trying.

URX paired with UC (UC,URX) is a common combination: upstream keeps closing connections, retries keep failing, and the retry budget is exhausted.

Downstream disconnects: DC and DPE

These describe what the client did, not what Envoy did wrong. They are often noise.

DC is the most over-alerted flag. Clients close connections for many benign reasons: page navigation, mobile app backgrounding, cancellation of slow requests. A baseline DC rate is normal. Investigate only when DC spikes alongside elevated latency, which suggests clients are timing out waiting for Envoy.

Local resets and rate limiting: LR and RL

RL pairs with the rate limit service over_limit result counter. LR is less specific; check circuit breaker and filter stats to find the cause.

Authorization and headers: UAEX and IH

UAEX spikes after a credential rotation or policy change. If ext_authz.error is also climbing, the auth service itself is degraded.

Idle and injection: SI and DI

DI only appears when a fault injection filter is configured. Seeing it unexpectedly means someone enabled chaos testing in production.

Multiple flags at once

Envoy emits flags as a comma-separated list when more than one applies. The combination carries information that a single flag does not.

A retry-related flag (URX) paired with a connection flag (UC, UF) is the signature of a retry storm: the upstream is failing, retries are firing, and the budget is gone. Check the upstream_rq_retry / upstream_rq_total ratio. See Envoy connection pool exhaustion: a slow upstream that fills the pool for the broader pattern.

A diagram for the common 503

When you see a 503, the response flag tells you which subsystem produced it. The flow maps the common cases.

flowchart TD
    REQ[Client request arrives] --> ROUTE{Route exists?}
    ROUTE -- no --> NR[NR: 503]
    ROUTE -- yes --> CLUSTER{Cluster exists?}
    CLUSTER -- no --> NC[NC: 503]
    CLUSTER -- yes --> BREAKER{Circuit breaker open?}
    BREAKER -- yes --> UO[UO: 503, fast-fail]
    BREAKER -- no --> CONNECT{TCP connect ok?}
    CONNECT -- no --> UF[UF: 503]
    CONNECT -- yes --> MIDCONN{Upstream stays connected?}
    MIDCONN -- no --> UC[UC: 502 or 503]
    MIDCONN -- yes --> TIMEOUT{Responds in time?}
    TIMEOUT -- no --> UT[UT: 504]
    TIMEOUT -- yes --> RETRY{Retries needed?}
    RETRY -- exhausted --> URX[URX: last response]
    RETRY -- ok --> OK[2xx, 3xx, or 4xx]

Patterns worth alerting on

Because response flags are access-log only, you need a log pipeline or a custom Lua/Wasm counter to alert on them. Once you have that, common severity guidance is:

• Any NR or NC in production. Configuration error. Page or ticket immediately.
• Sustained UO. A breaker is open. Investigate pool and limit configuration; usually the upstream is slow, not Envoy.
• Elevated UF, UC, or UT. Upstream is failing or slow. Correlate with upstream_cx_connect_fail, upstream_rq_rx_reset, upstream_rq_timeout.
• Baseline DC. Track it but do not page on it alone. Page only when it correlates with latency spikes.
• URX climbing with retry / total above 0.3. Retry storm. The retry policy is amplifying the failure.

Gotchas

• - is not “healthy”. A dash means Envoy did not set a flag. The request may still have failed (for example, a 500 forwarded from the upstream). The flag tells you why Envoy intervened, not whether the response was successful.
• DC is mostly noise. Mobile clients, page navigations, and request cancellations all produce DC. Investigate only when it correlates with latency.
• NR after an xDS push is a bad deploy. If NR appears right after a control plane push, the route table is wrong. Check update_rejected and the control plane push logs.
• Flags are access-log only. There is no Prometheus counter for UO. If you need real-time alerting, add a Lua filter that increments a custom stat, or feed logs to a pipeline that counts them.
• Multiple flags concatenate. A grep anchored on UO will not match UO,URX. Account for comma-separated values in log parsing.
• Newer flags exist. Envoy has added flags for DNS resolution failure, overload manager actions, drop overload, and others not covered here. If you see a flag you do not recognize, source/common/stream_info/utility.h in the Envoy repository is the canonical list.

How Netdata helps

• Correlate response flags with the stats that explain them. A spike in UO in the logs should be read against circuit_breakers.default.cx_open, rq_pending_open, upstream_rq_pending_overflow, and upstream_cx_active. Per-second metrics make the timing of the breaker trip visible against the upstream latency rise that caused it.
• Distinguish Envoy-generated 5xx from upstream-forwarded 5xx. cluster.<name>.upstream_rq_503 counts both. Overlay log-derived UO and NR counts on the 503 rate to see which fraction Envoy produced versus forwarded.
• Catch the upstream-side causes early. UF lines up with upstream_cx_connect_fail. UC lines up with upstream_rq_rx_reset. UT lines up with upstream_rq_timeout. Per-second collection shows these leading indicators before the flags appear in logs.
• Surface retry storms. URX in the logs pairs with upstream_rq_retry, upstream_rq_retry_overflow, and the upstream_rq_total / downstream_rq_total ratio. Watching these together tells you whether retries are helping or amplifying.
• Catch configuration regressions fast. Cross-check NR and NC against control_plane.connected_state, update_rejected, and listener_manager.listener_create_failure so a bad xDS push is visible within seconds, not at the next log scan.

Related guides

• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy connection pool exhaustion: a slow upstream that fills the pool
• Envoy health checks vs outlier detection: two systems that eject hosts differently
• How Envoy actually works in production: a mental model for operators
• Envoy membership_healthy dropping: reading the single most important cluster signal
• Envoy monitoring checklist: the signals every production proxy needs
• Envoy monitoring maturity model: from survival to expert
• Envoy no healthy upstream: the 503 when a cluster has no host to route to
• Envoy outlier detection mass ejection: when passive health checks empty a cluster
• Envoy panic threshold: why traffic routes to unhealthy hosts at 50%
• Envoy upstream_cx_active near max_connections: the pool filling up
• Envoy upstream_cx_connect_fail: failed TCP connections to upstream hosts