Envoy 502 and upstream resets: rx_reset, tx_reset, and mid-response failures

A client request made it past Envoy’s routing, established a TCP connection to an upstream host, and then the connection died before the response completed. The access log shows a UC or UPE response flag, upstream_rq_rx_reset is climbing, and the client received a 502 or 503.

This is a different failure class from “no healthy upstream” (503, NR flag) or “connection refused” (503, UF flag). Those happen before a connection is established. Resets happen after the handshake succeeds, which means the upstream was reachable and then failed during request processing. The diagnosis and fix are completely different.

Operators often call these “502 errors,” but Envoy’s response code depends on when and why the reset happened. Most of these are actually 503s with a UC flag, not 502s. A true 502 requires a UPE flag (protocol error). The distinction matters because UC points to the upstream process or network path, while UPE points to a protocol mismatch.

What this means

Envoy tracks two reset counters that tell you who initiated the teardown:

• upstream_rq_rx_reset: the upstream sent the reset. Envoy received a RST from the remote side. This fires when the upstream process crashes, restarts, closes an idle connection out from under Envoy, sends an HTTP/2 RST_STREAM, or sends a GOAWAY frame. A companion counter, upstream_rq_rx_reset_no_error, distinguishes graceful resets (HTTP/2 GOAWAY with NO_ERROR) from error-driven RSTs.

• upstream_rq_tx_reset: Envoy itself reset the upstream connection. This fires when a circuit breaker trips, a local timeout expires, or the overload manager intervenes. The upstream did not initiate this failure, Envoy did.

The response code the client sees depends on the timing and cause of the reset:

The UC flag returns 503. A true 502 from this path requires the UPE flag, which fires when the upstream sends malformed protocol data. If your dashboards label all of these as “502s,” you are likely looking at 503s with UC flags.

flowchart TD
    A["Envoy sends request to upstream"] --> B{"Reset before response headers?"}
    B -- "Yes, upstream RST or close" --> C["rx_reset increments
503 UC (connection_termination)
or 502 UPE (protocol_error)"]
    B -- "Yes, Envoy local reset" --> D["tx_reset increments
503 with local flag
(circuit breaker, timeout, overload)"]
    B -- "No" --> E{"Reset after headers started?"}
    E -- "Yes, upstream resets mid-response" --> F["rx_reset increments
Downstream gets truncated response
rq_reset_after_downstream_response_started"]
    E -- "No" --> G["Response completes normally"]

Metrics gap to know about. When a reset occurs before response headers are received, Envoy sends a local reply to the client and charges the local reply status code to the upstream_rq_5xx (or upstream_rq_2xx) cluster HTTP response code counters, alongside upstream_rq_completed. The reset-specific counters upstream_rq_rx_reset and upstream_rq_tx_reset tell you that the response was a local reply due to a reset, not a genuine upstream response. If your error-rate dashboards rely solely on upstream_rq_5xx, they cannot distinguish real upstream 5xx responses from Envoy local replies caused by resets. Always correlate with the reset counters.

Common causes

Quick checks

All commands below query the Envoy admin API (read-only). Ensure the admin port is not exposed outside localhost.

# rx_reset and tx_reset rates for a specific cluster
curl -s http://localhost:9901/stats | grep -E 'cluster\.my_cluster\.upstream_rq_(rx_reset|tx_reset|total)'

# Connection-level reset context (stream reset vs full connection teardown)
curl -s http://localhost:9901/stats | grep -E 'upstream_cx_destroy_(remote|local)_with_active_rq'

# Circuit breaker state (rules out tx_reset from breaker trips)
curl -s http://localhost:9901/stats | grep 'circuit_breakers'

# Timeout stats (rules out tx_reset from local timeouts)
curl -s http://localhost:9901/stats | grep -E 'upstream_rq_(timeout|per_try_timeout)'

# Upstream health (rules out host-level failure)
curl -s http://localhost:9901/stats | grep -E 'membership_healthy|membership_total'

# TLS errors on the upstream path (rules out mTLS issues)
curl -s http://localhost:9901/stats | grep 'ssl\.'

# Reset ratio: take two samples 10s apart and compute delta_rx_reset / delta_upstream_rq_total

In Istio sidecar mode, the admin port is 15000, not 9901. The health endpoint is on port 15021 at /healthz/ready.

How to diagnose it

1. Compute the reset ratio. Divide upstream_rq_rx_reset (or upstream_rq_tx_reset) by upstream_rq_total over the same window. The absolute count is meaningless without traffic context. A ratio above 0.001 (0.1%) sustained on a healthy cluster warrants investigation. During deploys, short spikes to 1-5% are common and self-resolving.

2. Determine who is resetting. If rx_reset is climbing, the upstream is tearing down connections. If tx_reset is climbing, Envoy is tearing them down locally. These have different root causes and different fixes. Check both counters.

3. Check the access log response flags. UC confirms upstream-initiated termination. UO points to a circuit breaker. UT points to a timeout. UPE points to a protocol error. Multiple flags can be set simultaneously (for example, UC,URX means the upstream closed and retries were exhausted). Response flags are access-log only, they are not exposed as aggregate Prometheus stats.

4. Check the response code detail string. If your access log format includes %RESPONSE_CODE_DETAILS%, look for upstream_reset_before_response_started{connection_termination} vs {protocol_error}. This string tells you the exact reset reason. For protocol errors, the detail includes {protocol_error}. If the codec provided a transport failure reason, it appears after a pipe separator (for example {protocol_error|HTTP/2 framing error}); when no transport reason was recorded, the detail shows only {protocol_error}.

5. Correlate with connection-level stats. Check upstream_cx_destroy_remote_with_active_rq alongside upstream_rq_rx_reset. If both spike together, the upstream is killing entire connections with active requests, not just individual streams. This points to process-level failure (crash, OOM, restart) rather than per-request issues.

6. Check timing against events. Correlate the reset spike with deployment timestamps, certificate rotation events, upstream scaling events, and network infrastructure changes. Most rx_reset spikes during deploys are expected and self-resolve.

7. Rule out the upstream being genuinely broken. Check upstream_rq_time for latency shifts, upstream_cx_connect_fail for connectivity issues, and membership_healthy for host ejection. If the upstream is slow or failing health checks, resets are a symptom, not the root cause.

Metrics and signals to monitor

The stat name in the docs is rq_reset_after_downstream_response_started (without a downstream_ prefix) under http.<stat_prefix>.

Fixes

rx_reset from backend deploys or restarts

The most common and usually least concerning cause. During a rolling deploy, old upstream instances receive SIGTERM and close active connections. Envoy sees these as resets. If the deploy is graceful and new instances come up quickly, the spike resolves on its own.

If the spike is sustained or customer-visible, check whether the upstream sends a graceful shutdown signal (HTTP/2 GOAWAY) that lets Envoy drain before closing. Without GOAWAY, Envoy has no warning before the RST arrives. Consider adding retry-on-reset policy if the deploy window is brief and the requests are idempotent.

rx_reset from idle timeout mismatch

Envoy holds idle upstream connections in its pool for reuse. If the upstream closes idle connections faster than Envoy expects (for example, the upstream’s keepalive timeout is 30s but Envoy’s idle_timeout is 60s), Envoy will occasionally send a request on a connection the upstream is about to close or has already closed. This produces a low, steady rate of rx_reset with no deploy correlation.

Fix: align the timeouts. Envoy’s upstream idle_timeout should be shorter than the upstream’s keepalive timeout so Envoy closes the connection first. This is a common source of sporadic 1-in-100k reset errors.

rx_reset from upstream crash or OOM

If the upstream process is crashing or getting OOM-killed, the resets are a symptom. Check the upstream’s own logs and metrics, not Envoy’s. Envoy is correctly reporting that the connection died. The fix is in the upstream: address the memory leak, resource exhaustion, or crash bug.

tx_reset from circuit breaker trips

When tx_reset correlates with circuit_breakers.*.cx_open = 1 or rq_pending_open = 1, Envoy is resetting upstream connections because it hit a configured limit. The response flag is UO. This is Envoy protecting itself and the upstream from overload.

Do not reflexively increase the circuit breaker limits. The circuit breaker is a symptom of the upstream being slow or the pool being undersized. Check upstream_rq_time for latency growth and upstream_cx_active against max_connections. If the upstream is genuinely slow, fix the upstream. If the limits are genuinely too low for normal traffic, then adjust the limits. See the circuit breaker guide for detailed triage.

tx_reset from timeouts

When tx_reset correlates with upstream_rq_timeout or upstream_rq_per_try_timeout, the upstream did not respond within the configured timeout window and Envoy gave up. The response flag is UT. Check whether the timeout is appropriate for the workload and whether the upstream’s latency distribution has shifted. A per-try timeout (x-envoy-upstream-rq-per-try-timeout-ms) fires per attempt, while the overall request timeout covers all retries combined.

Mid-response resets (truncated responses)

When the upstream resets after response headers have started flowing downstream, the client receives a partial or truncated response. There is no clean HTTP status code for this case because headers were already sent. This is particularly damaging for gRPC streaming, SSE, and long-poll workloads.

The router filter tracks these via rq_reset_after_downstream_response_started. Monitor this counter separately for streaming workloads. The access log detail is upstream_reset_after_response_started{remote_reset}. The root causes are the same as pre-header resets (crash, deploy, network issue), but the user impact is worse because the client has already started consuming the response.

Prevention

• Align idle timeouts. Set Envoy’s upstream idle_timeout lower than the upstream’s keepalive timeout. This eliminates the most common source of sporadic baseline resets.

• Configure retry-on-reset policy. The router filter’s retry policies cover disconnects and resets. The 5xx policy retries on disconnect, reset, and read timeout. The reset policy retries on any disconnect, reset, or read timeout specifically. The gateway-error policy covers 502, 503, 504 plus disconnect/reset. Configure retries only for idempotent requests, and set a per-try timeout so retries have time to complete within the overall request budget.

• Alert on reset ratio, not absolute count. Alert on upstream_rq_rx_reset / upstream_rq_total exceeding a threshold (for example, 0.5% sustained for 5 minutes with a meaningful traffic floor). Absolute counts are meaningless without traffic context.

• Include %RESPONSE_FLAGS% and %RESPONSE_CODE_DETAILS% in access logs. Without these fields, you cannot distinguish a UC reset from a UO circuit breaker trip from a UPE protocol error. These are the single most important debugging signals Envoy provides for this failure class.

• Track protocol errors separately. A 502 UPE is a different problem from a 503 UC. Protocol errors point to upstream format changes, header size violations, or HTTP/2 framing issues.

Monitoring with Netdata

• Per-second reset counters show exactly when upstream_rq_rx_reset or upstream_rq_tx_reset begins climbing, at the granularity needed to correlate with deploy events, upstream latency shifts, or circuit breaker state changes.

• Computed reset ratios (rx_reset / upstream_rq_total, tx_reset / upstream_rq_total) as derived metrics, so alerts fire on proportional impact rather than absolute counts that scale with traffic.

• Baseline-aware alerting on the reset ratio distinguishes a deploy-correlated spike (expected) from a genuine trend shift, reducing alert noise during routine rollouts.

• Correlation with connection-level stats (upstream_cx_destroy_remote_with_active_rq, upstream_cx_active, circuit_breakers.*_open) in a single timeline view, shortening the path from “resets are up” to “the upstream is killing connections” or “the circuit breaker is tripping.”

• Upstream latency histograms (upstream_rq_time) alongside reset counters, to determine whether resets correlate with slow upstreams (backend degradation) or occur independently (crashes, deploys, timeout mismatches).

Related guides

• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy connection pool exhaustion: a slow upstream that fills the pool
• Envoy health checks vs outlier detection: two systems that eject hosts differently
• How Envoy actually works in production: a mental model for operators
• Envoy membership_healthy dropping: reading the single most important cluster signal
• Envoy monitoring checklist: the signals every production proxy needs
• Envoy monitoring maturity model: from survival to expert
• Envoy no healthy upstream: the 503 when a cluster has no host to route to
• Envoy outlier detection mass ejection: when passive health checks empty a cluster
• Envoy panic threshold: why traffic routes to unhealthy hosts at 50%
• Envoy connection churn: a low reuse ratio and keepalive misconfiguration
• Envoy upstream_cx_active near max_connections: the pool filling up