Envoy upstream connect error or disconnect/reset before headers: reading the reset reason

The 503 body upstream connect error or disconnect/reset before headers. reset reason: <reason> means Envoy selected an upstream host, tried to open a request stream on a pooled connection, and the stream was reset before any response headers came back. The body looks generic, but the trailing reset reason: field is the diagnostic payload. It is one of the values in Envoy’s StreamResetReason enum — 14 values in the current source, including ConnectError, ProtocolError, OverloadManager, Http1PrematureUpstreamHalfClose, and RemoteResetNoError — and each value points at a different failure mechanism.

The reset reason usually maps to a single access-log response flag, which is what monitoring pipelines actually surface. ConnectionFailure becomes UF, ConnectionTermination becomes UC, Overflow becomes UO, ProtocolError becomes UPE, the local reset variants become LR, and the remote reset variants (including ConnectError) become UR. The body and the flag are two views of the same event: the body is what the client sees, the flag is what you correlate against cluster stats.

The trap is treating all of these as “upstream is down”. A host refusing TCP connections (UF) has a different root cause and remediation than an idle-timeout mismatch closing sockets mid-flight (UC), or a circuit breaker fast-failing because the per-cluster pool is saturated (UO). The sections below cover how to read the reset reason, map it to the right signal, and stop chasing the wrong subsystem.

What this means

Envoy has already picked a host (this is not the “no healthy upstream” NR/UH case) and attempted to open a request stream on a connection from the per-worker, per-host pool. The reset happened before any response headers arrived.

flowchart TD
  A["503 body:
upstream connect error or
disconnect/reset before headers"] --> B{"Read trailing
reset reason"}
  B -->|ConnectionFailure| C["UF: TCP connect failed
Check upstream_cx_connect_fail"]
  B -->|ConnectionTermination| D["UC: socket closed or RST
Check idle timeout, mTLS"]
  B -->|Overflow| E["UO: circuit breaker tripped
Check cx_open, rq_pending_open"]
  B -->|RemoteReset / ConnectError| F["UR: upstream sent RST
Check upstream_rq_rx_reset"]
  B -->|ProtocolError| G["UPE: framing mismatch
Check http2_protocol_options"]
  B -->|LocalReset| H["LR: Envoy reset locally
Check overload manager, filters"]

Notes that affect how you read it:

• In access logs, the same event appears as response_code_details = upstream_reset_before_response_started{<reason>} plus the matching flag in %RESPONSE_FLAGS%. Envoy warns that response-code detail strings are not stable across versions; the response flags are.
• Multiple flags can coexist on a single log line. UC,URX means the upstream closed the connection and the retry budget was also exhausted.
• The body string and the access-log detail come from the same code path. If you only have the body (a client screenshot, for instance), the reset reason is enough to start the diagnostic.

Common causes

Quick checks

Run these on a suspect Envoy instance. They are read-only. Use port 9901 for standalone Envoy and 15000 for Istio sidecars.

# Confirm the response flag and detail string for recent 503s.
# Look in the access log path for your deployment. In Istio this is sidecar stdout.
# Sample fields: response_code, %RESPONSE_FLAGS%, response_code_details.

# Is the cluster's host set healthy?
curl -s http://localhost:9901/stats | grep -E 'cluster.<name>.(membership_healthy|membership_total)'

# Are TCP connects actually failing?
curl -s http://localhost:9901/stats | grep -E 'cluster.<name>.(upstream_cx_connect_fail|upstream_cx_connect_timeout)'

# Are resets coming from upstream (rx) or from Envoy (tx)?
curl -s http://localhost:9901/stats | grep -E 'cluster.<name>.(upstream_rq_rx_reset|upstream_rq_tx_reset)'

# Is a circuit breaker currently open?
curl -s http://localhost:9901/stats | grep 'circuit_breakers'

# Is the pending queue saturating?
curl -s http://localhost:9901/stats | grep -E 'cluster.<name>.(upstream_rq_pending_active|upstream_rq_pending_overflow)'

# Per-host view of the cluster (health flags, ejection state).
curl -s http://localhost:9901/clusters?format=json | jq '.cluster_statuses[] | select(.name=="<name>") | .host_statuses[] | {address: .address, health_flags: .health_status}'

# Is the overload manager intervening (LR / local reset path)?
curl -s http://localhost:9901/stats | grep 'overload'

How to diagnose it

1. Pull the exact reset reason out of the 503 body or access log. The body gives you the trailing string (connection failure, connection termination, overflow, remote reset, protocol error, and so on). If you only have access logs, read %RESPONSE_FLAGS% and response_code_details. Do not proceed without one of these two signals; a raw 503 count is not enough.

2. Map the reason to a response flag and a subsystem.

   • UF: data plane cannot reach the host. Network, host process, or FD exhaustion on the upstream.
   • UC: a connection that was usable got torn down before the stream completed. Usually idle-timeout mismatch, mTLS renegotiation, or HTTP/1.1 close without Connection: close.
   • UO: Envoy is rejecting locally because the cluster hit a circuit breaker. The upstream may be perfectly healthy but slow.
   • UR: upstream-initiated reset. Backend is shutting down, refusing streams, or sending GOAWAY.
   • UPE: protocol mismatch. Suspect http2_protocol_options, ALPN, or plaintext-vs-TLS on the upstream.
   • LR: Envoy itself refused the stream. Check overload manager actions and filter denials.

3. Correlate the flag with cluster stats. A flag without a moving counter is a single blip. You want the sustained pattern:

   • UF plus climbing upstream_cx_connect_fail and falling membership_healthy is a real outage.
   • UO plus circuit_breakers.default.cx_open=1 or rq_pending_open=1 plus a growing upstream_rq_pending_active is a slow-upstream / pool-exhaustion cascade.
   • UC plus rising upstream_rq_rx_reset with stable membership_healthy points at connection lifecycle problems, not host death.
   • UPE plus cluster.<name>.ssl.connection_error or ssl.fail_verify_error suggests TLS or ALPN negotiation is wrong.

4. Decide whether the upstream is actually broken. Cross-check membership_healthy / membership_total and outlier_detection.ejections_active. If hosts are healthy and not ejected, the problem is in the connection lifecycle, the breaker config, or the protocol layer, not the backend dying.

5. Look at the deploy and config timeline. UC, UR, and UPE clusters often correlate with backend rollouts, mTLS policy changes, or xDS pushes. update_rejected on the cluster is a separate but adjacent failure: Envoy is connected to the control plane but silently NACKing bad config.

Metrics and signals to monitor

Note: the upstream_rq_pending_overflow counter is no longer incremented by default for the active-request circuit breaker path in recent Envoy versions. The runtime flag envoy.reloadable_features.skip_pending_overflow_count_on_active_rq controls this (confirmed in the Envoy source and release notes; introduced with the upstream_rq_active_overflow counter in 1.38.0). If you rely on that counter, verify it is still being incremented in your build.

Fixes

Connection failure (UF)

The TCP connect itself failed. The host is not accepting connections: process down, port not listening, FD exhaustion on the upstream, firewall/ACL change, or, in containers, the cluster endpoint points at 127.0.0.1 from inside a network namespace where loopback is not the host’s loopback.

• Verify the host process is up and listening on the expected port.
• Confirm security groups, network policies, and ACLs allow Envoy to reach the upstream.
• For Docker-based Envoy, do not point clusters at 127.0.0.1 unless you are using --network host or referencing sibling containers by service name.
• Watch upstream_cx_connect_timeout alongside connect_fail. Timeouts (SYN with no SYN-ACK) imply network path issues; immediate failures (RST) imply the port is closed.

Connection termination (UC)

A previously good connection was torn down before the stream completed. Three common patterns:

• Idle timeout mismatch. Envoy holds an idle pooled connection longer than the upstream’s idle timeout. The upstream closes the socket; Envoy reuses it for a new stream before noticing the FIN, and the stream dies with connection termination. Align Envoy’s idle_timeout with (or below) the upstream’s. See the connection churn guide for the reuse-ratio view.
• HTTP/1.1 close without Connection: close. A known race exists where an upstream closes the TCP connection without signaling. Ensure upstreams send Connection: close when they intend to close.
• Strict mTLS. In Istio, STRICT peer authentication can produce connection termination when the sidecar and application are not configured to speak TLS to each other. Check cluster.<name>.ssl.connection_error and the peer-auth policy.

A low, sporadic UC rate (for example, 1-2 per 100k requests) that you cannot reproduce is almost always upstream-initiated close behavior, not an Envoy bug.

Overflow (UO)

A circuit breaker is fast-failing requests. Default limits are max_connections=1024, max_pending_requests=1024, max_requests=1024, max_retries=3. Tripping them means the pool is saturated, usually because the upstream got slow, not because Envoy is broken.

• Check circuit_breakers.default.cx_open, rq_pending_open, rq_open to identify which limit is hit.
• Confirm the upstream is slow (upstream_rq_time climbing) before touching limits.
• Raising limits without fixing the upstream removes protection and makes the next failure worse. Add capacity or fix the backend first.
• Verify whether upstream_rq_pending_overflow is still meaningful in your Envoy build (see note above).

Remote reset (UR)

The upstream sent an explicit reset: HTTP/2 RST_STREAM or GOAWAY, a refused stream, or a ConnectError on a new connection. Common during graceful shutdown, backend overload, or backend-side bugs.

• Cross-reference with backend deploy windows and backend logs.
• Watch upstream_rq_rx_reset to confirm directionality.
• If a single host dominates, outlier detection should be ejecting it; check outlier_detection.ejections_active.

Protocol error (UPE)

Protocol framing failed. The cluster is speaking a different protocol than Envoy expects.

• For gRPC-web or HTTP/2 upstreams, verify http2_protocol_options is set on the cluster and that the upstream actually speaks HTTP/2.
• Check ALPN negotiation and whether upstream TLS is configured when the backend expects plaintext (or vice versa).
• cluster.<name>.ssl.connection_error and ssl.fail_verify_error will move if the cause is the TLS layer rather than pure framing.

Local reset (LR)

Envoy itself refused the stream. The usual causes are overload manager actions (stop_accepting_requests, disable_http_keepalive), filter denials, or local resource exhaustion.

• Check server.overload_manager.envoy.overload_actions.*.active. Any active=1 means Envoy is degrading to survive.
• Check server.memory_allocated against the container limit and max_heap_size_bytes.
• If overload manager is not configured, Envoy has no self-protection and will OOM instead of resetting gracefully. Configure it.

Prevention

• Alert on response flags, not just 503 counts. UF, UC, UO, UR, UPE, and LR each have different playbooks. A flat 5xx rate tells you nothing actionable.
• Treat upstream_rq_pending_active as the leading indicator for UO. By the time pending_overflow is nonzero, users are already failing.
• Align idle timeouts. Envoy’s upstream idle_timeout should be shorter than the upstream’s own idle close to avoid UC from late FIN detection.
• Verify circuit breaker headroom during normal load. Track upstream_cx_active / max_connections. If you are consistently above 80%, you have no burst absorption.
• Monitor both health checks and outlier detection. membership_healthy shows active-probe health; outlier_detection.ejections_active shows real-traffic health. A host can pass one and fail the other.
• Watch the xDS path. control_plane.connected_state=0 and update_rejected both produce stale config that eventually surfaces as 503s for new routes or endpoints.
• Track mTLS cert runway. In service mesh, silent SDS failure expires certs and then breaks TLS catastrophically across the mesh.
• Enable overload manager in every production deployment. Without it, Envoy has no graceful degradation path under memory pressure.

How Netdata helps

• Per-second collection of upstream_cx_connect_fail, upstream_cx_connect_timeout, upstream_rq_rx_reset, and upstream_rq_tx_reset lets you see the directionality of resets at the resolution of a single blip, not a 30-second scrape window.
• Circuit breaker gauges (cx_open, rq_pending_open, rq_open) and the pending queue (upstream_rq_pending_active, upstream_rq_pending_overflow) sit on the same dashboard, so a UO diagnosis is a visual check rather than a multi-command session.
• membership_healthy, membership_total, and outlier_detection.ejections_active sit alongside connect-fail counters, so you can distinguish “host is dead” (UF) from “host is ejected by real-traffic errors” without leaving the chart.
• ML anomaly detection on upstream_rq_time surfaces the slow-upstream trend that precedes an Overflow cascade, giving you minutes of warning before the breaker trips.
• Overload manager action gauges and server.memory_allocated against the container limit explain LR resets in real time, including the CFS-throttling case in Kubernetes where CPU limits cause watchdog misses without obvious CPU saturation.

Related guides

• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy connection pool exhaustion: a slow upstream that fills the pool
• Envoy health checks vs outlier detection: two systems that eject hosts differently
• How Envoy actually works in production: a mental model for operators
• Envoy membership_healthy dropping: reading the single most important cluster signal
• Envoy monitoring checklist: the signals every production proxy needs
• Envoy monitoring maturity model: from survival to expert
• Envoy no healthy upstream: the 503 when a cluster has no host to route to
• Envoy outlier detection mass ejection: when passive health checks empty a cluster
• Envoy panic threshold: why traffic routes to unhealthy hosts at 50%
• Envoy connection churn: a low reuse ratio and keepalive misconfiguration
• Envoy upstream_cx_active near max_connections: the pool filling up