Envoy upstream_rq_time high: upstream latency as the proxy sees it

cluster.<name>.upstream_rq_time is the histogram that answers “how long did the upstream interaction take, as Envoy observed it?” When it climbs, the instinct is to page the backend team. That instinct is often wrong, or at least incomplete. The metric is wall-clock time measured at the HTTP router filter. It bundles several distinct latencies: TCP connect, upstream TLS handshake (for new connections), request transmission, upstream processing, and response transfer. It is not pure backend service time.

A second trap: when a request is retried and a later attempt succeeds, the histogram records only one sample — the total wall-clock duration from downstream request receipt to upstream response receipt, including all retry overhead. For client-observed latency, use downstream_rq_time.

A third trap: the default histogram buckets are extremely wide. The 100ms to 250ms to 500ms gaps mean histogram_quantile() in Prometheus interpolates across a large range, and percentile values can be coarse or misleading. Custom buckets aligned with your SLOs are necessary for accurate percentile work.

What this means

upstream_rq_time measures the time from when the downstream request is fully received by Envoy’s HTTP router filter until the entire upstream response has been received. It includes:

• upstream TCP connection establishment (when a new connection is needed, not pooled reuse)
• upstream TLS handshake (when upstream TLS is configured and a new connection is established)
• network round-trip to the upstream host
• upstream processing time
• response transfer time back to Envoy

It does not include Envoy’s downstream-side processing (filter chain execution, TLS termination on the client side, compression). The delta downstream_rq_time - upstream_rq_time gives a rough sense of Envoy’s own overhead, but this delta also includes downstream upload/download behavior, retry time across attempts, and buffering delays. Treat it as a trend signal, not a clean overhead measurement.

Each retried request produces a single upstream_rq_time sample: the total wall-clock time from downstream request receipt to upstream response receipt, including all retry attempts and back-off. The route-level request timeout (x-envoy-upstream-rq-timeout-ms) covers all attempts. If retries are firing, the histogram’s tail widens because each surviving request carries the cumulative retry overhead, not because more samples are added.

When the route timeout fires, Envoy aborts the request and increments upstream_rq_timeout, but the aborted request is not recorded in the upstream_rq_time histogram at all. The histogram only contains requests whose upstream response was fully received. A timeout that fires mid-retry-sequence can still produce a recorded value in the histogram if a later attempt succeeds, in which case the sample reflects the total elapsed time including the timed-out attempt.

The metric only exists for HTTP router filter paths. TCP proxy, UDP proxy, and CONNECT-UDP tunnels do not produce upstream_rq_time histograms because the router filter is not involved.

flowchart TD
    A["upstream_rq_time high"] --> B{"P50 also past baseline P99"}
    B -- "Yes, whole distribution shifted" --> C["Backend in severe distress"]
    B -- "No, only P99 tail moved" --> D{"Tail at or near timeout value"}
    D -- "Yes" --> E["Retry overhead shaping tail"]
    D -- "No" --> F{"upstream_cx_connect_ms also high"}
    F -- "Yes" --> G["Network or TLS handshake issue"]
    F -- "No" --> H{"Pool saturated, cx_active near max"}
    H -- "Yes" --> I["Connection pool exhaustion"]
    H -- "No" --> J{"upstream_rq much greater than downstream_rq"}
    J -- "Yes" --> K["Retry amplification"]
    J -- "No" --> L{"watchdog_miss nonzero"}
    L -- "Yes" --> M["Hot worker or CFS throttle"]
    L -- "No" --> N["Backend slow at tail, check per-host"]

Common causes

Quick checks

Run these read-only against the admin interface. Replace 9901 with 15000 for Istio sidecar deployments, and my_cluster with your cluster name.

# Pull the upstream_rq_time histogram for a specific cluster
curl -s http://localhost:9901/stats/prometheus | grep 'envoy_cluster_upstream_rq_time.*my_cluster'

# Raw admin format with all histogram buckets
curl -s http://localhost:9901/stats | grep 'cluster.my_cluster.upstream_rq_time'

# TCP connect time to upstream - distinguishes network from backend
curl -s http://localhost:9901/stats | grep 'cluster.my_cluster.upstream_cx_connect_ms'

# Connection pool state - is the pool saturated?
curl -s http://localhost:9901/stats | grep -E 'cluster.my_cluster.(upstream_cx_active|upstream_rq_pending_active|upstream_cx_total)$'

# Retry behavior - is amplification inflating the histogram?
curl -s http://localhost:9901/stats | grep -E 'cluster.my_cluster.upstream_rq_retry'

# Timeout counters - is the ceiling artifact present?
curl -s http://localhost:9901/stats | grep -E 'cluster.my_cluster.upstream_rq_(timeout|per_try_timeout)'

# Circuit breaker state - are we rejecting before even trying?
curl -s http://localhost:9901/stats | grep 'cluster.my_cluster.circuit_breakers'

# Compare to client-observed latency
curl -s http://localhost:9901/stats | grep 'downstream_rq_time'

# Worker thread health - is Envoy itself the bottleneck?
curl -s http://localhost:9901/stats | grep -E 'watchdog_miss|watchdog_mega_miss'

# Per-thread CPU (OS-level, not admin)
top -H -p $(pgrep -x envoy) -b -n 1 | head -20

How to diagnose it

1. Confirm the increase is real and cluster-scoped. Pull upstream_rq_time for the affected cluster specifically. Aggregate across all clusters hides which backend is slow. Look at P50 and P99 together. If only P99 moved, the tail is the problem. If P50 moved past the normal P99, the whole distribution shifted and the backend is in severe distress.

2. Check whether retry overhead is distorting the histogram. If the histogram’s tail extends toward the configured route timeout while upstream_rq_timeout and upstream_rq_per_try_timeout are incrementing, retry overhead is shaping the tail. Aborted requests (those that actually time out) do not appear in the histogram. The backend’s real tail latency for those requests is unknowable from this histogram alone.

3. Separate network from backend. Compare upstream_cx_connect_ms to upstream_rq_time. If connect time is also elevated, the network path or the upstream’s SYN backlog is the contributor. If connect time is flat but upstream_rq_time is up, the backend is the contributor. For mTLS upstreams, the TLS handshake cost is folded into connect time, so a certificate or cipher regression will show up here.

4. Check the connection pool. If upstream_cx_active is near max_connections and upstream_rq_pending_active > 0, the pool is saturated. New requests wait for a connection, and the wait is included in upstream_rq_time. High upstream_cx_total rate means connections are being churned (not reused), which adds TCP and TLS cost to every request. For HTTP/1.1 upstreams without keepalive, every request pays this cost.

5. Check for retry amplification. Compute upstream_rq_total / downstream_rq_total. If above 1.3, retries are inflating upstream load. If above 2.0, it is a retry storm. Each retry is a separate upstream_rq_time sample, so retries make the histogram look busier without representing the client experience. Check upstream_rq_retry_success to see whether retries are even helping. If retry success rate is low and retry rate is high, retries are accelerating the failure.

6. Check circuit breaker and outlier detection state. If circuit_breakers.default.cx_open or rq_pending_open is 1, Envoy is fast-failing new requests with 503 (response flag UO). These fast-fails are recorded at near-zero latency, which can pull the histogram down. If latency “improves” during an incident, check whether error rate spiked: the improvement may be circuit-breaker rejections, not backend recovery.

7. Check worker thread health. Aggregate Envoy CPU can look moderate while one worker is at 100%. Requests on that worker suffer event-loop queuing delay, which shows up in upstream_rq_time and downstream_rq_time alike. Run top -H -p $(pgrep -x envoy) to see per-thread CPU. Check server.watchdog_miss and server.watchdog_mega_miss. In Kubernetes, also check for CFS throttling, which causes the same symptom without Envoy being CPU-saturated internally.

8. Check HTTP/2-specific issues. For HTTP/2 upstreams, low upstream_cx_active with high RPS is normal (multiplexing). But if max_concurrent_streams on the upstream is low, or if one slow stream is blocking others on the same connection, latency can spike. Per-host breakdown via GET /clusters?format=json reveals whether the problem is host-specific.

Metrics and signals to monitor

Fixes

Backend service degradation

Address the backend. Envoy is reporting accurately. Check the backend’s own GC pauses, database contention, disk I/O, and resource saturation. Do not raise Envoy’s circuit breaker limits to “absorb” a slow backend. That removes Envoy’s protection without fixing the cause, and the next failure involves an unprotected upstream collapsing completely.

Network latency increase

Investigate the network path. Cross-AZ or cross-region traffic adds fixed latency. Check whether routing changed, whether security groups or firewall rules shifted, or whether DNS is resolving upstreams to distant endpoints. For mTLS, check whether the TLS handshake cost included in upstream_cx_connect_ms has inflated due to a cipher or certificate regression.

Connection pool contention

If upstream_cx_active is near max_connections and the pool is saturated, the upstream is slow, which holds connections longer, which fills the pool. The fix is the upstream, not the limit. Temporarily raising max_connections can buy headroom during a spike, but the underlying problem is that connections are being held longer than the design assumes. For HTTP/1.1 upstreams, ensure keepalive is enabled; without it, every request pays TCP and TLS cost. Check upstream_cx_total rate for churn caused by idle timeout mismatches between Envoy and the upstream.

Retry amplification

If upstream_rq_total / downstream_rq_total is high and retries are not succeeding, disable or reduce retries for the affected cluster (for example by pushing upstream.use_retry to 0 via the runtime layer, or by removing the retry policy via xDS). Retries on non-idempotent requests can cause duplicate operations. Review the retry policy: which status codes trigger retries, how many attempts, and whether a retry budget (max_retries circuit breaker) is configured. During an active retry storm, temporarily disabling retries via runtime flag or xDS config push stops the amplification immediately.

Retry overhead shaping the tail

If the histogram’s tail extends toward the route timeout while upstream_rq_timeout is incrementing, retry overhead is contributing samples. Aborted requests are not in the histogram; only requests that eventually complete (including retried ones) contribute. Either the backend needs to be faster (the real fix) or the timeout needs to reflect the actual SLO (if the current value is too aggressive). Do not raise the timeout to “make the metric look better.” A higher timeout means clients wait longer before getting an error. Tune the timeout to the SLO, then fix the backend.

HTTP/2 head-of-line blocking

If HTTP/2 upstreams show latency spikes tied to specific connections, check max_concurrent_streams on the upstream. A low limit with high concurrency forces serialization. Consider whether HTTP/1.1 with a larger pool would actually be faster for this workload. Check for TCP-level issues (retransmits, window size) that affect one connection and all its multiplexed streams.

Hot worker thread

Identify the hot thread via top -H. Check for expensive filters (Lua, Wasm, body manipulation), synchronous DNS or ext_authz calls, or stats-sink contention. In Kubernetes, check for CFS throttling via cgroup stats (cpu.stat), which causes watchdog_miss without internal CPU saturation. If one worker is handling disproportionate load due to SO_REUSEPORT imbalance, that is a deployment-level problem.

Prevention

• Configure custom histogram buckets. The default buckets are 0.5, 1, 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000, 30000, 60000, 300000, 600000, 1800000, and 3600000 ms, with wide gaps in the 100-1000 ms range. Set buckets that bracket your SLO targets so percentile calculations are meaningful.
• Alert on baseline deviation, not fixed thresholds. Different services have different baselines. P99 greater than 2x rolling-24h average is more robust than a fixed millisecond cutoff. P50 greater than baseline P99 signals the entire distribution has shifted.
• Monitor the connection pool with headroom in mind. Track upstream_cx_active / max_connections and upstream_rq_pending_active. The pending queue is the leading indicator before 503s appear. Any sustained nonzero pending_active means the pool is becoming a bottleneck.
• Track retry ratios as a standard dashboard metric. The ratio upstream_rq_retry / upstream_rq_total should be a first-class signal. A sudden jump is often the first sign of upstream degradation, before latency or error rates move.
• Separate upstream_rq_time from upstream_cx_connect_ms. The ratio between them distinguishes network issues from application issues. If connect time is a growing fraction of total time, the problem is in the network path or connection establishment, not the backend.
• Watch for the retry-overhead tail. If upstream_rq_timeout is nonzero in steady state, your timeout is too tight or your backend is too slow. Either way, aborted requests are excluded from the histogram, so the real tail is worse than what the histogram shows.
• Monitor per-worker CPU, not just aggregate. Aggregate Envoy CPU masks single-threaded-per-worker behavior. watchdog_miss is the canary for event-loop blocking that does not show up in aggregate CPU.

How Netdata helps

• Per-second collection of upstream_rq_time, downstream_rq_time, and upstream_cx_connect_ms on the same timeline. Correlating the three side by side separates backend latency from network latency from Envoy-side overhead without manual curl sampling.
• Connection pool signals (upstream_cx_active, upstream_rq_pending_active, upstream_rq_pending_overflow) and circuit breaker gauges (cx_open, rq_pending_open) are collected per cluster. The pending queue depth is the leading indicator before 503s, and per-second resolution makes the cliff-edge visible.
• Retry signals (upstream_rq_retry, retry_success, retry_overflow) and the upstream-to-downstream request rate ratio are tracked continuously. A retry storm shows up as a ratio spike before error rates cascade.
• Timeout counters (upstream_rq_timeout, upstream_rq_per_try_timeout) are collected alongside the latency histogram, so the timeout ceiling artifact is distinguishable from real backend latency.
• Anomaly detection on the latency histogram flags baseline deviations that fixed thresholds miss, which matters for upstream_rq_time where baselines vary widely across services and traffic patterns.
• Worker thread health (watchdog_miss, watchdog_mega_miss) and per-second CPU per container surface hot-worker and CFS-throttling issues that aggregate CPU hides.

Related guides

• Envoy 502 and upstream resets: rx_reset, tx_reset, and mid-response failures
• Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend
• Envoy 504 upstream timeout: upstream_rq_timeout, per-try timeouts, and the UT flag
• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy connection pool exhaustion: a slow upstream that fills the pool
• Envoy health checks vs outlier detection: two systems that eject hosts differently
• How Envoy actually works in production: a mental model for operators
• Envoy membership_healthy dropping: reading the single most important cluster signal
• Envoy monitoring checklist: the signals every production proxy needs
• Envoy monitoring maturity model: from survival to expert
• Envoy no healthy upstream: the 503 when a cluster has no host to route to
• Envoy outlier detection mass ejection: when passive health checks empty a cluster