Envoy upstream_rq_time high: upstream latency as the proxy sees it

cluster.<name>.upstream_rq_time is the histogram that answers “how long did the upstream interaction take, as Envoy observed it?” When it climbs, the instinct is to page the backend team. That instinct is often wrong, or at least incomplete. The metric is wall-clock time measured at the HTTP router filter. It bundles several distinct latencies: TCP connect, upstream TLS handshake (for new connections), request transmission, upstream processing, and response transfer. It is not pure backend service time.

A second trap: upstream_rq_time records each retry attempt as a separate measurement. If a request is retried three times and the third succeeds, you get three latency samples, none of which represents the client’s actual wait. For client-observed latency, use downstream_rq_time.

A third trap: the default histogram buckets are extremely wide. The 100ms to 250ms to 500ms gaps mean histogram_quantile() in Prometheus interpolates across a large range, and percentile values can be coarse or misleading. Custom buckets aligned with your SLOs are necessary for accurate percentile work.

What this means

upstream_rq_time measures the time from when the downstream request is fully received by Envoy’s HTTP router filter until the entire upstream response has been received. It includes:

• upstream TCP connection establishment (when a new connection is needed, not pooled reuse)
• upstream TLS handshake (when upstream TLS is configured and a new connection is established)
• network round-trip to the upstream host
• upstream processing time
• response transfer time back to Envoy

It does not include Envoy’s downstream-side processing (filter chain execution, TLS termination on the client side, compression). The delta downstream_rq_time - upstream_rq_time gives a rough sense of Envoy’s own overhead, but this delta also includes downstream upload/download behavior, retry time across attempts, and buffering delays. Treat it as a trend signal, not a clean overhead measurement.

Each retry attempt is a separate sample in upstream_rq_time. The route-level request timeout (x-envoy-upstream-rq-timeout-ms) covers all attempts, but the histogram does not record the cumulative retry time. If retries are firing, the histogram looks busier than the client experience, with each retry counted at its own duration rather than the sum.

When upstream_rq_timeout fires, the request is recorded at the timeout duration. This creates an artificial ceiling in the histogram: if your timeout is 15s, you will see a pile of samples near 15s and nothing above it. The backend did not “get faster at 15s.” Envoy gave up.

The metric only exists for HTTP router filter paths. TCP proxy, UDP proxy, and CONNECT-UDP tunnels do not produce upstream_rq_time histograms because the router filter is not involved.

flowchart TD
    A["upstream_rq_time high"] --> B{"P50 also past baseline P99"}
    B -- "Yes, whole distribution shifted" --> C["Backend in severe distress"]
    B -- "No, only P99 tail moved" --> D{"Pile of samples at timeout value"}
    D -- "Yes" --> E["Timeout ceiling artifact"]
    D -- "No" --> F{"upstream_cx_connect_ms also high"}
    F -- "Yes" --> G["Network or TLS handshake issue"]
    F -- "No" --> H{"Pool saturated, cx_active near max"}
    H -- "Yes" --> I["Connection pool exhaustion"]
    H -- "No" --> J{"upstream_rq much greater than downstream_rq"}
    J -- "Yes" --> K["Retry amplification"]
    J -- "No" --> L{"watchdog_miss nonzero"}
    L -- "Yes" --> M["Hot worker or CFS throttle"]
    L -- "No" --> N["Backend slow at tail, check per-host"]

Common causes

Quick checks

Run these read-only against the admin interface. Replace 9901 with 15000 for Istio sidecar deployments, and my_cluster with your cluster name.

# Pull the upstream_rq_time histogram for a specific cluster
curl -s http://localhost:9901/stats/prometheus | grep 'envoy_cluster_upstream_rq_time.*my_cluster'

# Raw admin format with all histogram buckets
curl -s http://localhost:9901/stats | grep 'cluster.my_cluster.upstream_rq_time'

# TCP connect time to upstream - distinguishes network from backend
curl -s http://localhost:9901/stats | grep 'cluster.my_cluster.upstream_cx_connect_ms'

# Connection pool state - is the pool saturated?
curl -s http://localhost:9901/stats | grep -E 'cluster.my_cluster.(upstream_cx_active|upstream_rq_pending_active|upstream_cx_total)$'

# Retry behavior - is amplification inflating the histogram?
curl -s http://localhost:9901/stats | grep -E 'cluster.my_cluster.upstream_rq_retry'

# Timeout counters - is the ceiling artifact present?
curl -s http://localhost:9901/stats | grep -E 'cluster.my_cluster.upstream_rq_(timeout|per_try_timeout)'

# Circuit breaker state - are we rejecting before even trying?
curl -s http://localhost:9901/stats | grep 'cluster.my_cluster.circuit_breakers'

# Compare to client-observed latency
curl -s http://localhost:9901/stats | grep 'downstream_rq_time'

# Worker thread health - is Envoy itself the bottleneck?
curl -s http://localhost:9901/stats | grep -E 'watchdog_miss|watchdog_mega_miss'

# Per-thread CPU (OS-level, not admin)
top -H -p $(pgrep -x envoy) -b -n 1 | head -20

How to diagnose it

1. Confirm the increase is real and cluster-scoped. Pull upstream_rq_time for the affected cluster specifically. Aggregate across all clusters hides which backend is slow. Look at P50 and P99 together. If only P99 moved, the tail is the problem. If P50 moved past the normal P99, the whole distribution shifted and the backend is in severe distress.

2. Check whether the timeout ceiling is distorting the histogram. If you see a spike at a round millisecond value (15000ms, 30000ms) and nothing above it, you are looking at timeout artifacts, not backend behavior. Cross-reference with upstream_rq_timeout and upstream_rq_per_try_timeout counters. The backend may be slower than the timeout, in which case the real latency is unknowable from this histogram alone.

3. Separate network from backend. Compare upstream_cx_connect_ms to upstream_rq_time. If connect time is also elevated, the network path or the upstream’s SYN backlog is the contributor. If connect time is flat but upstream_rq_time is up, the backend is the contributor. For mTLS upstreams, the TLS handshake cost is folded into connect time, so a certificate or cipher regression will show up here.

4. Check the connection pool. If upstream_cx_active is near max_connections and upstream_rq_pending_active > 0, the pool is saturated. New requests wait for a connection, and the wait is included in upstream_rq_time. High upstream_cx_total rate means connections are being churned (not reused), which adds TCP and TLS cost to every request. For HTTP/1.1 upstreams without keepalive, every request pays this cost.

5. Check for retry amplification. Compute upstream_rq_total / downstream_rq_total. If above 1.3, retries are inflating upstream load. If above 2.0, it is a retry storm. Each retry is a separate upstream_rq_time sample, so retries make the histogram look busier without representing the client experience. Check upstream_rq_retry_success to see whether retries are even helping. If retry success rate is low and retry rate is high, retries are accelerating the failure.

6. Check circuit breaker and outlier detection state. If circuit_breakers.default.cx_open or rq_pending_open is 1, Envoy is fast-failing new requests with 503 (response flag UO). These fast-fails are recorded at near-zero latency, which can pull the histogram down. If latency “improves” during an incident, check whether error rate spiked: the improvement may be circuit-breaker rejections, not backend recovery.

7. Check worker thread health. Aggregate Envoy CPU can look moderate while one worker is at 100%. Requests on that worker suffer event-loop queuing delay, which shows up in upstream_rq_time and downstream_rq_time alike. Run top -H -p $(pgrep -x envoy) to see per-thread CPU. Check server.watchdog_miss and server.watchdog_mega_miss. In Kubernetes, also check for CFS throttling, which causes the same symptom without Envoy being CPU-saturated internally.

8. Check HTTP/2-specific issues. For HTTP/2 upstreams, low upstream_cx_active with high RPS is normal (multiplexing). But if max_concurrent_streams on the upstream is low, or if one slow stream is blocking others on the same connection, latency can spike. Per-host breakdown via GET /clusters?format=json reveals whether the problem is host-specific.

Metrics and signals to monitor

Fixes

Backend service degradation

Address the backend. Envoy is reporting accurately. Check the backend’s own GC pauses, database contention, disk I/O, and resource saturation. Do not raise Envoy’s circuit breaker limits to “absorb” a slow backend. That removes Envoy’s protection without fixing the cause, and the next failure involves an unprotected upstream collapsing completely.

Network latency increase

Investigate the network path. Cross-AZ or cross-region traffic adds fixed latency. Check whether routing changed, whether security groups or firewall rules shifted, or whether DNS is resolving upstreams to distant endpoints. For mTLS, check whether the TLS handshake cost included in upstream_cx_connect_ms has inflated due to a cipher or certificate regression.

Connection pool contention

If upstream_cx_active is near max_connections and the pool is saturated, the upstream is slow, which holds connections longer, which fills the pool. The fix is the upstream, not the limit. Temporarily raising max_connections can buy headroom during a spike, but the underlying problem is that connections are being held longer than the design assumes. For HTTP/1.1 upstreams, ensure keepalive is enabled; without it, every request pays TCP and TLS cost. Check upstream_cx_total rate for churn caused by idle timeout mismatches between Envoy and the upstream.

Retry amplification

If upstream_rq_total / downstream_rq_total is high and retries are not succeeding, disable or reduce retries for the affected cluster. Retries on non-idempotent requests can cause duplicate operations. Review the retry policy: which status codes trigger retries, how many attempts, and whether a retry budget (max_retries circuit breaker) is configured. During an active retry storm, temporarily disabling retries via runtime flag or xDS config push stops the amplification immediately.

Timeout ceiling artifact

If the histogram piles at the timeout value, the backend is slower than the configured timeout. Either the backend needs to be faster (the real fix) or the timeout needs to reflect the actual SLO (if the current value is too aggressive). Do not raise the timeout to “make the metric look better.” A higher timeout means clients wait longer before getting an error. Tune the timeout to the SLO, then fix the backend.

HTTP/2 head-of-line blocking

If HTTP/2 upstreams show latency spikes tied to specific connections, check max_concurrent_streams on the upstream. A low limit with high concurrency forces serialization. Consider whether HTTP/1.1 with a larger pool would actually be faster for this workload. Check for TCP-level issues (retransmits, window size) that affect one connection and all its multiplexed streams.

Hot worker thread

Identify the hot thread via top -H. Check for expensive filters (Lua, Wasm, body manipulation), synchronous DNS or ext_authz calls, or stats-sink contention. In Kubernetes, check for CFS throttling via cgroup stats (cpu.stat), which causes watchdog_miss without internal CPU saturation. If one worker is handling disproportionate load due to SO_REUSEPORT imbalance, that is a deployment-level problem.

Prevention

• Configure custom histogram buckets. The default buckets span 0.5ms to 1 hour with wide gaps (100ms to 250ms to 500ms). Set buckets that bracket your SLO targets so percentile calculations are meaningful.
• Alert on baseline deviation, not fixed thresholds. Different services have different baselines. P99 greater than 2x rolling-24h average is more robust than a fixed millisecond cutoff. P50 greater than baseline P99 signals the entire distribution has shifted.
• Monitor the connection pool with headroom in mind. Track upstream_cx_active / max_connections and upstream_rq_pending_active. The pending queue is the leading indicator before 503s appear. Any sustained nonzero pending_active means the pool is becoming a bottleneck.
• Track retry ratios as a standard dashboard metric. The ratio upstream_rq_retry / upstream_rq_total should be a first-class signal. A sudden jump is often the first sign of upstream degradation, before latency or error rates move.
• Separate upstream_rq_time from upstream_cx_connect_ms. The ratio between them distinguishes network issues from application issues. If connect time is a growing fraction of total time, the problem is in the network path or connection establishment, not the backend.
• Watch for the timeout ceiling. If upstream_rq_timeout is nonzero in steady state, your timeout is too tight or your backend is too slow. Either way, the histogram is hiding the real tail.
• Monitor per-worker CPU, not just aggregate. Aggregate Envoy CPU masks single-threaded-per-worker behavior. watchdog_miss is the canary for event-loop blocking that does not show up in aggregate CPU.

How Netdata helps

• Per-second collection of upstream_rq_time, downstream_rq_time, and upstream_cx_connect_ms on the same timeline. Correlating the three side by side separates backend latency from network latency from Envoy-side overhead without manual curl sampling.
• Connection pool signals (upstream_cx_active, upstream_rq_pending_active, upstream_rq_pending_overflow) and circuit breaker gauges (cx_open, rq_pending_open) are collected per cluster. The pending queue depth is the leading indicator before 503s, and per-second resolution makes the cliff-edge visible.
• Retry signals (upstream_rq_retry, retry_success, retry_overflow) and the upstream-to-downstream request rate ratio are tracked continuously. A retry storm shows up as a ratio spike before error rates cascade.
• Timeout counters (upstream_rq_timeout, upstream_rq_per_try_timeout) are collected alongside the latency histogram, so the timeout ceiling artifact is distinguishable from real backend latency.
• Anomaly detection on the latency histogram flags baseline deviations that fixed thresholds miss, which matters for upstream_rq_time where baselines vary widely across services and traffic patterns.
• Worker thread health (watchdog_miss, watchdog_mega_miss) and per-second CPU per container surface hot-worker and CFS-throttling issues that aggregate CPU hides.

Related guides

• Envoy 502 and upstream resets: rx_reset, tx_reset, and mid-response failures
• Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend
• Envoy 504 upstream timeout: upstream_rq_timeout, per-try timeouts, and the UT flag
• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy connection pool exhaustion: a slow upstream that fills the pool
• Envoy health checks vs outlier detection: two systems that eject hosts differently
• How Envoy actually works in production: a mental model for operators
• Envoy membership_healthy dropping: reading the single most important cluster signal
• Envoy monitoring checklist: the signals every production proxy needs
• Envoy monitoring maturity model: from survival to expert
• Envoy no healthy upstream: the 503 when a cluster has no host to route to
• Envoy outlier detection mass ejection: when passive health checks empty a cluster