$ guides / envoy / envoy-downstream-rq-time-high ▌

Operations Guides

Envoy downstream_rq_time high: client-observed latency and proxy overhead

High downstream_rq_time means client-observed latency through the proxy is climbing. The histogram lives under http.<stat_prefix>.downstream_rq_time, measured in milliseconds, and it maps most directly to user-perceived slowness. When it breaches SLO, you are in an incident whether the upstreams are healthy or not.

The common reflex is to subtract upstream_rq_time from downstream_rq_time and label the remainder “Envoy overhead.” That delta is useful as a trend, but it is not a clean measurement of proxy processing time. It folds in filter execution, downstream upload and download behavior, retry time across multiple attempts, buffering delays, streaming duration, and how fast the client reads the response. A high delta can mean Envoy is working hard, a client is slow, a request was retried, or a long-lived SSE stream is open. The histogram alone will not tell you which.

Treat downstream_rq_time as the SLO signal and the delta as a hint worth trending, not a number worth thresholding.

What this means

downstream_rq_time is total wall-clock time from Envoy receiving the first byte of a downstream request to sending the last byte of the response. It is a histogram, so you read P50/P95/P99 from its buckets. Default buckets span 0.5ms to 3,600,000ms (one hour), which is wide enough to hide structure below 100ms. Custom buckets aligned to your SLO are worth configuring.

upstream_rq_time is measured at the router filter level: it starts when the downstream request has been received in full (the router’s onRequestComplete) and ends when the upstream response completes (onUpstreamComplete). Downstream upload time is therefore excluded from upstream_rq_time but included in downstream_rq_time, so clients that upload slowly widen the delta without adding any upstream latency. For buffered request-response traffic this approximately brackets upstream processing.

Each retry attempt is a separate upstream_rq_time observation; the stat does not accumulate retry time across attempts.

The delta downstream_rq_time - upstream_rq_time therefore includes everything outside that upstream window:

HCM request parsing
Request-side filter chain execution (Lua, Wasm, ext_authz, rate limit, RBAC)
Connection pool acquisition when a new upstream connection is needed
Response-side filter chain execution (compression, buffering)
Time spent pushing the response body to the client, including slow client reads
For retried requests, the full wall-clock across all attempts, because downstream_rq_time measures the whole client-facing transaction while upstream_rq_time records each attempt individually

flowchart LR
    A["downstream_rq_time START\nclient first byte"] --> B["HCM parses request"]
    B --> C["request filters: Lua, Wasm,\next_authz, rate limit, RBAC"]
    C --> D["upstream_rq_time START\nupstream attempt begins"]
    D --> E["upstream: connect +\nrequest + response"]
    E --> F["upstream_rq_time END\nlast upstream byte"]
    F --> G["response filters:\ncompression, buffer"]
    G --> H["downstream_rq_time END\nlast byte to client"]

None of the delta windows is “pure Envoy overhead.” Some are proxy work (filter execution, TLS, compression CPU), but others are client behavior (slow reads), protocol semantics (streaming duration), or retry policy (cumulative attempts). Trend the delta; do not alert on a fixed delta threshold.

downstream_rq_time is not available per-route by default. It is emitted at the http.<stat_prefix>. level. If your stat prefix covers many routes with different latency profiles, the histogram is a blend. A slow route and a fast route sharing a stat prefix will look like a bimodal distribution that resists clean SLO reasoning.

Common causes

Cause	What it looks like	First thing to check
Filter chain overhead	Delta grows, `upstream_rq_time` stable, filter-specific stats (`ext_authz.*`, rate limit) elevated	ext_authz and ratelimit stats, Lua/Wasm filter cost
Worker thread saturation	Sporadic P99 spikes, P50 stable, aggregate CPU moderate	`server.watchdog_miss`, per-thread CPU via `top -H`
Retry inflation	`downstream_rq_time` high but any single `upstream_rq_time` looks fine	`upstream_rq_retry` ratio against `upstream_rq_total`
Streaming responses	Long-tail dominates P99, request rate low, `downstream_rq_active` sustained	Whether the workload is SSE, long-poll, or gRPC streams
Slow downstream reads	Delta grows, response sizes large, `downstream_cx_active` high	Client read rate, buffer watermarks, idle timeout config
Downstream TLS handshake storms	Delta spikes on new connections, CPU spikes on workers	`listener.<address>.downstream_cx_ssl_total` rate, plus `listener.<address>.ssl.handshake` (successful handshakes)
Compression CPU	Delta grows on compressible responses, worker CPU correlated	Whether compression filter is enabled and at what level
Upstream is actually slow	Delta flat or small, but `downstream_rq_time` high because `upstream_rq_time` is high	`upstream_rq_time` per cluster, `upstream_cx_connect_ms`

If the delta is stable and small while downstream_rq_time climbs, the upstream is the driver, not Envoy. Chasing proxy overhead when upstream_rq_time is high wastes the investigation.

Quick checks

Run these read-only against the admin API. All are safe under load.

# downstream_rq_time histogram for the HCM stat prefix
curl -s http://localhost:9901/stats | grep 'downstream_rq_time'

# upstream_rq_time for a specific cluster
curl -s http://localhost:9901/stats | grep 'cluster.<name>.upstream_rq_time'

# retry rate against total upstream requests
curl -s http://localhost:9901/stats | grep -E 'upstream_rq_retry|upstream_rq_total'

# worker thread blocking signals
curl -s http://localhost:9901/stats | grep -E 'watchdog_miss|watchdog_mega_miss|loop_duration|poll_delay'

# per-thread CPU to find a hot worker
top -H -p $(pgrep -x envoy) -b -n 1

# TLS connection rate on the listener (counter; derive rate over time)
curl -s http://localhost:9901/stats | grep 'downstream_cx_ssl_total'

# ext_authz latency and error signals
curl -s http://localhost:9901/stats | grep 'ext_authz'

# active downstream requests (sustained nonzero means in-flight, possibly streaming)
curl -s http://localhost:9901/stats | grep 'downstream_rq_active'

In Istio sidecar mode, the admin port is 15000, not 9901.

How to diagnose it

Confirm the breach is real, not a histogram artifact. Look at the P95/P99 trend over the last hour, not a single scrape. Default buckets are wide; a small distribution shift crossing a bucket boundary can look dramatic. If you see NaN or zero values, your stats configuration may be suppressing histogram emission, which is a separate problem.
Check whether upstream_rq_time is also high. If yes, the upstream is the driver and the delta is irrelevant. Go work the upstream. See the related guides on 504 timeouts, circuit breakers, and connection pool exhaustion.
If upstream_rq_time is normal, compute the delta trend. Is the delta growing, or stable while both metrics shifted together? A growing delta points at the proxy path or client behavior. A stable delta with both metrics high still points upstream.
Check the retry ratio. Divide upstream_rq_retry by upstream_rq_total. Elevated retries mean downstream_rq_time includes cumulative retry wall-clock. A request that retried three times with per-try timeouts will show a downstream_rq_time far above any single upstream_rq_time observation.
Check for streaming or long-lived requests. Sustained nonzero downstream_rq_active with low request rate suggests SSE, long-poll, or gRPC streams. These dominate the histogram tail. If your SLO mixes streaming and request-response traffic under one stat prefix, separate them or set SLOs per traffic class.
Check worker thread health. watchdog_miss or watchdog_mega_miss incrementing means a worker event loop blocked long enough to trip the watchdog. Use top -H -p $(pgrep -x envoy) to find a hot worker. Aggregate CPU looking moderate while one thread is pinned at 100% is the classic hot-worker pattern.
Check filter latency. If ext_authz is configured, look at ext_authz.ok, ext_authz.error, and any latency stat the filter exposes. ext_authz latency is additive: every millisecond the auth service takes is a millisecond on downstream_rq_time. Lua and Wasm filters do not always emit per-filter latency stats; if suspected, enable dispatcher stats or run a profiling pass.
Check TLS connection rate. A burst of new downstream connections means a burst of TLS handshakes, which are CPU-intensive. Correlate new-connection rate with worker CPU. If clients are not reusing connections, every request pays handshake cost.
Check for CFS throttling in Kubernetes. If Envoy is containerized with a CPU limit, CFS throttling causes latency spikes and watchdog_miss without Envoy being computationally overloaded. Check container_cpu_cfs_throttled_periods_total at the container level, or read /sys/fs/cgroup/cpu.stat (cgroup v2) for nr_throttled.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`http.<stat_prefix>.downstream_rq_time`	The SLO metric. Client-observed latency.	P99 above 2x rolling baseline, or SLO breach sustained
`cluster.<name>.upstream_rq_time`	The upstream window. Determines whether the delta is the issue.	P99 above 2x rolling baseline independently of downstream
Delta `downstream_rq_time - upstream_rq_time`	Hint at proxy-path cost. Trend only.	Sustained upward trend, not a fixed threshold
`cluster.<name>.upstream_rq_retry`	Retry inflation folds into downstream time.	Ratio above 0.1 of `upstream_rq_total`
`server.watchdog_miss` / `watchdog_mega_miss`	Worker event loop blocking. Direct latency cause.	Any nonzero increment
`listener.<address>.downstream_cx_ssl_total`	TLS handshake CPU cost on new connections.	Burst rate correlated with CPU and delta spikes
`http.<stat_prefix>.ext_authz.*`	Additive per-request auth latency.	`ext_authz.error` elevated, or latency stat growing
`http.<stat_prefix>.downstream_rq_active`	In-flight request count. Sustained nonzero with low rate hints at streams.	Distribution shape inconsistent with request-response workload
Per-worker CPU (`top -H`)	Reveals hot-worker pattern hidden by aggregate CPU.	One thread pinned while others idle

Fixes

Filter chain overhead

Profile which filter adds latency. ext_authz is the usual suspect: check its timeout and failure mode. If the auth service is slow, every proxied request pays. Increase the ext_authz timeout only if the auth service genuinely needs it; otherwise the fix is the auth service, not Envoy. For Lua and Wasm filters, review whether the filter does synchronous work (DNS lookups, external calls) that belongs elsewhere. Removing an expensive filter from the hot path is more effective than tuning around it.

Worker thread saturation

If one worker is pinned, the options are: increase --concurrency if the machine has spare cores, reduce per-filter cost, or investigate CFS throttling. In Kubernetes sidecars, CPU limits are the most common hidden cause of worker stalls that look like Envoy bugs. Removing or raising the CPU limit often resolves watchdog_miss and the associated tail latency without any Envoy config change.

Retry inflation

If retries are inflating downstream_rq_time, fix the retry policy. Reduce the retry count, narrow retry_on to genuinely retriable conditions, and ensure the retry budget (the max_retries circuit breaker) is not so large it allows amplification. Retries on non-idempotent endpoints are a correctness risk as well as a latency risk.

Streaming responses

If SSE, long-poll, or gRPC streams dominate the histogram, the metric is not broken; your SLO framing is. Separate streaming traffic into its own stat prefix or cluster, and set SLOs per traffic class. Mixing a 30-second SSE stream and a 50ms API call under one downstream_rq_time histogram guarantees a useless P99.

Slow downstream reads

If clients are slow to read responses, Envoy holds the stream open longer, inflating downstream_rq_time. Check idle timeout configuration, buffer watermarks, and whether the client is genuinely slow or network-constrained. This is often a client-side or network issue, not an Envoy issue.

Compression

Compression adds CPU time to the response path. If the delta grows on compressible responses and worker CPU correlates, review whether compression is applied to already-compressed content (images, pre-compressed assets) where it adds cost without benefit. Tune the compression level if the filter supports it.

Upstream is the cause

If upstream_rq_time is the driver, stop tuning the proxy. Address the upstream. See the related guides on upstream timeouts, circuit breakers, and connection pool exhaustion.

Prevention

Set SLO-based alerts on downstream_rq_time, not fixed thresholds. Different services have different baselines; a fixed-ms threshold fires on every workload change.
Trend the delta, do not alert on it. A fixed delta threshold fires on every workload shift and misses real regressions within the threshold.
Configure custom histogram buckets aligned to your SLO. Default buckets are too wide to see structure in the 1-100ms range where most request-response traffic lives.
Separate streaming and request-response traffic into different stat prefixes. A blended histogram is unactionable for both traffic classes.
Monitor watchdog_miss and per-worker CPU. Hot workers are invisible in aggregate CPU and cause sporadic tail latency.
Monitor the retry ratio. Retries silently inflate client-observed latency during partial upstream failures.
Verify histograms are being emitted. A stats configuration that suppresses histogram output leaves you blind to latency regressions with no error signal.

How Netdata helps

Per-second collection of downstream_rq_time and upstream_rq_time histograms makes latency shifts visible within a single scrape window rather than after coarse bucket aggregation.
ML anomaly detection on the delta trend surfaces proxy-path regressions before they cross an SLO threshold.
Correlating downstream_rq_time with watchdog_miss, per-worker CPU, TLS connection rate, and ext_authz latency in one view makes it faster to isolate whether the driver is proxy, client, or upstream.
The retry ratio (upstream_rq_retry against upstream_rq_total) shown alongside downstream_rq_time makes retry-driven latency inflation visible without manual calculation.
Anomaly flags on downstream_rq_active help distinguish streaming workloads from request-response workloads when the histogram shape changes unexpectedly.