Envoy downstream request rate dropping to zero: the traffic blackhole

A sudden collapse of http.<stat_prefix>.downstream_rq_total to near zero is one of the loudest availability signals Envoy can emit, and one of the easiest to misread. Operators trained to chase 5xx spikes often treat a falling request rate as “the system is calm.” It is not. A drop below roughly 10% of baseline on a listener that should be receiving traffic is a traffic blackhole: clients have stopped reaching Envoy, Envoy has stopped accepting requests, or traffic is reaching Envoy but being answered by local replies before any upstream work happens.

Most traffic blackholes are not Envoy bugs. They are failures upstream of Envoy: a cloud LB stopped sending traffic, DNS is misresolving, a certificate expired and clients refuse the TLS handshake, or an xDS push removed a route or cluster. The diagnostic discipline: prove whether traffic is arriving at Envoy, localize where it stopped, then examine the configuration and TLS state that explains why.

This article assumes you have a baseline for downstream_rq_total on the affected listener, with a page threshold around 10% of rolling baseline and a traffic floor so idle off-hours environments do not page on legitimate zero.

What this means

“Traffic blackhole” describes the symptom, not the diagnosis. Three failure classes produce the same flat line on downstream_rq_total:

1. Traffic is not arriving at Envoy. A device upstream of Envoy (cloud LB, DNS resolver, ingress controller, mesh gateway) stopped sending, or clients cannot establish a connection (TLS handshake failures, SYN drops, firewall or security group changes).
2. Envoy is refusing or unable to accept traffic. A listener is gone, the process is draining or stuck in INITIALIZING, the overload manager has triggered stop_accepting_connections, or file descriptors are exhausted at the OS or container level.
3. Traffic is arriving but is being answered locally. Local replies (no route, no cluster, no healthy upstream, rate limited, ext_authz denied) still increment downstream_rq_total, but no application work is performed. This is not strictly a blackhole on the counter, but it is the same user-visible outcome and the easiest class to mistake for one.

First distinction: did the request rate drop, or did the request success rate drop? If downstream_rq_total is stable but downstream_rq_2xx collapsed while downstream_rq_5xx spiked, you have an error problem (see Envoy 503 with response flag UO, Envoy 504 upstream timeout, and Envoy no healthy upstream). This article is about the rate itself going to zero.

Two traps to watch for:

• HTTP/2 connection stability masks request collapse. downstream_cx_active can look healthy while the request rate has collapsed. One HTTP/2 connection multiplexes many streams; if the upstream or a filter has stalled the streams, the connection stays open and idle. Always read request rate alongside connection count. The same applies to gRPC, where long-lived streams make connection count nearly meaningless as a load signal.
• Load balancer health checks inflate the counter. A listener that has gone dark for application traffic can still show a steady trickle of requests if the LB health probe is the only thing hitting it. Filter by route, virtual host, or user agent before declaring the rate “non-zero.”

flowchart TD
    A["downstream_rq_total below 10% baseline"] --> B{server.state == LIVE?}
    B -- No --> C["Draining or stuck initializing"]
    B -- Yes --> D{Listeners active?}
    D -- No --> E["LDS removed the listener"]
    D -- Yes --> F{downstream_cx_active stable?}
    F -- Yes --> G["HTTP/2 streams stalled or local replies"]
    F -- Dropping --> H{TLS handshakes failing?}
    H -- Yes --> I["Cert or CA expiry"]
    H -- No --> J["Upstream of Envoy: LB, DNS, firewall"]

Common causes

Quick checks

All read-only and safe on a live proxy.

# Confirm the counter really dropped (two samples, 10s apart)
curl -s http://localhost:9901/stats | grep 'downstream_rq_total'

# Server lifecycle state (0=LIVE, 1=DRAINING, 2=PRE_INITIALIZING, 3=INITIALIZING)
curl -s http://localhost:9901/stats | grep '^server.state'
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:9901/ready

# Listener and route config presence
curl -s http://localhost:9901/stats | grep -E 'total_listeners_active|listener_create_failure'
curl -s http://localhost:9901/config_dump | jq '.configs[] | select(.["@type"]|contains("Routes")) | .route_config'

# Downstream connection and stream state
curl -s http://localhost:9901/stats | grep -E 'downstream_cx_active|downstream_rq_active|downstream_rq_completed'

# TLS handshake health on the listener
curl -s http://localhost:9901/stats | grep -E 'ssl\.(handshake|connection_error|fail_verify_error)'

# Overload manager and FD pressure
curl -s http://localhost:9901/stats | grep -E 'overload_actions.*active|downstream_cx_overload_reject'
ENVOY_PID=$(pgrep -x envoy | head -1)
ls /proc/$ENVOY_PID/fd | wc -l
grep 'Max open files' /proc/$ENVOY_PID/limits

# xDS state
curl -s http://localhost:9901/stats | grep -E 'control_plane.connected_state|update_rejected|update_failure'

# Upstream of Envoy: is anything actually reaching the listener port?
ss -s
# From a client vantage point:
dig +short <envoy-facing-hostname>

In Istio sidecar mode, substitute port 15000 for 9901 and use port 15021 with path /healthz/ready for the readiness check.

How to diagnose it

Work top-down. Each step eliminates a layer.

1. Confirm the drop is real, not a scraping artifact. Sample downstream_rq_total twice with a short interval and compute the delta. A flat admin endpoint scrape during heavy load can look like a counter freeze. Compare against access log volume for the same window.

2. Is Envoy itself up and accepting? Check server.state. Anything other than 0 (LIVE) means the process is draining or initializing. Combine with listener_manager.total_listeners_active and uptime > 300s to confirm the listener is actually bound. A non-LIVE state during a rolling deploy is expected; sustained non-LIVE outside a deploy window is the incident.

3. Is the listener still configured? A bad LDS push can remove a listener. Check listener_manager.total_listeners_active for the affected address and listener_manager.listener_create_failure for rejected configs. update_rejected confirms an Envoy-side NACK.

4. Is traffic arriving at the port? Compare downstream_cx_active to baseline. If connections are stable but request rate is zero, you are in HTTP/2 stream stall or local-reply territory. If connections are dropping, the failure is upstream of Envoy or at the TLS layer.

5. Are TLS handshakes failing? Spikes in ssl.connection_error or ssl.fail_verify_error on the listener mean clients are connecting but cannot complete the handshake. Check /certs for expiry. In mTLS meshes, a CA rotation that did not propagate produces the same pattern on upstream clusters via cluster.<name>.ssl.fail_verify_error.

6. Is Envoy generating local replies? If downstream_rq_total is non-zero but upstream_rq_total is flat, requests are being answered without forwarding. Pull response codes and, critically, response flags from the access log. NR (no route) and NC (no cluster) are configuration errors and should be zero in production. UO (circuit breaker), RL (rate limited), and UAEX (ext_authz denied) are policy-driven local replies that look like outages to clients.

7. Is the overload manager active? Check overload_actions.stop_accepting_connections.active and stop_accepting_requests.active. If either is 1, Envoy is refusing traffic to survive. The cause is upstream: memory pressure, FD exhaustion, or a missing max_heap_size_bytes configuration that lets Envoy run straight to OOM with no graceful degradation.

8. Look upstream of Envoy. If handshakes are clean, listeners are live, and the process is LIVE, the failure is outside Envoy. Check the cloud LB target group health, DNS resolution from a real client vantage point, security group and firewall changes, and any ingress or gateway controller that selects Envoy as a backend.

9. Verify xDS state. control_plane.connected_state = 0 means Envoy is serving stale config. Stale config alone does not blackhole traffic, but a listener or cluster removed from the active config via a bad push will. Cross-check connected_state, update_rejected, and config_dump against what the control plane thinks it pushed.

Metrics and signals to monitor

Response flags are access-log only. They are not exposed as aggregate Prometheus stats. For real-time alerting on NR or NC, you need a log pipeline that counts them, or a Lua or Wasm filter that increments custom counters.

Fixes

Group the response by which layer the diagnosis pointed at.

Traffic is not arriving at Envoy

Restore the path. Common actions: re-register the Envoy target in the cloud LB, fix DNS records, roll back the security group or firewall change, or fail the ingress controller back to its previous config. None of these are Envoy-side changes. The trap is spending an hour inside Envoy’s admin interface when the fix is one API call to the LB.

If the cause is a TLS cert or CA expiry, rotate the cert and verify SDS propagation. Check /certs after rotation. In mTLS meshes, also verify cluster.<name>.ssl.fail_verify_error is falling on upstream clusters, not just the downstream listener. See Envoy membership_healthy dropping for related cluster-side symptoms.

Envoy is refusing or unable to accept traffic

If server.state is non-LIVE outside a planned restart, investigate why. A process stuck in INITIALIZING is usually waiting on initial xDS config; check control_plane.connected_state and the control plane’s own health. A draining process that never exits is a hot restart race; check server.parent_connections and server.hot_restart_epoch.

If the overload manager is active, the root cause is memory or connection pressure. Increasing max_heap_size_bytes or raising connection limits is a temporary measure; the durable fix is to address whatever is consuming the memory (buffering filters, stats cardinality, connection leak). The overload manager exists to prevent OOM; do not disable it.

If file descriptors are exhausted, raise the ulimit (in Kubernetes, verify the actual limit inside the container, since securityContext or LimitRange may override the runtime default) and then hunt the leak. Common culprits: access log file descriptor leaks, missing keepalive on HTTP/1.1 upstreams, and excessive health check connections across many clusters.

Local replies (NR, NC, UO, RL, UAEX)

For NR and NC, the fix is a configuration correction. Pull config_dump, find the route or cluster that should have matched, and reconcile against what the control plane pushed. A bad xDS push is the usual cause. See How Envoy actually works in production for the routing model.

For UO, the fix is upstream, not in Envoy. Raising circuit breaker limits without addressing the slow upstream removes protection and makes the next failure worse. See Envoy circuit breaker open and Envoy connection pool exhaustion.

For RL and UAEX, verify the rate limit and ext_authz services are healthy and that the policy in effect matches intent. failure_mode_allow on ext_authz is a security-relevant decision: if it is true and the service is down, unauthenticated traffic is flowing.

HTTP/2 stream stall

If downstream_cx_active is stable but downstream_rq_active is growing and downstream_rq_completed is flat, streams are arriving and not completing. Look at upstream_rq_active on the target cluster, circuit breaker state, and filter latencies (ext_authz, rate limit, Lua, Wasm). A blocked worker thread produces the same pattern; check server.watchdog_miss and per-thread CPU via top -H -p $(pgrep -x envoy).

Prevention

• Alert on downstream_rq_total rate below 10% of rolling baseline with a traffic floor. The floor eliminates false pages from idle off-hours environments.
• Alert on server.state != 0 combined with listener_manager.total_listeners_active == 0 and uptime > 300s to confirm actual capacity loss rather than a transition state.
• Monitor TLS certificate runway via /certs. In SDS-managed meshes, treat dropping runway as a rotation pipeline failure, not a future concern.
• Alert on control_plane.connected_state = 0 sustained for more than 5 minutes, and on any non-zero update_rejected or listener_create_failure. These catch bad-config blackholes before users do.
• Track file descriptor utilization as a percentage of ulimit. Keep below 80%, and below 50% if hot restart is in use, since FDs briefly double during rollover.
• Configure the overload manager. Without it, Envoy goes straight from “fine” to OOM with no graceful degradation.
• Filter load balancer health check paths out of application traffic analysis. They inflate downstream_rq_total and mask real drops.
• Run a log pipeline that counts response flags. Without it, NR, NC, and UO are invisible in aggregate metrics.

How Netdata helps

• Per-second collection of downstream_rq_total and downstream_rq_active surfaces the collapse within seconds, before a typical 15-30s Prometheus scrape interval would.
• Correlated charts for downstream_cx_active, ssl.connection_error, ssl.fail_verify_error, and server.state on the same timeline let you distinguish “no traffic arriving” from “TLS failing” from “Envoy draining” in one view.
• Upstream upstream_rq_total, upstream_rq_active, and circuit breaker gauges on the same dashboard confirm whether the blackhole is downstream-side or upstream-side without pivoting.
• control_plane.connected_state, listener_manager.total_listeners_active, and overload manager action gauges surface Envoy-internal causes (stale config, removed listener, self-protection) alongside the symptom.
• Container-level CPU throttling and FD utilization charts catch kernel-side and OS-side causes (CFS throttling, FD exhaustion) that look like Envoy bugs from inside Envoy’s own stats.

Related guides

• Envoy 502 and upstream resets: rx_reset, tx_reset, and mid-response failures
• Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend
• Envoy 504 upstream timeout: upstream_rq_timeout, per-try timeouts, and the UT flag
• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy connection pool exhaustion: a slow upstream that fills the pool
• Envoy health checks vs outlier detection: two systems that eject hosts differently
• How Envoy actually works in production: a mental model for operators
• Envoy membership_healthy dropping: reading the single most important cluster signal
• Envoy monitoring checklist: the signals every production proxy needs
• Envoy monitoring maturity model: from survival to expert
• Envoy no healthy upstream: the 503 when a cluster has no host to route to
• Envoy outlier detection mass ejection: when passive health checks empty a cluster