Envoy downstream connection flood: slowloris, the cx-to-rq ratio, and oversized requests

The proxy is rejecting new connections, latency is climbing, and upstreams look healthy. The problem is at the front door. A downstream connection flood exhausts Envoy’s resources before a request reaches a filter chain. The classic signal: a high new-connection rate paired with a low request rate. Many TCP handshakes, few HTTP requests. That ratio is the cx-to-rq ratio, and when it inverts you are looking at a slowloris-style attack or a misconfigured client pool.

Three distinct failure shapes show up as “too many downstream connections”:

1. A pure connection flood that trips the listener or global connection limit.
2. A slowloris-style attack that holds connections open without completing requests.
3. Oversized request bodies that consume per-connection buffer budget and trigger 413s.

Each has a different primary signal. Conflating them leads to wrong mitigation.

The signals that matter live at the listener and HTTP connection manager (HCM) layers: listener.<address>.downstream_cx_overflow, listener.<address>.downstream_global_cx_overflow, http.<stat_prefix>.downstream_cx_total (rate), http.<stat_prefix>.downstream_rq_total (rate), and http.<stat_prefix>.downstream_rq_too_large. Response flags in access logs (DPE, LR, SI) and overload manager action gauges (stop_accepting_connections) confirm whether Envoy is actively protecting itself.

What this means

A downstream connection flood saturates the listener accept queue, the file descriptor budget, or the overload manager’s configured memory or connection thresholds. Envoy does not gracefully degrade here. Once the cliff is hit, new TCP handshakes are RST’d or dropped, and clients see connection refused or timeout. The damage is at the door, before any HTTP filter or route resolution runs. Connection-based DDoS is turned away at the listener, not in the filter chain.

The cx-to-rq ratio distinguishes a flood from legitimate burst traffic. Under normal load, downstream_cx_total (new connections per interval) and downstream_rq_total (new requests per interval) move together. When downstream_cx_total spikes but downstream_rq_total stays flat, connections are being opened but not used. In HTTP/1.1 that is the slowloris fingerprint. In HTTP/2 it can also indicate stream-exhaustion patterns where a client opens a connection but never sends headers, or holds the flow-control window open to stall the stream.

Oversized request bodies surface differently. They show up as downstream_rq_too_large increments and 413 responses, not as connection rejection. The connection completes; the body is rejected when a buffering filter or the HCM body limit is exceeded. Envoy streams request bodies by default, so this counter only increments when a non-streaming filter requires the full body.

flowchart TD
  A[downstream_cx_active climbs] --> B{downstream_cx_overflow nonzero?}
  B -- yes --> C[Listener or global limit hit]
  B -- no --> D{cx_total much greater than rq_total?}
  D -- yes --> E[Slowloris or idle connection hold]
  D -- no --> F{downstream_rq_too_large climbing?}
  F -- yes --> G[Oversized body or payload attack]
  F -- no --> H[Check FD budget and overload manager]
  C --> I[Reject at listener, pre-filter-chain]
  E --> I
  G --> J[Reject in HCM, post-accept]
  H --> I

Common causes

Quick checks

# Listener-level connection rejections (per-listener limit and global limit)
curl -s http://localhost:9901/stats | grep -E 'downstream_cx_(overflow|overload_reject|global_cx_overflow)'

# Current active downstream connections per listener
curl -s http://localhost:9901/stats | grep 'downstream_cx_active'

# cx-to-rq ratio inputs: take two samples 10s apart, compute deltas
curl -s http://localhost:9901/stats | grep -E 'http\..*\.(downstream_cx_total|downstream_rq_total)'

# Oversized request bodies (413 source)
curl -s http://localhost:9901/stats | grep 'downstream_rq_too_large'

# Overload manager active actions (1 = Envoy is self-protecting)
curl -s http://localhost:9901/stats | grep 'overload_actions.*active'

# File descriptor budget
ENVOY_PID=$(pgrep -x envoy | head -1)
ls /proc/$ENVOY_PID/fd | wc -l
grep 'Max open files' /proc/$ENVOY_PID/limits

# TLS handshake error rate (floods often half-complete TLS)
curl -s http://localhost:9901/stats | grep -E 'ssl\.(connection_error|handshake)'

# Source concentration from access logs (tune path and field to your format)
tail -n 10000 /var/log/envoy/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head

The admin port is 9901 in vanilla Envoy and 15000 in Istio sidecar mode. Adjust the curl target accordingly. The awk field for source IP depends on your access log format; inspect a sample line first to pick the right column.

How to diagnose it

1. Confirm the symptom is downstream-side. Check downstream_cx_active and downstream_cx_overflow. If active connections are climbing or rejections are nonzero, the problem is at the listener, not the upstream. Verify cluster.<name>.membership_healthy is stable to rule out upstream-driven backpressure cascading into connection pileup.

2. Compute the cx-to-rq ratio. Sample http.<stat_prefix>.downstream_cx_total and http.<stat_prefix>.downstream_rq_total twice, 10 seconds apart. Compute cx_total_delta / rq_total_delta. A healthy HTTP/1.1 keepalive deployment has a ratio well below 1.0 (many requests per connection). A ratio above 5.0 with sustained growth in downstream_cx_active is a slowloris fingerprint.

3. Distinguish flood from slowloris. Both have a high cx rate. The discriminator is the downstream_cx_active trajectory and idle timeout behavior. In a flood, active connections rise quickly then plateau at the listener limit. In slowloris, active connections rise steadily because each connection is held open near the idle timeout. Check connection duration via access log timestamps (the gap between start_time and close) to confirm connections are being held near the idle timeout.

4. Check whether Envoy is self-protecting. Look at server.overload_manager.envoy.overload_actions.stop_accepting_connections.active and stop_accepting_requests.active. If either is 1, the overload manager has tripped. Cross-reference with server.memory_allocated against the configured heap limit and with FD utilization. If the overload manager is not configured at all, Envoy has no self-protection and will go straight from “fine” to OOM-killed under memory pressure.

5. Identify the noisy source. Parse access logs for source IP and user-agent concentration. The signal is the top-k distribution: a single source accounting for more than 10% of new connections during the incident window is the likely attacker or misconfigured client. Filter on response flag DPE (downstream protocol error) and LR (local reset) to find connections Envoy terminated early.

6. For oversized bodies, confirm the rejection layer. downstream_rq_too_large only increments when a non-streaming filter requires the full body and the body exceeds the configured limit. If you have a buffer filter or an ext_authz filter that buffers the body, that is where the limit applies. Verify whether the 413 was generated by Envoy (stat increments) or by the upstream application (no stat increment, response forwarded).

Metrics and signals to monitor

Fixes

Slowloris and connection floods

The first mitigation is listener-level connection limits enforced before the filter chain. Configure a per-listener connection limit and a global downstream connection limit. The global limit is configured via the overload manager resource monitor envoy.resource_monitors.global_downstream_max_connections; set it below half the system FD limit so Envoy cannot exhaust its own FD budget.

The connection limit filter (envoy.extensions.filters.network.connection_limit.v3.ConnectionLimit) closes overlimit connections immediately after accept. It does not prevent the TCP handshake from completing; it rejects after accept. For pre-accept rejection, the overload manager load shed point envoy.load_shed_points.tcp_listener_accept rejects new connections before the listener filter chain is created.

Reduce idle timeouts. A long common_http_protocol_options.idle_timeout inflates downstream_cx_active during a slowloris attack because each held connection lingers. Under attack, reducing the idle timeout shrinks the held-connection footprint without affecting well-behaved clients that send requests regularly. Set it low enough that held-without-request connections expire quickly, but high enough that legitimate keepalive clients are not disrupted.

Set max_requests_per_connection on the HCM. Once the limit is reached, Envoy drains the connection (GOAWAY for HTTP/2, Connection: close for HTTP/1.1). This caps the value of any single connection to an attacker and forces re-handshake cost on the next connection.

Oversized request bodies

The fix depends on where the limit should apply. If the body limit is intentional (you do not accept uploads larger than N bytes), ensure the HCM or buffer filter body size limit is set explicitly and documented. If the bodies are unexpected, investigate the client: missing Content-Length, wrong transfer encoding, or a retry loop that re-sends a large payload on every attempt.

Do not raise the limit reflexively. A higher limit means more memory per connection, which lowers the connection count at which the overload manager trips. Every additional byte of allowed body size is a byte of per-connection memory an attacker can consume.

Misconfigured client pools

If the noisy source is a known internal client, the fix is on the client side: connection pool sizing, keepalive settings, retry policy. A client that opens a new connection per request (HTTP/1.1 without keepalive) produces a cx-to-rq ratio near 1.0 and high TLS handshake CPU. A client that holds connections idle inflates downstream_cx_active. Both look like floods from Envoy’s side but are configuration issues on the client.

Prevention

• Set both per-listener and global connection limits. They are enforced independently. A listener with a smaller per-listener limit is enforced even when the global limit has headroom.
• Configure the overload manager with a heap limit and the global downstream connection monitor. Without it, Envoy has no self-protection and goes straight to OOM under memory pressure.
• Alert on the cx-to-rq ratio, not just on connection count. Absolute connection count is workload-dependent. The ratio is a stable abuse signal across traffic changes.
• Monitor FD utilization against ulimit, not just server.total_connections. The stat is a proxy; the ulimit is the hard cliff. During hot restart, FD usage briefly doubles, so keep baseline below 50% of the limit if hot restart is in use.
• Track downstream_rq_too_large and 413 rate as a ratio of total requests. Occasional 413s are client misconfiguration; sustained 413s are a payload attack or a client regression.
• Prefer /stats/prometheus?usedonly for scraping at 30s or longer intervals. The admin stats endpoint is served on the main thread. On a high-cardinality proxy, frequent scraping adds lock contention that affects request processing.

How Netdata helps

• Per-second resolution on downstream_cx_active and downstream_cx_total catches the cx-to-rq ratio inversion within the first minute, before the listener limit is hit. Counter-based scrapers with 30s intervals can miss a fast-ramp flood entirely.
• Anomaly detection on the cx-to-rq ratio flags the inversion as anomalous even when absolute connection counts are within historical bounds, which distinguishes a flood from a legitimate traffic burst.
• Correlation of downstream_cx_overflow with overload manager action gauges distinguishes “limit misconfiguration” from “Envoy is self-protecting under memory pressure.” Without that correlation you cannot tell whether raising the limit is safe.
• FD utilization alongside connection count shows runway to the cliff. Per-process resource views expose /proc/<pid>/fd counts and ulimit without a separate exporter.
• Anomaly flags on downstream_rq_too_large and 413 rate surface payload attacks that would otherwise be buried in aggregate 4xx noise.
• Source IP concentration from access log pipelines (when configured) gives the top-k distribution that identifies the noisy source without ad hoc shell commands against log files.

Related guides

• Envoy 502 and upstream resets: rx_reset, tx_reset, and mid-response failures
• Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend
• Envoy 504 upstream timeout: upstream_rq_timeout, per-try timeouts, and the UT flag
• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy clusters stuck warming: warming_clusters non-zero and routes returning 503
• Envoy connection pool exhaustion: a slow upstream that fills the pool
• Envoy control_plane.connected_state = 0: running on stale xDS config
• Envoy downstream_rq_time high: client-observed latency and proxy overhead
• Envoy file descriptor exhaustion: the FD cliff that refuses every new connection
• Envoy health checks vs outlier detection: two systems that eject hosts differently
• How Envoy actually works in production: a mental model for operators
• Envoy listener_create_failure: a listener config Envoy could not apply