$ guides / envoy / envoy-downstream-cx-overflow-overload-reject ▌

Operations Guides

Envoy downstream_cx_overflow and overload_reject: connections turned away at the door

When Envoy rejects a new TCP connection at the listener, the client gets no HTTP error or retry hint: the SYN is answered with a RST or dropped, and the client sees connection refused or a connect timeout. The only evidence is three listener-level counters that should always be zero: downstream_cx_overflow, downstream_cx_overload_reject, and downstream_global_cx_overflow.

Each counter corresponds to a distinct rejection mechanism with a different root cause and fix. Confusing them wastes time tuning the wrong limit while the real problem is memory pressure, file descriptor exhaustion, or a missing overload manager.

What this means

Envoy rejects a new downstream connection at one of three checkpoints, each controlled by different configuration.

Per-listener connection limit increments downstream_cx_overflow. The listener has hit its configured maximum active connections, set via the runtime key envoy.resource_limits.listener.<listener_name>.connection_limit.

Global downstream connection limit increments downstream_global_cx_overflow. Total active downstream connections across all participating listeners have hit the ceiling configured via the envoy.resource_monitors.downstream_connections resource monitor in the overload manager (max_active_downstream_connections field). Listeners can opt out with Listener.ignore_global_conn_limit.

Overload manager action increments downstream_cx_overload_reject. The overload manager has activated a protective action. Likely triggers are the overload actions envoy.overload_actions.stop_accepting_connections and envoy.overload_actions.reject_incoming_connections, and the load shed point envoy.load_shed_points.tcp_listener_accept.

In all three cases Envoy does not send an HTTP error. The TCP SYN is RST’d or dropped.

The critical distinction is between limit-based rejections (the first two) and resource-pressure rejections (the third). Limit-based rejections mean a configured number is too low for the traffic pattern. Resource-pressure rejections mean Envoy is protecting itself from exhausting memory or connections; fix the underlying pressure, do not raise a limit.

Common causes

Cause	What it looks like	First thing to check
Per-listener limit too low	Only `downstream_cx_overflow` increments; traffic bursty or clients open many connections	`downstream_cx_active` vs configured limit
Global limit too low	Only `downstream_global_cx_overflow` increments; multiple listeners share the global cap	Total `downstream_cx_active` across all listeners vs global limit
Memory pressure (overload manager)	`downstream_cx_overload_reject` increments alongside `overload_actions.*.active = 1`	`server.memory_allocated` vs `max_heap_size_bytes` or container limit
Overload manager not configured	Envoy OOM-killed; no overload actions ever fire	Whether overload manager exists in config
File descriptor exhaustion	All counters may increment; `server.total_connections` near limit	`/proc/<pid>/fd` count vs `Max open files`
Connection leak	`downstream_cx_active` grows monotonically without traffic increase	`downstream_cx_active` trend vs `downstream_cx_total` rate
Health check listener caught in overload	Kubernetes restarts pod during overload; probes fail	Whether health check listener has `bypass_overload_manager: true` (v1.31.0+)

Quick checks

Commands use port 9901 (standalone Envoy). In Istio sidecar mode, use 15000.

# Which rejection counters are incrementing
curl -s http://localhost:9901/stats | grep -E 'downstream_cx_(overflow|overload_reject|global_cx_overflow)'

# Current active connections per listener
curl -s http://localhost:9901/stats | grep 'downstream_cx_active'

# Overload manager state - which actions are active right now
curl -s http://localhost:9901/stats | grep 'overload'

# Memory pressure (if overload_reject is firing)
curl -s http://localhost:9901/stats | grep 'server.memory'

# File descriptor utilization
ENVOY_PID=$(pgrep -x envoy | head -1)
ls /proc/$ENVOY_PID/fd | wc -l
grep 'Max open files' /proc/$ENVOY_PID/limits

# Connection-based FD estimate (each proxied connection ~= 2 FDs)
curl -s http://localhost:9901/stats | grep 'server.total_connections'

# Downstream connection rate vs destruction rate (detect leaks)
curl -s http://localhost:9901/stats | grep -E 'downstream_cx_(total|destroy)'

How to diagnose it

flowchart TD
    A["downstream_cx_overflow > 0?"] -->|Yes| B["Per-listener limit hit"]
    A -->|No| C["downstream_global_cx_overflow > 0?"]
    C -->|Yes| D["Global downstream limit hit"]
    C -->|No| E["downstream_cx_overload_reject > 0?"]
    E -->|Yes| F["Overload manager active"]
    E -->|No| G["Look elsewhere: FD exhaustion, kernel conntrack, listener down"]
    B --> H["Check limit vs downstream_cx_active"]
    D --> I["Check global limit vs total active across listeners"]
    F --> J["Check overload_actions.*.active and server.memory"]

Identify which counter is incrementing. Run the first quick check. Usually only one of the three is climbing. If all three fire simultaneously, file descriptor exhaustion is the likely root cause because it starves every mechanism at once.
If downstream_cx_overflow: the per-listener limit is the cause. Check envoy.resource_limits.listener.<listener_name>.connection_limit against downstream_cx_active. If active is near the limit, either the limit is too low for the traffic, or connections are accumulating (idle timeout too long, leak, spike).
If downstream_global_cx_overflow: the global limit is the cause. Check max_active_downstream_connections in the resource monitor configuration. This value cannot be updated at runtime. Also check whether the deprecated runtime key overload.global_downstream_max_connections is set with a conflicting value.
If downstream_cx_overload_reject: the overload manager is actively rejecting connections. This is the most serious case: Envoy is in resource protection mode. Identify the active action via server.overload_manager.envoy.overload_actions.<action>.active gauges and the resource monitor that triggered it.
Correlate with memory and FD usage. If the overload manager fired on memory pressure, check server.memory_allocated against max_heap_size_bytes or the container memory limit. If FDs are the constraint, count entries in /proc/<pid>/fd against Max open files.
Check for health check listener impact. If Kubernetes restarts the pod during the incident, the overload manager may be blocking liveness and readiness probes. This is a known issue (envoyproxy/envoy#23843). Fix it with bypass_overload_manager: true on the health check listener (requires v1.31.0+) or ignore_global_conn_limit: true on the admin listener.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`downstream_cx_overflow` rate	Per-listener limit being hit	Any sustained nonzero rate
`downstream_global_cx_overflow` rate	Global connection limit being hit	Any sustained nonzero rate
`downstream_cx_overload_reject` rate	Overload manager rejecting connections	Any sustained nonzero rate
`downstream_cx_active`	Current connection load per listener	Trending toward configured limit
`overload_actions.*.active`	Which overload actions are firing right now	Any gauge at 1
`server.memory_allocated`	Memory pressure that triggers the overload manager	Approaching container limit or `max_heap_size_bytes`
`server.total_connections`	FD proxy (each connection ~= 2 FDs)	Approaching FD ulimit
FD count (`/proc/<pid>/fd`)	Hard cliff when exhausted	Above 80% of `Max open files`

These counters should be zero in normal operation. For alerting, use a ratio of rejected / (accepted + rejected) with a meaningful traffic floor and a duration threshold above 5 minutes. Small instances trip listener limits on routine bursts; alerting on any nonzero value produces false positives.

If overload manager action gauges are available, page on those instead of the rejection counters. overload_actions.stop_accepting_connections.active = 1 indicates hard resource exhaustion, a stronger root-cause signal than a rejection counter that could be a misconfigured limit.

Fixes

Per-listener connection limit too low

If downstream_cx_overflow is the only counter firing and active connections reflect traffic rather than a leak, raise the limit via envoy.resource_limits.listener.<listener_name>.connection_limit.

Keep the limit below half of the FD ulimit to leave room for upstream connections, files, and other FD usage. Raising the limit without FD headroom pushes the problem to FD exhaustion, a harder cliff with no graceful degradation.

Global downstream connection limit too low

If downstream_global_cx_overflow is firing, raise max_active_downstream_connections in the envoy.resource_monitors.downstream_connections resource monitor.

Version note: in v1.28.0 the runtime key overload.global_downstream_max_connections was deprecated in favor of the resource monitor. In v1.29.0 it was undeprecated until the resource monitor extension becomes stable. Conflicting values produce duplicate resource monitor errors. The resource monitor value cannot be updated at runtime; if you need dynamic limits you must use the deprecated runtime key, which will be removed in a future release.

Overload manager triggered by memory pressure

If downstream_cx_overload_reject fires alongside an active overload action, the root cause is resource exhaustion, not a connection limit. Fix the underlying pressure:

Check server.memory_allocated. If it grows without a traffic increase, investigate buffer accumulation from slow downstream clients, stats cardinality explosion, or a memory leak.
If memory is near the container limit, raise the limit or address the source of growth. Do not raise the overload manager threshold to suppress the action. That removes Envoy’s self-protection and leads to OOM kills.

Overload manager not configured at all

If Envoy is OOM-killed and downstream_cx_overload_reject never increments, the overload manager may not be configured. Many default and tutorial deployments omit it. Without it Envoy goes straight from fine to dead.

Configure the overload manager with a fixed_heap resource monitor (max_heap_size_bytes) and threshold-based actions. The official edge proxy best practices guide recommends shrink_heap at 95% and stop_accepting_requests at 98%.

If no global downstream connection limit is configured, Envoy emits a startup warning. To silence it without setting a limit, set the runtime value to a very large number (approximately 2e9).

File descriptor exhaustion

If all three counters increment simultaneously and /proc/<pid>/fd is near the limit, FD exhaustion is the root cause. Each proxied connection consumes approximately 2 FDs (downstream plus upstream), plus FDs for listen sockets, access logs, xDS connections, and health checks.

Raise the FD ulimit via the container security context, the systemd unit (LimitNOFILE), or /etc/security/limits.conf, then restart Envoy. A running process cannot have its FD limit raised; ulimit -n in a shell only affects child processes started after the change. Keep baseline usage below 50% of the limit if hot restart is used, because both old and new processes hold open FDs during drain.

Health check listener blocked by overload manager

If Kubernetes restarts the pod during an overload event, the overload manager is likely blocking liveness and readiness probes on the health check listener.

On v1.31.0 and later, set bypass_overload_manager: true on the health check listener. This is the recommended way to exempt health check listeners from overload actions.

On older versions, set ignore_global_conn_limit: true on the admin listener, or use a dedicated listener for health checks exempt from global limits.

Prevention

Configure the overload manager in every production deployment with a fixed_heap resource monitor and threshold-based actions. Without it, Envoy has no self-protection and goes straight to OOM.
Set per-listener and global connection limits proactively. Do not wait for the first incident to discover the limits are missing.
Keep connection limits below half of the FD ulimit. Each proxied connection uses approximately 2 FDs, plus overhead.
Monitor FD utilization directly, not just connection counts. The FD cliff is invisible if you only watch Envoy stats.
Set idle timeouts appropriately to prevent connection accumulation from long-lived but idle clients.
Exempt health check listeners from overload protection with bypass_overload_manager: true (v1.31.0+) to prevent pod restarts during overload.
Track downstream_cx_active trends to catch connection leaks before they hit any limit.
Verify which connection limit mechanism is active in service mesh deployments. Both the deprecated runtime key and the resource monitor configured together produce conflicting limits and duplicate errors.

How Netdata helps

Per-second granularity on downstream_cx_active, downstream_cx_overflow, downstream_cx_overload_reject, and downstream_global_cx_overflow shows the exact moment rejections start and correlates them with connection count growth, instead of waiting for a 30-second scrape to catch the transition.
ML anomaly detection on downstream_cx_active catches connection leaks and approaching limits before the cliff, with lead time that aggregate scrapes miss.
Correlating listener rejection counters with server.memory_allocated, overload_actions.*.active, and FD utilization in a single view distinguishes a too-low limit from genuine resource pressure without cross-referencing separate dashboards.
The Envoy integration surfaces overload manager action state alongside connection metrics, so you can immediately see whether stop_accepting_connections is the cause of rejections or a per-listener limit is the trigger.
Alerting on the ratio of rejected to accepted connections with a configurable traffic floor and duration window suppresses false positives from small instances tripping limits on routine bursts while still catching sustained rejection events.