Envoy overload manager actions: stop_accepting_connections and the last line before OOM

When server.overload_manager.envoy.overload_actions.stop_accepting_connections.active flips to 1, Envoy has stopped accepting new TCP connections on its listeners. When stop_accepting_requests is active, Envoy returns 503 to new HTTP requests before they reach an upstream. These are not bugs. They are Envoy’s last intentional actions before the kernel OOM-kills the process.

The overload manager connects resource pressure (heap size, connection counts) to a cascade of protective actions. When it fires, you are looking at both a root cause (memory or connection pressure) and a symptom (traffic being refused). Many deployments never configure it, skipping every graceful degradation step and going straight from “fine” to OOM-killed with nothing in between.

What this means

The overload manager watches resource monitors. When a monitor reports pressure above a configured threshold, it activates actions. Each action exposes two stats:

• server.overload_manager.envoy.overload_actions.<action>.active (gauge: 1 if currently active, 0 otherwise)
• server.overload_manager.envoy.overload_actions.<action>.scale_percent (gauge: 0-100, where 100 means fully applied)

These actions fire as a cascade. A typical configuration sequences them from soft degradation to hard traffic rejection:

flowchart TD
    A["Heap grows toward max_heap_size_bytes"] --> B["disable_http_keepalive"]
    B --> C["reduce_timeouts"]
    C --> D["close_idle_http_connections"]
    D --> E["stop_accepting_requests: 503 new HTTP"]
    E --> F["stop_accepting_connections: no new TCP"]
    F --> G["shrink_heap: tcmalloc release"]
    G -->|"pressure relieved"| H["Actions clear, traffic resumes"]
    G -.->|"pressure persists"| I["OOM Kill"]

stop_accepting_requests and stop_accepting_connections are the last traffic-serving actions. After those, the only remaining lever is shrink_heap, which asks tcmalloc to return memory. If the allocator cannot release enough, the next event is an OOM kill.

When either action is active, the question is not “why is Envoy broken” but “what is consuming memory faster than Envoy can shed load.”

If append_local_overload is configured, Envoy sets the x-envoy-local-overloaded header on local replies generated by overload actions. This lets downstream clients and log pipelines distinguish overload rejections from other 503s.

Envoy also supports load shed points (for example, envoy.load_shed_points.tcp_listener_accept) as an alternative to stop_accepting_connections for rejecting new TCP connections at the listener accept stage. Load shed points are evaluated at specific junctions in the connection lifecycle and are more reactive to changing conditions. Overload actions like stop_accepting_connections are better suited for cases where worker threads are not actively processing connections. Check your Envoy version’s docs for which mechanism applies to your use case.

Common causes

Quick checks

All commands are read-only and safe to run during an incident. The admin port defaults to 9901 in standalone Envoy and 15000 in Istio sidecar mode.

# Check which overload actions are currently active
curl -s http://localhost:9901/stats | grep 'overload'

# Check memory trajectory (allocated vs heap size vs physical)
curl -s http://localhost:9901/stats | grep 'server.memory'

# Check downstream connections rejected by overload manager
curl -s http://localhost:9901/stats | grep 'downstream_cx_overload_reject'

# Check total stat count for cardinality (non-zero stats only)
curl -s http://localhost:9901/stats?usedonly | wc -l

# Check active connection counts
curl -s http://localhost:9901/stats | grep -E 'downstream_cx_active|upstream_cx_active'

# Check container memory limit vs actual usage (cgroup v2)
cat /sys/fs/cgroup/memory.max
cat /sys/fs/cgroup/memory.current

# Check container memory limit vs actual usage (cgroup v1)
cat /sys/fs/cgroup/memory/memory.limit_in_bytes
cat /sys/fs/cgroup/memory/memory.usage_in_bytes

How to diagnose it

1. Confirm which actions are active. Grep for overload in stats. Any active = 1 is an event. The action name tells you where in the escalation cascade Envoy currently sits. If stop_accepting_requests or stop_accepting_connections is active, Envoy is refusing new traffic right now.

2. Check the memory trajectory. Look at server.memory_allocated over time. Monotonic increase without a corresponding traffic increase means a leak or unbounded accumulation. If it tracks traffic, the issue is load-driven: large bodies, many connections, or high-cardinality stats.

3. Verify max_heap_size_bytes is configured and correct. Check the bootstrap config (the YAML file passed via --config-path). The overload manager section should contain a fixed_heap resource monitor with max_heap_size_bytes set to a value below the container memory limit. The official example uses 2 GiB (2147483648). If this value is absent or set too high, the overload manager never triggers and Envoy is OOM-killed instead.

4. Check for the fixed_heap vs cgroup mismatch. The fixed_heap monitor reports pressure based on tcmalloc’s view of the heap, which can be substantially lower than what the cgroup memory controller reports. The kernel OOM-kills based on cgroup memory, not tcmalloc’s internal accounting. If server.memory_heap_size looks healthy but the container is near its memory limit, this mismatch is the root cause.

5. Check whether probes are being blocked. In Kubernetes, if you see pod restarts coinciding with overload actions, the overload manager may be blocking liveness or readiness probes. Probes that hit the same listeners as application traffic are disabled by stop_accepting_connections, so the kubelet sees probe failures and restarts the pod. This is most common in standalone Envoy deployments; Istio typically routes probes through the pilot-agent on a separate port.

Metrics and signals to monitor

Fixes

Missing or misconfigured overload manager

If the overload manager is not configured, Envoy has no graceful degradation path. It goes from serving traffic to OOM-killed with nothing in between. Every production deployment should have the overload manager in the bootstrap config.

The critical field is max_heap_size_bytes on the fixed_heap resource monitor. This value must reflect the container’s memory limit with headroom for non-heap memory: thread stacks, shared memory regions for stats, file-backed pages. Setting it to the container limit or above means the overload manager triggers only after the kernel has already decided to kill the process.

Fixed_heap vs cgroup memory monitor

In containerized deployments, the fixed_heap monitor can underreport pressure because tcmalloc’s heap size can lag behind actual cgroup memory usage. If you are seeing OOM kills without overload manager actions firing first, the mismatch between tcmalloc’s view and cgroup memory is likely the cause.

Probe blocking during overload

When stop_accepting_connections or stop_accepting_requests fires, Kubernetes liveness and readiness probes that hit the same listeners are also blocked. The kubelet sees probe failures, restarts the pod, and the restart clears the overload state, masking the root cause. Operators see “pod restarted, looks fine now” and never learn the overload manager was active.

The fix is bypass_overload_manager: true on the listener that serves health probes. This tells the overload manager to skip that listener when applying actions. The main application listener still gets protection; the probe listener stays reachable so the kubelet can confirm liveness.

Deprecated downstream connection limit runtime key

The overload.global_downstream_max_connections runtime key has been superseded by the envoy.resource_monitors.downstream_connections resource monitor. If your config uses the runtime key, migrate to the resource monitor for proper stats integration and forward compatibility.

reset_high_memory_stream and HTTP/2

The reset_high_memory_stream action resets streams consuming the most memory. Its primary use case is HTTP/2, where multiplexed long-lived streams can accumulate large buffers per connection. If your workload is HTTP/1.1-heavy or uses long-polling over HTTP/1.1, do not rely on this action alone; you are depending on stop_accepting_connections and shrink_heap to shed load.

Prevention

• Configure the overload manager in every deployment. Its absence is the most common cause of preventable Envoy OOM kills in production. It belongs in every bootstrap config, sidecar injection template, and Helm chart default.
• Set max_heap_size_bytes below the container limit. Leave headroom for non-heap memory. A practical starting point is 80-85% of the container memory limit.
• Prefer cgroup_memory for containerized deployments. It reports what the kernel sees, which is what triggers OOM kills. The fixed_heap monitor is more appropriate for bare-metal or VM deployments where Envoy is the primary memory consumer.
• Set bypass_overload_manager on probe listeners. Without this, overload actions cause probe failures that restart the pod and hide the root cause.
• Monitor scale_percent for early warning. Actions activate at 100%, but scale_percent trending upward gives you minutes of warning before traffic rejection starts. Alert on sustained values above 80.
• Load test the cascade. Simulate memory pressure in a non-production environment and confirm actions fire in the expected order before the OOM kill.

How Netdata helps

• Per-second collection of server.overload_manager.envoy.overload_actions.* gauges means you see the exact second an action activates and when it clears, not just “Envoy was down for 3 minutes.”
• Overlaying overload action state with server.memory_allocated and server.memory_heap_size on the same timeline shows whether memory growth preceded the action, distinguishing a leak from a traffic-driven spike.
• The downstream_cx_overload_reject counter, plotted alongside the action gauges, quantifies how many connections were refused during the overload window.
• Container-level cgroup memory metrics alongside Envoy’s internal memory stats expose the fixed_heap vs cgroup mismatch that causes OOM kills without overload manager activation.

Related guides

• Envoy 502 and upstream resets: rx_reset, tx_reset, and mid-response failures
• Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend
• Envoy 504 upstream timeout: upstream_rq_timeout, per-try timeouts, and the UT flag
• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy clusters stuck warming: warming_clusters non-zero and routes returning 503
• Envoy connection pool exhaustion: a slow upstream that fills the pool
• Envoy control_plane.connected_state = 0: running on stale xDS config
• Envoy downstream_rq_time high: client-observed latency and proxy overhead
• Envoy health checks vs outlier detection: two systems that eject hosts differently
• How Envoy actually works in production: a mental model for operators
• Envoy listener_create_failure: a listener config Envoy could not apply
• Envoy membership_healthy dropping: reading the single most important cluster signal