Envoy memory pressure spiral: server.memory_allocated climbing toward OOM

server.memory_allocated is climbing, traffic is flat, and you are hours or minutes from an OOM kill. In a sidecar deployment, the pod dies and the application gets the blame. In an edge or gateway deployment, Envoy vanishes and clients see connection resets across the board.

Envoy has a built-in protection mechanism (the overload manager), but many deployments never configure it. Without it, there is no graceful degradation: Envoy goes straight from “memory looks fine” to “OOM killed” with nothing in between.

What this means

server.memory_allocated tracks the bytes Envoy’s allocator (typically tcmalloc) considers in use by live data structures. server.memory_heap_size tracks the total bytes reserved from the operating system, including freed-but-not-returned regions. The gap between them is where fragmentation and allocator free lists live.

When memory_allocated grows without a corresponding traffic increase, the process is accumulating data structures faster than it releases them. The sources are finite: connection and stream buffers, stats storage, config caches, and the deferred-delete list. One of those buckets is growing without bound.

The critical diagnostic step is comparing memory_allocated against memory_heap_size:

• If memory_allocated is climbing but memory_heap_size is climbing faster, fragmentation is dominating. The allocator holds freed memory rather than returning it to the OS. This is normal tcmalloc behavior and is not itself a leak, but it reduces effective headroom inside the container.
• If memory_allocated tracks closely with memory_heap_size, the process is holding live objects. This points to buffering, connection metadata accumulation, stats cardinality growth, or an actual leak in a filter such as Lua or Wasm.

flowchart TD
    A[memory_allocated grows] --> B{Overload manager?}
    B -->|Not configured| C[OOM kill]
    C --> D[Pod restarts, app blamed]
    B -->|Configured| E[shrink_heap]
    E --> F[disable_http_keepalive]
    F --> G[reduce_timeouts]
    G --> H[stop_accepting_connections]
    H --> I[stop_accepting_requests]
    I --> J[503s to all clients]

The overload manager, if configured, monitors heap pressure via the fixed_heap resource monitor against a configured max_heap_size_bytes and triggers protective actions as pressure increases: shrink_heap, disable_http_keepalive, reduce_timeouts, stop_accepting_connections, and stop_accepting_requests. Each action trades user-visible impact for process survival.

If the overload manager is not configured, Envoy accumulates memory until the container’s cgroup limit is hit and the kernel OOM-kills the process. No warning, no progressive degradation, no signal to correlate.

Common causes

Quick checks

Safe, read-only commands. Use the port that matches your deployment (9901 for standalone Envoy, 15000 for Istio sidecar).

# Current memory stats (allocated, heap_size, physical_size)
curl -s http://localhost:9901/stats | grep 'server.memory'

# Detailed allocator breakdown from the admin interface
curl -s http://localhost:9901/memory

# Overload manager configuration and active actions
curl -s http://localhost:9901/stats | grep 'overload'

# Total stat count to detect cardinality explosion
curl -s http://localhost:9901/stats | wc -l

# Active connections (each connection holds buffers and metadata)
curl -s http://localhost:9901/stats | grep -E 'downstream_cx_active|upstream_cx_active'

# Overload manager action gauges
curl -s http://localhost:9901/stats | grep 'overload_actions.*active'

# Process uptime and state
curl -s http://localhost:9901/server_info | python3 -m json.tool | grep -E 'uptime|state'

# Container memory usage vs limit (cgroup v2, then v1 fallback)
cat /sys/fs/cgroup/memory.current 2>/dev/null || cat /sys/fs/cgroup/memory/memory.usage_in_bytes 2>/dev/null
cat /sys/fs/cgroup/memory.max 2>/dev/null || cat /sys/fs/cgroup/memory/memory.limit_in_bytes 2>/dev/null

How to diagnose it

1. Confirm memory is growing, not just high. Take two samples of server.memory_allocated 60 seconds apart. If the delta is positive and traffic is flat, you have growth. If the value is high but stable, you may have a headroom problem (container limit too low for the workload) rather than a spiral.

2. Check whether the overload manager is configured. Look for server.overload_manager.envoy.overload_actions.shrink_heap.active in stats. If no overload stats appear, the overload manager is not configured. Without it, Envoy has no self-protection and will OOM without warning.

3. Separate allocation from fragmentation. Compare server.memory_allocated to server.memory_heap_size. A low ratio (allocated much smaller than heap_size) means the allocator is holding freed memory. This is normal tcmalloc behavior after a traffic spike but can persist for hours. A high ratio means live objects are accumulating.

4. Check for stats cardinality growth. Run curl -s http://localhost:9901/stats | wc -l. If the count is above 100,000 or growing over time, stats are consuming the shared-memory region. Dynamic stat tags (user IDs, request IDs, per-route names with variable components) are the usual cause. When the stats region fills, new metrics silently fail to register, creating monitoring blind spots.

5. Correlate memory with connections. If downstream_cx_active or upstream_cx_active tracks with memory_allocated, each connection is holding significant buffers or metadata. This points to buffering filters, large HTTP/2 window sizes, or long-lived connections accumulating state.

6. Check the admin /memory endpoint for the detailed breakdown. The endpoint exposes pageheap_unmapped, pageheap_free, and total_thread_cache. A large pageheap_free with small allocated bytes means the allocator is holding freed pages it could release. A large total_thread_cache means per-worker caches are bloated.

7. Look for filter-related growth. If you run Lua or Wasm filters, check whether memory growth started after a filter or Envoy version change. Filter leaks are less common than the other causes but are the hardest to diagnose without heap profiling.

Metrics and signals to monitor

Fixes

If the overload manager is not configured

This is the highest-impact fix. Configure the overload manager in the bootstrap with a fixed_heap resource monitor and max_heap_size_bytes set below the container limit. Envoy’s edge proxy example currently triggers stop_accepting_requests at 95% of max_heap_size_bytes and the tcp_listener_accept load-shed point at 95%, with disable_http_keepalive at 92%. Older docs examples used shrink_heap at 95% and stop_accepting_requests at 98%; the shrink_heap action still exists if you want it.

Set max_heap_size_bytes to about 80 to 90% of the container memory limit to leave room for non-heap overhead (thread stacks, shared memory regions, access log buffers).

If the cause is unbounded buffering

Check for buffer filters and connection-level buffer limits:

• per_connection_buffer_limit_bytes: limits the buffer size per connection. A high default combined with large-body traffic causes memory to scale with connection count.
• HTTP/2 initial_stream_window_size and initial_connection_window_size: large windows allow Envoy to buffer significant data per stream and per connection. If clients or upstreams send large bodies and the peer is slow to read, this memory accumulates.

If a buffering filter is in the chain and buffering entire request or response bodies, evaluate whether full buffering is necessary. Streaming passthrough uses far less memory.

If the cause is stats cardinality

Identify the high-cardinality stat tag source. Common culprits are dynamic route names, per-request metadata injected into stat tags, or user and request IDs used as stat dimensions. Use the stats_matcher configuration (inclusion list or exclusion list) to control which stats are recorded. Alternatively, increase the stats shared-memory region size if legitimate cardinality is high.

After fixing, verify that curl /stats | wc -l stops growing and stabilizes.

If the cause is fragmentation

tcmalloc holds freed memory in free lists and does not aggressively return pages to the OS. After a traffic spike, memory_heap_size can remain elevated for hours even as memory_allocated drops. If the overload manager has shrink_heap configured, it periodically calls the allocator’s release function, which helps return memory to the OS.

If the container’s RSS stays high after traffic subsides and this causes repeated OOM kills, verify that shrink_heap is configured as an overload action and that max_heap_size_bytes is set so the overload manager triggers shrink_heap before the kernel kills the process.

If the cause is a filter leak

Filter leaks require reproduction and profiling. If a Lua or Wasm filter was recently added or updated and memory growth started at the same time, try disabling the filter in a staging environment to confirm. For Wasm filters, check the runtime (V8 vs wamr vs wavm) and the filter version. For Lua filters, check for unbounded table growth or accumulated state in filter-local variables.

Prevention

Configure the overload manager in every production deployment. Set max_heap_size_bytes below the container limit and configure shrink_heap and stop_accepting_requests at appropriate thresholds.

Monitor server.memory_allocated against the container limit, not in isolation. Alert when the ratio exceeds 70 to 80% during steady state.

Track total stat count as a metric. Alert if the count exceeds 100,000 or grows by more than 20% week-over-week without corresponding infrastructure growth.

Set connection-level buffer limits deliberately. Large defaults are fine for low-connection, high-throughput workloads but dangerous for high-connection or large-body workloads.

Run memory growth tests during deployments. Before deploying a new filter, Envoy version, or config change, run a sustained load test in staging and monitor memory_allocated over time.

How Netdata helps

• Netdata collects server.memory_allocated, server.memory_heap_size, and server.memory_physical_size per second, so growth trends are visible within minutes rather than after the OOM.
• Overload manager action gauges (overload_actions.*.active) appear alongside memory metrics in the same dashboard. When memory spikes and stop_accepting_requests activates, the correlation is immediate rather than a post-incident reconstruction.
• ML-based anomaly detection on memory_allocated flags non-linear growth early, before the process reaches the container limit.
• Connection metrics (downstream_cx_active, upstream_cx_active) plotted against memory usage make it obvious whether memory tracks connection count (buffering) or diverges from it (leak or cardinality).
• Per-container memory metrics from the kernel side sit next to Envoy’s own view, so the actual distance to OOM kill is visible in real time.

Related guides

• Envoy 502 and upstream resets: rx_reset, tx_reset, and mid-response failures
• Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend
• Envoy 504 upstream timeout: upstream_rq_timeout, per-try timeouts, and the UT flag
• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy connection pool exhaustion: a slow upstream that fills the pool
• Envoy control_plane.connected_state = 0: running on stale xDS config
• Envoy downstream_rq_time high: client-observed latency and proxy overhead
• Envoy health checks vs outlier detection: two systems that eject hosts differently
• How Envoy actually works in production: a mental model for operators
• Envoy membership_healthy dropping: reading the single most important cluster signal
• Envoy monitoring checklist: the signals every production proxy needs
• Envoy monitoring maturity model: from survival to expert