$ guides / memcached / memcached-response-obj-oom ▌

Operations Guides

Memcached response_obj_oom: connections killed for lack of buffer memory

If response_obj_oom is climbing in your stats output, the daemon is closing client connections because it cannot allocate internal response buffers. This is not an item-storage problem and not an eviction problem. A cache can be otherwise healthy (zero evictions, free slab memory, low CPU) and still kill clients over this.

The stat has existed since 1.6.0, when memcached reworked its connection buffer management to allocate buffers on demand instead of reserving them per connection. That cut idle connection overhead but introduced a new failure surface: when the pool of memory for response objects runs out, the daemon closes the connection rather than blocking or queueing. The close is immediate when the allocation fails. There is no retry, no backpressure, no graceful degradation. Applications see timeouts or resets against a daemon that still answers version and stats probes.

The most common trigger is raising -c (max connections) without accounting for the per-connection buffer memory that high connection counts now demand. If you recently raised -c from a few thousand to tens of thousands, start there.

What this means

response_obj_oom is a monotonically increasing counter. Each increment is one client connection closed because a response buffer object (the internal structure used to assemble outgoing data before writing to the client socket) could not be allocated. Response buffers live outside the slab allocator and do not compete with cached items for the -m budget.

Three stats are easy to confuse. Which one is climbing changes the diagnosis entirely:

store_no_memory: SET operations rejected because the slab allocator has no room. Item-storage exhaustion, addressed by adding cache memory or fixing slab imbalance.
response_obj_oom: connections closed because response-buffer memory could not be allocated. Connection-buffer exhaustion, addressed by reducing concurrent connection pressure or raising the buffer memory budget.
read_buf_oom: the parallel stat for the read side. Same failure shape, different pool.

A cache with abundant free slab memory, zero evictions, and an excellent hit ratio can still be killing clients over response-buffer pressure. If you only watch eviction rate and hit ratio, you miss this class of failure.

From the client side, the close looks like a reset or timeout mid-request. From the server side, it is a deliberate close under memory pressure, not a crash or hang.

Common causes

Cause	What it looks like	First thing to check
Raised `-c` without buffer memory	`curr_connections` high, `response_obj_oom` climbing, evictions zero	`max_connections` setting vs available process memory for buffers
Connection storm	`total_connections` rate spiking, `response_obj_oom` climbing alongside	`curr_connections` trend and client reconnect behavior
Buffer memory limit set too low	`response_obj_oom` climbing at modest connection counts	`stats settings` for buffer memory limits
Large pending responses	`response_obj_bytes` high relative to connection count	Average response size and multiget batch sizes
Host memory pressure	`VmSwap` nonzero, RSS climbing toward system limit	`/proc/<pid>/status` and system free memory

Quick checks

All read-only and safe to run during an incident.

# Check response_obj_oom and the parallel read_buf_oom
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT (response_obj_oom|read_buf_oom)"

# Current response buffer usage (count and bytes)
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT response_obj_(count|bytes)"

# Connection count vs configured limit
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT (curr_connections|max_connections)"

# Connection yields (fairness mechanism under pipeline pressure)
echo "stats" | nc -q1 localhost 11211 | grep "STAT conn_yields"

# Connection churn rate (sample twice, 10s apart)
echo "stats" | nc -q1 localhost 11211 | grep "STAT total_connections"

# Confirm evictions are NOT the problem
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT (evictions|store_no_memory)"

# Check configured buffer memory limit
echo "stats settings" | nc -q1 localhost 11211 | grep -iE "read_buf_mem_limit"

# Process RSS and swap
PID=$(pgrep memcached); grep -E "Vm(RSS|Swap)" /proc/$PID/status

If response_obj_oom is climbing while evictions and store_no_memory are flat, you have confirmed a buffer-memory problem, not a slab problem.

How to diagnose it

flowchart TD
    A["response_obj_oom climbing"] --> B{"evictions also high?"}
    B -- yes --> C["Item storage pressure
not buffer pressure"]
    B -- no --> D{"curr_connections high?"}
    D -- yes --> E{"Recently raised -c?"}
    E -- yes --> F["Buffer pool undersized
for connection count"]
    E -- no --> G["Connection storm
or client misbehavior"]
    D -- no --> H{"response_obj_bytes high?"}
    H -- yes --> I["Large pending responses
big values or multigets"]
    H -- no --> J["Check buffer limit
and host memory"]

Confirm the counter is actually moving. Sample response_obj_oom twice, 10 seconds apart. A flat counter is residual from a past event; a rising counter is an active problem.
Confirm evictions are flat. If evictions is also climbing, you have a slab pressure problem that may be triggering buffer OOM as a side effect. Diagnose the slab problem first.
Check curr_connections against max_connections. If connections are near the limit, the buffer pool is carrying close to its maximum concurrent load. Each connection with a pending response holds buffer memory.
Check response_obj_count and response_obj_bytes. These gauges show how many response objects are in flight and how much memory they consume. High response_obj_bytes relative to connection count means individual responses are large.
Check conn_yields. If yields are climbing alongside response_obj_oom, a small number of clients are pipelining aggressively, inflating pending responses per connection.
Check total_connections rate. High churn (rapid connect/disconnect) stresses the buffer allocator differently from a stable high connection count. Churn indicates missing client-side connection pooling.
Check read_buf_oom alongside response_obj_oom. If both are climbing, the entire connection-buffer subsystem is under pressure. If only response_obj_oom is climbing, the problem is specifically on the write path.
Check host-level memory. VmSwap for the memcached process must be zero. Any swap is catastrophic for latency and suggests the host is oversubscribed. Compare RSS against system free memory.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`response_obj_oom`	Direct count of connections killed for buffer memory	Any sustained nonzero rate at scale
`read_buf_oom`	Parallel signal for read-buffer exhaustion	Climbing alongside `response_obj_oom` means systemic buffer pressure
`response_obj_count`	Gauge of response objects currently in flight	Trending toward a ceiling suggests buffer pool saturation
`response_obj_bytes`	Memory consumed by in-flight response objects	High relative to connection count means large responses
`curr_connections`	Concurrent connection pressure on buffer pool	Approaching `max_connections` after a `-c` raise
`conn_yields`	Pipeline fairness mechanism activity	Rising alongside `response_obj_oom` indicates aggressive clients
`evictions`	Rules out slab pressure as the cause	Flat while `response_obj_oom` rises confirms buffer-only problem
`VmSwap`	Host memory oversubscription	Any nonzero value is a production incident for a cache

Fixes

Raised `-c` without buffer memory

The most common cause and the easiest to confirm. If curr_connections is high and response_obj_oom started climbing after a -c increase, the buffer pool cannot keep up with the concurrent connection count.

Short term, reduce connection pressure. The fastest lever is fixing client-side connection pooling so fewer connections carry the same workload. Most memcached client libraries support persistent connections; verify they are enabled and that pool sizes are bounded.

Longer term, either lower -c back to a level the buffer pool can sustain, or provision more memory for the process. Each active connection with a pending response holds buffer memory on the order of 10KB. Tens of thousands of concurrent connections means tens to hundreds of megabytes of buffer memory on top of the -m cache budget. Plan for it.

Memcached 1.6.0 introduced a tunable to cap connection-buffer memory: read_buf_mem_limit, specified in megabytes with a default of 0 (unlimited). If this was set explicitly as a guardrail, verify the value is not artificially low.

Connection storm

If total_connections rate is high and curr_connections is volatile, clients are churning connections. Each new connection demands buffer allocation, and rapid churn stresses the allocator and the freelist.

Fix the client side. Enable persistent connections, set sensible idle timeouts, and verify the client library is not opening a connection per request. Check for deployment events, failovers, or config changes that triggered mass reconnection.

Large pending responses

If response_obj_bytes is high relative to response_obj_count, individual responses are large. Big values, large multigets, and bulk response assembly all inflate per-connection buffer demand.

Options: reduce average value size (compression at the client, smaller cache payloads), reduce multiget batch sizes, or provision more buffer memory. If you recently started caching larger objects, this is a likely trigger. 1.6.17 fixed several OOM and excess-eviction bugs specifically for caches with mostly large objects; if you are on an older 1.6.x, upgrading is worth considering.

Buffer memory limit set too low

If response_obj_oom is climbing at connection counts that should be sustainable, someone may have set read_buf_mem_limit as a guardrail. Check stats settings.

If a limit is set and is the constraint, either raise it or set it to 0 for unlimited, and instead size the host memory to absorb the working set plus connection-buffer overhead.

Host memory pressure

If VmSwap is nonzero, the host is oversubscribed and the kernel is swapping memcached pages. This is catastrophic for latency on its own and can cascade into buffer allocation failures.

Add memory to the host, reduce the -m budget, or move neighboring processes off the box. Run memcached with -k (mlockall) to prevent swapping, provided the process has CAP_IPC_LOCK or runs as root.

Prevention

Size buffer memory when you raise -c. Back-of-envelope: concurrent connections times roughly 10KB of buffer headroom each, on top of -m. If you cannot afford that memory, your -c is too high.
Track response_obj_oom and read_buf_oom as Level 3 signals. They are cheap to collect and catch this class of failure before clients notice.
Correlate with curr_connections and conn_yields. A connection-count spike without a corresponding capacity review is the leading indicator.
Verify client-side connection pooling. Churn is the second-most-common trigger after an undersized -c raise.
Keep VmSwap at zero. Any swap on a cache process is a separate incident that compounds buffer pressure.
Know your version. 1.6.0 introduced the on-demand buffer system and these stats. 1.6.17 fixed large-object OOM bugs. Behavior differs across the 1.6.x line.

How Netdata helps

Per-second collection of response_obj_oom and read_buf_oom shows the exact moment connections start being killed, not just the cumulative total at poll time.
Correlating response_obj_oom with curr_connections, conn_yields, and total_connections rate on a single timeline separates a connection-count problem from a large-response problem from a connection-storm problem.
ML anomaly detection flags a rising response_obj_oom rate even when absolute values are still low, which matters because the stat can go from zero to killing clients in minutes after a -c change.
RSS and swap charts for the memcached process catch host-level memory pressure that would otherwise surface as buffer allocation failures.
Eviction and hit-ratio charts let you confirm in one view that the cache itself is healthy while connections are being dropped.