Memcached curr_connections climbing: connection leaks and missing pooling

curr_connections trends upward with no corresponding traffic increase, edging toward the -c ceiling (default 1024). When it hits the limit, the listen socket disables: accepting_conns flips to 0 and new client TCP connects are refused. To the application it looks like memcached is down. To memcached it is idle and healthy on the connections it already holds.

Most of the time this is not a memcached bug. It is a client-side problem: connections opened and never returned, a broken or absent pool, or a fleet of application instances each holding more sockets than you accounted for. The server is the victim, not the cause.

The reflex fix is “raise -c”. That buys runway but masks the leak, and it pushes you toward response_obj_oom once connection buffer memory becomes the constraint instead of slot count. The real work is deciding whether you have a leak (curr_connections grows, total_connections rate is normal) or churn (curr_connections is stable, total_connections rate is very high), then tracing it to the offending client.

What this means

Memcached has a hard connection limit set by -c (default 1024). A main listener thread accepts connections and hands them round-robin to worker threads (-t, default 4). When curr_connections reaches max_connections, the listen socket is temporarily disabled: listen_disabled_num increments, accepting_conns reads 0, and new connects are rejected. The cumulative duration is captured in time_in_listen_disabled_us.

The cliff is binary: below the limit everything works, at the limit new clients are denied with no graceful degradation. The useful window for diagnosis is the slope, not the cliff.

curr_connections is a gauge that includes memcached’s own internal connections (the listening socket, internal pipe, and management sockets), typically a baseline of 3 to 5. It also counts your monitoring agents. The “real” client count is a few lower than the raw number, and capacity planning needs to reserve slots for both.

Two distinct failure modes produce a “connections are a problem” symptom, and the remediation differs for each:

• Leak. curr_connections grows monotonically. total_connections rate is normal or only mildly elevated. Sockets are opened and held, never closed. This is the dangerous one: it does not self-correct, and the only outcomes are hitting -c or restarting the offending clients.
• Churn. curr_connections is roughly stable, but total_connections rate is very high. Clients connect, issue a request, disconnect, and repeat. curr_connections can look fine in a dashboard and then spike when churn outpaces TIME_WAIT reaping on the host.

Churn tends to show up as client-side TIME_WAIT exhaustion or as elevated total_connections rate rather than as a monotonic curr_connections rise.

flowchart TD
    A["curr_connections trending up"] --> B{"total_connections rate high?"}
    B -- No --> C["Leak: sockets held open"]
    B -- Yes --> D{"curr_connections stable?"}
    D -- Yes --> E["Churn: connect/disconnect cycling"]
    D -- No --> F["Leak + churn: investigate both"]
    C --> G["Per-client socket count via ss"]
    E --> H["Enable client-side pooling"]
    C --> I["Fix pool / close paths in app"]
    F --> G

Common causes

Quick checks

These are read-only and safe to run during an incident.

# Daemon responsiveness and version
echo "version" | nc -q1 localhost 11211

# Current vs max connections, accept state, and rejection counters
echo "stats" | nc -q1 localhost 11211 \
  | grep -E "STAT (curr_connections|max_connections|accepting_conns|listen_disabled_num|total_connections|rejected_connections|time_in_listen_disabled_us)"

# Two samples of total_connections to compute the new-connection rate
T0=$(echo "stats" | nc -q1 localhost 11211 | awk '/STAT total_connections/{print $3}'); sleep 10
T1=$(echo "stats" | nc -q1 localhost 11211 | awk '/STAT total_connections/{print $3}')
echo "new_conns_per_sec: $(( (T1 - T0) / 10 ))"

# OS file descriptor limit for the memcached process (must be >= max_connections)
PID=$(pgrep -x memcached); cat /proc/$PID/limits | grep "open files"

# Per-client connection count, ranked
# $5 is the peer address:port; -p adds local process names if run as root
ss -tnp | grep ":11211" | grep ESTAB | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head

# TIME_WAIT count on the memcached port (client-side churn indicator)
ss -tan state time-wait | grep ":11211" | wc -l

The ss -tnp peer IPs are visible without root. Root is needed for -p to resolve the local process name, which confirms the connections belong to memcached but is not required to identify offending clients.

How to diagnose it

1. Confirm it is a leak, not churn. Sample total_connections twice over a 10 to 30 second window. If curr_connections is climbing but total_connections rate is low, sockets are being held open: a leak. If curr_connections is roughly flat but total_connections rate is high, clients are churning through connects. The two need different fixes.
2. Compute runway. Use (max_connections - curr_connections) / new_connection_rate. If the leak adds 5 connections per minute and you have 200 slots free, you have 40 minutes. That tells you whether this is a “page now” or a “fix in the next deploy” problem.
3. Find the offending client. Group established connections by peer IP. A single application host holding hundreds of sockets is the leak source. A broad, even distribution means the problem is fleet-wide pooling policy, not one bad instance.
4. Check whether the OS fd limit is the real ceiling. cat /proc/$PID/limits | grep "open files" must be at least max_connections. If the kernel limit is lower than -c, the kernel wins and connections fail before memcached’s own counter trips.
5. Check for server-side buffer pressure. If curr_connections is very high, watch response_obj_oom (since 1.6.0). Connection buffers are allocated outside the -m slab pool, and raising -c without considering process memory can trigger response object OOMs that close otherwise healthy connections.
6. Rule out a server-side leak. connection_structures should track curr_connections roughly. If it grows well out of proportion to live connections, that points to an internal accounting issue rather than a client problem.

Metrics and signals to monitor

Fixes

Leaking client connections

A leak means sockets are opened and never closed. The memcached side cannot fix this. The work is in the application.

• Identify the offending client from the ss grouping above.
• Audit every code path that constructs a memcached client. Look for paths that create a client per request, per thread, or per loop iteration without a corresponding close or return-to-pool.
• If the client library exposes pool metrics (max size, idle count, active count), check whether active count grows without bound under load. A correctly bounded pool should plateau, not climb.
• Restarting the offending application instances reclaims the sockets immediately, but only buys time. Without a code or config fix the leak returns.

Restarting clients is disruptive and loses in-process state, but it is faster than waiting for a code deploy when you are hours from the limit.

Missing or disabled pooling

If every request opens a new TCP connection, total_connections rate is high and curr_connections may look stable until TIME_WAIT or ephemeral port exhaustion on the client causes a spike. This is churn, not a leak.

• Enable persistent connections in the client library. Most memcached clients support a persistent connection mode or a built-in pool.
• For PHP specifically, persistent connections only help if the PHP process survives between requests. With short-lived PHP-CGI or mod_php workers, “persistent” connections are effectively non-persistent because the process dies and takes the socket. PHP-FPM with persistent connection IDs is the working pattern.
• Set an idle timeout on the pool so abandoned connections are reaped. Without one, a slow leak in pool accounting still accumulates.
• Verify the pool size is bounded. A pool with no max is just a leak with extra steps.

-c too low for the fleet

The default 1024 is often too low for production. If you have 50 application instances each holding 30 persistent connections, you are already at 1500.

• Raise -c. This requires a restart unless your deployment supports runtime reconfiguration.
• Account for monitoring agents and the 3 to 5 internal baseline connections.
• Keep curr_connections below 70% of max_connections as a planning target. The headroom absorbs reconnection storms after deploys and brief bursts.

Each connection consumes roughly 10KB of buffer memory outside the slab pool. Raising -c to 50000 without considering process memory risks response_obj_oom. Size the limit against actual peak persistent connection count plus headroom, not against a round number.

OS ulimit below -c

If /proc/$PID/limits shows an open-files limit lower than max_connections, the kernel enforces the lower number and connections fail before memcached’s own counter trips.

• Raise the memcached process fd limit via systemd LimitNOFILE, an init script ulimit -n, or the orchestrator’s equivalent.
• In containerized deployments, check the container runtime’s ulimit. Docker inherits a default that may be much lower than expected. A common workaround is --ulimit nofile=16384 on the docker run command.
• The fd limit must cover max_connections plus listening sockets plus internal use.

Prevention

• Alert on curr_connections / max_connections above 0.7. This is the warning band. The composite Connection Exhaustion page (accepting_conns = 0 sustained) is the emergency band.
• Track total_connections as a rate, not just a counter. Churn shows up here before it shows up in curr_connections, and a sudden rate increase often precedes a deploy-induced reconnection storm.
• Size -c against peak persistent connection count plus 30% headroom. Recompute whenever the application fleet scales horizontally.
• Verify OS fd limits in the same runbook step as -c. The two ceilings must agree.
• Instrument client-side pool metrics. Active, idle, and pending-acquire counts per pool tell you whether the problem is server pressure or client pressure before you touch memcached.
• Watch response_obj_oom when running high connection counts. It is the signal that you raised -c past what the process memory can support.
• Alert on any increment of rejected_connections or listen_disabled_num. Both mean a client was denied service.

How Netdata helps

• Per-second curr_connections and max_connections ratio lets you see the slope, not just snapshots. The rate of climb tells you whether you have minutes or hours of runway.
• total_connections rate as a derived metric separates leak from churn at a glance. A climbing curr_connections with a flat total_connections rate is the leak signature.
• accepting_conns, listen_disabled_num, time_in_listen_disabled_us, and rejected_connections together form the connection exhaustion composite. Correlating them with cmd_get and cmd_set rates confirms whether saturation is traffic-driven or leak-driven.
• ML anomaly detection on the curr_connections slope catches slow leaks that never trip a static threshold but still drain the slot pool over hours.
• conn_yields and response_obj_oom surface the secondary effects of high connection counts: unfair pipelining and buffer memory exhaustion. These appear before accepting_conns flips.
• Correlation with per-host TCP connection state (TIME_WAIT, ESTAB counts from the OS-level collector) localizes churn to specific client hosts without needing root on the memcached box.

Related guides

• Memcached connection limit reached: accepting_conns=0 and clients being refused
• Memcached connection refused: telling a dead process from a hung or full one
• How Memcached actually works in production: a mental model for operators
• Memcached eviction cascade: when a full cache overloads the backend
• Memcached evictions climbing: the cache is full and discarding live data
• Memcached evicted_time low: distinguishing healthy turnover from cache thrash
• Memcached memory utilization: bytes vs limit_maxbytes and why the global number lies
• Memcached hit ratio dropping: reading get_hits, get_misses, and cache effectiveness
• Memcached high miss rate: separating cold start, new key patterns, and memory pressure
• Memcached evicted_unfetched and expired_unfetched: caching data nobody ever reads
• Memcached cas_badval climbing: check-and-set contention and lost updates
• Memcached incr/decr misses: evicted counters that silently break rate limiters and locks