$ guides / memcached / memcached-connection-churn ▌

Operations Guides

Memcached connection churn: total_connections racing and TIME_WAIT buildup

total_connections is a cumulative counter that only moves up. When it climbs fast while curr_connections stays flat, clients are opening a fresh TCP connection per request and closing it immediately instead of pooling. Each cycle costs a syscall pair on both ends, a socket on both ends, and on Linux the closed socket sits in TIME_WAIT on the client host for roughly 60 seconds.

The daemon stays responsive, hit ratio is unaffected, and memory looks fine. The damage shows up as inflated p99 latency, wasted accept-thread CPU, and a slow march toward either the -c ceiling or ephemeral port exhaustion on the client side. When either cliff is reached, a previously quiet cache starts refusing connections and clients cascade to the backend.

What this means

The signature is a wide gap between two rates. curr_connections is a live gauge that tracks the number of application workers or pool members touching the cache. total_connections is monotonic, so what matters is its first derivative. If the rate of total_connections is many multiples of curr_connections, every slot is being opened, used once or briefly, and closed.

The memcached protocol explicitly encourages clients to cache their connections rather than reopen them every time. When that advice is ignored, three resource pools come under pressure simultaneously.

First, memcached’s single accept thread. The main listener accepts new TCP connections and hands them round-robin to worker threads (-t, default 4). The official performance guide warns that cycling connections very quickly can overwhelm the thread. Throughput on existing connections stays high, but new connection setup latency climbs.

Second, the -c connection limit (default 1024). Churn itself does not raise curr_connections, but it makes the limit far easier to hit. A burst of clients connecting at once, a deploy-time reconnection storm, or a slow client that holds a socket a few hundred milliseconds longer than usual can push curr_connections to the ceiling. At that point accepting_conns flips to 0, listen_disabled_num and rejected_connections increment, and time_in_listen_disabled_us starts accumulating.

Third, the client host’s TCP stack. Closed sockets linger in TIME_WAIT on the side that initiated the close, which for memcached clients is almost always the application host. With a default ephemeral port range of roughly 32768 to 60999 (about 28,000 ports), sustained churn above roughly 470 new connections per second per client host exhausts local ports. Stateful firewalls between client and cache track each connection too, and conntrack tables can fill before the port range does.

flowchart TD
    A[Client opens conn per request] --> B[memcached accept thread]
    B --> C[Worker thread serves request]
    C --> D[Client closes conn]
    D --> E[TIME_WAIT on client ~60s]
    E --> F{Port range full?}
    F -->|no| A
    F -->|yes| G[EADDRNOTAVAIL on client]
    A --> H{curr_connections near -c?}
    H -->|yes| I[accepting_conns=0]
    I --> J[rejected_connections++]
    J --> K[Clients fall to backend]

Common causes

Cause	What it looks like	First thing to check
No client-side pool	`total_connections` rate is a large multiple of `curr_connections`; `cmd_get` rate is modest	Client library config; look for a new connection per request
Pool too small or idle timeout too aggressive	`curr_connections` is stable but low; `total_connections` rate spikes on every traffic burst	Pool min/max and idle timeout settings
PHP-FPM persistent_id misuse	`curr_connections` scales with worker count times server count; one socket per worker per server	`addServer()` call sites and `persistent_id` usage
Deploy or failover reconnect storm	Brief spike in `total_connections` rate, resolves in 1 to 2 minutes	Deployment timeline correlation
Load balancer or NAT terminating idle conns	Connections drop unexpectedly; clients reconnect; churn is steady not bursty	LB idle timeout vs client keepalive
Client library leaking sockets	`curr_connections` trends up monotonically; `connection_structures` diverges from `curr_connections`	`connection_structures` vs `curr_connections` over time

Quick checks

# Confirm the churn signature: high total_connections rate, stable curr_connections
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT (total_connections|curr_connections|max_connections)"
# Sample twice a few seconds apart and compute the delta on total_connections.

# Check whether the server is currently refusing connections
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT (accepting_conns|listen_disabled_num|rejected_connections|time_in_listen_disabled_us)"

# Look for connection_structures divergence (potential leak)
echo "stats" | nc -q1 localhost 11211 | grep -E "STAT (connection_structures|curr_connections)"

# Verify the configured connection ceiling (-c) and the process file descriptor limit
echo "stats settings" | nc -q1 localhost 11211 | grep "STAT maxconns"
grep "open files" /proc/$(pgrep memcached)/limits

# Count TIME_WAIT sockets on the CLIENT host (not the memcached host)
ss -tan state time-wait | wc -l
# Break them down by peer to confirm they point at memcached.
ss -tan state time-wait | awk 'NR>1{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head

# Check the client host's ephemeral port range and fin timeout
cat /proc/sys/net/ipv4/ip_local_port_range
cat /proc/sys/net/ipv4/tcp_fin_timeout

# Identify which clients hold the most open connections to memcached
ss -tnp | grep ":11211" | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head

# Check conntrack saturation if a stateful firewall sits in the path
cat /proc/sys/net/netfilter/nf_conntrack_count 2>/dev/null
cat /proc/sys/net/netfilter/nf_conntrack_max 2>/dev/null

How to diagnose it

Confirm the churn signature. Sample total_connections and curr_connections twice, 5 to 10 seconds apart. Compute delta(total_connections) / delta(seconds). If that rate is a large multiple of curr_connections, you have churn. A rate equal to or below curr_connections means connections are being reused.
Check whether churn has already tipped into rejection. If accepting_conns is 0, or rejected_connections or listen_disabled_num are non-zero and climbing, the -c ceiling has been hit. Clients are already falling through to the backend.
Localize the TIME_WAIT pressure. Run ss -tan state time-wait on the application hosts, not the memcached host. Closed sockets accumulate on the side that initiated the close. If the memcached host itself shows TIME_WAIT, something else is closing first (load balancer, proxy, or the server timing out).
Identify the churning clients. Use ss -tnp on the memcached host grouped by peer IP. A small number of hosts contributing the bulk of total_connections rate points at specific application tiers or deployments.
Distinguish a leak from churn. If curr_connections is monotonically increasing, that is a leak, not churn. Leaks close the gap between curr_connections and max_connections over time and eventually trigger accepting_conns = 0 without any burst. connection_structures climbing while curr_connections drops is a secondary leak indicator.
Correlate with deploy or traffic events. A churn spike that aligns with a deploy, autoscale event, or failover is likely transient. Sustained churn across hours or days points to a structural client-side issue.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`total_connections` rate	Primary churn signal; cumulative counter, derive the rate	Rate is many multiples of `curr_connections`
`curr_connections`	Live connection count; should track pool size, not traffic	Monotonic climb (leak) or pegged at `max_connections`
`max_connections`	The `-c` ceiling; default 1024 is often too low for production	`curr_connections / max_connections` above 0.8
`accepting_conns`	Real-time boolean for whether the listener is disabled	Flips to 0
`rejected_connections`	Count of connections turned away at the ceiling	Any non-zero increment
`listen_disabled_num`	Count of listener-disable transitions	Any non-zero value
`time_in_listen_disabled_us`	Cumulative microseconds spent not accepting	Increasing rate over a 5-minute window
`connection_structures`	Allocated internal connection structs	Diverges upward from `curr_connections`
`conn_yields`	Connections hitting the `-R` per-event request limit (default 20)	Sustained rate above 100/sec
`response_obj_oom`	Connections closed for lack of response buffer memory	Any sustained non-zero rate
Client TIME_WAIT count	Port exhaustion risk on the application host	Count approaching ephemeral range size
Conntrack usage	Stateful firewall table saturation	Count approaching `nf_conntrack_max`

Fixes

Enable or size the client connection pool

This is almost always the real fix. The goal: each application worker or process holds a small set of long-lived connections to each memcached server and reuses them across requests. Expect curr_connections to settle near (workers per host) * (memcached servers) * (hosts) plus a small baseline.

A pool that opens one connection per request is the bug. A pool sized to peak concurrency with aggressive idle eviction is also wrong because it defeats pooling under sustained load. Pick a min and max that match expected steady-state concurrency, and set idle timeouts long enough to survive quiet periods between bursts.

Handle PHP persistent_id carefully

The php-memcached persistent_id feature pools per worker. Each PHP-FPM worker opens one socket per memcached server the first time it touches the pool. With 200 FPM workers and 10 memcached servers that is 2000 connections, regardless of traffic. Either cap FPM workers, shard across fewer memcached servers per pool, or accept the steady-state connection count and size -c accordingly. Calling addServer() repeatedly with the same persistent_id can also inflate connection counts.

Tune the OS only after pooling is in place

If pooling is fixed and TIME_WAIT pressure persists during legitimate bursts, kernel tuning can buy headroom. This is mitigation, not a fix.

net.ipv4.ip_local_port_range: widen the range to give the client more ephemeral ports. This directly raises the ceiling on outstanding TIME_WAIT sockets.
net.ipv4.tcp_tw_reuse: allows new outgoing connections to reuse sockets in TIME_WAIT. This only helps the client side (outgoing connections), which is the memcached client case. The default is 0 on current kernels.
Do not set net.ipv4.tcp_tw_recycle. It was removed in Linux 4.12. Older guides still reference it.
net.ipv4.tcp_fin_timeout controls how long sockets sit in FIN-WAIT-2, not TIME_WAIT. Lowering it does not shorten TIME_WAIT duration, which is hardcoded at 60 seconds in the Linux kernel .

tcp_tw_reuse does nothing for server-side TIME_WAIT. If the memcached host itself is accumulating TIME_WAIT, the close is happening on the server side and the fix is on the clients or intermediaries causing it.

Raise `-c` deliberately, not reflexively

Increasing -c raises memory cost. Each connection consumes roughly 10KB of buffer memory outside the slab allocator. At very high connection counts without corresponding buffer headroom, response_obj_oom can trigger and close connections. Size -c to expected peak concurrency plus headroom for monitoring connections and reconnection storms, and confirm the OS ulimit -n for the memcached process is at least as high. Changing -c requires a restart.

Keep curr_connections below 70 percent of max_connections to absorb bursts. Monitoring agents consume connections too, and curr_connections includes a small baseline of internal connections (listening socket, pipe), typically 3 to 5.

Consider UDP for read-heavy, loss-tolerant workloads

The performance guide suggests UDP for get-heavy workloads where occasional drops are acceptable. UDP avoids TCP connection setup entirely, eliminating churn and TIME_WAIT. This is a workload-specific tradeoff, not a general fix. UDP must be explicitly enabled and firewalled carefully given the historical amplification attack surface. UDP is disabled by default since 1.5.6.

Prevention

Alert on the rate of total_connections, not just curr_connections. A stable curr_connections with a racing total_connections is invisible if you only watch the gauge.
Alert on the composite connection exhaustion pattern. Combine accepting_conns = 0, curr_connections / max_connections > 0.98, a non-zero rate of rejected_connections or time_in_listen_disabled_us, sustained for 5 minutes, with a max_connections > 50 guard to filter tiny instances.
Track client-side TIME_WAIT counts on application hosts that talk to memcached. A slow upward trend predicts port exhaustion before it happens.
Document the expected connection math. (workers per host) * (memcached servers) * (hosts) plus baseline gives the expected steady-state curr_connections. Any large gap between expected and observed is a pooling bug.
Watch connection_structures vs curr_connections. Divergence indicates either a client not closing connections properly or an internal leak.
Size -c and ulimit -n together. A high -c with a low OS FD limit means the OS ceiling is hit first and the symptoms look identical to memcached rejection.

How Netdata helps

Per-second collection of total_connections and curr_connections makes the churn signature visible without manual nc sampling. The rate-of-change on total_connections is computed automatically, so a flat curr_connections alongside a steep total_connections slope is immediately obvious on the same chart.
Composite connection alerts correlate accepting_conns, curr_connections / max_connections, rejected_connections, and time_in_listen_disabled_us rather than relying on any single threshold, which suppresses noise from brief deploy-time spikes.
Anomaly detection on connection counters flags unusual churn patterns, such as a sudden multiple-of-normal total_connections rate after a deploy, before curr_connections reaches the ceiling.
Cross-host correlation lets you overlay client-side TCP state metrics from application hosts with memcached-side connection metrics, so TIME_WAIT buildup on the client lines up timewise with churn on the server.
connection_structures alongside curr_connections surfaces divergence that points to leaks rather than pooling bugs, narrowing the fix from client config to client code.