Memcached connection refused: telling a dead process from a hung or full one

“Connection refused” from a memcached client is commonly misdiagnosed. Operators run a TCP port check, see that the port is open, and act on that single data point. A port test tells you whether the kernel accepts a TCP SYN on 11211, not whether memcached is processing commands.

Three distinct failure modes all surface to clients as “cannot reach memcached”:

1. Process dead: crashed, OOM-killed, or never owned the port. The kernel refuses the connection.
2. Process hung: alive but unresponsive. The kernel accepts the connection because the listen socket is open, but commands never return.
3. Connection limit reached: healthy but saturated. New connections are rejected, queued indefinitely, or accepted but never serviced.

Each requires a different response. Restarting a hung process without understanding why it hung will reproduce the hang. Raising the connection limit on a dead process does nothing.

The diagnostic move is to send an actual command (version or stats) and require a response within a few seconds. This separates all three modes before you touch dmesg, ss, or stats settings. Require three consecutive failures over 30-60 seconds before escalating: transient blips during deploys and reconnection storms are real.

What this means

The kernel returns ECONNREFUSED (errno 111) when nothing is listening on the destination IP:port. A full listen backlog on Linux does not produce ECONNREFUSED by default; the kernel drops the SYN silently and the client sees a timeout. For memcached, ECONNREFUSED almost always means the daemon is not bound to the address your client is using, or the process has exited.

A TCP connection that succeeds only tells you the listen socket accepted your SYN. It does not confirm that a worker thread read your command, parsed it, or wrote a reply. A memcached process that is deadlocked, stopped, swap-thrashing, or stuck in a long-running internal operation can still have an open listen socket fed by the kernel. From the client’s perspective, this looks like a connection that hangs or times out rather than a clean refuse.

The third mode is subtler. When curr_connections reaches max_connections, memcached sets accepting_conns = 0 and stops calling accept(). New SYNs queue in the kernel backlog. Without the maxconns_fast option, those queued connections eventually time out from the client’s perspective. With maxconns_fast enabled, memcached rejects over-limit connections and increments rejected_connections.

flowchart TD
    A["Client reports
connection refused"] --> B{"TCP connect to
IP:port succeeds?"}
    B -- "refused (ECONNREFUSED)" --> C["Process dead/crashed/
OOM-killed/port conflict"]
    B -- "connects" --> D{"version command
responds in 2s?"}
    D -- "no response" --> E["Process hung:
silent degradation"]
    D -- "VERSION x.y.z" --> F{"accepting_conns = 0
or new conns refused?"}
    F -- "yes" --> G["At connection limit:
curr_connections near max"]
    F -- "no" --> H["Daemon healthy:
look upstream
(firewall, IP, LB)"]

Common causes

Quick checks

All read-only. Run as a first pass.

# Send a real command and require a response. This is the single most important check.
echo "version" | nc -w 2 localhost 11211
# Expected: VERSION x.y.z
# Empty or timeout: process is dead, hung, or unreachable.

# Confirm the process is running and capture its state.
pgrep -a memcached
ps -o pid,ppid,stat,etime,rss,cmd -p "$(pgrep -x memcached)"
# State D (uninterruptible sleep) or T (stopped) indicates a hang, not a crash.

# Confirm the listening socket: which interface, which port, which PID owns it.
ss -tlnp | grep 11211

# Check whether the server is at its connection limit.
echo "stats" | nc -w 2 localhost 11211 | grep -E 'STAT (curr_connections|max_connections|accepting_conns|rejected_connections|listen_disabled_num|time_in_listen_disabled_us)'

# Check for OOM kills.
dmesg -T | grep -Ei 'killed process|oom-killer|out of memory' | tail -20

# Check for unexpected restarts.
echo "stats" | nc -w 2 localhost 11211 | grep -E 'STAT (uptime|pid|time)'

# Check whether the process has been swapped (catastrophic for an in-memory cache).
grep -E 'Vm(RSS|Swap)' /proc/"$(pgrep -x memcached)"/status

# Check the configured bind address, threading, and maxconns.
echo "stats settings" | nc -w 2 localhost 11211 | grep -E 'STAT (inter|udpport|num_threads|maxconns)'

How to diagnose it

Place the failure into exactly one of three buckets before changing anything.

Step 1: send a command, not a SYN. From the memcached host, run echo "version" | nc -w 2 localhost 11211. Three outcomes:

• TCP connect refused immediately: go to step 2.
• TCP connects, no response within 2 seconds: go to step 3 (likely hung).
• VERSION x.y.z returns cleanly: the daemon is alive on localhost. Go to step 4.

Step 2: confirm the process is dead and find the cause. Run pgrep -a memcached. If nothing returns, the process is gone. Check:

• systemctl status memcached (or your supervisor) for exit status and recent restart attempts.
• journalctl -u memcached --since "30 min ago" for stderr output.
• dmesg -T | grep -Ei 'killed process|oom'. The Linux OOM killer log line names the process, PID, and oom_score.

If the process is running but the port is still refused, you have a port conflict or wrong bind address. Use ss -tlnp | grep 11211 to confirm which PID owns the port, and stats settings | grep inter to check the configured bind address. A memcached bound to 127.0.0.1 will refuse connections from an external interface even when perfectly healthy.

Step 3: confirm the process is hung, not just slow. A single slow version response is not proof. Require three consecutive failures spaced over 30-60 seconds, with uptime stable across checks. A fresh start (low uptime) that fails to respond is a startup failure, not a hang.

While the checks run:

• ps -o pid,stat,etime -p <pid>: state D (uninterruptible sleep, usually I/O) or T (stopped) confirms a kernel-level stall. State R or S with no response suggests userspace deadlock.
• grep VmSwap /proc/<pid>/status: any nonzero value means the process is partially on disk. This is a production incident in its own right.
• cat /proc/loadavg: severe memory pressure shows elevated load with no obvious CPU consumer, often the OOM killer winding up.

If you can still get a stats response from an existing connection (some monitoring tools hold one open), check cmd_get and cmd_set rates. Near-zero throughput with stable uptime and accepting_conns = 1 is the signature of silent process degradation.

Step 4: if localhost is healthy, check the connection limit and network path. Run stats | grep -E 'accepting_conns|curr_connections|max_connections|rejected_connections|listen_disabled_num'. Interpret:

• accepting_conns = 0 with curr_connections near max_connections: the server is saturated. See the connection-limit fix below.
• accepting_conns = 1 and rejected_connections is zero: memcached is fine. The refuse is happening upstream: firewall, network partition, wrong destination IP, or a load balancer with a stale health check.

Then test from the client host: echo "version" | nc -w 2 <memcached-ip> 11211. If that fails but the localhost check passes, run ss -tn state established '( sport = :11211 )' on the memcached host to see who is connected, and check iptables -L -n and any cloud security groups.

Metrics and signals to monitor

Fixes

Treat each branch separately. Do not restart memcached reflexively: every restart is total cache loss and a thundering herd on the backend.

Process dead or crashed

Recover via your supervisor (systemctl start memcached). Before you do, capture evidence: the OOM killer line, the supervisor’s exit status, and any stderr. A crash with no log usually means OOM kill or an external signal.

If the OOM killer is the cause, the fix is on the host:

• Check /proc/<pid>/status for VmRSS against the host’s free memory. Memcached RSS should be roughly limit_maxbytes plus overhead for the hash table and connection buffers.
• Verify the cgroup memory limit if running in a container. The cgroup OOM killer fires on cgroup limits independently of host memory.
• Consider -k (mlockall) to prevent swap and turn memory pressure into an earlier, louder allocation failure. Reduce -m or move other processes off the host.
• Confirm ulimit -n for the memcached process is at least as high as max_connections. The OS file descriptor limit silently caps the daemon below its configured connection limit.

Process hung (silent degradation)

A hung process requires a restart to recover service, but the diagnostic value of the hung state is high. Capture what you can before killing it:

• A few stack traces via gdb -p <pid> -batch -ex 'thread apply all bt'. Worker threads stuck in the same function point to a deadlock. Threads in __GI___libc_malloc or madvise point to allocator or memory pressure.
• /proc/<pid>/status for VmSwap and VmRSS.
• /proc/<pid>/wchan per thread for kernel wait reasons.

Then restart via your supervisor. Do not skip evidence collection: a hang that reproduces is much easier to root-cause with one captured trace than with three clean restarts.

If VmSwap is nonzero, the underlying problem is host memory pressure, not memcached. Adding -k converts silent swap thrash into an earlier allocation failure and prevents the disk-latency regime entirely.

Connection limit reached

Raising -c requires a restart and is the wrong first move if the cause is a connection leak. Before touching the limit:

• ss -tnp | grep :11211 | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn: which clients hold the most connections?
• Sample total_connections twice with a known interval. If curr_connections is pegged at max but the total_connections rate is very high, clients are churning connections (no pooling). If curr_connections is pegged and the total_connections rate is low, connections are leaking: held open and never closed.
• Check client-side connection pool settings. Persistent connections are the standard fix for churn.

If the workload legitimately needs more concurrent connections, raise -c on next restart and raise ulimit -n to match. Memcached will not exceed the OS limit. Consider enabling -o maxconns_fast so over-limit clients get a fast refuse instead of a multi-second queue-and-timeout, but expect rejected_connections to start incrementing.

Wrong bind address or port conflict

If ss -tlnp | grep 11211 shows a different PID owning the port, identify and stop that process before restarting memcached. If memcached is bound to 127.0.0.1 and clients use the external IP, change the -l flag to bind the correct interface and restart. Always pair a bind-address change with firewall rules: exposing memcached to untrusted networks without SASL is a serious security issue. UDP should remain off.

Prevention

• Command-probe alerting, not port-check alerting. Alert on version probe failure sustained for 3 checks over 30-60 seconds, with uptime previously stable. A bare TCP port check is not a liveness signal.
• Connection-exhaustion composite alert. Alert on accepting_conns = 0 sustained for 5 minutes with curr_connections / max_connections > 0.98. Brief reconnection storms after deploys should not page.
• Swap alerting. Alert on any nonzero VmSwap for the memcached process. Treat even single-digit megabytes as an incident.
• Flush alerting. Alert on any cmd_flush increment in production.
• Restart alerting. Alert on uptime discontinuities.
• Leading indicators. Track listen_disabled_num. Any non-zero value means the server has hit its connection limit at least once.
• File descriptor headroom. Set ulimit -n to at least max_connections + headroom, and verify it in the supervisor unit.
• Lock memory. Run with -k (mlockall) on hosts where you want swap pressure to surface as an allocation failure rather than silent latency.
• Know your version. Segmented LRU, slab_automove defaults, and UDP default behavior changed across 1.4.x, 1.5.x, and 1.6.x. The stats settings output tells you which features are active.

How Netdata helps

• The memcached collector issues real command probes (stats, version) per second rather than bare TCP checks. A failed command response with a live port surfaces as an anomaly instead of a false green.
• Per-second curr_connections, accepting_conns, rejected_connections, and listen_disabled_num let you correlate a connection-limit event with the exact moment accepting_conns flips to 0.
• ML anomaly detection on cmd_get and cmd_set rates catches the silent-degradation signature: throughput collapsing while uptime stays stable and the port stays open.
• Host-level collectors (OOM killer events, per-process RSS and swap, cgroup memory limits) sit alongside the memcached metrics, so cause and effect are visible on the same timeline.
• Composite alerting on sustained accepting_conns = 0 plus curr_connections / max_connections > 0.98 avoids pages from deploy-time storms while still paging on real saturation.