Memcached connection limit reached: accepting_conns=0 and clients being refused

Memcached enforces a hard ceiling on concurrent client connections via the -c flag (default 1024). When curr_connections reaches that limit, the daemon flips accepting_conns to 0, disables the listen socket, and turns away every new TCP connection. Clients see connection-refused errors or timeouts, fall through to the backend, and the cache stops working for anyone not already connected.

This is purely a connection-slot problem. Memory can be at 40% of limit_maxbytes, CPU idle, evictions zero, and the process fully serving existing connections. The only thing wrong is that the connection table is full.

The definitive real-time signal is accepting_conns = 0. The cumulative counters listen_disabled_num, rejected_connections, and time_in_listen_disabled_us tell you how often and how long this has been happening. The fix is almost always one of: raise -c, raise the OS file-descriptor limit, or stop the client-side behavior leaking or churning connections.

What this means

Memcached uses a single listener thread that accepts connections and distributes them round-robin across worker threads (set by -t, default 4). Each connection occupies a file descriptor and a small amount of buffer memory outside the slab allocator. The -c flag caps how many connections the daemon will hold open at once.

When curr_connections hits max_connections, the listener disables the accept socket and accepting_conns flips to 0. No new TCP connections are accepted. listen_disabled_num increments on each transition into this state, and time_in_listen_disabled_us accumulates the total microseconds spent with the listener off.

Whether rejected clients are tracked in rejected_connections depends on the maxconns_fast runtime option. Without -o maxconns_fast, new connections queue in the OS backlog or are dropped, and rejected_connections stays at 0. With -o maxconns_fast enabled , memcached explicitly accepts each surplus connection, writes an error response, closes it, and increments rejected_connections. If you monitor only rejected_connections and maxconns_fast is off, you miss the problem entirely. listen_disabled_num and accepting_conns are the reliable signals in both modes.

flowchart TD
    A[Clients open connections] --> B[curr_connections climbs]
    B --> C{curr_connections >= max_connections?}
    C -- no --> D[accepting_conns = 1 - normal]
    C -- yes --> E[accepting_conns = 0 - listener off]
    E --> F[listen_disabled_num increments]
    E --> G{maxconns_fast enabled?}
    G -- no --> H[new connections queue or drop]
    G -- yes --> I[accept, error, close - rejected_connections++]
    H --> J[clients see refused or timeout]
    I --> J
    J --> K[cache miss load hits backend]

From the client perspective, memcached looks down. The connection is refused or times out, so the application treats it as a cache failure and queries the database. If enough clients are refused simultaneously, the backend absorbs the full production read load and may saturate.

Common causes

Quick checks

These are safe read-only commands. Run them against the memcached instance on its configured port (default 11211).

# Real-time accept state and connection counts
echo "stats" | nc -w 2 localhost 11211 | grep -E "STAT (accepting_conns|curr_connections|max_connections|listen_disabled_num|rejected_connections|total_connections)"

# Configured maxconns and whether maxconns_fast is active
echo "stats settings" | nc -w 2 localhost 11211 | grep -E "STAT (maxconns|maxconns_fast)"

# Cumulative time spent with listener disabled
echo "stats" | nc -w 2 localhost 11211 | grep "STAT time_in_listen_disabled_us"

# OS file descriptor limit for the memcached process
PID=$(pgrep -x memcached)
cat /proc/$PID/limits | grep "Max open files"

# Connections to memcached by source IP, sorted by count
ss -tn | awk '$3 ~ /:11211$/ || $4 ~ /:11211$/ {split($4,a,":"); print a[1]}' | sort | uniq -c | sort -rn | head -20

# Connections in TIME_WAIT to the memcached port
ss -tn state time-wait | grep ":11211" | wc -l

Note: ss -tnp (with -p) requires root or CAP_NET_PTRACE to show process info.

How to diagnose it

1. Confirm the server is at its connection limit. Check accepting_conns. If it is 0, the server is currently refusing new connections. If it is 1 but you saw a brief refusal, check listen_disabled_num and time_in_listen_disabled_us for recent increments. A nonzero listen_disabled_num means the limit was hit at least once since process start.

2. Check the configured limit against actual usage. Pull max_connections from stats and compare to curr_connections. If curr_connections is at or within a few connections of max_connections, the limit is the bottleneck. curr_connections includes internal connections (listening socket, pipe, management), typically 3 to 5 baseline, so the actual client count is slightly lower than the stat shows.

3. Verify the OS file descriptor limit is not lower than maxconns. If the OS ulimit -n for the memcached process is lower than -c, the OS limit wins. The daemon may log a setrlimit warning at startup, or silently run with fewer available fds than configured. This is a common trap in containerized deployments.

4. Distinguish churn from a leak. This is the critical diagnostic fork:

   • Leak: curr_connections climbs monotonically and never drops, even when traffic is low. total_connections also climbs but at a moderate rate. Something is opening connections and never closing them.
   • Churn: curr_connections is high but relatively stable, while total_connections increases very rapidly. Clients are opening and closing connections constantly without pooling. Closed sockets linger in TIME_WAIT for roughly 60 seconds (Linux default), consuming ephemeral ports and sometimes fd slots.

5. Identify which clients hold the most connections. Use ss -tnp to see source IPs. If one application host or pod holds a disproportionate share, investigate its connection pool configuration. In Kubernetes or container environments, source IPs may be SNAT’d, so correlate with client-side metrics if possible.

6. Check whether monitoring itself is a significant consumer. Each monitoring agent that connects to memcached consumes a connection slot. Multiple health checks, stats scrapers, or dashboards against the same instance add up. Account for them in capacity planning.

Metrics and signals to monitor

For alerting, a composite rule avoids false positives better than any single threshold. Combine accepting_conns = 0, curr_connections / max_connections above a high threshold (for example 0.98), and a positive rate on rejected_connections or time_in_listen_disabled_us, sustained for several minutes. A sustained duration filters brief spikes from deploys, failover reconnection storms, and batch processing bursts that self-resolve in 1 to 2 minutes. Add a floor (for example max_connections > 50) to exclude tiny dev instances.

Fixes

maxconns is too low

The default -c 1024 is often insufficient for production. If your application fleet has grown beyond what the connection limit supports, raise it.

Changing -c requires a process restart. Memcached does not support changing the connection limit at runtime. Plan for a cold cache after restart.

When raising -c, also raise the OS file descriptor limit. Each connection consumes roughly 10KB of buffer memory outside the slab allocator, so 10,000 connections means about 100MB of non-cache memory. On very large connection counts (above 50,000), watch for response_obj_oom, which indicates the internal response buffer pool is undersized for the connection count.

Connection leak in the application

If curr_connections climbs steadily and never drops, a client is opening connections without closing them. This is a code-level fix in the application or its memcached client library.

Common sources: a new client object created per request without a destructor, a connection pool that grows under load but never shrinks, or a health check that opens a connection, runs a command, but does not close the socket.

Until the code is fixed, the only operational mitigation is to restart the leaking client process or periodically restart memcached. Restarting memcached loses all cached data and should be a last resort.

Connection churn with no pooling

If total_connections is climbing fast but curr_connections is relatively stable, clients are not reusing connections. Each request opens a new TCP connection, sends commands, and closes it. The closed socket enters TIME_WAIT for roughly 60 seconds, consuming ephemeral ports and potentially fd slots.

Fix this on the client side by enabling persistent connections or connection pooling in the memcached client library. Most mature client libraries support pooling. The application framework may need configuration to share a pool across requests or threads.

OS ulimit below maxconns

If memcached cannot raise its file descriptor limit to match -c, it runs with whatever the OS allows. Check cat /proc/<pid>/limits | grep "Max open files" for the actual limit.

For systemd-managed deployments, set LimitNOFILE in the service unit. For Docker, the official memcached image inherits the container runtime’s default, which can be as low as 1024. Use docker run --ulimit nofile=4096:4096 (or higher) to match your -c value. Verify after restart that the process limit matches expectations.

maxconns_fast considerations

If you rely on rejected_connections for alerting, make sure -o maxconns_fast is enabled. Without it, that counter stays at 0 even when connections are being refused. The tradeoff: with maxconns_fast, surplus connections are explicitly accepted, error-responded, and closed, which adds a small amount of per-rejection work. Without it, connections queue or are dropped by the kernel.

Most production deployments benefit from maxconns_fast because it gives clients a fast, explicit failure rather than a timeout. The client can immediately fall through to the backend or retry against a different node.

Prevention

• Size maxconns for peak plus headroom. Track peak curr_connections over weeks. Set -c to at least 1.5x the peak, accounting for reconnection storms after deploys and failovers.
• Monitor the ratio, not just the absolute count. Alert when curr_connections / max_connections exceeds 0.80 sustained. This gives lead time before the hard limit.
• Account for monitoring connections. Every stats scraper, health check, and dashboard consumes a slot. Include them in capacity planning.
• Verify the OS fd limit after every deployment change. Container orchestration, systemd unit updates, and base image changes can silently reset ulimits.
• Enforce client-side connection pooling. This is the single most effective preventive measure. Without pooling, a modest traffic increase can exhaust connection slots through churn alone.
• Watch for response_obj_oom on large maxconns values. If you push -c above 50,000, monitor response_obj_oom to ensure the internal buffer pool can keep up with the connection count.

How Netdata helps

Netdata collects memcached stats per second. For connection-limit incidents, per-second granularity matters because the transition from healthy to saturated can happen in seconds during a reconnection storm.

• accepting_conns as a real-time dimension: per-second collection shows the exact moment the listener disables, not a 5-minute-averaged blur.
• curr_connections vs max_connections on the same chart: the ratio is visible at a glance, and anomaly detection flags unusual connection growth patterns before the hard limit is reached.
• Correlation with total_connections rate: Netdata can show churn (high total_connections rate, stable curr_connections) alongside the ratio, making the leak-vs-chorn diagnostic immediate.
• Composite alerting: a connection-exhaustion alert can combine accepting_conns = 0, ratio thresholds, and sustained duration to filter transient spikes.
• Cross-layer correlation: when clients fall through to the backend, Netdata can correlate the memcached connection-refusal window with backend database load, CPU, and latency on the same timeline.

Related guides

• Memcached connection refused: telling a dead process from a hung or full one
• Memcached monitoring checklist: the signals every production cache needs
• How Memcached actually works in production: a mental model for operators
• Memcached eviction cascade: when a full cache overloads the backend
• Memcached memory utilization: bytes vs limit_maxbytes and why the global number lies