Memcached maxconns and ulimit: why the default 1024 is too low for production

Memcached is up, the port is open, existing connections work fine. But new clients get connection refused. Their requests fall through to the backend, backend load climbs, and the cache looks healthy from the outside: CPU is low, memory is nowhere near the limit, and the slab allocator is not evicting anything.

The most common cause is the default -c 1024 (maxconns) limit. It looks generous for a single application instance, but becomes a hard cliff-edge once you scale from 10 to 50 instances, each with its own connection pool. Monitoring agents, health checks, and load balancer probes also consume connections against this budget.

Raising -c looks simple. Two gotchas complicate it. First, the OS ulimit -n is a second ceiling that silently overrides -c. Second, each connection carries buffer memory outside the -m cache budget, and at high counts this can push the process toward OOM and trigger response_obj_oom.

What this means

When memcached reaches its -c limit, the main listener thread temporarily disables the accept socket. Three things happen immediately:

• accepting_conns flips to 0 (the real-time state bit)
• listen_disabled_num increments (count of listener-disable transitions)
• rejected_connections increments (count of connections turned away)

The process stays alive and keeps serving existing connections. Worker threads, the slab allocator, and the LRU crawler continue running. But no new clients can join until a slot frees up.

This is a binary cliff. One connection below the limit, everything works. At the limit, new connections are instantly refused. There is no graceful degradation, no queueing, no backpressure signal to clients. With -o maxconns_fast (the option exists since 1.4.8; on by default since 1.5.0), memcached instead accepts the connection, writes an error to the client, and immediately closes it. The outcome for the client is the same.

There are two ceilings, not one:

If ulimit -n is lower than -c, the OS limit wins. Memcached cannot open more file descriptors than the kernel allows, regardless of what -c says. In some configurations, memcached may fail to start entirely if -c exceeds the process’s rlimit.

curr_connections also includes 3-5 internal connections (listener socket, management pipe, internal bookkeeping). These count against the -c budget. When sizing, subtract them from the usable slots.

Each client connection allocates buffers outside the -m slab budget. The commonly cited figure is roughly 10KB per connection. At 10,000 connections, that is about 100MB of non-cache memory. At very high counts, this can trigger response_obj_oom (since 1.6.0), where the server closes connections because it cannot allocate response buffers. Increasing -c without sufficient memory headroom trades one problem for another.

flowchart TD
    A["New connections refused"] --> B["accepting_conns = 0"]
    B --> C{"curr_connections near
max_connections?"}
    C -->|Yes| D["-c limit is the ceiling"]
    C -->|No| E["OS ulimit -n is lower
than -c"]
    D --> F["Raise -c, verify
ulimit -n matches"]
    E --> F
    F --> G["Check response_obj_oom
for memory headroom"]

Common causes

Quick checks

All of these are read-only. None modify state.

# Check current connections and configured limit
echo "stats" | nc -w 1 localhost 11211 | grep -E "STAT (curr_connections|max_connections)"

# Check if the server is currently accepting connections
echo "stats" | nc -w 1 localhost 11211 | grep "STAT accepting_conns"

# Check if connections have been rejected or listener disabled (cumulative)
echo "stats" | nc -w 1 localhost 11211 | grep -E "STAT (rejected_connections|listen_disabled_num)"

# Check total connections ever opened (detects churn; this counter only goes up)
echo "stats" | nc -w 1 localhost 11211 | grep "STAT total_connections"

# Check time spent with listener disabled (cumulative microseconds)
echo "stats" | nc -w 1 localhost 11211 | grep "STAT time_in_listen_disabled_us"

# Check OS file descriptor limit for the process
cat /proc/$(pgrep -x memcached | head -1)/limits | grep "Max open files"

# Check response buffer OOM (since 1.6.0)
echo "stats" | nc -w 1 localhost 11211 | grep "STAT response_obj_oom"

# See which clients hold the most connections
ss -tn | grep ":11211" | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head -20

How to diagnose it

1. Confirm the server is at the connection limit. Check accepting_conns. If it is 0, the listener is disabled and connections are being rejected. If it is 1 but clients report connection failures, the problem is elsewhere: process health, network, or firewall.

2. Determine which ceiling is active. Compare curr_connections to max_connections. If curr_connections is near max_connections, the -c limit is the ceiling. If curr_connections is well below max_connections but connections are still failing, the OS ulimit -n is likely the real ceiling.

3. Check the OS limit. Read /proc/<pid>/limits for the memcached process. Look at “Max open files”. If this value is lower than -c, the kernel is enforcing a tighter limit than memcached’s configuration.

4. Distinguish a leak from churn from undersizing.

   • Leak: curr_connections increases monotonically and never drops. total_connections rate is moderate but steady.
   • Churn: curr_connections is moderate, but total_connections rate is very high. Clients open and close connections rapidly without pooling. Closed sockets may linger in TIME_WAIT for roughly 60 seconds, consuming fd slots.
   • Undersizing: curr_connections is stable near the limit. The configured -c is simply too low for the number of clients.

5. Check for connection buffer memory pressure. If response_obj_oom is incrementing, the process is closing connections because it cannot allocate response buffers. The connection count is straining total process memory, not just the -c counter. Raising -c without adding memory makes this worse.

6. Identify the source. Use ss -tn | grep 11211 to see which IPs hold the most connections. A single client with hundreds of connections may have a misconfigured pool or a leak.

Metrics and signals to monitor

Fixes

Raise -c to match your deployment

Size -c to at least 2x peak concurrent connections. Account for:

• All application instances times their connection pool size
• Monitoring and health-check connections
• The 3-5 internal connections memcached uses for itself
• Burst headroom for reconnection storms after deploys

For example: 50 app instances with 10 connections each = 500 client connections. Add 10 monitoring connections and 5 internal = 515. At 2x headroom, set -c to at least 1030. Round up generously.

Changing -c requires a restart. Memcached does not support changing the connection limit at runtime.

Before restarting, verify the OS ulimit -n is at least as high as the new -c. Otherwise you restart, the OS limit still applies, and nothing changes.

Fix the OS file descriptor limit

systemd (RHEL, CentOS, Ubuntu, Debian):

# Create a systemd override (non-destructive, does not edit the original unit file)
systemctl edit memcached.service

Add to the override:

[Service]
LimitNOFILE=65536

Then reload and restart. This drops all existing connections and clears the in-memory cache:

systemctl daemon-reload
systemctl restart memcached

If memcached fails to start after raising -c, with an error about failing to set rlimit for open files, the systemd unit file may be restricting capabilities. Unit files older than the 1.4.34-era upstream fix lacked CAP_SYS_RESOURCE in CapabilityBoundingSet, which prevented the process from raising its own rlimit above 1024; current units shipped by Fedora/RHEL, Debian, and Ubuntu include it, but custom or older unit files may not. The workaround is to rely on LimitNOFILE in the systemd override, which systemd enforces before the process starts, bypassing the capability issue.

Docker:

The official memcached Docker image runs the process as PID 1. Even if the container has a high ulimit -n via --ulimit, memcached defaults to -c 1024 unless explicitly overridden. You need to set both:

# Raise both the container fd limit and memcached's -c
docker run --ulimit nofile=65536:65536 memcached -c 4096

The --ulimit flag raises the OS ceiling. The -c 4096 tells memcached it can actually use that many connections. Setting only one is a common mistake.

Fix connection leaks and missing pooling

If the diagnosis reveals a leak or churn rather than genuine undersizing, raising -c only delays the problem. Root causes:

• Missing connection pooling: every request opens a new TCP connection. Fix on the client side by enabling persistent connections or a connection pool in the memcached client library.
• Connection leak: the client library opens connections but never closes them. curr_connections increases monotonically. Fix the client.
• TIME_WAIT accumulation: non-persistent connections leave sockets in TIME_WAIT for roughly 60 seconds. At high request rates these accumulate and consume fd slots. Connection pooling eliminates this.

Account for connection buffer memory

Each connection costs roughly 10KB of buffer memory outside the -m budget. Before raising -c to a large value:

• Estimate additional non-cache memory: new_maxconns * 10KB
• Compare against available system memory minus the -m allocation
• Monitor response_obj_oom after the change

If response_obj_oom is already incrementing, the process does not have enough memory for its current connection count. Raising -c further without adding memory makes this worse.

Prevention

• Monitor curr_connections / max_connections with a warning at 0.80 and critical at 0.95. This gives advance warning before the cliff.
• Alert on accepting_conns = 0 combined with curr_connections / max_connections > 0.98, sustained for 5 minutes. This is the composite connection exhaustion pattern that warrants paging.
• Alert on any increment of listen_disabled_num or rejected_connections in production. Even a single occurrence means clients were denied service.
• Verify ulimit -n after every deploy or host replacement. Orchestration and container platforms may reset the limit to defaults.
• Size -c proactively when scaling application instances. Recalculate: instances * pool_size + monitoring + headroom.
• Track response_obj_oom when running high connection counts. It signals that buffer memory, not the -c counter, is the binding constraint.

How Netdata helps

• Per-second curr_connections and max_connections collection catches spikes that 60-second polling intervals miss entirely.
• accepting_conns as a live state bit identifies the exact moment the listener disables, distinguishing connection exhaustion from a process crash or network issue.
• Correlating listen_disabled_num, rejected_connections, and time_in_listen_disabled_us distinguishes brief deploy-time spikes from sustained exhaustion that warrants paging.
• total_connections rate alongside curr_connections separates connection churn from missing pooling versus genuine undersizing.
• Anomaly detection on curr_connections flags unusual growth from connection leaks or deployment spikes before the hard limit is reached.

Related guides

• Memcached connection refused: telling a dead process from a hung or full one
• Memcached monitoring checklist: the signals every production cache needs
• How Memcached actually works in production: a mental model for operators
• Memcached eviction cascade: when a full cache overloads the backend
• Memcached memory utilization: bytes vs limit_maxbytes and why the global number lies
• Memcached evictions climbing: the cache is full and discarding live data
• Memcached high miss rate: separating cold start, new key patterns, and memory pressure
• Memcached hit ratio dropping: reading get_hits, get_misses, and cache effectiveness
• Memcached evicted_unfetched and expired_unfetched: caching data nobody ever reads
• Memcached cas_badval climbing: check-and-set contention and lost updates
• Memcached evicted_time low: distinguishing healthy turnover from cache thrash
• Memcached incr/decr misses: evicted counters that silently break rate limiters and locks