Apache Too many open files: file descriptor exhaustion (EMFILE)

Your error log starts showing lines like:

(24)Too many open files: AH00035: access to /some/path failed

or proxy requests begin failing with AH01114 connection errors, and users see intermittent 5xx responses. Sometimes only some requests fail. Sometimes the site looks fine for hours and then breaks at peak. The process is running, the port is open, and nothing in CPU or memory looks wrong.

This is file descriptor exhaustion: one or more Apache child processes have hit their per-process open-file limit and can no longer accept connections, open files, or reach backends. It is a cliff-edge failure, not a gradual one. Everything works until the limit, then everything that needs a new descriptor fails at once.

Two things make this incident confusing. First, the errors are sporadic because only the child at peak load hits the ceiling while the others are fine. Second, it often arrives before you ever touch MaxRequestWorkers, because someone sized the worker pool for production but nobody raised the file descriptor limit from its default.

What this means

Every Apache child process holds file descriptors for more than just client connections:

• Each accepted client connection costs one FD, including idle keepalive connections.
• Each open log file costs one FD per child. Every VHost with its own CustomLog and ErrorLog multiplies this across every child process.
• Each backend proxy socket (mod_proxy, mod_proxy_fcgi, mod_jk) costs one FD.
• Each pipe costs FDs: piped logging to rotatelogs, CGI processes, and similar.

The per-process limit comes from the operating system, not from Apache config. On systemd-based systems it is the unit’s LimitNOFILE; on older init systems it comes from ulimit and /etc/security/limits.conf. There is also a separate system-wide ceiling (fs.file-max). When a child hits its per-process limit, the kernel returns EMFILE, which Apache logs as (24)Too many open files. From that moment, that child cannot accept() new connections, cannot open static files, and cannot open new backend sockets. Requests that land on healthy children still work, which is why the failure looks intermittent.

flowchart TD
  C[Apache child process] --> A[Client connections: 1 FD each]
  C --> K[Keepalive idle conns: still hold FDs]
  C --> L[Log files: 1 FD per child per log]
  C --> B[Backend proxy sockets: 1 FD each]
  C --> P[Pipes: rotatelogs, CGI]
  A --> LIM[Per-process limit reached: EMFILE]
  K --> LIM
  L --> LIM
  B --> LIM
  P --> LIM
  LIM --> F1[accept fails: new clients refused]
  LIM --> F2[open fails: static files 500]
  LIM --> F3[connect fails: proxy AH01114, 502/503]

Common causes

Quick checks

All of these are read-only and safe to run during an incident.

# Confirm the error in the log (adjust path for your distro)
grep -c "Too many open files" /var/log/httpd/error_log
grep "Too many open files" /var/log/httpd/error_log | tail -20

# Per-process FD limit for the Apache parent
cat /proc/$(pgrep -o 'httpd|apache2')/limits 2>/dev/null | grep "Max open files"

# FD count per child process, with each child's limit
for pid in $(pgrep 'httpd|apache2'); do
  count=$(ls /proc/$pid/fd 2>/dev/null | wc -l)
  limit=$(awk '/Max open files/{print $4}' /proc/$pid/limits)
  echo "PID $pid: $count / $limit"
done

# System-wide FD picture: allocated, unused, maximum
cat /proc/sys/fs/file-nr

# What kinds of FDs is the busiest child holding?
pid=$(pgrep 'httpd|apache2' | head -1)
ls -l /proc/$pid/fd 2>/dev/null | awk '{print $NF}' | \
  sed 's/[0-9]*$//' | sort | uniq -c | sort -rn | head -20

# Connection states feeding FD usage
ss -tn '( sport = :80 or sport = :443 )' | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rn

# Current worker utilization, for context
curl -s 'http://localhost/server-status?auto' | grep -E "BusyWorkers|IdleWorkers"

Two things to look at first: whether any child’s FD count is near its limit, and what the FDs actually are. ls -l /proc/PID/fd tells you immediately whether you are drowning in sockets (traffic, keepalive, or proxy), log files (VHost sprawl), or something unexpected.

How to diagnose it

1. Confirm the signature. Grep the error log for Too many open files. Note the errno: (24) is the per-process limit (EMFILE), which is the common Apache case. If you see system-wide exhaustion instead, check fs.file-nr against fs.file-max before touching Apache.

2. Check the effective limit, not the config you think is applied. Read /proc/<pid>/limits for a running Apache process. This is the ground truth. On systemd systems, whatever you put in /etc/security/limits.conf does not apply to the service; only LimitNOFILE in the unit does. A stale value here explains most “but I already raised it” surprises.

3. Measure per-child usage over a traffic cycle. Sample the FD counts several times, including at peak. Exhaustion is often intermittent because only the peak child briefly touches the ceiling, then usage drops. A single point-in-time check during a quiet period can look completely healthy.

4. Classify the descriptors. On the busiest child, resolve what the FDs point to. Mostly sockets in ESTABLISHED or CLOSE_WAIT: a connection or proxy-side problem, and persistent CLOSE_WAIT specifically means the local side never closed a socket the peer already closed (a leak in the proxy path or an application handler). Mostly log files: VHost log sprawl. Mostly pipes: check piped logging and CGI.

5. Check the scoreboard for connection hoarding. On prefork or worker MPM, a high K (keepalive) count means idle connections are each holding both a worker and an FD. On event MPM, keepalive is handled by the listener thread and tracked via ConnsAsyncKeepAlive, which is normal even at high counts; significant K states in the scoreboard on event MPM are not normal and are worth investigating.

6. Rule out a leak versus a capacity problem. Plot FD count per PID over hours. A sawtooth that tracks traffic is a capacity problem (raise the limit, reduce consumers). A line that only goes up, surviving traffic troughs, is a leak (find the consumer; raising the limit only delays the incident).

Metrics and signals to monitor

Fixes

Raise the per-process limit (systemd)

Create an override for the unit rather than editing the packaged unit file:

# /etc/systemd/system/httpd.service.d/override.conf  (or apache2.service.d on Debian/Ubuntu)
[Service]
LimitNOFILE=65536

Then:

systemctl daemon-reload
systemctl restart httpd   # disruptive: drops all connections

This requires a full restart, not a graceful reload. systemctl reload and apachectl -k graceful send SIGUSR1 to the running process; existing children keep their old limits and new children forked from the running parent inherit them. The new limit only takes effect when systemd starts a fresh process. Plan the restart accordingly: drain the node from the load balancer first if you can.

Sizing rule from the playbook: the limit should be at least 2x your theoretical maximum FD usage per child. Estimate that as concurrent connections per child, plus backend proxy connections, plus one per log file, plus static overhead.

Consolidate VHost log files

If the FD inventory shows dozens or hundreds of log file descriptors per child, reduce the number of open logs instead of only raising the ceiling. Use a single access log with the VHost identifier in the log format (%v, or %V when UseCanonicalName Off is in play) and split per host later at analysis or rotation time. This removes the multiplicative cost of separate logs per VHost per child.

Fix proxy connection leaks

If FDs climb monotonically and the inventory shows accumulated backend sockets, raising the limit buys time but fixes nothing. Review the proxy configuration and backend behavior: connections should be returned to the pool and reused. Persistent CLOSE_WAIT toward the backend points at the local side not closing sockets; investigate the application or proxy path that holds them.

Reduce keepalive FD pressure

On prefork or worker MPM, a high KeepAliveTimeout lets idle connections hold FDs (and workers) for no benefit. Lowering it frees both. On event MPM this is largely a non-issue because the listener thread handles keepalive asynchronously. If you are on prefork or worker and keepalive hoarding is your pattern, moving to event MPM addresses the root cause, but treat an MPM change as a planned change with testing, not an incident fix.

System-wide limit

If fs.file-nr shows the system-wide ceiling is the binding constraint, raise fs.file-max via sysctl. This is rare compared to the per-process case, but check it before assuming the per-process limit is the only one that matters.

Prevention

• Set LimitNOFILE deliberately for any production Apache unit and verify it with /proc/PID/limits after the change, with a full restart. Do not assume the distribution default is adequate; on many systems it is far below what a proxied, multi-VHost deployment needs.
• Monitor FD usage as a first-class signal. Alert (ticket level) when any child exceeds 70% of its limit, so you hear about it during the growth phase rather than at the cliff.
• Alert on the error string. Too many open files in the error log with active traffic and confirmed request failures is a page per the playbook severity guidance. A single transient occurrence can be a ticket, but never silence it.
• Watch the trend, not just the threshold. Per-child FD count that grows across days independent of traffic is a leak. Catch it in review before it becomes an incident.
• Keep VHost log sprawl in check as you add hosts. Every new site with its own log pair raises per-child FD cost for the whole server.
• Include FD checks in capacity reviews. FD headroom, like worker and memory headroom, degrades silently as traffic and VHost count grow. The playbook’s headroom rule: limit at least 2x theoretical maximum usage.

How Netdata helps

• Netdata collects per-process open file descriptor counts continuously, so you see the slow climb toward the limit days before the first EMFILE, including which child process is leading.
• Error log monitoring surfaces Too many open files and related AH codes as they happen, correlated in time with the FD and connection charts.
• TCP connection state charts (ESTABLISHED, TIME_WAIT, CLOSE_WAIT) let you distinguish a traffic-driven capacity problem from a CLOSE_WAIT-style leak at a glance.
• Apache worker utilization from mod_status (BusyWorkers, IdleWorkers, scoreboard states) sits next to FD usage, so you can confirm the signature “workers fine, descriptors exhausted” that distinguishes this from worker exhaustion.
• Because collection is per-second, the intermittent pattern, where only the peak child briefly hits the ceiling, is visible instead of being missed by a point-in-time check.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• How Apache HTTPD actually works in production: a mental model for operators
• Apache 5xx error rate: 500 vs 502 vs 503 vs 504 and what each one means
• Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion
• Apache 502 Bad Gateway: a backend that returned an invalid response
• Apache CLOSE_WAIT and TIME_WAIT: connection leaks versus normal churn
• Apache keepalive consuming workers: KeepAliveTimeout, the K state, and MPM choice
• Apache BusyWorkers and IdleWorkers: reading worker utilization from mod_status
• Apache error log monitoring: severity levels, AH codes, and what to alert on
• Apache 504 Gateway Timeout: slow backends, ProxyTimeout, and worker pile-up
• Apache backend response time: telling ‘Apache is slow’ from ’the backend is slow’
• Apache balancer member in error state: reading balancer-manager and failover
• Apache listen queue overflow: Recv-Q growth, ListenBacklog, and refused connections