Apache open-file limits: raising ulimit and systemd LimitNOFILE correctly

The default per-process open-file limit on most Linux distributions is 1024, far too low for production Apache. Every client connection, every backend proxy connection, every log file, and every pipe consumes a file descriptor in every child process, and 1024 runs out long before MaxRequestWorkers does. When a child hits the limit, the failure is a cliff edge: “Too many open files” in the error log, failed accepts, failed backend connections, and intermittent 5xx responses, often in only some children at first, which makes the symptom look random.

The confusion comes from the limit not living in one place. Operators edit /etc/security/limits.conf, run ulimit -n in a shell, or tweak Apache configuration, and none of it sticks because the running httpd is managed by systemd and systemd sets the limit itself. This article covers where the limit actually comes from, how to raise it, why it requires a full restart, how to size it, and the adjacent limits that still bite after LimitNOFILE is fixed.

Where the limit actually comes from

For a service-managed httpd, the limit is set by systemd at process start, not by login configuration and not by Apache itself.

• systemd unit LimitNOFILE=: The authoritative source for httpd started by systemd. Whatever is in the unit file or in drop-in overrides under /etc/systemd/system/httpd.service.d/ (RHEL family) or /etc/systemd/system/apache2.service.d/ (Debian family) wins.
• /etc/security/limits.conf: Applied by PAM at login time for interactive sessions. It does not apply to services started by systemd. Editing it and restarting httpd changes nothing, which is the most common dead end in this investigation.
• ulimit -n in a shell or init script: Only affects processes started from that shell. On SysV-era systems this was the mechanism (setting ulimit in /etc/sysconfig/httpd or the init script). On systemd systems it is irrelevant for the service.
• Apache configuration: There is no Apache directive that raises the FD limit. Apache inherits whatever the process was started with.

flowchart TD
  A[httpd start requested] --> B{Started by systemd?}
  B -- yes --> C[LimitNOFILE from unit file]
  C --> D[Drop-in overrides in .d directory applied last]
  D --> E[Resulting limit visible in /proc/PID/limits]
  B -- no: shell or init script --> F[ulimit -n of the starting shell]
  G[/etc/security/limits.conf] -. ignored for systemd services .-> C

The definitive check is always against the running process, not against any config file:

# Check the effective FD limit of the running Apache parent
cat /proc/$(pgrep -o 'httpd|apache2')/limits | grep "Max open files"

If the value there does not match what you configured, you configured it in the wrong place.

Why raising the limit needs a full restart

systemctl reload httpd and apachectl graceful send SIGUSR1. The existing parent process stays alive and spawns new children under the same inherited limits. systemd applies LimitNOFILE when it starts a process, and a graceful reload never goes back through systemd’s process creation path, so the new value is never picked up.

The only way to apply a changed LimitNOFILE is a full stop and start of the unit. That is disruptive: it drops all in-flight connections. Plan it for a low-traffic window or drain the node from the load balancer first.

# Apply a new FD limit (DISRUPTIVE: drops all connections)
systemctl daemon-reload
systemctl restart httpd    # or apache2 on Debian/Ubuntu

Do not let log rotation or config management trigger hard restarts casually; sending the parent SIGHUP is also a hard restart and drops all connections.

Sizing the limit

The limit is per child process, and each child holds FDs for everything it touches. A sizing formula that covers production reality:

Required FDs per child = (concurrent connections per child x 2 if proxying) + one FD per distinct log file + internal overhead

• Client connections: one FD each. On worker/event MPM, concurrent connections per child tracks ThreadsPerChild plus async keepalive connections on event.
• Proxying: each backend connection is another FD. In reverse-proxy deployments, count client connection plus backend connection per in-flight request, hence the x2.
• Log files: every distinct CustomLog and ErrorLog target is held open by every child. With many VirtualHosts each writing separate access and error logs, this term alone can exceed 1024 before a single client connects. Hundreds of vhosts times two log files each means hundreds of FDs per child as a floor.
• Overhead: pipes, shared memory, SSL session cache, module internals. Leave room.

Then apply headroom: set the limit to at least 2x the theoretical maximum FD usage. FD exhaustion is a cliff-edge failure with no graceful degradation, and you want headroom for events like log rotation briefly opening new files. For most production deployments LimitNOFILE=65536 is a reasonable floor; vhost-heavy or high-concurrency proxy deployments may justify more.

Also check the system-wide ceiling: /proc/sys/fs/file-nr shows allocated, free, and maximum. fs.file-max must accommodate all httpd children plus everything else on the box. Per-process limits are the usual binding constraint, but on shared hosts the system limit can be hit first.

If the math surprises you, the fix is often not a bigger number. Consolidating vhost logs into a single log with the %v format directive (split later with split-logfile) collapses the per-child log FD count dramatically and is usually the right structural fix.

Procedure

This assumes systemd, which covers RHEL 7+ and derivatives, Debian 8+, and Ubuntu 16.04+. Adjust the service name: httpd on RHEL family, apache2 on Debian family.

1. Measure first. Record current FD usage per child so you have a baseline and can validate your sizing:

# FD count and limit per Apache process
for pid in $(pgrep 'httpd|apache2'); do
  count=$(ls /proc/$pid/fd 2>/dev/null | wc -l)
  limit=$(awk '/Max open files/{print $4}' /proc/$pid/limits)
  echo "PID $pid: $count / $limit"
done

2. Create a drop-in override. Do not edit the vendor unit file; package updates will overwrite it.

# Create the drop-in directory and file
mkdir -p /etc/systemd/system/httpd.service.d
cat > /etc/systemd/system/httpd.service.d/limits.conf <<'EOF'
[Service]
LimitNOFILE=65536
EOF

LimitNOFILE also accepts soft:hard syntax (e.g., LimitNOFILE=32768:65536) if you want a lower soft limit with a hard ceiling. A single value sets both.

3. Reload systemd and fully restart the service. Disruptive: drains connections.

systemctl daemon-reload
systemctl restart httpd

4. Verify against the running process (see next section). Do not skip this; a typo in the drop-in silently leaves you at the old limit.

Verifying it worked

# Confirm the running parent picked up the new limit
cat /proc/$(pgrep -o 'httpd|apache2')/limits | grep "Max open files"

# Spot-check children too; children inherit the parent's limits
for pid in $(pgrep 'httpd|apache2'); do
  awk -v p=$pid '/Max open files/{print p, $4, $5}' /proc/$pid/limits
done

You should see the new value in the “Max open files” row for parent and children. If it still shows 1024 or 4096, the drop-in was not read: check the path, the service name, that you ran daemon-reload, and that you did a full restart rather than reload.

Common pitfalls

• Editing /etc/security/limits.conf and expecting it to apply. It does not, for systemd services. This is the single most common wasted hour in this task.
• Running systemctl reload and assuming the new limit is live. The running parent keeps the old limit; only a full restart applies it. Worse, cat /proc/.../limits is the only honest check, because systemctl show httpd -p LimitNOFILE reports what systemd would apply at next start, not what the running process has.
• TasksMax capping you from the other direction. systemd also imposes a process-and-thread limit via the cgroup pids controller. If TasksMax is lower than what MaxRequestWorkers implies (processes plus threads), Apache cannot spawn the children you configured, and the kernel logs fork rejections from the pids controller. This looks nothing like an FD problem but shows up in the same investigation. Raise TasksMax= in the same drop-in if your worker math requires it. The default changed across systemd releases: 512 for services on older systemd (the v228-era DefaultTasksMax), then 15% of the system’s PIDs limit on newer releases (systemd 238 and later; check the effective value with systemctl show httpd -p TasksMax).
• Distro wrapper scripts that set ulimit themselves. Debian’s apache2ctl honors an APACHE_ULIMIT_MAX_FILES environment variable and runs ulimit -n before starting Apache. It is still present in current Debian/Ubuntu packaging (the default is ulimit -n 8192), and Debian’s apache2.service actually starts Apache through apachectl, so on Debian both mechanisms genuinely interact: a ulimit -n lower than the hard limit lowers it, and one higher fails with a warning. Pick one mechanism, preferably the systemd drop-in, and remove the other.
• Raising the limit to mask an FD leak. If per-child FD count grows monotonically over time, raising LimitNOFILE just delays the cliff. Common leak sources: backend connections in CLOSE_WAIT that never get reaped (often a backend that does not close connections properly), proxy pool connections not returned, or old log handles held across rotation. Track per-child FD counts over days before and after the change.
• select() and FDs above 1023. The systemd documentation warns that select(2) cannot handle file descriptors above 1023 on Linux. Apache’s worker and event MPMs use poll/epoll internally and are not affected, but a third-party module or CGI that calls select() can misbehave once a child holds more than 1023 FDs. If you run exotic modules, this is worth knowing; it is not a reason to keep the limit at 1024.
• mod_proxy CLOSE_WAIT accumulation. If backends keep sockets half-closed, FDs accumulate in CLOSE_WAIT inside Apache children. Raising the limit buys time; the real fix is proxy timeout tuning or fixing the backend’s connection handling.

Signals to monitor

How Netdata helps

• Netdata charts per-process file descriptor counts against their limits, so you see each Apache child’s FD usage trending toward the ceiling long before “Too many open files” appears.
• Correlating FD growth with connection-state charts (especially CLOSE_WAIT) separates a leak from legitimate load growth, which decides whether you raise the limit or fix the backend.
• Error log pattern monitoring catches “Too many open files” and AH01114 as they happen, instead of after users report intermittent 5xx.
• Restart and uptime context on the same dashboard makes it obvious whether a limit change actually took effect (new limit visible after a full restart) or was silently not applied (graceful reload only).
• System-wide fs.file-nr alongside per-process counts shows when the box-level ceiling, not the per-process limit, is the binding constraint.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• Apache 502 Bad Gateway: a backend that returned an invalid response
• Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion
• Apache 504 Gateway Timeout: slow backends, ProxyTimeout, and worker pile-up
• Apache 5xx error rate: 500 vs 502 vs 503 vs 504 and what each one means
• Apache backend response time: telling ‘Apache is slow’ from ’the backend is slow’
• Apache balancer member in error state: reading balancer-manager and failover
• Apache BusyWorkers and IdleWorkers: reading worker utilization from mod_status
• Apache CLOSE_WAIT and TIME_WAIT: connection leaks versus normal churn
• Apache error log monitoring: severity levels, AH codes, and what to alert on
• How Apache HTTPD actually works in production: a mental model for operators
• Apache keepalive consuming workers: KeepAliveTimeout, the K state, and MPM choice
• Apache listen queue overflow: Recv-Q growth, ListenBacklog, and refused connections