Apache capped by systemd: TasksMax, LimitNOFILE, and MemoryMax override your config

You raised MaxRequestWorkers, restarted Apache, and it still refuses to spawn more workers. The error log shows no AH00484, the host has free CPU and RAM, and connections queue and time out. Or Apache was OOM-killed even though free showed gigabytes available. Or you hit “Too many open files” at a fraction of the limit you set in /etc/security/limits.conf.

When httpd runs under systemd, the unit file is a second, invisible configuration layer. TasksMax, LimitNOFILE, and MemoryMax are enforced by the kernel (cgroups and rlimits), not by Apache, and they silently override whatever you put in httpd.conf. Apache does not know these limits exist. It just fails to fork, fails to open a file, or gets killed, and the errors it logs point at the wrong thing.

This article covers how the three limits work, how to read the effective values for a running httpd, and how to reconcile them with MaxRequestWorkers, ServerLimit, and ThreadsPerChild.

What this means

Every httpd process started by systemd inherits limits from the unit, and the whole unit lives inside a cgroup. There are two enforcement mechanisms, and confusing them is half the debugging pain:

• rlimits (LimitNOFILE=, and Apache’s own RLimitMEM/RLimitCPU) are per-process. Each child gets its own copy.
• cgroup limits (TasksMax=, MemoryMax=) apply to the unit as a whole. They count everything in the cgroup: the root parent, all children, all threads.

The consequence: you can tune Apache perfectly and still be capped by a unit file you never edited, often from distro packaging defaults. Worse, systemctl set-property writes overrides to /etc/systemd/system/httpd.service.d/, which do not appear in the original unit file, so a previous operator’s fix can itself become the invisible ceiling.

flowchart TD
  U[unit file and drop-ins] --> C[cgroup: TasksMax, MemoryMax - whole unit]
  U --> R[rlimits: LimitNOFILE - per process]
  C --> H[httpd parent and children]
  R --> H
  H --> W[worker processes and threads]
  HC[httpd.conf: MaxRequestWorkers, ServerLimit, ThreadsPerChild] --> W
  W --> S{symptom}
  C -- "hit first" --> S
  HC -- "hit first" --> S

Apache only logs cleanly when it hits its own limit (AH00484: server reached MaxRequestWorkers setting). When systemd’s limit is the lower one, Apache hits an OS error it does not understand and reports something vague, or nothing.

The three limits

TasksMax: the sneaky one

TasksMax maps to the cgroup pids controller (pids.max). It counts tasks, and in Linux every thread is a task. On worker and event MPM, a single httpd child running ThreadsPerChild=64 consumes 64 tasks plus listener threads. Your task budget is roughly:

worst-case tasks = ServerLimit x ThreadsPerChild + listener threads + parent + helpers

When the cgroup hits TasksMax, fork() and pthread_create() return EAGAIN. Apache logs something like:

(11)Resource temporarily unavailable: AH03104: apr_thread_create: unable to create worker thread

The result looks exactly like worker exhaustion: BusyWorkers climbs, idle capacity never appears, connections pile up in the listen backlog. But the scoreboard shows open slots (.) that Apache cannot fill, because the scoreboard has room while the cgroup does not. That mismatch is the tell.

Version history matters because defaults changed dramatically:

• systemd 228 (2015) introduced TasksMax. Earlier versions had no pids limit at all.
• systemd 228 to 241 defaulted DefaultTasksMax to 512, which capped many services far below their configured capacity.
• systemd 242+ defaults to 15% of the smaller of kernel.pid_max and kernel.threads-max. On a typical host that is around 4915, but on a VM with a low threads-max it can be a few hundred. Check the actual value; do not assume.

LimitNOFILE: FD exhaustion below your limits.conf

LimitNOFILE sets RLIMIT_NOFILE (the per-process open file limit) for every process in the unit. Two gotchas:

1. /etc/security/limits.conf does not apply to systemd services. PAM limits only apply to login sessions. Editing limits.conf and wondering why nothing changed is a classic dead end.
2. The apachectl wrapper script on some distributions contains ULIMIT_MAX_FILES logic that calls ulimit itself, overriding what systemd set. If your drop-in does not take effect, inspect the script (commonly /usr/sbin/apachectl) for that variable.

Each client connection, backend proxy connection, log file, and piped log process costs an FD per child. A practical sizing rule: MaxRequestWorkers x 2 (client plus backend) + log files + about 50 of static overhead, then set the limit to at least 2x that. VHost-heavy deployments with separate log files per vhost multiply FD usage in every child.

Symptoms are “Too many open files” (EMFILE) in the error log, intermittent 5xx, failed accepts, and proxy connection failures (AH01114). Failure is cliff-edge: one child hits the limit and starts failing while others look fine.

MemoryMax: cgroup OOM with free RAM everywhere

MemoryMax sets a hard cgroup memory ceiling for the whole unit. When the cgroup exceeds it, the kernel OOM killer runs inside that cgroup and kills httpd processes, even if the host has tens of gigabytes free. From the outside it looks like Apache crashed for no reason; dmesg shows an oom-kill with the cgroup path, not a system-wide OOM.

This interacts badly with Apache capacity math. The usual rule is that MaxRequestWorkers x max observed child RSS must stay under about 70% of system RAM. If someone also set MemoryMax=2G on the unit, your real ceiling is the lower of the two, and Apache has no idea. MemoryHigh is the throttling counterpart: the kernel starts reclaiming aggressively before the hard limit, which shows up as latency growth before any kill. Apache’s own RLimitMEM is a per-process rlimit, independent of the cgroup limit; both can apply, and the cgroup one is usually the one that bites.

Common causes

Quick checks

All read-only. Substitute apache2 for httpd on Debian/Ubuntu.

# 1. Effective unit limits, as systemd sees them
systemctl show httpd.service -p TasksMax,LimitNOFILE,MemoryMax,MemoryHigh

# 2. rlimits of a running child (the values actually enforced per process)
cat /proc/$(pgrep -o httpd)/limits | grep -E "Max open files|Max processes"

# 3. cgroup pids usage vs limit (cgroup v2 path; on v1 use /sys/fs/cgroup/pids/...)
cat /sys/fs/cgroup/system.slice/httpd.service/pids.current \
    /sys/fs/cgroup/system.slice/httpd.service/pids.max

# 4. cgroup memory usage vs limit (v2 path)
cat /sys/fs/cgroup/system.slice/httpd.service/memory.current \
    /sys/fs/cgroup/system.slice/httpd.service/memory.max

# 5. Drop-ins and set-property overrides not visible in the unit file
systemctl cat httpd.service

# 6. Worker/task footprint right now
ps -eLf | grep -c '[h]ttpd'

# 7. Evidence of the limits firing
grep -E "AH03104|Resource temporarily unavailable|Too many open files" /var/log/httpd/error_log | tail
dmesg -T | grep -i -E "oom|killed process" | tail
journalctl -u httpd.service --since today | grep -i -E "oom|limit" | tail

# 8. Apache's own configured ceiling, for comparison
apachectl -t -D DUMP_RUN_CFG 2>/dev/null; grep -rE "MaxRequestWorkers|ServerLimit|ThreadsPerChild" /etc/httpd/ 2>/dev/null

How to diagnose it

1. Confirm which ceiling you are hitting. Get BusyWorkers and the scoreboard from mod_status (curl -s http://localhost/server-status?auto). If BusyWorkers is stuck well below MaxRequestWorkers with open slots (.) that never fill, you are capped by something below Apache, not by Apache config.
2. Check for AH00484. Its presence means Apache hit its own MaxRequestWorkers. Its absence while workers refuse to grow points at systemd.
3. Read the effective TasksMax (check 1) and compute Apache’s worst case: ServerLimit x ThreadsPerChild, plus margin for listener threads and the parent. If TasksMax is lower, that is your cap.
4. Compare pids.current to pids.max (check 3) during the incident window. If current is pinned at max, the cgroup is actively blocking forks.
5. If the symptom is EMFILE, compare each child’s FD count (ls /proc/<pid>/fd | wc -l) against the “Max open files” line in /proc/<pid>/limits. Apache fails in children first, so check children, not just the parent.
6. If the symptom is sudden death with free RAM, look for a cgroup-scoped oom-kill in dmesg and check memory.max for the unit. A system-wide OOM lists the whole host’s memory state; a cgroup OOM names the unit’s cgroup.
7. Reconcile before changing anything. systemctl cat to see the full drop-in stack, and grep apachectl for overrides. Fix the value at the layer that actually enforces it.

Metrics and signals to monitor

Fixes

All fixes are unit-file changes. Use a drop-in, not an edit of the packaged unit file:

sudo systemctl edit httpd.service

[Service]
TasksMax=infinity
LimitNOFILE=65536
# Remove or raise only if you understand why it was set:
# MemoryMax=

Then systemctl daemon-reload and restart. rlimit changes require a full restart, not a graceful reload. Cgroup property changes via systemctl set-property apply immediately and persist (they write their own drop-in), but existing processes may need a restart to pick up new rlimits.

Per limit:

• TasksMax: size it above ServerLimit x ThreadsPerChild with margin, or set infinity and let MaxRequestWorkers be the single ceiling. Tradeoff: removing the pids cap removes a fork-bomb guardrail for that unit. Prefer a deliberate number over infinity on shared hosts.
• LimitNOFILE: set to at least 2x your computed FD worst case. Also fix or remove the ULIMIT_MAX_FILES logic in apachectl if present, or it will silently override you.
• MemoryMax: either remove it and rely on Apache capacity math (MaxRequestWorkers x child RSS < 70% RAM), or set it deliberately above Apache’s worst case with headroom for the parent and page-fault spikes. Do not leave a stale low value fighting your worker math.

Do not “fix” any of these by lowering MaxRequestWorkers to match a systemd cap you have not investigated. That masks the real ceiling and leaves the mismatch in place.

Prevention

• One source of truth per limit. Decide whether Apache config or the unit file owns each ceiling, and set the other one to not interfere. Document the choice in the drop-in.
• Validate after every change. After editing either layer, run systemctl show and /proc/<pid>/limits checks and confirm effective values match intent. Add this to your config-management runbooks.
• Alert on approach, not on failure. Track pids.current, FD usage per child, and cgroup memory.current as percentages of their limits. These are cliff-edge resources; you get no gradual warning from Apache itself.
• Watch set-property drift. systemctl cat httpd.service in reviews catches overrides that do not appear in the packaged unit file.
• Size VMs consistently. The 15%-of-threads-max default means identical Apache configs have different caps on different hosts. Pin TasksMax explicitly on any host running production httpd.

How Netdata helps

• Cgroup and per-application charts show the httpd unit’s task count, memory usage, and FD usage against their limits, so a systemd cap becoming the binding constraint is visible before workers stall.
• The Apache collector charts BusyWorkers, IdleWorkers, and scoreboard states from mod_status at per-second resolution, making the “workers stuck below MaxRequestWorkers with open slots” pattern obvious.
• Correlating the two layers on one dashboard is what shortens diagnosis: worker count plateauing at exactly the TasksMax value, while host CPU and RAM sit idle, is a signature you can spot in seconds.
• Web log and error log monitoring surfaces AH03104, EMFILE, and AH00484 as counted events, so you can tell “Apache hit its own limit” from “the OS refused to fork” without grepping at 3 a.m.
• Cgroup memory charts separate unit-level memory pressure from host-level free RAM, which is the difference between a MemoryMax OOM and a system OOM.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion
• Apache BusyWorkers and IdleWorkers: reading worker utilization from mod_status
• Apache error log monitoring: severity levels, AH codes, and what to alert on
• Apache 5xx error rate: 500 vs 502 vs 503 vs 504 and what each one means
• Apache 504 Gateway Timeout: slow backends, ProxyTimeout, and worker pile-up
• How Apache HTTPD actually works in production: a mental model for operators