Apache No space left on device: a full log disk that stops the server serving

The error string operators usually search for is (28)No space left on device in the Apache error log. Sometimes it appears at startup and Apache refuses to run. The nastier version appears at runtime: the site goes dark, but the process is alive, the port is open, and TCP connections are accepted. Health checks that only test the socket say everything is fine.

What is actually happening in the runtime case is the log stall deadlock. Apache workers finish their requests and then block trying to write the access or error log entry. A worker blocked on a log write sits in the L (Logging) state on the scoreboard. It cannot take the next connection. As more workers finish requests and stall, the scoreboard fills with L states until no worker is available. The server is up. It serves nothing.

With piped logging (rotatelogs or similar), there is a second failure mode on top of the first: when the logger cannot write downstream, its pipe can stop draining. Apache workers then block on log writes even though the logger process remains alive.

What this means

Apache writes to its logs synchronously by default. When a worker finishes a request, it writes the access log line before it can pick up new work. If the filesystem hosting the log is full, the write blocks or fails, and the worker waits. The scoreboard state L is the visible symptom of that wait.

The distinguishing signature is the combination of three things:

1. The scoreboard dominated by L states, sustained.
2. df showing the log filesystem at or near 100%.
3. Throughput collapsed while the process and listener look healthy.

This is why the incident is confusing from the outside. TCP connects succeed because the kernel still accepts into the listen backlog. Load balancer health checks that probe a real URL start timing out. Existing connections hang. New connections queue, then get refused as the backlog fills.

flowchart TD
  A[Log filesystem fills] --> B[Log writes block]
  B --> C[Workers stall in L state]
  C --> D[IdleWorkers falls to 0]
  D --> E[New connections queue in listen backlog]
  E --> F[Backlog full: connections refused]
  A -.piped logging.-> G[rotatelogs dies on write failure]
  G --> H[SIGPIPE kills Apache children]
  H --> I[Child churn adds to the stall]

One caveat before you assume the disk: the identical error string can appear at startup for a completely different reason. On Linux, SysV semaphore exhaustion (orphaned semaphore arrays left behind after unclean Apache shutdowns) also produces No space left on device even when every filesystem has free space. That failure looks different: Apache refuses to start, and the error log shows messages like couldn't create the accept lock or mutex creation failures rather than log write failures. If Apache fails at startup and df shows free space, check ipcs -s for orphaned semaphore arrays owned by the Apache user. The rest of this article is about the runtime, disk-full case.

Common causes

Quick checks

Read-only and fast. Run them in this order.

# 1. Confirm the log filesystem is full
df -h /var/log/apache2/ 2>/dev/null || df -h /var/log/httpd/

# 2. Check inodes too; 100% inode use gives the same error with free blocks
df -i /var/log/apache2/ 2>/dev/null || df -i /var/log/httpd/

# 3. See the scoreboard state distribution (expect many L states)
curl -s http://localhost/server-status?auto | grep Scoreboard | \
  sed 's/Scoreboard: //' | fold -w1 | sort | uniq -c | sort -rn

# 4. Busy/idle worker summary
curl -s http://localhost/server-status?auto | grep -E "BusyWorkers|IdleWorkers"

# 5. Find the largest files on the log filesystem
ls -lhS /var/log/httpd/ 2>/dev/null | head
ls -lhS /var/log/apache2/ 2>/dev/null | head

# 6. Check for deleted files Apache still holds open (space not reclaimed)
lsof +L1 2>/dev/null | grep -E "httpd|apache2"

# 7. Check whether the piped log process is alive
pgrep -af rotatelogs

# 8. Confirm the error string in the error log
grep -i "No space left on device" /var/log/httpd/error_log /var/log/apache2/error.log 2>/dev/null | tail

# 9. Look for the last access log write time; a frozen mtime confirms the stall
stat /var/log/httpd/access_log 2>/dev/null || stat /var/log/apache2/access.log

# 10. Probe the critical path; expect timeout or failure
curl -sf -o /dev/null -w "%{http_code}\n" --max-time 5 http://localhost/

Two readings confirm the diagnosis: the scoreboard count from step 3 shows L dominating, and the access log mtime from step 9 is stale even though traffic is arriving.

How to diagnose it

1. Confirm the symptom class. curl the listener. If the TCP connect succeeds but the HTTP request hangs or times out, you have a serving stall, not a down process. Confirm the process is alive with pgrep -o 'httpd|apache2'.

2. Check the disk. df -h and df -i on the log filesystem. 100% on either is your answer. If both are fine, stop and look at the startup semaphore case (ipcs -s) or at file descriptor exhaustion (“Too many open files” in the error log), which produces similar stall behavior.

3. Read the scoreboard. Many L states with IdleWorkers at or near zero confirms workers are stalling on log writes, not on a backend (which would show W states) or slow clients (which would show R states).

4. Identify what filled the disk. ls -lhS on the log directory. If an Apache log is the largest file, it is your logging. If the biggest consumers are elsewhere, another process filled a shared filesystem and Apache is collateral damage.

5. Check for the deleted-but-open trap. If someone already “rotated” by moving or deleting the log file without signalling Apache, df may still show 100% because Apache holds the deleted inode open. lsof +L1 shows these. Space is not reclaimed until the file handle closes, which requires a graceful restart or restart.

6. Check the piped logger. If CustomLog or ErrorLog pipes to rotatelogs and pgrep -af rotatelogs returns nothing, the pipe target died. Expect SIGPIPE child crashes in the error log alongside the disk errors.

Metrics and signals to monitor

Fixes

Free space now

Delete or archive old rotated logs first. *.gz files from previous rotations are the safest candidates. Truncating an actively written Apache log is fragile because it loses the existing log window; Apache opens log files with append semantics, so the next append starts at the newly truncated end, but rotation plus graceful reopen is still the safer operation.

Force rotation, then signal Apache correctly

# Force rotation, then verify Apache reopened its logs
logrotate -f /etc/logrotate.d/httpd
apachectl graceful

logrotate -f is disruptive only in that it rotates immediately; it is safe for the service. The critical part is what follows: Apache must reopen its log files, or it keeps writing to the old (now renamed or deleted) handle and the space is never freed. Most distro logrotate configs do this with a postrotate script that runs apachectl graceful. If yours does not, or if the graceful restart silently failed during the incident, run it yourself and verify with lsof +L1 that no deleted log files remain open.

Avoid copytruncate in logrotate configs for Apache. It races with Apache’s writes and loses log lines. Rename plus graceful restart, or piped logging, is the correct pattern.

Restart if the stall persists

If workers remain wedged in L after space is freed and logs are reopened, restart the service: apachectl restart (or systemctl restart httpd / apache2). This drops in-flight connections, so prefer the graceful path first. Once workers are blocked on a failed log write, a restart is often the fastest way to clear them. Note that some log rotation setups use kill -HUP, which is a hard restart and drops all connections; know which one your tooling sends.

Fix the piped logger

If the logger exited, freeing space lets httpd’s reliable piped logging respawn it; restart Apache only if the replacement does not appear or the pipe remains stalled. If it keeps exiting, check that its target directory is writable and on a filesystem with space, and inspect the downstream consumer.

Clean up semaphores only if that is your actual case

If the failure was at startup with accept lock errors and free disk, the standard recovery is removing orphaned semaphore arrays owned by the Apache user with ipcrm, for example ipcs -s | grep apache | awk '{print $2}' | xargs -n 1 ipcrm sem (adjust the owner pattern to your Apache user). This is safe for semaphores owned by the web server user while Apache is stopped, but destructive if you target the wrong owner, so verify with ipcs -s first. Raising kernel.sem limits addresses chronic recurrence.

Prevention

• Put logs on their own filesystem. This is the single most effective control. When logs share a filesystem with the OS or application, a log explosion takes down the whole server, and any other process can fill the disk and stall Apache. A dedicated log partition caps the blast radius at “Apache stalls,” which your monitoring can catch early.
• Make rotation real. Verify logrotate runs (timer or cron), rotates on size as well as time for high-traffic servers, and signals Apache correctly. Test it: run logrotate -f and confirm lsof +L1 is clean afterward.
• Cap log verbosity in production. Default LogLevel warn for the error log. Debug-level logging and request body logging fill disks at rates rotation cannot keep up with.
• Alert at 80%, not 100%. Disk-full is a cliff-edge failure. A log filesystem above 80% is a ticket; by 100% you are already in the stall.
• Monitor the L state. A sustained L fraction in the scoreboard is the earliest Apache-side signal, ahead of probe failures.
• Watch the piped logger as a process. If you pipe logs, the pipe target is part of your serving path. Alert if it disappears.
• Bound error log growth from noisy modules. Per-module LogLevel (for example LogLevel warn rewrite:info rather than global debug) keeps diagnostics available without flooding the disk.

How Netdata helps

• Scoreboard state distribution over time: Netdata charts every worker state including L, so a log stall shows up as a visible band forming minutes before probes fail.
• Disk usage and inode metrics per mount: the log filesystem trend is charted continuously, and alerts at warning thresholds fire long before 100%.
• Correlation in one view: worker states, BusyWorkers/IdleWorkers, request rate, and disk fullness on the same dashboard let you confirm “disk full causes L states causes zero throughput” in seconds instead of three terminals.
• Process monitoring for rotatelogs: the piped-log process disappearing is visible as a process count change, catching the SIGPIPE churn variant early.
• Request rate collapse: the gap between “port open” and “requests served” is exactly what per-second request metrics expose.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• Apache error log monitoring: severity levels, AH codes, and what to alert on
• Apache BusyWorkers and IdleWorkers: reading worker utilization from mod_status
• How Apache HTTPD actually works in production: a mental model for operators
• Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion
• Apache 5xx error rate: 500 vs 502 vs 503 vs 504 and what each one means
• Apache backend response time: telling ‘Apache is slow’ from ’the backend is slow’