Piped logging, where ErrorLog or CustomLog points at a program instead of a file (CustomLog "| /usr/bin/rotatelogs /var/log/httpd/access_log.%Y-%m-%d 86400" combined), adds a new failure mode to Apache: the log program itself. When rotatelogs (or whatever sits at the end of the pipe) dies, wedges, or cannot write because its target disk is full, the pipe breaks. Log writes then fail silently, or worse, the Apache children writing to the broken pipe receive SIGPIPE and can crash.

The operational symptoms are confusing on first encounter: access logs stop updating, the error log may show nothing or a burst of child crashes, workers pile up in the Logging (L) state on the scoreboard, and the server looks alive while quietly losing observability or shedding capacity. If you are running piped logs and something looks off, the log pipeline is a prime suspect.

This article covers how to detect a dead or stalled log-pipe process, what the failure cascade looks like, and the trade-offs between piped logging and plain files with logrotate.

What this means

With file-based logging, each Apache child holds an open file descriptor per log file and writes directly. With piped logging, Apache spawns the log program at startup and each child writes log lines into a pipe. The program on the other end (rotatelogs, cronolog, a syslog forwarder, mod_security’s audit log pipe) owns the actual files and handles rotation.

That indirection buys you rotation without restarts, but it makes the log program part of the request-serving critical path:

  • If the log program exits, the read end of the pipe closes. The next write() from a child delivers SIGPIPE, which can kill the child.
  • If the log program is alive but blocked (for example, its output filesystem is full or I/O is saturated), the pipe fills and children block in the Logging state. Enough blocked children and you get the Log Stall Deadlock: workers finish requests but cannot log them, so effective capacity collapses while the process looks healthy.
  • Each piped logger consumes file descriptors. On VHost-heavy servers with a pipe per vhost log, FD usage multiplies fast.
flowchart TD
  A[rotatelogs dies
or output disk fills] --> B[pipe breaks or backs up]
  B --> C[child write fails:
SIGPIPE or blocked write]
  C --> D[child process crash
sporadic segfaults/exits]
  C --> E[workers stuck in L state
on scoreboard]
  D --> F[parent respawns children
churn and instability]
  E --> G[BusyWorkers climbs
IdleWorkers falls to zero]
  G --> H[connections queue in backlog
then refused]
  B --> I[access log goes silent
observability lost]

One mitigation worth knowing: Apache 2.4 treats piped loggers as “reliable” and will restart the piped log process if it crashes while the server is running. That helps with transient crashes, but it does not save you if the program keeps dying (bad path, full disk), and it does not unblock children already blocked on a full pipe.

Common causes

CauseWhat it looks likeFirst thing to check
rotatelogs process diedAccess log timestamps stop advancing; sporadic child exits; pipe FDs show as brokenpgrep -af rotatelogs shows nothing or fewer instances than configured
Log filesystem fulldf at 100%; rotatelogs emits “messages lost” or dies; workers in L statedf -h on the log filesystem
Bad pipe configuration after restartApache fails to start or logs AH00104/AH00088 style pipe errorsapachectl configtest, then error log at startup
Graceful restart raceAH00646: Error writing to pipe errors after reloads; transient broken pipeError log around “resuming normal operations” events
Too many piped loggersHigh FD count per child, “Too many open files” under loadPer-process FD counts vs ulimit
Slow consumer in the pipeIntermediate filter buffers or stalls; workers accumulate in LScoreboard state distribution

Quick checks

All of these are read-only.

# 1. Are the piped log processes actually running?
pgrep -af rotatelogs

# 2. Scoreboard: how many workers are stuck in Logging state?
curl -s http://localhost/server-status?auto | grep "Scoreboard:" | \
  awk '{print $2}' | fold -w1 | sort | uniq -c | sort -nr

# 3. Is the log filesystem full?
df -h /var/log/apache2/ 2>/dev/null || df -h /var/log/httpd/

# 4. Recent pipe and disk errors in the error log
grep -E "AH00646|piped log|No space left|Too many open files" \
  /var/log/apache2/error.log /var/log/httpd/error_log 2>/dev/null | tail -20

# 5. FD usage per Apache process (each pipe costs FDs)
for pid in $(pgrep 'httpd|apache2'); do
  echo -n "PID $pid: "; ls /proc/$pid/fd 2>/dev/null | wc -l
done

# 6. Did the access log stop advancing?
ls -l --time-style=full-iso /var/log/apache2/access.log 2>/dev/null || \
  ls -l --time-style=full-iso /var/log/httpd/access_log
date

Two readings matter most: an L-heavy scoreboard (check 2) means children are blocked writing logs right now, and a stalled log mtime with a running server (check 6) means the pipeline is broken even if nothing is obviously blocked yet.

How to diagnose it

  1. Confirm the pipeline is the problem. Compare the access log mtime with the current time and with request activity. If mod_status shows requests completing (Total Accesses climbing) but the access log is frozen, the write path is broken, not the traffic.

  2. Check the log program’s process table entry. You should see one rotatelogs instance per piped log directive, parented to the Apache parent. If the count is lower than the number of piped directives, one died and either was restarted with a new output file or the pipe is broken. If you see zombie rotatelogs processes, that points to a hung parent or a debugging build flag rather than normal operation.

  3. Check the scoreboard for L and G states. Sustained L above a few percent of workers is the log-stall signature. Many G states after a reload, combined with AH00646 broken-pipe errors, points at the graceful restart race (below).

  4. Check disk space and I/O on the log filesystem. df -h for space, iostat -xz 1 3 for saturation. rotatelogs on a full filesystem emits “error writing to logfile … messages lost”, truncates, and continues, or dies outright. On platforms without large-file support, the same message appears when a file hits 2GB.

  5. Check FD headroom. Each piped logger costs pipe FDs in the parent and children. If per-child FD counts are near the limit (cat /proc/$(pgrep -o 'httpd|apache2')/limits | grep "Max open files"), a burst of connections plus the pipe FDs can tip a child into EMFILE, and log writes are among the first things to fail.

  6. Correlate with restart events. grep -E "resuming normal operations|caught SIGTERM" error_log | tail -20. Broken-pipe errors clustered right after graceful restarts are the known piped-logging race: during overlapping graceful restarts the piped log process can be terminated before all old-generation workers finish writing to it (tracked upstream as Apache Bug 61926).

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Scoreboard L state countDirect view of workers blocked on log writesSustained L above ~5% of workers
Log-pipe process countA dead rotatelogs is the failure itselfFewer instances than piped directives; zombies
Log filesystem free spaceFull disk kills or stalls the log programAbove 80% used
Log file mtime / write rateSilent pipeline failure shows up here firstmtime stale while requests complete
Per-process FD count vs limitPipes multiply FD usage, especially per-vhostAny child above 70% of its limit
Error log: AH00646, “No space left”, “Too many open files”The explicit pipe/disk/FD failure messagesAny occurrence outside a rotation window
Child exit / segfault rateSIGPIPE can kill children writing to a broken pipeSporadic exits correlated with log gaps

Fixes

Dead log-pipe process

Apache 2.4 restarts a crashed piped logger on its own, so if the process is gone and staying gone, the restart itself is failing. The most common reasons are configuration-level: the logger path must be absolute, and the logfile path passed to rotatelogs must also be absolute. A relative path produces pipe-open errors at startup or respawn. Fix the directive, run apachectl configtest, then do a full restart. A graceful reload is not guaranteed to cleanly recover a broken pipe state, and on some versions the parent itself can fault during rapid successive graceful reloads, so prefer a planned full restart when repairing piped logging.

Note the 2.2 to 2.4 behavior change: in 2.4, piped commands run directly without a shell. If your directive relies on shell syntax (redirection, command chaining), prefix it with |$ to invoke it via /bin/sh -c, or it will fail to spawn.

Full log filesystem

Free space first: compress or move old rotated files, then confirm rotatelogs resumed writing (mtime advancing). If rotatelogs truncated and continued, expect a gap. Do not delete the file a running rotatelogs has open; the space will not be released until the process reopens it. A graceful restart forces loggers to reopen their outputs, but see the race warning below. Longer term, put logs on their own filesystem so a log explosion cannot starve the OS or application, and size rotation intervals to the disk.

Graceful restart broken-pipe errors (AH00646)

If broken-pipe errors cluster around graceful restarts, you are hitting the piped-logging race where the logger is torn down before old workers drain. Practical mitigations: reduce graceful restart frequency (batch config changes, do not reload per logrotate run), set GracefulShutdownTimeout so old generations cannot linger indefinitely, and avoid multiple overlapping reload scripts. If your logrotate setup triggers an Apache reload per rotation, that alone can produce this pattern under load.

FD exhaustion from too many piped loggers

Per-vhost CustomLog pipes scale FD usage with vhost count. Consolidate to a single piped logger and split by vhost in the log program or downstream, and raise LimitNOFILE in the systemd unit with headroom (the playbook guidance is at least 2x theoretical maximum usage). Raising the limit requires a full restart, not a reload.

Deciding: piped logs vs files plus logrotate

Piped logging survives rotation better than file logging: rotation happens inside the logger, so there is no rename-and-reopen window and no need to signal Apache when files rotate. Plain files with logrotate using copytruncate can lose log lines in the race between copy and truncate; the safe file-based approach is rename plus SIGUSR1 (graceful restart) so Apache reopens its logs, but that reintroduces reload frequency and the graceful-restart failure modes above.

The trade-off: piped logs put a process on the critical write path (SIGPIPE and stall risk, extra FDs), while file logging keeps writes kernel-direct but makes rotation an orchestration problem. For high-traffic servers, a single piped rotatelogs for access logs plus a dedicated log filesystem is a solid default. For many vhosts, prefer one pipe with vhost splitting over one pipe per vhost.

Prevention

  • Monitor the log pipeline as a first-class dependency. Track rotatelogs process count, log mtime freshness, and scoreboard L state, not just disk space.
  • Use absolute paths everywhere in piped directives, both the logger binary and its output files, and configtest before every reload.
  • Dedicate a filesystem to logs and alert at 80% so a full disk never reaches the pipe.
  • Bound reload frequency. Batch config changes, avoid logrotate-triggered graceful restarts where possible, and set GracefulShutdownTimeout (for example 30s) so old workers drain or die on a deadline.
  • Budget FDs for pipes. Each pipe costs FDs in every child; size LimitNOFILE with at least 2x headroom over the theoretical maximum.
  • Keep the pipe simple. Every extra program in the chain adds buffering and a new way to stall; pipe directly to rotatelogs and do filtering downstream.

How Netdata helps

  • Scoreboard state tracking over time. Netdata collects the Apache scoreboard continuously, so a sustained L-state buildup shows up as a trend, not a snapshot you happened to catch, and you can see exactly when it started relative to a reload or disk event.
  • Correlation with disk and process signals. A log stall is confirmed when L states rise at the same moment the log filesystem hits full or the rotatelogs process count drops. Seeing those on one dashboard removes the guesswork.
  • FD and memory context. Per-process FD usage and child RSS alongside worker utilization tells you whether pipe FDs are contributing to a limit problem.
  • Error-log and uptime correlation. Restart events and child churn lined up against log-pipeline metrics make the graceful-restart race visible instead of anecdotal.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.