$ guides / apache-httpd / apache-httpd-piped-logging-rotatelogs ▌

Operations Guides

Apache piped logging failures: dead rotatelogs, AH00106, AH00646, and Logging-state workers

Piped logging, where ErrorLog or CustomLog points at a program instead of a file (CustomLog "| /usr/bin/rotatelogs /var/log/httpd/access_log.%Y-%m-%d 86400" combined), adds a new failure mode to Apache: the log program itself. When rotatelogs (or whatever sits at the end of the pipe) dies, wedges, or cannot write because its target disk is full, logging can silently lose records or block workers. Upstream Apache restarts a crashed reliable piped logger and logs AH00106; it does not normally crash children on SIGPIPE.

The operational symptoms are confusing on first encounter: access logs stop updating, the error log may show nothing or a burst of child crashes, workers pile up in the Logging (L) state on the scoreboard, and the server looks alive while quietly losing observability or shedding capacity. If you are running piped logs and something looks off, the log pipeline is a prime suspect.

This article covers how to detect a dead or stalled log-pipe process, what the failure cascade looks like, and the trade-offs between piped logging and plain files with logrotate.

What this means

With file-based logging, each Apache child holds an open file descriptor per log file and writes directly. With piped logging, Apache spawns the log program at startup and each child writes log lines into a pipe. The program on the other end (rotatelogs, cronolog, a syslog forwarder, mod_security’s audit log pipe) owns the actual files and handles rotation.

That indirection buys you rotation without restarts, but it makes the log program part of the request-serving critical path:

If the log program exits, Apache’s reliable piped-logging maintenance restarts it and logs AH00106; a persistent restart failure is the remaining hazard.
If the log program is alive but blocked (for example, its output filesystem is full or I/O is saturated), the pipe fills and children block in the Logging state. Enough blocked children and you get the Log Stall Deadlock: workers finish requests but cannot log them, so effective capacity collapses while the process looks healthy.
Each piped logger consumes file descriptors. On VHost-heavy servers with a pipe per vhost log, FD usage multiplies fast.

flowchart TD
  A[rotatelogs dies
or output disk fills] --> B[pipe restarts/back logs or backs up]
  B --> C[write fails/blocks:
AH00646 or blocked logger]
  C --> D[logger restart failure
or logged write failures]
  C --> E[workers stuck in L state
on scoreboard]
  D --> F[parent respawns children
churn and instability]
  E --> G[BusyWorkers climbs
IdleWorkers falls to zero]
  G --> H[connections queue in backlog
then refused]
  B --> I[access log goes silent
observability lost]

One mitigation worth knowing: Apache 2.4 treats piped loggers as “reliable” and will restart the piped log process if it crashes while the server is running. That helps with transient crashes, but it does not save you if the program keeps dying (bad path, full disk), and it does not unblock children already blocked on a full pipe.

Common causes

Cause	What it looks like	First thing to check
rotatelogs process died	`AH00106`; access-log timestamps stop advancing; fewer process instances than configured	`pgrep -af rotatelogs` and `AH00106` in the error log
Log filesystem full	`df` at 100%; rotatelogs emits “messages lost” or dies; workers in `L` state	`df -h` on the log filesystem
Bad pipe configuration after restart	Apache fails to start or logs AH00104/AH00088 style pipe errors	`apachectl configtest`, then error log at startup
Graceful restart race	`AH00646: Error writing to` pipe errors after reloads; transient broken pipe	Error log around “resuming normal operations” events
Too many piped loggers	High FD count per child, “Too many open files” under load	Per-process FD counts vs ulimit
Slow consumer in the pipe	Intermediate filter buffers or stalls; workers accumulate in `L`	Scoreboard state distribution

Quick checks

All of these are read-only.

# 1. Are the piped log processes actually running?
pgrep -af rotatelogs

# 2. Scoreboard: how many workers are stuck in Logging state?
curl -s http://localhost/server-status?auto | grep "Scoreboard:" | \
  awk '{print $2}' | fold -w1 | sort | uniq -c | sort -nr

# 3. Is the log filesystem full?
df -h /var/log/apache2/ 2>/dev/null || df -h /var/log/httpd/

# 4. Recent pipe and disk errors in the error log
grep -E "AH00646|piped log|No space left|Too many open files" \
  /var/log/apache2/error.log /var/log/httpd/error_log 2>/dev/null | tail -20

# 5. FD usage per Apache process (each pipe costs FDs)
for pid in $(pgrep 'httpd|apache2'); do
  echo -n "PID $pid: "; ls /proc/$pid/fd 2>/dev/null | wc -l
done

# 6. Did the access log stop advancing?
ls -l --time-style=full-iso /var/log/apache2/access.log 2>/dev/null || \
  ls -l --time-style=full-iso /var/log/httpd/access_log
date

Two readings matter most: an L-heavy scoreboard (check 2) means children are blocked writing logs right now, and a stalled log mtime with a running server (check 6) means the pipeline is broken even if nothing is obviously blocked yet.

How to diagnose it

Confirm the pipeline is the problem. Compare the access log mtime with the current time and with request activity. If mod_status shows requests completing (Total Accesses climbing) but the access log is frozen, the write path is broken, not the traffic.
Check the log program’s process table entry. You should see one rotatelogs instance per piped log directive, parented to the Apache parent. If the count is lower than the number of piped directives, one died and either was restarted with a new output file or the pipe is broken. If you see zombie rotatelogs processes, that points to a hung parent or a debugging build flag rather than normal operation.
Check the scoreboard for L and G states. Sustained L above a few percent of workers is the log-stall signature. Many G states after a reload, combined with AH00646 broken-pipe errors, points at the graceful restart race (below).
Check disk space and I/O on the log filesystem. df -h for space, iostat -xz 1 3 for saturation. rotatelogs on a full filesystem emits “error writing to logfile … messages lost”, truncates, and continues, or dies outright. On platforms without large-file support, the same message appears when a file hits 2GB.
Check FD headroom. Each piped logger costs pipe FDs in the parent and children. If per-child FD counts are near the limit (cat /proc/$(pgrep -o 'httpd|apache2')/limits | grep "Max open files"), a burst of connections plus the pipe FDs can tip a child into EMFILE, and log writes are among the first things to fail.
Correlate with restart events. grep -E "resuming normal operations|caught SIGTERM" error_log | tail -20. Broken-pipe errors clustered right after graceful restarts are the known piped-logging race: during overlapping graceful restarts the piped log process can be terminated before all old-generation workers finish writing to it (tracked upstream as Apache Bug 61926; its public Bugzilla status was not independently readable, so verify it directly if you need its current state).

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Scoreboard `L` state count	Direct view of workers blocked on log writes	Sustained `L` above ~5% of workers
Log-pipe process count	A dead rotatelogs is the failure itself	Fewer instances than piped directives; zombies
Log filesystem free space	Full disk kills or stalls the log program	Above 80% used
Log file mtime / write rate	Silent pipeline failure shows up here first	mtime stale while requests complete
Per-process FD count vs limit	Pipes multiply FD usage, especially per-vhost	Any child above 70% of its limit
Error log: AH00646, “No space left”, “Too many open files”	The explicit pipe/disk/FD failure messages	Any occurrence outside a rotation window
Child exit / segfault rate	Separate persistent logger failures from unrelated crashes	Exits correlated with `AH00106`/`AH00646` and log gaps

Fixes

Dead log-pipe process

Apache 2.4 restarts a crashed piped logger on its own, so if the process is gone and staying gone, the restart itself is failing. The most common reasons are configuration-level: the logger path must be absolute, and the logfile path passed to rotatelogs must also be absolute. A relative path produces pipe-open errors at startup or respawn. Fix the directive, run apachectl configtest, then do a full restart. A graceful reload is not guaranteed to cleanly recover a broken pipe state, and on some versions the parent itself can fault during rapid successive graceful reloads, so verify your exact 2.4.x behavior and prefer a planned restart when repairing piped logging.

Note the 2.2 to 2.4 behavior change: in 2.4, piped commands run directly without a shell. If your directive relies on shell syntax (redirection, command chaining), prefix it with |$ to invoke it via /bin/sh -c, or it will fail to spawn.

Full log filesystem

Free space first: compress or move old rotated files, then confirm rotatelogs resumed writing (mtime advancing). If rotatelogs truncated and continued, expect a gap. Do not delete the file a running rotatelogs has open; the space will not be released until the process reopens it. A graceful restart forces loggers to reopen their outputs, but see the race warning below. Longer term, put logs on their own filesystem so a log explosion cannot starve the OS or application, and size rotation intervals to the disk.

Graceful restart broken-pipe errors (AH00646)

If broken-pipe errors cluster around graceful restarts, you are hitting the piped-logging race where the logger is torn down before old workers drain. Practical mitigations: reduce graceful-restart frequency (batch config changes, do not reload per logrotate run), bound request/keepalive/proxy timeouts, and avoid multiple overlapping reload scripts; GracefulShutdownTimeout applies to graceful-stop, not a graceful reload. If your logrotate setup triggers an Apache reload per rotation, that alone can produce this pattern under load. The current 2.4.x source still terminates a reliable piped logger on restart, but the public Bugzilla status of the race report was not independently readable; verify your exact release if you need its fix state.

FD exhaustion from too many piped loggers

Per-vhost CustomLog pipes scale FD usage with vhost count. Consolidate to a single piped logger and split by vhost in the log program or downstream, and raise LimitNOFILE in the systemd unit with headroom (the playbook guidance is at least 2x theoretical maximum usage). Raising the limit requires a full restart, not a reload.

Deciding: piped logs vs files plus logrotate

Piped logging survives rotation better than file logging: rotation happens inside the logger, so there is no rename-and-reopen window and no need to signal Apache when files rotate. Plain files with logrotate using copytruncate can lose log lines in the race between copy and truncate; the safe file-based approach is rename plus SIGUSR1 (graceful restart) so Apache reopens its logs, but that reintroduces reload frequency and the graceful-restart failure modes above.

The trade-off: piped logs put a process on the critical write path (restart/stall risk and extra FDs), while file logging keeps writes kernel-direct but makes rotation an orchestration problem. For high-traffic servers, a single piped rotatelogs for access logs plus a dedicated log filesystem is a solid default. For many vhosts, prefer one pipe with vhost splitting over one pipe per vhost.

Prevention

Monitor the log pipeline as a first-class dependency. Track rotatelogs process count, log mtime freshness, and scoreboard L state, not just disk space.
Use absolute paths everywhere in piped directives, both the logger binary and its output files, and configtest before every reload.
Dedicate a filesystem to logs and alert at 80% so a full disk never reaches the pipe.
Bound reload frequency. Batch config changes, avoid logrotate-triggered graceful restarts where possible, and tune request/keepalive/proxy timeouts; GracefulShutdownTimeout is for graceful-stop.
Budget FDs for pipes. Each pipe costs FDs in every child; size LimitNOFILE with at least 2x headroom over the theoretical maximum.
Keep the pipe simple. Every extra program in the chain adds buffering and a new way to stall; pipe directly to rotatelogs and do filtering downstream.

How Netdata helps

Scoreboard state tracking over time. Netdata collects the Apache scoreboard continuously, so a sustained L-state buildup shows up as a trend, not a snapshot you happened to catch, and you can see exactly when it started relative to a reload or disk event.
Correlation with disk and process signals. A log stall is confirmed when L states rise at the same moment the log filesystem hits full or the rotatelogs process count drops. Seeing those on one dashboard removes the guesswork.
FD and memory context. Per-process FD usage and child RSS alongside worker utilization tells you whether pipe FDs are contributing to a limit problem.
Error-log and uptime correlation. Restart events and child churn lined up against log-pipeline metrics make the graceful-restart race visible instead of anecdotal.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

The Netdata solution

Apache HTTP Server monitoring with Netdata

Netdata monitors Apache HTTP Server with per-second metrics from mod_status, pre-built dashboards, and ML-powered anomaly detection. Watch busy versus idle workers and the scoreboard state mix, requests per second, bytes served per second, and request processing duration alongside the rest of your stack, so you catch the worker-exhaustion, slow-backend, and memory incidents in these runbooks before they page anyone.

See web server monitoring → Start monitoring free