The Apache parent process is running. The port is open. A TCP connect succeeds. But nothing is served: requests hang, throughput is at or near zero, and the scoreboard is a wall of L characters. Every worker has finished its request and is now blocked writing the log line for it.

Workers write access log entries synchronously at the end of the request cycle, before returning to the idle pool. If that write blocks, because the log filesystem is full or a piped logging program has stopped reading, the worker never becomes available again. New workers accept new connections, finish those requests, and block on the same write. Within minutes the entire worker pool is frozen in the Logging state.

The error log usually stops updating too, because it lives on the same filesystem or travels through the same kind of pipe. Your primary diagnostic output freezes at the moment you need it most.

What this means

Each scoreboard slot shows what that worker is doing right now: R reading, W writing, K keepalive, L logging, _ idle. A healthy scoreboard under load is mostly W, K, and _, with a scattering of L that clears in milliseconds per request.

A scoreboard dominated by L for more than a couple of minutes means the log write path itself is blocked. Two root causes produce this:

  • Disk full. The filesystem hosting the access or error log hits 100%. Writes fail or stall, and workers cannot complete the request cycle. df shows 100% on the log filesystem, the error log may stop updating entirely, and the scoreboard fills with L.
  • Blocked log pipe. With piped logging (CustomLog "|/path/to/program"), all workers share a single pipe to one logger process. Pipes have a finite kernel buffer (the Linux default is 64 KB). If the piped program dies, stalls, or stops reading, the buffer fills and every worker’s next log write blocks. rotatelogs is the usual program in this role; if it crashes or wedges, Apache goes down with it.
flowchart TD
  A[Disk full or log pipe blocked] --> B[Log write blocks in worker]
  B --> C[Worker stuck in L state after finishing request]
  C --> D[BusyWorkers climbs, IdleWorkers hits zero]
  D --> E[New connections queue in listen backlog]
  E --> F[Throughput collapses to zero, probes time out]
  C --> G[Error log stops updating]

Brief L spikes are normal. During log rotation the old file handle closes and a new one opens, and you may catch a few workers in L for a second or two. Page only when all four hold: L dominates the scoreboard, the condition persists for more than 2 minutes, a critical-path probe is failing or throughput has collapsed, and disk is full or log I/O is saturated. Anything short of that combination is a ticket-level investigation.

Common causes

CauseWhat it looks likeFirst thing to check
Log filesystem fulldf at 100% on the log partition; error log frozen mid-linedf -h on the log directory
Piped logger died or stalledScoreboard all L; pipe target process missing or not readingIs the rotatelogs (or custom) process alive and consuming?
Log I/O saturationDisk not full but writes extremely slow; L builds during burstsiostat on the log device
Log rotation gone wrongRotation script deleted files Apache still holds open, so space never frees; or rotation triggered a hangDeleted-but-open files on the log filesystem; recent rotation in cron logs
Another process filled the shared filesystemLog partition is the root or app partition; something else ate the spacedu on the largest directories of that filesystem

Quick checks

# 1. Scoreboard state distribution: is L dominant?
curl -s http://localhost/server-status?auto | grep "Scoreboard:" | \
  awk '{print $2}' | fold -w1 | sort | uniq -c | sort -nr

# 2. Worker utilization
curl -s http://localhost/server-status?auto | grep -E "BusyWorkers|IdleWorkers"

# 3. Critical-path probe (does an actual request complete?)
curl -sf -o /dev/null -w "%{http_code}\n" --max-time 5 http://localhost/health

# 4. Disk space on the log filesystem
df -h /var/log/apache2/ 2>/dev/null || df -h /var/log/httpd/

# 5. Is the error log still being written?
ls -l --time-style=full-iso /var/log/apache2/error.log 2>/dev/null || \
  ls -l --time-style=full-iso /var/log/httpd/error_log
tail -5 /var/log/apache2/error.log 2>/dev/null || tail -5 /var/log/httpd/error_log

# 6. Piped logging processes: alive?
ps aux | grep -E "rotatelogs|cronolog" | grep -v grep

# 7. Listen backlog building up behind the frozen workers?
ss -ltn | grep -E ':80\s|:443\s'

Notes on reading these:

  • Check 1 is the diagnostic. If L is the plurality of slots and stays that way across two samples 30 seconds apart, you have the stall. A scoreboard mostly in W with a slow backend is a different incident (see the related 504 guide).
  • Check 5 is subtle: compare the file mtime against the current time. An error log that has not been written in minutes on a busy server is itself a symptom, not a sign of health.
  • Check 7 confirms the cascade stage: a non-zero, growing Recv-Q on the listen socket means new connections are queuing because no worker can accept them.

How to diagnose it

  1. Confirm the L-state dominance. Take two scoreboard samples 30 seconds apart. If L is dominant in both and IdleWorkers is at or near zero, treat it as the log stall, not a transient rotation blip.

  2. Rule out the lookalikes. A scoreboard full of W (slow backend cascade) and a scoreboard full of R (Slowloris) also zero out throughput. The state character is the discriminator. Do not skip this step, because the fix for each is completely different.

  3. Check disk space. df -h on the log filesystem. At 100%, you have your cause. Also check df -i if the filesystem has many small files; inode exhaustion produces the same write failures with free blocks remaining.

  4. If disk is not full, check the pipe. Look at your config for CustomLog "|..." and ErrorLog "|..." directives. Find the piped process in the process table. If it is missing, that is the cause: the pipe write end still exists but nobody is reading. If it exists but is stuck, check what it is doing (it may itself be blocked writing downstream, for example to a full disk or a dead log shipper).

  5. Check I/O saturation. If the disk is neither full nor piped, iostat -xz 1 3 on the log device tells you whether writes are simply too slow to drain. This variant shows up on shared or network storage, and as brief L storms during backup windows or log compression.

  6. Decide severity. Page when all four hold: L dominates, sustained over 2 minutes, probe failing or throughput collapsed, and disk full or log I/O saturated. Otherwise ticket and work the cause.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Scoreboard L countDirect measure of workers blocked on log writesL >5% of slots sustained; dominant state >2 min
BusyWorkers / IdleWorkersShows the pool draining as workers freezeIdleWorkers at zero with L dominant
Log filesystem free spaceThe leading indicator that prevents this incident>80% full
Log device I/O latencyCatches the slow-disk variant before the full-disk oneSustained high await on the log filesystem
Critical-path HTTP probeDetects that the frozen server is actually failing requestsTimeout or non-2xx >5s
Total Accesses delta (RPS)Confirms throughput collapseRate near zero while LB still sends traffic
Listen backlog Recv-QShows connections queuing behind dead workersSustained non-zero and growing
Error log mtimeA frozen error log is both a symptom and a blind spotNo writes for minutes on a busy server

Fixes

Disk full

Free space first, restart second.

# Find what is consuming space on the log filesystem
du -xh /var/log | sort -rh | head -20

Delete or compress old rotated logs, then check for the classic trap: files deleted while Apache still holds them open do not free space. If df still shows 100% after deleting files, look for deleted-but-open files (lsof +L1, or lsof | grep deleted on the log filesystem) and restart Apache to release them. Also verify nothing else shares the filesystem: if the log directory sits on the root partition, the thing that filled it may be an application temp directory, a core dump, or a package cache, and it will fill again.

After space is freed, restart httpd. Workers frozen in L do not reliably recover on their own once the write would succeed; a restart is the clean way back. This is disruptive: it drops all in-flight connections. Here it is justified, because the server is already serving nothing, but on a partially healthy server use a graceful restart first.

Blocked log pipe

If the piped logger (rotatelogs, cronolog, or a custom shipper) has died, the only reliable recovery is to restart Apache so the pipe and the logger process are recreated. Same disruption warning applies. Then fix why it died: a downstream destination that stopped accepting writes, the logger crashing on rotation, or a shipper that blocks when its output stalls. A piped logger whose own output can block (for example, forwarding to a remote system over a socket that stops draining) will take Apache down with it every time; treat the whole chain as part of Apache’s critical path.

Slow log I/O

Move logs to a dedicated local filesystem, reduce log volume (drop noisy health-check requests from the access log with conditional logging), and make sure rotation-time compression is not competing with live writes on the same device.

Prevention

  • Dedicated log filesystem. Put Apache logs on their own partition so a log explosion cannot take down the OS or the application, and so application growth cannot fill the log disk. This is the single most effective structural defense against this pattern.
  • Working log rotation. Verify rotation actually runs: check cron/systemd timer logs and confirm rotated files appear on schedule. A rotation config silently broken for months is how most log disks fill.
  • Alert on log disk at 80%. This incident has a long, visible runway. There is no excuse for being paged by a full disk that took weeks to fill.
  • Alert on L state. Scoreboard monitoring with an L-state threshold (sustained >5% of workers, or any L dominance over 2 minutes) catches the pipe variant, which has no disk-space warning.
  • Watch piped logger health. If you use piped logging, the logger process is a single point of failure for the entire server. Monitor that it is alive and that its downstream destination is draining.
  • Include No space left in error log grep patterns. Process, probe, disk, and error log keyword checks catch this entire failure class cheaply.

How Netdata helps

  • Netdata’s Apache collector polls server-status continuously and charts the scoreboard state distribution over time, so you see L climbing toward dominance minutes before the pool freezes, instead of discovering it from user reports.
  • BusyWorkers and IdleWorkers are graphed alongside request rate, making the signature obvious: workers pinned, throughput collapsing, process still up.
  • Disk space and I/O latency for the log filesystem are collected per-mount, so the disk-full cause is visible in the same dashboard as the scoreboard symptom, which is what shortens diagnosis from “why is Apache hung” to “the log partition hit 100% at 03:12.”
  • Critical-path HTTP checks and TCP listener metrics corroborate that the frozen pool is user-impacting, matching the paging criteria rather than alerting on a transient rotation blip.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.