$ guides / apache-httpd / apache-httpd-log-stall-deadlock ▌

Operations Guides

Apache log stall deadlock: workers stuck in the L state, throughput at zero

The Apache parent process is running. The port is open. A TCP connect succeeds. But nothing is served: requests hang, throughput is at or near zero, and the scoreboard is a wall of L characters. Every worker has finished its request and is now blocked writing the log line for it.

Workers write access log entries synchronously at the end of the request cycle, before returning to the idle pool. If that write blocks, because the log filesystem is full or a piped logging program has stopped reading, the worker never becomes available again. New workers accept new connections, finish those requests, and block on the same write. Within minutes the entire worker pool is frozen in the Logging state.

The error log usually stops updating too, because it lives on the same filesystem or travels through the same kind of pipe. Your primary diagnostic output freezes at the moment you need it most.

What this means

Each scoreboard slot shows what that worker is doing right now: R reading, W writing, K keepalive, L logging, _ idle. A healthy scoreboard under load is mostly W, K, and _, with a scattering of L that clears in milliseconds per request.

A scoreboard dominated by L for more than a couple of minutes means the log write path itself is blocked. Two root causes produce this:

Disk full. The filesystem hosting the access or error log hits 100%. Writes fail or stall, and workers cannot complete the request cycle. df shows 100% on the log filesystem, the error log may stop updating entirely, and the scoreboard fills with L.
Blocked log pipe. With piped logging (CustomLog "|/path/to/program"), workers write to the pipe that feeds the logger. Pipes have a finite kernel buffer (the Linux default is 64 KB). If the logger exits, httpd’s reliable piped logging restarts it; if it stalls without exiting or the downstream consumer stops reading, the buffer fills and workers block on log writes. rotatelogs is the usual program in this role, and the whole downstream chain matters.

flowchart TD
  A[Disk full or log pipe blocked] --> B[Log write blocks in worker]
  B --> C[Worker stuck in L state after finishing request]
  C --> D[BusyWorkers climbs, IdleWorkers hits zero]
  D --> E[New connections queue in listen backlog]
  E --> F[Throughput collapses to zero, probes time out]
  C --> G[Error log stops updating]

Brief L spikes are normal. During log rotation the old file handle closes and a new one opens, and you may catch a few workers in L for a second or two. Page only when all four hold: L dominates the scoreboard, the condition persists for more than 2 minutes, a critical-path probe is failing or throughput has collapsed, and disk is full or log I/O is saturated. Anything short of that combination is a ticket-level investigation.

Common causes

Cause	What it looks like	First thing to check
Log filesystem full	`df` at 100% on the log partition; error log frozen mid-line	`df -h` on the log directory
Piped logger died or stalled	Scoreboard all `L`; pipe target process missing or not reading	Is the `rotatelogs` (or custom) process alive and consuming?
Log I/O saturation	Disk not full but writes extremely slow; `L` builds during bursts	`iostat` on the log device
Log rotation gone wrong	Rotation script deleted files Apache still holds open, so space never frees; or rotation triggered a hang	Deleted-but-open files on the log filesystem; recent rotation in cron logs
Another process filled the shared filesystem	Log partition is the root or app partition; something else ate the space	`du` on the largest directories of that filesystem

Quick checks

# 1. Scoreboard state distribution: is L dominant?
curl -s http://localhost/server-status?auto | grep "Scoreboard:" | \
  awk '{print $2}' | fold -w1 | sort | uniq -c | sort -nr

# 2. Worker utilization
curl -s http://localhost/server-status?auto | grep -E "BusyWorkers|IdleWorkers"

# 3. Critical-path probe (does an actual request complete?)
curl -sf -o /dev/null -w "%{http_code}\n" --max-time 5 http://localhost/health

# 4. Disk space on the log filesystem
df -h /var/log/apache2/ 2>/dev/null || df -h /var/log/httpd/

# 5. Is the error log still being written?
ls -l --time-style=full-iso /var/log/apache2/error.log 2>/dev/null || \
  ls -l --time-style=full-iso /var/log/httpd/error_log
tail -5 /var/log/apache2/error.log 2>/dev/null || tail -5 /var/log/httpd/error_log

# 6. Piped logging processes: alive?
ps aux | grep -E "rotatelogs|cronolog" | grep -v grep

# 7. Listen backlog building up behind the frozen workers?
ss -ltn | grep -E ':80\s|:443\s'

Notes on reading these:

Check 1 is the diagnostic. If L is the plurality of slots and stays that way across two samples 30 seconds apart, you have the stall. A scoreboard mostly in W with a slow backend is a different incident (see the related 504 guide).
Check 5 is subtle: compare the file mtime against the current time. An error log that has not been written in minutes on a busy server is itself a symptom, not a sign of health.
Check 7 confirms the cascade stage: a non-zero, growing Recv-Q on the listen socket means new connections are queuing because no worker can accept them.

How to diagnose it

Confirm the L-state dominance. Take two scoreboard samples 30 seconds apart. If L is dominant in both and IdleWorkers is at or near zero, treat it as the log stall, not a transient rotation blip.
Rule out the lookalikes. A scoreboard full of W (slow backend cascade) and a scoreboard full of R (Slowloris) also zero out throughput. The state character is the discriminator. Do not skip this step, because the fix for each is completely different.
Check disk space. df -h on the log filesystem. At 100%, you have your cause. Also check df -i if the filesystem has many small files; inode exhaustion produces the same write failures with free blocks remaining.
If disk is not full, check the pipe. Look at your config for CustomLog "|..." and ErrorLog "|..." directives. Find the piped process in the process table. If it is missing, that is the cause: the pipe write end still exists but nobody is reading. If it exists but is stuck, check what it is doing (it may itself be blocked writing downstream, for example to a full disk or a dead log shipper).
Check I/O saturation. If the disk is neither full nor piped, iostat -xz 1 3 on the log device tells you whether writes are simply too slow to drain. This variant shows up on shared or network storage, and as brief L storms during backup windows or log compression.
Decide severity. Page when all four hold: L dominates, sustained over 2 minutes, probe failing or throughput collapsed, and disk full or log I/O saturated. Otherwise ticket and work the cause.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Scoreboard `L` count	Direct measure of workers blocked on log writes	`L` >5% of slots sustained; dominant state >2 min
BusyWorkers / IdleWorkers	Shows the pool draining as workers freeze	IdleWorkers at zero with `L` dominant
Log filesystem free space	The leading indicator that prevents this incident	>80% full
Log device I/O latency	Catches the slow-disk variant before the full-disk one	Sustained high await on the log filesystem
Critical-path HTTP probe	Detects that the frozen server is actually failing requests	Timeout or non-2xx >5s
Total Accesses delta (RPS)	Confirms throughput collapse	Rate near zero while LB still sends traffic
Listen backlog Recv-Q	Shows connections queuing behind dead workers	Sustained non-zero and growing
Error log mtime	A frozen error log is both a symptom and a blind spot	No writes for minutes on a busy server

Fixes

Disk full

Free space first, restart second.

# Find what is consuming space on the log filesystem
du -xh /var/log | sort -rh | head -20

Delete or compress old rotated logs, then check for the classic trap: files deleted while Apache still holds them open do not free space. If df still shows 100% after deleting files, look for deleted-but-open files (lsof +L1, or lsof | grep deleted on the log filesystem) and restart Apache to release them. Also verify nothing else shares the filesystem: if the log directory sits on the root partition, the thing that filled it may be an application temp directory, a core dump, or a package cache, and it will fill again.

After space is freed, restart httpd. Workers frozen in L do not reliably recover on their own once the write would succeed; a restart is the clean way back. This is disruptive: it drops all in-flight connections. Here it is justified, because the server is already serving nothing, but on a partially healthy server use a graceful restart first.

Blocked log pipe

If the logger exited, verify that httpd’s reliable piped logging recreated it; avoid assuming that a stale process name in an old snapshot means the new process is healthy. If the logger is alive but stalled, inspect and fix its downstream: a destination that stopped accepting writes, a crash during rotation, or a shipper blocked on a socket that stops draining. A piped logger whose own output can block will hold Apache workers; treat the whole chain as part of Apache’s critical path.

Slow log I/O

Move logs to a dedicated local filesystem, reduce log volume (drop noisy health-check requests from the access log with conditional logging), and make sure rotation-time compression is not competing with live writes on the same device. BufferedLogs On batches log entries before one disk write and can reduce write frequency, but it does not remove the eventual blocking flush on a saturated device and risks losing buffered entries on a crash.

Prevention

Dedicated log filesystem. Put Apache logs on their own partition so a log explosion cannot take down the OS or the application, and so application growth cannot fill the log disk. This is the single most effective structural defense against this pattern.
Working log rotation. Verify rotation actually runs: check cron/systemd timer logs and confirm rotated files appear on schedule. A rotation config silently broken for months is how most log disks fill.
Alert on log disk at 80%. This incident has a long, visible runway. There is no excuse for being paged by a full disk that took weeks to fill.
Alert on L state. Scoreboard monitoring with an L-state threshold (sustained >5% of workers, or any L dominance over 2 minutes) catches the pipe variant, which has no disk-space warning.
Watch piped logger health. If you use piped logging, the logger process is a single point of failure for the entire server. Monitor that it is alive and that its downstream destination is draining.
Include No space left in error log grep patterns. Process, probe, disk, and error log keyword checks catch this entire failure class cheaply.

How Netdata helps

Netdata’s Apache collector polls server-status continuously and charts the scoreboard state distribution over time, so you see L climbing toward dominance minutes before the pool freezes, instead of discovering it from user reports.
BusyWorkers and IdleWorkers are graphed alongside request rate, making the signature obvious: workers pinned, throughput collapsing, process still up.
Disk space and I/O latency for the log filesystem are collected per-mount, so the disk-full cause is visible in the same dashboard as the scoreboard symptom, which is what shortens diagnosis from “why is Apache hung” to “the log partition hit 100% at 03:12.”
Critical-path HTTP checks and TCP listener metrics corroborate that the frozen pool is user-impacting, matching the paging criteria rather than alerting on a transient rotation blip.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

The Netdata solution

Apache HTTP Server monitoring with Netdata

Netdata monitors Apache HTTP Server with per-second metrics from mod_status, pre-built dashboards, and ML-powered anomaly detection. Watch busy versus idle workers and the scoreboard state mix, requests per second, bytes served per second, and request processing duration alongside the rest of your stack, so you catch the worker-exhaustion, slow-backend, and memory incidents in these runbooks before they page anyone.

See web server monitoring → Start monitoring free