Apache log rotation losing lines: logrotate copytruncate vs graceful reopen

You notice it after the fact: the access log has a gap. Requests you know happened, because the application processed them and the client got a response, never appear in any log file. Not in the current log, not in the rotated one. The gap lines up with the moment logrotate ran.

Or the opposite symptom: after rotation, Apache keeps writing to the old file. access.log.1 grows for days while access.log stays empty. Or every night at rotation time you see a blip of dropped connections and a spike of 499s or client resets, because the postrotate script does a hard restart instead of a graceful one.

All three symptoms have the same root: rotation acting on files Apache holds open. Apache keeps log file descriptors open across requests. Whatever you do to the file underneath those descriptors, you have to tell Apache about it, or accept that something gets lost.

What this means

Apache opens its access and error logs at startup (or reload) and keeps the descriptors open in the parent and every child. Writes go to the descriptor, not to the path. Three consequences:

• copytruncate races the writer. logrotate’s copytruncate mode copies the live file, then truncates the original in place. Between the copy finishing and the truncate executing, Apache keeps appending. Those lines are wiped out by the truncate. The logrotate documentation itself warns that data can be lost in this window. At high request rates the window is small but the loss is real: every line written in that interval is gone.

• Rename without reopen strands the descriptor. If logrotate renames access.log to access.log.1 and creates a fresh access.log, Apache’s children still hold a descriptor to the renamed inode. They keep writing to access.log.1 indefinitely, and the new file stays empty until something makes Apache reopen its logs. Compression of the rotated file then destroys data that is still being written.

• SIGHUP is a hard restart. Some distro-shipped or hand-rolled rotation configs signal Apache with kill -HUP or call apachectl restart in postrotate. SIGHUP makes the parent kill all children immediately and re-exec: every in-flight connection is dropped. The correct signal for reopening logs without dropping traffic is SIGUSR1, which is what apachectl graceful sends.

Bottom line: copytruncate races Apache’s writes and loses lines. The correct approaches are rename plus SIGUSR1, or piped logging with rotatelogs.

Common causes

Quick checks

All read-only. Run on the Apache host.

# 1. Find which rotation method is configured
grep -rE 'copytruncate|postrotate|sharedscripts' /etc/logrotate.d/ /etc/logrotate.conf 2>/dev/null

# 2. See exactly what the postrotate script does
cat /etc/logrotate.d/apache2 2>/dev/null || cat /etc/logrotate.d/httpd 2>/dev/null

# 3. Confirm Apache is writing to the file you think it is
#    (if this shows access.log.1 or a deleted file, the reopen never happened)
ls -l /proc/$(pgrep -o 'httpd|apache2' | head -1)/fd 2>/dev/null | grep -i log

# 4. Dry-run logrotate to see what it would do, without doing it
logrotate -d /etc/logrotate.conf 2>&1 | grep -A20 -i 'apache\|httpd'

# 5. Check restart history around the last rotation
grep -E "resuming normal operations|caught SIGTERM|graceful restart" \
  /var/log/apache2/error.log /var/log/httpd/error_log 2>/dev/null | tail -20

# 6. Look for a timestamp gap in the access log around rotation time
#    (compare the last line of the rotated file with the first line of the new one)
tail -2 /var/log/apache2/access.log.1 2>/dev/null
head -2 /var/log/apache2/access.log 2>/dev/null

# 7. If using piped logging, confirm the pipe processes are alive
pgrep -af rotatelogs

# 8. Check scoreboard for stuck Logging or Graceful states
curl -s http://localhost/server-status?auto | grep "Scoreboard:" | \
  awk '{print $2}' | fold -w1 | sort | uniq -c | sort -nr

Check 3 is the single most diagnostic one. If the parent and children have the rotated (or deleted) file open, every other fix is cosmetic until Apache reopens its logs.

How to diagnose it

1. Establish which symptom you have. Missing lines around the rotation instant points at copytruncate. A rotated file that keeps growing points at rename-without-reopen. Dropped connections at rotation time points at SIGHUP in postrotate. They are frequently mixed on hosts where the config evolved over years.

2. Read the actual logrotate config for Apache. Do not trust memory. Distributions ship different defaults, and old configs survive upgrades. Look for copytruncate, and look at what postrotate runs: apachectl graceful, systemctl reload httpd, kill -HUP, or invoke-rc.d apache2 reload. The first two are graceful (SIGUSR1) on standard distro units; kill -HUP is a hard restart. If your unit files or init scripts are customized, verify what reload actually does instead of assuming.

3. Verify the reopen happened. After the next rotation, run check 3 on several children, not just the parent. If any child still holds the old file, the graceful signal did not reach Apache. Common reasons: the postrotate command fails silently (wrong PID file path, permissions, the cron job running in an environment where apachectl is not on PATH), or old-generation children still have the old file open.

4. Quantify the loss. Compare the last timestamp of the rotated file with the first timestamp of the new file, and check whether requests you can prove happened (application logs, client records) appear anywhere. Missing lines plus copytruncate in the config confirms the race. Note that delaycompress does not help here: it only postpones compression, it does not close the copy-then-truncate window.

5. Check for restart pile-up side effects. If rotation triggers a graceful restart under load, old and new children overlap. Look for many G (gracefully finishing) states in the scoreboard and multiple resuming normal operations lines close together. If you have several logrotate blocks for different Apache logs (per-vhost files), each may fire its own reload; consolidate with sharedscripts so postrotate runs once per cycle.

6. If logs stopped entirely, check the pipe. With CustomLog "|/usr/sbin/rotatelogs ...", a dead rotatelogs process means writes go to a broken pipe. Workers can then stall in the L (Logging) scoreboard state, which is the leading edge of the log-stall failure mode.

Metrics and signals to monitor

Fixes

Replace copytruncate with rename plus graceful reopen

The standard fix. Remove copytruncate and add a postrotate block that gracefully restarts Apache so it reopens its logs:

# /etc/logrotate.d/apache2 (or httpd) - pattern, adapt paths to your distro
/var/log/apache2/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    sharedscripts
    postrotate
        if [ -f /var/run/apache2/apache2.pid ]; then
            apachectl graceful
        fi
    endscript
}

Key points:

• sharedscripts matters when the block matches multiple files (several vhosts). Without it, postrotate runs once per matched file and you get N graceful restarts back to back.
• Test the postrotate command by hand as the same user logrotate runs as. Silent postrotate failure is the most common reason “we switched to graceful and it still writes to the old file.”
• Give old children time before compressing. After SIGUSR1, old-generation children finish in-flight requests against the old descriptor. Apache’s own guidance is that a rotation script cannot know when all children have finished writing the pre-restart log, so you need a delay before touching the old file. delaycompress covers this in practice; if you compress immediately in postrotate, add a sleep. Bound the linger with GracefulShutdownTimeout (for example 30s) so stuck requests cannot hold the old file open indefinitely.
• apachectl graceful does not always validate config first (distribution-dependent). A graceful that hits a config error can leave new children failing to start while old ones keep the old config. Run apachectl configtest in your change process, and watch the error log after reloads.

If you are stuck in an environment where signaling Apache is impossible, copytruncate is not the only copy-based mode. The safe move remains: signal Apache, or pipe the logs.

Switch to piped logging with rotatelogs

Piped logging removes logrotate from the write path entirely. Apache writes to a pipe; a per-log rotatelogs process handles rotation:

# Time-based rotation, no signals or cron needed
CustomLog "|/usr/sbin/rotatelogs -l /var/log/apache2/access.%Y-%m-%d.log 86400" combined
ErrorLog  "|/usr/sbin/rotatelogs -l /var/log/apache2/error.%Y-%m-%d.log 86400"

Tradeoffs:

• One process per piped log directive. Many vhosts each with piped access and error logs means many rotatelogs processes. Usually fine, but not free.
• Do not share one rotatelogs instance across vhosts unless you accept interleaved lines. Separate instances per log are safe.
• The pipe is a new failure mode. If rotatelogs dies (disk full, killed), children write to a broken pipe and can stall in L state or die on SIGPIPE. Monitor pgrep -af rotatelogs and the scoreboard L count.
• Compression needs -p (a post-rotation program) or an external job that only touches files rotatelogs has already closed. Never compress the file rotatelogs currently has open.

Fix a hard-restart postrotate

If postrotate runs kill -HUP, apachectl restart, or service httpd restart, replace it with apachectl graceful or systemctl reload httpd (verify your unit’s reload sends SIGUSR1). SIGHUP drops every in-flight connection; on a busy server that is a nightly micro-outage showing up as client retries and 499s. It also resets SSL session caches, causing a CPU burst of full handshakes right after rotation.

Choosing the right approach:

flowchart TD
    A[Need to rotate Apache logs] --> B{Can rotation scripts signal Apache?}
    B -->|yes| C[logrotate: rename + postrotate apachectl graceful]
    B -->|no| D[Piped logging with rotatelogs]
    C --> E{Multiple log files matched?}
    E -->|yes| F[Add sharedscripts so graceful runs once]
    E -->|no| G[Delay compression until old children drain]
    F --> G
    D --> H[Monitor rotatelogs process and scoreboard L state]
    A -.->|never| I[copytruncate at production write rates]
    A -.->|never| J[kill -HUP in postrotate]

Prevention

• Standardize on one rotation strategy per host. Mixed configs (one vhost block with copytruncate, another with graceful) guarantee someone misdiagnoses the next gap.
• Alert on log write gaps, not just disk space. A check that the access log’s mtime is fresh during known traffic catches both the race and a dead pipe.
• Watch rotation as an event. Correlate the error log (resuming normal operations), scoreboard G and L states, and connection resets around the rotation minute. Rotation should be boring; if it shows up in any signal, the config is wrong.
• Keep logs on their own filesystem. Rotation is your main defense against a full log disk, and a full log disk produces the log-stall failure mode: workers blocked in L, throughput collapsing while the process looks alive.
• Force-rotate manually after fixing the config (logrotate -f /etc/logrotate.d/apache2) during a quiet window to prove the new postrotate works end to end, then verify open FDs on several children. Forcing rotation briefly overlaps old and new children; on a memory-tight server, do it off-peak.

How Netdata helps

• Log continuity as a signal. Netdata’s web log collector parses Apache access logs continuously, so a rotation gap or a stranded descriptor shows up immediately as a drop in parsed requests per second while traffic is unchanged.
• Scoreboard correlation. BusyWorkers, IdleWorkers, and worker state distribution are collected per second, so G pile-ups and L stalls around the rotation minute are visible instead of anecdotal.
• Restart detection. Uptime and restart-event tracking makes it obvious when rotation is triggering hard restarts or repeated graceful restarts back to back.
• Error spike alignment. HTTP 5xx and connection-level metrics on the same timeline as rotation confirm or rule out a hard-restart postrotate in one look.
• Disk and FD headroom. Log filesystem usage and per-process file descriptor counts are collected alongside Apache metrics, so you catch the “log disk filling because rotation silently stopped working” case before the log-stall failure.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• Apache error log monitoring: severity levels, AH codes, and what to alert on
• Apache BusyWorkers and IdleWorkers: reading worker utilization from mod_status
• Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion
• Apache CPU saturation: TLS handshakes, mod_deflate, mod_rewrite, and mod_security
• Apache 5xx error rate: 500 vs 502 vs 503 vs 504 and what each one means
• Apache backend response time: telling ‘Apache is slow’ from ’the backend is slow’