Unlike the 502/503/504 family, which Apache generates while proxying to a broken or exhausted backend, a 500 is generated inside Apache itself or by the handler Apache invoked: a module crashed, a CGI script failed, a rewrite rule looped, an .htaccess directive is invalid, or the server hit a permission problem reaching the content.

That distinction is the diagnostic strategy. For a 502 you look at the backend. For a 500 you look at Apache’s own error log, because every 500 Apache emits has a corresponding error-log line with the real cause. The access log tells you that it happened and which URL triggered it. The error log tells you why.

If your access log shows 502, 503, or 504 instead, you are in proxy territory; see Apache 5xx error rate: 500 vs 502 vs 503 vs 504 for the split.

What this means

Apache’s request pipeline passes every request through phases: URI translation, access control, authentication, content generation by a handler, logging. A 500 means one of those phases failed in a way Apache could not map to a more specific status code. The failure happened on this server, in this process, during this request. There is no upstream to blame.

The useful property of a 500 is that it is never silent. Apache logs the underlying error at [error] level or higher, usually with an AH-prefixed code in 2.4 that identifies the subsystem. Two traps to avoid:

  • The inverse case: applications that render error pages with a 200 status are invisible to status-code monitoring. If users report errors but the access log is clean, suspect the app, not Apache.
  • Default verbosity: at LogLevel warn, some diagnostics logged at info level in 2.4 never reach the log. If you have 500s in the access log but an empty error log, raise verbosity before concluding anything.
flowchart TD
  A[500 in access log] --> B[Find matching error-log line by timestamp and vhost]
  B --> C{What does the line say?}
  C -->|AH code, e.g. AH00124| D[Config or rewrite fault - fix directive]
  C -->|Premature end of script headers| E[CGI or handler failure - fix script or interpreter]
  C -->|Permission denied| F[Filesystem, SELinux, or suexec]
  C -->|exit signal Segmentation fault| G[Module or library crash]
  C -->|Nothing logged| H[LogLevel too quiet - raise to info and reproduce]

Common causes

CauseWhat it looks likeFirst thing to check
Rewrite loop in .htaccess or config500 on specific URLs after a rules change; AH00124: Request exceeded the limit of 10 internal redirectsThe most recently edited RewriteRule; look for [L] in per-directory context
Invalid directive in .htaccess500 on every request under that directory tree, immediately after the file changedAH00670 or a directive not allowed in that context
Old 2.2 access-control syntax on 2.4500 or unexpected denials after an upgrade or config migrationOrder/Allow/Deny instead of Require
Missing module for the configured handler500 with AH01144: No protocol handler was valid for the URLWhether the module the handler needs (e.g. proxy_fcgi_module) is loaded
CGI script failure500 with Premature end of script headersShebang line, execute bit, interpreter path
Permission or SELinux denial500 with (13)Permission denied on files that look readablegetenforce, audit log, file contexts
suexec refusing to run a script500 with Permission denied: exec of ... failedOwnership and writability of the script and its directory
Module or library segfaultChild PIDs dying; child pid X exit signal Segmentation faultdmesg, which module the failing requests share

Quick checks

All read-only, safe during an incident.

# 1. Confirm the 500s and get failing URLs and timestamps
tail -2000 /var/log/apache2/access.log | awk '$9 == 500' | tail -20
# RHEL path: /var/log/httpd/access_log

# 2. Pull the error-log lines for the same window - this is where the cause lives
grep -E "\[error\]|\[crit\]|\[alert\]|\[emerg\]" /var/log/apache2/error.log | tail -30

# 3. Tally the AH codes to identify the subsystem
grep -oE "AH[0-9]{5}" /var/log/apache2/error.log | sort | uniq -c | sort -rn | head

# 4. Check for child crashes
grep -i "segfault\|segmentation\|exit signal" /var/log/apache2/error.log | tail -10
dmesg | grep -i "segfault" | tail -10

# 5. Validate the configuration
apachectl configtest 2>&1

# 6. On SELinux systems (RHEL family), check enforcement and denials
getenforce
ausearch -m avc -ts recent 2>/dev/null | grep -i httpd | tail -10

# 7. Reproduce one failing request locally to confirm it is deterministic
curl -s -o /dev/null -w "%{http_code}\n" http://localhost/the/failing/path

apachectl configtest validates syntax only. It does not prove the server is running, and it does not catch runtime errors such as an .htaccess directive that is syntactically fine but forbidden in that context, or a module that loads but crashes on specific requests. Clean configtest plus live 500s means the fault is at runtime, not parse time.

How to diagnose it

  1. Pin one failing request. From the access log, take a single 500: exact timestamp, vhost, URL, client. Do not start with aggregates; you want one concrete event to trace.

  2. Match it to the error-log line. Search the error log for that timestamp on that vhost. Vhosts can have their own ErrorLog files; if the main error log is silent, check the vhost-specific one. In 2.4 the line carries the module name, severity, PID, client address, and usually an AH code.

  3. If the error log is empty for a confirmed 500, raise verbosity. Set LogLevel warn core:info, or per-module, e.g. LogLevel info rewrite:trace5 for rewrite debugging (replacing the removed 2.2-era RewriteLog). Reload, reproduce, then drop the level back; trace levels are extremely verbose.

  4. Classify by the error line:

    • AH00124 (internal redirect limit exceeded): a rewrite loop. Almost always [L] in .htaccess context, where Apache restarts processing from the top after each pass, or a rule missing a RewriteCond %{REQUEST_FILENAME} !-f guard.
    • AH00670: mod_rewrite refused because neither FollowSymLinks nor SymLinksIfOwnerMatch is enabled in that directory context.
    • AH01144 (no protocol handler valid for the URL): the configured handler needs a module that is not loaded, classically mod_proxy_fcgi for a PHP-FPM setup.
    • Premature end of script headers: a CGI or wrapped script died before emitting valid headers. Check the shebang, the execute bit, and the interpreter path.
    • (13)Permission denied on a file that ls -l says is readable: think SELinux context or suexec policy before Unix mode bits.
    • exit signal Segmentation fault: a module or library crashed the child.

  5. Scope the blast radius. One directory tree points at that tree’s .htaccess. One vhost points at that vhost’s config or handler. Everything starting at a specific time points at a config change, module update, or deployment.

  6. Diff against the last change. 500s that begin suddenly almost always correlate with a change: an edited .htaccess, a config reload, a package update, new content with wrong ownership. Find the last resuming normal operations line in the error log and compare mtimes on recently edited config files.

  7. For segfaults, identify the common module. Which URLs were the dead children serving? Prefork loses only the crashed child; on worker or event MPM a segfault in one thread can take the whole process and its threads, so the impact per crash is larger. Core-dump analysis is covered in the official Apache debugging documentation if the crash is repeatable.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
500 rate as % of requestsThe user-facing symptom; baseline should be near zeroSustained rate above ~0.1%, or a step change after a deploy
Error-log rate at [error] and aboveEvery 500 has a matching line; the rate tracks real fault volume>10 lines/minute for most deployments, or any [crit]/[alert]/[emerg]
Specific AH codes (AH00124, AH00670, AH01144)Identifies the failing subsystem without reading every lineAny appearance of a code you have not seen before
Child segfault eventsModule crashes produce 500s for the in-flight request and hint at exploitable bugsAny segfault in production; multiple per minute degrading service
Configuration reload successA failed graceful reload leaves stale config running silently, a common source of “I fixed it but it still 500s”Reload attempted but no matching resuming normal operations, or configtest failing
Scoreboard / BusyWorkersA flood of 500s that also exhausts workers is worse than a bad directiveIdleWorkers at zero alongside rising 500s

Fixes

Rewrite loops (AH00124)

In per-directory context (.htaccess), [L] only stops the current pass; Apache re-injects the result and runs the ruleset again from the top, which is the loop. Use [END] instead of [L] to terminate rewriting for the request entirely (available since 2.3.9, so present in all 2.4.x). Add existence guards such as RewriteCond %{REQUEST_FILENAME} !-f and !-d so real files and directories bypass the rule. While testing redirect rules, use [R=302], not [R=301]; browsers cache 301s aggressively and will keep sending users to the wrong place after you fix the rule.

Invalid or forbidden .htaccess directives

Not every directive is legal in every context. RewriteMap, for example, can only be declared in server or virtualhost context, never in .htaccess or <Directory>. If the error is AH00670, enable Options FollowSymLinks (or SymLinksIfOwnerMatch) for that directory, because mod_rewrite refuses to operate per-directory without one of them. The durable fix for anything non-trivial is moving the rules into the main config and setting AllowOverride None, which also removes the per-request stat() overhead .htaccess imposes on every directory component of every URL.

Leftover 2.2 access-control syntax

Apache 2.2 is fully end of life. On 2.4, replace Order allow,deny / Allow from all blocks with Require all granted (or the appropriate Require variant). Mixed old and new syntax after an upgrade is a recurring source of 500s and unexpected denials; convert the whole file rather than patching line by line.

Missing handler module (AH01144)

Load the module the handler needs. For a PHP-FPM setup via SetHandler "proxy:fcgi://..." or ProxyPassMatch, that means mod_proxy plus mod_proxy_fcgi; without the submodule, Apache has no protocol handler for the URL and returns 500. Confirm with apachectl -M that proxy_fcgi_module appears.

CGI and suexec failures

The script needs a valid shebang pointing at an existing interpreter, the execute bit set, and (under suexec) ownership by the expected user with no group- or world-writable bits on the script or its directory; suexec refuses to run writable scripts by design. Test the script from the shell as the target user before blaming Apache.

Permission denials and SELinux

If mode bits look correct but the error log says (13)Permission denied, check SELinux. On RHEL-family systems in enforcing mode, restore expected contexts with restorecon -R /var/www/html (adjust for your document root) rather than disabling enforcement. Use audit2allow on the denial only when the access itself is legitimate policy-wise.

Module segfaults

Treat any production segfault as a bug to root-cause, not noise to restart away. Identify the module shared by the crashing requests, check for recent package updates, and confirm you are on a supported 2.4.x release; running an old point release can mean the crash is already patched upstream. Occasional single-child crashes are absorbed by design (the parent respawns the child), but recurring crashes on a request pattern will keep producing 500s for that pattern.

Prevention

  • Alert on the 500 rate and the error-log rate together. The rate is the symptom; the log rate is the cause arriving in real time. Alerting on either alone leaves you blind to half the incident.
  • Run apachectl configtest before every reload, in automation. A graceful reload that fails its config check can leave stale config running while everyone believes the fix is live.
  • Keep trace-level logging ready but off. Know the per-module LogLevel syntax before the incident; turning on rewrite:trace5 during an outage is not the time to learn it.
  • Minimize .htaccess in production. Set AllowOverride None where you can and move rules into the main config. Fewer 500 classes, no per-request filesystem stat tax.
  • Keep current on 2.4.x point releases. Several recent CVEs include crash behavior that surfaces to users as 500s. The 2.2 branch receives nothing.
  • Exercise the real path in health checks. A health check against a static file will pass while the CGI handler 500s on every request. Probe the critical path, not just the port.
  • Fix apps that return 200 for errors. Status-code monitoring will never see an incident rendered as a 200. Return honest status codes.

How Netdata helps

  • Netdata charts the 5xx rate broken down by status code from the access log, so a 500 spike is distinguishable from the 502/503/504 proxy family at a glance, which decides whether you debug Apache or a backend.
  • Error-log rate and severity tracking puts the matching error-log lines next to the 500 spike on one timeline, the correlation this diagnosis depends on.
  • Child process crash and restart events surface alongside request metrics, making segfault-driven 500s visible without grepping dmesg mid-incident.
  • Scoreboard and BusyWorkers metrics show whether a 500 flood is also consuming worker capacity, separating a contained config fault from developing saturation.
  • Per-second collection catches short 500 bursts during deploys that minute-resolution polling averages away.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.