Apache 5xx error rate: 500 vs 502 vs 503 vs 504 and what each one means

Your Apache 5xx error rate just spiked. The status code is the first fork in the diagnostic path, and it is a sharp one: a 500 means the failure happened inside Apache or a module, while 502, 503, and 504 are mod_proxy telling you something about the relationship between Apache and a backend. Treating them as one bucket of “server errors” wastes the best lead you have.

There is a second trap. A sustained 5xx rate almost always means something real is broken, because 5xx is server-side failure by definition. But the reverse is not true: a proxied application that returns error pages with HTTP 200 is invisible to status-based monitoring. Low 5xx is necessary, not sufficient, evidence of health.

This guide covers what each code means mechanically, how to confirm which failure you have from logs and mod_status, and how to alert on the rate without drowning in crawler noise.

What this means

Apache returns 5xx when the server side of the request failed. The four codes you will see in practice split into two families:

• 500 is local. Something inside this Apache process failed: a module crashed, a CGI script died, a handler threw, or the configuration is broken for that request path.
• 502, 503, and 504 are proxy codes. They only appear when Apache is proxying (mod_proxy_http, mod_proxy_fcgi, mod_proxy_balancer). They describe three different ways the backend relationship failed: the backend said something invalid (502), Apache could not get a backend at all (503), or the backend connected but never answered in time (504).

The failure cascades also differ. A 502 storm usually tracks one broken backend. A 503 storm is often Apache-side saturation: all workers busy, or the proxy connection pool exhausted. A 504 storm is the classic slow-backend cascade: workers pile up in the W state waiting on the backend, IdleWorkers drains to zero, the listen backlog fills, and the site goes down while Apache’s own CPU and memory look fine.

flowchart TD
  A[5xx in access log] --> B{Which code?}
  B -->|500| C[Local failure: module, CGI, handler, config]
  B -->|502| D[Backend responded with invalid data or refused connection]
  B -->|503| E{Workers or proxy pool?}
  B -->|504| F[Backend connected but timed out - ProxyTimeout]
  E -->|AH00484 in error log| G[Worker pool exhausted]
  E -->|backend down or pool full| H[Proxy pool / balancer member error]
  D --> I[Check backend directly, check AH01114 / AH00898]
  F --> J[Check backend response time, scoreboard W states]

What each code means in production

Two distinctions worth internalizing:

503 has two very different causes with the same status code. The access log cannot tell them apart. “All Apache workers busy” (AH00484, server reached MaxRequestWorkers) is Apache-side saturation. “Proxy pool exhausted or backend marked down” is a backend connectivity problem. The error log tells you which one you have. Note also that mod_proxy keeps a failed backend in error state for its retry interval (default 60 seconds), so a brief backend blip produces a tail of 503s after the backend has recovered.

504 is about ProxyTimeout, not about how slow the backend is allowed to be. ProxyTimeout defaults to the value of the core Timeout directive (60 seconds) if not set explicitly. The core Timeout applies per I/O operation, not to the total request, so a backend that trickles data continuously can hold a worker for minutes without ever triggering a timeout. The 504s you do see are backends that went completely silent.

Quick checks

All read-only. Run these before changing anything.

# 5xx rate and breakdown by code, last 1000 requests
tail -1000 /var/log/apache2/access.log | awk '
  $9 ~ /^5/ {e[$9]++} END {
    for (c in e) printf "%s: %d (%.2f%%)\n", c, e[c], e[c]/NR*100
  }' | sort

This is the single most important first command. The code distribution tells you which family of failure you are in. (On RHEL the log is /var/log/httpd/access_log.)

# Worker exhaustion confirmation
grep "AH00484" /var/log/apache2/error.log | tail -5

Any occurrence of “server reached MaxRequestWorkers setting” is definitive: the 503s are Apache-side worker saturation.

# Proxy-side errors: connection failures, bad backend responses, dispatch failures
grep -E "AH01114|AH00898|AH01075" /var/log/apache2/error.log | tail -20

AH01114 is a failed connection to the backend (502 territory). AH00898 is an error reading the status line from the backend (502). AH01075 “Error dispatching request” on mod_proxy_fcgi typically pairs with 503/504 when the PHP-FPM pool is saturated.

# Current worker utilization
curl -s http://localhost/server-status?auto | grep -E "BusyWorkers|IdleWorkers"

# Scoreboard state distribution: where are workers stuck?
curl -s http://localhost/server-status?auto | grep "Scoreboard:" | \
  awk '{print $2}' | fold -w1 | sort | uniq -c | sort -nr

Many W states during a 504 storm confirms workers waiting on backends. On Event MPM, significant K states are abnormal since keepalive connections should not occupy workers; on prefork/worker they point at keepalive hoarding, which also produces 503s under load.

# Is the listen backlog filling? (worker saturation corroborator)
ss -ltn | grep -E ':80\s|:443\s'

Sustained non-zero Recv-Q means connections are arriving faster than Apache accepts them. That is the runway to connection refused.

# Backend member state, if using mod_proxy_balancer
curl -s http://localhost/balancer-manager 2>/dev/null | grep -E 'Worker|Status'

# Bypass Apache: is the backend itself healthy and fast?
curl -s -o /dev/null -w "code=%{http_code} ttfb=%{time_starttransfer}s\n" \
  --max-time 10 http://backend-host:backend-port/health

How to diagnose it

1. Split the 5xx count by status code (first quick check above). This chooses the branch: 500 goes local, the rest go to the proxy path.

2. For 500s, go straight to the error log. Every 500 should have a corresponding error log entry at the same timestamp with the actual cause: script error, permission denied, segfault, module exception. The access log only tells you that it happened, not why. If the 500s started right after a deploy or config change, diff that change first. Also check whether mod_security is in play: it can return 403 or 500 for blocked requests, which is policy working as intended, not an application failure.

3. For 503s, decide: worker pool or proxy pool. Grep for AH00484. If present, you have worker exhaustion: check BusyWorkers against your configured MaxRequestWorkers, look at the scoreboard to see what workers are stuck on, and check the listen backlog. If AH00484 is absent, look at the proxy side: balancer member in error state, backend connection failures in the error log, or a proxy connection pool that is too small for the request rate. The default proxy pool max per child is small (ThreadsPerChild, which is 1 on prefork) and is a classic cause of 503s under moderate load while MaxRequestWorkers looks fine.

4. For 502s, test the backend directly. Curl the backend’s health endpoint bypassing Apache. Connection refused or an immediate error means the backend process is down or crashed mid-response. If the backend answers cleanly on its own but Apache still 502s, suspect protocol-level breakage: the backend sending invalid response headers or a malformed status line (AH00898 in the error log).

5. For 504s, measure backend latency and watch the scoreboard. Compare backend response time against baseline. If the backend is slow and the scoreboard is filling with W states while IdleWorkers drains, you are in the slow-backend cascade: Apache is healthy but being strangled. Check CPU and memory on the Apache host: normal values with high latency is the signature. See Apache backend response time: telling ‘Apache is slow’ from ’the backend is slow’.

6. Correlate the timeline. Did the 5xx spike start with a deploy, a traffic event, or a backend change? A sudden 5xx spike after a config change is a bad change until proven otherwise.

Metrics and signals to monitor

Two logging details matter for all of this:

• Log %>s, not %s. %s records the original status before internal redirects; %>s records the final status sent to the client. For user-facing error rates, %>s is the one that matches what users experienced. If ErrorDocument is in play, %s and %>s can disagree.
• Latency fields: use %D (microseconds) or %{ms}T, never %T. %T rounds to whole seconds and makes sub-second latency monitoring useless. Remember that %D includes client transfer time, so a slow client on a big download inflates it.

Fixes

Grouped by cause. None of these start with “restart Apache.”

500: local failures. Fix what the error log names. Module or mod_php segfaults: consider moving PHP to PHP-FPM, which isolates crashes from Apache workers. Misconfiguration: run apachectl configtest and check what changed. If a graceful reload recently failed, you may be running a stale config while thinking the new one applied.

502: invalid backend responses. Fix the backend: it is crashing mid-response or speaking a broken protocol. On the Apache side, verify the proxy is pointing where you think it is and that the backend process is supervised. If the backend intermittently sends bad headers, that is a backend bug to fix, not an Apache knob to tune.

503: worker exhaustion. Do not just raise MaxRequestWorkers blindly: worker memory multiplies. Derive the ceiling from available memory / per-child RSS (see Apache MaxRequestWorkers tuning: sizing the worker pool against memory). First check what workers are stuck on. If keepalive connections are hoarding workers on prefork or worker MPM, lowering KeepAliveTimeout or moving to Event MPM buys real capacity. If workers are stuck on a slow backend, fixing the backend is the fix; raising worker count just lets more workers wait.

503: proxy pool or balancer member error. Size the proxy pool for your actual concurrency: the per-child max times the number of child processes is your total backend connection budget, and the defaults are small. If a balancer member flaps into error state, check its retry interval and failure thresholds, and see Apache balancer member in error state: reading balancer-manager and failover.

504: backend timeout. The durable fix is backend latency. As a mitigation, a shorter ProxyTimeout makes Apache fail fast and release workers instead of holding them for 60 seconds each, which contains the cascade. Tradeoff: too short and slow-but-successful requests start failing. Dropping ProxyTimeout is a pressure valve, not a repair.

Prevention

• Alert on rate, not count. 5xx as a percentage of total requests, per status code. Under 0.1% is healthy; sustained above 1% pages someone. A flat count alert either sleeps through quiet hours or screams during peak.
• Baseline crawler and bot 4xx separately so background noise does not desensitize the team to the error stream. 404 floods from scanners are normal; treat them as a different metric from 5xx entirely.
• Close the 200-shaped hole. Status-code monitoring cannot see an application that renders error pages with HTTP 200. Add a synthetic check that validates response content on the critical path, and make your health check exercise the real path (through the proxy to the backend), not a static file.
• Watch the leading indicators: BusyWorkers utilization, listen backlog, and backend response time trend upward before the 5xx rate moves. Alerting only on 5xx means you find out at the same time your users do.
• After any deploy or config change, watch the 5xx mix for 15 minutes. A shift in the code distribution is the fastest bad-deploy detector you have.

How Netdata helps

• Netdata’s Apache collector reads mod_status continuously, so BusyWorkers, IdleWorkers, request rate, and the scoreboard are time-series data, not snapshots you happened to catch during the incident.
• Correlating the 5xx rate against worker utilization and request rate on one dashboard is what separates the two 503 causes quickly: 503s with maxed workers is exhaustion, 503s with idle workers is the proxy/backend path.
• Backend response time next to the scoreboard W-state trend makes the slow-backend cascade visible as it develops, before 504s start.
• Error log monitoring surfaces AH00484 and proxy error codes as they happen, which is the corroboration that turns “5xx is elevated” into a named cause.
• Per-second collection catches short 5xx bursts tied to deploys and graceful restarts that minute-level polling smooths away.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• Apache 502 Bad Gateway: a backend that returned an invalid response
• Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion
• Apache 504 Gateway Timeout: slow backends, ProxyTimeout, and worker pile-up
• Apache backend response time: telling ‘Apache is slow’ from ’the backend is slow’
• Apache balancer member in error state: reading balancer-manager and failover
• Apache BusyWorkers and IdleWorkers: reading worker utilization from mod_status
• Apache CLOSE_WAIT and TIME_WAIT: connection leaks versus normal churn
• How Apache HTTPD actually works in production: a mental model for operators
• Apache keepalive consuming workers: KeepAliveTimeout, the K state, and MPM choice
• Apache listen queue overflow: Recv-Q growth, ListenBacklog, and refused connections
• Apache AH00484: server reached MaxRequestWorkers setting - worker pool exhausted
• Apache MaxRequestWorkers tuning: sizing the worker pool against memory