Apache 5xx error rate: 500 vs 502 vs 503 vs 504 and what each one means

Your Apache 5xx error rate just spiked. The status code is the first fork in the diagnostic path, and it is a sharp one: a 500 means the failure happened inside Apache or a module, while 502, 503, and 504 are mod_proxy telling you something about the relationship between Apache and a backend. Treating them as one bucket of “server errors” wastes the best lead you have.

There is a second trap. A sustained 5xx rate almost always means something real is broken, because 5xx is server-side failure by definition. But the reverse is not true: a proxied application that returns error pages with HTTP 200 is invisible to status-based monitoring. Low 5xx is necessary, not sufficient, evidence of health.

This guide covers what each code means mechanically, how to confirm which failure you have from logs and mod_status, and how to alert on the rate without drowning in crawler noise.

What this means

Apache returns 5xx when the server side of the request failed. The four codes you will see in practice split into two families:

• 500 is local. Something inside this Apache process failed: a module crashed, a CGI script died, a handler threw, or the configuration is broken for that request path.
• 502, 503, and 504 are proxy codes. 502 and 504 are only produced when Apache is proxying; in a proxied deployment, 503 is too (mod_proxy_http, mod_proxy_fcgi, mod_proxy_balancer). They describe three different ways the backend relationship failed: the backend said something invalid (502), Apache could not get a backend at all (503), or the backend connected but never answered in time (504).

The failure cascades also differ. A 502 storm usually tracks one broken backend. A 503 storm is often Apache-side saturation: all workers busy, or the proxy connection pool exhausted. A 504 storm is the classic slow-backend cascade: workers pile up in the W state waiting on the backend, IdleWorkers drains to zero, the listen backlog fills, and the site goes down while Apache’s own CPU and memory look fine.

flowchart TD
  A[5xx in access log] --> B{Which code?}
  B -->|500| C[Local failure: module, CGI, handler, config]
  B -->|502| D[Backend responded with invalid data]
  B -->|503| E{Which proxy-side cause?}
  B -->|504| F[Backend connected but timed out - ProxyTimeout]
  E -->|AH00484 in error log| G[Worker pool saturated - excess connections queue]
  E -->|backend down or pool full| H[Proxy pool / balancer member error]
  D --> I[Check backend directly, check AH01102 / AH00898]
  F --> J[Check backend response time, scoreboard W states]

What each code means in production

Two distinctions worth internalizing:

503 is proxy-side, and it often coexists with worker saturation. The 503s themselves come from mod_proxy: pool exhaustion or a balancer member in error. Apache-side worker saturation (AH00484, server reached MaxRequestWorkers) is a different mechanism: excess connections queue in the listen backlog rather than getting 503 answers, which shows up as hangs and 504s. Both can be present during one incident, and the error log tells you which one you have. Note also that mod_proxy keeps a failed backend in error state for its retry interval (default 60 seconds), so a brief backend blip produces a tail of 503s after the backend has recovered.

504 is about ProxyTimeout, not about how slow the backend is allowed to be. ProxyTimeout defaults to the value of the core Timeout directive (60 seconds) if not set explicitly, and like Timeout it is a socket timeout: it applies to each I/O operation on the backend connection, not to the whole proxied request. The core Timeout applies per I/O operation, not to the total request, so a backend that trickles data continuously can hold a worker for minutes without ever triggering a timeout. The 504s you do see are backends that went completely silent.

Quick checks

All read-only. Run these before changing anything.

# 5xx rate and breakdown by code, last 1000 requests
tail -1000 /var/log/apache2/access.log | awk '
  $9 ~ /^5/ {e[$9]++} END {
    for (c in e) printf "%s: %d (%.2f%%)\n", c, e[c], e[c]/NR*100
  }' | sort

This is the single most important first command. The code distribution tells you which family of failure you are in. (On RHEL the log is /var/log/httpd/access_log.)

# Worker exhaustion confirmation
grep "AH00484" /var/log/apache2/error.log | tail -5

Any occurrence of “server reached MaxRequestWorkers setting” is definitive: the 503s are Apache-side worker saturation.

# Proxy-side errors: connection failures, bad backend responses, dispatch failures
grep -E "AH01114|AH00898|AH01075" /var/log/apache2/error.log | tail -20

AH01114 is a failed connection to the backend, which returns 503, not 502. AH01102 is an unreadable status line from the backend and AH00898 a read failure mid-response; both return 502. AH01075 “Error dispatching request” on mod_proxy_fcgi typically pairs with 503/504 when the PHP-FPM pool is saturated.

# Current worker utilization
curl -s http://localhost/server-status?auto | grep -E "BusyWorkers|IdleWorkers"

# Scoreboard state distribution: where are workers stuck?
curl -s http://localhost/server-status?auto | grep "Scoreboard:" | \
  awk '{print $2}' | fold -w1 | sort | uniq -c | sort -nr

Many W states during a 504 storm confirms workers waiting on backends. On Event MPM, significant K states are abnormal since keepalive connections should not occupy workers; on prefork/worker they point at keepalive hoarding, which also produces 503s under load.

# Is the listen backlog filling? (worker saturation corroborator)
ss -ltn | grep -E ':80\s|:443\s'

Sustained non-zero Recv-Q means connections are arriving faster than Apache accepts them. That is the runway to connection refused.

# Backend member state, if using mod_proxy_balancer
curl -s http://localhost/balancer-manager 2>/dev/null | grep -E 'Worker|Status'

# Bypass Apache: is the backend itself healthy and fast?
curl -s -o /dev/null -w "code=%{http_code} ttfb=%{time_starttransfer}s\n" \
  --max-time 10 http://backend-host:backend-port/health

How to diagnose it

1. Split the 5xx count by status code (first quick check above). This chooses the branch: 500 goes local, the rest go to the proxy path.

2. For 500s, go straight to the error log. Every 500 should have a corresponding error log entry at the same timestamp with the actual cause: script error, permission denied, segfault, module exception. The access log only tells you that it happened, not why. If the 500s started right after a deploy or config change, diff that change first. Also check whether mod_security is in play: it can return 403 or 500 for blocked requests, which is policy working as intended, not an application failure.

3. For 503s, look at the proxy path first, then check for compounding saturation. The 503s come from mod_proxy: balancer member in error state, backend connection failures in the error log, or a proxy connection pool that is too small for the request rate. Grep for AH00484 as well: if present, workers are saturated too. Saturation itself queues excess connections in the listen backlog (hangs and 504s) rather than answering 503, but it makes any proxy-side failure worse and usually needs fixing alongside. The default proxy pool max per child is small (ThreadsPerChild, which is 1 on prefork) and is a classic cause of 503s under moderate load while MaxRequestWorkers looks fine.

4. For 502s, test the backend directly. Curl the backend’s health endpoint bypassing Apache. Connection refused or an immediate error means the backend process is down or crashed mid-response. If the backend answers cleanly on its own but Apache still 502s, suspect protocol-level breakage: the backend sending invalid response headers or a malformed status line (AH00898 in the error log).

5. For 504s, measure backend latency and watch the scoreboard. Compare backend response time against baseline. If the backend is slow and the scoreboard is filling with W states while IdleWorkers drains, you are in the slow-backend cascade: Apache is healthy but being strangled. Check CPU and memory on the Apache host: normal values with high latency is the signature. See Apache backend response time: telling ‘Apache is slow’ from ’the backend is slow’.

6. Correlate the timeline. Did the 5xx spike start with a deploy, a traffic event, or a backend change? A sudden 5xx spike after a config change is a bad change until proven otherwise.

Metrics and signals to monitor

Two logging details matter for all of this:

• Log %>s, not %s. %s records the original status before internal redirects; %>s records the final status sent to the client. For user-facing error rates, %>s is the one that matches what users experienced. If ErrorDocument is in play, %s and %>s can disagree.
• Latency fields: use %D (microseconds) or %{ms}T, never %T. %T rounds to whole seconds and makes sub-second latency monitoring useless. Remember that %D includes client transfer time, so a slow client on a big download inflates it.

Fixes

Grouped by cause. None of these start with “restart Apache.”

500: local failures. Fix what the error log names. Module or mod_php segfaults: consider moving PHP to PHP-FPM, which isolates crashes from Apache workers. Misconfiguration: run apachectl configtest and check what changed. If a graceful reload recently failed, you may be running a stale config while thinking the new one applied.

502: invalid backend responses. Fix the backend: it is crashing mid-response or speaking a broken protocol. On the Apache side, verify the proxy is pointing where you think it is and that the backend process is supervised. If the backend intermittently sends bad headers, that is a backend bug to fix, not an Apache knob to tune.

503: worker exhaustion. Do not just raise MaxRequestWorkers blindly: worker memory multiplies. Derive the ceiling from available memory / per-child RSS (see Apache MaxRequestWorkers tuning: sizing the worker pool against memory). First check what workers are stuck on. If keepalive connections are hoarding workers on prefork or worker MPM, lowering KeepAliveTimeout or moving to Event MPM buys real capacity. If workers are stuck on a slow backend, fixing the backend is the fix; raising worker count just lets more workers wait.

503: proxy pool or balancer member error. Size the proxy pool for your actual concurrency: the per-child max times the number of child processes is your total backend connection budget, and the defaults are small. If a balancer member flaps into error state, check its retry interval and failure thresholds, and see Apache balancer member in error state: reading balancer-manager and failover.

504: backend timeout. The durable fix is backend latency. As a mitigation, a shorter ProxyTimeout makes Apache fail fast and release workers instead of holding them for 60 seconds each, which contains the cascade. Tradeoff: too short and slow-but-successful requests start failing. Dropping ProxyTimeout is a pressure valve, not a repair.

Prevention

• Alert on rate, not count. 5xx as a percentage of total requests, per status code. Under 0.1% is healthy; sustained above 1% pages someone. A flat count alert either sleeps through quiet hours or screams during peak.
• Baseline crawler and bot 4xx separately so background noise does not desensitize the team to the error stream. 404 floods from scanners are normal; treat them as a different metric from 5xx entirely.
• Close the 200-shaped hole. Status-code monitoring cannot see an application that renders error pages with HTTP 200. Add a synthetic check that validates response content on the critical path, and make your health check exercise the real path (through the proxy to the backend), not a static file.
• Watch the leading indicators: BusyWorkers utilization, listen backlog, and backend response time trend upward before the 5xx rate moves. Alerting only on 5xx means you find out at the same time your users do.
• After any deploy or config change, watch the 5xx mix for 15 minutes. A shift in the code distribution is the fastest bad-deploy detector you have.

How Netdata helps

• Netdata’s Apache collector reads mod_status continuously, so BusyWorkers, IdleWorkers, request rate, and the scoreboard are time-series data, not snapshots you happened to catch during the incident.
• Correlating the 5xx rate against worker utilization and request rate on one dashboard is what separates the two 503 causes quickly: 503s with maxed workers is exhaustion, 503s with idle workers is the proxy/backend path.
• Backend response time next to the scoreboard W-state trend makes the slow-backend cascade visible as it develops, before 504s start.
• Error log monitoring surfaces AH00484 and proxy error codes as they happen, which is the corroboration that turns “5xx is elevated” into a named cause.
• Per-second collection catches short 5xx bursts tied to deploys and graceful restarts that minute-level polling smooths away.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• Apache 502 Bad Gateway: a backend that returned an invalid response
• Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion
• Apache 504 Gateway Timeout: slow backends, ProxyTimeout, and worker pile-up
• Apache backend response time: telling ‘Apache is slow’ from ’the backend is slow’
• Apache balancer member in error state: reading balancer-manager and failover
• Apache BusyWorkers and IdleWorkers: reading worker utilization from mod_status
• Apache CLOSE_WAIT and TIME_WAIT: connection leaks versus normal churn
• How Apache HTTPD actually works in production: a mental model for operators
• Apache keepalive consuming workers: KeepAliveTimeout, the K state, and MPM choice
• Apache listen queue overflow: Recv-Q growth, ListenBacklog, and refused connections
• Apache AH00484: server reached MaxRequestWorkers setting - worker pool exhausted
• Apache MaxRequestWorkers tuning: sizing the worker pool against memory