A 504 on an Apache-proxied path means a gateway in the request chain timed out. There are two cases:

  1. Apache generated the 504: mod_proxy waited longer than the effective proxy timeout for the backend.
  2. The backend generated the 504: Apache received that status and passed it through, often because the backend was proxying to a dead dependency of its own.

The access log alone does not distinguish those cases. Correlate the 504 with Apache’s error log.

In the Apache-generated case, the connection and request often succeeded, but a response read stalled. Do not assume a clean connection from the status code alone. Every stalled proxy operation occupies an Apache worker, usually appearing in W (sending reply) state while Apache waits to receive or stream the response.

With the stock Apache 2.4 TimeOut of 60 seconds and no explicit ProxyTimeout, proxy waits effectively get 60 seconds. That is a long time to hold a worker. When a backend goes slow, workers pile up, IdleWorkers falls to zero, the listen backlog fills, and new connections queue or fail. By the time 504s are steady, worker exhaustion may already be underway.

This article covers how to identify the source of the 504, how ProxyTimeout really behaves, and how to keep backend slowness from consuming the entire worker pool.

What this means

The 504 sits in a specific place in the proxy failure family:

  • 502 Bad Gateway: the backend connection failed or reset, or the backend sent an invalid response.
  • 503 Service Unavailable: the proxy has no eligible backend, a connection-pool acquisition fails, or the balancer has marked workers unavailable. MaxRequestWorkers exhaustion more commonly causes connections to queue or stall than an immediate Apache-generated 503.
  • 504 Gateway Timeout: a proxied wait exceeded the effective timeout.

The timeout case is dangerous for Apache itself. A refused backend connection fails quickly and frees the worker. A timed-out request can hold a worker for the full timeout window. One slow backend can therefore consume the worker pool with modest traffic.

Two properties of ProxyTimeout matter:

  1. It is per I/O operation, not per request. The clock applies to each read or write against the backend and resets when data moves. A backend that sends one byte every 50 seconds will never trip a 60-second timeout, even if the full request takes 20 minutes. Those requests appear as extreme %D values, not 504s.
  2. It is not the same directive as the global TimeOut. For proxied paths, the effective response timeout chain is the per-worker timeout= parameter, then ProxyTimeout, then the global TimeOut. Backend connection establishment has a separate per-worker connectiontimeout= parameter. Raising TimeOut to fix 504s often changes more client-side behavior than intended.
flowchart TD
  A[Backend slows down] --> B[Proxied requests hold Apache workers]
  B --> C[Proxy timeout expires - Apache returns 504]
  B --> D[IdleWorkers falls toward zero]
  D --> E[Listen backlog Recv-Q grows]
  E --> F[New connections queue or fail]
  D --> G[AH00484 MaxRequestWorkers reached]
  F --> H[LB health checks fail - site appears down]
  G --> H

Common causes

CauseWhat it looks likeFirst thing to check
Backend application overload or deadlockScoreboard filling with W, backend latency many times baseline, Apache CPU often lowCurl a known read-only backend URL directly, bypassing Apache
Backend database lock contention or slow queries504s concentrated on data-heavy endpoints; cheap endpoints remain fastBackend query, lock, and connection-pool metrics
Backend GC pause or memory exhaustionPeriodic 504 bursts aligned with backend pausesBackend process RSS and GC logs
Network loss between Apache and backend504s mixed with proxy connection errors, intermittent patternError log for proxy failures; ping or mtr to the backend
Backend’s own downstream dependency timed outBackend returns 504s on paths that call one dependencyWhether Apache logged a proxy timeout or merely passed the backend’s 504 through
Proxy timeout too low for a legitimately slow endpoint504s only on known-expensive endpoints such as reports or exportsEndpoint P99 compared with the effective proxy timeout

Quick checks

These checks do not change Apache state. Paths shown are Debian-style; on RHEL use /var/log/httpd/ and /etc/httpd/. Checks 2 and 3 require mod_status to be enabled and permitted from localhost.

# 1. Count recent 5xx responses, assuming the standard combined log fields
tail -5000 /var/log/apache2/access.log | awk '$9 ~ /^5/ {print $9}' | sort | uniq -c

# 2. Scoreboard state distribution: many W states often means workers waiting on backends
curl -s 'http://localhost/server-status?auto' | grep '^Scoreboard:' | \
  awk '{print $2}' | fold -w1 | sort | uniq -c | sort -nr

# 3. Worker utilization
curl -s 'http://localhost/server-status?auto' | grep -E '^(BusyWorkers|IdleWorkers):'

# 4. Proxy errors, including vhost error logs
grep -hE '\[proxy' /var/log/apache2/*error*.log 2>/dev/null | tail -30

# 5. Test the backend directly, bypassing Apache
# Use a known read-only URL and the Host header expected by the backend.
curl -s -o /dev/null -w "TTFB: %{time_starttransfer}s Total: %{time_total}s Code: %{http_code}\n" \
  --max-time 70 http://backend-host:backend-port/health

# 6. Listen backlog: are new connections queuing?
ss -ltn | grep -E ':80[[:space:]]|:443[[:space:]]'

# 7. Check whether MaxRequestWorkers has been reached
grep -h 'AH00484' /var/log/apache2/*error*.log 2>/dev/null | tail -5

# 8. Find the effective proxy configuration
grep -rE '^[[:space:]]*(ProxyTimeout|ProxyPass|ProxySet)([[:space:]]|$)' /etc/apache2/ 2>/dev/null

Checks 2 and 3 show whether 504s have progressed to worker exhaustion. Check 5 is decisive only if it exercises a representative backend path; a cheap health endpoint can stay fast while the failing API hangs. The 70-second client limit assumes a 60-second proxy timeout, so adjust it after check 8. Check 8 reveals the configured chain: per-worker timeout=, then ProxyTimeout, then TimeOut.

How to diagnose it

  1. Confirm the error class and origin. Run checks 1 and 4. A 504 paired with a mod_proxy timeout error at the same timestamp was generated by Apache. A 504 with no corresponding Apache proxy error may be a backend response passed through unchanged. A mix of 502s and 504s points toward backend crashes or connection failures in addition to slowness.

  2. Look at the scoreboard. Run check 2. A slow-backend cascade usually shows many workers in W, low CPU relative to worker count, and a deceptively normal request rate because older requests are still completing. Many R states instead indicate slow clients or a Slowloris-style pattern. See Apache scoreboard states explained.

  3. Test the backend directly. Run check 5 from the Apache host. Three outcomes matter: the backend is dead, the backend is slow but responds, or the backend is fast. The last case points to the network path, proxy configuration, proxy pool, or a mismatched Host header or request path.

  4. Identify which URLs time out. Group recent 504s by request path:

tail -5000 /var/log/apache2/access.log | \
  awk '$9 == 504 {print $7}' | sort | uniq -c | sort -nr | head -20

If 504s concentrate on expensive endpoints, the backend may be healthy overall and those paths may need more time or asynchronous execution. Uniform 504s across proxied paths indicate global backend degradation.

  1. Check for the trickler case. Pull the slowest completed requests by %D. This assumes %D is the final access-log field; adjust the fields for your LogFormat.
tail -5000 /var/log/apache2/access.log | \
  awk '{print $NF, $9, $7}' | sort -rn | head -20

Completed 200 responses lasting many minutes indicate a backend that trickles data and may never trip ProxyTimeout. Those requests still hold workers the entire time.

  1. Check whether the cascade has reached exhaustion. Run checks 3, 6, and 7. IdleWorkers at zero, a growing Recv-Q, or AH00484 means the slow-backend incident has become an Apache saturation incident. See Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
504 rateDirect symptom of a gateway timeoutAny sustained non-zero rate
Scoreboard W countProxy waits commonly appear as W; shows the pile-up formingW above 50% of workers for several minutes
IdleWorkersRemaining capacity before queuing startsApproaching zero
Backend response time, probed directlySeparates backend slowness from Apache slownessP95 above twice baseline
Listen backlog Recv-QConnections are queuing because workers are unavailableSustained non-zero and growing
AH00484MaxRequestWorkers has been reachedAny occurrence
Proxy errors such as AH00898 or AH01114Distinguishes backend read failures from connection failuresRising rate alongside 504s
Request duration P99 from %DFinds trickling requests that never time outCompleted requests lasting minutes

Fixes

The backend is slow or hung

Fix the backend. No Apache-side timeout can make an unresponsive application healthy. While that work is underway:

  • Lower ProxyTimeout temporarily to fail fast. A worker held for 60 seconds serves nothing; releasing it sooner preserves capacity for healthy requests. Expect a higher 504 rate and failures for legitimately long requests while this is active. A configuration reload is required, and waits already in progress may continue under their original timeout.
  • Remove the Apache instance from load balancer rotation if other instances have healthy backends and the load balancer supports draining. Do not leave one instance accumulating held workers until its health checks fail.
  • Isolate expensive endpoints behind a separate ProxyPass worker and timeout if the backend serves both fast API paths and slow reports. This prevents one endpoint class from starving the rest.

A legitimately slow endpoint exceeds the timeout

If a report or export legitimately needs more time and the backend is healthy:

  • Set the per-worker timeout= parameter for that ProxyPass target above the endpoint’s P99. Use ProxyTimeout when the setting should apply to the whole server or virtual host. Avoid raising global TimeOut unless you also intend to change client-side timeouts.
  • Budget workers for concurrent slow requests. Ten simultaneous 90-second requests hold ten workers. Size memory according to the active MPM’s actual process and thread model; on worker or event MPMs, do not multiply MaxRequestWorkers by per-process RSS because threads share an address space. See Apache MaxRequestWorkers tuning.
  • Make the endpoint asynchronous when possible. Submit the job, return quickly, and poll for completion so no Apache worker remains held.

The backend trickles data and never times out

ProxyTimeout cannot enforce a total request deadline because every successful read resets the per-I/O clock.

  • Fix the backend to buffer and send complete responses where possible.
  • Enforce a total request deadline in the application or framework.
  • Detect outliers through %D and account for their worker holds in pool sizing.

Worker exhaustion has already happened

If IdleWorkers is zero and AH00484 has fired, do not raise MaxRequestWorkers first. More workers against a hung backend creates more waiting workers and more memory pressure. Reduce the timeout, remove the instance from rotation if appropriate, fix the backend, and then revisit worker sizing. For the exhaustion side of the incident, see Apache AH00484: server reached MaxRequestWorkers setting.

Prevention

  • Monitor backend latency separately from Apache health. From outside, an Apache outage and a backend outage look identical. A direct backend probe from the Apache host is the fastest discriminator.
  • Alert on scoreboard trends, not only status codes. Rising W states with flat CPU are the early stage of the cascade. Waiting for 504s means workers have already been held for a full timeout each.
  • Log %D, not only %T. Microsecond resolution catches a backend drifting from 200 ms to 5 seconds before users see 504s.
  • Set proxy timeouts deliberately. Use per-worker timeout= and connectiontimeout= where backends have different latency and connection profiles. Do not tune global TimeOut when the intent is only to tune proxy behavior.
  • Health-check a lightweight proxied path. A static-file check can pass while every proxied request times out. Monitoring should exercise the same proxy path users depend on.

How Netdata helps

  • Netdata’s Apache collector charts the scoreboard, BusyWorkers, and IdleWorkers continuously, making the worker pile-up visible before AH00484 fires.
  • With access-log collection enabled, status-code and request-duration distributions help separate Apache-generated 504s from slow completed responses.
  • Apache metrics can be correlated with backend process, database, and service metrics on the same dashboard, shortening the “Apache or backend?” decision.
  • Alerts on worker utilization and proxy error rates can fire while the cascade is still recoverable.

Apache HTTP Server monitoring with Netdata brings these signals together for incident correlation.