$ guides / apache-httpd / apache-httpd-request-rate-drop ▌

Operations Guides

Apache request rate dropped to near zero: silent load shedding and its causes

Your dashboard shows requests per second falling off a cliff, but nobody changed anything, the load balancer still reports healthy demand, and there is no error spike to point at. The most natural reading, “traffic went away,” is usually wrong.

The trap is in how the number is produced. The request rate most operators watch comes from mod_status Total Accesses, which counts completed requests. It is not a measure of arriving demand. When Apache cannot complete requests, because every worker is stuck or a backend has stopped answering, the completed-request rate collapses even while clients are still hammering the front door. The server has not lost its traffic. It is silently shedding load, and the metric you trust is the last place that shows up.

This guide covers why the counter lies during saturation, how to confirm which of the four usual causes you are dealing with, and what to do about each one.

What this means

A request travels a short path before it shows up in your RPS graph: the kernel accepts the TCP connection into the listen backlog, an Apache worker takes it off the queue, the request is processed (possibly via a backend), the response is sent, and only then is the request counted in Total Accesses and written to the access log.

If anything downstream of “worker takes the connection” stalls, requests pile up in two invisible places: the kernel listen backlog and the clients themselves. Neither increments your request counter. So you get the signature pattern of silent load shedding:

Total Accesses rate drops toward zero.
The access log develops a gap, because the access log only records completed requests.
Clients experience timeouts or connection resets that your metrics do not show.
CPU often looks calm, because waiting is free.

The four causes that produce this pattern are worker exhaustion (all slots busy on something slow), a hung backend holding every proxied request open, an upstream change (load balancer or DNS) that actually did divert traffic away, and, less commonly, a local resource wall like file descriptor exhaustion that prevents Apache from accepting at all. Distinguishing them fast matters, because the fixes are completely different.

flowchart LR
  client[Clients] --> lb[Load balancer]
  lb --> backlog[Kernel listen backlog]
  backlog --> workers[Apache workers]
  workers --> backend[Backend]
  backend --> done[Request completed]
  done --> counted[Total Accesses + access log]
  backlog -. "queued, invisible" .-> gap[RPS drop]
  workers -. "stuck in W state" .-> gap

Common causes

Cause	What it looks like	First thing to check
Worker exhaustion	BusyWorkers at MaxRequestWorkers, IdleWorkers zero, Recv-Q growing, MPM worker-limit code in error log	server-status `BusyWorkers` and `IdleWorkers`
Hung backend (proxy deployments)	Scoreboard full of `W` states, normal CPU, 5xx trickling in, request rate paradoxically low	Curl the backend directly, bypassing Apache
Upstream routing change	RPS drops but workers are idle, backlog empty, no errors anywhere	LB target health and DNS resolution for your hostname
Local resource wall (FDs, disk)	Intermittent 5xx, “Too many open files” or “No space left” in error log, scoreboard stuck in `L`	Error log tail and `df -h` on the log filesystem

Quick checks

All of these are read-only and safe to run during an incident.

# Worker utilization: the single most important first look
curl -s http://localhost/server-status?auto | grep -E "BusyWorkers|IdleWorkers"

# Scoreboard state distribution: where are workers stuck?
curl -s http://localhost/server-status?auto | grep "Scoreboard:" | \
  awk '{print $2}' | fold -w1 | sort | uniq -c | sort -nr

# Listen backlog depth: Recv-Q = queued connections, Send-Q = limit
ss -ltn | grep -E ':80\s|:443\s'

# MaxRequestWorkers reached events (definitive exhaustion signal)
# AH00161 = prefork, AH00286 = worker, AH00484 = event
grep -E 'AH(00161|00286|00484)' /var/log/apache2/error.log | tail -20

# Access log gap tell: is the log still being written at all?
tail -5 /var/log/apache2/access.log; date

# Error log: FD exhaustion, disk full, proxy failures
grep -E "\[error\]|\[crit\]|\[alert\]" /var/log/apache2/error.log | tail -20

# Backend direct probe (adjust host:port), bypassing Apache entirely
curl -s -o /dev/null -w "TTFB: %{time_starttransfer}s Total: %{time_total}s\n" \
  --max-time 10 http://backend-host:8080/health

# Established connections to your backend, if proxying
ss -tn state established dport = :8080 | wc -l

On RHEL-family systems the logs live under /var/log/httpd/ (error_log, access_log) instead of /var/log/apache2/.

How to diagnose it

Establish the baseline first. Compare the current RPS to the same time of day from previous days, not to an absolute number. A drop from 500 to 50 req/s at 03:00 may be normal; the same drop at 14:00 is an incident. If you have no time-of-day baseline, treat any sudden halving as suspect until proven otherwise.
Check worker utilization. If BusyWorkers equals your configured MaxRequestWorkers and IdleWorkers is zero, Apache is saturated and queuing. Requests are arriving but not completing, so your RPS drop is load shedding, not lost demand. MaxRequestWorkers is not shown in server-status; read it from the config, and remember ServerLimit can silently cap it.
Read the scoreboard states. Saturation alone does not tell you why. A scoreboard dominated by W (sending reply) with calm CPU and memory means workers are waiting on something, almost always a backend in proxy deployments. Many R (reading) states points at slow clients or a Slowloris pattern. Many L (logging) states means the log pipe or disk is the stall.
Confirm with the access-log gap tell. If the access log has stopped advancing while the error log is quiet, Apache is alive but unable to complete requests. Queued connections never reach a worker and are never logged. A gap in the access log concurrent with an MPM worker-limit event is expected behavior, not a logging bug.
Check the listen backlog. A sustained non-zero Recv-Q on ports 80/443 means connections are arriving faster than Apache accepts them. This is the earliest saturation signal and confirms real arriving demand even though RPS has collapsed. The kernel caps the effective backlog at net.core.somaxconn regardless of the ListenBacklog directive.
If workers are idle, look upstream. Low BusyWorkers, empty backlog, and no errors means the traffic genuinely is not arriving. Check the load balancer: has this node been removed from rotation by a failing health check? Did a DNS change or expiry repoint the hostname? Did a WAF or CDN rule start blocking? This is the one cause where “traffic went away” is actually true, and no amount of Apache tuning will bring it back.
If proxying, test the backend directly. A hung backend holds workers in W until ProxyTimeout fires, so requests trickle out as 5xx or never complete at all. Curl the backend from the Apache host, bypassing the proxy. If it hangs, your problem is the backend and Apache is the victim.
Rule out local resource walls. “Too many open files” in the error log means FD exhaustion; “No space left on device” plus a scoreboard full of L states is the log-stall deadlock. Both stop completions while the process looks alive.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
BusyWorkers / MaxRequestWorkers	Primary saturation gauge; the drop to “zero RPS” is nearly instantaneous at 100%	Sustained above 80%, IdleWorkers at zero
Scoreboard state distribution	Tells you what workers are stuck on, not just that they are stuck	`W` dominating with calm CPU; any `L` pile-up; `R` above ~20%
Listen backlog Recv-Q	Leading indicator that arrives before user-visible failure	Sustained non-zero during normal traffic
MPM worker-limit events	Apache explicitly reporting worker exhaustion	Any occurrence
Access log write rate	The gap tell: completions stopped even if the process is alive	Log stops advancing while error log is quiet
Backend response time	The number one cause of worker holding in proxy setups	P95 above 2x baseline
503/504 rate	Backpressure surfacing as errors after the silent phase	Any sustained occurrences

Fixes

Workers saturated by a hung backend

This is the most common root cause and the most commonly mistreated one. Raising MaxRequestWorkers does not help; it just gives the backend more workers to hang.

Take the node out of LB rotation if the backend is dead, and fix the backend.
If the backend is slow but alive, temporarily lower ProxyTimeout so Apache fails fast and frees workers instead of holding them for the full default. Tradeoff: clients see quick 5xx errors instead of long hangs, which is usually the better failure mode but still an error.
Longer term, size the proxy connection pool deliberately. The default max equals ThreadsPerChild (1 on prefork), which is far too small for production and produces 503s under moderate load that look exactly like Apache saturation.

Workers saturated by real demand

If the scoreboard shows healthy turnover, backends are fast, and demand simply exceeds capacity:

Raise MaxRequestWorkers, but only after the memory math: MaxRequestWorkers x per-child RSS must stay under roughly 70% of RAM. Setting it from a round number without measuring per-process memory is how you trade a queuing incident for an OOM incident.
On prefork or worker MPM, check KeepAliveTimeout. High values hold worker slots on idle keepalive connections; event MPM largely removes this concern by handling keepalive connections asynchronously instead of occupying a worker.
If the scoreboard shows many R states instead, treat it as slow clients or Slowloris: confirm source IP concentration and make sure mod_reqtimeout is active, then block offenders at the firewall, not in Apache.

Upstream routing change

Fix it where it lives: LB target group health, DNS records, CDN or WAF rules.
If the LB removed the node because its health check times out during saturation, that is worker exhaustion feeding back into routing. Fix the saturation first; the LB will re-add the node.
Verify the LB health check tests the real service path. A health check that fetches a static file passes while the proxied application is dead, which is how nodes stay in rotation while serving nothing.

Local resource walls

FD exhaustion: raise LimitNOFILE in the systemd unit (this needs a restart, not a graceful reload), and consolidate per-VHost log files if you have many virtual hosts.
Log stall: free space on the log filesystem immediately, then restart Apache after the disk is fixed so blocked workers clear. Keep logs on their own filesystem so a log explosion cannot take the server with it.

Prevention

Alert on the ratio, not the rate. RPS drops are a lagging, ambiguous signal. Alert when BusyWorkers / MaxRequestWorkers exceeds 80% sustained, and page when IdleWorkers is zero with a growing backlog or an MPM worker-limit event. That catches silent shedding minutes before the RPS graph flatlines.
Watch the backlog. Sustained non-zero Recv-Q is the earliest warning that acceptance cannot keep up.
Monitor backends separately. In proxy deployments, backend response time is the single highest-value signal for predicting this exact incident.
Set MaxConnectionsPerChild to a finite value (5000-10000) so a leaky module cannot slowly grow children into memory exhaustion, which is another path to the same symptom.
Baseline by time of day. “Dropped to near zero” is only meaningful against the same-hour history.

How Netdata helps

The Apache collector polls server-status?auto every second, so Total Accesses becomes a true per-second completed-request rate rather than a lifetime average like the ReqPerSec field, making drops visible within seconds.
BusyWorkers and IdleWorkers are charted alongside the request rate, so you can see the telltale divergence, rate falling while workers pin at maximum, on one screen instead of correlating by hand.
The scoreboard state breakdown over time shows whether a drop came with a W pile-up (backend), R pile-up (slow clients), or L pile-up (log stall), which is the key diagnostic branch.
Correlating Apache worker saturation with backend latency, system CPU, and socket statistics (listen backlog, connection states) in one place is what turns a 30-minute guessing session into a two-minute root cause.
Anomaly detection on the request rate flags deviations from the learned time-of-day pattern, which is exactly the baseline comparison this symptom demands.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

The Netdata solution

Apache HTTP Server monitoring with Netdata

Netdata monitors Apache HTTP Server with per-second metrics from mod_status, pre-built dashboards, and ML-powered anomaly detection. Watch busy versus idle workers and the scoreboard state mix, requests per second, bytes served per second, and request processing duration alongside the rest of your stack, so you catch the worker-exhaustion, slow-backend, and memory incidents in these runbooks before they page anyone.

See web server monitoring → Start monitoring free