Apache Slowloris: R-state workers, slow-read attacks, and mod_reqtimeout

Your Apache server stopped serving requests, but nothing looks broken. CPU is normal. Memory is normal. The backend is healthy when you test it directly. Request rate is flat or dropping. Yet clients time out, and the load balancer is pulling the node from rotation.

This is the Slowloris signature: a slow-read denial of service where an attacker opens many connections and drips request data byte by byte, holding each worker in the Reading (R) state indefinitely. With enough slow connections, the entire worker pool is consumed reading requests that never complete. The server is not overloaded. It is held hostage.

The attack is cheap to run, needs almost no bandwidth, and produces almost no error-log noise. The scoreboard is where you see it, and most teams never look at the scoreboard until their first incident. This guide covers detection, confirmation, immediate mitigation, and durable defense.

What this means

Apache workers are a finite pool, bounded by MaxRequestWorkers. A worker assigned to a connection stays assigned until the request completes or a timeout fires. In a normal workload, a worker spends a few percent of its life in the R state (reading the request line and headers). Slowloris inverts that ratio: attackers send headers one byte at a time, or send periodic junk headers to reset timers, so workers park in R for minutes.

Once all workers are stuck in R, new connections queue in the kernel listen backlog, then get refused. From the outside the site is down. From the inside, every infrastructure metric looks fine because nobody is doing any work.

flowchart TD
  A[Attacker opens many TCP connections] --> B[Sends request bytes slowly]
  B --> C[Workers park in R state]
  C --> D[BusyWorkers rises, IdleWorkers hits zero]
  D --> E[New connections queue in listen backlog]
  E --> F[Backlog fills, connections refused]
  F --> G[Legitimate users see timeouts]
  C -.-> H[CPU, memory, backend all normal]

Common causes

A high R-state count is not automatically an attack. Rule these out before you block IPs.

Quick checks

All read-only and safe to run during the incident.

# 1. Scoreboard state distribution: is R abnormally high?
curl -s http://localhost/server-status?auto | grep "Scoreboard:" | \
  awk '{print $2}' | fold -w1 | sort | uniq -c | sort -nr

Normal traffic rarely has more than a few percent of workers in R. Above 20% sustained is anomalous. If R dominates and _ (idle) slots are gone, you are in an active event.

# 2. Worker utilization: BusyWorkers climbing, IdleWorkers at zero?
curl -s http://localhost/server-status?auto | grep -E "BusyWorkers|IdleWorkers"

# 3. Source-IP concentration on the listening port
ss -tn sport = :80 | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head -20
# Repeat for :443 if you terminate TLS locally

The attack signature is a small number of IPs holding a large number of connections. A healthy traffic mix shows the reverse.

# 4. Human-readable scoreboard: look for long-stuck reading slots
curl -s http://localhost/server-status | less

In the per-worker table, the classic Slowloris row shows state R, an SS value (seconds since the request started) in the hundreds, client shown as ?, and request shown as ..reading... A worker that has been “reading” a request for 400 seconds is not reading a request.

# 5. Is mod_reqtimeout loaded and firing?
apachectl -M 2>/dev/null | grep reqtimeout
awk '$9 == 408' /var/log/apache2/access.log | wc -l   # adjust path for your distro

408 responses are mod_reqtimeout disconnecting slow clients. Some 408s are normal background noise. A flood of them during the incident confirms slow reads.

# 6. Rule out resource saturation (should all look boring)
ps -C httpd -o pid,%cpu --sort=-%cpu 2>/dev/null | head -5 || \
  ps -C apache2 -o pid,%cpu --sort=-%cpu | head -5
df -h /var/log/apache2/ 2>/dev/null || df -h /var/log/httpd/

Boring CPU, memory, and disk with a full scoreboard is exactly what distinguishes Slowloris from the slow backend cascade and worker saturation patterns.

How to diagnose it

1. Confirm the R-state pattern. Scoreboard shows R well above 20% of slots, BusyWorkers near MaxRequestWorkers, IdleWorkers at zero. Request rate (Total accesses delta) is flat or falling because nothing completes.

2. Confirm source concentration. The ss check shows few IPs holding many connections. If the distribution is broad, you are looking at slow legitimate clients or an upload problem, not an attack.

3. Check the bytes-per-connection ratio. Attack connections transfer almost nothing over their lifetime. Legitimate slow clients still complete requests with real bodies. In the server-status table, long-SS R-state rows with ? clients and ..reading.. requests are definitive.

4. Verify what Apache is logging. If mod_reqtimeout is active, access log 408s spike during the event. If nothing is logged at all, the connections are dying before a request line completes, or the timeout module is not configured.

5. Check what you cannot see: the AcceptFilter blind spot. On Linux, AcceptFilter http data is the default: the kernel holds the socket until at least one byte arrives, and only then hands it to a worker. A variant that opens TCP connections and sends nothing never reaches a worker. BusyWorkers stays normal while the kernel connection table fills. If ss shows thousands of ESTABLISHED connections with zero bytes transferred but few R-state workers, this is your case. The handshake and header timeouts in mod_reqtimeout only start after the worker receives the socket, so mod_reqtimeout cannot catch this variant. Detection has to happen at the connection layer, not the scoreboard.

6. Correlate with the error log. Look for the MPM worker-limit message — AH00484 (event), AH00286 (worker), or AH00161 (prefork) followed by “server reached MaxRequestWorkers setting” — as corroboration that the pool actually exhausted, and to timestamp when saturation began.

Metrics and signals to monitor

Fixes

Immediate: block at the firewall, not in Apache

Once workers are exhausted, Apache-level access rules do nothing useful. There are no free workers to evaluate them. Block the offending sources in the host firewall or, better, upstream at the edge or load balancer.

# Emergency: drop an offending source IP at the host firewall
# Disruptive to that source only; verify the IP list before applying
iptables -A INPUT -s 203.0.113.17 -j DROP

Prefer rate or connection limits over hard drops when the sources might be shared (NAT, corporate egress). If you are behind a load balancer, Apache sees the LB’s addresses unless mod_remoteip is configured, and blocking at the host firewall may be the wrong layer entirely. Block at the LB or edge instead.

Durable: mod_reqtimeout

mod_reqtimeout is loaded by default in Apache 2.4 and is the primary in-Apache defense. It enforces deadlines for receiving the request headers and body, with a minimum data rate:

# In server config or virtual host context
RequestReadTimeout header=20-40,MinRate=500 body=20,MinRate=500

This reads as: headers must complete within 20 seconds, but each 500 bytes per second of progress extends the deadline up to a 40-second ceiling. The body must complete in 20 seconds at a minimum rate of 500 bytes per second. Slowloris connections cannot sustain 500 bytes per second, so they are disconnected with a 408 well inside the window. The defaults shipped since 2.4.39 also include handshake=0 (no limit) for the TLS handshake stage; the handshake= parameter only exists on 2.4.39 and later and causes a configuration error on older builds, so check your version before copying directives that include it.

Tradeoffs to understand:

• It does not prevent exhaustion during the grace period. A 20-second header timeout means each attack connection still holds a worker for up to 20 seconds. An attacker opening connections faster than your pool turns over can still exhaust workers. mod_reqtimeout raises the attacker’s cost; it does not close the hole by itself.
• Legitimate slow clients get cut. If you serve users on genuinely slow links, tighten MinRate carefully and watch the 408 rate after changes.
• Timeouts are logged at info level. To see them explicitly, use LogLevel reqtimeout:info rather than raising global verbosity.
• The AcceptFilter gap remains. Connect-and-send-nothing connections never reach a worker on Linux, so this module never sees them. That variant needs kernel or firewall-level controls.

Durable: per-IP connection limits

Because mod_reqtimeout alone leaves the grace-period gap, pair it with a per-IP connection limiter so no single source can hold more than a handful of slots:

• Firewall-level limiting (connection tracking rules) rejects excess connections before Apache touches them. This is the layer that also catches the AcceptFilter-masked variant.
• Third-party modules such as mod_antiloris hook the connection early enough to reject excess per-IP connections before they occupy a worker. Modules such as mod_evasive and mod_limitipconn act after headers are read, so they never see a Slowloris connection that never finishes sending headers. mod_antiloris availability varies by distribution: Debian and Ubuntu do not currently ship it in their package indexes, while some distributions package it; otherwise build it from its maintained upstream repository.
• Edge limiting (CDN, cloud LB, or reverse proxy in front) is the strongest option because attack connections never reach your Apache hosts at all.

Patch HTTP/2 slow-request CVEs

If you serve HTTP/2, several fixed CVEs are Slowloris-class attacks over h2: CVE-2018-17189 (slow request bodies, fixed in 2.4.38), CVE-2019-9517 (flooding with requests while never reading responses, fixed in 2.4.41), and CVE-2023-43622 (streams blocked indefinitely via initial window size 0, fixed in 2.4.58). If your version predates these fixes, HTTP/2 gives an attacker a second slow-read surface that RequestReadTimeout tuning alone does not cover. Upgrade.

Prevention

• Monitor the R-state ratio continuously. This is the signal most teams only discover during their first incident. Alert on R above 20% of slots sustained, correlated with per-IP connection concentration.
• Load mod_reqtimeout with explicit, reviewed values. It is on by default in 2.4, but confirm apachectl -M shows it and that the timeouts match your client population.
• Track 408s as their own signal. A rising 408 baseline is an early indicator of probing before a full attack.
• Enforce per-IP connection limits at the firewall or edge. Assume mod_reqtimeout alone is not enough, because it is not.
• Restrict server-status. Slowloris reconnaissance and the scoreboard itself both live there. Bind it to localhost or an IP allowlist.
• Baseline connection counts per IP. Without a baseline you cannot tell a distributed slow-client event from a concentrated attack under pressure.
• Keep Apache patched. The HTTP/2 slow-request CVEs above are all fixed versions behind current releases.

How Netdata helps

• Scoreboard state tracking over time turns the R-state ratio from a point-in-time curl into a trend, so you see the slow build before IdleWorkers hits zero, not after.
• Worker utilization and request rate on the same view makes the Slowloris signature obvious: BusyWorkers climbing while completed requests fall is not a traffic surge.
• Connection and socket-level metrics alongside scoreboard data expose the AcceptFilter-masked variant, where connections pile up without workers moving.
• Error and access log correlation puts 408 spikes and MPM worker-limit events on the same timeline as the worker saturation they corroborate.
• ML anomaly detection on the R-state ratio catches low-and-slow attacks that stay under static thresholds but deviate from your baseline.

Netdata’s web server monitoring brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• Apache BusyWorkers and IdleWorkers: reading worker utilization from mod_status
• Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion
• Apache 504 Gateway Timeout: slow backends, ProxyTimeout, and worker pile-up
• Apache CLOSE_WAIT and TIME_WAIT: connection leaks versus normal churn
• Apache backend response time: telling ‘Apache is slow’ from ’the backend is slow’
• Apache 5xx error rate: 500 vs 502 vs 503 vs 504 and what each one means