Your access log is filling with 401s against a login endpoint, or 403s across paths like /.env and /wp-admin. Some of this is the normal background radiation of the public internet. Some of it is an active credential stuffing run against your users. The status code alone does not tell you which; the shape of the traffic does.

The three patterns look different once you group the log lines:

  • A burst of 401s from one IP against one login endpoint, same or few usernames: brute force.
  • 401s from one IP (or a rotating set) against many different usernames: credential stuffing with a leaked password list.
  • 403s spread across sensitive paths (/.env, /.git, /wp-admin, /phpmyadmin, /actuator, /server-status): enumeration by a scanner.

All three are worth detecting. Only some are worth paging on.

What this means

A 401 means authentication was required and failed or was missing. A 403 means the server understood the request and refused it: authentication succeeded but authorization failed, or an access control rule denied the request outright.

Both codes are generated constantly by internet noise. The operational problem is not the existence of these responses but the deviation from baseline: a sudden concentration in source, target, or username distribution.

One caveat before any IP-based analysis: if Apache sits behind a load balancer or CDN and mod_remoteip is not configured, every request in your log carries the proxy’s IP. Every signal in this guide (per-IP rate, source concentration, blocking) is then wrong. Check that first.

flowchart TD
  A[401/403 rate spike] --> B{Which status?}
  B -->|401| C{One IP, one endpoint?}
  C -->|Yes, few usernames| D[Brute force]
  C -->|Yes, many usernames| E[Credential stuffing]
  C -->|Many IPs, many usernames| E
  B -->|403| F{Sensitive paths?}
  F -->|Yes: .env, .git, wp-admin, phpmyadmin| G[Scanner enumeration]
  F -->|No, real app paths| H[Authorization or config problem]
  D --> I[Block source, check for successes]
  E --> I
  G --> J[Verify nothing real responded 200]
  H --> K[Review Require rules and file permissions]

Common causes

CauseWhat it looks likeFirst thing to check
Credential stuffing401s to a login endpoint, many distinct usernames, one IP or rotating IPsCount distinct usernames per source IP in the log
Password brute force401s from one IP, one endpoint, one or few usernames, steady cadenceawk '$9 == 401' grouped by IP and URI
Scanner enumeration403/404 across /.env, /.git, /wp-admin, /phpmyadmin, /actuatorGrep the access log for the scanner path regex
Exposed admin surface/server-status or /server-info returning 200 to the internetCurl the path from an external host
Misconfigured access control403s on legitimate application paths, affecting real usersRecent config changes; Require directives; file permissions
Broken client or integration401s at a steady low rate from one known internal IPExpired token, stale credentials in a cron job or CI system
LB/CDN without mod_remoteipAll “per-IP” analysis shows one IP: the proxyCheck whether %h/%a in the log is the proxy address

Quick checks

All read-only. Paths shown for Debian/Ubuntu; on RHEL-family systems the logs are /var/log/httpd/access_log and /var/log/httpd/error_log.

# Top source IPs in the last 10000 requests
tail -10000 /var/log/apache2/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -20

# 401/403 events grouped by IP, URI, and status
tail -1000 /var/log/apache2/access.log | awk '$9 == 401 || $9 == 403 {print $1, $7, $9}' | sort | uniq -c | sort -rn | head -20

# 401 concentration: which IPs are hitting which endpoints
tail -10000 /var/log/apache2/access.log | awk '$9 == 401 {print $1, $7}' | sort | uniq -c | sort -rn | head -10

# Scanner probe check: common enumeration targets
grep -iE '/(\.env|\.git|wp-admin|wp-login|phpmyadmin|actuator|graphql|\.aws|api/v[0-9]+/admin|server-status|server-info)' /var/log/apache2/access.log | tail -20

# Connection concentration right now (is one source holding many sockets?)
ss -tn sport = :80 | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head -20

# Confirm mod_remoteip is loaded when behind a proxy
apachectl -M 2>/dev/null | grep remoteip

The status field position ($9) assumes combined log format; if your LogFormat differs, adjust. As a working threshold, a 401 rate over roughly 50 per minute from a single IP indicates an active attack rather than background noise.

How to diagnose it

  1. Establish the baseline first. Look at the same log window from yesterday and last week. Every internet-facing server has a background hum of scanner 403s and 404s. If today’s rate matches the baseline, you are looking at noise, not an incident. Alerting on absolute counts instead of deviation from baseline is how teams end up ignoring these alerts entirely.

  2. Verify the client IPs are real. If requests arrive via a load balancer or CDN, the log’s IP field shows the proxy unless mod_remoteip rewrites it from X-Forwarded-For or equivalent. If every “top talker” is your LB, stop and fix that before any further analysis; nothing downstream of a wrong IP field is trustworthy. See the Fixes section for the configuration.

  3. Classify by shape, not by code. Group the 401s by source IP, then by URI, then look at the username dimension. One IP, one endpoint, one username, high frequency: brute force. One IP, one endpoint, dozens of distinct usernames: credential stuffing. For HTTP Basic auth, the attempted username appears in auth failure messages in the error log; note that the %u field in the access log is not trustworthy on a 401 response, because the user was never authenticated. For application-level logins (a POST to your app’s /login), the username lives in the request body, which Apache does not log, so you need application logs for that dimension.

  4. Check whether anything succeeded. This sets severity. For credential stuffing, look for 200 responses on the login endpoint interleaved with the 401s from the same sources. For scanner enumeration, check whether any probed path returned 200 instead of 403/404. A 403 against /phpmyadmin on a server that runs no PHP is noise. A 200 against /.git/config is an incident: your repository metadata is exposed.

  5. Check the operational side effects. Failed authentication attempts that close connections drive reconnect churn: more connections per second, more TIME_WAIT, more TLS handshakes on 443. High churn is mostly the kernel’s problem (TIME_WAIT consumes no Apache workers), but a heavy flood can still show up as elevated connection rates and, on prefork or worker MPM, increased worker pressure from the request volume itself. Compare ss connection counts and request rate against baseline before concluding the flood is “only noise.”

  6. Confirm the 403s are the denial you expect. If real users are getting 403s on real paths, this is not an attack. Check for a recent config change: a modified Require directive, a permissions change on the document root, or mixing legacy Order/Allow/Deny directives with Require in different scopes, a known source of unexpected 403s on Apache 2.4. apachectl configtest validates syntax but will not catch an authorization rule that is syntactically fine and semantically wrong.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
401 rate per source IPPrimary brute force and stuffing detector>50/min from one IP, or any sustained deviation from baseline
Distinct usernames per source IPSeparates credential stuffing from brute forceMany usernames, one or few IPs
403s on sensitive pathsEnumeration detectionProbes against /.env, /.git, /wp-admin, /phpmyadmin, /actuator that return 200
4xx rate as % of totalCatches the spike without staring at raw countsSudden jump >3x baseline
Request rate per source IPDetects floods and scrapersSingle IP >100x the average IP rate (excluding known LB/CDN)
New connection rate and TIME_WAIT countShows reconnect churn from failed-auth floodsConnection rate rising faster than completed request rate
200s on login endpoints from attacking IPsTells you whether the attack workedAny success interleaved with a 401 burst

Fixes

Blocking the source

Block at the firewall, not in Apache. By the time a request reaches an Apache worker, the worker is already spent. During an active flood, denying at the network layer (iptables/nftables, or the cloud security group / CDN edge rule) keeps Apache out of the path entirely. For recurring offenders, fail2ban with Apache log-based jails automates the parse-and-ban loop; for application-layer logins, rate limiting belongs at the CDN edge or in the application, where per-account and per-IP limits can be enforced before Apache sees the request.

Before blocking anything, confirm the IP is a real client and not your load balancer. If mod_remoteip is missing, every block you add at the firewall is aimed at your own proxy and will take down all traffic, not just the attacker’s.

Do not rely on mod_evasive for this. It is largely unmaintained, and its behavior under modern Apache 2.4 MPMs is unreliable. mod_ratelimit is also not the tool: it throttles response bandwidth, not request rate per IP.

Fixing the client IP behind a proxy

If Apache is behind a load balancer or CDN, configure mod_remoteip so the log and all IP-based logic see the real client:

# Example: trust the LB and use its forwarded header
RemoteIPHeader X-Forwarded-For
RemoteIPInternalProxy 10.0.0.0/8

Only list proxies you actually control in RemoteIPInternalProxy; the header is trivially spoofable by direct clients, so trusting it from untrusted sources makes your logs less accurate, not more. Some CDNs use their own header (for example Cloudflare’s CF-Connecting-IP) rather than standard X-Forwarded-For; use the header your edge actually sets.

Reducing the attack surface

  • Restrict /server-status and /server-info to localhost or a management network with Require ip. Publicly reachable, they disclose internal IPs, request URLs, and vhost structure, which is exactly the reconnaissance a scanner is looking for.
  • Return 404, not 403, for paths that do not exist. A 403 confirms something is there, and enumeration tools treat it as “keep digging.” (This is an application and config design point; Apache’s default behavior for denied-but-existing paths is 403.)
  • Move admin panels off predictable paths or behind authentication at the edge, so /wp-admin and /phpmyadmin probes are guaranteed noise.
  • Disable unused methods: TraceEnable Off at minimum, and deny PUT/DELETE/PROPFIND unless the application needs them.

After a confirmed stuffing run

If the log shows 200s interleaved with the 401 burst, treat it as an account-security incident, not an Apache incident: force password resets for the affected accounts, notify per your incident process, and add rate limiting plus MFA at the application layer. Apache’s job was to record it; the fix lives elsewhere.

Prevention

  • Baseline the noise. Track 401/403 rates per day and time-of-day so “spike” has a numeric meaning. Alert on deviation (for example >3x baseline, or >50 401s/min per IP), not on absolute counts.
  • Keep the real client IP end to end. mod_remoteip configured and tested after any change to the LB/CDN chain. Verify periodically that access log IPs are client IPs, not proxy IPs.
  • Minimize what exists to be denied. No .git directories under the document root, no .env files reachable by the web server, no phpMyAdmin on production hosts. Enumeration that finds nothing is permanently noise.
  • Separate the filesystem and the blast radius. Floods inflate the access log. Log growth from an attack should fill a log partition, not the root filesystem.
  • Use only Require-based access control on 2.4. The legacy Order/Allow/Deny directives are deprecated, and mixing the two models across scopes produces surprise 403s.

How Netdata helps

Netdata’s Apache collector and log-based monitoring line up directly with the classification workflow above:

  • Per-second request and response metrics from mod_status show the traffic spike as it happens, so you can correlate a 401/403 burst with overall request rate and worker utilization instead of discovering it in a log review.
  • Status-code breakdowns from weblog parsing separate 401s, 403s, and 404s into distinct series, which makes baseline-vs-spike comparison a chart glance rather than an awk pipeline.
  • Correlation with saturation signals: if a flood is driving connection churn, you can see BusyWorkers, connection counts, and the error rate in one view and decide whether the attack is merely noisy or actually degrading service.
  • Anomaly detection on the 4xx series flags deviations from the learned baseline, which is the practical answer to alert fatigue from constant scanner background noise.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.