Apache HTTPD Operations Guides on Netdata

Apache /server-status exposed: the reconnaissance leak hiding in mod_status

Mon, 01 Jan 0001 00:00:00 +0000

A scanner, pentest, or audit just flagged your Apache server: /server-status is reachable from the internet. mod_status renders a live view of your server’s internals, and anyone who can load it can watch your traffic in near real time.

The operational problem is that mod_status is also the best monitoring source Apache has. BusyWorkers, the scoreboard, requests per second: all of it comes from /server-status?auto. So the fix is not “disable mod_status.” The fix is to restrict who can reach it, verify the restriction holds on every vhost, and keep scraping it locally.

Apache %D includes client transfer time: the latency alert that cries wolf

Mon, 01 Jan 0001 00:00:00 +0000

Your p99 latency alert fired at 03:00. You pull the access log, sort by %D, and find requests with durations of 400, 600, even 800 seconds. The server looks fine: CPU normal, workers mostly idle, backend healthy. The “slow” requests are large file downloads to clients on slow connections. Apache served the response instantly; the client took thirteen minutes to read it.

%D does not measure how long Apache took to process the request. It measures the time from when Apache read the request to when the last byte of the response was handed to the OS network stack, and that second part is paced by how fast the client consumes data. TCP send buffers are finite. When the buffer fills, Apache blocks in write() until the client drains it. A 100MB download to a 1Mbps client shows %D of roughly 800 seconds even though server-side work took milliseconds.

Apache %T vs %D vs %{ms}T: stop rounding latency to whole seconds

Mon, 01 Jan 0001 00:00:00 +0000

Open your Apache access log and look at the request duration field. If every line shows 0, your monitoring pipeline is probably fine. Your LogFormat is the problem. The %T directive logs time to serve the request in whole seconds, so any request that completes in under one second logs as 0. On a fleet where most requests finish in well under 100ms, %T collapses the entire latency distribution into a single useless value.

Apache 401/403 floods: credential stuffing, scanner probes, and access denials

Mon, 01 Jan 0001 00:00:00 +0000

Your access log is filling with 401s against a login endpoint, or 403s across paths like /.env and /wp-admin. Some of this is the normal background radiation of the public internet. Some of it is an active credential stuffing run against your users. The status code alone does not tell you which; the shape of the traffic does.

The three patterns look different once you group the log lines:

A burst of 401s from one IP against one login endpoint, same or few usernames: brute force.
401s from one IP (or a rotating set) against many different usernames: credential stuffing with a leaked password list.
403s spread across sensitive paths (/.env, /.git, /wp-admin, /phpmyadmin, /actuator, /server-status): enumeration by a scanner.

All three are worth detecting. Only some are worth paging on.

Apache 500 Internal Server Error: modules, handlers, and misconfiguration

Mon, 01 Jan 0001 00:00:00 +0000

Unlike the 502/503/504 family, which Apache generates while proxying to a broken or exhausted backend, a 500 is generated inside Apache itself or by the handler Apache invoked: a module crashed, a CGI script failed, a rewrite rule looped, an .htaccess directive is invalid, or the server hit a permission problem reaching the content.

That distinction is the diagnostic strategy. For a 502 you look at the backend. For a 500 you look at Apache’s own error log, because every 500 Apache emits has a corresponding error-log line with the real cause. The access log tells you that it happened and which URL triggered it. The error log tells you why.

Apache 502 Bad Gateway: a backend that returned an invalid response

Mon, 01 Jan 0001 00:00:00 +0000

You are seeing 502 Bad Gateway responses from Apache, usually in bursts, often on some requests and not others. Users report intermittent failures. Apache is running, the port is open, static content may even work. The proxied application path is the thing failing.

A 502 from Apache is almost never an Apache bug. In a mod_proxy deployment, 502 means Apache acted as a reverse proxy, forwarded the request to a backend, and got back something it could not use: a garbled response, an empty response, a connection closed mid-reply, or a connection that refused to carry the request at all. Apache is surfacing a backend signal.

Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion

Mon, 01 Jan 0001 00:00:00 +0000

Your access log is filling with 503s and users are reporting the site is down. The status code tells you almost nothing: Apache returns 503 for two root causes that look identical in the access log but need completely different fixes.

The first is frontend worker exhaustion. Every worker slot in the scoreboard is busy, Apache has hit MaxRequestWorkers, and new connections queue in the kernel backlog until they time out. The second is a mod_proxy failure: a backend is down, marked errored, or the per-child proxy connection pool is too small, so Apache refuses to forward requests even though its own workers are mostly idle.

Apache 504 Gateway Timeout: slow backends, ProxyTimeout, and worker pile-up

Mon, 01 Jan 0001 00:00:00 +0000

A 504 on an Apache-proxied path means a gateway in the request chain timed out. There are two cases:

Apache generated the 504: mod_proxy waited longer than the effective proxy timeout for the backend.
The backend generated the 504: Apache received that status and passed it through, often because the backend was proxying to a dead dependency of its own.

The access log alone does not distinguish those cases. Correlate the 504 with Apache’s error log.

Apache 5xx error rate: 500 vs 502 vs 503 vs 504 and what each one means

Mon, 01 Jan 0001 00:00:00 +0000

Your Apache 5xx error rate just spiked. The status code is the first fork in the diagnostic path, and it is a sharp one: a 500 means the failure happened inside Apache or a module, while 502, 503, and 504 are mod_proxy telling you something about the relationship between Apache and a backend. Treating them as one bucket of “server errors” wastes the best lead you have.

There is a second trap. A sustained 5xx rate almost always means something real is broken, because 5xx is server-side failure by definition. But the reverse is not true: a proxied application that returns error pages with HTTP 200 is invisible to status-based monitoring. Low 5xx is necessary, not sufficient, evidence of health.

Apache accepting deprecated TLS: TLS 1.0/1.1 and weak-cipher exposure

Mon, 01 Jan 0001 00:00:00 +0000

If a compliance scan flags “TLS 1.0 enabled” or “weak cipher suites supported” on an Apache vhost, the finding is almost always real. Apache’s default SSLProtocol in the 2.4 branch is all -SSLv3, so TLS 1.0 and TLS 1.1 are offered unless someone explicitly removed them. A vhost that was correct three years ago may still be accepting protocol versions that PCI DSS and modern browser policy treat as deprecated.

Apache AH00484: server reached MaxRequestWorkers setting - worker pool exhausted

Mon, 01 Jan 0001 00:00:00 +0000

You found this line in the Apache error log:

AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

This is Apache explicitly telling you that every worker slot in its pool was occupied and it had nowhere to put a new connection. From that moment, new connections queue in the kernel’s TCP listen backlog, and once the backlog fills, clients get connection refused. Users experience timeouts or a dead site while the Apache process looks perfectly healthy in ps.

Apache AH00558: Could not reliably determine the server's fully qualified domain name

Mon, 01 Jan 0001 00:00:00 +0000

You restart Apache, run apachectl configtest, or check the error log after a deploy, and you see it again:

AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message

The short answer: Apache is still serving traffic. This warning does not stop startup, does not drop requests, and is not a security issue. It means no global ServerName directive is set, so Apache guessed a name for itself and is telling you what it guessed.

Apache and nf_conntrack: table full and the invisible packet drops

Mon, 01 Jan 0001 00:00:00 +0000

Users report timeouts and intermittent connection failures, the load balancer is flapping the backend out of rotation, and yet everything on the Apache host looks fine. BusyWorkers are normal. The 5xx rate is flat. The listen backlog is empty. CPU and memory are unremarkable. Apache is not logging errors, because from Apache’s point of view, nothing is wrong.

The failure is one layer below Apache, in the kernel’s netfilter connection tracking subsystem. When the nf_conntrack table fills up, the kernel emits nf_conntrack: table full, dropping packet and starts silently discarding packets. New SYN packets never reach the TCP stack, so they never reach the listen backlog, so they never reach an Apache worker. Clients see their SYNs dropped and retransmit until they give up: a timeout, not a refusal, which makes it look like a network problem.

Apache backend response time: telling 'Apache is slow' from 'the backend is slow'

Mon, 01 Jan 0001 00:00:00 +0000

Users report the site is slow. Apache is up, requests eventually complete. Someone says “Apache is slow,” someone else says “the app is slow,” and both are guessing, because from the outside the two are indistinguishable: the client just sees a slow response.

When Apache proxies to a backend, the %D value in your access log bundles three things together: Apache’s own processing, time waiting for the backend, and time transferring the response to the client. A slow backend, a saturated Apache, and a client on a bad connection all produce the same inflated %D. Without separate instrumentation per component, you cannot attribute the latency, and teams routinely burn hours tuning Apache when the database is the problem.

Apache balancer member in error state: reading balancer-manager and failover

Mon, 01 Jan 0001 00:00:00 +0000

You opened balancer-manager, or someone pasted a screenshot into the incident channel, and one of your BalancerMembers shows Err in the status column. Traffic is still flowing, but the pool is quietly running on fewer backends than you think. If enough members flip to error state, Apache stops proxying entirely and returns 503s even though httpd itself is healthy.

This state is mod_proxy_balancer doing its job: it detected failures against a backend and pulled that member out of rotation so requests stop dying on it. The problem is that the balancer tells you almost nothing about why, and it keeps the member sidelined on its own retry schedule regardless of whether the backend has recovered.

Apache BusyWorkers and IdleWorkers: reading worker utilization from mod_status

Mon, 01 Jan 0001 00:00:00 +0000

BusyWorkers and IdleWorkers are the two most-quoted numbers from Apache’s mod_status output, and the two most frequently misread. Together they are the primary saturation gauge for the server: how much of the worker pool is currently occupied. Read them wrong and you either miss the onset of worker exhaustion or page someone for healthy autoscaling churn.

This guide covers what the two counters actually count, how to turn them into a utilization ratio you can alert on, why Apache degrades at a cliff edge rather than gradually, and how to distinguish the two very different situations that both show IdleWorkers: 0.

Apache capped by systemd: TasksMax, LimitNOFILE, and MemoryMax override your config

Mon, 01 Jan 0001 00:00:00 +0000

You raised MaxRequestWorkers, restarted Apache, and it still refuses to spawn more workers. The error log shows no AH00484, the host has free CPU and RAM, and connections queue and time out. Or Apache was OOM-killed even though free showed gigabytes available. Or you hit “Too many open files” at a fraction of the limit you set in /etc/security/limits.conf.

When httpd runs under systemd, the unit file is a second, invisible configuration layer. TasksMax, LimitNOFILE, and MemoryMax are enforced by the kernel (cgroups and rlimits), not by Apache, and they silently override whatever you put in httpd.conf. Apache does not know these limits exist. It just fails to fork, fails to open a file, or gets killed, and the errors it logs point at the wrong thing.

Apache child pid exit signal Segmentation fault: crashing workers

Mon, 01 Jan 0001 00:00:00 +0000

You found this line in the error log:

[core:notice] [pid 1234] AH00052: child pid 5678 exit signal Segmentation fault (11)

A child process died on a memory access violation. The parent logged the exit status and spawned a replacement. That is Apache’s designed recovery path, and a single occurrence with no user impact is that mechanism working. But a segfault is never “normal noise”: a bug executed in Apache, a loaded module, or a shared library, and any segfault in production deserves root-cause analysis. Multiple per minute means the service is actively degrading.

Apache CLOSE_WAIT and TIME_WAIT: connection leaks versus normal churn

Mon, 01 Jan 0001 00:00:00 +0000

You ran ss -tan on an Apache host and saw thousands of connections that are not ESTABLISHED. Some are in TIME_WAIT, some in CLOSE_WAIT, and the numbers look alarming. The first question is not “how do I get rid of them” but “which of these is actually a problem”.

The two states look similar in a socket listing but mean opposite things. TIME_WAIT is the kernel doing its job after a connection closes cleanly; it is normal churn on any busy web server. CLOSE_WAIT means the remote peer closed the connection and Apache never finished closing its side; a sustained, growing CLOSE_WAIT count is an application-side leak that consumes file descriptors and, eventually, workers.

Apache configtest and Include wildcards: catching bad config before it bites

Mon, 01 Jan 0001 00:00:00 +0000

apachectl configtest (equivalent to httpd -t) parses your Apache configuration and reports Syntax OK or a specific error. It is the cheapest safety check in your toolchain, and it is the main thing standing between a bad edit and the stale-config trap: the state where you believe a new configuration is live, but Apache rejected it at reload time and is still running the old one.

The trap works like this. You edit a config file, run a graceful reload, and move on. If the reload fails the config check, the old configuration keeps running and the new one is discarded. Nothing in the process table or access log tells you. The only evidence is a few lines in the error log that nobody greps. Hours later, someone restarts Apache for an unrelated reason and the server fails to come up, because the config on disk was never valid.

Apache CPU saturation: TLS handshakes, mod_deflate, mod_rewrite, and mod_security

Mon, 01 Jan 0001 00:00:00 +0000

Apache serving static content is almost never CPU-bound. The request path (accept, read, sendfile, log) is cheap. When httpd processes start eating cores, the cause is nearly always something doing per-request or per-connection computation: TLS handshakes, mod_deflate compression, mod_rewrite regex evaluation, or mod_security rule processing.

The symptom is gradual, not a cliff. Latency creeps up as CPU saturates. Unlike worker exhaustion, which fails hard at 100% utilization, CPU saturation degrades service progressively, which makes it easy to ignore until p99 latency is unacceptable.

Apache error log monitoring: severity levels, AH codes, and what to alert on

Mon, 01 Jan 0001 00:00:00 +0000

Metrics tell you that something is wrong with Apache. The error log tells you what. When BusyWorkers pegs at MaxRequestWorkers, the scoreboard shows saturation, but the error log hands you AH00484: “server reached MaxRequestWorkers setting.” When a child dies, the process table shows a respawn; the error log shows “Segmentation fault.”

This guide covers how to read the log efficiently: the severity hierarchy, the AH code scheme introduced in 2.4, the patterns worth alerting on, and the configuration gotchas that silently suppress the messages you need.

Apache graceful reload ran the old config: the silent stale-configuration trap

Mon, 01 Jan 0001 00:00:00 +0000

You edited a vhost, ran apachectl graceful, saw no error, and moved on. Hours later someone notices the new TLS certificate is not being served, the new redirect is missing, or the old ProxyPass target is still receiving traffic. Apache never went down and no alert fired. The reload failed and the old configuration kept running.

This is one of Apache’s worst failure modes because nothing looks broken. The server keeps serving and request rates stay normal. The only thing wrong is that the running configuration is not the configuration on disk, which quietly invalidates every assumption you make while debugging the next incident.

Apache graceful restart pile-up: G states, stacked generations, and doubled memory

Mon, 01 Jan 0001 00:00:00 +0000

Memory on an Apache host climbs in steps, each step landing a few minutes after a deploy, a config push, or a log rotation. The scoreboard shows an unusual number of G (gracefully finishing) workers. ps shows far more httpd children than MaxRequestWorkers should allow. Nothing is erroring yet, but the host is drifting toward swap, and the next restart will make it worse.

This is the graceful restart pile-up pattern. Each graceful restart spawns a new generation of children while the old generation drains. Old children do not exit until their in-flight requests complete. If requests are slow (large downloads, slow proxied backends, long-lived connections), the old generation lingers. If another graceful restart arrives before the previous generation has drained, you now have three generations of children, each holding memory. Stack enough generations and the host OOMs.

Apache HTTPD monitoring checklist: the signals every production web server needs

Mon, 01 Jan 0001 00:00:00 +0000

This is a working checklist for engineers running Apache HTTPD in production. It organizes the signals worth collecting into four maturity levels, from “is the process alive” through “which worker is stuck and why.” Use it to audit an existing setup for gaps or to build one without over-instrumenting on day one.

Two things before the list. First, every saturation signal in Apache is interpreted through the active Multi-Processing Module (MPM), so the checklist starts there. Second, the levels are cumulative: Level 2 assumes Level 1 is in place. Do not skip ahead. The scoreboard state distribution is useless if nobody is watching the error log for AH00484.

Apache HTTPD monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Most teams monitor Apache at whatever level their last outage forced them to reach. A disk fills up, the site goes down, and “check log disk space” appears in the runbook. A backend hangs, workers exhaust, and “watch BusyWorkers” gets added. This model organizes that organic growth into four deliberate levels so you can see what you have, what you are missing, and which blind spot will produce your next incident.

Apache keepalive consuming workers: KeepAliveTimeout, the K state, and MPM choice

Mon, 01 Jan 0001 00:00:00 +0000

The symptom looks like a capacity problem: Apache logs AH00484: server reached MaxRequestWorkers setting, new connections start queuing, and users see slow responses or 503s. But the request rate is low, and nothing in CPU, memory, or bandwidth explains it. Then you open the scoreboard and see it: a wall of K states. Most of your workers are not serving requests. They are parked on idle keepalive connections, waiting for a next request that may never come.

Apache killed by the OOM killer: the memory exhaustion cascade

Mon, 01 Jan 0001 00:00:00 +0000

Your monitoring says Apache is up. The parent process exists, the port is listening, but the site is down or intermittently dead. When you finally check the kernel log, there it is: Out of memory: Killed process ... (httpd). Not once. Dozens of times, at regular intervals, going back hours.

This is the Apache OOM death spiral. The kernel OOM killer terminates httpd child processes because total memory demand exceeded RAM. The Apache parent survives, respawns the children, the new children immediately start serving queued requests and allocating memory, and the OOM killer kills them again. The service looks “running” while serving little or nothing.

Apache listen queue overflow: Recv-Q growth, ListenBacklog, and refused connections

Mon, 01 Jan 0001 00:00:00 +0000

Users report “connection refused” or intermittent timeouts, but Apache is running, the port is open, and a TCP connect from localhost sometimes works. The load balancer flaps the backend in and out of rotation. Nothing in the access log explains it, because the failing connections never got far enough to be logged.

This is the signature of listen queue overflow. Connections complete the TCP handshake in the kernel and sit in the accept queue waiting for an Apache worker to call accept(). When workers cannot keep up, the queue fills, and the kernel starts ignoring or resetting new connection attempts. The server looks up. It is effectively down for a slice of arriving connections.

Apache ListenBacklog vs net.core.somaxconn: the silently truncated accept queue

Mon, 01 Jan 0001 00:00:00 +0000

You tuned Apache for burst absorption. You set ListenBacklog 2048 (or left the default 511, reasoning it was generous). Then a traffic spike arrived, workers saturated, and connections were refused far earlier than your capacity model predicted. The scoreboard and MaxRequestWorkers get the blame, but the real culprit is often one layer down: the kernel quietly rewrote your backlog at listen() time and never told anyone.

Apache passes ListenBacklog as the backlog argument to listen(2). Per the listen(2) man page, if the backlog argument is greater than the value in /proc/sys/net/core/somaxconn, the kernel silently caps it to that value. No log line, no warning from Apache, no error return. The effective maximum accept queue on any listening socket is always min(ListenBacklog, net.core.somaxconn).

Apache log rotation losing lines: logrotate copytruncate vs graceful reopen

Mon, 01 Jan 0001 00:00:00 +0000

You notice it after the fact: the access log has a gap. Requests you know happened, because the application processed them and the client got a response, never appear in any log file. Not in the current log, not in the rotated one. The gap lines up with the moment logrotate ran.

Or the opposite symptom: after rotation, Apache keeps writing to the old file. access.log.1 grows for days while access.log stays empty. Or every night at rotation time you see a blip of dropped connections and a spike of 499s or client resets, because the postrotate script does a hard restart instead of a graceful one.

Apache log stall deadlock: workers stuck in the L state, throughput at zero

Mon, 01 Jan 0001 00:00:00 +0000

The Apache parent process is running. The port is open. A TCP connect succeeds. But nothing is served: requests hang, throughput is at or near zero, and the scoreboard is a wall of L characters. Every worker has finished its request and is now blocked writing the log line for it.

Workers write access log entries synchronously at the end of the request cycle, before returning to the idle pool. If that write blocks, because the log filesystem is full or a piped logging program has stopped reading, the worker never becomes available again. New workers accept new connections, finish those requests, and block on the same write. Within minutes the entire worker pool is frozen in the Logging state.

Apache MaxConnectionsPerChild: bounding leaky modules by recycling children

Mon, 01 Jan 0001 00:00:00 +0000

The symptom usually arrives as a slow memory problem, not a clean Apache failure. Per-child RSS climbs for hours or days, total Apache memory grows linearly, swap starts to move, and then the kernel OOM killer begins shooting httpd children. The parent respawns them, the new children leak too, and the host enters a respawn and OOM loop.

When that pattern is present, MaxConnectionsPerChild 0 is often part of the story. The default value of 0 means children never recycle. A leaky module, or plain APR pool fragmentation in a long-lived prefork child, gets unlimited time to grow. Setting a finite value, commonly 5000 to 10000, forces each child to exit after handling that many connections so the parent can replace it with a fresh, smaller process.

Apache MaxRequestWorkers tuning: sizing the worker pool against memory

Mon, 01 Jan 0001 00:00:00 +0000

MaxRequestWorkers is the single most misconfigured directive in Apache HTTPD. The failure pattern is always the same: someone picks a round number like 1000, deploys it on a 4GB server running mod_php children at 50MB RSS each, and the arithmetic (1000 x 50MB = 50GB of potential demand on 4GB of RAM) guarantees an OOM cascade the first time traffic reaches the limit. The setting feels like a performance knob. It is a memory budget.

Apache No space left on device: a full log disk that stops the server serving

Mon, 01 Jan 0001 00:00:00 +0000

The error string operators usually search for is (28)No space left on device in the Apache error log. Sometimes it appears at startup and Apache refuses to run. The nastier version appears at runtime: the site goes dark, but the process is alive, the port is open, and TCP connections are accepted. Health checks that only test the socket say everything is fine.

What is actually happening in the runtime case is the log stall deadlock. Apache workers finish their requests and then block trying to write the access or error log entry. A worker blocked on a log write sits in the L (Logging) state on the scoreboard. It cannot take the next connection. As more workers finish requests and stall, the scoreboard fills with L states until no worker is available. The server is up. It serves nothing.

Apache OCSP stapling failure: the silent handshake that adds client latency

Mon, 01 Jan 0001 00:00:00 +0000

OCSP stapling failure is a textbook “silently catastrophic” signal: nothing in the error log at default levels, no 5xx, no worker pile-up, and yet every new TLS handshake is slower than it should be. When Apache cannot fetch or attach a stapled OCSP response, each client that wants revocation information makes its own OCSP request to the CA’s responder, adding hundreds of milliseconds before the handshake completes. The server looks healthy from the inside. The slowness only exists on the client side.

Apache open-file limits: raising ulimit and systemd LimitNOFILE correctly

Mon, 01 Jan 0001 00:00:00 +0000

The default per-process open-file limit on most Linux distributions is 1024, far too low for production Apache. Every client connection, every backend proxy connection, every log file, and every pipe consumes a file descriptor in every child process, and 1024 runs out long before MaxRequestWorkers does. When a child hits the limit, the failure is a cliff edge: “Too many open files” in the error log, failed accepts, failed backend connections, and intermittent 5xx responses, often in only some children at first, which makes the symptom look random.

Apache per-child RSS climbing: the memory-leak slow death

Mon, 01 Jan 0001 00:00:00 +0000

Your Apache server has been up for three weeks. Nothing is erroring. Request rate is normal, latency is fine, the scoreboard looks healthy. But free memory keeps shrinking, swap usage is creeping up, and last night the OOM killer took out two httpd children. The parent respawned them, and the cycle started over.

This is the memory-leak slow death: a module loaded into Apache (most commonly mod_php, mod_perl, or a custom module) leaks a small amount of memory per request. Each child’s RSS grows monotonically over hours or days. Nothing looks broken until the system runs out of memory, the OOM killer starts shooting children, and the parent respawns fresh ones that also leak. You are now in a kill-and-respawn spiral.

Apache piped logging failures: when rotatelogs dies and workers SIGPIPE

Mon, 01 Jan 0001 00:00:00 +0000

Piped logging, where ErrorLog or CustomLog points at a program instead of a file (CustomLog "| /usr/bin/rotatelogs /var/log/httpd/access_log.%Y-%m-%d 86400" combined), adds a new failure mode to Apache: the log program itself. When rotatelogs (or whatever sits at the end of the pipe) dies, wedges, or cannot write because its target disk is full, the pipe breaks. Log writes then fail silently, or worse, the Apache children writing to the broken pipe receive SIGPIPE and can crash.

Apache proxy connection refused: AH01114 / (111) Connection refused to backend

Mon, 01 Jan 0001 00:00:00 +0000

Your error log is filling with lines like these:

AH00957: HTTP: attempt to connect to 127.0.0.1:8080 (localhost) failed
AH01114: HTTP: failed to make connection to backend: localhost

Clients see 502 or 503 responses. Apache itself is running fine. The failure is on the outbound leg: mod_proxy tried to open a TCP connection to the backend and the kernel told it no.

The important thing about (111)Connection refused is what it rules out. ECONNREFUSED means the backend host’s network stack answered immediately with a TCP RST. The host is reachable; nothing is listening on that port, or something actively rejected the connection. That is a different failure from a backend that is slow, hung, or firewalled, which produces a connect timeout after a long wait and a different error path (AH01075: Error dispatching request to, or a 504 after ProxyTimeout). Learning to separate “fast refusal” from “slow timeout” is the core diagnostic move on this page, and it is the difference between “restart the backend” and “go look at the network.”

Apache request latency: p50/p95/p99 from the access log %D field

Mon, 01 Jan 0001 00:00:00 +0000

“Apache is slow” is one of the least actionable statements in operations. The follow-up questions are always: slow for whom, on which URLs, and how slow at the tail? Averages cannot answer any of those. A server where every request takes 200ms and a server where half the requests take 5ms and half take 400ms have the same mean latency and completely different operational problems.

Apache gives you one high-resolution per-request number: the %D field in the access log, the request duration in microseconds. Logged on every completed request, it is the raw material for p50, p95, and p99 percentiles. This article covers what %D actually measures, how to compute percentiles from the log, how to read the resulting distribution, and the traps that make latency data lie to you.

Apache request rate dropped to near zero: silent load shedding and its causes

Mon, 01 Jan 0001 00:00:00 +0000

Your dashboard shows requests per second falling off a cliff, but nobody changed anything, the load balancer still reports healthy demand, and there is no error spike to point at. The most natural reading, “traffic went away,” is usually wrong.

The trap is in how the number is produced. The request rate most operators watch comes from mod_status Total Accesses, which counts completed requests. It is not a measure of arriving demand. When Apache cannot complete requests, because every worker is stuck or a backend has stopped answering, the completed-request rate collapses even while clients are still hammering the front door. The server has not lost its traffic. It is silently shedding load, and the metric you trust is the last place that shows up.

Apache reverse proxy worker busy: proxy connection pool exhaustion (AH01136)

Mon, 01 Jan 0001 00:00:00 +0000

You are seeing intermittent 503s on proxied endpoints under what looks like moderate load. The error log shows AH01136: Reverse proxy worker busy. BusyWorkers is nowhere near MaxRequestWorkers, CPU and memory are fine, and requests keep failing. Raising MaxRequestWorkers changed nothing.

That is the signature of proxy connection pool exhaustion. The bottleneck is not Apache’s worker pool. It is the smaller, less visible pool of backend connections that mod_proxy maintains per child process, and its default size is too small for production load.

Apache scoreboard states explained: what _ S R W K D C L G tell you

Mon, 01 Jan 0001 00:00:00 +0000

When someone asks “why is Apache slow?”, the first place to look is not CPU, not memory, not the access log. It is the scoreboard. The scoreboard is a shared-memory segment where every worker slot records what it is doing right now, one character per slot. It is what mod_status reads, and it is the most diagnostic structure Apache exposes.

A scoreboard snapshot answers the question metrics alone cannot: not “is Apache busy?” but “busy doing what?” A server with 200 workers stuck in W is a different incident from 200 workers stuck in R or L, even though all three look identical from the outside: the site is down.

Apache slow backend cascade: how one slow upstream starves the whole worker pool

Mon, 01 Jan 0001 00:00:00 +0000

Users report the site down. The load balancer has pulled the node from rotation. You SSH in expecting to find Apache melting, and instead find a perfectly calm process: normal CPU, normal memory, no crash dumps in the error log. The port is listening, but new connections hang or get refused.

This is the slow backend cascade: the classic “Apache outage” in reverse-proxy deployments. Apache is not broken. Every worker is blocked waiting on a backend that has gone slow, and the frontend has run out of execution slots as a consequence.

Apache Slowloris: R-state workers, slow-read attacks, and mod_reqtimeout

Mon, 01 Jan 0001 00:00:00 +0000

Your Apache server stopped serving requests, but nothing looks broken. CPU is normal. Memory is normal. The backend is healthy when you test it directly. Request rate is flat or dropping. Yet clients time out, and the load balancer is pulling the node from rotation.

This is the Slowloris signature: a slow-read denial of service where an attacker opens many connections and drips request data byte by byte, holding each worker in the Reading (R) state indefinitely. With enough slow connections, the entire worker pool is consumed reading requests that never complete. The server is not overloaded. It is held hostage.

Apache SSL certificate expired: the total, preventable HTTPS outage

Mon, 01 Jan 0001 00:00:00 +0000

Every HTTPS client is failing at once. Browsers show NET::ERR_CERT_DATE_INVALID, API clients throw certificate validation errors, monitoring probes time out, and your access logs have gone quiet because no TLS handshake ever completes. Apache itself is running fine: the process is up, workers are idle, and port 443 accepts TCP connections. The failure is at the TLS layer, before any HTTP request exists.

This is one of the most deterministic outages in operations. A certificate has a notAfter date baked into it at issuance. The moment that date passes, every client that validates certificates rejects the connection. There is no degraded mode and no partial failure. It works until the second it does not, and then it works for nobody.

Apache SSL session cache: shmcb sizing, sharing, and silent eviction

Mon, 01 Jan 0001 00:00:00 +0000

A full TLS handshake is the most CPU-intensive thing Apache does per connection. Session resumption exists to avoid paying that cost on every reconnect: a client that recently completed a handshake can resume instead of redoing the asymmetric cryptography. When resumption silently stops working, every connection pays full price, and the only symptom is rising CPU with no error in the log.

The most common way resumption breaks in Apache 2.4 is the server-side session cache: it was never configured (the default is no cache at all), it is not shared across child processes, or it has quietly filled up. A full shmcb cache logs nothing. It evicts old sessions, the resumption rate drops, and CPU climbs. This article covers how the cache works, how session IDs differ from session tickets, how to size shmcb, and how to detect the silent-full condition from resumption rate and cache statistics.

Apache swap thrashing: the latency cliff before the OOM kill

Mon, 01 Jan 0001 00:00:00 +0000

The symptom arrives before the failure. P95 latency doubles, then triples, across every endpoint at once, including static files that should be served from page cache. No 5xx spike yet. No OOM kills in dmesg yet. The Apache parent process is fine, the scoreboard shows workers in normal states, and every request is suddenly slow. This is the swap window: the period between “memory is tight” and “the kernel starts killing children,” where Apache is technically up but effectively degraded.

Apache TLS handshake CPU: broken session resumption and the HTTPS CPU wall

Mon, 01 Jan 0001 00:00:00 +0000

The symptom looks like a capacity problem: Apache processes pinned near 100% CPU, request latency climbing, and no obvious cause in the access log. Traffic is up, but not absurdly. The workers are not stuck on a backend. The scoreboard is busy but not exhausted. And yet the box is melting.

On HTTPS-heavy servers, the usual suspect is the TLS handshake. A full TLS handshake with an RSA-2048 certificate costs on the order of 15 ms of CPU time per connection. Session resumption cuts that cost by roughly 10x. When resumption works, repeat clients skip the expensive public-key operation entirely. When resumption is broken, every connection pays the full price, and at a few hundred new connections per second the math stops working.

Apache Too many open files: file descriptor exhaustion (EMFILE)

Mon, 01 Jan 0001 00:00:00 +0000

Your error log starts showing lines like:

(24)Too many open files: AH00035: access to /some/path failed

or proxy requests begin failing with AH01114 connection errors, and users see intermittent 5xx responses. Sometimes only some requests fail. Sometimes the site looks fine for hours and then breaks at peak. The process is running, the port is open, and nothing in CPU or memory looks wrong.

This is file descriptor exhaustion: one or more Apache child processes have hit their per-process open-file limit and can no longer accept connections, open files, or reach backends. It is a cliff-edge failure, not a gradual one. Everything works until the limit, then everything that needs a new descriptor fails at once.

Apache unexpected restarts: SIGTERM vs SIGUSR1, crashes, and OOM respawns

Mon, 01 Jan 0001 00:00:00 +0000

mod_status shows ServerUptimeSeconds at 340 again, and it was 340 an hour ago too. Something is restarting Apache, and unless you deployed at that exact moment, it is not you. The error log says resuming normal operations every few minutes, or caught SIGTERM, or nothing at all between restarts. Each signature points at a different failure mode.

Apache restarts fall into three buckets: intentional graceful reloads (SIGUSR1), hard restarts (SIGTERM or SIGHUP, often from logrotate or config management), and crash respawns (child segfaults, OOM kills, or the parent dying and being restarted by systemd or a wrapper script). The first is usually harmless. The second drops every in-flight connection. The third means something is actively wrong and will keep happening until you find it.

Apache with mod_php: why per-child memory explodes and when to move to PHP-FPM

Mon, 01 Jan 0001 00:00:00 +0000

Each apache2 or httpd child sits at 80, 120, sometimes 200MB of RSS, the sum of all children creeps toward total RAM, and the OOM killer starts picking off workers at peak traffic. MaxRequestWorkers is already set conservatively, but it does not matter: per-child memory is so large that any worker count high enough to serve your traffic exceeds what the machine can hold.

This is the mod_php signature. When PHP runs as an Apache module, the entire PHP runtime - interpreter, every loaded extension, the opcode cache, and whatever your framework loads per request - lives inside every Apache child process. A child that would cost 10-15MB serving static files now costs 50-100MB or more, and every concurrent connection pays that full price because mod_php forces the prefork MPM, where one connection equals one process.

How Apache HTTPD actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

Most Apache incidents are misdiagnosed for the same reason: the operator is reading signals without knowing which execution model produced them. A scoreboard full of K states is a crisis on prefork and a rounding error on event. “Apache is down” and “the backend is down” look identical from outside a reverse proxy. MaxRequestWorkers means something different depending on whether your concurrency unit is a process or a thread.

This article is the model layer. It covers the abstractions every Apache runbook, alert, and tuning decision depends on: the three MPMs, the scoreboard, worker pool arithmetic, the kernel listen backlog, the module pipeline, mod_proxy connection pools, and APR memory pools. Almost every Apache outage is one of a small set of resource exhaustion patterns, and each maps directly onto a structure described here.