Apache CLOSE_WAIT and TIME_WAIT: connection leaks versus normal churn

You ran ss -tan on an Apache host and saw thousands of connections that are not ESTABLISHED. Some are in TIME_WAIT, some in CLOSE_WAIT, and the numbers look alarming. The first question is not “how do I get rid of them” but “which of these is actually a problem”.

The two states look similar in a socket listing but mean opposite things. TIME_WAIT is the kernel doing its job after a connection closes cleanly; it is normal churn on any busy web server. CLOSE_WAIT means the remote peer closed the connection and Apache never finished closing its side; a sustained, growing CLOSE_WAIT count is an application-side leak that consumes file descriptors and, eventually, workers.

This article is about making that distinction quickly, finding which Apache child owns the leaked sockets, and fixing the actual cause instead of reaching for kernel sysctls.

What this means

A TCP connection close is a two-sided handshake. When the remote peer sends FIN and your side acknowledges it, the local socket enters CLOSE_WAIT. It stays there until the local application calls close(). If the application never does, the socket sits in CLOSE_WAIT indefinitely: no timeout cleans it up. The remote end is gone; only Apache can release the FD.

TIME_WAIT is the mirror image. The side that initiates the close enters TIME_WAIT after the final exchange and stays there for a fixed interval (on Linux, 60 seconds, hardcoded; it is not tunable). Its purpose is to absorb stray packets from the old connection so they cannot corrupt a new connection reusing the same tuple. A high-traffic web server terminating thousands of short-lived connections per minute will legitimately carry tens of thousands of TIME_WAIT sockets. These consume no Apache FDs or workers; they are kernel objects.

The diagnostic rule of thumb:

• CLOSE_WAIT should be near zero. A persistent count, especially one that grows monotonically or clusters against the same remote address, is a leak in Apache or one of its modules.
• TIME_WAIT tracks connection churn. If it correlates with your request rate and turns over on roughly a 60-second cycle, it is background noise, not a symptom.

One modern caveat: with HTTP/2, fewer TCP connections carry more requests through multiplexing. Total connection counts drop while request counts stay flat. Do not alarm on “low” connection counts, and do not use raw connection count as a proxy for load. Judge connection-state data by state distribution and trend, not by absolute totals.

Common causes

The mod_proxy case is the one seen most often in production. Apache maintains pools of backend connections and does not actively scan for peer-closed sockets; a backend that closes idle connections on its own schedule leaves the Apache side in CLOSE_WAIT until the pool slot is reused. If the backend’s keepalive timeout is shorter than Apache’s pool TTL, these accumulate steadily.

Quick checks

All of these are read-only and safe to run on a live server. The -p variants need root (or sudo) to show processes you do not own.

# Connection state breakdown for Apache ports
ss -tanH '( sport = :80 or sport = :443 )' | awk '{print $1}' | sort | uniq -c | sort -rn

# CLOSE_WAIT sockets with owning process (needs root)
ss -tanp state close-wait

# TIME_WAIT count only (-H drops the header line so the count is exact)
ss -tanH state time-wait | wc -l

# CLOSE_WAIT grouped by remote address (leaks cluster)
ss -tanH state close-wait | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head -20

# Per-child FD counts: a leaking child stands out
for pid in $(pgrep 'httpd|apache2'); do
  echo -n "PID $pid: "; ls /proc/$pid/fd 2>/dev/null | wc -l
done

# FD limit for the parent
grep "Max open files" /proc/$(pgrep -o 'httpd|apache2')/limits 2>/dev/null

# Scoreboard state distribution (requires mod_status)
curl -s http://localhost/server-status?auto | grep "Scoreboard:" | \
  awk '{print $2}' | fold -w1 | sort | uniq -c | sort -nr

# Error log: FD exhaustion and proxy connection failures (Debian and RHEL paths)
grep -E "Too many open files|AH01114" /var/log/apache2/error.log /var/log/httpd/error_log 2>/dev/null | tail -20

Notes on the output:

• The cut -d: -f1 grouping breaks on IPv6 listeners; if Apache listens on ::, inspect $5 directly or strip the trailing :port instead.
• Look for CLOSE_WAIT sockets with a non-zero Recv-Q: the peer sent a FIN (and possibly trailing bytes) that Apache never read, a strong sign the socket was abandoned rather than deliberately held.
• The -p flag gives you PID and FD number, which is how you find the specific child holding the leaked sockets.

How to diagnose it

flowchart TD
    A[High non-ESTABLISHED count on Apache ports] --> B{Dominant state?}
    B -->|TIME_WAIT| C[Compare to connection churn rate]
    C -->|Tracks request rate, turns over ~60s| D[Normal kernel behavior: no action]
    C -->|Port exhaustion symptoms| E[Client-side issue: widen ephemeral range on the connecting side]
    B -->|CLOSE_WAIT| F{Count near zero and stable?}
    F -->|Yes| D
    F -->|No, growing or clustered| G[ss -tanp: identify PID and remote address]
    G -->|Remote = backend IP| H[mod_proxy keepalive mismatch or pool handling]
    G -->|Remote = client IPs| I[Module not closing sockets]
    H --> J[Align ttl with backend keepalive timeout]
    I --> J

1. Get the state distribution. Run the breakdown command above. If TIME_WAIT dominates and CLOSE_WAIT is in single or low double digits, you are almost certainly looking at normal churn. Stop here unless you have a specific symptom (port exhaustion, conntrack pressure).

2. Establish the CLOSE_WAIT trend, not the snapshot. Take three samples a few minutes apart. A leak grows monotonically; noise oscillates. A count that climbs at a steady rate per minute is the signature.

3. Identify the remote addresses. Group CLOSE_WAIT sockets by peer. If they concentrate on one or two addresses, that is your backend, and mod_proxy is the suspect. If they spread across many client IPs, the leak is on the frontend side (a module failing to close client connections, often on error or timeout paths).

4. Identify the owning child. ss -tanp state close-wait shows PID and FD. Cross-reference with the per-child FD counts: a leaking child’s FD count grows in step with the CLOSE_WAIT count while its siblings stay flat. If mod_status is enabled with ExtendedStatus On, match the PID to what that child is serving.

5. Check whether workers are being consumed. Sockets in CLOSE_WAIT hold an FD; depending on where they leak (frontend keepalive handling versus proxy pool) they may or may not hold a worker slot. Watch BusyWorkers and the scoreboard distribution alongside the CLOSE_WAIT trend. If workers are accumulating in C (closing) or the pool is shrinking, the leak is starting to hurt, not just accumulate.

6. Check FD headroom. The failure mode is cliff-edge: when a child hits its FD limit, it cannot accept connections or open files, and you get “Too many open files” in the error log plus intermittent 5xx. Compare per-child FD counts against the limit from /proc/<pid>/limits.

Metrics and signals to monitor

Fixes

Fix the mod_proxy keepalive mismatch

If CLOSE_WAIT sockets point at your backends, the root cause is usually a lifecycle mismatch: the backend closes idle pooled connections, and Apache does not notice until it tries to reuse them. Align the pool’s idle timeout with the backend’s keepalive behavior by setting the ttl parameter on ProxyPass (or the balancer member) to a value shorter than the backend’s keepalive timeout, so Apache closes pooled connections before the backend does. Note that ttl expiry means the connection “will be closed at some later time”, not immediately, so leave margin. A graceful restart applies config changes without dropping in-flight requests.

The heavier hammer is disablereuse=On, which forces a fresh backend connection per request and eliminates pooled-socket staleness entirely. The tradeoff is real: you lose connection reuse, add TCP setup latency to every proxied request, and increase connection churn against the backend (which raises TIME_WAIT on the backend side). Prefer tuning ttl first; use disablereuse when the backend’s close behavior is outside your control.

Also confirm your MPM. Under prefork, backend connection reuse happens per child process rather than through a shared pool, so pooling behavior and the effect of these parameters differ from the threaded MPMs. If you are on prefork and seeing backend-side CLOSE_WAIT, verify whether ttl is actually taking effect in your configuration before assuming the mismatch theory holds.

Fix frontend-side leaks

If CLOSE_WAIT sockets are client-facing, some module or handler is failing to close the socket, typically on an error or aborted-request path. There is no config directive that makes Apache close abandoned sockets; the fix is in the code path. Practical steps:

• Identify which URLs the leaking child serves (mod_status PID correlation, or per-URL error rates in the access log).
• Check for module or version bugs: if you are behind on 2.4.x patch releases, upgrade first. Connection and FD lifecycle bugs in modules do get fixed, and running an old minor release while debugging a socket leak is wasted effort.
• As containment, MaxConnectionsPerChild set to a finite value (5000 to 10000) forces periodic child recycling, which releases everything the child holds, including leaked sockets. This bounds the damage; it does not fix the leak. Treat it the same way you treat it for memory leaks: a band-aid that buys time for a real fix.

What to do about TIME_WAIT: mostly nothing

If TIME_WAIT is high but the server is healthy, leave it alone. TIME_WAIT sockets are kernel objects consuming minimal memory each; they do not hold Apache FDs or workers. The failure mode people fear, ephemeral port exhaustion, is a property of the side initiating the connections (your clients, or Apache itself when proxying), not of inbound TIME_WAIT on the server.

The one legitimate sysctl in this space is net.ipv4.tcp_tw_reuse=1, and it only helps for outgoing connections: it matters when Apache as a proxy is churning connections to a backend fast enough to exhaust local ports toward a single backend address. It does nothing for inbound TIME_WAIT from clients.

Do not set net.ipv4.tcp_tw_recycle. It was removed from the kernel in Linux 4.12 and was broken for NAT environments before that. Any runbook recommending it is dangerously outdated.

If you are genuinely exhausting ports toward a backend, the correct fix is more connection tuples: widen ip_local_port_range, reuse connections properly (keepalive=On on the proxy pool), or add backend addresses. Kernel hacks are not the answer.

Prevention

• Monitor connection counts by state, continuously. CLOSE_WAIT trend belongs on every Apache dashboard. If you have ever chased an FD exhaustion incident, it belongs in your alerts at “sustained non-zero and growing”.
• Alert on the trend, not the threshold. A static threshold on CLOSE_WAIT will either page you on noise or miss a slow leak. Alert on sustained growth over tens of minutes.
• Set MaxConnectionsPerChild to a finite value on any deployment with leaky modules (mod_php, mod_perl, custom). It bounds both memory and socket leaks.
• Size FD limits for the worst case. Per-process limit of at least 2x theoretical maximum usage; set via LimitNOFILE in the systemd unit, and remember it takes a full restart, not a graceful reload, to apply.
• Keep proxy ttl aligned with backend keepalive timeouts as a standing configuration review item whenever backend settings change.
• Watch per-child FD counts. The first child to hit the limit fails while the server looks fine. Catching the outlier child early is the whole game.

How Netdata helps

• Netdata collects TCP connection states per socket, so CLOSE_WAIT and TIME_WAIT trends are visible as time series instead of ss snapshots you happened to run during the incident.
• Per-process file descriptor counts against limits show which specific Apache child is accumulating leaked sockets, and how much headroom remains before the cliff edge.
• The Apache collector scrapes server-status?auto, so BusyWorkers, IdleWorkers, and the scoreboard state distribution sit next to the socket-state data; you can see whether a CLOSE_WAIT build-up is starting to consume workers or is still just consuming FDs.
• Error log pattern monitoring surfaces “Too many open files” and proxy connection failures the moment they appear, rather than after users report 5xx errors.
• Because all of these are per-second metrics on one dashboard, the diagnostic sequence in this article (state breakdown, then owning child, then worker impact) becomes correlation across charts instead of a shell session under pressure.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion
• Apache BusyWorkers and IdleWorkers: reading worker utilization from mod_status
• How Apache HTTPD actually works in production: a mental model for operators
• Apache AH00484: server reached MaxRequestWorkers setting - worker pool exhausted
• Apache MaxRequestWorkers tuning: sizing the worker pool against memory
• Apache HTTPD monitoring checklist: the signals every production web server needs
• Apache HTTPD monitoring maturity model: from survival to expert
• Apache scoreboard states explained: what _ S R W K D C L G tell you