Apache and nf_conntrack: table full and the invisible packet drops

Users report timeouts and intermittent connection failures, the load balancer is flapping the backend out of rotation, and yet everything on the Apache host looks fine. BusyWorkers are normal. The 5xx rate is flat. The listen backlog is empty. CPU and memory are unremarkable. Apache is not logging errors, because from Apache’s point of view, nothing is wrong.

The failure is one layer below Apache, in the kernel’s netfilter connection tracking subsystem. When the nf_conntrack table fills up, the kernel emits nf_conntrack: table full, dropping packet and starts silently discarding packets. New SYN packets never reach the TCP stack, so they never reach the listen backlog, so they never reach an Apache worker. Clients see their SYNs dropped and retransmit until they give up: a timeout, not a refusal, which makes it look like a network problem.

This is one of the classic “Apache looks fine but connections fail” root causes. This guide walks through confirming it, sizing the fix, and making sure it does not recur.

What this means

Each tracked connection gets an entry in the kernel connection-tracking hash table; Linux represents it with tuples for both the original and reply directions and records its state, such as NEW, ESTABLISHED, or TIME_WAIT. That table has a hard maximum size, nf_conntrack_max. When the table is full and a packet arrives that would need a new entry, the kernel drops the packet and ratelimited-logs the table full, dropping packet message.

Two properties make this nasty to diagnose:

• The drop happens before the socket layer. Apache’s accept queue, ss output, worker states, and access logs show nothing because the connection never existed as far as userspace is concerned.
• The default timeouts are enormous. The default TCP established timeout is 432000 seconds (5 days). Dead entries linger for days, so on a busy server the table fills with connections that ended long ago.

flowchart LR
  client[Client SYN] --> nf{nf_conntrack}
  nf -->|table has room| tcp[TCP stack]
  nf -->|table full| drop[Packet dropped]
  tcp --> backlog[Listen backlog]
  backlog --> apache[Apache accept and worker]
  drop -.->|client sees| timeout[Timeout and retransmit]
  apache --> healthy[Apache metrics look healthy]

Any host running netfilter-based firewalling or NAT tracks connections. The two biggest amplifiers in web-tier deployments:

• Docker. Every published port (-p) installs DNAT rules, which forces conntrack for that traffic. A busy web container can exhaust the host table.
• Kubernetes with kube-proxy in iptables mode. Every Service IP and NodePort flow creates conntrack entries. Nodes with many pods routinely need a much larger table than the default.

Common causes

Quick checks

All read-only and safe to run during an incident.

# 1. The smoking gun: kernel log
dmesg -T | grep -i "conntrack"

# 2. Current utilization vs ceiling
sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max

# 3. Bucket count (hash table size)
sysctl net.netfilter.nf_conntrack_buckets

# 4. Per-CPU conntrack statistics: look at insert_failed and drop counters
conntrack -S

# 5. The established timeout (default is 432000 = 5 days)
cat /proc/sys/net/netfilter/nf_conntrack_tcp_timeout_established

# 6. Socket state summary: how much churn is TIME_WAIT generating
ss -s

# 7. Confirm Apache itself is NOT the problem: listen backlog should be near zero
ss -ltn | grep -E ':80\s|:443\s'

One caution: conntrack -L dumps the full table. On a host with hundreds of thousands of entries this produces a lot of output and some overhead. Prefer conntrack -S and the sysctls during an active incident; if you need table contents, pipe conntrack -L through wc -l or aggregate it.

How to diagnose it

1. Check dmesg first. dmesg -T | grep -i conntrack. If you see nf_conntrack: table full, dropping packet, you have your root cause. The message is ratelimited, so a few lines can represent a large number of dropped packets.

2. Measure headroom. Divide nf_conntrack_count by nf_conntrack_max. In current kernels, the GC worker starts aggressively early-dropping eligible entries once count exceeds 95% of nf_conntrack_max; when count exceeds max and early drop cannot make room, insertion fails and the packet is dropped. If count is glued to max, the table is the bottleneck.

3. Confirm Apache is innocent. Check the scoreboard and listen backlog. BusyWorkers well under MaxRequestWorkers, an empty Recv-Q on ports 80/443, a normal 5xx rate, and no worker-limit message (AH00484/AH00286/AH00161) in the error log together prove the application tier is fine and the loss is below it. This is the correlation that saves you hours: user-facing failures with a completely clean Apache.

4. Identify what is filling the table. Look at ss -s for TIME_WAIT volume (high churn), check whether Docker DNAT rules or kube-proxy are in play, and check your new-connection rate. Health checks from a load balancer that opens a fresh TCP connection per check, per backend, at short intervals are a classic contributor.

5. Check the odd case: drops with count below max. A sampled count below max can coexist with a prior transient crossing, race-conditioned insert failure, or hash-chain pressure; the table full warning itself is emitted only after count exceeds max and early drop fails. If insert_failed climbs while count is comfortably below max, look at bucket count and race conditions.

Metrics and signals to monitor

Fixes

Raise the table size

The direct fix. Increase the max and, if needed, the bucket count:

# Runtime (takes effect immediately, lost on reboot)
sysctl -w net.netfilter.nf_conntrack_max=1048576

Each entry is kernel- and configuration-dependent non-swappable slab memory. Check the nf_conntrack slab object size in /proc/slabinfo on your kernel before sizing the table; the table is cheap compared to an outage, but do not set it blindly on a small instance.

For persistence, drop a file in /etc/sysctl.d/ rather than relying on /etc/sysctl.conf alone. There is a module-load ordering risk: a sysctl for a namespaced conntrack setting cannot apply until the module and its sysctls exist. Verify the values after reboot; if they did not stick, load the module earlier or reapply the sysctls after it loads.

Note on defaults: they are version-dependent. On kernels before 5.15, nf_conntrack_max defaulted to 4x the bucket count; since 5.15, max defaults to the same value as buckets, and the bucket default on systems with more than 4GB of RAM is 262144. Check your actual values rather than assuming.

Shorten the timeouts

The 5-day established timeout is the single biggest reason tables fill on busy servers. Reducing it reclaims entries far sooner:

# Example: 2 hours instead of 5 days. Pick a value above your longest legitimate idle connection.
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=7200

Tradeoff: if you have legitimately long-idle TCP connections (some database proxies, WebSockets without application-level pings, long-lived LB connections without keepalive), an aggressive timeout will expire the tracking state mid-connection. On a pure HTTP web tier this is rarely a problem, but verify what long-lived flows exist before going below a few hours.

Reduce connection churn at the source

Fewer, longer-lived connections means a smaller, stabler table:

• Enable Apache keepalive so clients reuse connections instead of opening one per request.
• Use HTTP/2 where possible; multiplexing collapses many requests onto one connection.
• Configure your load balancer to use persistent connections to backends rather than a fresh connection per health check or per request.
• Under heavy TIME_WAIT churn on a single IP:port pair, net.ipv4.tcp_tw_reuse can help when it is protocol-safe. The kernel documentation explicitly says it should not be changed without expert advice; verify your kernel and traffic mix before changing it.

Exempt high-volume traffic from tracking

For traffic you do not need stateful firewalling or NAT on, you can bypass conntrack entirely. With iptables this is the NOTRACK target in the raw table; with nftables it is the notrack statement. Typical use: exemption for high-rate internal health-check or monitoring traffic, or for a trusted LB-to-backend path.

The tradeoff is real: untracked traffic cannot match stateful rules (-m state / ct state), cannot be NATed, and does not benefit from conntrack-based helpers. Apply NOTRACK narrowly, to specific source/destination pairs, never as a blanket bypass on an internet-facing host.

If Docker or Kubernetes is involved

Do not try to unload or disable conntrack on a host running Docker: Docker’s bridge networking and DNS depend on it, and removing the module breaks container networking. The correct moves are sizing the table for the combined host plus container load, and shortening timeouts. On Kubernetes nodes, treat a large conntrack table as a standard part of node provisioning, not a remediation.

Prevention

• Alert on the ratio, not the outage. Drops start at 100%, but the leading indicator is nf_conntrack_count / nf_conntrack_max trending up over weeks. Alert at 70-80% and you fix it on a Tuesday instead of during a traffic spike.
• Persist the tuning properly. Whatever you set at runtime must survive reboot and the module-load ordering race. Verify after a reboot that the values stuck.
• Include conntrack in capacity reviews. Connection rate growth fills the table just as surely as request rate growth fills the worker pool. On any high-traffic or containerized web tier, conntrack utilization belongs in your standard signal set.
• Prefer connection reuse everywhere. Keepalive, HTTP/2, and persistent LB connections reduce churn, which is the cheapest conntrack capacity you can buy.

How Netdata helps

• Conntrack utilization as a first-class chart: Netdata collects the kernel’s conntrack count and max, so the ratio is visible per second and alarmable before the table fills.
• The correlation that cracks the case: Apache worker utilization, request rate, and 5xx rate charted alongside conntrack utilization makes the “Apache healthy, clients failing” signature obvious in one view instead of across three terminals.
• TCP connection state visibility: TIME_WAIT and established-connection churn is charted from the system TCP stack, so you can see whether the table is filling from churn or from long-lived flows.
• Container and node context together: on Docker and Kubernetes hosts, per-container network activity next to node-level conntrack saturation shows which workload is driving table growth.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• Apache 500 Internal Server Error: modules, handlers, and misconfiguration
• Apache 502 Bad Gateway: a backend that returned an invalid response
• Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion
• Apache 504 Gateway Timeout: slow backends, ProxyTimeout, and worker pile-up
• Apache 5xx error rate: 500 vs 502 vs 503 vs 504 and what each one means
• Apache AH00558: Could not reliably determine the server’s fully qualified domain name
• Apache backend response time: telling ‘Apache is slow’ from ’the backend is slow’
• Apache balancer member in error state: reading balancer-manager and failover
• Apache BusyWorkers and IdleWorkers: reading worker utilization from mod_status
• Apache CLOSE_WAIT and TIME_WAIT: connection leaks versus normal churn
• Apache error log monitoring: severity levels, AH codes, and what to alert on
• How Apache HTTPD actually works in production: a mental model for operators