Users report timeouts and intermittent connection failures, the load balancer is flapping the backend out of rotation, and yet everything on the Apache host looks fine. BusyWorkers are normal. The 5xx rate is flat. The listen backlog is empty. CPU and memory are unremarkable. Apache is not logging errors, because from Apache’s point of view, nothing is wrong.

The failure is one layer below Apache, in the kernel’s netfilter connection tracking subsystem. When the nf_conntrack table fills up, the kernel emits nf_conntrack: table full, dropping packet and starts silently discarding packets. New SYN packets never reach the TCP stack, so they never reach the listen backlog, so they never reach an Apache worker. Clients see their SYNs dropped and retransmit until they give up: a timeout, not a refusal, which makes it look like a network problem.

This is one of the classic “Apache looks fine but connections fail” root causes. This guide walks through confirming it, sizing the fix, and making sure it does not recur.

What this means

Every packet that traverses a netfilter hook (iptables or nftables rules, NAT, stateful firewall matches) gets an entry in a kernel hash table that tracks the connection’s state: NEW, ESTABLISHED, TIME_WAIT, and so on. That table has a hard maximum size, nf_conntrack_max. When the table is full and a packet arrives that would need a new entry, the kernel drops the packet and ratelimited-logs the table full, dropping packet message.

Two properties make this nasty to diagnose:

  • The drop happens before the socket layer. Apache’s accept queue, ss output, worker states, and access logs show nothing because the connection never existed as far as userspace is concerned.
  • The default timeouts are enormous. The default TCP established timeout is 432000 seconds (5 days). Dead entries linger for days, so on a busy server the table fills with connections that ended long ago.
flowchart LR
  client[Client SYN] --> nf{nf_conntrack}
  nf -->|table has room| tcp[TCP stack]
  nf -->|table full| drop[Packet dropped]
  tcp --> backlog[Listen backlog]
  backlog --> apache[Apache accept and worker]
  drop -.->|client sees| timeout[Timeout and retransmit]
  apache --> healthy[Apache metrics look healthy]

Any host running netfilter-based firewalling or NAT tracks connections. The two biggest amplifiers in web-tier deployments:

  • Docker. Every published port (-p) installs DNAT rules, which forces conntrack for that traffic. A busy web container can exhaust the host table.
  • Kubernetes with kube-proxy in iptables mode. Every Service IP and NodePort flow creates conntrack entries. Nodes with many pods routinely need a much larger table than the default.

Common causes

CauseWhat it looks likeFirst thing to check
Table simply too small for connection ratenf_conntrack_count pinned at nf_conntrack_max, table full in dmesgsysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
Stale entries from the 5-day established timeoutCount grows monotonically, entries far outlive real connectionscat /proc/sys/net/netfilter/nf_conntrack_tcp_timeout_established
High connection churn (short-lived connections, no keepalive, health checks, scanners)High new-connection rate, many TIME_WAIT entriesconntrack -S and connection state distribution via ss -s
Docker or kube-proxy NAT amplifying tracked flowsHost running containers; drops correlate with container traffic, not just Apacheiptables -t nat -L -n rule volume; count vs max
Hash pressure with too few bucketsDrops even though count is below max (rare; hash chains degenerate)Compare nf_conntrack_buckets to max; see “Fixes”

Quick checks

All read-only and safe to run during an incident.

# 1. The smoking gun: kernel log
dmesg -T | grep -i "conntrack"

# 2. Current utilization vs ceiling
sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max

# 3. Bucket count (hash table size)
sysctl net.netfilter.nf_conntrack_buckets

# 4. Per-CPU conntrack statistics: look at insert_failed and drop counters
conntrack -S

# 5. The established timeout (default is 432000 = 5 days)
cat /proc/sys/net/netfilter/nf_conntrack_tcp_timeout_established

# 6. Socket state summary: how much churn is TIME_WAIT generating
ss -s

# 7. Confirm Apache itself is NOT the problem: listen backlog should be near zero
ss -ltn | grep -E ':80\s|:443\s'

One caution: conntrack -L dumps the full table. On a host with hundreds of thousands of entries this produces a lot of output and some overhead. Prefer conntrack -S and the sysctls during an active incident; if you need table contents, pipe conntrack -L through wc -l or aggregate it.

How to diagnose it

  1. Check dmesg first. dmesg -T | grep -i conntrack. If you see nf_conntrack: table full, dropping packet, you have your root cause. The message is ratelimited, so a few lines can represent a large number of dropped packets.

  2. Measure headroom. Divide nf_conntrack_count by nf_conntrack_max. Near the ceiling the kernel’s early-drop logic starts evicting old entries to make room; at 100%, insertions fail and packets drop. If count is glued to max, the table is the bottleneck.

  3. Confirm Apache is innocent. Check the scoreboard and listen backlog. BusyWorkers well under MaxRequestWorkers, an empty Recv-Q on ports 80/443, a normal 5xx rate, and no AH00484 in the error log together prove the application tier is fine and the loss is below it. This is the correlation that saves you hours: user-facing failures with a completely clean Apache.

  4. Identify what is filling the table. Look at ss -s for TIME_WAIT volume (high churn), check whether Docker DNAT rules or kube-proxy are in play, and check your new-connection rate. Health checks from a load balancer that opens a fresh TCP connection per check, per backend, at short intervals are a classic contributor.

  5. Check the odd case: drops with count below max. Some operators see table full while count is well under max, caused by hash collision pressure when buckets are too few relative to max, or by per-CPU accounting races. If count is comfortably below max but insert_failed in conntrack -S is climbing, look at bucket count.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
nf_conntrack_count / nf_conntrack_max ratioDirect saturation measure of the tableSustained above 70-80%
insert_failed and drop counters (conntrack -S)Proof packets are being dropped right nowAny sustained increase
dmesg table full messagesKernel explicitly reporting dropsAny occurrence in production
New connection rate to 80/443Drives table growth; churn fills it faster than steady loadSpike without matching request-rate growth
TIME_WAIT count (ss -s)Closed connections still hold conntrack entries until timeoutLarge and growing relative to established
Apache BusyWorkers and listen backlogNegative evidence: proves the app tier is healthyHealthy Apache plus failing clients = look at conntrack

Fixes

Raise the table size

The direct fix. Increase the max and, if needed, the bucket count:

# Runtime (takes effect immediately, lost on reboot)
sysctl -w net.netfilter.nf_conntrack_max=1048576

Each entry costs roughly 300-350 bytes of non-swappable kernel memory, so 1M entries costs on the order of 320MB of RAM. Size accordingly: the table is cheap compared to an outage, but do not set it blindly on a small instance.

For persistence, drop a file in /etc/sysctl.d/ rather than relying on /etc/sysctl.conf alone. There is a known boot-order race: sysctl may be applied before the nf_conntrack module loads, in which case your setting is silently ignored. The standard workaround is a udev rule that reapplies netfilter sysctls when the module loads.

Note on defaults: they are version-dependent. On kernels before 5.15, nf_conntrack_max defaulted to 4x the bucket count; since 5.15, max defaults to the same value as buckets, and the bucket default on systems with more than 4GB of RAM is 262144. Check your actual values rather than assuming.

Shorten the timeouts

The 5-day established timeout is the single biggest reason tables fill on busy servers. Reducing it reclaims entries far sooner:

# Example: 2 hours instead of 5 days. Pick a value above your longest legitimate idle connection.
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=7200

Tradeoff: if you have legitimately long-idle TCP connections (some database proxies, WebSockets without application-level pings, long-lived LB connections without keepalive), an aggressive timeout will expire the tracking state mid-connection. On a pure HTTP web tier this is rarely a problem, but verify what long-lived flows exist before going below a few hours.

Reduce connection churn at the source

Fewer, longer-lived connections means a smaller, stabler table:

  • Enable Apache keepalive so clients reuse connections instead of opening one per request.
  • Use HTTP/2 where possible; multiplexing collapses many requests onto one connection.
  • Configure your load balancer to use persistent connections to backends rather than a fresh connection per health check or per request.
  • Under heavy TIME_WAIT churn on a single IP:port pair, net.ipv4.tcp_tw_reuse helps. Verify your kernel and client mix before enabling it.

Exempt high-volume traffic from tracking

For traffic you do not need stateful firewalling or NAT on, you can bypass conntrack entirely. With iptables this is the NOTRACK target in the raw table; with nftables it is the notrack statement. Typical use: exemption for high-rate internal health-check or monitoring traffic, or for a trusted LB-to-backend path.

The tradeoff is real: untracked traffic cannot match stateful rules (-m state / ct state), cannot be NATed, and does not benefit from conntrack-based helpers. Apply NOTRACK narrowly, to specific source/destination pairs, never as a blanket bypass on an internet-facing host.

If Docker or Kubernetes is involved

Do not try to unload or disable conntrack on a host running Docker: Docker’s bridge networking and DNS depend on it, and removing the module breaks container networking. The correct moves are sizing the table for the combined host plus container load, and shortening timeouts. On Kubernetes nodes, treat a large conntrack table as a standard part of node provisioning, not a remediation.

Prevention

  • Alert on the ratio, not the outage. Drops start at 100%, but the leading indicator is nf_conntrack_count / nf_conntrack_max trending up over weeks. Alert at 70-80% and you fix it on a Tuesday instead of during a traffic spike.
  • Persist the tuning properly. Whatever you set at runtime must survive reboot and the module-load ordering race. Verify after a reboot that the values stuck.
  • Include conntrack in capacity reviews. Connection rate growth fills the table just as surely as request rate growth fills the worker pool. On any high-traffic or containerized web tier, conntrack utilization belongs in your standard signal set.
  • Prefer connection reuse everywhere. Keepalive, HTTP/2, and persistent LB connections reduce churn, which is the cheapest conntrack capacity you can buy.

How Netdata helps

  • Conntrack utilization as a first-class chart: Netdata collects the kernel’s conntrack count and max, so the ratio is visible per second and alarmable before the table fills.
  • The correlation that cracks the case: Apache worker utilization, request rate, and 5xx rate charted alongside conntrack utilization makes the “Apache healthy, clients failing” signature obvious in one view instead of across three terminals.
  • TCP connection state visibility: TIME_WAIT and established-connection churn is charted from the system TCP stack, so you can see whether the table is filling from churn or from long-lived flows.
  • Container and node context together: on Docker and Kubernetes hosts, per-container network activity next to node-level conntrack saturation shows which workload is driving table growth.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.