Conntrack on Netdata

Docker Container Can't Connect To Internet: Fixes

Mon, 01 Jan 0001 00:00:00 +0000

The host may be online while Docker’s bridge, iptables rules, DNS proxy, or connection tracking table is in a broken state. Applications fail with connection timeouts, package managers stall, and external health checks return unhealthy. Isolate whether the failure is DNS, routing, packet filtering, or daemon state before fixing it.

What this means

Outbound connectivity from a container traverses the container’s network namespace, a veth pair attached to a bridge (docker0 or user-defined), iptables NAT and filter rules managed by Docker’s libnetwork, the host’s routing table, and the upstream physical interface. On user-defined networks, Docker’s embedded DNS resolver at 127.0.0.11 proxies queries to the host’s configured resolvers. On the default bridge network, there is no embedded DNS; containers inherit the host’s /etc/resolv.conf directly. A failure at any layer produces the same symptom: requests time out.

Kubernetes conntrack exhaustion: dropped connections under load

Mon, 01 Jan 0001 00:00:00 +0000

Intermittent connection timeouts under load in Kubernetes often trace to a full nf_conntrack table on the node. Existing TCP sessions stay open, but new connections fail silently. DNS resolution becomes unreliable. Application logs show timeouts to healthy dependencies. The root cause is usually not the application, network policy, or CNI, but kernel connection tracking exhaustion.

Every connection that traverses kube-proxy NAT rules creates an entry in the node’s nf_conntrack table. This finite, node-level table is shared by all workloads and invisible to most application monitoring. When it fills, the kernel drops new connection attempts without sending a TCP reset or ICMP error. The application sees a timeout.

Kubernetes DNS resolution failures inside pods

Mon, 01 Jan 0001 00:00:00 +0000

DNS failures inside pods break service discovery. A single overloaded CoreDNS replica or saturated conntrack table on one node can look like a multi-service outage. Before fixing, determine whether the failure is cluster-wide, node-specific, or workload-specific.

What this means

Kubernetes injects an /etc/resolv.conf into every pod that points to the cluster DNS service, typically CoreDNS. CoreDNS resolves cluster-internal names via the kubernetes plugin and forwards external queries to an upstream resolver. A failure at any point produces the same symptom: the name cannot be resolved.

Kubernetes kube-proxy iptables sync stall: causes and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Pods fail to start. Services intermittently route traffic to dead endpoints. The kube-proxy health endpoint still returns HTTP 200, so the DaemonSet looks healthy, yet rules drift further behind with every sync cycle.

An iptables sync stall is not a crash. It is a slowdown or blockage in the control loop that translates Service and EndpointSlice state into kernel NAT rules. When kube-proxy cannot acquire the global xtables lock, when iptables-restore hangs, or when the rule set grows too large to reconcile within the sync period, the node forwards packets using stale rules. New endpoints are invisible. Terminated pods still receive connections. CNI plugins that also need the xtables lock time out, and pod sandbox creation fails.

Kubernetes kube-proxy IPVS: stale rules and session affinity issues

Mon, 01 Jan 0001 00:00:00 +0000

DNS queries start timing out from one node after a CoreDNS rolling update. A UDP Service returns timeouts for some clients but not others. New Services are unreachable from a specific node while older Services continue to work.

In IPVS mode, kube-proxy programs the kernel’s IPVS table with virtual servers and real servers. The IPVS connection table lives outside kube-proxy’s direct control and outside nf_conntrack. That separation creates two IPVS-specific failure modes: stale rules that diverge from EndpointSlice state, and UDP session affinity that sticks to dead backends long after a pod terminates.

Kubernetes Service Not Reachable: Kube-Proxy, Endpoints & DNS

Mon, 01 Jan 0001 00:00:00 +0000

A Service fails when the chain between the client and backend breaks. That chain depends on EndpointSlices to list healthy pods, kube-proxy to program kernel rules, and cluster DNS to resolve names to ClusterIPs. This guide covers the gap between healthy backend pods and an unreachable Service, focusing on kube-proxy data-plane programming, endpoint state, and DNS dependencies. It does not cover application-level bugs inside the pod.

What This Means

Reachability is a control-loop problem. kube-proxy watches Services and EndpointSlices, then programs iptables, IPVS, or nftables rules to DNAT traffic to healthy endpoints. These rules persist in the kernel if kube-proxy crashes, but updates stop until it resumes. DNS resolution targets the CoreDNS ClusterIP, so a kube-proxy failure often appears as a DNS outage before a direct Service timeout.

NAT and session-table exhaustion: catching it before connections fail

Mon, 01 Jan 0001 00:00:00 +0000

New connections fail while existing ones keep working. Applications report “connection refused” or timeouts. Open SSH sessions stay alive, but new SSH attempts hang. Your monitoring shows the firewall or NAT gateway is up, interfaces are healthy, and CPU is normal. The session or NAT translation table is full.

Session-table exhaustion is a cliff-edge failure. The table degrades gracefully until it hits its limit, then every new connection is denied. Existing flows continue because their entries are already in the table. The symptom pattern is distinctive but easy to misdiagnose as application failure, DNS issues, or upstream provider problems, because the applications are the ones reporting errors.