What the control plane is doing under the hood, where clusters tend to break, what to monitor as your operation matures, and which mistakes to stop making before the next incident.
Kubernetes is easy to demo and hard to operate.
A pod can be Running while the application inside it is broken. A node can report Ready while its kubelet is too wedged to start anything new. The API server can pass /healthz while etcd is too slow to answer real writes. A service can have endpoints and still be unreachable because conntrack is full, an iptables sync is stalled, or NetworkPolicy quietly denied the packet. A pod evicted for memory pressure tells you something — a pod evicted because another pod broke a node tells you something different.
These guides are written for engineers who already run Kubernetes, not for people learning what a pod is. The goal is to give you the mental model of the control plane, the failure patterns that keep recurring, the monitoring story that catches issues before they page anyone, and the runbooks you wish someone had handed you before your last incident.
Kubernetes is not one thing. It is a stack of cooperating components, and most production failures happen between these layers, not inside any single one of them.
Why this matters: a pod can be Running while the application is broken. NodeReady can be true while the kubelet's PLEG is stalling. The API server can answer /healthz while etcd is too slow to commit writes. A 'service not reachable' might live in kube-proxy, endpoints, DNS, NetworkPolicy, or the application — and each one looks different from the outside.
Most Kubernetes incidents are not exotic. They cluster into a small set of recurring patterns. Recognise the shape, and triage gets dramatically faster.
The API server is up but slow. kubectl responses crawl, controllers stop reconciling, the scheduler queue drains too slowly. Underneath it is usually etcd latency, a misbehaving admission webhook, or APF starvation of legitimate traffic.
One node hits MemoryPressure or DiskPressure. The kubelet evicts pods. Replacement pods land on other nodes, push them over their thresholds, and the cascade keeps going. The cluster looks alive while replicas vanish.
A workload never reaches Ready. The pod is stuck in Pending, ContainerCreating, ImagePullBackOff, or CrashLoopBackOff. Each state has a different root cause and a different runbook.
Services with endpoints that aren't reachable. Pod-to-pod traffic dropping. Conntrack tables full, iptables sync stalled, CNI plugin in an inconsistent state, NetworkPolicy denying silently. Most of these look healthy from outside the data path.
Resolution inside pods is slow or intermittent. Applications hit a 5-second resolver timeout. Upstream services get classified as flaky when the real problem is CoreDNS, the cluster DNS service IP, or ndots.
kubelet or API-server certs expire without warning. Nodes drop out one at a time. kubectl stops authenticating. There is no preceding load event, no obvious trigger, just a date on a certificate nobody was watching.
Kubernetes monitoring works in four practical levels. Each level is a complete operation, not a stepping stone you must climb. Pick the level that matches how much your cluster's reliability matters and how much investment your team has the bandwidth for. Most production clusters should aim for the second level.
Survival monitoring is the floor. With these signals you can answer one question: is the cluster still functioning? You will not learn what broke or why, but you will learn that something broke before users do. Survival is enough for hobby clusters, dev environments, and teams running stateless workloads where Kubernetes reliability is not in the critical path.
Operational monitoring is what most production clusters should target. Once survival signals tell you something is wrong, operational signals tell you what. With this coverage your team can usually diagnose an incident on its own: scheduling failures, evictions, control-plane latency, network drops, image pull issues. If you only invest in one level above survival, this is the one to invest in.
Mature monitoring catches problems before they wake anyone up. APF throttling drifting upward, kubelet certificates approaching expiry, conntrack tables filling, iptables sync time creeping, controller queue depth growing under invisible load. None of these will page you on day one. They turn into pages on day thirty if no one is watching.
Expert signals are reactive, not aspirational. Each one tends to enter your stack the day after a specific incident proved you needed it. kubelet pprof captures, scheduler attempt latency by predicate, audit log forensics, network policy hit-miss tracking, etcd compaction frequency. Most teams never need every signal at this level. Add the ones your incident history tells you to add.
The traps teams keep falling into. Each has a clear, well-known fix. Most teams only learn it after an incident.
Pods without requests get evicted first under pressure and are scheduled greedily by the scheduler. Set CPU and memory requests for every production workload, not just limits.
The API server can answer /readyz while etcd is gasping. Watch etcd commit duration and the API-server-to-etcd latency separately.
One slow admission webhook can stall the entire control plane. Set failurePolicy and timeoutSeconds explicitly, alert on webhook latency, and exclude critical namespaces from webhooks that need them.
Ready only means the kubelet is reporting. The node may still be degraded (PLEG slow, runtime stalls, disk pressure). Watch every node condition individually, not just the Ready summary.
In a multi-tenant or microservice cluster, default-allow is a blast-radius hazard. Default-deny per namespace and explicitly allow the traffic each workload needs.
Most clusters auto-rotate kubelet and apiserver certs. A few don't. The day they expire silently, you find out which kind you have. Verify rotation works on a non-production cluster before you need it.
Each guide is a focused runbook for one symptom or topic. Pick one when you have an incident, or use the categories to learn the area.
If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, jump straight to the symptom that matches what you're seeing.
