Kubernetes Operations Guides on Netdata

How the Kubernetes control plane works: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How the Kubernetes control plane works: a mental model for operators

The Kubernetes control plane is often drawn as a single box labeled “Master.” In production, that abstraction fails the moment kubectl hangs, a node disappears, or a Deployment stays pending. The control plane is not a monolith. It is a pipeline of specialized components that hand off state through a single API server, and understanding those handoffs is what lets you triage a cluster outage without guessing.

Kubernetes admission webhook death spiral: detection and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes admission webhook death spiral: detection and recovery

You deploy a mutating webhook to enforce policy. Later, a node drains and the webhook pod evicts. Now no pods can start, including the webhook’s own replacement. The cluster does not crash, but it stops moving. Every kubectl apply hangs. Horizontal autoscalers freeze. Rolling updates stall. This is the admission webhook death spiral: a circular dependency where the webhook must admit a pod it itself requires.

Kubernetes anonymous API access: detection, audit, and lockdown

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes anonymous API access: detection, audit, and lockdown

Anonymous requests to the Kubernetes API server are authenticated as system:anonymous and evaluated as part of the system:unauthenticated group. On many self-managed clusters this behavior is enabled by default, which means requests that arrive without a valid client certificate, bearer token, or other credential are passed to the authorization layer instead of being rejected immediately. Public discovery endpoints and health checks are common legitimate uses, but anonymous access to namespaced resources, secrets, or RBAC objects is a direct security exposure.

Kubernetes API server audit logging: policy, backends, and forensics

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes API server audit logging: policy, backends, and forensics

Kubernetes API server audit logging is the authoritative record of every request that reaches the control plane. It captures the identity of the caller, the resource and verb, the timestamp, the stage, and the outcome. Without it, a security investigation into unauthorized access, a compliance audit, or a postmortem into a failed certificate rotation is built on inference rather than evidence.

Kubernetes API server certificate rotation: detection and grace handling

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes API server certificate rotation: detection and grace handling

Kubernetes control plane certificates created by kubeadm expire after one year. The API server does not auto-rotate its serving certificate. When it expires, etcd rejects control plane connections, kubelets cannot authenticate, and the cluster becomes unreachable. The failure is sudden and total.

This guide covers kubeadm-managed clusters where you own the control plane. Distinguish between the API server serving certificate and the broader control plane bundle, detect expiration before the outage, and renew with minimal disruption.

Kubernetes API server etcd latency: detection and cascading failures

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes API server etcd latency: detection and cascading failures

When etcd slows down, the entire control plane slows with it. A few extra milliseconds on disk fsync turns into hung kubectl commands, backed-up controller queues, and eventually a cluster that cannot schedule pods or update endpoints. Detect the etcd latency cascade, confirm whether storage is the root cause, and break the feedback loop before the cluster becomes effectively read-only.

What this means

etcd serializes every Kubernetes mutation. Every API server write becomes a Raft proposal that must fsync to the WAL before etcd acknowledges it. When the disk under etcd is slow, every fsync waits longer. The API server holds mutating requests open until etcd responds. Requests pile up in the inflight queue. Once the queue hits the limit, the API server returns 429 Too Many Requests. Controllers that depend on writes (scheduler, replica set controller, and others) fall behind and retry. Retries generate more write load. The result is a feedback loop: slow disk -> slow etcd -> slow API server -> retry storm -> amplified etcd load.

Kubernetes API server FlowSchemas and PriorityLevels: design and tuning

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes API server FlowSchemas and PriorityLevels: design and tuning

Before Kubernetes 1.20, the API server protected itself with two global hard limits: --max-requests-inflight and --max-mutating-requests-inflight. Every request, whether a kubelet heartbeat or a runaway controller LIST, competed for the same pool. API Priority and Fairness (APF), enabled by default since 1.20, replaces that coarse model with a two-stage classification and fair-queuing system. It separates requests into priority levels, isolates flows within each level, and rejects or queues traffic before it can starve critical control plane operations.

Kubernetes API server memory pressure: OOM cycle and tuning

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes API server memory pressure: OOM cycle and tuning

Your control plane is crash-looping. The kube-apiserver process climbs toward its container memory limit, the Go garbage collector stalls trying to reclaim space, and the kernel OOM killer terminates it. The replacement pod starts with cold caches; every connected client immediately re-lists watched resources, and memory spikes again before caches warm. The root cause is usually a mismatch between how the API server uses memory and how much headroom you have given it.

Kubernetes API server rate limiting: APF priority levels and starvation

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes API server rate limiting: APF priority levels and starvation

Your API server is running. /healthz returns 200. /readyz passes. Yet nodes drop to NotReady, the scheduler stops placing pods, and controller logs fill with context deadline exceeded. The cluster is not down, but it is frozen. This pattern often points to API Priority and Fairness (APF) starvation: low-priority traffic consumes the API server’s concurrency budget, and critical control plane requests queue or get rejected.

Kubernetes API server slow or unresponsive: causes and fixes

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes API server slow or unresponsive: causes and fixes

When kubectl hangs, controllers log context deadline exceeded, and deployments stall, the Kubernetes API server is usually the bottleneck. It is the single funnel for every read and write to cluster state. Slowness propagates to scheduling, pod lifecycle, service discovery, and external automation.

This article covers operational causes and gives a step-by-step diagnostic flow to run during an incident. Use it to distinguish etcd latency, admission webhook stalls, request saturation, and memory pressure.

Kubernetes API server watch storm: re-list cascades and connection floods

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes API server watch storm: re-list cascades and connection floods

A sudden wall of LIST requests pins API server CPU, climbs memory, spikes etcd read latency, and lags controllers. The culprit is usually a watch storm: hundreds or thousands of clients simultaneously re-listing because their watch connections failed or fell behind. Each re-list triggers an expensive etcd range scan and serializes all matching objects. Under load, this saturates CPU, fills network bandwidth, and can trigger APF throttling or memory pressure. If the API server restarts before the storm subsides, the cycle repeats.

Kubernetes bound service account tokens: rotation, audience, and expiry

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes bound service account tokens: rotation, audience, and expiry

Pods fail with 401 Unauthorized, CSI volume mounts hang with token errors, or security audits surface long-lived credentials that never rotate. Bound tokens are projected, audience-scoped, and short-lived. Legacy tokens are static Secrets that persist forever. After Kubernetes 1.24, both coexist in most upgraded clusters. This guide covers the TokenRequest API lifecycle, kubelet rotation behavior, audience binding, and version-specific changes from 1.24 through 1.33 to help you diagnose auth failures and remove stale credentials safely.

Kubernetes conntrack exhaustion: dropped connections under load

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes conntrack exhaustion: dropped connections under load

Intermittent connection timeouts under load in Kubernetes often trace to a full nf_conntrack table on the node. Existing TCP sessions stay open, but new connections fail silently. DNS resolution becomes unreliable. Application logs show timeouts to healthy dependencies. The root cause is usually not the application, network policy, or CNI, but kernel connection tracking exhaustion.

Every connection that traverses kube-proxy NAT rules creates an entry in the node’s nf_conntrack table. This finite, node-level table is shared by all workloads and invisible to most application monitoring. When it fills, the kernel drops new connection attempts without sending a TCP reset or ICMP error. The application sees a timeout.

Kubernetes container runtime shim failures: containerd, CRI-O troubleshooting

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes container runtime shim failures: containerd, CRI-O troubleshooting

Pods stuck in ContainerCreating, nodes flapping NotReady, and PLEG timeouts that clear only after a node reboot usually point to the container runtime shim layer, not the kubelet or network. The shim sits between the kubelet and the low-level runtime. When it hangs, crashes, or leaks, the kubelet cannot enumerate containers, start sandboxes, or reap terminated pods. Existing containers may keep running, but the node stops accepting new work.

Kubernetes controller-manager leader election failures

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes controller-manager leader election failures

Your Deployment has stopped scaling. Nodes cordoned hours ago are still draining. Garbage collection is paused, and orphaned volumes are not being cleaned up. The kube-controller-manager runs these reconciliation loops, and in an HA cluster only the leader performs work. When leader election fails, the controller-manager exits, and the control plane stops acting on desired state. Existing workloads keep running, but nothing new is managed.

Kubernetes CSI driver failures: detection, recovery, and version skew

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes CSI driver failures: detection, recovery, and version skew

When a workload pod hangs in ContainerCreating with FailedMount or FailedAttach events, the root cause is often a CSI driver pod that has crashed, a node plugin that is missing on the target node, or a version mismatch between the driver and its sidecars. Unlike application pods, CSI drivers sit on the critical path for every volume operation. Their failure is not isolated; it blocks scheduling, provisioning, and recovery.

Kubernetes DaemonSet pods Pending: scheduling and tolerations

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes DaemonSet pods Pending: scheduling and tolerations

A DaemonSet pod in Pending on a node where it should run is a scheduling failure, not a workload crash. Since Kubernetes 1.18, DaemonSet pods pass through the default scheduler like any other pod. The DaemonSet controller creates the pod and pins it to a target node via nodeAffinity, but the scheduler still evaluates predicates: taints, tolerations, resource requests, and node state. If the scheduler rejects the pod, it stays Pending. The controller will not create a replacement; it waits for the scheduler to succeed.

Kubernetes Deployment rollout stuck: stalled rollouts and ready replicas

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes Deployment rollout stuck: stalled rollouts and ready replicas

A Deployment rollout that stalls is a silent capacity leak. The old ReplicaSet scales down, the new ReplicaSet stops halfway, and kubectl rollout status blocks indefinitely. Kubernetes does not automatically recover. The controller sets ProgressDeadlineExceeded only after progressDeadlineSeconds elapses, and takes no corrective action. You need to distinguish between a Pod lifecycle blockage, a readiness probe or gate failure, and a rare controller bug that freezes the rollout entirely.

Kubernetes DNS resolution failures inside pods

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes DNS resolution failures inside pods

DNS failures inside pods break service discovery. A single overloaded CoreDNS replica or saturated conntrack table on one node can look like a multi-service outage. Before fixing, determine whether the failure is cluster-wide, node-specific, or workload-specific.

What this means

Kubernetes injects an /etc/resolv.conf into every pod that points to the cluster DNS service, typically CoreDNS. CoreDNS resolves cluster-internal names via the kubernetes plugin and forwards external queries to an upstream resolver. A failure at any point produces the same symptom: the name cannot be resolved.

Kubernetes etcd defragmentation: when, how, and what breaks

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes etcd defragmentation: when, how, and what breaks

etcd’s database file grows over time even when you are not adding objects. Compaction removes old revisions logically, but the bbolt backend does not shrink the file on disk. Without defragmentation, the gap between physical file size and actual data footprint widens until the cluster hits a NOSPACE alarm and rejects all writes. This guide covers measuring that gap, sequencing defragmentation across an HA cluster without triggering leader elections, and the monitoring signals that predict when you need to act.

Kubernetes etcd disk fsync latency: detection and tuning

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes etcd disk fsync latency: detection and tuning

etcd serializes every Kubernetes write. When disk fsync latency rises, Raft heartbeats stall, leaders step down, and the API server returns 500s and 429s while controllers retry into a death spiral. Unlike CPU or memory pressure, etcd disk latency is invisible until it is catastrophic. This guide shows how to detect it early, isolate the root cause between storage hardware, database size, and configuration, and tune the cluster to survive production load.

Kubernetes etcd snapshot failures: backup, restore, and verification

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes etcd snapshot failures: backup, restore, and verification

An etcd snapshot failure usually surfaces during an incident, not during backup. A snapshot from an unhealthy member, a corrupted transfer to object storage, or a restore that writes no data renders disaster recovery useless. This guide gives the checks, commands, and decision logic to verify snapshot integrity, fix backup failures, and perform clean restores.

What this means

An etcd snapshot captures the entire key-value store at a point in time. Because committed Raft log entries exist on a majority of members, a snapshot from any healthy member contains the full cluster state. The snapshot file includes a SHA-256 hash computed at save time. If that hash does not match after transfer, or if the snapshot is taken while etcd is under NOSPACE alarm or leader instability, the file may be inconsistent.

Kubernetes eviction cascade: when one node failure takes down the cluster

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes eviction cascade: when one node failure takes down the cluster

You see pods entering Evicted status across multiple nodes. Nodes flap between Ready and MemoryPressure or DiskPressure. The scheduler keeps placing replacements, but the new pods are evicted again before they become ready. Workloads never stabilize, and every remediation attempt seems to make the cluster more volatile.

This is a node-pressure eviction cascade. It happens when the scheduler’s view of capacity diverges from the kubelet’s view. One node under pressure evicts pods; those pods land on other nodes that are also overcommitted; those nodes tip into pressure and evict more pods. The result is a cluster-wide feedback loop that looks like a resource shortage but is often a scheduling and configuration problem.

Kubernetes headless service resolution: SRV records and pod discovery

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes headless service resolution: SRV records and pod discovery

You deployed a StatefulSet with a headless Service so peers can discover each other, but nslookup returns NXDOMAIN or only a single IP when several pods are running. Your application might rely on SRV records for port discovery and the lookup returns nothing. Or a pod rescheduled onto a new node and clients kept trying the old IP for minutes because the TTL behavior surprised you.

Kubernetes imagePullSecrets: configuration, propagation, and rotation

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes imagePullSecrets: configuration, propagation, and rotation

Pods stuck in ImagePullBackOff are rarely caused by a missing or incorrect image tag. More often, the kubelet lacks valid registry credentials. Kubernetes uses imagePullSecrets, namespaced Secrets of type kubernetes.io/dockerconfigjson, to inject registry auth into a pod. These can be attached directly to the pod spec or propagated through a ServiceAccount.

This guide covers propagation from registry to kubelet, verification at each link, and rotation without forcing a rolling restart of every workload.

Kubernetes init container fails: blocking main container start

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes init container fails: blocking main container start

An init container failure blocks the entire pod. Until every init container exits zero, no main container starts. In automated clusters, one failing init container can stall a deployment while pods sit in PodInitializing. This guide covers init container retry mechanics, how to read pod status to find the failing step, and how to distinguish application bugs, resource limits, and kubelet issues that leave pods permanently stuck.

Kubernetes Job and CronJob troubleshooting: history, backoff, and missed runs

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes Job and CronJob troubleshooting: history, backoff, and missed runs

You deployed a CronJob to run every minute, but the last successful run was three hours ago. Or a critical data-processing Job failed with BackoffLimitExceeded after six silent retries, leaving a trail of failed Pods and no clear signal about what broke. Batch workloads fail differently from long-running services: they are time-bound, retry-sensitive, and leave debris in etcd if you do not clean them up. Read the failure signals, distinguish retry storms from control plane delays, and fix the root cause without guessing.

Kubernetes kube-proxy and CNI rule conflicts: detection and fix

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes kube-proxy and CNI rule conflicts: detection and fix

Pods stuck in ContainerCreating while the node reports Ready. Services time out despite existing endpoints. Intermittent connection resets during rolling updates. These symptoms usually indicate a conflict between kube-proxy and the container network interface (CNI) plugin over netfilter rules. Both components program the same kernel tables, compete for the same locks, and can corrupt each other’s chains. The data plane degrades while the control plane stays healthy.

Kubernetes kube-proxy iptables sync stall: causes and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes kube-proxy iptables sync stall: causes and recovery

Pods fail to start. Services intermittently route traffic to dead endpoints. The kube-proxy health endpoint still returns HTTP 200, so the DaemonSet looks healthy, yet rules drift further behind with every sync cycle.

An iptables sync stall is not a crash. It is a slowdown or blockage in the control loop that translates Service and EndpointSlice state into kernel NAT rules. When kube-proxy cannot acquire the global xtables lock, when iptables-restore hangs, or when the rule set grows too large to reconcile within the sync period, the node forwards packets using stale rules. New endpoints are invisible. Terminated pods still receive connections. CNI plugins that also need the xtables lock time out, and pod sandbox creation fails.

Kubernetes kube-proxy IPVS: stale rules and session affinity issues

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes kube-proxy IPVS: stale rules and session affinity issues

DNS queries start timing out from one node after a CoreDNS rolling update. A UDP Service returns timeouts for some clients but not others. New Services are unreachable from a specific node while older Services continue to work.

In IPVS mode, kube-proxy programs the kernel’s IPVS table with virtual servers and real servers. The IPVS connection table lives outside kube-proxy’s direct control and outside nf_conntrack. That separation creates two IPVS-specific failure modes: stale rules that diverge from EndpointSlice state, and UDP session affinity that sticks to dead backends long after a pod terminates.

Kubernetes kubelet certificate expired: detection, rotation, and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes kubelet certificate expired: detection, rotation, and recovery

A healthy node suddenly shows NotReady. Pods keep running, but the kubelet stops reporting status. kubectl logs and kubectl exec fail with TLS errors. The cluster event stream is quiet. This is usually an expired kubelet client certificate that failed to rotate.

Every kubelet maintains two independent TLS credentials: a client certificate that authenticates it to the kube-apiserver, and a serving certificate that secures the kubelet’s own HTTPS endpoints. Both typically have a one-year validity. When the client certificate expires, the kubelet cannot authenticate to the API server. The node goes NotReady. Workloads may continue running, but they are unmanaged: no evictions, no probe execution, no status updates, and no new pod scheduling. When the serving certificate expires, metrics-server, kubectl exec, and kubectl logs break even if the node is otherwise Ready.

Kubernetes kubelet goroutine leaks: detection and bisection

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes kubelet goroutine leaks: detection and bisection

A kubelet goroutine leak is a slow-burn failure. The process stays up, the node often remains Ready for hours, and then PLEG timeouts start, sync loops lag, and the kubelet is eventually OOM-killed or unresponsive. By the time the node flips to NotReady, the original leak signature is usually obscured by secondary symptoms.

What this means

The kubelet spawns goroutines for pod workers, probes, API server watches, volume operations, PLEG relists, and CRI calls. In a healthy node, goroutine count correlates with pod density and returns to a stable floor when churn stops. A leak means goroutines are created but never exit, causing three cascading effects:

Kubernetes kubelet memory leak: detection and OOM cycle

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes kubelet memory leak: detection and OOM cycle

Kubelet memory growth ends one of two ways: the process hits its cgroup limit or the node runs out of memory. The kernel OOM killer sends SIGKILL. Systemd restarts kubelet, but the new process has cold caches and immediately runs a full reconciliation pass: relisting all containers, re-syncing every pod status, and re-attaching every volume. On a busy node, that burst spikes CPU and memory, which can push the fresh kubelet back over the edge and create a Ready/NotReady flap cycle.

Kubernetes kubelet not responding: PLEG, runtime, and certificate issues

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes Kubelet Not Responding: PLEG, Runtime & Certificate Issues

A Kubernetes node flipping to NotReady while containers keep running is one of the most confusing production failure modes. The kubelet is the node agent that reconciles API server intent with running containers. When it stops responding or reports unhealthy subsystems, the control plane marks the node NotReady and reschedules workloads, even though the data plane may still serve traffic.

This guide covers three failure domains: Pod Lifecycle Event Generator (PLEG) stalls, container runtime disconnections, and kubelet certificate expiration or rotation failures. Distinguish these symptoms, run safe targeted diagnostics, and apply fixes without blind node reboots.

Kubernetes kubelet pod CIDR changes: detection and rolling fix

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes kubelet pod CIDR changes: detection and rolling fix

Pod sandbox creation errors, nodes registering without an IP range, and cross-node traffic failures are symptoms of pod CIDR drift. The kubelet does not assign its own pod CIDR; the kube-controller-manager node IPAM controller writes the range into node.spec.podCIDR and node.spec.podCIDRs. When that assignment fails or diverges, the result is usually a slow fracture: some nodes host pods while others cannot, or pods on different nodes lose connectivity.

Kubernetes kubelet pprof troubleshooting: capturing heap and goroutine profiles

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes kubelet pprof troubleshooting: capturing heap and goroutine profiles

When a node starts flapping between Ready and NotReady, or kubelet memory climbs steadily while pod count stays flat, node-level metrics like kubelet_goroutines and process_resident_memory_bytes will tell you that the kubelet process is struggling. They will not tell you whether the leak is in the PLEG relist path, the volume manager, or a probe goroutine pool. For that, you need a profile.

Kubernetes kubelet volume deadlock: detection and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes kubelet volume deadlock: detection and recovery

When pods hang in ContainerCreating or Terminating while the node stays Ready=True, the kubelet volume manager is often the culprit. A blocked mount, unmount, attach, or detach operation consumes a goroutine in the volume manager’s finite pool. Once enough operations hang, the queue saturates. Subsequent pods needing volumes stall indefinitely, while pods without volumes start normally. The scheduler, seeing a healthy node, may keep placing volume-bound pods onto the saturated node, deepening the backlog. This guide covers how to confirm a volume deadlock, distinguish it from slow storage, and recover safely.

Kubernetes monitoring checklist: the signals every production cluster needs

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes Monitoring Checklist: The Signals Every Production Cluster Needs

This article is a reference checklist for senior engineers who are wiring up, auditing, or hardening monitoring for a production Kubernetes cluster. It assumes you already understand the control plane architecture and focuses on what to collect, where to find it, and which symptoms matter. Use it during greenfield instrumentation, post-incident gap analysis, or routine health audits.

The signals are grouped by domain. Each entry leads with a short noun phrase, followed by one sentence explaining why it matters, and a concrete warning sign to alert on. Thresholds are drawn from upstream SLOs, kubelet defaults, and etcd operational limits documented in the Kubernetes source and production playbooks. If you run a managed service such as EKS, GKE, or AKS, treat control-plane metrics as provider-mediated; many etcd and API server internals are opaque in those environments.

Kubernetes monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes monitoring maturity model: from survival to expert

Kubernetes failures rarely announce themselves. A slow etcd disk cascades into API latency, which backs up controller workqueues, which delays pod scheduling, which triggers autoscaling, which amplifies the load that caused the original latency spike. Without the right signals, you will debug the autoscaling event while the real problem is a WAL fsync that crossed 100ms ten minutes earlier.

A monitoring maturity model is not a tooling checklist. It is a coverage framework that tells you which signals you are missing and what those gaps cost you during an incident. The four levels below map the progression from “Is the cluster on fire?” to “Why did that single pod take three milliseconds longer to start on node twelve?” Each level assumes the previous and adds signals that change your mean time to detection and your mean time to root cause.

Kubernetes NetworkPolicy debugging: when traffic is denied silently

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes NetworkPolicy debugging: when traffic is denied silently

A pod that could reach its dependency yesterday now times out today. There is no TCP RST, no ICMP unreachable, and often no application log. The packet is dropped in the CNI data plane. If a policy change, namespace reorganization, or cluster upgrade preceded the outage, you are likely dealing with silent NetworkPolicy denial. This guide shows how to confirm it, find the rule or semantic gap responsible, and restore connectivity without opening the cluster.

Kubernetes node CPU saturation: load, throttling, and runqueue depth

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes node CPU saturation: load, throttling, and runqueue depth

Application latency climbs and pods slow down. kubectl top nodes reports 70 percent CPU, so you assume headroom exists. It does not. CPU percent is a time-average that masks micro-bursts, runqueue backlog, and CFS throttling. A container can throttle to a crawl while node utilization looks comfortable, and a node can show 50 percent utilization with every runnable thread queued behind a noisy neighbor. Distinguish node-level CPU contention from limit-induced throttling using runqueue depth, CFS bandwidth metrics, and Pressure Stall Information (PSI).

Kubernetes node DiskPressure: detection, eviction, and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes Node DiskPressure: Detection, Eviction & Recovery

A node reporting DiskPressure is actively shedding workloads. The kubelet has detected that nodefs or imagefs has crossed an eviction threshold. It is garbage collecting images, terminating pods, and applying the node.kubernetes.io/disk-pressure taint to block new scheduling. Existing pods may continue running, but any pod requiring disk for logs, emptyDir volumes, or image pulls is at risk.

Disk pressure builds predictably, unlike memory pressure. This guide covers how the kubelet evaluates disk pressure, how to distinguish nodefs from imagefs exhaustion, how to find the specific consumer, and how to recover without causing a cascading eviction loop.

Kubernetes node MemoryPressure: detection, eviction order, and prevention

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes node MemoryPressure: detection, eviction order, and prevention

Before adding RAM, determine whether kubelet is evicting because workloads are genuinely starving or because memory requests are misaligned with reality.

What this means

Kubelet evaluates memory.available against an eviction threshold. On Linux the default hard threshold is memory.available < 100Mi. Kubelet derives this from cgroup stats, not free -m. It measures working-set memory (RSS plus active file-backed pages) and subtracts that from total capacity. When the threshold is crossed, kubelet sets the node condition MemoryPressure=True and adds the taint node.kubernetes.io/memory-pressure:NoSchedule. New pods are blocked from scheduling until the condition clears.

Kubernetes node NotReady: kubelet, runtime, and network diagnosis

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes Node NotReady: Kubelet, Runtime & Network Diagnosis

When a Kubernetes node becomes NotReady, existing containers usually keep running, but the cluster stops scheduling new pods, removes endpoints from Services, and eventually evicts workloads after the pod eviction timeout. Root causes fall into three domains: kubelet health, container runtime responsiveness, and CNI or control plane connectivity.

What This Means

Kubernetes marks a node NotReady when the kubelet Ready condition is False, or when the node controller has not received a heartbeat within --node-monitor-grace-period (default 40 seconds). The node receives the node.kubernetes.io/not-ready:NoSchedule taint. If the condition persists longer than the pod eviction timeout (default 5 minutes), the controller manager marks pods on the node for rescheduling.

Kubernetes node PIDPressure: detection and remediation

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes node PIDPressure: detection and remediation

PID exhaustion is a cliff-edge failure: once the kernel cannot fork, containers fail to start, health checks fail, and ssh to the node may hang. Kubernetes surfaces this through the PIDPressure node condition, but many clusters ship without PID-based eviction thresholds. Without them, the first symptom is usually EAGAIN or ENOMEM from fork failures, not a kubelet eviction.

This guide shows how to detect PIDPressure before it triggers an outage, distinguish between application leaks, runtime shim accumulation, and kernel limits, and remediate the root cause. You will correlate node-level PID utilization with specific pods, validate kubelet cgroup enforcement, and configure thresholds that provide lead time.

Kubernetes PLEG is not healthy: runtime stalls and node degradation

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes PLEG is not healthy: runtime stalls and node degradation

A node suddenly flips to NotReady with the message “PLEG is not healthy.” Containers on the node keep running, but the control plane evicts workloads and reschedules them elsewhere. New pods cannot start, and existing pods run without health checks or status updates. This is one of the most common kubelet failure modes in production.

The Pod Lifecycle Event Generator (PLEG) is the kubelet subsystem that polls the container runtime every second and emits events when containers start, stop, or change state. When the runtime becomes slow or unresponsive, the PLEG relist loop stalls. If the elapsed time since the last successful relist exceeds three minutes, kubelet declares PLEG unhealthy, marks the node NotReady, and skips pod synchronization.

Kubernetes pod CrashLoopBackOff: causes, diagnosis, and fixes

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes Pod CrashLoopBackOff: Causes, Diagnosis & Fixes

CrashLoopBackOff means a container in a Pod has terminated after starting, and the kubelet is delaying the next restart with exponential backoff. The status describes behavior, not root cause. Underlying failures include application panics, OOM kills, misconfigured liveness probes, missing secrets, or node-level resource pressure.

Use pod status, previous container logs, node conditions, and kubelet events to narrow the cause. Monitor restart rate, node pressure, and probe failures to catch loops before they degrade capacity.

Kubernetes pod creation fails: admission, quota, and CRI errors

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes pod creation fails: admission, quota, and CRI errors

Pre-scheduling failures happen when the API server or container runtime rejects a Pod before the scheduler assigns it. You apply a Deployment, but kubectl get pods returns nothing. Or a Pod hangs in ImagePullBackOff before ContainerCreating. These cases surface as missing Pods, FailedCreate events on ReplicaSets or Jobs, or explicit API rejections. This guide covers admission control, quota and policy limits, and CRI-level image pull and sandbox failures.

Kubernetes pod Evicted: detection, root cause, and prevention

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes pod Evicted: detection, root cause, and prevention

Pods with status Evicted are not application crashes. They are the kubelet’s emergency response to node-level resource pressure. When memory, disk, inodes, or PIDs approach exhaustion, the kubelet terminates pods to reclaim resources and protect node availability. The pod phase changes to Failed with reason Evicted, and the node reports conditions such as MemoryPressure or DiskPressure.

This guide covers node-pressure eviction triggered by the kubelet, not voluntary disruption from kubectl drain or PodDisruptionBudget enforcement.

Kubernetes pod exits immediately: how to diagnose it

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes pod exits immediately: how to diagnose it

When a pod shows Completed or Error with zero restarts, the container exited on its first run. The diagnostic evidence lives in termination metadata, not in a growing restart count. This is distinct from CrashLoopBackOff, where the kubelet has already applied exponential backoff after multiple restarts.

This guide covers how to distinguish a clean exit, an OOM kill, an application crash, and a configuration error using only the kubelet’s reported state and the previous container logs, plus which node-level and control-plane signals to check when the container produced no logs.

Kubernetes pod ImagePullBackOff: registry, auth, and network diagnosis

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes pod ImagePullBackOff: registry, auth, and network diagnosis

ImagePullBackOff means the kubelet cannot pull a required image. After each ErrImagePull failure, the kubelet retries with exponential backoff capped at five minutes. When serializeImagePulls is true, a single slow pull blocks every subsequent pull on that node. Read the exact error from the CRI in pod events, test the registry directly from the node, and fix the root cause without blindly recreating pods.

Kubernetes pod liveness probe killing healthy containers

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes pod liveness probe killing healthy containers

A container that is processing requests, not OOMKilled, and not crashed can still be restarted repeatedly by the kubelet because a liveness probe failed. The application is alive, but the probe says it is not. This usually shows up as a pod stuck in CrashLoopBackOff with Liveness probe failed events, even though application logs show no fatal error. The restarts waste resources, break active connections, and can trigger cascading load on the cluster as other pods absorb the shifted traffic.

Kubernetes pod network isolation: when one node loses pod connectivity

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes pod network isolation: when one node loses pod connectivity

One node in your cluster drops off the pod network. Pods still show Running, but they cannot reach Services, other pods, or external endpoints. The rest of the cluster is unaffected. This is single-node pod network isolation, and it is almost always a node-local CNI or data path failure.

Symptoms are subtle compared to a full cluster outage. Applications log connection timeouts. Health checks fail. New pods stick in ContainerCreating. Because the node often stays Ready, operators may blame the application rather than the network layer.

Kubernetes pod OOMKilled: cgroup limits, evictions, and fixes

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes pod OOMKilled: cgroup limits, evictions, and fixes

A pod status of OOMKilled means the container restarted after the kernel sent SIGKILL because it could not satisfy a memory allocation. There is no graceful shutdown.

Distinguish whether the kill happened at the container cgroup level (a limit you set) or at the node level (a system-wide shortage). Then separate kernel OOM kills from kubelet evictions, identify the correct fix, and prevent recurrence without guessing at memory limits.

Kubernetes pod readiness probe failures: traffic exclusion and debug

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes pod readiness probe failures: traffic exclusion and debug

You check a Deployment and see pods in Running phase but not Ready. Traffic to the Service drops. The containers have not restarted. This is a readiness probe failure. Unlike liveness, readiness does not restart the container. It removes the pod from Service endpoints. The container may be starting slowly, temporarily overloaded, or waiting on a dependency that should not be in the probe path. This guide explains how Kubernetes excludes traffic, how to distinguish readiness from liveness failures, and how to debug the root cause.

Kubernetes Pod Security Standards violations: detection and remediation

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes Pod Security Standards violations: detection and remediation

You apply a Deployment and the ReplicaSet creates zero pods. You roll out a DaemonSet update and new pods vanish without events. You add a namespace label to improve security posture, and CI pipelines fail with opaque admission errors. Pod Security Admission (PSA) blocks non-compliant workloads silently when enforce mode is active. To fix these violations, correlate namespace labels with enforcement outcomes, parse audit log annotations into exact spec changes, distinguish admission-time rejections from runtime failures, and prevent enforcement from silently blocking future rollouts.

Kubernetes pod stuck ContainerCreating: volume, network, and image issues

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes pod stuck ContainerCreating: volume, network, and image issues

A pod stuck in ContainerCreating never produces logs or readiness events. The kubelet accepted the spec but blocked during initialization after scheduling and before the container runtime starts the user process. The dominant failure domains are volume mount deadlocks, CNI sandbox creation failures, and image pull problems. They all surface the same status but need different fixes. This guide shows how to identify the stuck subsystem and resolve it.

Kubernetes pod stuck on volume mount: CSI, permissions, and timeouts

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes pod stuck on volume mount: CSI, permissions, and timeouts

You scale a StatefulSet and the new pods sit in ContainerCreating for ten minutes. The node is Ready. The CSI driver pods are running. kubectl describe shows no FailedMount events, yet the containers never start. The absence of volume events often misleads operators into checking image registries or resource quotas instead of the storage path. The kubelet volume manager is blocked somewhere between attach and mount, and Kubernetes will not retry fast enough to hide the problem.

Kubernetes pod stuck Pending: scheduling failures explained

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes pod stuck Pending: scheduling failures explained

A Deployment scales up and the new replicas stay Pending. No containers start, no logs appear, and kubectl logs returns nothing because no node is assigned yet. When a pod is stuck in Pending, the scheduler has either not yet evaluated it or has rejected every candidate node. Containers cannot start until the pod is assigned, so this blocks rollouts, autoscaling, and recovery.

Kubernetes pod stuck Terminating: finalizers, grace periods, and force delete

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes Pod Stuck Terminating: Finalizers, Grace Periods & Force Delete

A pod stuck in Terminating stays visible in the API server after kubectl delete, sometimes for minutes or hours. Usually a finalizer blocks removal, a CSI volume is still attached, or the container is ignoring SIGTERM. Force deleting without diagnosis orphans containers and can violate StatefulSet guarantees. Check the signals first, then decide whether to wait, patch a finalizer, or force delete.

Kubernetes priority class evictions: how preemption actually works

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes priority class evictions: how preemption actually works

Scheduler preemption and kubelet node-pressure eviction both use Pod Priority, but they follow different rules, interact differently with PodDisruptionBudgets, and produce different failure signatures. Misunderstanding the two mechanisms leads to critical workloads being terminated unexpectedly, high-priority pods staying pending while lower-priority pods run, or system pods being preempted after a PriorityClass change.

The scheduler preempts lower-priority pods to make room for a higher-priority pending pod. The kubelet evicts pods when a node is under resource pressure. A PodDisruptionBudget protects against voluntary disruptions, but the scheduler treats PDB violations during preemption as best-effort, not guaranteed.

Kubernetes PV reclaim policies: Retain, Delete, Recycle in practice

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes PV reclaim policies: Retain, Delete, Recycle in practice

Deleting a PVC does not always delete the underlying storage. If the PV enters Released state and the cloud bill keeps growing, or the application fails after redeployment because the PV is still bound to a deleted claim, the reclaim policy is misaligned with operator intent. This guide explains what happens when a bound PVC is deleted under each reclaim policy, how to recover a Retained volume, why Delete can orphan cloud resources, and why Recycle should not be used in modern clusters. The focus is on practical checks, CSI-specific gotchas, and preventing data loss.

Kubernetes PV volumeBindingMode WaitForFirstConsumer: when and why

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes PV volumeBindingMode WaitForFirstConsumer: when and why

A PersistentVolumeClaim that stays Pending after creation often triggers an incident response. When the StorageClass uses volumeBindingMode: WaitForFirstConsumer, that Pending state is usually intentional. The cluster defers provisioning and binding until a Pod that references the PVC is created and the scheduler tentatively selects a node. This article explains what Immediate and WaitForFirstConsumer mean, why Kubernetes defers binding, where the mechanism appears in production, and the operational tradeoffs you accept when you choose one mode over the other.

Kubernetes PVC stuck Pending: storage class, provisioner, and quota

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes PVC stuck Pending: storage class, provisioner, and quota

A PersistentVolumeClaim stuck in Pending is a storage-layer failure. Unlike a pod that is Pending due to CPU or memory, a PVC in Pending means the cluster cannot provision the volume. The cause usually sits in one of three layers: the StorageClass and its provisioner, namespace-level ResourceQuota limits, or topology and cloud constraints that prevent volume creation and attachment.

When a PVC stays unbound, dependent pods cannot start. StatefulSets stall, rolling updates hang, and storage-dependent applications remain offline.

Kubernetes RBAC permission denied: detection and minimum-permission fix

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes RBAC permission denied: detection and minimum-permission fix

A 403 Forbidden from the Kubernetes API server means the caller was authenticated but RBAC refused the action. In production, this appears as a Deployment stuck creating pods, a CI pipeline failing to patch a ConfigMap, a controller logging repeated forbidden errors, or an operator unable to finalize a custom resource. One missing verb on one resource in one namespace blocks the entire workflow.

Kubernetes ResourceQuota exceeded: detection and remediation

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes ResourceQuota exceeded: detection and remediation

A Deployment looks healthy in kubectl get deployment but the new ReplicaSet has zero pods. A Job is accepted but never creates a pod. A CI/CD pipeline fails with opaque 403 errors from the API server. These symptoms point to ResourceQuota exhaustion. Confirm the quota is the blocker, identify the exhausted resource, and fix it.

What this means

ResourceQuota is a namespace-scoped admission controller that enforces hard aggregate limits on resource consumption. When a tracked resource hits its hard limit, the API server rejects subsequent create requests with HTTP 403 Forbidden and a message containing exceeded quota: <quota-name>. Existing pods keep running; quota is enforced at admission time, not by terminating workloads.

Kubernetes scheduler not scheduling pods: queue depth and failure reasons

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes scheduler not scheduling pods: queue depth and failure reasons

Pods stay Pending for many reasons, but the scheduler process being down is rarely one. More often, pods accumulate in internal queues because the cluster is out of capacity, a control plane dependency stalls the binding cycle, or a filter plugin rejects every candidate node. Distinguishing “unschedulable” (no node fits) from “not scheduling” (the scheduler cannot keep up or the binding cycle is failing) prevents wasted node scaling when the real problem is an etcd latency spike or a volume affinity conflict.

Kubernetes secrets mount failures: detection and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes secrets mount failures: detection and recovery

Pods that reference a Secret volume can hang in ContainerCreating for minutes, crash on startup with missing files, or silently run with stale credentials. The kubelet mounts Secret data as a tmpfs-backed volume and validates the Secret object and its requested keys before any container starts. If the Secret does not exist, if a specific key is absent, or if the kubelet cannot reach the API server, the mount fails. Non-optional mounts block every container from starting. Optional mounts allow startup but leave the mount point empty, which can cause failures later.

Kubernetes Service Not Reachable: Kube-Proxy, Endpoints & DNS

Mon, 01 Jan 0001 00:00:00 +0000

A Service fails when the chain between the client and backend breaks. That chain depends on EndpointSlices to list healthy pods, kube-proxy to program kernel rules, and cluster DNS to resolve names to ClusterIPs. This guide covers the gap between healthy backend pods and an unreachable Service, focusing on kube-proxy data-plane programming, endpoint state, and DNS dependencies. It does not cover application-level bugs inside the pod.

What This Means

Reachability is a control-loop problem. kube-proxy watches Services and EndpointSlices, then programs iptables, IPVS, or nftables rules to DNAT traffic to healthy endpoints. These rules persist in the kernel if kube-proxy crashes, but updates stop until it resumes. DNS resolution targets the CoreDNS ClusterIP, so a kube-proxy failure often appears as a DNS outage before a direct Service timeout.

Kubernetes stale conntrack during rolling updates: intermittent connection resets

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes stale conntrack during rolling updates: intermittent connection resets

You deploy a new version of a service. The rollout reports healthy. Error budgets look fine. Then you notice sporadic connection resets in application logs, a handful of timeout errors, or brief latency spikes that correlate exactly with pod terminations. The failures are intermittent, last only seconds, and never trigger a full outage. This is the stale conntrack race during rolling updates.

Kubernetes StatefulSet pod not ready: ordering, PVCs, and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes StatefulSet pod not ready: ordering, PVCs, and recovery

A StatefulSet pod stuck at 0/1 Ready, Pending, or Unknown blocks the entire chain when the default OrderedReady policy is in effect. The controller creates pods sequentially from ordinal 0 to N-1, so one failure halts all higher ordinals. A volumeClaimTemplate adds dependencies on storage binding, node affinity, and CSI attachment that can outlive the pod. Force-deleting a pod risks violating at-most-one semantics and causing split-brain in quorum-sensitive workloads.

Kubernetes volume snapshot failures: VolumeSnapshot CRDs and CSI snapshotter

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes volume snapshot failures: VolumeSnapshot CRDs and CSI snapshotter

A VolumeSnapshot that never becomes readyToUse, a restore yielding an empty PVC, or a CSI snapshotter sidecar logging GRPC errors with no cluster progress all indicate the same problem: the snapshot pipeline broke between the API server and the storage backend.

This guide covers failure modes between applying a VolumeSnapshot manifest and the storage backend completing the operation. It covers verifying CRDs, the snapshot-controller, the CSI snapshotter sidecar, driver names, capabilities, and secrets, and reading the error signatures each misalignment produces.

Monitoring Kubernetes across vanilla, EKS, GKE, AKS, k3s, and rke2

Mon, 01 Jan 0001 00:00:00 +0000

Monitoring Kubernetes across vanilla, EKS, GKE, AKS, k3s, and rke2

Every Kubernetes cluster exposes the same core control plane components, but what you can actually scrape, query, and alert on depends entirely on how the distribution packages those components. An operator migrating between vanilla kubeadm, Amazon EKS, Google GKE, Azure AKS, k3s, and rke2 needs to know which metrics endpoints exist, which are reachable without extra tooling, and which are gated behind vendor lock-in or architectural quirks. This article maps the native monitoring surface of each variant so you can adapt your collection strategy without discovering gaps during an incident.