Kubelet on Netdata

K8s Kubelet Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

What is K8s Kubelet?

Kubelet is an agent that runs on each node in the cluster. It makes sure that containers are running in a pod.

Monitoring K8s Kubelet with Netdata

The prerequisite for monitoring K8s Kubelet with Netdata is to Netdata installed on your system.

Netdata auto discovers hundreds of services, and for those it doesn’t turning on manual discovery is a one line configuration. For more information on configuring Netdata for K8s Kubelet monitoring please read the collector documentation.

Kubelet Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Kubelet Monitoring

In today’s fast-paced DevOps environments, monitoring the Kubelet effectively is crucial for maintaining healthy Kubernetes clusters. Understanding its metrics allows teams to ensure high availability and performance efficiency for their applications.

What Is Kubelet?

Kubelet is a core component of Kubernetes that runs on each node in the cluster. It ensures that containers are running as expected, watching for changes in Pod specifications and reporting back to the Kubernetes API server.

Kubernetes eviction cascade: when one node failure takes down the cluster

Mon, 01 Jan 0001 00:00:00 +0000

You see pods entering Evicted status across multiple nodes. Nodes flap between Ready and MemoryPressure or DiskPressure. The scheduler keeps placing replacements, but the new pods are evicted again before they become ready. Workloads never stabilize, and every remediation attempt seems to make the cluster more volatile.

This is a node-pressure eviction cascade. It happens when the scheduler’s view of capacity diverges from the kubelet’s view. One node under pressure evicts pods; those pods land on other nodes that are also overcommitted; those nodes tip into pressure and evict more pods. The result is a cluster-wide feedback loop that looks like a resource shortage but is often a scheduling and configuration problem.

Kubernetes kubelet certificate expired: detection, rotation, and recovery

Mon, 01 Jan 0001 00:00:00 +0000

A healthy node suddenly shows NotReady. Pods keep running, but the kubelet stops reporting status. kubectl logs and kubectl exec fail with TLS errors. The cluster event stream is quiet. This is usually an expired kubelet client certificate that failed to rotate.

Every kubelet maintains two independent TLS credentials: a client certificate that authenticates it to the kube-apiserver, and a serving certificate that secures the kubelet’s own HTTPS endpoints. Both typically have a one-year validity. When the client certificate expires, the kubelet cannot authenticate to the API server. The node goes NotReady. Workloads may continue running, but they are unmanaged: no evictions, no probe execution, no status updates, and no new pod scheduling. When the serving certificate expires, metrics-server, kubectl exec, and kubectl logs break even if the node is otherwise Ready.

Kubernetes kubelet memory leak: detection and OOM cycle

Mon, 01 Jan 0001 00:00:00 +0000

Kubelet memory growth ends one of two ways: the process hits its cgroup limit or the node runs out of memory. The kernel OOM killer sends SIGKILL. Systemd restarts kubelet, but the new process has cold caches and immediately runs a full reconciliation pass: relisting all containers, re-syncing every pod status, and re-attaching every volume. On a busy node, that burst spikes CPU and memory, which can push the fresh kubelet back over the edge and create a Ready/NotReady flap cycle.

Kubernetes kubelet not responding: PLEG, runtime, and certificate issues

Mon, 01 Jan 0001 00:00:00 +0000

A Kubernetes node flipping to NotReady while containers keep running is one of the most confusing production failure modes. The kubelet is the node agent that reconciles API server intent with running containers. When it stops responding or reports unhealthy subsystems, the control plane marks the node NotReady and reschedules workloads, even though the data plane may still serve traffic.

This guide covers three failure domains: Pod Lifecycle Event Generator (PLEG) stalls, container runtime disconnections, and kubelet certificate expiration or rotation failures. Distinguish these symptoms, run safe targeted diagnostics, and apply fixes without blind node reboots.

Kubernetes Monitoring Checklist: The Signals Every Production Cluster Needs

Mon, 01 Jan 0001 00:00:00 +0000

This article is a reference checklist for senior engineers who are wiring up, auditing, or hardening monitoring for a production Kubernetes cluster. It assumes you already understand the control plane architecture and focuses on what to collect, where to find it, and which symptoms matter. Use it during greenfield instrumentation, post-incident gap analysis, or routine health audits.

The signals are grouped by domain. Each entry leads with a short noun phrase, followed by one sentence explaining why it matters, and a concrete warning sign to alert on. Thresholds are drawn from upstream SLOs, kubelet defaults, and etcd operational limits documented in the Kubernetes source and production playbooks. If you run a managed service such as EKS, GKE, or AKS, treat control-plane metrics as provider-mediated; many etcd and API server internals are opaque in those environments.

Kubernetes node CPU saturation: load, throttling, and runqueue depth

Mon, 01 Jan 0001 00:00:00 +0000

Application latency climbs and pods slow down. kubectl top nodes reports 70 percent CPU, so you assume headroom exists. It does not. CPU percent is a time-average that masks micro-bursts, runqueue backlog, and CFS throttling. A container can throttle to a crawl while node utilization looks comfortable, and a node can show 50 percent utilization with every runnable thread queued behind a noisy neighbor. Distinguish node-level CPU contention from limit-induced throttling using runqueue depth, CFS bandwidth metrics, and Pressure Stall Information (PSI).

Kubernetes node DiskPressure: detection, eviction, and recovery

Mon, 01 Jan 0001 00:00:00 +0000

A node reporting DiskPressure is actively shedding workloads. The kubelet has detected that nodefs or imagefs has crossed an eviction threshold. It is garbage collecting images, terminating pods, and applying the node.kubernetes.io/disk-pressure taint to block new scheduling. Existing pods may continue running, but any pod requiring disk for logs, emptyDir volumes, or image pulls is at risk.

Disk pressure builds predictably, unlike memory pressure. This guide covers how the kubelet evaluates disk pressure, how to distinguish nodefs from imagefs exhaustion, how to find the specific consumer, and how to recover without causing a cascading eviction loop.

Kubernetes node MemoryPressure: detection, eviction order, and prevention

Mon, 01 Jan 0001 00:00:00 +0000

Before adding RAM, determine whether kubelet is evicting because workloads are genuinely starving or because memory requests are misaligned with reality.

What this means

Kubelet evaluates memory.available against an eviction threshold. On Linux the default hard threshold is memory.available < 100Mi. Kubelet derives this from cgroup stats, not free -m. It measures working-set memory (RSS plus active file-backed pages) and subtracts that from total capacity. When the threshold is crossed, kubelet sets the node condition MemoryPressure=True and adds the taint node.kubernetes.io/memory-pressure:NoSchedule. New pods are blocked from scheduling until the condition clears.

Kubernetes node NotReady: kubelet, runtime, and network diagnosis

Mon, 01 Jan 0001 00:00:00 +0000

When a Kubernetes node becomes NotReady, existing containers usually keep running, but the cluster stops scheduling new pods, removes endpoints from Services, and eventually evicts workloads after the pod eviction timeout. Root causes fall into three domains: kubelet health, container runtime responsiveness, and CNI or control plane connectivity.

What This Means

Kubernetes marks a node NotReady when the kubelet Ready condition is False, or when the node controller has not received a heartbeat within --node-monitor-grace-period (default 40 seconds). The node receives the node.kubernetes.io/not-ready:NoSchedule taint. If the condition persists longer than the pod eviction timeout (default 5 minutes), the controller manager marks pods on the node for rescheduling.

Kubernetes node PIDPressure: detection and remediation

Mon, 01 Jan 0001 00:00:00 +0000

PID exhaustion is a cliff-edge failure: once the kernel cannot fork, containers fail to start, health checks fail, and ssh to the node may hang. Kubernetes surfaces this through the PIDPressure node condition, but many clusters ship without PID-based eviction thresholds. Without them, the first symptom is usually EAGAIN or ENOMEM from fork failures, not a kubelet eviction.

This guide shows how to detect PIDPressure before it triggers an outage, distinguish between application leaks, runtime shim accumulation, and kernel limits, and remediate the root cause. You will correlate node-level PID utilization with specific pods, validate kubelet cgroup enforcement, and configure thresholds that provide lead time.

Kubernetes Operations Guides

Mon, 01 Jan 0001 00:00:00 +0000

Learn what actually breaks in production Kubernetes clusters, how to diagnose it, and what to monitor so the next incident is shorter than the last one.

Kubernetes PLEG is not healthy: runtime stalls and node degradation

Mon, 01 Jan 0001 00:00:00 +0000

A node suddenly flips to NotReady with the message “PLEG is not healthy.” Containers on the node keep running, but the control plane evicts workloads and reschedules them elsewhere. New pods cannot start, and existing pods run without health checks or status updates. This is one of the most common kubelet failure modes in production.

The Pod Lifecycle Event Generator (PLEG) is the kubelet subsystem that polls the container runtime every second and emits events when containers start, stop, or change state. When the runtime becomes slow or unresponsive, the PLEG relist loop stalls. If the elapsed time since the last successful relist exceeds three minutes, kubelet declares PLEG unhealthy, marks the node NotReady, and skips pod synchronization.

Kubernetes pod CrashLoopBackOff: causes, diagnosis, and fixes

Mon, 01 Jan 0001 00:00:00 +0000

CrashLoopBackOff means a container in a Pod has terminated after starting, and the kubelet is delaying the next restart with exponential backoff. The status describes behavior, not root cause. Underlying failures include application panics, OOM kills, misconfigured liveness probes, missing secrets, or node-level resource pressure.

Use pod status, previous container logs, node conditions, and kubelet events to narrow the cause. Monitor restart rate, node pressure, and probe failures to catch loops before they degrade capacity.

Kubernetes pod creation fails: admission, quota, and CRI errors

Mon, 01 Jan 0001 00:00:00 +0000

Pre-scheduling failures happen when the API server or container runtime rejects a Pod before the scheduler assigns it. You apply a Deployment, but kubectl get pods returns nothing. Or a Pod hangs in ImagePullBackOff before ContainerCreating. These cases surface as missing Pods, FailedCreate events on ReplicaSets or Jobs, or explicit API rejections. This guide covers admission control, quota and policy limits, and CRI-level image pull and sandbox failures.

What this means

Pod creation is a pipeline. The API server first authenticates and authorizes the request, then runs the Pod spec through admission controllers: mutating webhooks, validating webhooks, LimitRanger, ResourceQuota, and PodSecurity. If any controller rejects the request, the Pod object is never persisted to etcd. If admission succeeds, the object is stored, the scheduler assigns a node, and the kubelet asks the container runtime (via CRI) to create the pod sandbox and pull images. Failures in this guide happen before the pod is running or scheduled.

Kubernetes pod Evicted: detection, root cause, and prevention

Mon, 01 Jan 0001 00:00:00 +0000

Pods with status Evicted are not application crashes. They are the kubelet’s emergency response to node-level resource pressure. When memory, disk, inodes, or PIDs approach exhaustion, the kubelet terminates pods to reclaim resources and protect node availability. The pod phase changes to Failed with reason Evicted, and the node reports conditions such as MemoryPressure or DiskPressure.

This guide covers node-pressure eviction triggered by the kubelet, not voluntary disruption from kubectl drain or PodDisruptionBudget enforcement.

Kubernetes pod exits immediately: how to diagnose it

Mon, 01 Jan 0001 00:00:00 +0000

When a pod shows Completed or Error with zero restarts, the container exited on its first run. The diagnostic evidence lives in termination metadata, not in a growing restart count. This is distinct from CrashLoopBackOff, where the kubelet has already applied exponential backoff after multiple restarts.

This guide covers how to distinguish a clean exit, an OOM kill, an application crash, and a configuration error using only the kubelet’s reported state and the previous container logs, plus which node-level and control-plane signals to check when the container produced no logs.

Kubernetes pod ImagePullBackOff: registry, auth, and network diagnosis

Mon, 01 Jan 0001 00:00:00 +0000

ImagePullBackOff means the kubelet cannot pull a required image. After each ErrImagePull failure, the kubelet retries with exponential backoff capped at five minutes. When serializeImagePulls is true, a single slow pull blocks every subsequent pull on that node. Read the exact error from the CRI in pod events, test the registry directly from the node, and fix the root cause without blindly recreating pods.

What this means

The kubelet asks the container runtime to pull any image not cached locally. The runtime resolves the registry, authenticates, downloads layers, and unpacks them into node storage. A failure at any step returns a CRI error that the kubelet surfaces as a pod event. kubectl get pod shows only the state; the reason lives in the events.

Kubernetes pod OOMKilled: cgroup limits, evictions, and fixes

Mon, 01 Jan 0001 00:00:00 +0000

A pod status of OOMKilled means the container restarted after the kernel sent SIGKILL because it could not satisfy a memory allocation. There is no graceful shutdown.

Distinguish whether the kill happened at the container cgroup level (a limit you set) or at the node level (a system-wide shortage). Then separate kernel OOM kills from kubelet evictions, identify the correct fix, and prevent recurrence without guessing at memory limits.

What this means

OOMKilled means the Linux OOM killer selected a process in your container and terminated it with SIGKILL (signal 9). Exit code 137 (128 + 9) often maps to OOMKilled, but the code alone is ambiguous: it can also result from a manual docker kill or an escalated graceful termination. Verify the pod status Reason field.

Kubernetes pod stuck ContainerCreating: volume, network, and image issues

Mon, 01 Jan 0001 00:00:00 +0000

A pod stuck in ContainerCreating never produces logs or readiness events. The kubelet accepted the spec but blocked during initialization after scheduling and before the container runtime starts the user process. The dominant failure domains are volume mount deadlocks, CNI sandbox creation failures, and image pull problems. They all surface the same status but need different fixes. This guide shows how to identify the stuck subsystem and resolve it.

What this means

ContainerCreating is the phase where the kubelet pulls the image, creates the pod sandbox via CRI, attaches and mounts volumes, injects ConfigMaps and Secrets, and starts the container. These steps run largely synchronously in the pod worker. If a mount hangs, a CNI plugin errors, or the registry rejects the pull, the worker blocks and the pod stays in ContainerCreating. The node condition can remain Ready because the kubelet sync loop and PLEG are still healthy, so cluster monitoring may miss the problem. Identifying whether the block is in storage, network, or images is the first step.

Kubernetes pod stuck on volume mount: CSI, permissions, and timeouts

Mon, 01 Jan 0001 00:00:00 +0000

You scale a StatefulSet and the new pods sit in ContainerCreating for ten minutes. The node is Ready. The CSI driver pods are running. kubectl describe shows no FailedMount events, yet the containers never start. The absence of volume events often misleads operators into checking image registries or resource quotas instead of the storage path. The kubelet volume manager is blocked somewhere between attach and mount, and Kubernetes will not retry fast enough to hide the problem.

Kubernetes pod stuck Terminating: finalizers, grace periods, and force delete

Mon, 01 Jan 0001 00:00:00 +0000

A pod stuck in Terminating stays visible in the API server after kubectl delete, sometimes for minutes or hours. Usually a finalizer blocks removal, a CSI volume is still attached, or the container is ignoring SIGTERM. Force deleting without diagnosis orphans containers and can violate StatefulSet guarantees. Check the signals first, then decide whether to wait, patch a finalizer, or force delete.

What This Means

When you delete a pod, the API server sets metadata.deletionTimestamp and starts a graceful deletion window controlled by terminationGracePeriodSeconds (default 30). The kubelet executes any preStop hooks, sends SIGTERM to all containers, and waits up to the grace period before sending SIGKILL.