Kubernetes monitoring checklist: the signals every production cluster needs

Production Kubernetes failures announce themselves early: PLEG relist latency climbs before a node goes NotReady; etcd WAL fsync duration creeps toward the heartbeat timeout; conntrack utilization sits at 90% until a traffic spike drops SYN packets. This checklist maps signals to failure modes across four layers: node and container runtime, control plane, workload state, and network data plane. Use it to wire a greenfield pipeline or audit a brownfield one.

Common causes

Quick checks

Run these commands to validate health across layers before relying on dashboards.

# Control plane liveness
kubectl get --raw /livez

# etcd leader and disk health (run from etcd node)
etcdctl endpoint health --cluster
etcdctl endpoint status --cluster -w table

# Node Ready conditions and pressure
kubectl get nodes -o json | jq -r '.items[] | "\(.metadata.name): Ready=\(.status.conditions[] | select(.type=="Ready") | .status), MemoryPressure=\(.status.conditions[]? | select(.type=="MemoryPressure") | .status)"'

# Pending pods
kubectl get pods --all-namespaces --field-selector=status.phase=Pending

# Conntrack utilization on a node
echo "scale=2; $(cat /proc/sys/net/netfilter/nf_conntrack_count) * 100 / $(cat /proc/sys/net/netfilter/nf_conntrack_max)" | bc

# kube-proxy sync freshness
curl -s http://127.0.0.1:10249/metrics | grep kubeproxy_sync_proxy_rules_last_timestamp_seconds

# CRI runtime responsiveness
time crictl ps

# API server error rate
kubectl get --raw /metrics | grep 'apiserver_request_total' | grep 'code="5"'

# Critical service endpoints
kubectl get endpoints --all-namespaces -o json | jq -r '.items[] | select((.subsets // []) | length == 0) | "\(.metadata.namespace)/\(.metadata.name)"'

How to diagnose it

When an alert fires, follow this flow to isolate the layer.

1. Control plane liveness. kubectl get --raw /livez must return 200. If it fails, check etcd leader stability (etcd_server_has_leader) and disk latency (etcd_disk_wal_fsync_duration_seconds). A missing leader or fsync p99 over 100 ms explains downstream symptoms.
2. Node Ready conditions. Any node with Ready=Unknown for more than one minute is past its grace period. Check kubelet_pleg_relist_duration_seconds. If it is above 60 seconds, the container runtime is the bottleneck.
3. Workload scheduling. Sustained pending pods mean either capacity exhaustion (check node allocatable vs requests) or scheduling constraints (taints, affinity, ResourceQuotas). Look at scheduler_pending_pods: unschedulableQ indicates hard constraints; backoffQ indicates transient failures.
4. Network programming. If pods are Running but services are unreachable, verify kubeproxy_sync_proxy_rules_last_timestamp_seconds is recent. Then check nf_conntrack_count against nf_conntrack_max. If utilization is above 90%, new connections drop.
5. Resource pressure per node. MemoryPressure=True or DiskPressure=True means the eviction manager is actively killing pods. Cross-reference with kubelet_evictions_total and actual memory utilization from /proc/meminfo or cgroup stats. If actual usage exceeds requests, you have invisible overcommit.

Metrics and signals to monitor

Node and container runtime

Control plane

Workload and cluster state

Network and data plane

Fixes

If the cause is node or runtime pressure

Cordon the node. Identify evicted pods: kubectl get events --field-selector reason=Evicted. Check whether evicted workloads have memory requests matching actual usage. If BestEffort pods consume unbounded memory, enforce limits via LimitRange. If the runtime is slow, check for defunct or hung containerd-shim processes.

If the cause is control plane saturation

Identify the traffic source using apiserver_request_total user-agent labels. If APF throttles critical controllers, increase concurrency shares for system and leader-election priority levels. If a single operator floods the API, reduce its concurrency or add client-side rate limits. If admission webhooks are the bottleneck, verify webhook endpoint health. As a temporary mitigation only, you can change failurePolicy to Ignore, but this bypasses policy enforcement and can allow invalid or insecure objects into the cluster.

If the cause is etcd latency

Move etcd data directories to dedicated SSD storage with low fsync latency. Verify etcd does not share disks with log-heavy workloads. Check compaction effectiveness: if the gap between etcd_debugging_mvcc_db_total_size_in_bytes and etcd_mvcc_db_total_size_in_use_in_bytes is large, schedule defragmentation during a maintenance window, one member at a time. Increase --quota-backend-bytes if the database approaches its limit.

If the cause is network or conntrack exhaustion

Immediately increase nf_conntrack_max on affected nodes: sysctl -w net.netfilter.nf_conntrack_max=<2x current>. This is a temporary fix; investigate the root cause to avoid masking a connection leak. Inspect connection states with conntrack -L to identify TIME_WAIT or UDP accumulation. If kube-proxy sync duration in iptables mode exceeds 10 seconds at scale, plan a migration to IPVS or nftables mode. For UDP services in IPVS mode, reduce the UDP timeout to prevent dead backend stickiness.

If the cause is workload misconfiguration

Set resource requests and limits for all production pods. Remove CPU limits to avoid CFS throttling if your cluster can tolerate the burst, but keep memory limits to contain blast radius. Configure PodDisruptionBudgets with minAvailable below total replicas to allow drains. Ensure liveness probes detect deadlocks while readiness probes detect temporary traffic inability. Do not configure them identically.

Prevention

• Monitor leading indicators, not just lagging ones. PLEG relist duration, etcd WAL fsync, and conntrack ratio warn early. Node Ready and pod eviction are late.
• Forward Kubernetes Events to persistent storage. Events have a one-hour TTL. Without external forwarding, you lose the incident narrative.
• Alert on certificate TTL at 30 days. Control plane, kubelet, and webhook certificates expire on predictable schedules; auto-rotation fails silently.
• Enforce resource hygiene. Use LimitRange and ResourceQuota to prevent BestEffort pods from entering the cluster. The scheduler sees zero requests and over-packs nodes.
• Right-size kube-proxy for scale. If you run more than a few thousand Services, iptables mode degrades. Size for IPVS or nftables before you hit the cliff.
• Audit admission webhooks quarterly. Verify failurePolicy, namespaceSelector exclusions for kube-system, and timeout values. A webhook matching all resources with failurePolicy: Fail is a cluster-wide single point of failure.

How Netdata helps

• Correlates node-level PSI stall metrics with kubelet PLEG relist latency and CRI operation duration to surface runtime slowdown before nodes go NotReady.
• Tracks conntrack utilization, API server request latency, and APF rejection rates on one timeline to expose network and control plane saturation.
• Surfaces container-level CPU throttling, memory pressure, and OOM kills alongside pod restart counts and deployment readiness gaps.
• Maps etcd disk latency and leader changes to scheduler pending pods and controller work queue depth to show control plane cascades.