Kubernetes pod stuck on volume mount: CSI, permissions, and timeouts

You scale a StatefulSet and the new pods sit in ContainerCreating for ten minutes. The node is Ready. The CSI driver pods are running. kubectl describe shows no FailedMount events, yet the containers never start. The absence of volume events often misleads operators into checking image registries or resource quotas instead of the storage path. The kubelet volume manager is blocked somewhere between attach and mount, and Kubernetes will not retry fast enough to hide the problem.

Unlike image pull errors or scheduling failures, stuck volume mounts often leave the node looking healthy. Existing pods keep running. Only workloads that need storage fail to start, which makes the symptom easy to blame on the application instead of the storage path.

What this means

The kubelet volume manager runs two asynchronous reconcilers: attach/detach at the node level, and mount/unmount into pod directories. For CSI volumes, the kubelet calls node RPCs to stage and publish the volume. Each operation holds a goroutine. If an operation hangs, that goroutine blocks. When enough hang, the volume manager saturates, and subsequent pods with volumes stall while the node stays Ready.

One hung NFS mount or a cloud volume still finishing force-detach can queue up and delay every new stateful pod on the node. Additionally, the kubelet applies fsGroup ownership changes recursively during mount; on large volumes this can stall pod startup without producing FailedMount events.

Common causes

CauseWhat it looks likeFirst thing to check
CSI driver unavailable on the nodeEvents mention RPC errors or driver name; mount setup failsCSI node plugin pod on the target node
Stale VolumeAttachment or multi-attachEvents mention multi-attach errors or attach waits indefinitelyVolumeAttachment objects and which node owns them
Permission or fsGroup mismatchContainer exits with permission denied on the volume path, or mount fails with access errorsPod securityContext fsGroup and storage backend ownership
NFS or network storage hangNo FailedMount events; pod ContainerCreating for more than 10 min; node ReadyManual mount test from the node to the storage endpoint
Cloud provider detach delayAttach operation exceeds 2 minutes after previous node lossCloud provider console or volume status for pending detach

Quick checks

# Confirm the pod phase is ContainerCreating
kubectl get pod $POD_NAME -o jsonpath='{.status.phase}'

# Inspect events for MountVolume, AttachVolume, or RPC errors
kubectl describe pod $POD_NAME | grep -A 20 Events

# Identify if the volume is attached to a different node
kubectl get volumeattachment -o wide

# Verify the CSI node plugin is present and running
kubectl get pods -n $CSI_NAMESPACE -l app=$CSI_DRIVER --field-selector spec.nodeName=$NODE_NAME

# Scan cluster-wide volume errors
kubectl get events --all-namespaces | grep -E "FailedMount|AttachVolume|MountVolume"

# Read kubelet logs for low-level volume and CSI errors
journalctl -u kubelet --since "30 minutes ago" | grep -i "mount\|volume\|csi"

# Check storage operation latency from kubelet metrics
kubectl get --raw /api/v1/nodes/$NODE_NAME/proxy/metrics | grep storage_operation_duration_seconds

# Verify fsGroup is configured when containers run as non-root
kubectl get pod $POD_NAME -o jsonpath='{.spec.securityContext.fsGroup}'

# Inspect currently mounted volumes under the kubelet directory
mount | grep kubelet

# Test NFS reachability directly from the node with a timeout to avoid indefinite hangs
timeout 10 mount -t nfs $SERVER:$PATH /tmp/testmnt
# Unmount after testing: umount /tmp/testmnt

# Check CSI node plugin logs for RPC timeouts or driver errors
kubectl logs -n $CSI_NAMESPACE -l app=$CSI_DRIVER --field-selector spec.nodeName=$NODE_NAME --tail=200 | grep -iE "error|timeout|rpc"

How to diagnose it

  1. Determine whether the delay is in attach or mount. Look at kubectl describe pod events and scroll past image pull and scheduling messages to find volume entries. If you see AttachVolume.Attach with a timestamp older than two minutes, the problem is attach. If you see MountVolume.SetUp without completion, the problem is mount.
  2. Inspect VolumeAttachment objects. Compare each VolumeAttachment’s .spec.nodeName against the pod’s .spec.nodeName. If the volume is attached to a different node, flag it as stale for cleanup after you verify the previous workload has stopped.
  3. Verify the CSI node driver pod on the target node. If the pod is not running, volume operations cannot proceed.
  4. Check kubelet storage operation metrics. As rough thresholds, local volumes should mount in under one second, block devices under ten seconds, network volumes under sixty seconds, and cloud attach operations under two minutes. Operations exceeding these indicate a backend hang or slowness.
  5. Examine permission errors. If the container starts but cannot write to the mount path, compare the pod’s securityContext.fsGroup and runAsUser against the filesystem ownership on the storage backend. Correct mismatches in the pod spec.
  6. For network storage, test reachability from the node directly. If a manual mount command hangs, the issue is outside Kubernetes. If it succeeds but the pod still stalls, the problem is likely in the kubelet volume manager or CSI driver.
  7. Review kubelet logs for RPC errors from CSI drivers or mount syscall failures. RPC errors with codes like Internal or DeadlineExceeded point to driver-specific bugs or resource exhaustion.
  8. Check for volume manager saturation. If multiple pods on the same node are stuck in ContainerCreating and the node is Ready, look for the first hung operation. Resolving that operation often unblocks the rest.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
storage_operation_duration_secondsReveals attach and mount latency before pods visibly stallp99 exceeds 30 s for network volumes or 2 min for cloud attach
storage_operation_errors_totalTracks failing volume operationsSustained increase over a 5-minute window
kubelet_volume_manager_total_volumesCount of volumes the kubelet is managingSudden spike correlating with ContainerCreating pods
Pod phase ContainerCreating durationDirect symptom of stuck mountsDuration > 5 minutes
Node Ready conditionNodes can remain Ready while the volume manager is blockedReady=True but volume-mounted pods fail to start
kubelet_pod_worker_duration_secondsSlow reconciliation can indicate blocked volume operationsSustained > 30 s
container_runtime_operations_duration_secondsDistinguishes runtime slowness from storage issueslist_containers p99 > 5 s
VolumeAttachment object ageStale attachments block reattachmentOlder than the cloud provider’s normal detach window

Fixes

If the cause is CSI driver failure

Restart the CSI node plugin pod on the affected node. Check the CSI controller and node plugin logs for RPC errors such as Internal, DeadlineExceeded, or Unavailable. These indicate the driver cannot reach the storage backend, the node plugin socket is unreachable, or the driver is exhausting its memory limits. If the driver is crashing due to resource pressure, increase its memory limit or move it to a node pool with headroom. Do not restart kubelet until the driver is healthy.

If the cause is stale VolumeAttachment or multi-attach

Delete the stale VolumeAttachment object:

kubectl delete volumeattachment <name>

Warning: Only delete a VolumeAttachment after you verify the volume is no longer in use on the previous node. Deleting it while the volume is still mounted can cause data corruption or unclean filesystem state. The attach/detach controller will recreate it. For cloud-backed volumes after a node failure, allow the cloud provider’s force-detach window to complete before manually detaching the volume, or detach it through the provider’s console if the cluster is stuck.

If the cause is permission or ownership mismatch

Add or correct securityContext.fsGroup in the pod spec to align with the group owning the files on the backing storage. For SELinux enforcing environments, ensure the volume context is compatible or configure seLinuxOptions in the securityContext. Recreate the pod to apply changes.

If the cause is network storage hang

For NFS, prefer mount options that prevent indefinite hard hangs, such as soft and a reasonable timeo value, while accepting that soft can return I/O errors to the application if the server becomes unreachable. If a mount is already hung, identify the blocking process before acting. Do not kill kubelet or the mount helper without first assessing workload impact; an unclean termination can leave the volume in an inconsistent state and prevent clean unmount. Restarting the kubelet without fixing the storage endpoint rarely helps and can extend recovery time.

If the cause is cloud API throttling or detach delay

Reduce the rate of new volume attachments to stay within the cloud provider’s API limits and volume queue depths. Verify that the previous node has fully released the volume before rescheduling stateful pods. If a node was preempted or terminated, expect a forced-detach delay.

Prevention

  • Monitor storage_operation_duration_seconds and alert on p99 thresholds before pods stall.
  • Set fsGroup and expected runAsUser explicitly in pod specs for volumes written by non-root containers.
  • Configure NFS and network storage mounts with soft and timeo to avoid hard hangs that saturate the volume manager.
  • Keep CSI driver and sidecar versions current, and monitor their pod health independently from kubelet health.
  • Maintain node disk and inode headroom; disk pressure slows image pulls and can indirectly delay volume reconciliation.
  • Track the gap between running and desired pod counts on each node to detect volume manager saturation early.
  • Document your storage backend’s attach limits and enforce them through CSI driver flags or scheduler constraints.
  • Test storage endpoint fail-over and recovery procedures in staging so that hang behavior is understood before it happens in production.

How Netdata helps

  • Correlates storage_operation_duration_seconds with node disk I/O wait and network latency to distinguish a slow storage backend from a saturated kubelet volume manager.
  • Tracks kubelet sync loop and PLEG relist latency alongside pod start duration to identify whether volume mounts are the bottleneck during startup storms.
  • Alerts on node filesystem usage and inode consumption before disk pressure triggers evictions that complicate volume recovery.