ContainerCreating stuck
This error occurs when a Pod remains in the ContainerCreating state due to volume mounting issues, CNI network timeouts, or runtime bugs.
Usually happens because:
- ☑ PersistentVolume is locked by an inactive node or fails to attach
- ☑ CNI network plug-in runs out of allocatable IP addresses
- ☑ Containerd fails to construct the namespace sandbox container
🔍 Quick Checklist:
What is ContainerCreating stuck?
A Pod stuck in the 'ContainerCreating' status indicates that the scheduler assigned the Pod to a node, but the local Kubelet cannot start the container runtime processes. In Kubernetes v1.36, this is commonly caused by Container Storage Interface (CSI) volume mount timeouts, Container Network Interface (CNI) IP address allocation timeouts, or host-level kernel resource bottlenecks.
Common Causes
- CSI Volume mount locks: A Persistent Volume (PV) is locked by another node or fails to attach (common with multi-attach errors for block storage).
- CNI IP allocation failure: The Container Network Interface (e.g. Calico, Flannel, AWS VPC CNI) has exhausted its IP subnet or fails to bind virtual ethernet interfaces.
- Runtime or sandbox creation failure: The container runtime (containerd) fails to initialize the sandbox namespace due to host memory limits or OS resource leaks.
| Cause | Frequency |
|---|---|
| CSI volume attachment timeouts or locks (Multi-Attach-Error) | ⭐⭐⭐⭐⭐ |
| CNI network subnet IP pool exhaustion | ⭐⭐⭐⭐ |
| Containerd sandbox namespace creation failures (OS limits) | ⭐⭐⭐ |
Common Mistakes
- Re-deploying stateful workloads across zones assuming PersistentVolumes dynamically cross boundaries (read-write-once block storage cannot attach to a node in a different AZ, leaving the Pod stuck in ContainerCreating).
- Exhausting host file descriptors or node process ID limits (PID limits).
How to Fix
Kubernetes Operations & Verification
Inspect active volume attachments to locate stuck locks.
# 1. List all volume attachments and check if attachment status is false
kubectl get volumeattachments
# 2. Describe stuck attachment metadata detail
kubectl describe volumeattachment vault-volume-attachment-hashPlatform Specific Fixes
Log into the worker node directly and review containerd and Kubelet log lines.
# 1. Inspect Kubelet logs for mounting errors
journalctl -u kubelet -n 100 --no-pager
# 2. Inspect Containerd sandbox creation logs
journalctl -u containerd -n 100 | grep -i sandboxBest Practices
- Utilize ReadWriteMany (RWX) storage provisioners (like NFS, EFS) for workloads that must scale across multiple nodes.
- Adopt proper subnets allocations sizing configurations during initial CNI installations.
Frequently Asked Questions (FAQ)
Q: Why is my Pod stuck in ContainerCreating status?
This means the Kubelet on the scheduled node is attempting to spin up the container sandbox but is blocked. The most common reasons are volumes failing to attach/mount, or network plug-in (CNI) failures.
Q: How do I troubleshoot volume mount issues keeping a Pod in ContainerCreating?
Run 'kubectl describe pod <pod-name>'. If you see events like 'Multi-Attach error for volume' or 'FailedMount', the volume is likely still attached to an old node. You may need to delete the stuck volume attachment or wait for the CSI driver to force detach it.
Q: What is a CNI error in this state?
The Container Network Interface (CNI) is responsible for provisioning the pod's IP. If you see 'Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container...', the CNI is failing to allocate an IP or set up the network interface.
Q: How do I inspect local node runtime issues in K8s v1.36?
Log into the worker node and inspect containerd events using 'journalctl -u containerd' or check Kubelet logs using 'journalctl -u kubelet'.