FailedMount
This error occurs when the Kubelet cannot mount a volume inside a container, commonly due to directory locks, wrong credentials, or CSI driver failures.
Usually happens because:
- ☑ ReadWriteOnce volume is locked by another node due to delayed detachment
- ☑ CSI driver container crashed or is unreachable on the scheduled node
- ☑ Reference to missing ConfigMaps or Secrets within the volume definition
🔍 Quick Checklist:
What is FailedMount?
A 'FailedMount' event is raised by the Kubelet when it fails to mount the designated volume path into the container sandbox. In Kubernetes v1.36 architectures, this is mediated by Container Storage Interface (CSI) drivers. Typical causes include network latency blocking remote attachments, target directory path locks, using ReadWriteOnce (RWO) volumes across multiple nodes, or missing sub-path keys.
Common Causes
- Multi-Attach error for RWO volume: Attempting to attach a ReadWriteOnce volume to a new Pod on Node B while it is still locked by a terminated Pod on Node A.
- CSI driver connection timeouts: The node's CSI plugin container cannot communicate with the cloud storage API (e.g. AWS EBS or Azure Disk) to finalize the mount.
- Missing Secret keys for storage credentials: The volume mount configuration references a Secret or ConfigMap containing access keys that does not exist in the namespace.
| Cause | Frequency |
|---|---|
| ReadWriteOnce volume multi-attach locks (stuck detach) | ⭐⭐⭐⭐⭐ |
| Missing access credential Secrets in Pod namespace | ⭐⭐⭐⭐ |
| CSI node daemon container crash or timeout | ⭐⭐⭐ |
Common Mistakes
- Mounting ConfigMaps or Secrets as volumes using non-existent keys, causing the Kubelet mount daemon to fail indefinitely.
- Attempting to scale a deployment with a ReadWriteOnce (RWO) volume to multiple replicas, causing FailedMount on all replicas scheduled to different nodes.
How to Fix
Kubernetes Operations & Verification
Locate and remove stuck RWO volume attachments to allow the volume to bind on new scheduled nodes.
# 1. List all active VolumeAttachments
$ kubectl get volumeattachments
# 2. Delete the stuck attachment locking the volume
$ kubectl delete volumeattachment csi-attachment-hashPlatform Specific Fixes
Log into the worker node directly to check local mounting directories under the Kubelet directory structure.
# 1. Inspect Kubelet journal logs filtering by mounts
journalctl -u kubelet -n 100 --no-pager | grep -i "mount"
# 2. Check mount points in system directories
df -h | grep kubeletBest Practices
- Use ReadWriteMany (RWX) PV storage types (e.g. AWS EFS, Azure Files) for replica workloads.
- Adopt GitOps sync checks (like ArgoCD or Flux) to ensure config maps exist before pods launch.
Frequently Asked Questions (FAQ)
Q: What causes FailedMount in Kubernetes?
This means the Kubelet successfully scheduled the Pod but cannot mount the requested storage volume. It's usually due to RWO volume locks, CSI driver issues, or missing configuration secrets.
Q: How do I solve 'Multi-Attach error for volume' during mount?
A ReadWriteOnce (RWO) volume can only be mounted by a single node at a time. If a Pod is rescheduled to a new node but the old node hasn't released the volume, you get this error. You can delete the stuck 'VolumeAttachment' resource or restart the old node to force detach the disk.
Q: Why does my ConfigMap mount fail?
If you mount a ConfigMap or Secret as a volume but it doesn't exist in the same namespace, the Kubelet cannot compile the virtual files and raises a FailedMount error. Ensure the ConfigMap/Secret is created first.
Q: How do I check Kubelet mount logs on worker nodes?
Log into the worker node and inspect mount processes logs: 'journalctl -u kubelet | grep -E "MountVolume|AttachVolume"'.