Node NotReady
This error occurs when a worker node transitions to the NotReady status due to resource pressure, Kubelet process failures, or network splits.
Usually happens because:
- ☑ Kubelet service daemon crashed or is locked due to memory leaks
- ☑ containerd container runtime crashed or blocks on Docker queries
- ☑ Worker host network partition isolates node from API servers
🔍 Quick Checklist:
What is Node NotReady?
A 'Node NotReady' status indicates that the control plane (specifically the kube-controller-manager) has lost connection or stopped receiving heartbeats from the worker node's Kubelet service. In Kubernetes v1.36 clusters, node heartbeat mechanisms utilize Lease resources ('coordination.k8s.io') in addition to NodeStatus objects. Typical causes include Kubelet service crashes, memory/disk exhaustion on the host, network partitions, or container runtime (containerd) hangs.
Common Causes
- Kubelet service failure: The local Kubelet service crashes or stops responding due to host kernel bugs or configuration mismatches.
- Container runtime crash: The container runtime (containerd) crashes or blocks, preventing Kubelet from querying container states.
- Network partition split: The host node experiences a network failure, blocking outbound connections to control-plane API servers.
| Cause | Frequency |
|---|---|
| Kubelet systemd service crash or out-of-memory lock | ⭐⭐⭐⭐⭐ |
| Container runtime daemon (containerd) hang or failure | ⭐⭐⭐⭐ |
| Network partition isolating the worker host from control plane | ⭐⭐⭐ |
Common Mistakes
- Evicting pods prematurely due to short toleration timings during transient CNI network drops.
- Ignoring disk capacity fills (if node root folder hits 85%+, Kubelet triggers DiskPressure evictions, which can lead to NotReady transitions).
How to Fix
Kubernetes Operations & Verification
Inspect Kube Lease heartbeats inside the system leases namespace in Kubernetes v1.36.
# 1. List active leases in kube-node-lease
$ kubectl get leases -n kube-node-lease
# 2. Check the specific lease detail for worker-node-1
$ kubectl describe lease worker-node-1 -n kube-node-leasePlatform Specific Fixes
Log into the affected worker node via SSH and diagnose Kubelet / Containerd systemd services status.
# 1. Inspect services status
sudo systemctl status kubelet
sudo systemctl status containerd
# 2. Restart services if hanging
sudo systemctl restart containerd kubelet
# 3. Read Kubelet tail logs
journalctl -u kubelet -n 100 --no-pagerBest Practices
- Configure node disk cleanup routines to automatically prune dangling container images.
- Implement high-availability (HA) master endpoints configurations to ensure nodes can always reach at least one API server instance.
Frequently Asked Questions (FAQ)
Q: What causes 'Node NotReady' status?
This means the control plane has lost contact with the Kubelet on that node. It is usually caused by the Kubelet service crashing, containerd hanging, or node network partitions.
Q: How does the node send heartbeats in Kubernetes v1.36?
Kubernetes uses Node Lease objects under the 'kube-node-lease' namespace. Instead of updating the heavy Node object every few seconds, the Kubelet updates a lightweight Lease. If it fails to renew the Lease within the timeout, the node is marked NotReady.
Q: How do I troubleshoot NotReady nodes?
SSH into the affected node and check the services status: 'sudo systemctl status kubelet' and 'sudo systemctl status containerd'. Restarting these services ('sudo systemctl restart containerd kubelet') often resolves transient hangs.
Q: What happens to Pods running on a NotReady node?
If a node stays NotReady for longer than the toleration seconds (default is 5 minutes), the control plane automatically schedules the Pods for eviction, recreating them on other healthy Ready nodes.