Back-off restarting failed container
This error occurs when the Kubelet enters a delayed back-off phase after a container in a Pod repeatedly fails and restarts.
Usually happens because:
- ☑ Application process crashes immediately due to uncaught runtime exceptions
- ☑ Kubelet kills container due to repeated liveness probe failures
- ☑ Host security rules (AppArmor/Seccomp) block internal system calls
🔍 Quick Checklist:
What is Back-off restarting failed container?
The 'Back-off restarting failed container' status is an event raised by the Kubelet indicating that a container has terminated repeatedly (crashed). Rather than immediately launching another instance and exhausting CPU/IO resources on the host, the Kubelet enforces an exponential back-off restart delay starting at 10 seconds, doubling at each failure up to a maximum limit of 5 minutes. This state corresponds directly with the 'CrashLoopBackOff' status column.
Common Causes
- Application startup runtime crash: Uncaught runtime exceptions (like SegFaults, out of memory, or script syntax errors) cause the container process to exit immediately.
- Failed liveness or startup probes: The Kubelet forcefully kills and restarts the container because health checks repeatedly fail or time out.
- Unsupported host kernel calls: Enforcing security profiles (like custom AppArmor or Seccomp configurations) that block kernel syscalls required by the container's executable.
| Cause | Frequency |
|---|---|
| Application runtime process crash (uncaught code exceptions) | ⭐⭐⭐⭐⭐ |
| Forceful container termination due to failed liveness probes | ⭐⭐⭐⭐ |
| Kernel system calls blocked by AppArmor or Seccomp profiles | ⭐⭐⭐ |
Common Mistakes
- Attempting to query active logs (`kubectl logs pod-name`) instead of historical logs (`kubectl logs pod-name --previous`), showing empty screens if the container is currently paused.
- Ignoring the exit code details inside describe results (e.g. exit code 139 indicates a Segmentation Fault crash, meaning the container binary has internal runtime code bugs).
How to Fix
Kubernetes Operations & Verification
Retrieve exit code reasons and timestamps for terminated container instances dynamically.
# 1. Retrieve the container lastState termination details
$ kubectl get pod api-pod-hash -o jsonpath='{.status.containerStatuses[*].lastState.terminated}'
# 2. Check logs of the crashed container instance
$ kubectl logs api-pod-hash --previousPlatform Specific Fixes
Inspect container runtime system logs on worker nodes to trace kernel calls blocks.
# 1. Inspect containerd event logs on linux node
journalctl -u containerd -n 50 --no-pager
# 2. Check audit logs for AppArmor access blocks
sudo grep -i "apparmor" /var/log/audit/audit.logBest Practices
- Verify container entrypoint scripts run correctly under unprivileged non-root users locally.
- Implement comprehensive health check probes delay configurations to protect slow-starting services.
Frequently Asked Questions (FAQ)
Q: What does 'Back-off restarting failed container' mean?
This is the Event description equivalent to 'CrashLoopBackOff'. It means the container has crashed multiple times, and the Kubelet is waiting before trying to start it again.
Q: How do I find out why the container failed?
First run 'kubectl logs <pod-name> --previous'. If the container crashed, this command prints the logs of the failed instance. If that's empty, run 'kubectl describe pod <pod-name>' to check the exit code.
Q: What is exit code 139?
Exit code 139 means the container exited due to a Segmentation Fault (SIGSEGV). This is a low-level C/C++ memory access crash inside the container's binary.
Q: How do AppArmor or Seccomp profiles trigger restarts?
If your cluster enforces strict security profiles, and your application tries to run a restricted system call (e.g. modifying network settings), the kernel blocks the syscall, forcing the process to crash immediately.