Liveness probe failed
This error occurs when a container's liveness health check fails, causing the Kubelet to terminate and restart the container.
Usually happens because:
- ☑ Application is frozen due to thread deadlocks or memory leaks
- ☑ Liveness check starts firing before the container finishes bootstrap
- ☑ Slow response response time exceeding timeoutSeconds limit parameters
🔍 Quick Checklist:
What is Liveness probe failed?
A 'Liveness probe failed' error indicates that the container is running but has entered an unrecoverable deadlocked or frozen state. When a liveness probe fails, the Kubelet immediately restarts the container according to its restartPolicy. In Kubernetes v1.36, liveness probes can use HTTP, TCP, gRPC, or Exec commands. Misconfiguration (like placing liveness checks on slow startup code) can trigger infinite crash loops.
Common Causes
- Application deadlock or freeze: The application stops responding to requests due to thread locks, memory leaks, or endless loops.
- Insufficient probe timeout limits: The container is heavily loaded and cannot return a health check within the configured 'timeoutSeconds'.
- Liveness check overlapping startup: The liveness probe begins firing before a heavy Java or Node process completes its cold-start initialization.
| Cause | Frequency |
|---|---|
| Thread deadlocks or application infinite loops (freezing) | ⭐⭐⭐⭐⭐ |
| Liveness checks firing during slow container startup | ⭐⭐⭐⭐ |
| Too low timeoutSeconds threshold during load spikes | ⭐⭐⭐ |
Common Mistakes
- Setting `timeoutSeconds` equal to `periodSeconds`, causing probes to pile up if single checks block, triggering false-positive container restarts.
- Pointing liveness probes to the same heavy database-checking endpoint as readiness probes, crashing the container during brief network fluctuations.
How to Fix
Kubernetes Operations & Verification
Establish startupProbes to protect slow-starting applications from premature liveness kills in Kubernetes v1.36.
apiVersion: v1
kind: Pod
metadata:
name: heavy-java-pod
spec:
containers:
- name: java-app
image: openjdk:17-slim
# 1. Startup probe permits up to 5 minutes bootstrap window
startupProbe:
httpGet:
path: /health/startup
port: 8080
failureThreshold: 30
periodSeconds: 10
# 2. Liveness check activates only after startupProbe succeeds
livenessProbe:
httpGet:
path: /health/live
port: 8080
periodSeconds: 15
timeoutSeconds: 3
failureThreshold: 2Platform Specific Fixes
Query container termination details (such as exit code 137 or SIGKILL signal tags) from Pod specs.
# 1. Inspect container restart numbers
kubectl get pods web-app -o jsonpath='{.status.containerStatuses[*].restartCount}'
# 2. Get last termination state logs
kubectl get pod web-app -o jsonpath='{.status.containerStatuses[*].lastState.terminated}'Best Practices
- Ensure liveness check code is highly optimized, resolving immediately without querying external DBs or APIs.
- Configure appropriate `terminationGracePeriodSeconds` to permit cleanup tasks before SIGKILL runs.
Frequently Asked Questions (FAQ)
Q: What is a liveness probe?
A liveness probe is a diagnostic check run by the Kubelet to determine if a container needs to be restarted. If it fails, Kubernetes kills the container and starts a new one.
Q: Why does my Pod keep restarting automatically?
If your container crashes every few minutes, run 'kubectl describe pod' and check the 'Events' section. If you see 'Liveness probe failed', it means the check returned a non-200 code or timed out, prompting the Kubelet to kill the container.
Q: How do I prevent restarts during application startup?
Do not use a high liveness probe frequency or low initial delay for startup. Instead, use a 'startupProbe' (native in v1.36) that keeps liveness checks disabled until the startup probe succeeds.
Q: What exit code does a liveness restart trigger?
When the Kubelet kills a container due to liveness failures, it sends a SIGTERM, followed by a SIGKILL (Exit Code 137) if it doesn't shut down gracefully within the terminationGracePeriodSeconds.