Context deadline exceeded
This error occurs when a client request (like kubectl commands or webhook calls) fails to complete within its allocated context timeout period.
Usually happens because:
- ☑ Validating/Mutating webhooks block requests longer than 10 seconds
- ☑ Slow disk I/O on etcd nodes causes transaction database timeouts
- ☑ Aggregated custom API registrations (like metrics-server) are offline
🔍 Quick Checklist:
What is Context deadline exceeded?
A 'Context deadline exceeded' error is a Go-specific timeout exception raised when a context is cancelled because its deadline passed. In Kubernetes v1.36 clusters, this typically manifests when kubectl commands fail to receive replies from the API server, when the API server times out communicating with etcd databases, or when validating/mutating admission webhooks block request execution.
Common Causes
- Admission webhook timeouts: A validating or mutating admission webhook fails to respond within its configured timeout limit (default is often 10 seconds).
- etcd storage backend latency: The API server times out waiting for etcd to read or write cluster state due to slow disk I/O or network latency.
- APIService or Custom Resource Definition (CRD) unavailability: An aggregated API server (like metrics-server) is offline or unreachable, blocking queries to its resources.
| Cause | Frequency |
|---|---|
| Failing or slow validating/mutating admission webhooks | ⭐⭐⭐⭐⭐ |
| etcd storage database write latency (slow disk I/O) | ⭐⭐⭐⭐ |
| Unreachable aggregated API services (e.g. metrics-server) | ⭐⭐⭐ |
Common Mistakes
- Running heavy validation webhooks that perform synchronous DNS checks or external API calls without timeout protections, blocking all cluster operations if the external system slows down.
- Neglecting to monitor etcd disk latency warnings in logs, leading to client connection dropouts.
How to Fix
Kubernetes Operations & Verification
Configure validating webhook configurations to fail open (Ignore) to prevent blocking resource creation during timeouts in Kubernetes v1.36.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: policy-checker
webhooks:
- name: check.policy.io
rules:
- apiGroups: [""]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["pods"]
# Configure to ignore failures if webhook times out
failurePolicy: Ignore
timeoutSeconds: 5
clientConfig:
service:
name: policy-service
namespace: default
path: "/validate"Platform Specific Fixes
Review API server logs to track etcd write latencies and timeout events.
# 1. Read API server logs on control plane nodes
journalctl -u kube-apiserver -n 50 --no-pager | grep -i "deadline"
# 2. Query etcd write durations
curl -s http://localhost:2379/metrics | grep etcd_disk_wal_write_secBest Practices
- Implement asynchronous patterns for admission webhooks, returning quickly and executing auditing out-of-band.
- Provision dedicated high-speed SSD storage classes (like gp3 on AWS with dedicated IOPS) for etcd nodes.
Frequently Asked Questions (FAQ)
Q: What does 'context deadline exceeded' mean in Kubernetes?
This is a Go language error indicating a timeout. A component (like kubectl or the API server) started an operation with a time limit, but the limit was reached before the operation finished.
Q: How do I fix webhook-related context deadline exceeded errors?
Run 'kubectl get mutatingwebhookconfigurations,validatingwebhookconfigurations' to list webhooks. If a webhook is down or slow, it will block all resource creations. Check the webhook pod logs, or modify its config to 'failurePolicy: Ignore' to bypass it temporarily.
Q: Why does etcd cause this error?
etcd is the single source of truth for Kubernetes. If etcd has slow disk I/O (common on cloud nodes with small IOPS limits), write requests will block. The API server waits, eventually hitting its context deadline and throwing this error.
Q: How do I troubleshoot metrics-server timeouts?
If you see this error when running 'kubectl top nodes', the metrics API service is likely offline. Run 'kubectl get apiservice v1beta1.metrics.k8s.io' to check if the registration is active and healthy.