Dial TCP timeout
This error occurs when a network socket fails to establish a three-way TCP handshake within the connection timeout limit.
Usually happens because:
- ☑ VPC Security Groups block VXLAN (UDP 4789) or Geneve (UDP 6081) ports between nodes
- ☑ Kube-proxy fails to synchronize iptables/IPVS routing tables
- ☑ Underlying cloud provider subnet routes are broken
🔍 Quick Checklist:
What is Dial TCP timeout?
A 'dial tcp: i/o timeout' error is raised when a client attempts to open a TCP connection to a server, but receives no response (SYN-ACK) before the connection context expires. In Kubernetes v1.36 clusters, this typically happens when Pods attempt to communicate across nodes while CNI vxlan/geneve encapsulation packets are blocked by security group firewalls, when kube-proxy rules fail to apply, or when CoreDNS fails to resolve target service IP routes.
Common Causes
- Security group port blocks: AWS VPC Security Groups or local firewalls block internal network traffic on CNI overlay ports (like UDP 4789 for VXLAN or UDP 6081 for Geneve).
- Kube-proxy routing failures: The local kube-proxy daemon fails to synchronize iptables or IPVS rules, causing traffic routed to ClusterIPs to drop silently.
- Node routing partitions: Underlying cloud network routes between worker subnet groups are severed or misconfigured.
| Cause | Frequency |
|---|---|
| Security groups blocking CNI overlay ports (UDP 4789 / 6081) | ⭐⭐⭐⭐⭐ |
| Kube-proxy sync failures (missing iptables/IPVS rules) | ⭐⭐⭐⭐ |
| Cloud provider subnet routing misconfigurations | ⭐⭐⭐ |
Common Mistakes
- Configuring custom firewall profiles on worker nodes that drop UDP traffic, blocking VXLAN container tunnels.
- Using public IP targets inside network policies instead of ClusterIP names.
How to Fix
Kubernetes Operations & Verification
Verify kube-proxy synchronization state events and inspect iptables rules maps.
# 1. Inspect kube-proxy pods logs
$ kubectl logs -n kube-system daemonset/kube-proxy --tail=100
# 2. Confirm rules are synchronized without errorPlatform Specific Fixes
Inspect Linux host iptables rules tables directly from node shells.
# 1. List nat table routing rules
sudo iptables -t nat -L -n -v
# 2. Probe specific port route with timeout limit
nc -w 5 -zv 10.96.0.10 53Best Practices
- Adopt Network Policies analyzers to verify intra-cluster security rules.
- Adopt health checker agents (like goldpinger) to monitor cross-node pod routing paths continuously.
Frequently Asked Questions (FAQ)
Q: What is a dial TCP timeout?
This means a TCP connection request (SYN packet) was sent, but the client received no reply at all, causing the connection attempt to time out. This is a routing or firewall issue.
Q: How does CNI VXLAN port blocking cause this?
CNIs like Calico, Flannel, or Cilium route pod-to-pod traffic across nodes by encapsulating it in UDP packets. If your cloud security groups block UDP port 4789 (VXLAN) or UDP 6081 (Geneve) between worker nodes, cross-node pod traffic will time out.
Q: How do I troubleshoot kube-proxy iptables issues?
Check if kube-proxy is running: 'kubectl get pods -n kube-system -l k8s-app=kube-proxy'. If it is running, check the logs. You can inspect the generated NAT rules on a node using: 'sudo iptables -t nat -L -n -v'.
Q: How do I test TCP ports directly?
Use curl or nc: 'nc -w 5 -zv <target-ip> <port>'. The '-w 5' sets a 5-second timeout, helping you quickly identify connection timeout behaviors.