Connection refused
This error occurs when a Pod or network client fails to establish a TCP handshake because the target IP address is not listening on the specified port.
Usually happens because:
- ☑ Service selector labels mismatch, causing endpoints list to be empty
- ☑ Service targetPort does not match containerPort inside Pod manifest
- ☑ The application process crashed or has not completed startup listener binds
🔍 Quick Checklist:
What is Connection refused?
A 'Connection refused' error indicates that the network host was reached, but the target operating system kernel actively rejected the connection because no process was listening on the requested TCP port. In Kubernetes v1.36 clusters, this typically happens when microservices attempt to call peer Pods before they are fully initialized, when a Service has no matching backend Pods in the endpoints list, or when CoreDNS resolves service names to incorrect ClusterIPs.
Common Causes
- Service has zero active endpoints: The Service exists, but all backend pods are either crash-looping or failing readiness checks, leaving the endpoints list empty.
- Target port mismatch: The Service 'targetPort' does not match the actual 'containerPort' exposed by the application inside the Pod.
- Local container process offline: The target container process crashed or has not finished starting up, leaving the socket closed.
| Cause | Frequency |
|---|---|
| Service endpoints list is empty (no ready backend Pods) | ⭐⭐⭐⭐⭐ |
| Port mismatch between Service targetPort and containerPort | ⭐⭐⭐⭐ |
| Target container daemon process is down or restarting | ⭐⭐⭐ |
Common Mistakes
- Assuming a Service routes traffic correctly when selector labels are mismatched (the Service is created successfully but endpoints remain empty, returning connection refused to client calls).
- Sending traffic to the wrong Service port number when multiple ports are exposed.
How to Fix
Kubernetes Operations & Verification
Align Service ports configuration with container ports exposed in the Pod manifest to ensure traffic matches active socket listeners in Kubernetes v1.36.
apiVersion: v1
kind: Service
metadata:
name: api-service
spec:
selector:
app: backend-worker
ports:
- protocol: TCP
port: 8080 # Port clients connect to outside the Pod
targetPort: 9000 # Must match containerPort
---
apiVersion: v1
kind: Pod
metadata:
name: api-pod
labels:
app: backend-worker
spec:
containers:
- name: server-container
image: node-backend:latest
ports:
- containerPort: 9000 # App process listens on port 9000Platform Specific Fixes
Exec into a container to list active port listeners using ss or netstat commands.
# 1. Connect to target container
kubectl exec -it api-pod -- sh
# 2. Check active TCP listeners
ss -tuln
# Or: netstat -tulnBest Practices
- Implement client-side retries with jitter in microservice codebases to tolerate transient container starts.
- Adopt automated helm templates validation schemas to lock ports alignment across resources.
Frequently Asked Questions (FAQ)
Q: What causes 'Connection refused' in Kubernetes?
This means the network package reached the target Pod or Service, but the target system rejected it because no process was listening on that port. Common reasons are missing Service endpoints or port mismatches.
Q: How do I troubleshoot 'Connection refused' between two microservices?
First check the endpoints: 'kubectl get endpoints <service-name>'. If the list is empty, the target pods are either crashing or failing readiness probes. Fix the pods first.
Q: Why does my Service connection fail but pod-to-pod IP works?
Check your port definitions. The Service spec defines 'port' (what clients connect to) and 'targetPort' (what the container listens on). If 'targetPort' doesn't match the 'containerPort' in the Pod spec, traffic gets routed to the wrong port, causing connection refused.
Q: How do I inspect active listeners inside a running container?
Exec into the container and use a netstat tool: 'kubectl exec -it <pod-name> -- netstat -tuln'. This shows which ports the application is actively listening on.