Kubernetes: node(s) didn't match Pod's node affinity/selector
No node carries the labels the pod insists on, so the scheduler rejects every candidate. It turns up after an upgrade that replaced nodes with a different label set, and in multi zone clusters where a pod is pinned to the zone of a volume it can no longer be placed near.
Quick fix
Read the commands before running them. Anything that restarts a service, deletes data or changes permissions should be tried on a non-production system first.
# What the pod is asking for
kubectl get pod web-0 -o jsonpath='{.spec.nodeSelector}{"\n"}{.spec.affinity}{"\n"}'
# What the nodes actually offer
kubectl get nodes --show-labels
kubectl get nodes -L topology.kubernetes.io/zone,node.kubernetes.io/instance-type
# Label a node, or soften the rule from required to preferred
kubectl label node ip-10-0-1-5 workload=batch
# A bound volume pins the zone, so check the volume before the scheduler
kubectl get pv -o custom-columns=NAME:.metadata.name,CLAIM:.spec.claimRef.name,ZONE:.spec.nodeAffinity
# Autoscaler logs say why no group could satisfy the pod
kubectl -n kube-system logs -l app=cluster-autoscaler --tail=50
How to diagnose Kubernetes errors
Kubernetes errors are best read as a state machine that got stuck. Pending means the scheduler could not place the pod (resources, taints, or an unbound volume). ImagePullBackOff means kubelet could not fetch the image (name, credentials, or registry). CrashLoopBackOff means the container starts and exits, so the answer is in the container's own logs. OOMKilled means the kernel killed it for exceeding its memory limit. Each state points at a different subsystem, and kubectl describe almost always contains the exact reason in its events.
If the quick fix above does not resolve it, work through these steps. They apply to this whole class of error, not just to this one message, which is usually what saves the time.
- Start with
kubectl describe pod <name>and read the Events section at the bottom. It names the precise failure, including registry errors and scheduling constraints. - For CrashLoopBackOff, read the previous container's logs:
kubectl logs <pod> --previous. The current container may not have produced output yet. - Check resource pressure with
kubectl top podandkubectl top node, and compare against the pod's requests and limits. - Test RBAC directly:
kubectl auth can-i <verb> <resource> --as=system:serviceaccount:<ns>:<sa>. This answers permission questions definitively. - For networking, confirm the Service has endpoints (
kubectl get endpoints) before suspecting DNS, Ingress or the CNI.
Tools worth reaching for
kubectl describekubectl logs --previouskubectl auth can-ikubectl topkubectl events --sort-by=.lastTimestampk9s
Authoritative references
Primary documentation for this error, worth reading before applying any fix in production.
Related Kubernetes errors
- Argo CD: Application OutOfSync / DegradedThe live cluster state differs from Git (OutOfSync) or resources are unhealthy (Degraded)…
- Helm: another operation (install/upgrade) is in progressA previous Helm operation crashed and left the release stuck in…
- Helm: cannot patch (field is immutable)An upgrade changed a field Kubernetes does not allow to be updated in place, such as a…
- Helm: release stuck in failed or pending stateA previous Helm install or upgrade failed and left the release in a state that blocks later…
- Istio: 503 upstream connect error (UH/UF/NR)Envoy returned 503 with flags like UH (no healthy upstream), UF (upstream connection…
- kubectl apply: metadata.annotations: Too longkubectl apply stores the whole previous manifest in the last-applied-configuration…
- kubectl top: error: Metrics API not availablekubectl top and every horizontal pod autoscaler read from metrics-server, which no upstream…
- kubectl: error: You must be logged in to the server (Unauthorized)The API server rejected the credentials in your kubeconfig. Typically an expired token or…
Browse other categories
- HTTP 494xx client errors, 5xx server errors, redirects, headers and protocol problems.
- JavaScript 42npm resolution, async pitfalls, hydration, memory limits and runtime type…
- Database 41Connections, deadlocks, constraints, replication and memory limits.
- AI 35Rate limits, context windows, GPU memory and model-serving failures.
- Network 35Refused connections, timeouts, resets, MTU problems and port exhaustion.
- Python 35Imports, virtual environments, encoding, concurrency and dependency conflicts.
- Docker 27Daemon connectivity, disk space, image pulls, ports and architecture mismatches.
- System 26Disk space, systemd units, file descriptors, OOM killer and scheduled jobs.
- Cloud 25IAM permissions, quotas, service limits and credential failures.
- Security 25JWT validation, CSRF, OAuth grants, SELinux, SSH host keys and CSP.
- TLS 24Untrusted authorities, expiry, hostname mismatch, chains and cipher negotiation.
- Frontend 23Hydration mismatches, bundler resolution, layout shift and font loading.
Something missing or wrong?
This entry is maintained by hand. If the fix is out of date, incomplete, or you have a better one, email a correction and it will be reviewed.