A pod in Pending is a pod accepted by the API server, one or more of whose
containers have not been created yet. In practice this very often
means a scheduling problem (no acceptable node found), but the Pending
phase can also cover other steps that come before the actual start. This guide
gives you a 4-step method to identify the cause and unblock the situation.
What you will learn
- Understand why a pod stays in Pending and what the scheduler does
- Identify the 7 main causes of a blockage, one of which, the quota, is a case where the Pod is never created
- Apply a reproducible 4-step diagnostic method
- Use
kubectl describe podandkubectl get eventsto find the real cause - Reproduce and fix a Pending pod on a test cluster
Prerequisites
- A working Kubernetes cluster (the lesson is written for 1.37)
kubectlconfigured and connected to the cluster- Read rights on pods, nodes and events
- Basic knowledge of Pods and Pod placement (labels, taints, affinities and tolerations)
What is a Pending pod?
Pending is an official phase of the Pod lifecycle. It means
that the Pod has been accepted by the API server, but that one or more containers
have not been created yet. This includes scheduling time (finding a
node) and preparation time (downloading images, for example).
This guide focuses on the scheduling case: the pod
stays stuck because the scheduler cannot find an acceptable node.
A Pending Pod consumes neither CPU nor memory, since no container is
running. Its requests do count against the namespace ResourceQuota,
however, which can block other creations.
The role of the scheduler
The Kubernetes scheduler (kube-scheduler) evaluates each waiting pod and
looks for a node that satisfies all the constraints:
| Constraint | What the scheduler checks |
|---|---|
| Resources | Does the node have enough CPU and memory available (requests)? |
| Taints/Tolerations | Does the pod tolerate the node's taints? |
| Node Selector | Does the node have the requested labels? |
| Node Affinity | Does the node match the affinity rules? |
| Pod Affinity/Anti-Affinity | Is the placement compatible with the other pods? |
| PVC | Is the requested volume available in the same zone? |
| Topology constraints | Are the topology spread constraints met? |
If no node passes all the filters, the pod stays Pending and a
FailedScheduling event is emitted.
The 7 main causes
Here are the seven causes to know. For each one, the table gives the symptom to spot and where to read it:
| # | Cause | Typical symptom | Where to look |
|---|---|---|---|
| 1 | Insufficient resources | Insufficient cpu or Insufficient memory | kubectl describe pod → Events |
| 2 | Taint without toleration | had untolerated taint | kubectl describe pod → Events |
| 3 | Impossible node selector / node affinity | didn't match Pod's node affinity/selector | kubectl describe pod → Events |
| 4 | Unbound PVC | pod has unbound immediate PersistentVolumeClaims | kubectl describe pod + kubectl get pvc |
| 5 | Pod anti-affinity / topology spread too strict | placement constraint that cannot be satisfied | kubectl describe pod → Events |
| 6 | Quota or admission blocking (the Pod is not created) | exceeded quota or creation refused, with no Pod visible | FailedCreate events of the ReplicaSet / API response |
| 7 | Insufficient priority / preemption | the scheduler cannot free up useful room | kubectl describe pod → Events, then the lesson Priority and preemption |
Cause 6 is a special case: a Pod refused by a quota is never created, so it cannot be Pending. The refusal is written at creation time, and with a Deployment the exceeded quota message shows up on the ReplicaSet, in a FailedCreate event, not on a Pod.
4-step diagnostic method
-
Identify the Pending pods
List the pods and spot those whose STATUS is
Pending:Terminal window kubectl get pods -A --field-selector=status.phase=PendingIf the command returns nothing but a pod seems stuck, also check the recent events:
Terminal window kubectl get events -A --field-selector reason=FailedScheduling --sort-by='.lastTimestamp'Note the pod name, its namespace and how long it has been Pending.
-
Read the pod events
The essential command for a Pending pod is
kubectl describe pod, Events section:Terminal window kubectl describe pod <pod> -n <namespace>The
FailedSchedulingevent message tells you exactly why the scheduler found no node. Examples:0/3 nodes are available: 1 node(s) had untolerated taint{node-role.kubernetes.io/control-plane: }, 2 Insufficient memory.This message means: 3 nodes evaluated, 1 ruled out because of the control-plane taint, 2 ruled out for lack of memory.
The taint reason here is background noise: on a kubeadm cluster, or a multi-node kind, the control-plane node carries a default taint that always rules out an ordinary Pod, without being the cause of the blockage. A single-node kind removes this taint at creation. The real cause is the lack of memory.
0/3 nodes are available: 3 node(s) didn't match Pod's nodeaffinity/selector.This one means: no node has the labels requested by the
nodeSelectoror thenodeAffinity. -
Cross-check with the state of the nodes and resources
Depending on the error message, check:
For insufficient resources:
Terminal window # See the capacity and usage of each nodekubectl top nodes# Details of the allocations on a nodekubectl describe node <node-name> | grep -A10 "Allocated resources"For taints:
Terminal window # See the taints of each nodekubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taintsFor labels (nodeSelector/nodeAffinity):
Terminal window # See the labels of the nodeskubectl get nodes --show-labels# Check a specific labelkubectl get nodes -l <label-key>=<label-value>For PVCs:
Terminal window # See the state of the namespace's PVCskubectl get pvc -n <namespace>A PVC in
Pendingprevents the pod from starting, so the PVC has to be fixed first.For quotas:
Terminal window # See the namespace quotaskubectl describe resourcequota -n <namespace># Read the creation refusals (the exceeded quota message is here)kubectl get events -n <namespace> --field-selector reason=FailedCreateIf
kubectl get podsshows no Pod for your Deployment butkubectl describe deploymentdisplays theReplicaFailurecondition, this is the case: the ReplicaSet cannot create its Pods. -
Fix and verify
Depending on the cause identified:
Cause Corrective action Insufficient resources Reduce the pod's requests, or add nodes to the clusterTaint without toleration Add the tolerationto the manifest, or remove the taint from the nodeWrong Node Selector Fix the nodeSelectoror add the label to the nodeUnbound PVC Create the missing PV, check the StorageClass, or fix the zone Anti-affinity / topology spread Relax the rules ( preferredDuringSchedulinginstead ofrequired)Quota / admission Increase the quota, or reduce existing usage Insufficient priority Raise the pod's PriorityClass, or free up capacityAfter the fix, check that the pod leaves the Pending state:
Terminal window kubectl get pods -n <namespace> -wThe pod should go from
Pending→ContainerCreating→Running.
Practical examples: reproduce a Pending pod
Causing the failure on purpose is the best way to learn to read the scheduler's message: you already know the answer, so you can check that you would have found it. The four cases below cover the first four causes in the table, and can be played in a few minutes on a test cluster.
Case 1: insufficient resources
Create a pod that requests more CPU than the cluster can provide:
kubectl apply -f- <<'EOF'apiVersion: v1kind: Podmetadata: name: pending-resources namespace: defaultspec: containers: - name: app image: nginx:1.27@sha256:6784fb0834aa7dbbe12e3d7471e69c290df3e6ba810dc38b34ae33d3c1c05f7d resources: requests: cpu: "100" memory: "1Gi"EOFThe pod requests 100 CPUs, and no node can satisfy that request.
Diagnosis:
kubectl describe pod pending-resources | tail -10You will see a FailedScheduling event with the message
Insufficient cpu.
Fix: reduce requests.cpu to a realistic value (for example
100m for 0.1 CPU).
Case 2: nodeSelector with no match
kubectl apply -f- <<'EOF'apiVersion: v1kind: Podmetadata: name: pending-selector namespace: defaultspec: nodeSelector: disk: ssd-nvme containers: - name: app image: nginx:1.27@sha256:6784fb0834aa7dbbe12e3d7471e69c290df3e6ba810dc38b34ae33d3c1c05f7dEOFIf no node has the label disk=ssd-nvme, the pod will stay Pending.
Diagnosis:
kubectl describe pod pending-selector | tail -10kubectl get nodes --show-labels | grep ssd-nvmeFix: add the label to the node, or remove the nodeSelector:
kubectl label node <node-name> disk=ssd-nvmeCase 3: untolerated taint
If all the worker nodes carry a taint and the pod does not have the matching toleration, it will stay Pending. The order of the two commands matters: set the taint first, create the pod afterwards. This case assumes at least one worker node: on a single-node kind, the selector below matches nothing.
# 1. Set the taint on all worker nodeskubectl taint nodes -l '!node-role.kubernetes.io/control-plane' maintenance=true:NoSchedule
# 2. Only now, create the podkubectl apply -f- <<'EOF'apiVersion: v1kind: Podmetadata: name: pending-taint namespace: defaultspec: containers: - name: app image: nginx:1.27@sha256:6784fb0834aa7dbbe12e3d7471e69c290df3e6ba810dc38b34ae33d3c1c05f7dEOF
# 3. Check that the pod is Pendingkubectl describe pod pending-taint | tail -10In the reverse order, you would observe nothing: a NoSchedule taint
only affects placement, and it does not evict a pod that is already
running. A pod created earlier would stay quietly Running on its node,
taint or no taint. It is the NoExecute effect that dislodges existing pods, and this is
a distinction to know before setting a taint on a live cluster.
Fix: add the toleration to the manifest, or remove the taint:
# Remove the taintkubectl taint nodes -l '!node-role.kubernetes.io/control-plane' maintenance=true:NoSchedule-Case 4: unbound PVC
kubectl apply -f- <<'EOF'apiVersion: v1kind: PersistentVolumeClaimmetadata: name: pvc-inexistant namespace: defaultspec: accessModes: ["ReadWriteOnce"] storageClassName: "storage-class-inexistante" resources: requests: storage: 10Gi---apiVersion: v1kind: Podmetadata: name: pending-pvc namespace: defaultspec: containers: - name: app image: nginx:1.27@sha256:6784fb0834aa7dbbe12e3d7471e69c290df3e6ba810dc38b34ae33d3c1c05f7d volumeMounts: - name: data mountPath: /data volumes: - name: data persistentVolumeClaim: claimName: pvc-inexistantEOFThe PVC references a StorageClass that does not exist, so it stays Pending, and so does the pod.
Diagnosis:
kubectl get pvc pvc-inexistantkubectl describe pod pending-pvc | tail -10You will see pod has unbound immediate PersistentVolumeClaims.
Cleanup
These Pods will never start and clutter the output of
kubectl get pods. The --force --grace-period=0 options avoid a
pointless wait: since no container ever started, there is nothing to
stop cleanly. Also remove the taint set in case 3, otherwise the
node will refuse your next Pods.
kubectl delete pod pending-resources pending-selector pending-taint pending-pvc --force --grace-period=0 2>/dev/nullkubectl delete pvc pvc-inexistant 2>/dev/nullkubectl taint nodes -l '!node-role.kubernetes.io/control-plane' maintenance=true:NoSchedule- 2>/dev/nullSpecial cases
The three situations below do not appear in the causes table because they do not come from the pod itself: its manifest is correct, and it is the state of the cluster or a rule set by someone else that blocks the placement. They are recognized by a common detail: the pod waits and no line of the manifest explains why.
Prolonged Pending after a scale-up
When a Deployment increases its number of replicas, the scheduler handles
the Pods one by one. If the cluster is close to saturation, the
first ones are placed and the last ones stay Pending. The events
then show Insufficient cpu or Insufficient memory.
If you use a node autoscaling mechanism (Cluster Autoscaler, Karpenter...), the Pending pods can trigger the addition of capacity automatically. The delay depends heavily on the implementation and on the underlying infrastructure.
Adding a node by hand unblocks nothing until it is Ready, and a Pod's
request must fit on a single node: two half-free nodes do not add up.
Pending because of a PodDisruptionBudget
A PDB does not directly cause a Pending pod. But during a node drain, if the PDB prevents the eviction, the pods stay on the drained node and the new pods created to replace them can stay Pending for lack of resources on the remaining nodes.
Pending with topologySpreadConstraints
With topology spread constraints,
topologySpreadConstraints, set to whenUnsatisfiable: DoNotSchedule,
the scheduler refuses to place the Pod as soon as the spread can no longer
be respected. The events name the constraint that is not satisfied. If a perfect
spread is impossible, switch whenUnsatisfiable to ScheduleAnyway: the
constraint becomes a preference.
Quick troubleshooting
The first action is always the same, read the whole FailedScheduling event:
it gives the count of nodes ruled out and the reason for each. The
table below starts from what this message contains to take you to the
command that confirms it.
| Symptom | First action | Command |
|---|---|---|
Pending + Insufficient cpu/memory | Check node allocations | kubectl describe node <node> → Allocated resources |
Pending + untolerated taint | Check the taints | kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints |
Pending + didn't match node affinity/selector | Check the node labels | kubectl get nodes --show-labels |
Pending + unbound PersistentVolumeClaims | Check the PVC | kubectl get pvc -n <namespace> |
No Pod created + exceeded quota (FailedCreate event of the ReplicaSet) | Read the refusal events, then check the ResourceQuota | kubectl get events -n <namespace> --field-selector reason=FailedCreate, then kubectl describe resourcequota -n <namespace> |
Pending + no event | Scheduler down? | kubectl get pods -n kube-system -l component=kube-scheduler |
| Pod disappears instead of staying Pending | Check whether a webhook deletes it | kubectl get events -n <namespace> --sort-by='.lastTimestamp' |
The quota row is an exception to the FailedScheduling rule: the refusal happens before the scheduler, when the Pod is created, and it is the ReplicaSet that carries the event.
Key takeaways
Pendingmeans that one or more containers have not been created yet, and in most cases it is because the scheduler could not find an acceptable nodekubectl describe podis the key command, the Events section contains theFailedSchedulingmessage that explains exactly what is blockingInsufficient cpu/memoryis the first lead to rule out, check the pod'srequestsand the nodes'Allocated resources- Taints block silently: a node can have capacity but refuse the pod if it lacks the right toleration
- An unbound PVC blocks the pod, fix the PVC first (missing StorageClass, no PV available, wrong zone)
- On a cluster with PodPriority, a pod can stay Pending if its priority is too low to trigger a useful preemption
- An exceeded quota is not a
Pending: the Pod is not created, and theFailedCreateevent shows up on the ReplicaSet, not on a Pod - Set up an alert on prolonged Pending pods, it is a reliable symptom of a capacity or configuration problem
Knowledge check
Seven questions on reading the scheduler's message: the count of nodes, the reason that is only background noise on a small cluster, and what a taint changes or does not change.
Knowledge check
Check what you have learned with this interactive quiz
How it works
- The timer starts when you click Start
- Multiple choice, true/false and short answer questions
- You can move between questions
- Detailed results are shown at the end
Starts the quiz and the timer
Review
(0/0)Score by domain
A domain groups several guides and gets several questions, so its score is more reliable than a per-topic score, which often rests on a single question.
Summary
Strengths
No topic passed at 80% on at least two questions yet.
Weak areas
No topic below the pass mark.
Recommended next
Hints before you try again?
Continue the course: Diagnose a Kubernetes control plane failure
A new full quiz with random questions
Work again only on the questions you missed