Skip to content
Français
Account: not signed in, sync available My learning
Select theme
Français

Diagnose a Kubernetes Pending Pod

By updated

Improve this page

Logo Kubernetes

A pod in Pending is a pod accepted by the API server, one or more of whose containers have not been created yet. In practice this very often means a scheduling problem (no acceptable node found), but the Pending phase can also cover other steps that come before the actual start. This guide gives you a 4-step method to identify the cause and unblock the situation.

What you will learn

  • Understand why a pod stays in Pending and what the scheduler does
  • Identify the 7 main causes of a blockage, one of which, the quota, is a case where the Pod is never created
  • Apply a reproducible 4-step diagnostic method
  • Use kubectl describe pod and kubectl get events to find the real cause
  • Reproduce and fix a Pending pod on a test cluster

Prerequisites

  • A working Kubernetes cluster (the lesson is written for 1.37)
  • kubectl configured and connected to the cluster
  • Read rights on pods, nodes and events
  • Basic knowledge of Pods and Pod placement (labels, taints, affinities and tolerations)

What is a Pending pod?

Pending is an official phase of the Pod lifecycle. It means that the Pod has been accepted by the API server, but that one or more containers have not been created yet. This includes scheduling time (finding a node) and preparation time (downloading images, for example). This guide focuses on the scheduling case: the pod stays stuck because the scheduler cannot find an acceptable node.

A Pending Pod consumes neither CPU nor memory, since no container is running. Its requests do count against the namespace ResourceQuota, however, which can block other creations.

The role of the scheduler

The Kubernetes scheduler (kube-scheduler) evaluates each waiting pod and looks for a node that satisfies all the constraints:

ConstraintWhat the scheduler checks
ResourcesDoes the node have enough CPU and memory available (requests)?
Taints/TolerationsDoes the pod tolerate the node's taints?
Node SelectorDoes the node have the requested labels?
Node AffinityDoes the node match the affinity rules?
Pod Affinity/Anti-AffinityIs the placement compatible with the other pods?
PVCIs the requested volume available in the same zone?
Topology constraintsAre the topology spread constraints met?

If no node passes all the filters, the pod stays Pending and a FailedScheduling event is emitted.

The 7 main causes

Here are the seven causes to know. For each one, the table gives the symptom to spot and where to read it:

#CauseTypical symptomWhere to look
1Insufficient resourcesInsufficient cpu or Insufficient memorykubectl describe pod → Events
2Taint without tolerationhad untolerated taintkubectl describe pod → Events
3Impossible node selector / node affinitydidn't match Pod's node affinity/selectorkubectl describe pod → Events
4Unbound PVCpod has unbound immediate PersistentVolumeClaimskubectl describe pod + kubectl get pvc
5Pod anti-affinity / topology spread too strictplacement constraint that cannot be satisfiedkubectl describe pod → Events
6Quota or admission blocking (the Pod is not created)exceeded quota or creation refused, with no Pod visibleFailedCreate events of the ReplicaSet / API response
7Insufficient priority / preemptionthe scheduler cannot free up useful roomkubectl describe pod → Events, then the lesson Priority and preemption

Cause 6 is a special case: a Pod refused by a quota is never created, so it cannot be Pending. The refusal is written at creation time, and with a Deployment the exceeded quota message shows up on the ReplicaSet, in a FailedCreate event, not on a Pod.

4-step diagnostic method

  1. Identify the Pending pods

    List the pods and spot those whose STATUS is Pending:

    Terminal window
    kubectl get pods -A --field-selector=status.phase=Pending

    If the command returns nothing but a pod seems stuck, also check the recent events:

    Terminal window
    kubectl get events -A --field-selector reason=FailedScheduling --sort-by='.lastTimestamp'

    Note the pod name, its namespace and how long it has been Pending.

  2. Read the pod events

    The essential command for a Pending pod is kubectl describe pod, Events section:

    Terminal window
    kubectl describe pod <pod> -n <namespace>

    The FailedScheduling event message tells you exactly why the scheduler found no node. Examples:

    0/3 nodes are available: 1 node(s) had untolerated taint
    {node-role.kubernetes.io/control-plane: }, 2 Insufficient memory.

    This message means: 3 nodes evaluated, 1 ruled out because of the control-plane taint, 2 ruled out for lack of memory.

    The taint reason here is background noise: on a kubeadm cluster, or a multi-node kind, the control-plane node carries a default taint that always rules out an ordinary Pod, without being the cause of the blockage. A single-node kind removes this taint at creation. The real cause is the lack of memory.

    0/3 nodes are available: 3 node(s) didn't match Pod's node
    affinity/selector.

    This one means: no node has the labels requested by the nodeSelector or the nodeAffinity.

  3. Cross-check with the state of the nodes and resources

    Depending on the error message, check:

    For insufficient resources:

    Terminal window
    # See the capacity and usage of each node
    kubectl top nodes
    # Details of the allocations on a node
    kubectl describe node <node-name> | grep -A10 "Allocated resources"

    For taints:

    Terminal window
    # See the taints of each node
    kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

    For labels (nodeSelector/nodeAffinity):

    Terminal window
    # See the labels of the nodes
    kubectl get nodes --show-labels
    # Check a specific label
    kubectl get nodes -l <label-key>=<label-value>

    For PVCs:

    Terminal window
    # See the state of the namespace's PVCs
    kubectl get pvc -n <namespace>

    A PVC in Pending prevents the pod from starting, so the PVC has to be fixed first.

    For quotas:

    Terminal window
    # See the namespace quotas
    kubectl describe resourcequota -n <namespace>
    # Read the creation refusals (the exceeded quota message is here)
    kubectl get events -n <namespace> --field-selector reason=FailedCreate

    If kubectl get pods shows no Pod for your Deployment but kubectl describe deployment displays the ReplicaFailure condition, this is the case: the ReplicaSet cannot create its Pods.

  4. Fix and verify

    Depending on the cause identified:

    CauseCorrective action
    Insufficient resourcesReduce the pod's requests, or add nodes to the cluster
    Taint without tolerationAdd the toleration to the manifest, or remove the taint from the node
    Wrong Node SelectorFix the nodeSelector or add the label to the node
    Unbound PVCCreate the missing PV, check the StorageClass, or fix the zone
    Anti-affinity / topology spreadRelax the rules (preferredDuringScheduling instead of required)
    Quota / admissionIncrease the quota, or reduce existing usage
    Insufficient priorityRaise the pod's PriorityClass, or free up capacity

    After the fix, check that the pod leaves the Pending state:

    Terminal window
    kubectl get pods -n <namespace> -w

    The pod should go from Pending → ContainerCreating → Running.

Practical examples: reproduce a Pending pod

Causing the failure on purpose is the best way to learn to read the scheduler's message: you already know the answer, so you can check that you would have found it. The four cases below cover the first four causes in the table, and can be played in a few minutes on a test cluster.

Case 1: insufficient resources

Create a pod that requests more CPU than the cluster can provide:

Terminal window
kubectl apply -f- <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: pending-resources
namespace: default
spec:
containers:
- name: app
image: nginx:1.27@sha256:6784fb0834aa7dbbe12e3d7471e69c290df3e6ba810dc38b34ae33d3c1c05f7d
resources:
requests:
cpu: "100"
memory: "1Gi"
EOF

The pod requests 100 CPUs, and no node can satisfy that request.

Diagnosis:

Terminal window
kubectl describe pod pending-resources | tail -10

You will see a FailedScheduling event with the message Insufficient cpu.

Fix: reduce requests.cpu to a realistic value (for example 100m for 0.1 CPU).

Case 2: nodeSelector with no match

Terminal window
kubectl apply -f- <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: pending-selector
namespace: default
spec:
nodeSelector:
disk: ssd-nvme
containers:
- name: app
image: nginx:1.27@sha256:6784fb0834aa7dbbe12e3d7471e69c290df3e6ba810dc38b34ae33d3c1c05f7d
EOF

If no node has the label disk=ssd-nvme, the pod will stay Pending.

Diagnosis:

Terminal window
kubectl describe pod pending-selector | tail -10
kubectl get nodes --show-labels | grep ssd-nvme

Fix: add the label to the node, or remove the nodeSelector:

Terminal window
kubectl label node <node-name> disk=ssd-nvme

Case 3: untolerated taint

If all the worker nodes carry a taint and the pod does not have the matching toleration, it will stay Pending. The order of the two commands matters: set the taint first, create the pod afterwards. This case assumes at least one worker node: on a single-node kind, the selector below matches nothing.

Terminal window
# 1. Set the taint on all worker nodes
kubectl taint nodes -l '!node-role.kubernetes.io/control-plane' maintenance=true:NoSchedule
# 2. Only now, create the pod
kubectl apply -f- <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: pending-taint
namespace: default
spec:
containers:
- name: app
image: nginx:1.27@sha256:6784fb0834aa7dbbe12e3d7471e69c290df3e6ba810dc38b34ae33d3c1c05f7d
EOF
# 3. Check that the pod is Pending
kubectl describe pod pending-taint | tail -10

In the reverse order, you would observe nothing: a NoSchedule taint only affects placement, and it does not evict a pod that is already running. A pod created earlier would stay quietly Running on its node, taint or no taint. It is the NoExecute effect that dislodges existing pods, and this is a distinction to know before setting a taint on a live cluster.

Fix: add the toleration to the manifest, or remove the taint:

Terminal window
# Remove the taint
kubectl taint nodes -l '!node-role.kubernetes.io/control-plane' maintenance=true:NoSchedule-

Case 4: unbound PVC

Terminal window
kubectl apply -f- <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-inexistant
namespace: default
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: "storage-class-inexistante"
resources:
requests:
storage: 10Gi
---
apiVersion: v1
kind: Pod
metadata:
name: pending-pvc
namespace: default
spec:
containers:
- name: app
image: nginx:1.27@sha256:6784fb0834aa7dbbe12e3d7471e69c290df3e6ba810dc38b34ae33d3c1c05f7d
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: pvc-inexistant
EOF

The PVC references a StorageClass that does not exist, so it stays Pending, and so does the pod.

Diagnosis:

Terminal window
kubectl get pvc pvc-inexistant
kubectl describe pod pending-pvc | tail -10

You will see pod has unbound immediate PersistentVolumeClaims.

Cleanup

These Pods will never start and clutter the output of kubectl get pods. The --force --grace-period=0 options avoid a pointless wait: since no container ever started, there is nothing to stop cleanly. Also remove the taint set in case 3, otherwise the node will refuse your next Pods.

Terminal window
kubectl delete pod pending-resources pending-selector pending-taint pending-pvc --force --grace-period=0 2>/dev/null
kubectl delete pvc pvc-inexistant 2>/dev/null
kubectl taint nodes -l '!node-role.kubernetes.io/control-plane' maintenance=true:NoSchedule- 2>/dev/null

Special cases

The three situations below do not appear in the causes table because they do not come from the pod itself: its manifest is correct, and it is the state of the cluster or a rule set by someone else that blocks the placement. They are recognized by a common detail: the pod waits and no line of the manifest explains why.

Prolonged Pending after a scale-up

When a Deployment increases its number of replicas, the scheduler handles the Pods one by one. If the cluster is close to saturation, the first ones are placed and the last ones stay Pending. The events then show Insufficient cpu or Insufficient memory.

If you use a node autoscaling mechanism (Cluster Autoscaler, Karpenter...), the Pending pods can trigger the addition of capacity automatically. The delay depends heavily on the implementation and on the underlying infrastructure.

Adding a node by hand unblocks nothing until it is Ready, and a Pod's request must fit on a single node: two half-free nodes do not add up.

Pending because of a PodDisruptionBudget

A PDB does not directly cause a Pending pod. But during a node drain, if the PDB prevents the eviction, the pods stay on the drained node and the new pods created to replace them can stay Pending for lack of resources on the remaining nodes.

Pending with topologySpreadConstraints

With topology spread constraints, topologySpreadConstraints, set to whenUnsatisfiable: DoNotSchedule, the scheduler refuses to place the Pod as soon as the spread can no longer be respected. The events name the constraint that is not satisfied. If a perfect spread is impossible, switch whenUnsatisfiable to ScheduleAnyway: the constraint becomes a preference.

Quick troubleshooting

The first action is always the same, read the whole FailedScheduling event: it gives the count of nodes ruled out and the reason for each. The table below starts from what this message contains to take you to the command that confirms it.

SymptomFirst actionCommand
Pending + Insufficient cpu/memoryCheck node allocationskubectl describe node <node> → Allocated resources
Pending + untolerated taintCheck the taintskubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
Pending + didn't match node affinity/selectorCheck the node labelskubectl get nodes --show-labels
Pending + unbound PersistentVolumeClaimsCheck the PVCkubectl get pvc -n <namespace>
No Pod created + exceeded quota (FailedCreate event of the ReplicaSet)Read the refusal events, then check the ResourceQuotakubectl get events -n <namespace> --field-selector reason=FailedCreate, then kubectl describe resourcequota -n <namespace>
Pending + no eventScheduler down?kubectl get pods -n kube-system -l component=kube-scheduler
Pod disappears instead of staying PendingCheck whether a webhook deletes itkubectl get events -n <namespace> --sort-by='.lastTimestamp'

The quota row is an exception to the FailedScheduling rule: the refusal happens before the scheduler, when the Pod is created, and it is the ReplicaSet that carries the event.

Key takeaways

  • Pending means that one or more containers have not been created yet, and in most cases it is because the scheduler could not find an acceptable node
  • kubectl describe pod is the key command, the Events section contains the FailedScheduling message that explains exactly what is blocking
  • Insufficient cpu/memory is the first lead to rule out, check the pod's requests and the nodes' Allocated resources
  • Taints block silently: a node can have capacity but refuse the pod if it lacks the right toleration
  • An unbound PVC blocks the pod, fix the PVC first (missing StorageClass, no PV available, wrong zone)
  • On a cluster with PodPriority, a pod can stay Pending if its priority is too low to trigger a useful preemption
  • An exceeded quota is not a Pending: the Pod is not created, and the FailedCreate event shows up on the ReplicaSet, not on a Pod
  • Set up an alert on prolonged Pending pods, it is a reliable symptom of a capacity or configuration problem

Knowledge check

Seven questions on reading the scheduler's message: the count of nodes, the reason that is only background noise on a small cluster, and what a taint changes or does not change.

Knowledge check

Check what you have learned with this interactive quiz

7 questions
5 min
80% to pass

How it works

  • The timer starts when you click Start
  • Multiple choice, true/false and short answer questions
  • You can move between questions
  • Detailed results are shown at the end

Starts the quiz and the timer