Skip to content
Français
Account: not signed in, sync available My learning
Select theme
Français

Debug a Kubernetes application

By updated

Improve this page

Logo Kubernetes

A Pod does not start, crashes in a loop, or runs while the application does not answer: the diagnosis always starts from the symptom that kubectl get pods displays, and each symptom has a first command. In this lesson, you learn this method on four cases: a Pod blocked by a missing ConfigMap, a Pod that crashes, a Pod with no shell that you examine with an ephemeral container, and an application that is unreachable while its Pod is Running. The symptoms already covered elsewhere (image not found, Pending Pod, OOMKilled) are summarized with a link.

This lesson assumes that you have followed Pods, Services (the Service of the demonstration is provided without explanation), ConfigMaps and Requests and limits. You also have the demo cluster created in First cluster.

What you will learn

  • Choose the first diagnostic command from the status of the Pod
  • Read the events of a blocked Pod and unblock it without recreating it
  • Read the logs of a container, including kubectl logs --previous, and a State / Last State
  • Diagnose a failing kubectl exec and examine a Pod with no shell using kubectl debug
  • Locate in three steps an application that is unreachable while the Pod is running

Quick decision table

The status displayed by kubectl get pods guides the diagnosis: the table gives the first command for each symptom.

SymptomFirst commandFor what comes next
CreateContainerConfigErrorkubectl events --for pod/<pod>This lesson
ContainerCreating that lastskubectl events --for pod/<pod>This lesson
CrashLoopBackOff or Errorkubectl logs <pod> then --previousCrashLoopBackOff
ImagePullBackOffkubectl describe pod <pod>ImagePullBackOff (in French)
Pendingkubectl events --for pod/<pod>Pending Pods
OOMKilledkubectl describe pod <pod>Requests and limits
Running but unreachablekubectl describe service <svc>This lesson

For a complete incident method, see Incident diagnosis method (in French) and Analyze events (in French). To copy files (kubectl cp), measure the consumption (kubectl top) or open a direct access (kubectl port-forward), see Read the cluster state: kubectl get, describe, logs and top.

A Pod blocked by a missing ConfigMap

You work in a dedicated namespace, deleted at the end:

Terminal window
kubectl config use-context kind-demo && kubectl create namespace debogage
Switched to context "kind-demo".
namespace/debogage created

The names and messages of this lesson are in French on purpose, so that the outputs stay those of the tested cluster: debogage means "debugging", appli "app", reglages "settings", plante "crashes", minimal "minimal", lecteur "reader", debogueur "debugger", and sans-shell.yaml "no shell". The echo messages "démarrage", "erreur fatale : base introuvable", "même Pod" and "code de sortie" mean "startup", "fatal error: database not found", "same Pod" and "exit code". The file names config-absente.yaml, inaccessible.yaml and uid-avant.txt mean "missing config", "unreachable" and "uid before".

The appli Pod reads an environment variable from the reglages ConfigMap, which does not exist yet:

Terminal window
mkdir -p ~/atelier-k8s/debogage && cd ~/atelier-k8s/debogage
cat > config-absente.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: appli
spec:
containers:
- name: nginx
image: nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb
env:
- name: MODE
valueFrom:
configMapKeyRef:
name: reglages
key: MODE
EOF
kubectl apply -f config-absente.yaml -n debogage && sleep 10 && kubectl get pod appli -n debogage
pod/appli created
NAME READY STATUS RESTARTS AGE
appli 0/1 CreateContainerConfigError 0 10s

CreateContainerConfigError means that the kubelet could not prepare the configuration of the container, hence RESTARTS 0. The container was not even created, and the status does not say why. For the cause, read the Pod's events.

How do you read a Pod's events?

The kubectl events --for pod/<pod> command lists the events of an object in order. It is the first reflex when a Pod does not start, because the kubelet writes there what is blocking.

Terminal window
kubectl events -n debogage --for pod/appli
LAST SEEN TYPE REASON OBJECT MESSAGE
10s Normal Scheduled Pod/appli Successfully assigned debogage/appli to demo-control-plane
9s (x2 over 9s) Normal Pulled Pod/appli Container image "nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb" already present on machine and can be accessed by the pod
9s (x2 over 9s) Warning Failed Pod/appli Error: configmap "reglages" not found

The Warning Failed line gives the cause: configmap "reglages" not found. Create the missing ConfigMap, without touching the Pod, after noting its unique identifier (uid) to prove afterwards that the Pod was not recreated:

Terminal window
kubectl get pod appli -n debogage -o jsonpath='{.metadata.uid}' > uid-avant.txt && kubectl create configmap reglages -n debogage --from-literal=MODE=demo && kubectl wait --for=condition=Ready pod/appli -n debogage --timeout=180s
configmap/reglages created
pod/appli condition met

The Pod became ready on its own: the kubelet retries to create the container, and as soon as the ConfigMap exists, it succeeds. Here, the Pod is ready one second after the creation; the delay can be longer, because the wait between two attempts grows. Check that it is the same Pod:

Terminal window
test "$(cat uid-avant.txt)" = "$(kubectl get pod appli -n debogage -o jsonpath='{.metadata.uid}')" && echo "même Pod" && kubectl get pod appli -n debogage
même Pod
NAME READY STATUS RESTARTS AGE
appli 1/1 Running 0 11s

The uid did not change: fixing the cause is enough, there is nothing to recreate.

The same missing ConfigMap can also be mounted as a volume instead of being read in an environment variable. The symptom changes: the Pod stays in ContainerCreating, because the kubelet has to prepare the volume before creating the container. The Pod's events then carry a Warning with the reason FailedMount, whose message names the ConfigMap that cannot be found. The approach does not change: you read the events, you note the cause, then you create the ConfigMap with the expected content. The kubelet retries to mount the volume, and the Pod starts on its own, without being recreated.

Read a Pod: exec, logs and state

How do you open a shell in a container?

kubectl exec runs a command in a running container. To open an interactive shell, add -it and run sh: kubectl exec -it appli -n debogage -- sh. Here, the command is a one-off, to observe what exec returns. The exit code of the command is relayed: the command terminated with exit code N message repeats the code of the command that was run, not a state of the Pod.

Terminal window
kubectl exec appli -n debogage -- sh -c 'echo "MODE=$MODE"; exit 137'; echo "code de sortie : $?"
MODE=demo
command terminated with exit code 137
code de sortie : 137

The MODE variable comes from the ConfigMap created in the previous step. The code 137 is the one that the command chose, simulated here: a code 137 relayed by exec does not prove that the container was killed. To run a command in a specific container of a multi-container Pod, add -c <name>.

How do you read a container's logs?

kubectl logs <pod> displays the output of a container. On a multi-container Pod, the command picks the default container and announces it with the Defaulted container ... out of: ... line; to read another one, use -c, as shown in Init containers and sidecars.

The plante Pod prints a startup line, an error on the error output, waits 15 seconds then stops with a failure. The wait leaves time to read the logs of the previous attempt while the new attempt runs:

Terminal window
cat > plante.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: plante
spec:
containers:
- name: app
image: busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4
command:
- sh
- -c
- |
echo "démarrage"
echo "erreur fatale : base introuvable" >&2
sleep 15
exit 1
EOF
kubectl apply -f plante.yaml -n debogage && sleep 17 && kubectl get pod plante -n debogage && kubectl logs plante -n debogage --previous
pod/plante created
NAME READY STATUS RESTARTS AGE
plante 1/1 Running 1 (1s ago) 17s
erreur fatale : base introuvable
démarrage

RESTARTS 1: the first attempt failed and a second one is running. The --previous option reads the logs of the attempt that has just ended, which gives the cause of the crash: base introuvable (database not found). The order of the two lines is not guaranteed, because the standard output and the error output are two distinct streams: you may see démarrage before or after the error.

One limit is worth knowing: the kubelet only keeps the last stopped container. When the current attempt has also ended, the "previous" one has already been deleted and kubectl logs --previous answers unable to retrieve container logs for containerd://…. In that case, kubectl logs <pod> without --previous returns the logs of the last attempt.

For the state, kubectl describe pod plante -n debogage displays State (the current state of the container) and Last State (that of the previous attempt), with the reason (Reason: Error), the exit code (Exit Code: 1) and the start and end times. Just after the restart, the container's section looks like this (the times are those of your run):

State: Running
Started: Wed, 11 Nov 2026 16:09:43 +0000
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 11 Nov 2026 16:09:28 +0000
Finished: Wed, 11 Nov 2026 16:09:43 +0000
Ready: True

Last State keeps the first attempt: it lasted 15 seconds, between Started and Finished, then failed with code 1.

Fifteen seconds later, the second attempt has also ended: the Pod is 0/1 Error and State becomes Terminated with Reason: Error and Exit Code: 1, while Last State stays the same. The current state therefore depends on the moment: Terminated just after a failure, Waiting with CrashLoopBackOff during the wait between two attempts, Running during an attempt.

When kubectl exec fails: read the message

A failing exec gives a message that guides the diagnosis: the image has no shell, the container does not exist (or not yet), or you do not have the right. The minimal Pod uses the pause image, which contains a single executable:

Terminal window
cat > sans-shell.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: minimal
spec:
containers:
- name: pause
image: registry.k8s.io/pause:3.10@sha256:ee6521f290b2168b6e0935a181d4cff9be1ac3f505666ef0e3c98fae8199917a
EOF
kubectl apply -f sans-shell.yaml -n debogage && kubectl wait --for=condition=Ready pod/minimal -n debogage --timeout=90s && kubectl exec minimal -n debogage -- sh
pod/minimal created
pod/minimal condition met
error: Internal error occurred: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "95e2f199a9ac1b2e5512fb51243849f5d8b384f65f25374c5b9875556d4beaff": OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH

The useful message is at the end: exec: "sh": executable file not found in $PATH. The image has no shell at all, and this is not a permissions problem. The solution is an ephemeral container, seen further down.

The other possible cause remains, the permission. Entering a container is a permission distinct from reading Pods: kubectl exec first reads the Pod, then asks for the create right on the pods/exec subresource. The kubectl auth can-i create pods/exec command tests this second right. With your administrator account on the demo cluster, the answer is positive:

Terminal window
kubectl auth can-i create pods/exec -n debogage
yes

To see a refusal, create a service account (ServiceAccount, the identity that a program uses in the cluster) that has received no rights. The --as option then runs the check under its identity:

Terminal window
kubectl create serviceaccount lecteur -n debogage && kubectl auth can-i create pods/exec -n debogage --as=system:serviceaccount:debogage:lecteur
serviceaccount/lecteur created
no

The no explains a refused exec: this account does not have the create right on pods/exec. A refused exec answers with a Forbidden error, which must not be confused with the missing shell message. An account with no rights at all can already be refused when reading the Pod, before reaching pods/exec. To grant this right, see RBAC.

How do you examine a Pod with an ephemeral container?

An ephemeral container is a container added to a running Pod, without recreating it, to examine it with tools that the original image does not have. The --target option makes it share the process space of a container of the Pod. Add a busybox container to the minimal Pod:

Terminal window
kubectl debug minimal -n debogage -c debogueur --image=busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4 --target=pause -- sleep 600
Targeting container "pause". If you don't see processes from this container it may be because the container runtime doesn't support this feature.

The command returns immediately (without -it, it does not attach a terminal). The message says that the ephemeral container targets pause. An immediate exec in this new container fails, because it has not started yet:

Terminal window
kubectl exec minimal -n debogage -c debogueur -- ps
error: unable to upgrade connection: container not found ("debogueur")

container not found does not mean that the container is absent, but that the container is not yet ready to receive an exec. Wait for it with a short loop: it starts in a few seconds when the image is already on the node, and the loop displays the same message at each failed attempt before ps answers.

Terminal window
for i in $(seq 1 20); do kubectl exec minimal -n debogage -c debogueur -- ps && break; sleep 1; done
error: unable to upgrade connection: container not found ("debogueur")
error: unable to upgrade connection: container not found ("debogueur")
PID USER TIME COMMAND
1 65535 0:00 /pause
21 root 0:00 sleep 600
34 root 0:00 ps

The first two lines are the attempts where the container did not exist yet; the number of error lines varies with the startup time. The successful ps is displayed next and the loop stops (break).

The /pause process, PID 1, belongs to the target container: this is the effect of --target. Without this option, only the processes of the ephemeral container are visible. Finally, check that the original Pod was not touched:

Terminal window
kubectl get pod minimal -n debogage -o jsonpath='{.spec.ephemeralContainers[*].name}{"\n"}{.status.containerStatuses[0].restartCount}{"\n"}'
debogueur
0

The ephemeral container is recorded in the Pod's spec and the original container did not restart.

Pod Running but application unreachable

The Pod is Running and the application does not answer: the fault lies between the client and the application, and three steps locate it. The inaccessible.yaml file creates a healthy web Deployment, a web-svc Service whose selector contains a typo, and a client Pod for the tests:

Terminal window
cat > inaccessible.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
replicas: 1
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
containers:
- name: nginx
image: nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: web-svc
spec:
selector:
app: wev # ❌ faute de frappe : aucun Pod ne porte ce label
ports:
- port: 80
targetPort: 80
---
apiVersion: v1
kind: Pod
metadata:
name: client
spec:
containers:
- name: busybox
image: busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4
command: ["sh", "-c", "sleep 3600"]
EOF
kubectl apply -f inaccessible.yaml -n debogage && kubectl rollout status deployment/web -n debogage --timeout=90s && kubectl wait --for=condition=Ready pod/client -n debogage --timeout=90s
deployment.apps/web created
service/web-svc created
pod/client created
Waiting for deployment "web" rollout to finish: 0 of 1 updated replicas are available...
deployment "web" successfully rolled out
pod/client condition met

The comment in the manifest means "typo: no Pod carries this label".

Step 1: Check that the Pod answers directly

The client Pod calls the IP address of the web Pod, bypassing the Service. If the Pod answers, the application is not at fault:

Terminal window
POD_IP=$(kubectl get pod -n debogage -l app=web -o jsonpath='{.items[0].status.podIP}'); kubectl exec client -n debogage -- wget -qO- -T 3 "http://$POD_IP" | head -n 4
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>

The nginx welcome page is displayed: the Pod and the application work. The problem is therefore at the Service level.

Step 2: Check the Service and the Endpoints

Read the selector of the Service, the list of its addresses (Endpoints) and the real labels of the Pods:

Terminal window
kubectl describe service web-svc -n debogage | grep -E "Selector|Endpoints"; kubectl get pods -n debogage --show-labels -l app=web
Selector: app=wev
Endpoints:
NAME READY STATUS RESTARTS AGE LABELS
web-5f784567c7-sfkb7 1/1 Running 0 1s app=web,pod-template-hash=5f784567c7

The selector of the Service is app=wev, the Pod carries app=web: the typo jumps out. The Endpoints: line is empty, which confirms that no Pod is selected.

Step 3: Test connectivity from another Pod

Reproduce what a real client sees: call the Service by its name, from another Pod.

Terminal window
kubectl exec client -n debogage -- wget -qO- -T 3 http://web-svc; echo "code de sortie : $?"
wget: can't connect to remote host (10.96.146.126): Connection refused
command terminated with exit code 1
code de sortie : 1

The Service has an address, but it has no Pod behind it: the connection is refused. Fix the selector and run the test again:

Terminal window
kubectl patch service web-svc -n debogage -p '{"spec":{"selector":{"app":"web"}}}' && sleep 3 && kubectl exec client -n debogage -- wget -qO- -T 3 http://web-svc | head -n 4
service/web-svc patched
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>

By the name of the Service, the page is displayed. The three steps located the fault: the Pod answers, the Service does not select it, and once the selector is fixed the client reaches the application.

The other symptoms, summarized

Four symptoms have their own lesson. This page summarizes them in a few lines and points to the lesson concerned.

CrashLoopBackOff

The container starts, stops with an error, and the kubelet waits longer and longer before restarting it. Read kubectl logs <pod> and --previous as in this lesson, then kubectl describe for the exit code. The detail is in CrashLoopBackOff.

ImagePullBackOff

The kubelet cannot download the image: wrong name or tag, unreachable registry, missing credentials. The Pod's events give the cause. The detail is in ImagePullBackOff (in French).

Pending

The scheduler found no node for the Pod: requests too large, taint, missing volume. The FailedScheduling event says why, as in Requests and limits. The detail is in Pending Pods.

OOMKilled

The container exceeded its memory limit: exit code 137 and reason OOMKilled in Last State. The case, the fix and the resize are in Requests and limits.

Clean up the cluster

Delete the namespace: it takes away the Pods, the Service, the ConfigMap, the service account and the Deployment of the demonstration. Also remove the file that contained the uid of the first Pod.

Terminal window
kubectl delete namespace debogage && rm -f uid-avant.txt
namespace "debogage" deleted

Put it into practice

Two guided labs reproduce the gestures of the demonstration, for about 25 minutes in total.

In the first one, A Pod blocked by a ConfigMap that does not exist, a Pod does not start and says nothing in its logs. You read the cause in its events, you record it in a file before any repair, then you create the ConfigMap with the expected content. The validation checks that the cause is recorded before the repair, that the ConfigMap carries the right configuration, that the Pod runs without having been recreated and that the application serves its configuration. Here, the ConfigMap is mounted as a volume, the case described at the end of the section on the missing ConfigMap.

In the second one, Get inside a container with no shell using kubectl debug, you attach an ephemeral container to a Pod with no shell. The validation checks that this container shares the processes of the application and that the process list read from it shows those of the application.

Challenge: ship a robust manifest from a set of requirements

The challenge is the unaided proof of the module: it gives a set of requirements, without the commands, and it is graded on the state of the cluster that you leave behind, not on the commands typed. You deliver an application in two variants, dev and prod, from a single Kustomize base, with an init container, a native sidecar, requests and limits and three probes. You schedule a CronJob, you bring a Job to completion, and you repair a failing Pod, after recording its cause. The challenge builds on this lesson, on Init containers and sidecars, Application volumes, Requests and limits, Probes, Jobs and CronJobs and Manage images for the digest.

The set of requirements has ten requirements of equal weight, each checked by a test, and passing requires 70 points out of 100. Three requirements are blocking: they must pass whatever the total. They are the production application available on its image by digest, the three probes set with Pods ready and no restart, and the cause of the fault recorded before the repair. No hint is offered: you have only the specification and what the lesson taught you. Each requirement is worth 10 points, so at least seven requirements met, blocking ones included, reach the threshold of 70.

Key takeaways

  1. The status of kubectl get pods gives the first command: kubectl events --for pod/<pod> for a blocked Pod, kubectl logs for a Pod that crashes.
  2. CreateContainerConfigError means that the container was not created. A missing ConfigMap is read in the events: Failed when it is read in a variable, FailedMount (Pod in ContainerCreating) when it is mounted as a volume. In both cases, the Pod starts on its own as soon as it exists.
  3. kubectl logs --previous reads the previous attempt; the kubelet only keeps the last stopped container.
  4. Without -c, kubectl logs chooses the default container and announces it with Defaulted container.
  5. command terminated with exit code N after kubectl exec repeats the code of the command that was run, not a state of the Pod.
  6. exec: "sh": executable file not found means that the image has no shell; kubectl debug adds an ephemeral container, with --target to see its processes.
  7. The right to use exec is distinct from reading Pods: an exec requires reading the Pod and create on pods/exec, and kubectl auth can-i create pods/exec tests this second right.
  8. For an unreachable application, three steps: test the Pod directly, read the selector and the Endpoints of the Service, test the Service from another Pod.

Test your knowledge

This quiz covers the choice of the first command, the reading of the kubectl exec and kubectl logs messages, and the location of a Service fault. Allow five minutes and aim for 80%; below that, reread "Pod Running but application unreachable".

Knowledge check

Check what you have learned with this interactive quiz

7 questions
5 min
80% to pass

How it works

  • The timer starts when you click Start
  • Multiple choice, true/false and short answer questions
  • You can move between questions
  • Detailed results are shown at the end

Starts the quiz and the timer