Skip to content
Français
Account: not signed in, sync available My learning
Select theme
Français

Diagnose a Kubernetes CrashLoopBackOff

By updated

Improve this page

Logo Kubernetes

CrashLoopBackOff means that a container crashes, restarts, then crashes again in a loop. The first restart is immediate, then the kubelet waits 10 s, 20 s, 40 s, and so on up to 5 min at most, to avoid overloading the node. This guide gives you a 4-step method to identify the cause and fix the problem.

What you will learn

  • Understand the exponential backoff mechanism and why Kubernetes increases the delay between restarts
  • Identify the six common causes of a CrashLoopBackOff
  • Apply a repeatable diagnostic method in 4 steps
  • Use kubectl logs --previous and kubectl describe pod to find the real cause
  • Reproduce and fix a CrashLoopBackOff on a test cluster

Prerequisites

  • A working Kubernetes cluster (the lesson is written for 1.37)
  • kubectl configured and connected to the cluster
  • Read rights on pods, events and logs
  • Basic knowledge of Pods and probes

What is a CrashLoopBackOff?

CrashLoopBackOff is not a Pod phase. It is a waiting state visible in the STATUS column of kubectl get pods when a container fails then restarts in a loop with backoff. The phase of the Pod may very well stay Running during this time, and the status only shows during the wait between two attempts: what matters is that the container restarts in a loop.

The exponential backoff

When a container crashes, the kubelet applies a growing delay before each new restart:

RestartWait before this restartTime elapsed since the first restart
1stimmediate0 s
2nd10 seconds10 s
3rd20 seconds30 s
4th40 seconds1 min 10 s
5th80 seconds2 min 30 s
6th160 seconds5 min 10 s
7th and following300 seconds (maximum)+5 min per restart

If the container runs 10 minutes without crashing, the backoff counter is reset to zero.

These values are those of a default cluster. Since Kubernetes 1.35, an administrator can lower the 5-minute ceiling through crashLoopBackOff.maxContainerRestartPeriod in the kubelet configuration. If the delays you observe do not match the table, this is the first thing to check with the team that operates the cluster.

What are the six common causes of a CrashLoopBackOff?

Here are six common causes, with the symptom that gives them away and the place to look:

#CauseTypical symptomWhere to look
1Application errorException, panic, segfault in the logskubectl logs
2Missing configurationEnv variable absent, config file not foundkubectl describe pod (env, volumes)
3OOMKilledThe container exceeds its memory limitkubectl describe pod → Last State: OOMKilled
4Misconfigured probe (liveness or startup)Unhealthy, Killing, repeated restartskubectl describe pod → Events, probe definition
5Invalid command or entrypointThe container stops immediately, in StartError or with exit code 126/127kubectl describe pod, kubectl logs --previous
6Unavailable external dependencyThe database or the API cannot be reachedkubectl logs (timeout, connection refused)

Four-step diagnostic method

  1. Identify the pod in CrashLoopBackOff

    List the pods and spot those whose STATUS is CrashLoopBackOff or whose RESTARTS counter is growing:

    Terminal window
    kubectl get pods -A | grep -E 'CrashLoop|Error|BackOff'

    Note the pod name, its namespace and the number of RESTARTS.

  2. Read the logs of the crashed container

    The command to run first is kubectl logs --previous, which prints the logs of the previous run (the one that crashed):

    Terminal window
    kubectl logs <pod> -n <namespace> --previous

    If the pod contains several containers, specify the container:

    Terminal window
    kubectl logs <pod> -n <namespace> -c <container> --previous

    Look for clues: Exception, Error, panic, Traceback, connection refused, OOM, permission denied.

  3. Inspect the description of the pod

    Terminal window
    kubectl describe pod <pod> -n <namespace>

    The sections to look at first:

    • Last State: the reason for the previous stop (OOMKilled, Error, Completed)
    • Exit Code: 0 = normal stop, 1 = application error, 137 = OOMKilled or SIGKILL, 139 = segfault, 128 = the container did not even start
    • Events: Unhealthy (failed probe), BackOff, Pulling
    • Containers → Args/Command: is the start command correct?
    • Containers → Env: are the environment variables present?
    • Containers → Mounts: are the volumes mounted correctly?
  4. Fix and check

    Depending on the cause identified:

    CauseCorrective action
    Application errorFix the code or the config, redeploy
    Missing configAdd the ConfigMap/Secret, check the names
    OOMKilledIncrease resources.limits.memory
    Misconfigured probeIncrease initialDelaySeconds, timeoutSeconds, or add a startup probe
    Invalid command/entrypointCheck command: and args: in the manifest
    External dependencyCheck connectivity, add an init container

    After the fix, check that the pod is stable:

    Terminal window
    kubectl get pods -n <namespace> -w

    The -w flag follows the evolution live. The Pod must go to Running, and the RESTARTS counter must stop growing: this second condition proves the fix.

Practical example: reproduce a CrashLoopBackOff

To understand the mechanism well, deliberately create two Pods that crash for two different causes.

Case 1: application error (exit code 1)

Terminal window
kubectl apply -f- <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: crash-exit-error
namespace: default
spec:
restartPolicy: Always
containers:
- name: app
image: busybox:1.37@sha256:bdf57e528e45e4433820e045b29b4597825a1c9e38353532d90a01445013f82e
command: ["sh", "-c", "echo 'Application démarrée'; sleep 2; echo 'Erreur fatale!'; exit 1"]
EOF

The two echo messages are in French: Application démarrée means "application started" and Erreur fatale! means "fatal error!".

Wait 30 seconds then observe:

Terminal window
kubectl get pods crash-exit-error -w

You will see the STATUS go from Running to Error then to CrashLoopBackOff, with the RESTARTS counter growing.

Diagnosis:

Terminal window
# Logs of the run that crashed
kubectl logs crash-exit-error --previous
# Description with exit code
kubectl describe pod crash-exit-error | grep -A5 "Last State"

Case 2: OOMKilled (memory limit exceeded)

To trigger a real OOMKill, the container must allocate memory beyond its limit. A simple dd to /dev/null consumes CPU, not RAM. Here is an example that really allocates memory with one line of Python:

Terminal window
kubectl apply -f- <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: crash-oom
namespace: default
spec:
restartPolicy: Always
containers:
- name: app
image: python:3.13-slim@sha256:6771159cd4fa5d9bba1258caf0b82e6b73458c694d178ad97c5e925c2d0e1a91
command: ["python3", "-c", "data = []; [data.append('x' * 10**6) for _ in range(500)]"]
resources:
limits:
memory: "50Mi"
EOF

The Python script allocates blocks of 1 MB in a loop until it exceeds the 50 Mi limit, which triggers an OOMKill.

Diagnosis:

Terminal window
kubectl describe pod crash-oom | grep -A5 "Last State"

You will see Reason: OOMKilled and Exit Code: 137.

Cleanup

A simple kubectl delete pod is enough, but it gives each Pod its graceful shutdown delay, 30 seconds by default. The two options below delete the Pods without this delay, and there is nothing dangerous about them here, because these Pods hold no data.

Terminal window
kubectl delete pod crash-exit-error crash-oom --force --grace-period=0

The essential exit codes

Exit codes help you quickly identify the nature of the crash:

Exit codeMeaningProbable cause
0Normal stopThe container ended on its own, with restartPolicy: Always it will be restarted
1Generic errorApplication bug, unhandled exception
2Misuse of a shell commandInvalid command or argument in command:
126Permission deniedA shell refused to run a non-executable file
127Command not foundA shell did not find the requested binary
128StartErrorThe container never started: the runtime failed, not a program
137SIGKILL (128 + 9)Process killed by SIGKILL, often an OOMKill, but not exclusively
139SIGSEGV (128 + 11)Segmentation fault in the application
143SIGTERM (128 + 15)Graceful stop requested by Kubernetes

Special cases

Three situations produce a CrashLoopBackOff without the four-step method being enough, because the information you expect is not where you look for it: the logs are empty, the container works fine anyway, or the culprit is not the application container. Recognizing them saves you from rereading ten times an output that will never say anything.

The following diagram sums up the path. It starts from the logs of the previous run: as long as they are empty or silent, you go down one question at a time, and each "yes" tells you in which field to read the cause.

Decision tree for a CrashLoopBackOff. If kubectl logs --previous shows a message, read the error. Otherwise, an exit code 128 (StartError) points to the terminated.message field, an exit code 137 to the OOMKilled reason of kubectl describe pod, Unhealthy then Killing events to the liveness probe, and anything else to a rereading of kubectl describe pod (command, variables, volumes).Does kubectl logs --previousshow a message?Read the error:exception, not found,connection refusedyesExit code 128(StartError)?noRead the fieldterminated.messageyesExit code 137?noLook for Reason: OOMKilledin kubectl describe podyesEvents Unhealthythen Killing?noReview the liveness probeyesReread kubectl describe pod:Command, Env, Mountsno
Source · SVG
Esc to close, + and − to zoom, scroll to move around. 100 %

CrashLoopBackOff with empty logs

If kubectl logs --previous returns nothing, either the container crashed before producing anything, or it never started. The distinction is read in the exit code, and it completely changes the place to look.

Codes 126 and 127 come from a shell: they assume that a shell did start, then did not find or could not run what you asked it. The message is then in the logs.

Terminal window
kubectl logs mon-pod --previous
Output
sh: binaire-absent: not found

In this example, mon-pod stands for "my pod", the name of your own Pod, and binaire-absent for "missing binary", the command the shell could not find.

When the manifest calls the binary directly, without a shell, there is nobody to produce this message: the runtime fails to create the container, and Kubernetes returns a StartError with exit code 128. The logs are then permanently empty, and the cause lives in a field that few people think of reading:

Terminal window
kubectl get pod mon-pod \
-o jsonpath='{.status.containerStatuses[0].state.terminated.message}'
Output
failed to create containerd task: failed to create shim task:
OCI runtime create failed: runc create failed: unable to start container process

Remember the rule: 126 or 127, look at the logs; 128, look at the message. A 137 right at startup suggests an OOMKill, which kubectl describe pod confirms or not.

CrashLoopBackOff caused by a liveness probe

If the events show Unhealthy followed by Killing, the container works but the liveness probe considers it dead:

Warning Unhealthy 10s kubelet Liveness probe failed: HTTP probe failed with statuscode: 503
Normal Killing 10s kubelet Container app failed liveness probe, will be restarted

The solution is to review the probe configuration:

  • Increase initialDelaySeconds if the application takes time to start
  • Increase timeoutSeconds if the endpoint is slow
  • Use a startup probe for long startups

See the guide on Kubernetes probes for configuration best practices.

CrashLoopBackOff on an init container

An init container runs once, before the main container. If it fails, the Pod never reaches Running, and the STATUS column shows Init:CrashLoopBackOff: the Init: prefix puts you on the track.

Same diagnosis, specifying the container:

Terminal window
kubectl logs <pod> -c <init-container> --previous

Quick troubleshooting

This table is read from the left column, the one of the observed symptom. Each line gives the first action to take, the one that rules out or confirms a family of causes before going further. If none matches, go back to the four-step method rather than trying commands at random.

SymptomFirst actionCommand
CrashLoopBackOff with high RESTARTSRead the previous logskubectl logs <pod> --previous
Exit code 137, no logsCheck the memorykubectl describe pod <pod> → look for OOMKilled
Exit code 127Check the image and the commandkubectl describe pod <pod> → Containers section
Unhealthy + Killing eventsThe probe kills the containerReview livenessProbe in the manifest
Init:CrashLoopBackOffThe init container crasheskubectl logs <pod> -c <init-container> --previous
Logs show "connection refused"External dependency absentCheck the target Service/Endpoint

Put it into practice

Two guided labs replay the same symptom. The first puts back into service a Deployment whose Pods restart in a loop since the last delivery: you read what the process said before dying, then you fix the Deployment itself. The validation reads the state of the Deployment and of its Pods: two replicas available, no more restart loop, the configuration variable at the right value, and an application that answers.

The second lines up three Pods and three distinct causes: a command that does not exist, a missing environment variable, a memory limit that is too low. You recreate each fixed Pod, and the validation reads the state of each one, its memory limit, and checks that the restart counter no longer grows.

Key takeaways

  • CrashLoopBackOff is not a Pod phase, it is a waiting state between two restarts, with a delay that doubles at each crash (first restart immediate, then 10 s, 20 s, 40 s... up to 5 min max)
  • kubectl logs --previous is your main command: it prints the logs of the run that crashed
  • kubectl describe pod reveals the exit code, the events and the configuration, essential when the logs are empty
  • Exit code 137 = process killed by SIGKILL, often an OOMKill (but not exclusively), check with kubectl describe pod then increase resources.limits.memory if it is confirmed
  • Exit code 127 = a shell did not find the binary, the message is in the logs; StartError and 128 = the container never started, the logs are empty and the cause is in terminated.message
  • If the events show Unhealthy → Killing, it is the liveness probe that kills the container, not an application crash
  • After the fix, watch the pod with kubectl get pods -w and check that RESTARTS do not start again

Knowledge check

Seven questions to check three reflexes: telling a phase from a waiting state, knowing which field to read the cause in when the logs are empty, and recognizing the signature of an overly aggressive liveness probe.

Knowledge check

Check what you have learned with this interactive quiz

7 questions
5 min
80% to pass

How it works

  • The timer starts when you click Start
  • Multiple choice, true/false and short answer questions
  • You can move between questions
  • Detailed results are shown at the end

Starts the quiz and the timer