Skip to content
Français
Account: not signed in, sync available My learning
Select theme
Français

Kubernetes probes: liveness, readiness, startup without false positives

By updated

Improve this page

Logo Kubernetes

An application takes 40 seconds to start, a badly tuned probe declares it dead after 10 seconds, and Kubernetes restarts it in a loop even though it never crashes. Probes are checks that the kubelet, the Kubernetes agent on each node, runs against a container: the startup probe protects the startup, the liveness probe restarts a stuck container, the readiness probe decides whether the Pod receives traffic. In this lesson, you reproduce a liveness probe that is too short, you read its events, then you fix it with the three probes until the Pod is ready without any restart.

You need to know Deployments and Services (the Service of the demonstration is provided without explanation), and have the demo cluster created in First cluster.

What you will learn

  • Tell startup, liveness and readiness apart by what they trigger
  • Read the Unhealthy and Killing events of a failing probe
  • Fix a liveness probe that is too short with a startup probe and compute failureThreshold × periodSeconds
  • Read whether a Pod is ready in the Ready condition and in the EndpointSlice
  • Tell connection refused from an HTTP code in a probe message
  • Choose between HTTP, TCP, command and gRPC, and tune the parameters

What are Kubernetes probes?

A probe answers a question that Kubernetes cannot guess: the container is running, but is it doing its job? A running process is not necessarily ready: it may be initializing, stuck, or cut off from its database. The three probes answer three different questions.

The three types of probes

Each probe triggers its own action: it is this action, more than the check itself, that tells them apart.

ProbeQuestionIf it fails
startupProbeHas the application finished starting?The container is restarted
livenessProbeIs the container still alive, not stuck?The container is restarted
readinessProbeCan the Pod receive traffic?The Pod stays alive but no longer receives traffic

The order is as follows: as long as the startup probe has not succeeded, the other two do not run. Afterwards, liveness and readiness run in parallel for the whole life of the container.

Sequence of the probes: at Pod startup, the startup probe hands over to the liveness probe, then to the readiness probe; a startup or liveness failure restarts the container, a readiness failure marks the Pod not ready and the Service stops sending it traffic.Startup Probe OKStartup Probe failsLiveness Probe OKLiveness Probe failsReadiness Probe OKReadiness Probe failsPod startupLiveness ProbeReadiness ProbeContainer restartedPod in servicePod not ready: no traffic
Source · SVG
Esc to close, + and − to zoom, scroll to move around. 100 %

Look at the three exit arrows on failure: startup and liveness lead to a restart, readiness only leads to the removal from traffic. The Pod removed from traffic is not deleted: you can read this in the EndpointSlice, the object that lists the addresses of the Pods of a Service and says for each one whether it is ready.

Reproduce a liveness probe that is too short

The demonstration uses a simulated slow application: the nginx image whose command prints "initialisation en cours" ("initialization in progress"), waits 40 seconds, then starts nginx. During these 40 seconds, port 80 refuses connections. The lent Deployment ("slow") carries a liveness probe that checks / every 5 seconds and gives up after 2 failures, about 10 seconds of tolerance: that is too short.

You work in a dedicated namespace, deleted at the end:

Terminal window
kubectl config use-context kind-demo && kubectl create namespace sondes
Switched to context "kind-demo".
namespace/sondes created

The names and messages of this lesson are in French on purpose, so that the outputs stay those of the tested cluster: sondes means "probes", lent "slow", sans-route "no route", refus "refusal", sante-absente "health absent", "initialisation en cours" "initialization in progress", "prêt" "ready", and the RAISON column "REASON". The -corrige suffix of a file means "fixed".

Write and apply the manifest. The command waits 50 seconds before displaying the Pod:

Terminal window
mkdir -p ~/atelier-k8s/sondes && cd ~/atelier-k8s/sondes
cat > lent.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: lent
spec:
replicas: 1
selector:
matchLabels:
app: lent
template:
metadata:
labels:
app: lent
spec:
containers:
- name: app
image: nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb
command:
- /bin/sh
- -c
- |
echo "initialisation en cours"
sleep 40
echo "prêt"
exec nginx -g 'daemon off;'
ports:
- containerPort: 80
livenessProbe: # ❌ trop courte : l'application met 40 s à démarrer
httpGet:
path: /
port: 80
periodSeconds: 5
failureThreshold: 2
---
apiVersion: v1
kind: Service
metadata:
name: lent
spec:
selector:
app: lent
ports:
- port: 80
targetPort: 80
EOF
kubectl apply -f lent.yaml -n sondes && sleep 50 && kubectl get pods -n sondes -l app=lent
deployment.apps/lent created
service/lent created
NAME READY STATUS RESTARTS AGE
lent-85bdd4d975-9jtvj 1/1 Running 1 (9s ago) 50s

RESTARTS 1: the container has already restarted once in 50 seconds. Another detail: READY 1/1, even though nginx is not listening yet. Without a readinessProbe, a Pod is considered ready as soon as its container is running: this is the trap that the readiness probe fixes further on. Now look for why the container restarted, in the probe events:

Terminal window
kubectl get events -n sondes --field-selector reason=Unhealthy -o custom-columns=RAISON:.reason,MESSAGE:.message
RAISON MESSAGE
Unhealthy Liveness probe failed: Get "http://10.244.0.32:80/": dial tcp 10.244.0.32:80: connect: connection refused

The message names the probe (Liveness probe failed), the address queried and the cause: connection refused. The kubelet queries the IP address of the Pod from the node, and nothing is listening on port 80 yet. This is not an HTTP code: the application did not answer badly, it did not answer at all. The next event says what Kubernetes did about it:

Terminal window
kubectl get events -n sondes --field-selector reason=Killing -o custom-columns=RAISON:.reason,MESSAGE:.message
RAISON MESSAGE
Killing Container app failed liveness probe, will be restarted

It is indeed the probe that killed the container, not a crash. One last proof is in the logs of the previous instance:

Terminal window
kubectl logs deployment/lent -n sondes --previous
initialisation en cours

The container never wrote "prêt": it was killed in the middle of its initialization. The application is not broken, it is slow, and the liveness probe takes it for dead.

Fix it with startup, liveness and readiness

The fix has three parts. The startupProbe gives the application 150 seconds to start: the tolerance is computed by multiplying failureThreshold (the number of failures accepted) by periodSeconds (the delay between two attempts), here 30 × 5 s. The livenessProbe only takes over once the startup probe has succeeded, with a normal tolerance. The readinessProbe keeps the Pod out of traffic as long as the application does not answer.

The lent-corrige.yaml file takes the lent Deployment and its Service again, with these three probes in place of the liveness probe alone:

Terminal window
cat > lent-corrige.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: lent
spec:
replicas: 1
selector:
matchLabels:
app: lent
template:
metadata:
labels:
app: lent
spec:
containers:
- name: app
image: nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb
command:
- /bin/sh
- -c
- |
echo "initialisation en cours"
sleep 40
echo "prêt"
exec nginx -g 'daemon off;'
ports:
- containerPort: 80
startupProbe:
httpGet:
path: /
port: 80
periodSeconds: 5
failureThreshold: 30
livenessProbe:
httpGet:
path: /
port: 80
periodSeconds: 10
failureThreshold: 3
readinessProbe:
httpGet:
path: /
port: 80
periodSeconds: 5
failureThreshold: 3
---
apiVersion: v1
kind: Service
metadata:
name: lent
spec:
selector:
app: lent
ports:
- port: 80
targetPort: 80
EOF
kubectl apply -f lent-corrige.yaml -n sondes && sleep 15 && kubectl get pods -n sondes -l app=lent && kubectl get endpointslice -n sondes -l kubernetes.io/service-name=lent -o jsonpath='{.items[0].endpoints[*].conditions}{"\n"}'
deployment.apps/lent configured
service/lent unchanged
NAME READY STATUS RESTARTS AGE
lent-558f887648-tpjw4 0/1 Running 0 15s
lent-85bdd4d975-9jtvj 1/1 Running 1 (25s ago) 66s
{"ready":true,"serving":true,"terminating":false} {"ready":false,"serving":false,"terminating":false}

The Deployment replaces the Pod progressively, so two Pods coexist. The new one is Running but 0/1: the container is running, the readiness probe has not succeeded yet. The old one is still listed, 1/1.

The EndpointSlice contains both addresses: the old Pod with ready: true, the new one with ready: false. A Pod that is not ready therefore stays listed in the EndpointSlice, with ready: false: this flag is enough to remove its traffic. Wait for the end of the rollout:

Terminal window
kubectl rollout status deployment/lent -n sondes --timeout=120s && kubectl get pods -n sondes -l app=lent
Waiting for deployment "lent" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "lent" rollout to finish: 1 old replicas are pending termination...
deployment "lent" successfully rolled out
NAME READY STATUS RESTARTS AGE
lent-558f887648-tpjw4 1/1 Running 0 45s
lent-85bdd4d975-9jtvj 1/1 Terminating 2 (15s ago) 96s

The new Pod is 1/1 Running with 0 restarts: the startup probe waited for the end of the 40 seconds and the liveness probe killed nothing. The old one is Terminating. Read the EndpointSlice again:

Terminal window
kubectl get endpointslice -n sondes -l kubernetes.io/service-name=lent -o jsonpath='{.items[0].endpoints[*].conditions}{"\n"}'
{"ready":false,"serving":true,"terminating":true} {"ready":true,"serving":true,"terminating":false}

Two entries: the old Pod at the end of its life (terminating: true) and the new one with ready: true, which receives the traffic. Wait for the old Pod to disappear to be left with a single entry. The Ready condition of the Pod, the one the Service reads, can be read directly:

Terminal window
kubectl get pod -n sondes -l app=lent -o jsonpath='{.items[0].status.conditions[?(@.type=="Ready")].status}{"\n"}'
True

A readiness probe that fails without restarting anything

The readiness probe never restarts a container: it only removes the Pod from traffic. The sans-route Pod queries a path that does not exist:

Terminal window
cat > sans-route.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: sans-route
spec:
containers:
- name: nginx
image: nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb
ports:
- containerPort: 80
readinessProbe:
httpGet:
path: /sante-absente
port: 80
periodSeconds: 5
EOF
kubectl apply -f sans-route.yaml -n sondes && sleep 20 && kubectl get pod sans-route -n sondes && kubectl get events -n sondes --field-selector involvedObject.name=sans-route,reason=Unhealthy -o custom-columns=MESSAGE:.message
pod/sans-route created
NAME READY STATUS RESTARTS AGE
sans-route 0/1 Running 0 20s
MESSAGE
Readiness probe failed: Get "http://10.244.0.35:80/sante-absente": dial tcp 10.244.0.35:80: connect: connection refused
Readiness probe failed: HTTP probe failed with statuscode: 404

The Pod stays 0/1 with 0 restarts: a failing readiness probe kills nothing. The two messages read differently. The first, connection refused, dates from before nginx was listening: nothing was answering. The second, statuscode: 404, is an HTTP code: the application answers, but the path does not exist. With connection refused, look at the startup; with an HTTP code, look at the path and at the application code.

One setting remains forbidden: successThreshold, the number of consecutive successes that make the probe succeed, must be 1 for a liveness or startup probe. The liveness-refusee.yaml file describes an nginx Pod named refus, whose livenessProbe declares successThreshold: 2. Write it, then ask the server to validate it without creating anything, thanks to --dry-run=server:

Terminal window
cat > liveness-refusee.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: refus
spec:
containers:
- name: nginx
image: nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb
livenessProbe: # ❌ successThreshold vaut forcément 1 pour une liveness
httpGet:
path: /
port: 80
successThreshold: 2
EOF
kubectl apply -f liveness-refusee.yaml -n sondes --dry-run=server
The Pod "refus" is invalid: spec.containers[0].livenessProbe.successThreshold: Invalid value: 2: must be 1

The server refuses the value 2 on the liveness probe: this is a counter-example. Only the readiness probe accepts several consecutive successes. The comments of the manifests in French mean "too short: the application takes 40 s to start" and "successThreshold is necessarily 1 for a liveness probe".

Check methods

A probe chooses a method, and each method has its own success criterion:

  • httpGet calls an HTTP path: a code from 200 to 399 is a success;
  • tcpSocket opens a connection on a port: the opening is enough;
  • exec runs a command in the container: exit code 0 is a success;
  • grpc queries the standard gRPC health service of the application.

Except for exec, the kubelet sends the request from the node, to the IP address of the Pod: it does not run curl in the container. The table compares the four methods.

MethodSuitsLimit
httpGetA web application with a health pathOnly checks what the path tests
tcpSocketA service without HTTP (database, cache)An open port does not prove that the service is ready
execTesting a file or an internal commandEach probe starts a process in the container
grpcAn application that exposes the gRPC health protocolRequires the application to implement it

Configuration parameters

Five parameters set the pace of a probe:

  • initialDelaySeconds (default 0): the wait before the first attempt;
  • periodSeconds (default 10): the delay between two attempts;
  • timeoutSeconds (default 1): the time allowed for an answer before counting a failure;
  • successThreshold (default 1): the number of consecutive successes that make the probe succeed;
  • failureThreshold (default 3): the number of consecutive failures that make it fail.

The tolerance before a restart is failureThreshold multiplied by periodSeconds: with the default values, 30 seconds. This computation was missing from the liveness probe of the demonstration: 2 × 5 = 10 seconds for an application that needs 40.

When should you avoid a liveness probe?

A liveness probe restarts: it must only check what a restart repairs, that is a stuck container. It must not test an external dependency such as the database: if the database goes down, restarting the application does not bring it back, and all the Pods restart at the same time. This dependency check belongs to the readiness probe, which removes from traffic without restarting.

The mistakes that cause outages

Three settings are enough to cause an outage. A liveness probe whose tolerance is shorter than the startup restarts the container in a loop, as in the demonstration. A liveness probe that tests an external dependency causes cascading restarts. A startup probe whose failureThreshold is too low restarts the container before the end of the startup. For an application that always starts quickly, the startup probe is useless: a liveness and a readiness probe are enough.

Clean up the cluster

Deleting the namespace removes the Deployments, the Service and the Pods of the lesson:

Terminal window
kubectl delete namespace sondes
namespace "sondes" deleted

Put it into practice

In the lab, you set the three probes on a web server Pod: a startup probe that tolerates a slow startup of about one minute, then a liveness probe and a readiness probe that both query port 80 over HTTP. Compute the tolerance of the startup probe with the formula of the lesson. The validation reads the Pod: requested port and image, setting of each of the three probes, then Pod Ready with zero restarts.

Key takeaways

  1. The startup probe protects the startup, the liveness probe restarts a stuck container, the readiness probe decides about traffic.
  2. As long as the startup probe has not succeeded, liveness and readiness do not run.
  3. The tolerance of a probe is failureThreshold × periodSeconds: a liveness probe shorter than the startup causes restart loops.
  4. Without a readinessProbe, a Pod is ready as soon as its container runs, even if the application is not listening yet.
  5. A Pod that is not ready stays listed in the EndpointSlice with ready: false: the flag removes its traffic.
  6. The Unhealthy and Killing events name the faulty probe; connection refused says that nothing is listening, an HTTP code says that the application answers badly.
  7. The kubelet probes the IP address of the Pod from the node: it does not run curl in the container, except for an exec probe.
  8. successThreshold is 1 for a liveness or startup probe: the server refuses 2.

Test your knowledge

This quiz covers the action that distinguishes each probe, the reading of events and the EndpointSlice. Allow five minutes and aim for 80%; below that, reread "Reproduce a liveness probe that is too short".

Knowledge check

Check what you have learned with this interactive quiz

7 questions
5 min
80% to pass

How it works

  • The timer starts when you click Start
  • Multiple choice, true/false and short answer questions
  • You can move between questions
  • Detailed results are shown at the end

Starts the quiz and the timer