Skip to content
Français
Account: not signed in, sync available My learning
Select theme
Français

Manage container images in Kubernetes

By updated

Improve this page

Logo Kubernetes

A Deployment that references an image by its tag does not say which image is running: the tag can be republished with other content. In this lesson you read in the cluster the digest of the image that is running, you pin it in the manifest, then you trigger an ImagePullBackOff with an image stored in a private registry and you fix it with a Secret of type docker-registry. You finish with an admission policy that rejects the Pods whose image is not pinned by digest. A password-protected registry is deployed in the demo cluster for the occasion.

What you will learn

  • Read an image reference (registry, repository, tag, digest)
  • Predict the imagePullPolicy applied by default depending on whether the image carries a tag, the latest tag or a digest
  • Replace a tag with a digest in a manifest, and get this digest from the cluster
  • Give a Pod access to a private registry with a docker-registry Secret, referenced by the Pod or by its ServiceAccount
  • Diagnose an ImagePullBackOff or an ErrImagePull from kubectl describe
  • Recognize the rejection of a Pod by an admission policy that requires the digest

Prerequisites

You have followed the Kustomize lesson (the lesson that comes before this one in this module) and you know how to chain apply, rollout status and describe. You know Pods and Deployments, the demo cluster is running on the kind-demo context, and Docker is installed on the machine that hosts the cluster: only one part of the lesson uses it, to push an image into the registry.

Which three mechanisms do you need to know?

Three distinct questions arise when Kubernetes starts a container, and each has its own mechanism. They are independent: you can use a digest with any pull policy, and the registry Secret only acts on authentication.

QuestionMechanismExample
Which exact image to run?Tag or digestnginx:1.30.5 or nginx@sha256:...
When to contact the registry?imagePullPolicyAlways, IfNotPresent, Never
How to authenticate?imagePullSecretsSecret of type docker-registry

An image reference reads like this: [registry/][namespace/]name[:tag][@digest]. The registry is the server that hosts the image, and it is Docker Hub (docker.io) when absent. The namespace is the organization, library for official images. The tag is a version name that its author can move, latest when absent. The digest is the sha256 hash of the content: it designates a precise version. So nginx means docker.io/library/nginx:latest. The lab registry, localhost:30500, plays the role of docker.io for the private images of the lesson.

How do you know which image is really running?

Deploy an Nginx site with a tag alone. The tag alone is the counter-example of this step, hence the # ❌ marker in the file.

Terminal window
kubectl create namespace images
mkdir -p ~/atelier-k8s/images
cd ~/atelier-k8s/images
cat > site.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: site
labels:
app: site
spec:
replicas: 1
selector:
matchLabels:
app: site
template:
metadata:
labels:
app: site
spec:
containers:
- name: nginx
image: nginx:1.30.5 # ❌ tag only, counter-example: which image is really running?
EOF
Output
namespace/images created
Terminal window
kubectl apply -f site.yaml -n images && kubectl rollout status deployment/site -n images --timeout=90s
Output
deployment.apps/site created
Waiting for deployment "site" rollout to finish: 0 of 1 updated replicas are available...
deployment "site" successfully rolled out

Read again what the manifest declares, then what the cluster actually started.

Terminal window
kubectl get deployment site -n images -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
Output
nginx:1.30.5

The Deployment says nginx:1.30.5 and nothing else: if the author of the image republishes this tag tomorrow with different content, a new Pod would start another content under the same name. The real hash is read in the state of the container, in status.containerStatuses[].imageID.

Terminal window
kubectl get pods -n images -l app=site -o jsonpath='{.items[0].status.containerStatuses[0].imageID}{"\n"}'
Output
docker.io/library/nginx@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb

Read the part after @: it is the digest, here 9bf97bd7.... It is the digest of the index of the nginx:1.30.5 image: a multi-architecture image is published under an index that lists one image per architecture, and it is the digest of this index that the cluster reports when the image was pulled by tag.

Which pull policy applies by default?

imagePullPolicy decides whether the kubelet contacts the registry before starting a container: Always at every start, IfNotPresent only if the image is not already on the node, Never never (the container fails if the image is absent). When you do not write it, the API fills it in according to the shape of the reference. Three create deployment --dry-run=server commands return the chosen value without creating anything.

Terminal window
kubectl create deployment p-tag --image=nginx:1.30.5 --dry-run=server -o jsonpath='{.spec.template.spec.containers[0].imagePullPolicy}{"\n"}'
kubectl create deployment p-latest --image=nginx:latest --dry-run=server -o jsonpath='{.spec.template.spec.containers[0].imagePullPolicy}{"\n"}' # ❌ counter-example
kubectl create deployment p-digest --image=nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb --dry-run=server -o jsonpath='{.spec.template.spec.containers[0].imagePullPolicy}{"\n"}'
Output
IfNotPresent
Always
IfNotPresent

Read the three lines in the order of the commands: a precise tag gives IfNotPresent, the latest tag gives Always, a digest gives IfNotPresent. A precise tag, already present on the node, is therefore not downloaded again: if its content changes in the registry, the node keeps the old one. The latest tag, for its part, is rechecked against the registry at every start, but nothing then says which version you get. Only the digest describes a precise image, and the same image on all nodes.

How do you pin the image by its digest?

Take the digest read in the cluster and write it in the Deployment. The first command extracts the part after @ from the imageID; the second appends it to the current tag with kubectl set image.

Terminal window
DIGEST=$(kubectl get pods -n images -l app=site -o jsonpath='{.items[0].status.containerStatuses[0].imageID}')
kubectl set image deployment/site nginx="nginx:1.30.5@${DIGEST#*@}" -n images
Output
deployment.apps/site image updated

The ${DIGEST#*@} syntax removes from the variable everything that precedes the first @, so sha256:... remains. Wait for the end of the rollout and read the reference again.

Terminal window
kubectl rollout status deployment/site -n images --timeout=90s && kubectl get deployment site -n images -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
Output
Waiting for deployment "site" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "site" rollout to finish: 1 old replicas are pending termination...
deployment "site" successfully rolled out
nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb

The last line shows the tag and the digest together: the manifest now says precisely what is running. The digest designates the image, and the tag stays there so that humans read the version. A digest receives no security fix by itself: when the publisher fixes the image, you read the new digest and you deploy it.

Why does a private registry trigger an ImagePullBackOff?

An image stored in a registry that requires a password cannot be pulled without credentials. To see it, the cluster hosts its own protected registry. The following files use fields (env, volumes, mounted Secret) taught in other lessons: copy them as they are, they only serve to install the registry.

Terminal window
cat > registre.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: registre
labels:
app: registre
spec:
replicas: 1
selector:
matchLabels:
app: registre
template:
metadata:
labels:
app: registre
spec:
containers:
- name: registre
image: registry:3.1.2@sha256:38907510915c3f1604c7e83d6ab602ab3020d092f73bdaed8696773693ff768c
ports:
- containerPort: 5000
env:
- name: REGISTRY_AUTH
value: htpasswd
- name: REGISTRY_AUTH_HTPASSWD_REALM
value: formation
- name: REGISTRY_AUTH_HTPASSWD_PATH
value: /auth/htpasswd
volumeMounts:
- name: auth
mountPath: /auth
readOnly: true
volumes:
- name: auth
secret:
secretName: registre-auth
---
apiVersion: v1
kind: Service
metadata:
name: registre
spec:
type: NodePort
selector:
app: registre
ports:
- port: 5000
targetPort: 5000
nodePort: 30500
EOF

The registry reads its accounts from an htpasswd file. You build it with the htpasswd tool of an httpd image, then you store it in a Secret. The motdepasse-de-demo password ("demo password") is acceptable only for this disposable demonstration.

Terminal window
docker run --rm --entrypoint htpasswd httpd:2.4-alpine@sha256:610eddacdacf681838190f5e2fda06dba783384c71610b247a765e8f8faadfc6 -Bbn formation motdepasse-de-demo > htpasswd

The command prints nothing, because its output goes into the htpasswd file. Create the Secret, then the registry.

Terminal window
kubectl create secret generic registre-auth --from-file=htpasswd=htpasswd -n images
Output
secret/registre-auth created
Terminal window
kubectl apply -f registre.yaml -n images && kubectl rollout status deployment/registre -n images --timeout=90s
Output
deployment.apps/registre created
service/registre created
Waiting for deployment "registre" rollout to finish: 0 of 1 updated replicas are available...
deployment "registre" successfully rolled out

The registry answers. It is reachable from the node on localhost:30500 (the port of the NodePort Service), without any particular containerd setting.

To push an image into it from your machine, open a tunnel with kubectl port-forward to localhost:5000, log in, then push. Choose an image that the node has never pulled: if it was already there, the node would reuse it without going through the registry and you would not see the effect of authentication. The httpd image fits: Docker pulled it for htpasswd, but the kind node has its own containerd and does not have it.

Terminal window
kubectl port-forward -n images service/registre 5000:5000 > /dev/null 2>&1 &
PID=$!
sleep 3
docker login localhost:5000 -u formation -p motdepasse-de-demo
docker pull httpd:2.4-alpine@sha256:610eddacdacf681838190f5e2fda06dba783384c71610b247a765e8f8faadfc6
docker tag httpd:2.4-alpine@sha256:610eddacdacf681838190f5e2fda06dba783384c71610b247a765e8f8faadfc6 localhost:5000/site:1.0
docker push localhost:5000/site:1.0
kill $PID
Output
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'.
Configure a credential helper to remove this warning. See
https://docs.docker.com/go/credential-store/
Login Succeeded
docker.io/library/httpd@sha256:610eddacdacf681838190f5e2fda06dba783384c71610b247a765e8f8faadfc6: Pulling from library/httpd
Digest: sha256:610eddacdacf681838190f5e2fda06dba783384c71610b247a765e8f8faadfc6
Status: Image is up to date for httpd@sha256:610eddacdacf681838190f5e2fda06dba783384c71610b247a765e8f8faadfc6
docker.io/library/httpd:2.4-alpine@sha256:610eddacdacf681838190f5e2fda06dba783384c71610b247a765e8f8faadfc6
The push refers to repository [localhost:5000/site]
4f4fb700ef54: Unavailable
881fc932939e: Pushed
8590807ad028: Pushed
7d943e1ccc06: Pushed
cd9c4c98d974: Unavailable
d5f5f45757f1: Unavailable
4f4fb700ef54: Pushed
d5f5f45757f1: Pushed
8589b3d00f55: Pushed
cd9c4c98d974: Pushed
1.0: digest: sha256:ab148432aee464788f3b2fb739a3658a3060732606ee86272becefb63a8d934a size: 2287
Info -> Not all multiplatform-content is present and only the available single-platform image was pushed
sha256:610eddacdacf681838190f5e2fda06dba783384c71610b247a765e8f8faadfc6 -> sha256:ab148432aee464788f3b2fb739a3658a3060732606ee86272becefb63a8d934a

Look at the 1.0: digest: sha256:ab148432... line: it is the digest of the pushed manifest. Note it down. Just below, Docker announces Not all multiplatform-content is present and only the available single-platform image was pushed: the image pulled by tag@digest is a multi-architecture index, Docker sent only the platform of your machine, and the sha256:610eddac... -> sha256:ab148432... arrow shows that the pushed digest is not the one of the original index.

The two warnings of docker login point out that the password passed as an option and the stored credentials are in clear text: that is acceptable here, not elsewhere.

Now deploy an application that pulls this private image, without giving it any credentials. The 1.0 tag is here a temporary counter-example, until you know the digest.

Terminal window
cat > prive.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: prive
labels:
app: prive
spec:
replicas: 1
selector:
matchLabels:
app: prive
template:
metadata:
labels:
app: prive
spec:
containers:
- name: site
image: localhost:30500/site:1.0 # ❌ tag only, until it is pushed
EOF
kubectl apply -f prive.yaml -n images && sleep 30 && kubectl get pods -n images -l app=prive
Output
deployment.apps/prive created
NAME READY STATUS RESTARTS AGE
prive-8d748896c-f65km 0/1 ErrImagePull 0 30s

The Pod does not start: ErrImagePull means the download has just failed, and ImagePullBackOff appears afterwards, when the kubelet spaces out its new attempts. To know why, read the events of the Pod.

Terminal window
kubectl describe pod -n images -l app=prive | grep -A12 Events
Output
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 30s default-scheduler Successfully assigned images/prive-8d748896c-f65km to demo-control-plane
Normal Pulling 16s (x2 over 29s) kubelet spec.containers{site}: Pulling image "localhost:30500/site:1.0"
Warning Failed 16s (x2 over 29s) kubelet spec.containers{site}: Failed to pull image "localhost:30500/site:1.0": failed to pull and unpack image "localhost:30500/site:1.0": failed to resolve reference "localhost:30500/site:1.0": pull access denied, repository does not exist or may require authorization: authorization failed: no basic auth credentials
Warning Failed 16s (x2 over 29s) kubelet spec.containers{site}: Error: ErrImagePull
Normal BackOff 2s (x2 over 29s) kubelet spec.containers{site}: Back-off pulling image "localhost:30500/site:1.0"
Warning Failed 2s (x2 over 29s) kubelet spec.containers{site}: Error: ImagePullBackOff

Look at the Failed to pull image line, and in particular its end: pull access denied, repository does not exist or may require authorization ... no basic auth credentials. The registry did not say "image not found": it denied access for lack of credentials. This is the difference to remember between this diagnosis and the one of a tag that does not exist. The complete diagnosis of an ImagePullBackOff, with its other causes, is detailed in ImagePullBackOff: diagnose and fix (in French).

How do you give the credentials to the Pod?

Kubernetes stores the credentials of a registry in a Secret of type docker-registry. Create it with the address of the registry as the node contacts it, localhost:30500.

Terminal window
kubectl create secret docker-registry regcred -n images --docker-server=localhost:30500 --docker-username=formation --docker-password=motdepasse-de-demo
Output
secret/regcred created
Terminal window
kubectl get secret regcred -n images -o jsonpath='{.type}{"\n"}'
Output
kubernetes.io/dockerconfigjson

The kubernetes.io/dockerconfigjson type is the one the kubelet knows how to read to authenticate. It remains to reference it.

A ServiceAccount is the identity of a Pod in its namespace. Without any indication, the Pod uses the ServiceAccount named default, which exists in every namespace, and you pick another one with the serviceAccountName field of the Pod template. To go further on these accounts, see ServiceAccounts.

You can therefore reference the Secret Pod by Pod, with imagePullSecrets in the Pod template, or only once in a ServiceAccount: all the Pods created with this account, and which do not declare their own imagePullSecrets, inherit the Secret. The second method avoids forgetting a Deployment. The Secret is copied into the Pod when it is created: a Pod that is already running does not receive the imagePullSecrets added to the account afterwards, you have to replace it.

Terminal window
cat > compte-prive.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
name: lecteur-registre
imagePullSecrets:
- name: regcred
EOF
kubectl apply -f compte-prive.yaml -n images && kubectl set serviceaccount deployment/prive lecteur-registre -n images
Output
serviceaccount/lecteur-registre created
deployment.apps/prive serviceaccount updated

The prive Deployment now uses the lecteur-registre account, which changes its Pod template and starts a new rollout. Check that the new Pod starts.

Terminal window
kubectl rollout status deployment/prive -n images --timeout=120s && kubectl get pods -n images -l app=prive
Output
Waiting for deployment "prive" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "prive" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "prive" rollout to finish: 1 old replicas are pending termination...
deployment "prive" successfully rolled out
NAME READY STATUS RESTARTS AGE
prive-85dc774b7f-j59hj 1/1 Running 0 1s
prive-8d748896c-f65km 0/1 Terminating 0 32s

The new Pod is 1/1 Running, the old one (the one that was failing) is Terminating: the Secret made the difference. The events of the new Pod show the authenticated pull. Replace the name of the Pod with the one kubectl get pods has just displayed.

Terminal window
kubectl describe pod prive-85dc774b7f-j59hj -n images | grep -A10 Events
Output
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2s default-scheduler Successfully assigned images/prive-85dc774b7f-j59hj to demo-control-plane
Normal Pulling 2s kubelet spec.containers{site}: Pulling image "localhost:30500/site:1.0"
Normal Pulled 1s kubelet spec.containers{site}: Successfully pulled image "localhost:30500/site:1.0" in 863ms (863ms including waiting). Image size: 20796328 bytes.
Normal Created 1s kubelet spec.containers{site}: Container created
Normal Started 1s kubelet spec.containers{site}: Container started

The Successfully pulled image line replaces the Failed to pull image of earlier: with the credentials, the registry delivered the image. Now read the digest of the private image actually pulled. The --field-selector=status.phase=Running filter targets the new Pod, because the old one is still listed while it terminates.

Terminal window
kubectl get pods -n images -l app=prive --field-selector=status.phase=Running -o jsonpath='{.items[0].status.containerStatuses[0].imageID}{"\n"}'
Output
localhost:30500/site@sha256:ab148432aee464788f3b2fb739a3658a3060732606ee86272becefb63a8d934a

Compare this digest with the one on the 1.0: digest: line of your docker push: they are identical, sha256:ab148432.... So imageID returns the digest of the pushed manifest, and not the one of the original index (sha256:610eddac...). This comparison only works because httpd was absent from the node before the first pull: the kubelet pulled the image from your registry.

If the node already has the same image under another name, imageID may display that first name: it then proves nothing about the registry the image was pulled from.

You can now pin this image as in the previous step: replace localhost:30500/site:1.0 with localhost:30500/site@sha256: followed by the digest you noted.

What happens with a multi-architecture image?

An image published with a multi-architecture index (amd64, arm64) works on each type of node: the container runtime of the node (containerd in kind) picks the entry that matches its architecture. If the image does not publish the architecture of the node, the container fails at startup. The index digest designates all the entries, whereas the digest of a manifest designates a single architecture: this is what you saw with docker push, which sends a single platform.

How do you enforce the digest at admission?

Nothing yet prevents a colleague from deploying a tag alone. An admission policy rejects the Pods that do not follow a rule, before they are created. The ValidatingAdmissionPolicy is built into Kubernetes. It contains an expression written in CEL, a small rules language, that you do not need to understand here: it checks that the image of each container in spec.containers contains @sha256:, without looking at the init containers. The ValidatingAdmissionPolicy (in French) lesson teaches it, and Kyverno and Gatekeeper (in French) are alternatives. The binding restricts the policy to the namespaces that carry a given label. The message of the policy is in French on purpose, so that the outputs stay those of the tested cluster: "Chaque image doit être épinglée par digest" means "Every image must be pinned by digest".

Terminal window
cat > vap-digest.yaml <<'EOF'
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: digest-obligatoire
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: [""]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["pods"]
validations:
- expression: "object.spec.containers.all(c, c.image.contains('@sha256:'))"
message: "Chaque image doit être épinglée par digest (@sha256:...)."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: digest-obligatoire
spec:
policyName: digest-obligatoire
validationActions:
- Deny
matchResources:
namespaceSelector:
matchLabels:
images: digest-obligatoire
EOF
kubectl apply -f vap-digest.yaml && kubectl label namespace images images=digest-obligatoire
Output
validatingadmissionpolicy.admissionregistration.k8s.io/digest-obligatoire created
validatingadmissionpolicybinding.admissionregistration.k8s.io/digest-obligatoire created
namespace/images labeled

The policy is not active the millisecond it is created: a Pod launched right away can still get through. Before testing, wait until it rejects. This loop repeats a dry-run attempt (--dry-run=server, which writes nothing) as long as it is accepted.

Terminal window
while kubectl -n images run essai --image=nginx:1.30.5 --dry-run=server > /dev/null 2>&1; do sleep 1; done

The command prints nothing and returns as soon as the policy rejects, about one second later. Then run the real test with a tag alone.

Terminal window
kubectl -n images run web-tag --image=nginx:1.30.5; echo "code de sortie : $?"
Output
The pods "web-tag" is invalid: : ValidatingAdmissionPolicy 'digest-obligatoire' with binding 'digest-obligatoire' denied request: Chaque image doit être épinglée par digest (@sha256:...).
code de sortie : 1

The Pod is rejected with the message written in the policy, and the exit code is 1 ("code de sortie" means "exit code"). With a digest, the same Pod gets through.

Terminal window
kubectl -n images run web-digest --image=nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb
Output
pod/web-digest created

The policy only targets Pods. A Deployment with a tag alone is therefore created, but its Pods are rejected: it stays at 0/1, and the cause is read in the events of the ReplicaSet.

Terminal window
kubectl -n images create deployment api --image=nginx:1.30.5 && sleep 5 && kubectl -n images get deployment api
Output
deployment.apps/api created
NAME READY UP-TO-DATE AVAILABLE AGE
api 0/1 0 0 5s
Terminal window
kubectl -n images get events --field-selector reason=FailedCreate
Output
LAST SEEN TYPE REASON OBJECT MESSAGE
5s Warning FailedCreate replicaset/api-59c664567c Error creating: pods "api-59c664567c-lz7nv" is forbidden: ValidatingAdmissionPolicy 'digest-obligatoire' with binding 'digest-obligatoire' denied request: Chaque image doit être épinglée par digest (@sha256:...).
5s Warning FailedCreate replicaset/api-59c664567c Error creating: pods "api-59c664567c-ccrkb" is forbidden: ValidatingAdmissionPolicy 'digest-obligatoire' with binding 'digest-obligatoire' denied request: Chaque image doit être épinglée par digest (@sha256:...).
5s Warning FailedCreate replicaset/api-59c664567c Error creating: pods "api-59c664567c-qmc5j" is forbidden: ValidatingAdmissionPolicy 'digest-obligatoire' with binding 'digest-obligatoire' denied request: Chaque image doit être épinglée par digest (@sha256:...).
3s Warning FailedCreate replicaset/api-59c664567c (combined from similar events): Error creating: pods "api-59c664567c-zhnq4" is forbidden: ValidatingAdmissionPolicy 'digest-obligatoire' with binding 'digest-obligatoire' denied request: Chaque image doit être épinglée par digest (@sha256:...).

The ReplicaSet keeps trying to create the Pod: the command returns about ten lines in a few seconds. The excerpt above keeps the first three and the last one, which Kubernetes prefixes with (combined from similar events) when it groups neighboring events.

The rejection is the same as for web-tag, with a different form: is invalid for a Pod created directly, is forbidden when a ReplicaSet creates the Pod. A Deployment at 0/1 with no Pod and no visible error in get pods is therefore diagnosed in the events.

Clean up the cluster

Delete the policy and its binding, which are objects of the whole cluster, then the namespace, which takes with it the Deployments, the Secrets and the Pods. Then go back to the machine: log Docker out of the registry, remove the image tagged for it and delete the working folder.

Terminal window
kubectl delete -f vap-digest.yaml && kubectl delete namespace images
Output
validatingadmissionpolicy.admissionregistration.k8s.io "digest-obligatoire" deleted
validatingadmissionpolicybinding.admissionregistration.k8s.io "digest-obligatoire" deleted
namespace "images" deleted
Terminal window
docker logout localhost:5000; docker rmi localhost:5000/site:1.0; rm -rf ~/atelier-k8s/images
Output
Removing login credentials for localhost:5000
Untagged: localhost:5000/site:1.0

The pulled images (nginx, httpd) stay in the Docker cache and in the node cache: this cleanup does not remove them.

Troubleshooting

The diagnosis of a pull failure always starts with the events of the Pod (kubectl describe pod), where the kubelet writes the exact cause. The table gives the common causes. The other cases are in the dedicated diagnosis page.

SymptomLikely causeSolution
ErrImagePull then ImagePullBackOff, message no basic auth credentialsPrivate registry, no credentials providedCreate a docker-registry Secret and reference it by imagePullSecrets or by the ServiceAccount
ErrImagePull, image-not-found messageWrong name or tagFix the reference in the Deployment: a new Pod replaces the old one
The Secret is created but the Pod still failsThe Pod does not reference the Secret; the Secret is in another namespaceA Secret is local to its namespace: create it in the one of the Pod, then reference it
A Deployment is at 0/1 with no PodAn admission policy rejects the Podskubectl get events --field-selector reason=FailedCreate
A republished tag changes nothing on the nodeIfNotPresent reuses the image already present under that tagUse a digest, or a precise tag that is never republished
FailedToRetrieveImagePullSecret eventThe Pod or its ServiceAccount references a Secret that is absent from the namespaceCreate the Secret in the namespace of the Pod, or fix the referenced name

Put it into practice

The lab repeats this gesture on another set of images. You read again the real digest of what is running and you pin the Deployment on this digest. You then record why a Pod stays in ImagePullBackOff, before repairing it with a docker-registry Secret carried by its ServiceAccount. The validation covers the state of the cluster: the Deployment is pinned by the digest of the image it was running, the cause of the refused pull is recorded before the repair, and the Pod pulls its private image thanks to its ServiceAccount.

Key takeaways

  • A tag is mutable, a digest designates a precise content: the imageID of the Pod gives the digest actually pulled.
  • Without a written imagePullPolicy, the API sets IfNotPresent for a precise tag and for a digest, and Always for latest.
  • To pin an image, read the digest again in status.containerStatuses[].imageID and write it in the manifest with kubectl set image.
  • A Pod that cannot authenticate to a private registry shows ErrImagePull then ImagePullBackOff, with no basic auth credentials in kubectl describe.
  • The docker-registry Secret is referenced by imagePullSecrets, in the Pod or in the ServiceAccount, and stays limited to its namespace.
  • A ValidatingAdmissionPolicy can reject the Pods whose image is not pinned; a rejected Deployment stays at 0/1 and the cause is in the events of the ReplicaSet.

Test your knowledge

This quiz checks that you can predict the pull policy, read a digest, diagnose an ImagePullBackOff and recognize the rejection of an admission policy.

Knowledge check

Check what you have learned with this interactive quiz

8 questions
5 min
80% to pass

How it works

  • The timer starts when you click Start
  • Multiple choice, true/false and short answer questions
  • You can move between questions
  • Detailed results are shown at the end

Starts the quiz and the timer