Skip to content
Français
Account: not signed in, sync available My learning
Select theme
Français
Lire en français

Signing and verifying an image with Cosign on Kubernetes

By updated

Improve this page

Signing an image with Cosign means placing, in the registry, a cryptographic proof that binds a digest to a private key; verifying means checking that proof with the matching public key. This lesson is aimed at anyone preparing the CKS (Certified Kubernetes Security Specialist) or the KCSA (Kubernetes and Cloud Native Security Associate) who already knows how to pin an image by its digest. It shows, with cosign 3.1.3, how to generate the keys, sign, publish the public key in the cluster and prove that verification can also refuse.

Keyless mode, the Sigstore ecosystem and use cases outside Kubernetes are covered in Cosign: signing and verifying your container images and in Sigstore: keyless signing for the supply chain. This page is limited to key-based signing and its use in a cluster.

What you will learn

  • Say what an image signature proves and what it does not prove
  • Generate a key pair and keep the private key outside the cluster
  • Sign a digest and find the signature in the registry
  • Publish the public key in a ConfigMap
  • Verify an image and read the three exit codes
  • Say how admission relies on the signature, and diagnose a failure

Prerequisites

  • Know how to pin an image by its digest: Kubernetes supply chain security
  • A Kubernetes cluster with kubectl configured
  • A registry you can push an image to: a local registry is enough, started with Docker on your workstation
  • The crane tool (Crane), which reads an image's digest in a registry: its verified installation is given below, along with an alternative without crane
  • Internet access from the signing machine: cosign's default signing contacts a public Sigstore service (see the signing section)

What does an image signature prove?

A signature proves two things: authenticity (the image was indeed published by the holder of the private key) and integrity (its content has not changed since). It does not prove the absence of vulnerabilities: a signed image can contain critical CVEs, and it is the scan, described in Scanning Kubernetes images, that looks for them.

A signature is only worth the key that verifies it. If anyone can obtain the private key, the signature no longer distinguishes your image from a third party's. Hence the rule of this lesson: the private key stays on the signing machine and never enters a cluster object, whereas the public key is meant to be published.

Cosign, from the Sigstore project, is the usual signing tool in the container ecosystem. The commands on this page were played with cosign 3.1.3 and crane 0.22.1, a small command-line tool for querying registries.

How do you sign an image with Cosign?

Signing takes three gestures: install cosign, generate a key pair, then sign the image's digest. The signature is pushed to the registry, next to the image: you therefore need a registry you are allowed to write to. A throwaway local registry is enough, and the commands can be replayed on your workstation without an account at any provider. One exception: with cosign 3.1.3, the default signing also publishes an entry in Sigstore's public Rekor log, which requires network access (details in the section "Signing the digest, not the tag").

Terminal window
# A throwaway registry on the workstation
docker run -d --name registre -p 127.0.0.1:5555:5000 \
registry:3@sha256:ddf754342cfc8acc51a56d5d0ab6af06826461864460636d8bd5c546dab2a7b8

You then need two images in this registry: lab-signe:1.0.0, which you will sign, and lab-non-signe:1.0.0, which will never be signed and will be used to trigger exit code 10. Both come from public images pinned by digest, copied under another name into the local registry.

Terminal window
docker pull gcr.io/distroless/static-debian12@sha256:afa5c872c891853ca7fcf1f12c3edb23f7eeef36189728842dd51042ff57f7ab
docker tag gcr.io/distroless/static-debian12@sha256:afa5c872c891853ca7fcf1f12c3edb23f7eeef36189728842dd51042ff57f7ab localhost:5555/lab-signe:1.0.0
docker push localhost:5555/lab-signe:1.0.0
docker pull nginx@sha256:8541484afbc9c8a5a8a99b379568ebbc957f658583ec9448fc43104229c03cf8
docker tag nginx@sha256:8541484afbc9c8a5a8a99b379568ebbc957f658583ec9448fc43104229c03cf8 localhost:5555/lab-non-signe:1.0.0
docker push localhost:5555/lab-non-signe:1.0.0

Each docker push ends with a line 1.0.0: digest: sha256:… size: …: this digest is the registry's, and it is the one you will sign. Note the one for each image.

Installing cosign with checksum verification

A supply chain tool is downloaded with the same precautions as kubectl in the lesson Kubernetes supply chain security: pinned version and verified checksum. The block compares the binary's SHA-256 checksum with the one Sigstore publishes in the cosign_checksums.txt file of each release, before making it executable.

Terminal window
COSIGN_VERSION=3.1.3
base="https://github.com/sigstore/cosign/releases/download/v${COSIGN_VERSION}"
curl -sSfLO "${base}/cosign-linux-amd64"
curl -sSfLO "${base}/cosign_checksums.txt"
grep " cosign-linux-amd64$" cosign_checksums.txt | sha256sum -c -
chmod +x cosign-linux-amd64 && sudo mv cosign-linux-amd64 /usr/local/bin/cosign

The sha256sum -c - line must answer cosign-linux-amd64: OK. Without that OK, do not make the binary executable.

Installing crane with checksum verification, or doing without it

crane is not installed with the cluster or with Docker, and it is not taught elsewhere in this course: you install it with the same discipline as cosign. The go-containerregistry project publishes, for each release, an archive and a checksums.txt file. The block pins version v0.22.1 and only places the binary in the PATH if the checksum matches. The archive name is the one from the Crane page; other architectures are listed on the project's releases page.

Terminal window
VERSION=v0.22.1
ARCHIVE=go-containerregistry_Linux_x86_64.tar.gz
BASE="https://github.com/google/go-containerregistry/releases/download/${VERSION}"
curl -fsSL -o "$ARCHIVE" "${BASE}/${ARCHIVE}"
curl -fsSL -o checksums.txt "${BASE}/checksums.txt"
grep " ${ARCHIVE}$" checksums.txt | sha256sum --check -
tar -xzf "$ARCHIVE" crane
sudo install -m 0755 crane /usr/local/bin/crane
rm -f crane "$ARCHIVE" checksums.txt

The sha256sum --check - line must answer go-containerregistry_Linux_x86_64.tar.gz: OK. Without that OK, extract nothing. You can also do without crane: the digest that docker push prints at the end of each upload (previous step) is the one to sign; in that case, replace each crane digest call in the following sections with the digest you noted.

Generating the key pair

cosign generate-key-pair writes two files to the current directory: cosign.key (private) and cosign.pub (public). The tool asks for a passphrase, which protects cosign.key. To supply it without a dialogue, for example in a script, the COSIGN_PASSWORD variable does the job, even empty in a lab.

Terminal window
cosign generate-key-pair

Check that both files exist with ls cosign.*. cosign.key must never leave your vault: in this lesson's lab, it stays on the signing machine, and the cleanup removes it.

Signing the digest, not the tag

A tag can be reassigned to another image: so you sign the digest. crane digest resolves it, then cosign sign signs that precise digest.

Terminal window
crane digest localhost:5555/lab-signe:1.0.0
# sha256:afa5c872c891853ca7fcf1f12c3edb23f7eeef36189728842dd51042ff57f7ab
cosign sign --key cosign.key --allow-http-registry \
localhost:5555/lab-signe@sha256:afa5c872c891853ca7fcf1f12c3edb23f7eeef36189728842dd51042ff57f7ab

The digest above is the one read on the lab-signe:1.0.0 image of the trial. If crane digest prints a different one on your side (depending on how your Docker pushes the image to the registry), carry over the one you read into the cosign sign command.

The --allow-http-registry option exists only for this local registry without TLS: do not copy it to a production registry.

With its default signing configuration, cosign 3.1.3 also publishes an entry in Rekor, Sigstore's public transparency log. The signing machine must therefore reach the Internet, and this entry is public and cannot be erased. To publish nothing, for example in an isolated environment, the workaround is a signing configuration without a log: it is described in the troubleshooting section.

Signing by tag (lab-signe:1.0.0) also works, but the signature is attached in any case to the digest the tag points to at that moment. If the tag moves, the signature does not follow, and that is precisely what you want.

Where does the signature live, and how do you find it?

The signature is not in the image: it is a separate artifact, pushed to the same repository under a tag derived from the digest. The diagram follows the exchange over time: look at what the workstation pushes to the registry, then what the verifier asks it for again.

Sequence between a signing workstation, a registry and a verifier: the workstation pushes the image then its signature under a tag derived from the digest; the verifier asks for the signature again and checks it with the public key, with exit code 0, 1 or 10.Verifier(cosign.pub)RegistryWorkstation or CI(cosign.key)the repositoryholds twotags, 1.0.0 andsha256-abcchecks it withthe public keyresult 0 ifvalid, 1 ifwrong key, 10if nosignaturepushes the image, itsdigest is sha256:abcpushes the signatureof digest sha256:abc,tag sha256-abc, samerepositoryasks for thesignature of digestsha256:abcreturns the signature
Source · SVG
Esc to close, + and − to zoom, scroll to move around. 100 %

After signing, the repository therefore holds two tags. This command lists them, and crane ls localhost:5555/lab-signe gives the same information:

Terminal window
curl -s http://localhost:5555/v2/lab-signe/tags/list
Output
{"name":"lab-signe","tags":["1.0.0","sha256-afa5c872c891853ca7fcf1f12c3edb23f7eeef36189728842dd51042ff57f7ab"]}

The second tag is the signature. With cosign 3.x, it is stored as an artifact that references the image, following the OCI 1.1 specification (the one that defines container registries).

A registry without the Referrers API, the one that lists the artifacts attached to an image, stores this artifact under the tag sha256-<digest>, as here. A registry that exposes this API may not show this tag. This tag is no longer suffixed .sig by default: many articles still say otherwise, and a test that looked for .sig would wrongly declare that no signature exists.

Two practical consequences. Copying an image from one registry to another does not carry its signature, unless you also copy that artifact: a verification that fails after a registry migration may have this cause. And since the tag is derived from the digest, re-signing a rebuilt image produces a new artifact, the old one staying in place.

How do you publish the public key in the cluster?

For the cluster, or a third party, to verify your images, the public key must be readable in the cluster. It is published in a ConfigMap: here cosign-pub-key, in the chaine-signature namespace (the lab's, to be created with kubectl create namespace chaine-signature if it does not exist), under the cosign.pub key. The name of this key matters, because it is under this name that tools read it back.

Terminal window
kubectl -n chaine-signature create configmap cosign-pub-key \
--from-file=cosign.pub=cosign.pub

Never publish the private key cosign.key, neither in a ConfigMap nor in a Secret: anyone who read it could sign in your place, and the signature would then have no value at all.

To verify as a third party would, read the key back from the cluster rather than from your local file:

Terminal window
kubectl -n chaine-signature get configmap cosign-pub-key \
-o jsonpath='{.data.cosign\.pub}' > cosign-du-cluster.pub

The resulting file starts with -----BEGIN PUBLIC KEY-----. If it is empty, the ConfigMap's key name is the first suspect.

How do you check that the signature really protects?

A verification that succeeds does not yet prove that the signature protects: you must also see it fail when it should. A chain that accepts everything looks like a chain that works. The three situations in the table were played on the local registry with cosign 3.1.3 and return three different exit codes. The following command verifies the signed image with the right public key.

Terminal window
cosign verify --key cosign.pub --allow-http-registry \
localhost:5555/lab-signe@sha256:afa5c872c891853ca7fcf1f12c3edb23f7eeef36189728842dd51042ff57f7ab
echo "exit code: $?"

To trigger the other two cases, generate a second key pair in another directory, so as not to overwrite the first, and verify the same image with that public key. Then verify the lab-non-signe:1.0.0 image, which nobody signed, with the right key (replace <digest-non-signe> with the digest noted at docker push, or with the output of crane digest localhost:5555/lab-non-signe:1.0.0). Each time, echo prints the exit code.

Terminal window
mkdir autre-cle && (cd autre-cle && cosign generate-key-pair)
cosign verify --key autre-cle/cosign.pub --allow-http-registry \
localhost:5555/lab-signe@sha256:afa5c872c891853ca7fcf1f12c3edb23f7eeef36189728842dd51042ff57f7ab
echo "exit code: $?"
cosign verify --key cosign.pub --allow-http-registry \
localhost:5555/lab-non-signe@<digest-non-signe>
echo "exit code: $?"

With the right key, cosign details what it checked: the claims carried by the signature, their trace in the transparency log (verified offline), then the signature itself against the given public key.

Output
Verification for localhost:5555/lab-signe@sha256:afa5c872c891853ca7fcf1f12c3edb23f7eeef36189728842dd51042ff57f7ab --
The following checks were performed on each of these signatures:
- The cosign claims were validated
- Existence of the claims in the transparency log was verified offline
- The signatures were verified against the specified public key
SituationExit codeMessage
Valid signature0the detail of the checks above
Wrong public key1no matching attestations: failed to verify log inclusion (verification with the Rekor log)
Unsigned image10no signatures found

The distinction between 1 and 10 can be put to use. An unsigned image (10) signals an omission in the build chain; a signature that does not match the key (1) signals an identity problem. These two incidents do not call for the same response, and a script must test the exit code, not the text of the message. The message for a wrong key also depends on the verification configuration: with the default configuration, which checks inclusion in Rekor, it is no matching attestations: failed to verify log inclusion, whereas the Cosign guide, which verifies without a log, shows accepted signatures do not match threshold. The exit code stays the same.

Cosign also reserves codes 11, 12 and 13 for other verification failures: a nonexistent tag (11), no matching signature (12) and a certificate missing from the signature (13). For a wrong key, cosign 3.1.3 returned code 1, not 12. To refuse an image, therefore test for a code different from zero, and keep 10 to recognise the unsigned image.

When the trials are over, clean up: delete cosign.key, cosign.pub, cosign-du-cluster.pub and the autre-cle folder (the second key pair), then the signing-config.json file if you created it. Then delete the lab's namespace, which also removes the cosign-pub-key ConfigMap, and finally the throwaway registry.

Terminal window
rm -rf cosign.key cosign.pub cosign-du-cluster.pub autre-cle signing-config.json
kubectl delete namespace chaine-signature
docker rm -f registre

How does the cluster rely on the signature?

Cosign blocks nothing in the cluster: it answers one question, "is this signature valid for this key". It is admission, the control Kubernetes applies to every object before accepting it, that turns this answer into a refusal. In practice, a Kyverno policy verifies each image's signature before creating the Pod: see Kyverno for the image verification policy, and Admission controllers (in French) for the general mechanism.

Verification in Kubernetes with ImagePolicyWebhook

ImagePolicyWebhook is a historical admission controller, disabled by default, which delegates the authorisation decision to an external backend. It is still documented and useful for understanding image control mechanisms, but on recent clusters you rather meet generic admission webhooks (ValidatingAdmissionWebhook) or policy engines such as Kyverno.

One architecture point changes how to assess it: ImagePolicyWebhook decides nothing by itself. It forwards the request to an external service, reachable over TLS, which allows or refuses. The real security of the setup is therefore that of this backend, its availability and its rules, not that of the Kubernetes controller, which merely relays.

The diagram follows a Pod creation request: look at what the plugin does (it relays), what the backend does (it answers), and what happens when the backend does not answer.

Sequence between kubectl, the API server and an external backend: the admission plugin relays the Pod creation request to the backend, which allows or refuses, and the Pod is created or refused accordingly. When the backend is unreachable and defaultAllow is false, the Pod is refused. In a real setup, it is the backend that calls cosign verify.ExternalbackendAPI serverkubectlthe admissionplugin relays therequest, it doesnot decidethe Pod is createdor deniedaccording to theanswerdefaultAllow false:the Pod is deniedthe backend is theone that callscosign verifycreates a Podrequest over TLSallows or deniescreates another Pod,backend unreachable
Source · SVG
Esc to close, + and − to zoom, scroll to move around. 100 %

The demonstration below takes two configuration files and a change to the API server manifest, to be made on every control plane node. In /etc/kubernetes/manifests/kube-apiserver.yaml, the first flag enables the plugin and the second says where to read its configuration.

Since the API server is a Pod, the /etc/kubernetes/admission/ folder must be mounted into it, otherwise it will never see these files and will loop at startup. A mistake here prevents the API server from restarting: keep a copy of the manifest outside the manifests folder before modifying it, for example with sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.sauvegarde.

The Kubernetes documentation asks you to enable the imagepolicy.k8s.io/v1alpha1 API group with --runtime-config=imagepolicy.k8s.io/v1alpha1=true, to be added to the same command list. The excerpt below does not carry it: add it if the API server does not apply the policy.

/etc/kubernetes/manifests/kube-apiserver.yaml (excerpt)
spec:
containers:
- command:
- kube-apiserver
# ... other flags ...
- --enable-admission-plugins=NodeRestriction,ImagePolicyWebhook
- --admission-control-config-file=/etc/kubernetes/admission/admission-config.yaml
volumeMounts:
# ... existing mounts ...
- mountPath: /etc/kubernetes/admission
name: admission-config
readOnly: true
volumes:
# ... existing volumes ...
- name: admission-config
hostPath:
path: /etc/kubernetes/admission
type: DirectoryOrCreate
/etc/kubernetes/admission/admission-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: ImagePolicyWebhook
configuration:
imagePolicy:
kubeConfigFile: /etc/kubernetes/admission/webhook-kubeconfig.yaml
allowTTL: 50
denyTTL: 50
retryBackoff: 500
defaultAllow: false # Block if the webhook is unreachable

The second file is an ordinary kubeconfig: it tells the API server where to reach the backend and with which certificate authority to recognise it. It points to an address where no backend listens, which the demonstration that follows does not require you to fix. In a real setup, it is this backend, not ImagePolicyWebhook, that would call a verification such as cosign verify.

/etc/kubernetes/admission/webhook-kubeconfig.yaml
apiVersion: v1
kind: Config
clusters:
- name: image-policy
cluster:
certificate-authority: /etc/kubernetes/pki/ca.crt
server: https://127.0.0.1:9443/verify
contexts:
- name: image-policy
context:
cluster: image-policy
user: kube-apiserver
current-context: image-policy
users:
- name: kube-apiserver
user: {}

Replayed on a kind cluster (Kubernetes in Docker) running Kubernetes 1.37, the API server came back in 33 seconds with these three files. Since no backend listens on that port, the first Pod creation was refused:

Output
Error from server (Forbidden): pods "demo-ipw" is forbidden: Post "https://127.0.0.1:9443/verify?timeout=30s": dial tcp 127.0.0.1:9443: connect: connection refused

This is the expected behaviour of defaultAllow: false: the message names the cause, the unreachable backend, and nothing gets through until it answers. A production cluster must therefore monitor the availability of this backend like that of the API server itself.

When the demonstration is done, roll back: this configuration refuses every Pod as long as no backend listens. Put back the saved manifest, wait for the API server to answer again (kubectl get --raw /readyz must print ok), then check that a Pod can be created. Finally remove the /etc/kubernetes/admission folder, now useless since the original manifest no longer mounts it.

Terminal window
sudo cp /root/kube-apiserver.yaml.sauvegarde /etc/kubernetes/manifests/kube-apiserver.yaml
kubectl get --raw /readyz
kubectl run retour --image=nginx:1.31.3@sha256:8541484afbc9c8a5a8a99b379568ebbc957f658583ec9448fc43104229c03cf8
kubectl delete pod retour
sudo rm -r /etc/kubernetes/admission

As long as kubectl get --raw /readyz does not answer ok, give the kubelet time to restart the API server before creating the Pod.

Key-based or keyless signing?

Cosign offers two signing modes. Key-based signing, the one this lesson details, is explicit: you see what you generate and what you verify. It does leave you a long-lived private key to protect and rotate: if it leaks, a third party can sign in your place.

Keyless signing removes that problem by relying on an OIDC identity (OpenID Connect, an authentication protocol), that of your GitHub Actions or GitLab CI run, and on an ephemeral certificate. It suits an automated build chain that has an OIDC provider. Keep key-based signing to understand the mechanism and for isolated environments where no OIDC provider is reachable; in that case, without access to Sigstore's public services, sign with a signing configuration without a log (see the troubleshooting section). The Cosign guide details both.

Troubleshooting

When a verification fails, the exit code and the message almost always say where to look. The table starts from the observed symptom.

SymptomLikely causeSolution
no signatures found, code 10Image never signed, or signature not copied with the image during a registry migrationSign the digest, or also copy the signature artifact
no matching attestations, code 1Wrong public key, or image re-signed with another keyCheck the key; sign and verify on the same digest
A test looks for a .sig tag and finds noneWith cosign 3.x, the signature tag is sha256-<digest>Look for sha256-<digest>
--tlog-upload=false refused, or a signature that must not be published in RekorCosign 3.x refuses this option with its default signing configuration, which publishes to Rekor, and asks for a configuration without a log serviceCreate an empty signing configuration and pass it with --signing-config (commands below the table)
Pod refused with connection refusedImagePolicyWebhook backend unreachable with defaultAllow: falseRestore the backend: it is the default refusal doing its job
Key not found in the ConfigMapThe ConfigMap key is not named cosign.pubRecreate the ConfigMap with --from-file=cosign.pub=…

To sign without publishing anything in Rekor, follow the method of the Cosign guide: cosign signing-config create writes a configuration without any public service, which cosign sign receives with --signing-config (the -y option avoids the confirmation prompt). Since no log was used, verification requires --insecure-ignore-tlog=true.

Terminal window
cosign signing-config create --out signing-config.json # no public service
cosign sign --key cosign.key --signing-config signing-config.json \
--allow-http-registry -y \
localhost:5555/lab-signe@sha256:afa5c872c891853ca7fcf1f12c3edb23f7eeef36189728842dd51042ff57f7ab
cosign verify --key cosign.pub --insecure-ignore-tlog=true --allow-http-registry \
localhost:5555/lab-signe@sha256:afa5c872c891853ca7fcf1f12c3edb23f7eeef36189728842dd51042ff57f7ab

In this variant, verification no longer checks inclusion in a log: the public key alone commits the trust.

Put it into practice

The lab starts from a local registry mounted on the node, which holds two unsigned images, appli/web:1.0 and appli/outil:1.0. You generate a key pair, sign appli/web:1.0 and publish the public key in the cosign-pub-key ConfigMap of the chaine-signature namespace. Three tests read the state, never the commands typed: the public key is in the ConfigMap with no trace of the private key, the signature is pushed to the registry, and the key taken from the ConfigMap accepts the signed image and refuses the other one.

Key points

  • A signature proves authenticity and integrity, not the absence of vulnerabilities: the scan remains necessary.
  • The private key stays on the signing machine and never enters a cluster object; only cosign.pub is published, in the cosign-pub-key ConfigMap.
  • You sign the digest resolved with crane digest, never the tag.
  • Cosign 3.1.3's default signing publishes a public entry in Rekor; to publish nothing, sign with --signing-config.
  • The signature is a separate artifact, in the same repository, under the tag sha256-<digest> with cosign 3.x; copying an image does not copy it.
  • A verification is proved by its refusal: 0 if the signature is valid, 1 if the key is wrong, 10 if the image is unsigned.
  • The cluster relies on the signature at admission (Kyverno, ImagePolicyWebhook); ImagePolicyWebhook relays to a backend on which all the security depends, and defaultAllow: false refuses when it is unreachable.
  • Keyless signing avoids keeping a long-lived key when an OIDC provider is available; key-based signing serves to understand the mechanism and for isolated environments.

Test your knowledge

These questions ask you to predict the exit code of a verification, to choose what to sign and to diagnose a verification or admission failure.

Knowledge check

Check what you have learned with this interactive quiz

8 questions
8 min
70% to pass

How it works

  • The timer starts when you click Start
  • Multiple choice, true/false and short answer questions
  • You can move between questions
  • Detailed results are shown at the end

Starts the quiz and the timer