Skip to content
Français
Account: not signed in, sync available My learning
Select theme
Français

cert-manager: automatic TLS certificates in Kubernetes

By updated

Improve this page

Logo Kubernetes

cert-manager automates the issuing and renewal of your applications' TLS certificates in Kubernetes. You describe the certificate you want in a Certificate object, cert-manager requests it from an authority (an internal CA, Let's Encrypt through the ACME protocol, Vault...), stores the result in a Secret and renews it before it expires. This lesson covers the certificates your services present to their clients; the next one, Cluster PKI, covers those of the control plane. It is optional and is meant for whoever administers a cluster and wants HTTPS without watching expiry dates.

What you will learn

  • Say what cert-manager adds to the cluster PKI and read its three objects.
  • Install cert-manager and issue a first certificate with a self-signed Issuer.
  • Create an internal CA and sign a service certificate with it.
  • Get a Let's Encrypt certificate through ACME and secure an Ingress with an annotation.
  • Diagnose a certificate that stays not ready.

Prerequisites

  • A working Kubernetes cluster and a configured kubectl. The demonstration runs on the course's kind cluster, the lab on the catalog's kubeadm cluster.
  • Having followed the Ingress lesson, which has you create a TLS Secret and a certificate with openssl. The notions of authority and chain are recalled in the first section.
  • For the Let's Encrypt part only: a public domain name that points to the cluster. The lab does not need one.

What does cert-manager add to the cluster PKI?

A PKI (public key infrastructure) is the set of authorities and certificates that let programs trust each other. A certificate authority (CA) signs certificates with its private key; a client that trusts this authority accepts any certificate it has signed. The chain of trust is the verifiable link from a certificate up to the authority that signed it. TLS diagnostics with openssl goes deeper into these notions.

The cluster PKI, managed by kubeadm, serves authentication inside the cluster: it signs the certificates of the Kubernetes components, and those of users through a signing request (CertificateSigningRequest). It does not sign the certificates your services present to their clients. cert-manager does not replace it: it adds resources that issue these application certificates and renew them without human action.

You no longer handle openssl or .pem files: you declare what you want, and a controller (a program that keeps comparing the desired state with the actual state) does the work. Three objects are enough to get started.

ObjectRoleScope
IssuerA source of certificates (CA, ACME, Vault...)One namespace only
ClusterIssuerSame, but usable from the whole clusterWhole cluster
CertificateThe request for a concrete certificate (names, lifetime)One namespace

An Issuer only issues certificates to the Certificate objects of its own namespace. A ClusterIssuer serves every namespace: it is the usual choice for a company CA or a shared Let's Encrypt issuer.

When you create a Certificate, cert-manager runs through a chain of objects internally that you do not create yourself. The diagram shows it: the Certificate produces a CertificateRequest, which the issuer signs, and the result lands in a Secret of type kubernetes.io/tls. With ACME, the issuer also goes through Order and Challenge objects. These intermediate objects let you follow and diagnose an issuance, and the troubleshooting section works back through them one by one.

cert-manager issuance chain: the Certificate creates a CertificateRequest, the Issuer or ClusterIssuer signs it and the result is written to a kubernetes.io/tls Secret; for ACME, the issuer also goes through Order and Challenge objects.Certificatewhat you declareCertificateRequestthe signing requestIssuer or ClusterIssuersigns the requestSecret kubernetes.io/tlskey and certificateOrderACME onlyChallengeproof of domain control
Source · SVG
Esc to close, + and − to zoom, scroll to move around. 100 %

How do you install cert-manager?

Installation goes through a single manifest that deploys the three controllers (cert-manager, cert-manager-webhook and cert-manager-cainjector) and the CRDs, that is, the definitions of the new objects. Pin a precise version rather than a floating one. This lesson uses 1.21.3, the one the lab installs, with the server-side apply that the lab uses.

Terminal window
kubectl apply --server-side --force-conflicts \
-f https://github.com/cert-manager/cert-manager/releases/download/v1.21.3/cert-manager.yaml

Then wait for the deployments to be available, and read the controller image to confirm the version.

Terminal window
kubectl -n cert-manager wait --for=condition=Available deployment --all --timeout=300s
kubectl -n cert-manager get deploy cert-manager \
-o jsonpath='{.spec.template.spec.containers[0].image}'

The first command returns once cert-manager, cert-manager-webhook and cert-manager-cainjector are available. The second must print an image whose tag is v1.21.3.

cert-manager validates every resource through an admission webhook. An Issuer created right after installation can therefore fail with failed calling webhook: this is why you wait for the webhook's Available condition before going on.

The compatibility matrix of version 1.21 lists Kubernetes 1.33 to 1.36, while the course teaches 1.37. The lab of this lesson was played on a 1.37.2 cluster with no discrepancy. Read the official matrix for the version you install before deploying it to a production cluster.

How do you issue a first certificate with a self-signed Issuer?

A self-signed issuer (selfSigned) depends on no authority: each certificate is signed with its own key. No browser trusts it, so it is not for production. It is used to validate the installation, and to bootstrap an internal CA in the next section.

Create a self-signed ClusterIssuer, then a test Certificate.

first-certificate.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: selfsigned
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: test-cert
namespace: default
spec:
secretName: test-cert-tls
issuerRef:
name: selfsigned
kind: ClusterIssuer
commonName: test.example.com
dnsNames:
- test.example.com

Apply the file and read the certificate's Ready condition.

Terminal window
kubectl apply -f first-certificate.yaml
kubectl get certificate test-cert \
-o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'

The expected result is True: the test-cert-tls Secret then holds a private key and a certificate. To read the certificate, decode the tls.crt field of the Secret and pass it to openssl:

Terminal window
kubectl get secret test-cert-tls -o jsonpath='{.data.tls\.crt}' \
| base64 -d | openssl x509 -noout -subject -issuer -dates

The output gives the subject, the issuer and the validity dates. Here the subject and the issuer are identical, since the certificate signed itself.

How do you create an internal CA to sign your services?

A private network does not need Let's Encrypt: a single internal certificate authority can sign all the services. The principle comes down to three steps. The self-signed issuer creates a root CA certificate, a second issuer uses that certificate as its authority, and the application certificates are then signed by this second issuer.

  1. Create the root CA with isCA: true, signed by the self-signed issuer:

    internal-ca.yaml
    apiVersion: cert-manager.io/v1
    kind: Certificate
    metadata:
    name: ca-root
    namespace: cert-manager
    spec:
    isCA: true
    commonName: mon-lab-ca
    secretName: mon-lab-ca-secret
    issuerRef:
    name: selfsigned
    kind: ClusterIssuer
  2. Declare a ClusterIssuer of type ca that signs with the Secret of this CA:

    apiVersion: cert-manager.io/v1
    kind: ClusterIssuer
    metadata:
    name: mon-lab-ca
    spec:
    ca:
    secretName: mon-lab-ca-secret
  3. Issue a service certificate, with several names (SANs) and an early renewal:

    apiVersion: cert-manager.io/v1
    kind: Certificate
    metadata:
    name: app-cert
    namespace: default
    spec:
    secretName: app-cert-tls
    issuerRef:
    name: mon-lab-ca
    kind: ClusterIssuer
    commonName: app.interne.lab
    dnsNames:
    - app.interne.lab
    - api.interne.lab
    duration: 2160h # 90 days
    renewBefore: 360h # renews 15 days before expiry

The CA Secret of a ClusterIssuer lives in the namespace of cluster resources, which is cert-manager by default. The root CA is therefore created there. The lab of this lesson uses Issuer objects rather than ClusterIssuer objects: the CA and its Secret then live in the working namespace, and the reasoning is the same.

A Ready certificate is not necessarily signed by the right authority. Check the chain with openssl verify, against your CA's public certificate.

Terminal window
kubectl -n cert-manager get secret mon-lab-ca-secret \
-o jsonpath='{.data.tls\.crt}' | base64 -d > ca.pem
kubectl get secret app-cert-tls -o jsonpath='{.data.tls\.crt}' \
| base64 -d > app.pem
openssl verify -CAfile ca.pem app.pem

openssl verify answers OK when app.pem leads to ca.pem. For browsers and curl to trust the signed services, distribute this public ca.pem to the client machines. Never distribute the private key, which stays in the Secret.

How do you get a Let's Encrypt certificate with ACME?

For a service exposed on the Internet, Let's Encrypt issues browser-trusted certificates for free. cert-manager speaks the ACME protocol and proves that you control the domain with a challenge. The ACME certificates guide details the protocol itself; here we only look at what it changes in cert-manager.

Two challenges exist. With HTTP-01, cert-manager publishes a temporary file under http://your-domain/.well-known/acme-challenge/: it is simple, but the domain must be publicly reachable over HTTP. With DNS-01, it creates a TXT record in your DNS zone: it is more complex, but it is the only way to get a wildcard certificate (*.example.com), and the service does not need to be exposed.

Let's Encrypt applies strict quotas in production. Tune your configuration on the staging endpoint (acme-staging-v02), which issues untrusted certificates but with no tight quota, then switch to production once issuance works.

Here is a Let's Encrypt ClusterIssuer with an HTTP-01 challenge, solved through the course's traefik Ingress class:

letsencrypt.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: admin@example.com
privateKeySecretRef:
name: letsencrypt-prod-account
solvers:
- http01:
ingress:
ingressClassName: traefik

For a wildcard, replace the solver with a DNS-01 challenge tied to your DNS provider. The configuration depends on the provider; the cert-manager documentation lists the supported DNS-01 solvers. Neither this ClusterIssuer nor the Ingress of the next section is replayed in the lab, for lack of public access to the cluster.

How do you secure an Ingress automatically?

One annotation is enough for an Ingress to obtain and then renew its certificate. Add cert-manager.io/cluster-issuer with the issuer's name, and a tls block that names the Secret to fill.

ingress-tls.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: mon-app
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
ingressClassName: traefik
tls:
- hosts:
- app.example.com
secretName: app-example-tls
rules:
- host: app.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: mon-app
port:
number: 80

cert-manager spots the annotation, creates the matching Certificate, solves the challenge and fills the app-example-tls Secret. The Ingress then serves traffic over HTTPS. Annotations depend on the Ingress controller, but this one is read by cert-manager itself; the full Ingress configuration is in the Kubernetes Ingress lesson.

How does cert-manager renew certificates?

cert-manager watches every certificate and requests a new one before it expires, with no cron or script on your side. The renewBefore field sets the margin; without it, the default margin is one third of the lifetime. To read a certificate's expiry:

Terminal window
kubectl get certificate app-cert -o jsonpath='{.status.notAfter}'

At renewal time, cert-manager updates the same Secret. An Ingress controller reloads the new certificate by itself. For a Pod that mounts the Secret as a volume, the kubelet updates the mounted file after a short delay, unless the mount goes through subPath, which is never updated; the application must then re-read the file, which many only do on restart: check the behavior of yours.

Why does a certificate stay not ready, and how do you diagnose it?

When a Certificate does not reach Ready, the cause is not always visible on it: you have to work back through the chain of objects in the diagram, from the nearest to the farthest. A simple case to provoke is a Certificate whose issuerRef names an issuer that does not exist: this is the failure the lab sets up. The certificate then stays without Ready, and the cause shows in its events and in its CertificateRequest.

Terminal window
# 1. the certificate itself: Ready condition and events
kubectl describe certificate app-cert
# 2. the underlying request
kubectl describe certificaterequest
# 3. for ACME: the order and the challenge
kubectl describe order
kubectl describe challenge
# 4. the namespace events
kubectl get events

In each describe, read the Ready condition and the Events section at the end of the output: they say which object blocks and why. The table links the most common symptoms to their cause.

SymptomLikely causeFix
failed calling webhookWebhook not ready yetWait for the webhook's Available condition
Certificate without Ready, issuer not foundissuerRef points to an issuer that does not existFix issuerRef (name and kind) or create the issuer
HTTP-01 challenge pendingDomain not publicly reachable over HTTPCheck the DNS and the exposure of the Ingress
rateLimited on ACMELet's Encrypt production quota reachedGo back to staging while you test
CA Secret missingThe root CA is not ReadyFix the CA's Certificate first

How do you protect cert-manager and the keys it handles?

cert-manager handles private keys: treat it as a sensitive component. The first reflex is to keep it up to date. Version 1.20.3 fixed a HIGH severity flaw (GHSA-8rvj-mm4h-c258): the aggregated cert-manager-edit ClusterRole wrongly granted the right to create ACME Challenge and Order resources, which made it possible to hijack the configuration of a solver. The 1.21.3 of this lesson is later than that fix.

A kubernetes.io/tls Secret holds the service's private key: restrict the Role objects that can read it. The internal CA's Secret needs even more care, since it can sign any certificate: minimal access and a backup outside the cluster. Finally, separate the issuers by use (staging, production, internal CA) to partition their scopes. An authority driven from outside the cluster is also possible: cert-manager can rely on Vault PKI (in French) as a provider.

Put it into practice

The lab replays the core of this lesson on the catalog's kubeadm cluster, where cert-manager 1.21.3 is installed for you. You create a self-signed issuer auto-signe, an internal CA (a ca-interne Certificate with isCA, then an Issuer of the same name), the certificate of a vitrine-tls service signed by it, then you repair the api-tls certificate, left not ready because it targets a non-existent issuer. The tests read the issuers and the certificates, then check each chain with openssl verify against the internal CA: a certificate that is ready but signed by the wrong issuer does not pass. Let's Encrypt stays out of the lab, for lack of public access.

Key takeaways

  • cert-manager automates the issuing and renewal of your applications' TLS certificates; it complements the cluster PKI without replacing that of the control plane.
  • Issuer and ClusterIssuer are the source of certificates, Certificate is the request: the result lands in a kubernetes.io/tls Secret.
  • The self-signed issuer validates the installation, the internal CA signs your private services and is checked with openssl verify.
  • Let's Encrypt through ACME issues public certificates: HTTP-01 for a reachable domain, DNS-01 for a wildcard. Test on staging first.
  • The cert-manager.io/cluster-issuer annotation on an Ingress is enough to switch it to HTTPS.
  • A not ready certificate is diagnosed by working back through Certificate, CertificateRequest, Order then Challenge.

Knowledge check

Eight prediction and diagnosis questions: the role of each object, reading a signing chain, choosing an ACME challenge and working back through a not ready certificate.

Knowledge check

Check what you have learned with this interactive quiz

8 questions
8 min
70% to pass

How it works

  • The timer starts when you click Start
  • Multiple choice, true/false and short answer questions
  • You can move between questions
  • Detailed results are shown at the end

Starts the quiz and the timer