By default, traffic between Kubernetes Pods travels in clear text. Any Pod in the cluster can intercept another Pod's communications: all it takes is network access. Two families of tools answer this problem, and the CKS syllabus names both: Cilium's transparent encryption, which works inside the network without the application knowing, and Istio's mTLS, which gives every service a verifiable identity. This guide shows you both, how to enable them and, above all, how to prove that the traffic is really encrypted.
Pod-to-pod encryption belongs to the Minimize Microservice Vulnerabilities domain of the CKS, which weighs 20% of the exam according to the official CNCF curriculum. The competency is worded there as: Implement Pod-to-Pod encryption (Cilium, Istio). Both names appear, and they do not do the same job.
| Cilium with WireGuard | Istio with mTLS | |
|---|---|---|
| What encrypts | the CNI datapath, in the kernel | one proxy per Pod |
| What the application sees | nothing, no container added | a sidecar, or a node proxy in ambient mode |
| Scope | all traffic between nodes | traffic between Pods in the mesh |
| Service identity | no, encryption is at node level | yes, SPIFFE, usable in access policies |
| Identity-based access control | no, that stays with NetworkPolicies | yes, AuthorizationPolicy |
| Activation | one option of the chart | a mesh to install and inject |
Keep the line that decides: if the need is "nothing should travel in clear text anymore", Cilium is enough and costs one option. If the need is "service A may call B but not C", you need a service identity, so Istio.
What you will learn
- Why clear-text traffic between Pods is a security risk
- How mTLS works (SPIFFE identity, X.509 certificates, handshake)
- Configure Istio to enforce mTLS in STRICT mode
- Verify that encryption is active between two services
- Control access with AuthorizationPolicy and mTLS principals
- Enable and verify Cilium WireGuard encryption, then choose between Cilium and Istio
- Choose between the sidecar and ambient (ztunnel) approaches
Prerequisites
- A Kubernetes cluster with administrator access
- Familiarity with Pods and Network Policies
- Understanding of service mesh (in French) concepts (data plane, control plane)
- istioctl installed (the Istio command line)
Why encrypt traffic between Pods
In a Kubernetes cluster, the network is flat by default: every Pod can talk to every other Pod without restriction. This model simplifies deployment, but it creates a major risk: if an attacker compromises a single Pod, they can intercept (sniffing), modify (man-in-the-middle) or impersonate the traffic of other services.
Network Policies solve part of the problem by filtering traffic at the L3/L4 level (IPs and ports), but they do neither encryption nor authentication. The table below shows how the two complement each other:
| Mechanism | Layer | Encryption | Authentication | What it controls |
|---|---|---|---|---|
| NetworkPolicy | L3/L4 | No | No | Which Pods can talk to each other (IP/port) |
| mTLS | L4/L7 | Yes (TLS 1.2+) | Yes (X.509 certificates) | Who talks to whom (cryptographic identity) |
Combining the two gives you a defense in depth model: NetworkPolicies limit the allowed flows, and mTLS guarantees that every flow is encrypted and that both ends are who they claim to be.
How mTLS works
Classic TLS (the one behind HTTPS) is one-way: only the client verifies the server's identity. Mutual TLS (mTLS) adds a step: the server also verifies the client's identity. Both parties present an X.509 certificate.
The mTLS handshake, step by step
The five steps below describe a classic TLS exchange with one extra check: the client's certificate. Step 3 makes all the difference: in an ordinary HTTPS connection, the browser verifies the server and the server knows nothing about the client. With mTLS, each side proves its identity to the other.
Keep in mind that all of this happens before the first application request, and without any change to your code: in a service mesh, the injected proxy next to the container runs these five steps.
-
The client starts the TLS connection: it sends a "ClientHello" with the cipher suites it supports.
-
The server presents its certificate: the client checks that the certificate is signed by a trusted certificate authority (CA) and that the identity matches the expected service.
-
The server asks for the client's certificate: this is the extra step compared with classic TLS. The client presents its own certificate.
-
The server verifies the client's certificate: it confirms that the client is an authorized service, signed by the same CA.
-
The encrypted session is established: both parties exchange a symmetric session key. All the following traffic is encrypted and authenticated.
SPIFFE identity
In a Kubernetes cluster, the identity of each service is encoded in the certificate as a SPIFFE ID (Secure Production Identity Framework For Everyone). The standard format is:
spiffe://<trust-domain>/ns/<namespace>/sa/<service-account>For example, a service using the ServiceAccount api-server in the namespace production has the identity:
spiffe://cluster.local/ns/production/sa/api-serverThis identity based on the Kubernetes ServiceAccount is at the heart of the mTLS security model. It lets you define granular access policies: "only the frontend service in the production namespace can call the api-server service".
This identity is not theoretical, you can read it in the proxy. The
following command queries the proxy of the client Pod, which runs in the
lab-mesh namespace with the default ServiceAccount. This Pod and the mesh are created
in the "Create the demo application" section: run this
command again after creating them.
kubectl exec client -n lab-mesh -c istio-proxy -- curl -s localhost:15000/certs | grep -o 'spiffe://[^"]*' | head -1spiffe://cluster.local/ns/lab-mesh/sa/defaultThe head -1 keeps only one line: the proxy repeats the same identity
several times in its response (20 times during the test). The certificate
therefore carries exactly the format announced above.
The proxy is a native sidecar, not an ordinary container
Since Istio 1.27, injection places the proxy as a native sidecar by default. On a
Pod in a namespace labeled istio-injection=enabled, like the app Pod created
further down, the application container stays alone in containers, and the proxy
appears in initContainers:
kubectl get pod app -o jsonpath='{.spec.containers[*].name}{"\n"}'kubectl get pod app -o jsonpath='{.spec.initContainers[*].name}{"\n"}'appistio-init istio-proxyThe proxy is there, but in the initContainers, with the signature of the
native sidecar. This command prints the restartPolicy field of each
initContainer:
kubectl get pod app -o jsonpath='{range .spec.initContainers[*]}{.name}{" : restartPolicy="}{.restartPolicy}{"\n"}{end}'istio-init : restartPolicy=istio-proxy : restartPolicy=Alwaysistio-init has no restartPolicy: it is an ordinary init container,
which stops once its task is done. Only istio-proxy is a native
sidecar.
It is this restartPolicy: Always on an initContainer that distinguishes a native
sidecar from a simple init container: it starts before the application
containers, stays alive for the whole life of the Pod, and stops after them. The guaranteed order also fixes a historical flaw of the sidecars
of old, where the application started before its proxy and failed its
first requests.
Istio documents this itself in the Pod's sidecar.istio.io/status
annotation. This command reads it and prints the two lists of containers:
kubectl get pod app -o jsonpath='{.metadata.annotations.sidecar\.istio\.io/status}' | python3 -c 'import json,sys; d=json.load(sys.stdin); print("containers =", d.get("containers")); print("initContainers =", d.get("initContainers"))'containers = NoneinitContainers = ['istio-init', 'istio-proxy']Automatic certificate management
Without a service mesh, you would have to manage the certificates of each service by hand: generation, distribution, rotation, revocation. With a mesh like Istio, these four operations are automatic.
| Step | What happens |
|---|---|
| Generation | The proxy (Envoy or ztunnel) generates a private key and sends a CSR (Certificate Signing Request) to the control plane |
| Signing | The control plane (istiod) signs the CSR after verifying the Pod's identity through its ServiceAccount |
| Distribution | The signed certificate is delivered to the proxy through the SDS API (Secret Discovery Service) |
| Rotation | Certificates are renewed automatically before they expire (default lifetime: 24h) |
The application code sees none of this: the proxy handles mTLS transparently.
Encrypt with Cilium, the sidecar-free way
This is the simpler of the two solutions, and the one the syllabus cites first. Cilium encrypts traffic between nodes in its own datapath, with WireGuard: no container is added to the Pods, no application manifest is touched, and the application knows nothing about what happens.
The prerequisite is to have Cilium as your CNI. Its installation and concepts are covered in the dedicated guide, Cilium, the eBPF CNI (in French).
Enable WireGuard encryption
Two chart options are enough, set at installation or through an upgrade:
helm upgrade --install cilium cilium/cilium --version 1.20.1 \ --namespace kube-system \ --set encryption.enabled=true \ --set encryption.type=wireguardMeasured in a lab on Cilium 1.20.1 and Kubernetes 1.37: once the
installation is finished, the nodes become Ready right away.
Check that encryption is active
This is the step that proves encryption is active, and it does not consist of re-reading the option:
kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status | grep -i encryptionEncryption: Wireguard [NodeEncryption: Disabled, cilium_wg0 (Pubkey: v38E2wsPmrYwEnN5Y9rRs/..., Port: 51871, Peers: 1)]Three useful pieces of information in this line. Encryption is indeed
WireGuard, it goes through a cilium_wg0 interface on port
51871/UDP, and the number of peers must equal the number of other nodes in the
cluster. Peers: 0 on a multi-node cluster means the nodes did not
pair up.
NodeEncryption: Disabled is not an anomaly, it is the default. Traffic
between Pods is encrypted; traffic sent by the nodes themselves is
encrypted only if you enable encryption.nodeEncryption=true, which is a
separate and more intrusive setting.
The proof that counts: watch the tunnel
An enabled option and a green status do not prove that a single byte went through the tunnel. The interface counter does. Two series of 40 HTTP requests from the same client Pod, one to a Pod on the same node, the other to a Pod on a different node:
# On the node, before and after each seriescat /sys/class/net/cilium_wg0/statistics/tx_bytes| Destination | Bytes sent out by cilium_wg0 |
|---|---|
| Pod on the same node | 0 |
| Pod on another node | 42,880 |
The result says exactly what WireGuard does here: it encrypts what leaves the machine. Two co-located Pods communicate without ever touching the tunnel, which is logical: their traffic goes over no wire.
The negative control completes the demonstration. After
--set encryption.enabled=false and a restart of the agents, cilium-dbg status shows Encryption: Disabled, the interface is gone
(Device "cilium_wg0" does not exist), and traffic between the same two Pods
still goes through, with HTTP 200. Encryption was never a
condition for things to work, only a property of the transport.
What Cilium does not give you
Transparent encryption protects confidentiality on the wire. It gives no service identity: for Cilium, a Pod is a security identity derived from its labels, which you filter with NetworkPolicies. If you need to answer "is service A allowed to call B", you need the other approach, and that is the subject of the Istio section below.
Enable mTLS with Istio
Istio is a service mesh that the CKS syllabus names explicitly. It offers two deployment models: the classic sidecar mode and the ambient mode (no sidecar). In both cases, mTLS is configured with the same Kubernetes resources. To go from mTLS to timeouts, retries and authorizations between services, the guide Enforce timeouts, retries and authorizations between services (in French) takes over.
Install Istio
This section installs Istio so that you can practice, but it only serves to get a mesh in place. You can therefore skim it if you already have one, and come back when you build your own environment. The Istio documentation covers the options this guide does not detail.
The outputs on this page were captured with Istio 1.32.0 on a
kind cluster (kind v0.33.0, image kindest/node:v1.37.0, a single node), created
with kind create cluster --name demo-istio --image kindest/node:v1.37.0 --wait 120s.
Pod names carry a random suffix, and durations vary on your side.
For a lab cluster, the minimal installation with the demo profile:
# Download istioctlcurl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.32.0 sh -cd istio-1.32.0export PATH=$PWD/bin:$PATH
# Install with the demo profile (includes ingress + egress gateways)istioctl install --set profile=demo -yCheck:
kubectl get pods -n istio-systemExpected result, istiod and the gateways must be Running:
NAME READY STATUS RESTARTS AGEistio-egressgateway-84b6d9fcb9-fmrnb 1/1 Running 0 12sistio-ingressgateway-59859d889c-8plqj 1/1 Running 0 12sistiod-5b4f5c4699-8ffhk 1/1 Running 0 26sIf a Pod is not ready yet, this command waits until they all are:
kubectl -n istio-system wait --for=condition=Ready pod --all --timeout=300spod/istio-egressgateway-84b6d9fcb9-fmrnb condition metpod/istio-ingressgateway-59859d889c-8plqj condition metpod/istiod-5b4f5c4699-8ffhk condition metInject the sidecar proxies
In sidecar mode, Istio injects an Envoy proxy into every Pod. To enable automatic injection on a namespace:
kubectl label namespace default istio-injection=enablednamespace/default labeledEvery Pod created in this namespace now receives the Envoy proxy in addition to your application, and kubectl get pods shows 2/2 in the READY column.
Create the demo application
The following sections need a few workloads to observe. This
demo application comes in three blocks: an app Pod in default,
a client Pod in a lab-mesh namespace, then a small application in
production with a client outside the mesh in hors-maillage. The images are
pinned by digest.
The first block creates the app Pod, waits for it to be ready and displays it. The
READY column must show 2/2: the nginx container and the proxy, which is
counted even though it lives in the initContainers.
kubectl run app --image=nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb --port=80kubectl wait --for=condition=Ready pod/app --timeout=180skubectl get podspod/app createdpod/app condition metNAME READY STATUS RESTARTS AGEapp 2/2 Running 0 12sThe second block creates the lab-mesh namespace, with injection, and a client Pod
that sleeps for an hour, long enough to query it:
kubectl create namespace lab-meshkubectl label namespace lab-mesh istio-injection=enabledkubectl -n lab-mesh run client --image=curlimages/curl:8.23.0@sha256:3fb7e13e5d179a845c97d78204f3f1064d1907037f26a3ebf91710571151fd93 --command -- sleep 3600kubectl -n lab-mesh wait --for=condition=Ready pod/client --timeout=180snamespace/lab-mesh creatednamespace/lab-mesh labeledpod/client createdpod/client condition metThe third block is a manifest. It creates the production namespace with
injection, an api-server service (nginx behind a Service on port
8080), a frontend Pod that runs with its own ServiceAccount, an intrus
Pod that keeps the default ServiceAccount, and a hors-maillage
namespace whose dehors Pod has no proxy.
apiVersion: v1kind: Namespacemetadata: name: production labels: istio-injection: enabled---apiVersion: v1kind: ServiceAccountmetadata: name: frontend namespace: production---apiVersion: apps/v1kind: Deploymentmetadata: name: api-server namespace: productionspec: replicas: 1 selector: matchLabels: app: api-server template: metadata: labels: app: api-server spec: containers: - name: nginx image: nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb ports: - containerPort: 80---apiVersion: v1kind: Servicemetadata: name: api-server namespace: productionspec: selector: app: api-server ports: - port: 8080 targetPort: 80---apiVersion: v1kind: Podmetadata: name: frontend namespace: productionspec: serviceAccountName: frontend containers: - name: curl image: curlimages/curl:8.23.0@sha256:3fb7e13e5d179a845c97d78204f3f1064d1907037f26a3ebf91710571151fd93 command: ["sleep", "3600"]---apiVersion: v1kind: Podmetadata: name: intrus namespace: productionspec: containers: - name: curl image: curlimages/curl:8.23.0@sha256:3fb7e13e5d179a845c97d78204f3f1064d1907037f26a3ebf91710571151fd93 command: ["sleep", "3600"]---apiVersion: v1kind: Namespacemetadata: name: hors-maillage---apiVersion: v1kind: Podmetadata: name: dehors namespace: hors-maillagespec: containers: - name: curl image: curlimages/curl:8.23.0@sha256:3fb7e13e5d179a845c97d78204f3f1064d1907037f26a3ebf91710571151fd93 command: ["sleep", "3600"]Apply it, wait for the Pods, then check that the three Pods in
production are at 2/2 and that dehors is at 1/1: it has no proxy,
by design.
kubectl apply -f application.yamlkubectl -n production wait --for=condition=Ready pod --all --timeout=180skubectl -n hors-maillage wait --for=condition=Ready pod/dehors --timeout=180skubectl get pods -n productionkubectl get pods -n hors-maillagenamespace/production createdserviceaccount/frontend createddeployment.apps/api-server createdservice/api-server createdpod/frontend createdpod/intrus creatednamespace/hors-maillage createdpod/dehors createdpod/api-server-7ff9b5bf75-zm9xk condition metpod/frontend condition metpod/intrus condition metpod/dehors condition metNAME READY STATUS RESTARTS AGEapi-server-7ff9b5bf75-zm9xk 2/2 Running 0 58sfrontend 2/2 Running 0 58sintrus 2/2 Running 0 58sNAME READY STATUS RESTARTS AGEdehors 1/1 Running 0 4sAmbient mode, without sidecars
Since Istio 1.24, a second, stable architecture coexists with sidecars and changes the trade-off. Ambient mode replaces the proxy injected into each Pod with a ztunnel, deployed once per node as a DaemonSet. Encryption and L4 authentication are provided without modifying any application Pod, which removes injection, workload restarts and the cost of an extra container per Pod.
It is enabled with a label on the namespace:
kubectl label namespace default istio.io/dataplane-mode=ambientnamespace/default labeledBe careful: this label alone changes nothing on the installation done above. The
demo profile installs neither the ztunnel nor the istio-cni plugin that ambient
mode requires: the label is applied without error, and the Pods keep their sidecar.
This command confirms it, it finds no ambient Pod:
kubectl get pods -A | grep -i -E "ztunnel|istio-cni"It prints nothing and exits with code 1, the code grep returns when it finds
no line: this is the expected answer on the demo profile.
To try ambient, you must install Istio with the ambient profile
(istioctl install --set profile=ambient) instead of the demo profile. To
keep the sidecar path of this page, remove the label:
kubectl label namespace default istio.io/dataplane-mode-namespace/default unlabeledThe trade-off is worth knowing before you choose: the ztunnel only covers the L4 level. As soon as you need L7 features, HTTP routing or AuthorizationPolicy on URL paths, you must add a waypoint proxy per namespace, and the simplicity gain shrinks accordingly.
PeerAuthentication: enforce mTLS
The PeerAuthentication resource controls the mTLS mode for a namespace or a specific workload. It is the resource that enforces mTLS in Istio.
Three modes are available:
| Mode | Behavior | Usage |
|---|---|---|
| PERMISSIVE | Accepts mTLS and clear-text traffic | Gradual migration (Istio default) |
| STRICT | Accepts only mTLS traffic | Secure production |
| DISABLE | Disables mTLS | Not recommended except in very specific cases |
STRICT mTLS across the whole mesh
To enforce encryption on all namespaces of the cluster:
apiVersion: security.istio.io/v1kind: PeerAuthenticationmetadata: name: default namespace: istio-system # root namespace = applies to the whole meshspec: mtls: mode: STRICTkubectl apply -f - <<EOFapiVersion: security.istio.io/v1kind: PeerAuthenticationmetadata: name: default namespace: istio-systemspec: mtls: mode: STRICTEOFpeerauthentication.security.istio.io/default createdThe list confirms that the mesh policy is STRICT:
kubectl get peerauthentication -ANAMESPACE NAME MODE AGEistio-system default STRICT 1sSTRICT mTLS on a namespace
To apply STRICT mode only to the production namespace:
apiVersion: security.istio.io/v1kind: PeerAuthenticationmetadata: name: default namespace: productionspec: mtls: mode: STRICTSave this manifest as pa-namespace.yaml, then apply it:
kubectl apply -f pa-namespace.yamlpeerauthentication.security.istio.io/default createdmTLS on a specific workload
To target a precise workload with a selector:
apiVersion: security.istio.io/v1kind: PeerAuthenticationmetadata: name: api-strict namespace: productionspec: selector: matchLabels: app: api-server mtls: mode: STRICTSave it as pa-workload.yaml and apply it the same way
(kubectl apply -f pa-workload.yaml answers peerauthentication.security.istio.io/api-strict created).
With the three policies in place, the list shows them all:
kubectl get peerauthentication -ANAMESPACE NAME MODE AGEistio-system default STRICT 56sproduction api-strict STRICT 0sproduction default STRICT 0sAuthorizationPolicy: identity-based access control
STRICT mTLS guarantees that all traffic is encrypted and authenticated. But it does not control who can call what. That is the role of AuthorizationPolicy.
Thanks to mTLS, Istio knows the SPIFFE identity of every request. You can use that information to define granular access rules.
Allow only certain services
The following example allows only the frontend service (through its ServiceAccount) to access the api-server service:
apiVersion: security.istio.io/v1kind: AuthorizationPolicymetadata: name: api-server-access namespace: productionspec: selector: matchLabels: app: api-server action: ALLOW rules: - from: - source: principals: - "cluster.local/ns/production/sa/frontend"The source.principals field lists identities derived from the client's certificate, in the format <trust-domain>/ns/<namespace>/sa/<service-account>. Istio's documentation states that this field "requires mTLS enabled": a request received in clear text carries no identity, so a principals rule cannot allow it.
Save the policy as az-access.yaml, apply it, wait until it
is active, then call api-server from frontend (the allowed
principal) and from intrus (another ServiceAccount).
The wait is needed for every policy on this page: istiod pushes
the policy to the proxies within a few seconds, and the effect is not immediate.
A curl run right after kubectl apply can therefore still get
the old answer. The loop below retries every 2 seconds and returns
when intrus gets the expected 403:
kubectl apply -f az-access.yamluntil kubectl -n production exec intrus -c curl -- curl -s -o /dev/null -w '%{http_code}' --max-time 5 http://api-server.production.svc.cluster.local:8080/ | grep -q 403; do sleep 2; donekubectl -n production exec frontend -c curl -- curl -s -o /dev/null -w 'HTTP %{http_code}\n' --max-time 5 http://api-server.production.svc.cluster.local:8080/kubectl -n production exec intrus -c curl -- curl -s -w '\nHTTP %{http_code}\n' --max-time 5 http://api-server.production.svc.cluster.local:8080/authorizationpolicy.security.istio.io/api-server-access createdHTTP 200RBAC: access deniedHTTP 403frontend gets through, intrus receives an HTTP 403: the refusal comes from
the identity, not from the address.
Deny-by-default policy
For a namespace in "zero trust" mode, start by denying all traffic, then explicitly open the allowed flows. First remove the policy from the previous example, so that the measurements do not get mixed up:
kubectl -n production delete authorizationpolicy api-server-accessauthorizationpolicy.security.istio.io "api-server-access" deleted from production namespace# Step 1: deny everything by defaultapiVersion: security.istio.io/v1kind: AuthorizationPolicymetadata: name: deny-all namespace: productionspec: action: ALLOW # No rules → never matches → everything is deniedSave the policy as az-deny-all.yaml and apply it
(kubectl apply -f az-deny-all.yaml answers authorizationpolicy.security.istio.io/deny-all created).
The proxies receive it after a few seconds: wait until frontend
gets the 403, the loop then returns. A service that tries to call
api-server then gets an HTTP 403 error, even frontend:
until kubectl -n production exec frontend -c curl -- curl -s -o /dev/null -w '%{http_code}' --max-time 5 http://api-server.production.svc.cluster.local:8080/ | grep -q 403; do sleep 2; donekubectl -n production exec frontend -c curl -- curl -s -w '\nHTTP %{http_code}\n' --max-time 5 http://api-server.production.svc.cluster.local:8080/RBAC: access deniedHTTP 403# Step 2: explicitly allow the frontend → api-server flowapiVersion: security.istio.io/v1kind: AuthorizationPolicymetadata: name: allow-frontend-to-api namespace: productionspec: selector: matchLabels: app: api-server action: ALLOW rules: - from: - source: principals: - "cluster.local/ns/production/sa/frontend" to: - operation: methods: ["GET", "POST"] paths: ["/api/*"]Save this second policy as az-allow.yaml, then apply it with
kubectl apply -f az-allow.yaml (answer
authorizationpolicy.security.istio.io/allow-frontend-to-api created). The
loop waits until frontend gets the 404 on /api/x, a sign that
the authorization has reached the proxy. Three calls then show what
is open: frontend on /api/x, frontend on /, then intrus on
/api/x.
until kubectl -n production exec frontend -c curl -- curl -s -o /dev/null -w '%{http_code}' --max-time 5 http://api-server.production.svc.cluster.local:8080/api/x | grep -q 404; do sleep 2; donekubectl -n production exec frontend -c curl -- curl -s -o /dev/null -w 'HTTP %{http_code}\n' --max-time 5 http://api-server.production.svc.cluster.local:8080/api/xkubectl -n production exec frontend -c curl -- curl -s -w '\nHTTP %{http_code}\n' --max-time 5 http://api-server.production.svc.cluster.local:8080/kubectl -n production exec intrus -c curl -- curl -s -w '\nHTTP %{http_code}\n' --max-time 5 http://api-server.production.svc.cluster.local:8080/api/xHTTP 404RBAC: access deniedHTTP 403RBAC: access deniedHTTP 403The 404 is good news: the request got past authorization, and it is
nginx that has no /api/x page. Outside the /api/* path, or with another
principal, the answer stays a 403.
A habit to keep in production: apply the
principle of least privilege in this exact order. First set a
policy that denies everything, then explicitly allow each necessary flow
by naming the callers by their SPIFFE principal. Keep the chain in mind: a
PeerAuthentication in STRICT guarantees that the identity is proven, and an
AuthorizationPolicy then decides what that identity is allowed to
do.
Switching to STRICT, put to the test
This is the central claim of this page, and it is proven with three measurements on the same cluster. One server in a namespace of the mesh, one client outside the mesh, one client inside the mesh.
To make the PERMISSIVE measurement possible, you must first remove
what the previous sections set up: the STRICT of the whole mesh and the one on
production, which would already refuse the client outside the mesh, and the
AuthorizationPolicy resources, which would skew the answers.
kubectl -n production delete authorizationpolicy deny-all allow-frontend-to-apikubectl delete peerauthentication default -n istio-systemkubectl delete peerauthentication default api-strict -n productionkubectl get peerauthentication -Aauthorizationpolicy.security.istio.io "deny-all" deleted from production namespaceauthorizationpolicy.security.istio.io "allow-frontend-to-api" deleted from production namespacepeerauthentication.security.istio.io "default" deleted from istio-system namespacepeerauthentication.security.istio.io "default" deleted from production namespacepeerauthentication.security.istio.io "api-strict" deleted from production namespaceNo resources foundWith no PeerAuthentication at all, the mode is PERMISSIVE (the default). Deletions
propagate to the proxies just like creations: the loop waits until
dehors gets a 200 again. First measurement, from the dehors Pod,
which has no proxy:
until kubectl -n hors-maillage exec dehors -- curl -s -o /dev/null -w '%{http_code}' --max-time 5 http://api-server.production.svc.cluster.local:8080/ | grep -q 200; do sleep 2; donekubectl -n hors-maillage exec dehors -- curl -s -o /dev/null -w 'HTTP %{http_code}\n' --max-time 5 http://api-server.production.svc.cluster.local:8080/HTTP 200| Situation | PeerAuthentication mode | Result |
|---|---|---|
| Client outside the mesh | PERMISSIVE (default) | HTTP 200 |
| Client outside the mesh | STRICT | failure, curl exits with 56 |
| Client inside the mesh | STRICT | HTTP 200 |
curl code 56 means "failure in receiving network data": the TCP connection is established, then the proxy cuts it for lack of a client certificate. It
is therefore not an HTTP refusal, which is why no status code
appears. A client expecting a 403 will look for a long time.
Switching between the two middle rows takes a single object:
apiVersion: security.istio.io/v1kind: PeerAuthenticationmetadata: name: strict namespace: productionspec: mtls: mode: STRICTApply it, then wait until STRICT is active: the first loop
returns when dehors no longer gets an HTTP response (code 000), the
second when frontend gets a 200 again. This second wait
covers the removal of the AuthorizationPolicy resources, which propagates too.
The first loop hides the error output with 2>/dev/null: the expected refusal
makes curl exit with code 56, and kubectl reports it on every try. Then
repeat the call from dehors (outside the mesh) and from frontend
(inside the mesh):
kubectl apply -f peerauth-strict.yamluntil kubectl -n hors-maillage exec dehors -- curl -s -o /dev/null -w '%{http_code}' --max-time 5 http://api-server.production.svc.cluster.local:8080/ 2>/dev/null | grep -q 000; do sleep 2; doneuntil kubectl -n production exec frontend -c curl -- curl -s -o /dev/null -w '%{http_code}' --max-time 5 http://api-server.production.svc.cluster.local:8080/ | grep -q 200; do sleep 2; donekubectl -n hors-maillage exec dehors -- curl -s -o /dev/null -w 'HTTP %{http_code}\n' --max-time 5 http://api-server.production.svc.cluster.local:8080/kubectl -n production exec frontend -c curl -- curl -s -o /dev/null -w 'HTTP %{http_code}\n' --max-time 5 http://api-server.production.svc.cluster.local:8080/peerauthentication.security.istio.io/strict createdHTTP 000command terminated with exit code 56HTTP 200The client outside the mesh fails with code 56 and HTTP 000, the client in the
mesh gets its 200. To read curl's own message, add -S
(curl -sS): it prints curl: (56) Recv failure: Connection reset by peer.
Check that mTLS is active
After configuring mTLS, you must be able to verify that it really works. Istio provides several verification methods.
Check the configuration with istioctl
# See the effective mTLS policy for a workloadistioctl x describe pod <pod-name> -n <namespace>Replace <pod-name> with the name of your api-server Pod, which
kubectl get pods -n production gives you. Output, with the strict policy
set in the previous section:
Pod: api-server-7ff9b5bf75-zm9xk Pod Revision: default Pod Ports: 80 (nginx)--------------------Service: api-server Port: 8080/auto-detect targets pod port 80--------------------Effective PeerAuthentication: Workload mTLS mode: STRICTApplied PeerAuthentication: strict.productionThe output shows the PeerAuthentication policies actually present on the
workload, here strict.production, and the effective mTLS mode. If two policies
exist in the namespace, for example default and strict, the line
lists both: default.production, strict.production.
Check a proxy's certificate
# Inspect the certificate used by a Pod's Envoy proxyistioctl proxy-config secret <pod-name> -n <namespace>The output shows the active certificates, their validity and their serial number:
RESOURCE NAME TYPE STATUS VALID CERT SERIAL NUMBER NOT AFTER NOT BEFOREdefault Cert Chain ACTIVE true 75e2bd60635f15762c9aa890a51e3003 2026-11-13T19:33:30Z 2026-11-12T19:31:30ZROOTCA CA ACTIVE true 1a36f3e66de02a8fd7e74e3b9710c964 2036-11-09T19:31:15Z 2026-11-12T19:31:15ZThe default row carries the workload's mTLS certificate, SPIFFE ID included. The ROOTCA row is the Istio root certificate authority.
Look at the dates: the workload certificate lives 24 hours (plus 2 minutes of
margin on NOT BEFORE), the root ten years.
Test that plain traffic is rejected
To confirm that STRICT mode works, try to send clear-text traffic from a Pod without a proxy to a protected service:
# From a namespace without sidecar injectionkubectl create namespace no-meshkubectl run test-clear --image=curlimages/curl:8.23.0@sha256:3fb7e13e5d179a845c97d78204f3f1064d1907037f26a3ebf91710571151fd93 --rm -it --restart=Never -n no-mesh \ -- curl -s -o /dev/null -w "HTTP %{http_code}\n" --max-time 5 http://api-server.production.svc.cluster.local:8080Expected result with STRICT mTLS, the strict policy from the previous
section being in place:
namespace/no-mesh createdHTTP 000…pod "test-clear" deleted from no-mesh namespacepod no-mesh/test-clear terminated (Error)The real output also carries information lines from kubectl (All commands and output from this session will be recorded…, If you don't see a command prompt…, a couldn't attach to pod warning), and HTTP 000 shows up
twice: this comes from kubectl, not from mTLS. The point to read is the return
code, 56 (echo $? right after), which kubectl takes over from curl.
HTTP code 000 and curl exit code 56 (connection reset by peer) confirm that the server-side proxy rejects all non-mTLS traffic. The TCP connection is established but the TLS handshake fails.
Check the metrics
Istio metrics are read in Prometheus. The demo profile does not install it,
and the command fails as long as the addon is missing:
istioctl dashboard prometheusError: no pods found with selector app.kubernetes.io/name=prometheusThe return code is 1. The Istio archive downloaded above contains an addon
manifest, samples/addons/prometheus.yaml, in the istio-1.32.0 directory.
Apply it with kubectl apply -f samples/addons/prometheus.yaml,
wait until the Prometheus Pod is Running in istio-system, then
run istioctl dashboard prometheus again: the command opens the interface in
your browser.
The istio_tcp_connections_opened_total metric with the label connection_security_policy="mutual_tls" confirms that the connections are encrypted.
Gradual migration: PERMISSIVE to STRICT
Enabling STRICT mTLS all at once on an existing cluster breaks the services that do not have a proxy yet. The migration is done in three steps, and each one is verified before moving to the next.
-
Check proxy injection: make sure all application namespaces have sidecar injection enabled and that all Pods have restarted with their proxy.
Be careful with the command you use. Since Istio 1.27, the proxy is injected as a native sidecar: it lives in
spec.initContainerswithrestartPolicy: Always, and no longer inspec.containers. A check that readscontainerStatusestherefore does not see it and leads you to believe injection failed.Terminal window # ❌ no longer sees the proxy since Istio 1.27kubectl get pods -n production -o jsonpath='{.items[*].spec.containers[*].name}'Output nginx curl curlNo
istio-proxycontainer in this list, even though the Pods have one.Terminal window # ✅ the label set by the injector, independent of where the container liveskubectl get pods -n production -l security.istio.io/tlsMode=istioOutput NAME READY STATUS RESTARTS AGEapi-server-7ff9b5bf75-zm9xk 2/2 Running 0 58sfrontend 2/2 Running 0 58sintrus 2/2 Running 0 58skubectl get podsstill shows 2/2: native sidecars count in theREADYcolumn. Only the field-based inspection changes address. -
Enable PERMISSIVE mode (default): this mode accepts mTLS traffic and clear text. Services with a proxy already talk to each other over mTLS, and services without a proxy keep working.
Terminal window istioctl x describe pod <pod-name> -n productionCheck that connections between Pods with a proxy do use mTLS. If the
strictpolicy from the previous section still exists,x describealready showsWorkload mTLS mode: STRICT: this is expected, it enforces the mode. -
Switch to STRICT mode: once all Pods have a proxy, switch to STRICT. Watch the connection errors in the logs.
Terminal window kubectl apply -f - <<EOFapiVersion: security.istio.io/v1kind: PeerAuthenticationmetadata:name: defaultnamespace: productionspec:mtls:mode: STRICTEOFOutput peerauthentication.security.istio.io/default createdIf
strictstill exists,istioctl x describenow listsdefault.production, strict.production: both policies apply.
Troubleshooting
mTLS failures share a common signature: traffic used to go through, and no longer does, and the application does not say why since the refusal happens in the proxy, not in your code. The reflex is therefore to look at sidecar injection and the identity presented, never at the application.
Read the table starting from the exact symptom, error message included: two very different causes produce connection errors that look alike, and it is the precise wording that tells them apart.
| Symptom | Likely cause | Solution |
|---|---|---|
connection reset by peer from a Pod without a proxy | The target service is in STRICT mTLS and rejects clear-text traffic | Inject the Istio sidecar into the source Pod, or temporarily switch to PERMISSIVE |
upstream connect error or disconnect | The proxy has not received its certificate yet | Check the istiod logs: kubectl logs -n istio-system deploy/istiod | grep -i error |
| AuthorizationPolicy refuses legitimate requests | The SPIFFE principal in the rule does not match the actual ServiceAccount | Check the identity: istioctl proxy-config secret <pod> -n <namespace> and compare with the principals field |
Metrics do not show mutual_tls | Sidecar injection is not active on the namespace | Check the label: kubectl get ns <ns> --show-labels | grep istio |
| The policy you just applied has no effect yet: the call gives the old answer | istiod pushes the policy to the proxies within a few seconds (2 to 4 s measured, sometimes more than 10 s) | Retry until the expected answer with an until … sleep 2 loop, as in the sections above, before looking for an error in the policy |
| High latency after enabling mTLS | The TLS handshake adds latency on the first requests | Normal for short connections. Persistent connections (keep-alive) absorb the cost. Consider ambient mode (ztunnel) for a lighter footprint |
The istioctl and kubectl inspection commands look for the Pod in the
default namespace when you do not give -n. Without it, a Pod in production
is not found:
Error: failed to execute command on api-server-7ff9b5bf75-zm9xk.default sidecar: … pods "api-server-7ff9b5bf75-zm9xk" not found in the "default" namespaceSo always give -n <namespace>. Another point that worries people for no reason: right
after installation, kubectl logs -n istio-system deploy/istiod | grep -i error
returns a few webhook is not ready, retry lines (four during the test).
They are normal at istiod startup, while its validation webhook gets
ready, and do not signal a failure:
… error controllers error handling istio-validator-istio-system, retrying (retry count: 1): webhook is not ready, retry controller=validationKey takeaways
- Traffic between Pods is in clear text by default in Kubernetes, mTLS is essential to encrypt and authenticate it.
- mTLS verifies the identity of both ends through X.509 certificates, unlike classic TLS which only verifies the server.
- Identity is based on the SPIFFE ID (
spiffe://cluster.local/ns/<namespace>/sa/<service-account>), tied to the Kubernetes ServiceAccount. - PeerAuthentication controls the mTLS mode (PERMISSIVE, STRICT, DISABLE). STRICT mode is recommended in production.
- AuthorizationPolicy with SPIFFE principals allows granular access control based on the cryptographic identity of services.
- mTLS and NetworkPolicies are complementary: one encrypts and authenticates, the other filters network flows.
- Certificate rotation is automatic with a service mesh, no manual intervention needed.
- For the CKS, the competency reads Implement Pod-to-Pod encryption (Cilium, Istio): know how to enable and verify both.
- On the Cilium side, keep the chain in mind:
encryption.type=wireguard→ agent restart →cilium-dbg status→cilium_wg0counter. - On the Istio side, master the chain: proxy injection → PeerAuthentication STRICT → AuthorizationPolicy → SPIFFE principals.
Put it into practice
mTLS is proven by what is refused, not by what answers. This lab has you require mutual TLS in a mesh that still accepted clear-text traffic, then checks both sides: a Pod with its sidecar still reaches the service, a Pod without one no longer does.
Knowledge check
Seven questions on encrypting traffic between Pods: SPIFFE identity, PeerAuthentication, AuthorizationPolicy and diagnosing the handshake.
Knowledge check
Check what you have learned with this interactive quiz
How it works
- The timer starts when you click Start
- Multiple choice, true/false and short answer questions
- You can move between questions
- Detailed results are shown at the end
Starts the quiz and the timer
Review
(0/0)Score by domain
A domain groups several guides and gets several questions, so its score is more reliable than a per-topic score, which often rests on a single question.
Summary
Strengths
No topic passed at 80% on at least two questions yet.
Weak areas
No topic below the pass mark.
Recommended next
Hints before you try again?
Continue the course: Kubernetes supply chain security: securing the delivery chain
A new full quiz with random questions
Work again only on the questions you missed