Skip to content
Français
Account: not signed in, sync available My learning
Select theme
Français

Pod-to-pod mTLS: encrypting traffic between services

By updated

Improve this page

By default, traffic between Kubernetes Pods travels in clear text. Any Pod in the cluster can intercept another Pod's communications: all it takes is network access. Two families of tools answer this problem, and the CKS syllabus names both: Cilium's transparent encryption, which works inside the network without the application knowing, and Istio's mTLS, which gives every service a verifiable identity. This guide shows you both, how to enable them and, above all, how to prove that the traffic is really encrypted.

Pod-to-pod encryption belongs to the Minimize Microservice Vulnerabilities domain of the CKS, which weighs 20% of the exam according to the official CNCF curriculum. The competency is worded there as: Implement Pod-to-Pod encryption (Cilium, Istio). Both names appear, and they do not do the same job.

Cilium with WireGuardIstio with mTLS
What encryptsthe CNI datapath, in the kernelone proxy per Pod
What the application seesnothing, no container addeda sidecar, or a node proxy in ambient mode
Scopeall traffic between nodestraffic between Pods in the mesh
Service identityno, encryption is at node levelyes, SPIFFE, usable in access policies
Identity-based access controlno, that stays with NetworkPoliciesyes, AuthorizationPolicy
Activationone option of the charta mesh to install and inject

Keep the line that decides: if the need is "nothing should travel in clear text anymore", Cilium is enough and costs one option. If the need is "service A may call B but not C", you need a service identity, so Istio.

What you will learn

  • Why clear-text traffic between Pods is a security risk
  • How mTLS works (SPIFFE identity, X.509 certificates, handshake)
  • Configure Istio to enforce mTLS in STRICT mode
  • Verify that encryption is active between two services
  • Control access with AuthorizationPolicy and mTLS principals
  • Enable and verify Cilium WireGuard encryption, then choose between Cilium and Istio
  • Choose between the sidecar and ambient (ztunnel) approaches

Prerequisites

  • A Kubernetes cluster with administrator access
  • Familiarity with Pods and Network Policies
  • Understanding of service mesh (in French) concepts (data plane, control plane)
  • istioctl installed (the Istio command line)

Why encrypt traffic between Pods

In a Kubernetes cluster, the network is flat by default: every Pod can talk to every other Pod without restriction. This model simplifies deployment, but it creates a major risk: if an attacker compromises a single Pod, they can intercept (sniffing), modify (man-in-the-middle) or impersonate the traffic of other services.

Network Policies solve part of the problem by filtering traffic at the L3/L4 level (IPs and ports), but they do neither encryption nor authentication. The table below shows how the two complement each other:

MechanismLayerEncryptionAuthenticationWhat it controls
NetworkPolicyL3/L4NoNoWhich Pods can talk to each other (IP/port)
mTLSL4/L7Yes (TLS 1.2+)Yes (X.509 certificates)Who talks to whom (cryptographic identity)

Combining the two gives you a defense in depth model: NetworkPolicies limit the allowed flows, and mTLS guarantees that every flow is encrypted and that both ends are who they claim to be.

How mTLS works

Classic TLS (the one behind HTTPS) is one-way: only the client verifies the server's identity. Mutual TLS (mTLS) adds a step: the server also verifies the client's identity. Both parties present an X.509 certificate.

The mTLS handshake, step by step

The five steps below describe a classic TLS exchange with one extra check: the client's certificate. Step 3 makes all the difference: in an ordinary HTTPS connection, the browser verifies the server and the server knows nothing about the client. With mTLS, each side proves its identity to the other.

Keep in mind that all of this happens before the first application request, and without any change to your code: in a service mesh, the injected proxy next to the container runs these five steps.

  1. The client starts the TLS connection: it sends a "ClientHello" with the cipher suites it supports.

  2. The server presents its certificate: the client checks that the certificate is signed by a trusted certificate authority (CA) and that the identity matches the expected service.

  3. The server asks for the client's certificate: this is the extra step compared with classic TLS. The client presents its own certificate.

  4. The server verifies the client's certificate: it confirms that the client is an authorized service, signed by the same CA.

  5. The encrypted session is established: both parties exchange a symmetric session key. All the following traffic is encrypted and authenticated.

SPIFFE identity

In a Kubernetes cluster, the identity of each service is encoded in the certificate as a SPIFFE ID (Secure Production Identity Framework For Everyone). The standard format is:

spiffe://<trust-domain>/ns/<namespace>/sa/<service-account>

For example, a service using the ServiceAccount api-server in the namespace production has the identity:

spiffe://cluster.local/ns/production/sa/api-server

This identity based on the Kubernetes ServiceAccount is at the heart of the mTLS security model. It lets you define granular access policies: "only the frontend service in the production namespace can call the api-server service".

This identity is not theoretical, you can read it in the proxy. The following command queries the proxy of the client Pod, which runs in the lab-mesh namespace with the default ServiceAccount. This Pod and the mesh are created in the "Create the demo application" section: run this command again after creating them.

Terminal window
kubectl exec client -n lab-mesh -c istio-proxy -- curl -s localhost:15000/certs | grep -o 'spiffe://[^"]*' | head -1
Output
spiffe://cluster.local/ns/lab-mesh/sa/default

The head -1 keeps only one line: the proxy repeats the same identity several times in its response (20 times during the test). The certificate therefore carries exactly the format announced above.

The proxy is a native sidecar, not an ordinary container

Since Istio 1.27, injection places the proxy as a native sidecar by default. On a Pod in a namespace labeled istio-injection=enabled, like the app Pod created further down, the application container stays alone in containers, and the proxy appears in initContainers:

Terminal window
kubectl get pod app -o jsonpath='{.spec.containers[*].name}{"\n"}'
kubectl get pod app -o jsonpath='{.spec.initContainers[*].name}{"\n"}'
Output
app
istio-init istio-proxy

The proxy is there, but in the initContainers, with the signature of the native sidecar. This command prints the restartPolicy field of each initContainer:

Terminal window
kubectl get pod app -o jsonpath='{range .spec.initContainers[*]}{.name}{" : restartPolicy="}{.restartPolicy}{"\n"}{end}'
Output
istio-init : restartPolicy=
istio-proxy : restartPolicy=Always

istio-init has no restartPolicy: it is an ordinary init container, which stops once its task is done. Only istio-proxy is a native sidecar.

It is this restartPolicy: Always on an initContainer that distinguishes a native sidecar from a simple init container: it starts before the application containers, stays alive for the whole life of the Pod, and stops after them. The guaranteed order also fixes a historical flaw of the sidecars of old, where the application started before its proxy and failed its first requests.

Istio documents this itself in the Pod's sidecar.istio.io/status annotation. This command reads it and prints the two lists of containers:

Terminal window
kubectl get pod app -o jsonpath='{.metadata.annotations.sidecar\.istio\.io/status}' | python3 -c 'import json,sys; d=json.load(sys.stdin); print("containers =", d.get("containers")); print("initContainers =", d.get("initContainers"))'
Output
containers = None
initContainers = ['istio-init', 'istio-proxy']

Automatic certificate management

Without a service mesh, you would have to manage the certificates of each service by hand: generation, distribution, rotation, revocation. With a mesh like Istio, these four operations are automatic.

StepWhat happens
GenerationThe proxy (Envoy or ztunnel) generates a private key and sends a CSR (Certificate Signing Request) to the control plane
SigningThe control plane (istiod) signs the CSR after verifying the Pod's identity through its ServiceAccount
DistributionThe signed certificate is delivered to the proxy through the SDS API (Secret Discovery Service)
RotationCertificates are renewed automatically before they expire (default lifetime: 24h)

The application code sees none of this: the proxy handles mTLS transparently.

Encrypt with Cilium, the sidecar-free way

This is the simpler of the two solutions, and the one the syllabus cites first. Cilium encrypts traffic between nodes in its own datapath, with WireGuard: no container is added to the Pods, no application manifest is touched, and the application knows nothing about what happens.

The prerequisite is to have Cilium as your CNI. Its installation and concepts are covered in the dedicated guide, Cilium, the eBPF CNI (in French).

Enable WireGuard encryption

Two chart options are enough, set at installation or through an upgrade:

Terminal window
helm upgrade --install cilium cilium/cilium --version 1.20.1 \
--namespace kube-system \
--set encryption.enabled=true \
--set encryption.type=wireguard

Measured in a lab on Cilium 1.20.1 and Kubernetes 1.37: once the installation is finished, the nodes become Ready right away.

Check that encryption is active

This is the step that proves encryption is active, and it does not consist of re-reading the option:

Terminal window
kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status | grep -i encryption
Output
Encryption: Wireguard [NodeEncryption: Disabled,
cilium_wg0 (Pubkey: v38E2wsPmrYwEnN5Y9rRs/..., Port: 51871, Peers: 1)]

Three useful pieces of information in this line. Encryption is indeed WireGuard, it goes through a cilium_wg0 interface on port 51871/UDP, and the number of peers must equal the number of other nodes in the cluster. Peers: 0 on a multi-node cluster means the nodes did not pair up.

NodeEncryption: Disabled is not an anomaly, it is the default. Traffic between Pods is encrypted; traffic sent by the nodes themselves is encrypted only if you enable encryption.nodeEncryption=true, which is a separate and more intrusive setting.

The proof that counts: watch the tunnel

An enabled option and a green status do not prove that a single byte went through the tunnel. The interface counter does. Two series of 40 HTTP requests from the same client Pod, one to a Pod on the same node, the other to a Pod on a different node:

Terminal window
# On the node, before and after each series
cat /sys/class/net/cilium_wg0/statistics/tx_bytes
DestinationBytes sent out by cilium_wg0
Pod on the same node0
Pod on another node42,880

The result says exactly what WireGuard does here: it encrypts what leaves the machine. Two co-located Pods communicate without ever touching the tunnel, which is logical: their traffic goes over no wire.

The negative control completes the demonstration. After --set encryption.enabled=false and a restart of the agents, cilium-dbg status shows Encryption: Disabled, the interface is gone (Device "cilium_wg0" does not exist), and traffic between the same two Pods still goes through, with HTTP 200. Encryption was never a condition for things to work, only a property of the transport.

What Cilium does not give you

Transparent encryption protects confidentiality on the wire. It gives no service identity: for Cilium, a Pod is a security identity derived from its labels, which you filter with NetworkPolicies. If you need to answer "is service A allowed to call B", you need the other approach, and that is the subject of the Istio section below.

Enable mTLS with Istio

Istio is a service mesh that the CKS syllabus names explicitly. It offers two deployment models: the classic sidecar mode and the ambient mode (no sidecar). In both cases, mTLS is configured with the same Kubernetes resources. To go from mTLS to timeouts, retries and authorizations between services, the guide Enforce timeouts, retries and authorizations between services (in French) takes over.

Install Istio

This section installs Istio so that you can practice, but it only serves to get a mesh in place. You can therefore skim it if you already have one, and come back when you build your own environment. The Istio documentation covers the options this guide does not detail.

The outputs on this page were captured with Istio 1.32.0 on a kind cluster (kind v0.33.0, image kindest/node:v1.37.0, a single node), created with kind create cluster --name demo-istio --image kindest/node:v1.37.0 --wait 120s. Pod names carry a random suffix, and durations vary on your side.

For a lab cluster, the minimal installation with the demo profile:

Terminal window
# Download istioctl
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.32.0 sh -
cd istio-1.32.0
export PATH=$PWD/bin:$PATH
# Install with the demo profile (includes ingress + egress gateways)
istioctl install --set profile=demo -y

Check:

Terminal window
kubectl get pods -n istio-system

Expected result, istiod and the gateways must be Running:

Output
NAME READY STATUS RESTARTS AGE
istio-egressgateway-84b6d9fcb9-fmrnb 1/1 Running 0 12s
istio-ingressgateway-59859d889c-8plqj 1/1 Running 0 12s
istiod-5b4f5c4699-8ffhk 1/1 Running 0 26s

If a Pod is not ready yet, this command waits until they all are:

Terminal window
kubectl -n istio-system wait --for=condition=Ready pod --all --timeout=300s
Output
pod/istio-egressgateway-84b6d9fcb9-fmrnb condition met
pod/istio-ingressgateway-59859d889c-8plqj condition met
pod/istiod-5b4f5c4699-8ffhk condition met

Inject the sidecar proxies

In sidecar mode, Istio injects an Envoy proxy into every Pod. To enable automatic injection on a namespace:

Terminal window
kubectl label namespace default istio-injection=enabled
Output
namespace/default labeled

Every Pod created in this namespace now receives the Envoy proxy in addition to your application, and kubectl get pods shows 2/2 in the READY column.

Create the demo application

The following sections need a few workloads to observe. This demo application comes in three blocks: an app Pod in default, a client Pod in a lab-mesh namespace, then a small application in production with a client outside the mesh in hors-maillage. The images are pinned by digest.

The first block creates the app Pod, waits for it to be ready and displays it. The READY column must show 2/2: the nginx container and the proxy, which is counted even though it lives in the initContainers.

Terminal window
kubectl run app --image=nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb --port=80
kubectl wait --for=condition=Ready pod/app --timeout=180s
kubectl get pods
Output
pod/app created
pod/app condition met
NAME READY STATUS RESTARTS AGE
app 2/2 Running 0 12s

The second block creates the lab-mesh namespace, with injection, and a client Pod that sleeps for an hour, long enough to query it:

Terminal window
kubectl create namespace lab-mesh
kubectl label namespace lab-mesh istio-injection=enabled
kubectl -n lab-mesh run client --image=curlimages/curl:8.23.0@sha256:3fb7e13e5d179a845c97d78204f3f1064d1907037f26a3ebf91710571151fd93 --command -- sleep 3600
kubectl -n lab-mesh wait --for=condition=Ready pod/client --timeout=180s
Output
namespace/lab-mesh created
namespace/lab-mesh labeled
pod/client created
pod/client condition met

The third block is a manifest. It creates the production namespace with injection, an api-server service (nginx behind a Service on port 8080), a frontend Pod that runs with its own ServiceAccount, an intrus Pod that keeps the default ServiceAccount, and a hors-maillage namespace whose dehors Pod has no proxy.

application.yaml
apiVersion: v1
kind: Namespace
metadata:
name: production
labels:
istio-injection: enabled
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: frontend
namespace: production
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-server
namespace: production
spec:
replicas: 1
selector:
matchLabels:
app: api-server
template:
metadata:
labels:
app: api-server
spec:
containers:
- name: nginx
image: nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: api-server
namespace: production
spec:
selector:
app: api-server
ports:
- port: 8080
targetPort: 80
---
apiVersion: v1
kind: Pod
metadata:
name: frontend
namespace: production
spec:
serviceAccountName: frontend
containers:
- name: curl
image: curlimages/curl:8.23.0@sha256:3fb7e13e5d179a845c97d78204f3f1064d1907037f26a3ebf91710571151fd93
command: ["sleep", "3600"]
---
apiVersion: v1
kind: Pod
metadata:
name: intrus
namespace: production
spec:
containers:
- name: curl
image: curlimages/curl:8.23.0@sha256:3fb7e13e5d179a845c97d78204f3f1064d1907037f26a3ebf91710571151fd93
command: ["sleep", "3600"]
---
apiVersion: v1
kind: Namespace
metadata:
name: hors-maillage
---
apiVersion: v1
kind: Pod
metadata:
name: dehors
namespace: hors-maillage
spec:
containers:
- name: curl
image: curlimages/curl:8.23.0@sha256:3fb7e13e5d179a845c97d78204f3f1064d1907037f26a3ebf91710571151fd93
command: ["sleep", "3600"]

Apply it, wait for the Pods, then check that the three Pods in production are at 2/2 and that dehors is at 1/1: it has no proxy, by design.

Terminal window
kubectl apply -f application.yaml
kubectl -n production wait --for=condition=Ready pod --all --timeout=180s
kubectl -n hors-maillage wait --for=condition=Ready pod/dehors --timeout=180s
kubectl get pods -n production
kubectl get pods -n hors-maillage
Output
namespace/production created
serviceaccount/frontend created
deployment.apps/api-server created
service/api-server created
pod/frontend created
pod/intrus created
namespace/hors-maillage created
pod/dehors created
pod/api-server-7ff9b5bf75-zm9xk condition met
pod/frontend condition met
pod/intrus condition met
pod/dehors condition met
NAME READY STATUS RESTARTS AGE
api-server-7ff9b5bf75-zm9xk 2/2 Running 0 58s
frontend 2/2 Running 0 58s
intrus 2/2 Running 0 58s
NAME READY STATUS RESTARTS AGE
dehors 1/1 Running 0 4s

Ambient mode, without sidecars

Since Istio 1.24, a second, stable architecture coexists with sidecars and changes the trade-off. Ambient mode replaces the proxy injected into each Pod with a ztunnel, deployed once per node as a DaemonSet. Encryption and L4 authentication are provided without modifying any application Pod, which removes injection, workload restarts and the cost of an extra container per Pod.

It is enabled with a label on the namespace:

Terminal window
kubectl label namespace default istio.io/dataplane-mode=ambient
Output
namespace/default labeled

Be careful: this label alone changes nothing on the installation done above. The demo profile installs neither the ztunnel nor the istio-cni plugin that ambient mode requires: the label is applied without error, and the Pods keep their sidecar. This command confirms it, it finds no ambient Pod:

Terminal window
kubectl get pods -A | grep -i -E "ztunnel|istio-cni"

It prints nothing and exits with code 1, the code grep returns when it finds no line: this is the expected answer on the demo profile.

To try ambient, you must install Istio with the ambient profile (istioctl install --set profile=ambient) instead of the demo profile. To keep the sidecar path of this page, remove the label:

Terminal window
kubectl label namespace default istio.io/dataplane-mode-
Output
namespace/default unlabeled

The trade-off is worth knowing before you choose: the ztunnel only covers the L4 level. As soon as you need L7 features, HTTP routing or AuthorizationPolicy on URL paths, you must add a waypoint proxy per namespace, and the simplicity gain shrinks accordingly.

PeerAuthentication: enforce mTLS

The PeerAuthentication resource controls the mTLS mode for a namespace or a specific workload. It is the resource that enforces mTLS in Istio.

Three modes are available:

ModeBehaviorUsage
PERMISSIVEAccepts mTLS and clear-text trafficGradual migration (Istio default)
STRICTAccepts only mTLS trafficSecure production
DISABLEDisables mTLSNot recommended except in very specific cases

STRICT mTLS across the whole mesh

To enforce encryption on all namespaces of the cluster:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
name: default
namespace: istio-system # root namespace = applies to the whole mesh
spec:
mtls:
mode: STRICT
Terminal window
kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
name: default
namespace: istio-system
spec:
mtls:
mode: STRICT
EOF
Output
peerauthentication.security.istio.io/default created

The list confirms that the mesh policy is STRICT:

Terminal window
kubectl get peerauthentication -A
Output
NAMESPACE NAME MODE AGE
istio-system default STRICT 1s

STRICT mTLS on a namespace

To apply STRICT mode only to the production namespace:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
name: default
namespace: production
spec:
mtls:
mode: STRICT

Save this manifest as pa-namespace.yaml, then apply it:

Terminal window
kubectl apply -f pa-namespace.yaml
Output
peerauthentication.security.istio.io/default created

mTLS on a specific workload

To target a precise workload with a selector:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
name: api-strict
namespace: production
spec:
selector:
matchLabels:
app: api-server
mtls:
mode: STRICT

Save it as pa-workload.yaml and apply it the same way (kubectl apply -f pa-workload.yaml answers peerauthentication.security.istio.io/api-strict created). With the three policies in place, the list shows them all:

Terminal window
kubectl get peerauthentication -A
Output
NAMESPACE NAME MODE AGE
istio-system default STRICT 56s
production api-strict STRICT 0s
production default STRICT 0s

AuthorizationPolicy: identity-based access control

STRICT mTLS guarantees that all traffic is encrypted and authenticated. But it does not control who can call what. That is the role of AuthorizationPolicy.

Thanks to mTLS, Istio knows the SPIFFE identity of every request. You can use that information to define granular access rules.

Allow only certain services

The following example allows only the frontend service (through its ServiceAccount) to access the api-server service:

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
name: api-server-access
namespace: production
spec:
selector:
matchLabels:
app: api-server
action: ALLOW
rules:
- from:
- source:
principals:
- "cluster.local/ns/production/sa/frontend"

The source.principals field lists identities derived from the client's certificate, in the format <trust-domain>/ns/<namespace>/sa/<service-account>. Istio's documentation states that this field "requires mTLS enabled": a request received in clear text carries no identity, so a principals rule cannot allow it.

Save the policy as az-access.yaml, apply it, wait until it is active, then call api-server from frontend (the allowed principal) and from intrus (another ServiceAccount).

The wait is needed for every policy on this page: istiod pushes the policy to the proxies within a few seconds, and the effect is not immediate. A curl run right after kubectl apply can therefore still get the old answer. The loop below retries every 2 seconds and returns when intrus gets the expected 403:

Terminal window
kubectl apply -f az-access.yaml
until kubectl -n production exec intrus -c curl -- curl -s -o /dev/null -w '%{http_code}' --max-time 5 http://api-server.production.svc.cluster.local:8080/ | grep -q 403; do sleep 2; done
kubectl -n production exec frontend -c curl -- curl -s -o /dev/null -w 'HTTP %{http_code}\n' --max-time 5 http://api-server.production.svc.cluster.local:8080/
kubectl -n production exec intrus -c curl -- curl -s -w '\nHTTP %{http_code}\n' --max-time 5 http://api-server.production.svc.cluster.local:8080/
Output
authorizationpolicy.security.istio.io/api-server-access created
HTTP 200
RBAC: access denied
HTTP 403

frontend gets through, intrus receives an HTTP 403: the refusal comes from the identity, not from the address.

Deny-by-default policy

For a namespace in "zero trust" mode, start by denying all traffic, then explicitly open the allowed flows. First remove the policy from the previous example, so that the measurements do not get mixed up:

Terminal window
kubectl -n production delete authorizationpolicy api-server-access
Output
authorizationpolicy.security.istio.io "api-server-access" deleted from production namespace
# Step 1: deny everything by default
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
name: deny-all
namespace: production
spec:
action: ALLOW
# No rules → never matches → everything is denied

Save the policy as az-deny-all.yaml and apply it (kubectl apply -f az-deny-all.yaml answers authorizationpolicy.security.istio.io/deny-all created). The proxies receive it after a few seconds: wait until frontend gets the 403, the loop then returns. A service that tries to call api-server then gets an HTTP 403 error, even frontend:

Terminal window
until kubectl -n production exec frontend -c curl -- curl -s -o /dev/null -w '%{http_code}' --max-time 5 http://api-server.production.svc.cluster.local:8080/ | grep -q 403; do sleep 2; done
kubectl -n production exec frontend -c curl -- curl -s -w '\nHTTP %{http_code}\n' --max-time 5 http://api-server.production.svc.cluster.local:8080/
Output
RBAC: access denied
HTTP 403
# Step 2: explicitly allow the frontend → api-server flow
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
name: allow-frontend-to-api
namespace: production
spec:
selector:
matchLabels:
app: api-server
action: ALLOW
rules:
- from:
- source:
principals:
- "cluster.local/ns/production/sa/frontend"
to:
- operation:
methods: ["GET", "POST"]
paths: ["/api/*"]

Save this second policy as az-allow.yaml, then apply it with kubectl apply -f az-allow.yaml (answer authorizationpolicy.security.istio.io/allow-frontend-to-api created). The loop waits until frontend gets the 404 on /api/x, a sign that the authorization has reached the proxy. Three calls then show what is open: frontend on /api/x, frontend on /, then intrus on /api/x.

Terminal window
until kubectl -n production exec frontend -c curl -- curl -s -o /dev/null -w '%{http_code}' --max-time 5 http://api-server.production.svc.cluster.local:8080/api/x | grep -q 404; do sleep 2; done
kubectl -n production exec frontend -c curl -- curl -s -o /dev/null -w 'HTTP %{http_code}\n' --max-time 5 http://api-server.production.svc.cluster.local:8080/api/x
kubectl -n production exec frontend -c curl -- curl -s -w '\nHTTP %{http_code}\n' --max-time 5 http://api-server.production.svc.cluster.local:8080/
kubectl -n production exec intrus -c curl -- curl -s -w '\nHTTP %{http_code}\n' --max-time 5 http://api-server.production.svc.cluster.local:8080/api/x
Output
HTTP 404
RBAC: access denied
HTTP 403
RBAC: access denied
HTTP 403

The 404 is good news: the request got past authorization, and it is nginx that has no /api/x page. Outside the /api/* path, or with another principal, the answer stays a 403.

A habit to keep in production: apply the principle of least privilege in this exact order. First set a policy that denies everything, then explicitly allow each necessary flow by naming the callers by their SPIFFE principal. Keep the chain in mind: a PeerAuthentication in STRICT guarantees that the identity is proven, and an AuthorizationPolicy then decides what that identity is allowed to do.

Switching to STRICT, put to the test

This is the central claim of this page, and it is proven with three measurements on the same cluster. One server in a namespace of the mesh, one client outside the mesh, one client inside the mesh.

To make the PERMISSIVE measurement possible, you must first remove what the previous sections set up: the STRICT of the whole mesh and the one on production, which would already refuse the client outside the mesh, and the AuthorizationPolicy resources, which would skew the answers.

Terminal window
kubectl -n production delete authorizationpolicy deny-all allow-frontend-to-api
kubectl delete peerauthentication default -n istio-system
kubectl delete peerauthentication default api-strict -n production
kubectl get peerauthentication -A
Output
authorizationpolicy.security.istio.io "deny-all" deleted from production namespace
authorizationpolicy.security.istio.io "allow-frontend-to-api" deleted from production namespace
peerauthentication.security.istio.io "default" deleted from istio-system namespace
peerauthentication.security.istio.io "default" deleted from production namespace
peerauthentication.security.istio.io "api-strict" deleted from production namespace
No resources found

With no PeerAuthentication at all, the mode is PERMISSIVE (the default). Deletions propagate to the proxies just like creations: the loop waits until dehors gets a 200 again. First measurement, from the dehors Pod, which has no proxy:

Terminal window
until kubectl -n hors-maillage exec dehors -- curl -s -o /dev/null -w '%{http_code}' --max-time 5 http://api-server.production.svc.cluster.local:8080/ | grep -q 200; do sleep 2; done
kubectl -n hors-maillage exec dehors -- curl -s -o /dev/null -w 'HTTP %{http_code}\n' --max-time 5 http://api-server.production.svc.cluster.local:8080/
Output
HTTP 200
SituationPeerAuthentication modeResult
Client outside the meshPERMISSIVE (default)HTTP 200
Client outside the meshSTRICTfailure, curl exits with 56
Client inside the meshSTRICTHTTP 200

curl code 56 means "failure in receiving network data": the TCP connection is established, then the proxy cuts it for lack of a client certificate. It is therefore not an HTTP refusal, which is why no status code appears. A client expecting a 403 will look for a long time.

Switching between the two middle rows takes a single object:

peerauth-strict.yaml
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
name: strict
namespace: production
spec:
mtls:
mode: STRICT

Apply it, then wait until STRICT is active: the first loop returns when dehors no longer gets an HTTP response (code 000), the second when frontend gets a 200 again. This second wait covers the removal of the AuthorizationPolicy resources, which propagates too. The first loop hides the error output with 2>/dev/null: the expected refusal makes curl exit with code 56, and kubectl reports it on every try. Then repeat the call from dehors (outside the mesh) and from frontend (inside the mesh):

Terminal window
kubectl apply -f peerauth-strict.yaml
until kubectl -n hors-maillage exec dehors -- curl -s -o /dev/null -w '%{http_code}' --max-time 5 http://api-server.production.svc.cluster.local:8080/ 2>/dev/null | grep -q 000; do sleep 2; done
until kubectl -n production exec frontend -c curl -- curl -s -o /dev/null -w '%{http_code}' --max-time 5 http://api-server.production.svc.cluster.local:8080/ | grep -q 200; do sleep 2; done
kubectl -n hors-maillage exec dehors -- curl -s -o /dev/null -w 'HTTP %{http_code}\n' --max-time 5 http://api-server.production.svc.cluster.local:8080/
kubectl -n production exec frontend -c curl -- curl -s -o /dev/null -w 'HTTP %{http_code}\n' --max-time 5 http://api-server.production.svc.cluster.local:8080/
Output
peerauthentication.security.istio.io/strict created
HTTP 000
command terminated with exit code 56
HTTP 200

The client outside the mesh fails with code 56 and HTTP 000, the client in the mesh gets its 200. To read curl's own message, add -S (curl -sS): it prints curl: (56) Recv failure: Connection reset by peer.

Check that mTLS is active

After configuring mTLS, you must be able to verify that it really works. Istio provides several verification methods.

Check the configuration with istioctl

Terminal window
# See the effective mTLS policy for a workload
istioctl x describe pod <pod-name> -n <namespace>

Replace <pod-name> with the name of your api-server Pod, which kubectl get pods -n production gives you. Output, with the strict policy set in the previous section:

Output
Pod: api-server-7ff9b5bf75-zm9xk
Pod Revision: default
Pod Ports: 80 (nginx)
--------------------
Service: api-server
Port: 8080/auto-detect targets pod port 80
--------------------
Effective PeerAuthentication:
Workload mTLS mode: STRICT
Applied PeerAuthentication:
strict.production

The output shows the PeerAuthentication policies actually present on the workload, here strict.production, and the effective mTLS mode. If two policies exist in the namespace, for example default and strict, the line lists both: default.production, strict.production.

Check a proxy's certificate

Terminal window
# Inspect the certificate used by a Pod's Envoy proxy
istioctl proxy-config secret <pod-name> -n <namespace>

The output shows the active certificates, their validity and their serial number:

Output
RESOURCE NAME TYPE STATUS VALID CERT SERIAL NUMBER NOT AFTER NOT BEFORE
default Cert Chain ACTIVE true 75e2bd60635f15762c9aa890a51e3003 2026-11-13T19:33:30Z 2026-11-12T19:31:30Z
ROOTCA CA ACTIVE true 1a36f3e66de02a8fd7e74e3b9710c964 2036-11-09T19:31:15Z 2026-11-12T19:31:15Z

The default row carries the workload's mTLS certificate, SPIFFE ID included. The ROOTCA row is the Istio root certificate authority. Look at the dates: the workload certificate lives 24 hours (plus 2 minutes of margin on NOT BEFORE), the root ten years.

Test that plain traffic is rejected

To confirm that STRICT mode works, try to send clear-text traffic from a Pod without a proxy to a protected service:

Terminal window
# From a namespace without sidecar injection
kubectl create namespace no-mesh
kubectl run test-clear --image=curlimages/curl:8.23.0@sha256:3fb7e13e5d179a845c97d78204f3f1064d1907037f26a3ebf91710571151fd93 --rm -it --restart=Never -n no-mesh \
-- curl -s -o /dev/null -w "HTTP %{http_code}\n" --max-time 5 http://api-server.production.svc.cluster.local:8080

Expected result with STRICT mTLS, the strict policy from the previous section being in place:

Output
namespace/no-mesh created
HTTP 000
…
pod "test-clear" deleted from no-mesh namespace
pod no-mesh/test-clear terminated (Error)

The real output also carries information lines from kubectl (All commands and output from this session will be recorded…, If you don't see a command prompt…, a couldn't attach to pod warning), and HTTP 000 shows up twice: this comes from kubectl, not from mTLS. The point to read is the return code, 56 (echo $? right after), which kubectl takes over from curl.

HTTP code 000 and curl exit code 56 (connection reset by peer) confirm that the server-side proxy rejects all non-mTLS traffic. The TCP connection is established but the TLS handshake fails.

Check the metrics

Istio metrics are read in Prometheus. The demo profile does not install it, and the command fails as long as the addon is missing:

Terminal window
istioctl dashboard prometheus
Output
Error: no pods found with selector app.kubernetes.io/name=prometheus

The return code is 1. The Istio archive downloaded above contains an addon manifest, samples/addons/prometheus.yaml, in the istio-1.32.0 directory. Apply it with kubectl apply -f samples/addons/prometheus.yaml, wait until the Prometheus Pod is Running in istio-system, then run istioctl dashboard prometheus again: the command opens the interface in your browser.

The istio_tcp_connections_opened_total metric with the label connection_security_policy="mutual_tls" confirms that the connections are encrypted.

Gradual migration: PERMISSIVE to STRICT

Enabling STRICT mTLS all at once on an existing cluster breaks the services that do not have a proxy yet. The migration is done in three steps, and each one is verified before moving to the next.

  1. Check proxy injection: make sure all application namespaces have sidecar injection enabled and that all Pods have restarted with their proxy.

    Be careful with the command you use. Since Istio 1.27, the proxy is injected as a native sidecar: it lives in spec.initContainers with restartPolicy: Always, and no longer in spec.containers. A check that reads containerStatuses therefore does not see it and leads you to believe injection failed.

    Terminal window
    # ❌ no longer sees the proxy since Istio 1.27
    kubectl get pods -n production -o jsonpath='{.items[*].spec.containers[*].name}'
    Output
    nginx curl curl

    No istio-proxy container in this list, even though the Pods have one.

    Terminal window
    # ✅ the label set by the injector, independent of where the container lives
    kubectl get pods -n production -l security.istio.io/tlsMode=istio
    Output
    NAME READY STATUS RESTARTS AGE
    api-server-7ff9b5bf75-zm9xk 2/2 Running 0 58s
    frontend 2/2 Running 0 58s
    intrus 2/2 Running 0 58s

    kubectl get pods still shows 2/2: native sidecars count in the READY column. Only the field-based inspection changes address.

  2. Enable PERMISSIVE mode (default): this mode accepts mTLS traffic and clear text. Services with a proxy already talk to each other over mTLS, and services without a proxy keep working.

    Terminal window
    istioctl x describe pod <pod-name> -n production

    Check that connections between Pods with a proxy do use mTLS. If the strict policy from the previous section still exists, x describe already shows Workload mTLS mode: STRICT: this is expected, it enforces the mode.

  3. Switch to STRICT mode: once all Pods have a proxy, switch to STRICT. Watch the connection errors in the logs.

    Terminal window
    kubectl apply -f - <<EOF
    apiVersion: security.istio.io/v1
    kind: PeerAuthentication
    metadata:
    name: default
    namespace: production
    spec:
    mtls:
    mode: STRICT
    EOF
    Output
    peerauthentication.security.istio.io/default created

    If strict still exists, istioctl x describe now lists default.production, strict.production: both policies apply.

Troubleshooting

mTLS failures share a common signature: traffic used to go through, and no longer does, and the application does not say why since the refusal happens in the proxy, not in your code. The reflex is therefore to look at sidecar injection and the identity presented, never at the application.

Read the table starting from the exact symptom, error message included: two very different causes produce connection errors that look alike, and it is the precise wording that tells them apart.

SymptomLikely causeSolution
connection reset by peer from a Pod without a proxyThe target service is in STRICT mTLS and rejects clear-text trafficInject the Istio sidecar into the source Pod, or temporarily switch to PERMISSIVE
upstream connect error or disconnectThe proxy has not received its certificate yetCheck the istiod logs: kubectl logs -n istio-system deploy/istiod | grep -i error
AuthorizationPolicy refuses legitimate requestsThe SPIFFE principal in the rule does not match the actual ServiceAccountCheck the identity: istioctl proxy-config secret <pod> -n <namespace> and compare with the principals field
Metrics do not show mutual_tlsSidecar injection is not active on the namespaceCheck the label: kubectl get ns <ns> --show-labels | grep istio
The policy you just applied has no effect yet: the call gives the old answeristiod pushes the policy to the proxies within a few seconds (2 to 4 s measured, sometimes more than 10 s)Retry until the expected answer with an until … sleep 2 loop, as in the sections above, before looking for an error in the policy
High latency after enabling mTLSThe TLS handshake adds latency on the first requestsNormal for short connections. Persistent connections (keep-alive) absorb the cost. Consider ambient mode (ztunnel) for a lighter footprint

The istioctl and kubectl inspection commands look for the Pod in the default namespace when you do not give -n. Without it, a Pod in production is not found:

Output
Error: failed to execute command on api-server-7ff9b5bf75-zm9xk.default sidecar: … pods "api-server-7ff9b5bf75-zm9xk" not found in the "default" namespace

So always give -n <namespace>. Another point that worries people for no reason: right after installation, kubectl logs -n istio-system deploy/istiod | grep -i error returns a few webhook is not ready, retry lines (four during the test). They are normal at istiod startup, while its validation webhook gets ready, and do not signal a failure:

Output
… error controllers error handling istio-validator-istio-system, retrying (retry count: 1): webhook is not ready, retry controller=validation

Key takeaways

  • Traffic between Pods is in clear text by default in Kubernetes, mTLS is essential to encrypt and authenticate it.
  • mTLS verifies the identity of both ends through X.509 certificates, unlike classic TLS which only verifies the server.
  • Identity is based on the SPIFFE ID (spiffe://cluster.local/ns/<namespace>/sa/<service-account>), tied to the Kubernetes ServiceAccount.
  • PeerAuthentication controls the mTLS mode (PERMISSIVE, STRICT, DISABLE). STRICT mode is recommended in production.
  • AuthorizationPolicy with SPIFFE principals allows granular access control based on the cryptographic identity of services.
  • mTLS and NetworkPolicies are complementary: one encrypts and authenticates, the other filters network flows.
  • Certificate rotation is automatic with a service mesh, no manual intervention needed.
  • For the CKS, the competency reads Implement Pod-to-Pod encryption (Cilium, Istio): know how to enable and verify both.
  • On the Cilium side, keep the chain in mind: encryption.type=wireguard → agent restart → cilium-dbg status → cilium_wg0 counter.
  • On the Istio side, master the chain: proxy injection → PeerAuthentication STRICT → AuthorizationPolicy → SPIFFE principals.

Put it into practice

mTLS is proven by what is refused, not by what answers. This lab has you require mutual TLS in a mesh that still accepted clear-text traffic, then checks both sides: a Pod with its sidecar still reaches the service, a Pod without one no longer does.

Knowledge check

Seven questions on encrypting traffic between Pods: SPIFFE identity, PeerAuthentication, AuthorizationPolicy and diagnosing the handshake.

Knowledge check

Check what you have learned with this interactive quiz

7 questions
8 min
70% to pass

How it works

  • The timer starts when you click Start
  • Multiple choice, true/false and short answer questions
  • You can move between questions
  • Detailed results are shown at the end

Starts the quiz and the timer