Skip to content
Français
Account: not signed in, sync available My learning
Select theme
Français

Pod Networking: how Pods talk to each other across nodes

By updated

Improve this page

In Kubernetes, every Pod gets its own IP address and can reach every other Pod of the cluster directly, on the same node or on another one, with no address translation and no published port. This lesson makes you check that promise on a four-node kind cluster. You read the addresses, you read the routes that carry a packet from one node to another, then you cause three failures to learn where to look first. It is for readers who can already create a Pod and a Service and want to understand what happens underneath.

What you will learn

  • State the guarantees of the Kubernetes network model
  • Read the address of a Pod, the cluster range and the range of the node that hosts it
  • Say who assigns Pod addresses and what kube-proxy does
  • Test the connection from one Pod to another with wget and ping
  • Diagnose three failures: a Pod that cannot reach another one, a Pod stuck in ContainerCreating, and a communication that fails between two nodes
  • Explain why nslookup kubernetes.default fails from a busybox Pod while DNS works

Prerequisites

You have read Kubernetes Services: you know what a Service and a selector are. The commands of the lesson assume kind 0.33.0, kubectl 1.37.0 and nodes in v1.37.0, on a cluster with one control plane and three workers. Create it with this file, named kind-multi.yaml:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
labels:
topology.kubernetes.io/zone: zone-a
- role: worker
labels:
topology.kubernetes.io/zone: zone-b
- role: worker
labels:
topology.kubernetes.io/zone: zone-c
Terminal window
kind create cluster --name demo-reseau --image kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5 --config kind-multi.yaml
kubectl wait --for=condition=Ready nodes --all --timeout=120s
kubectl create namespace ws-podnet

The four nodes become Ready in about fifteen seconds. On kind, a node is a Docker container: to run a command on a node, this lesson uses docker exec demo-reseau-worker <command>. On a virtual machine, this would be an SSH connection to the node.

Some names in this lesson are in French on purpose, so that the commands and outputs stay those of the tested cluster: demo-reseau means "network-demo", cli-meme-noeud "cli-same-node", cli-autre-noeud "cli-other-node", pas-de-reseau "no-network", 10-kindnet.conflist.sauve "10-kindnet.conflist.saved", IP du client "client IP", and rc_meme_noeud and rc_autre_noeud "rc_same_node" and "rc_other_node".

What guarantees does the Kubernetes network model give?

The network model is a contract: it says what must be true of the network, not how to get it. Every network plugin, called a CNI plugin (see CNI, CSI and CRI), must respect it. This is why your applications behave the same way from one cluster to another, even when the technologies underneath differ.

The guarantees of the model

Three rules apply to every compliant plugin:

  1. Every Pod gets its own IP address.
  2. Two Pods reach each other through these addresses, whatever node hosts them, without publishing a port and without address translation (NAT).
  3. The address a Pod sees for itself is the one the other Pods see for it: there is no translation of the source address between Pods.

The third rule has a concrete consequence: an address found in an access log really designates the sending Pod, not its node.

Every Pod also has its own network namespace, so its own network stack and its 65,535 ports, with no conflict with the other Pods of the node. Unlike Docker, you do not publish a port to reach a Pod from another Pod.

Kubernetes distinguishes four kinds of communication. This lesson covers the first two; the other two go through Services and HTTP exposure.

CommunicationHow it goes through
Container to container, in the same Podlocalhost
Pod to Podthe Pod network, the subject of this lesson
Pod to Servicekube-proxy and DNS, see Services
Outside to ServiceNodePort, LoadBalancer or Ingress

How do you find a Pod's address and reach another Pod?

The simplest way to believe a guarantee is to test it. You create three Pods: an nginx server srv and a client cli-meme-noeud on the same node, and a client cli-autre-noeud on another node. The nodeSelector field places each Pod on the node you want. Save this file as pods.yaml:

apiVersion: v1
kind: Pod
metadata:
name: srv
namespace: ws-podnet
spec:
nodeSelector:
kubernetes.io/hostname: demo-reseau-worker
containers:
- name: nginx
image: nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb
ports:
- containerPort: 80
---
apiVersion: v1
kind: Pod
metadata:
name: cli-autre-noeud
namespace: ws-podnet
spec:
nodeSelector:
kubernetes.io/hostname: demo-reseau-worker2
containers:
- name: busybox
image: busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4
command: ["sleep", "3600"]
---
apiVersion: v1
kind: Pod
metadata:
name: cli-meme-noeud
namespace: ws-podnet
spec:
nodeSelector:
kubernetes.io/hostname: demo-reseau-worker
containers:
- name: busybox
image: busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4
command: ["sleep", "3600"]

See a Pod's IP

The address does not appear in the default output of kubectl get pods: you have to add -o wide, a jsonpath or custom columns.

Terminal window
kubectl apply -f pods.yaml
kubectl -n ws-podnet wait --for=condition=Ready pod --all --timeout=90s
kubectl -n ws-podnet get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
cli-autre-noeud 1/1 Running 0 3s 10.244.3.3 demo-reseau-worker2 <none> <none>
cli-meme-noeud 1/1 Running 0 3s 10.244.1.4 demo-reseau-worker <none> <none>
srv 1/1 Running 0 3s 10.244.1.3 demo-reseau-worker <none> <none>

Look at the IP and NODE columns. srv and cli-meme-noeud are on the node demo-reseau-worker and carry addresses in 10.244.1.x. cli-autre-noeud is on demo-reseau-worker2 and carries 10.244.3.3. The address of a Pod therefore starts with the prefix of its node: the next section explains where that prefix comes from. For a three-column output that you can paste into a script, ask for the columns yourself:

Terminal window
kubectl -n ws-podnet get pods -o custom-columns=NAME:.metadata.name,IP:.status.podIP,NODE:.spec.nodeName
NAME IP NODE
cli-autre-noeud 10.244.3.3 demo-reseau-worker2
cli-meme-noeud 10.244.1.4 demo-reseau-worker
srv 10.244.1.3 demo-reseau-worker

For a single address, kubectl get pod <name> -o jsonpath='{.status.podIP}' is enough. The value is empty until the network plugin has answered, which is normal for a Pod that is starting.

Test with wget/curl

From cli-autre-noeud, reach nginx through the address of srv, without going through a Service:

Terminal window
SRV=$(kubectl -n ws-podnet get pod srv -o jsonpath='{.status.podIP}'); kubectl -n ws-podnet exec cli-autre-noeud -- wget -qO- -T 3 http://$SRV; echo "rc=$?"
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
[...]
</html>
rc=0

The nginx welcome page arrives and rc=0 confirms success: two Pods on two different nodes reach each other through their address. Now look at the log of srv to see who talked to it:

Terminal window
CLI=$(kubectl -n ws-podnet get pod cli-autre-noeud -o jsonpath='{.status.podIP}'); echo "IP du client: $CLI"; kubectl -n ws-podnet logs srv --tail=3
IP du client: 10.244.3.3
2026/11/12 05:59:10 [notice] 1#1: start worker process 36
2026/11/12 05:59:10 [notice] 1#1: start worker process 37
10.244.3.3 - - [12/Nov/2026:05:59:13 +0000] "GET / HTTP/1.1" 200 896 "-" "Wget" "-"

The last line is the access: nginx saw 10.244.3.3, the address of the client Pod, not the address of its node. The two lines above are nginx start-up messages. This is the third guarantee, observed. The same test from cli-meme-noeud also returns rc=0, and a ping gives the same result:

Terminal window
SRV=$(kubectl -n ws-podnet get pod srv -o jsonpath='{.status.podIP}'); kubectl -n ws-podnet exec cli-autre-noeud -- ping -c 3 $SRV; echo "rc=$?"
PING 10.244.1.3 (10.244.1.3): 56 data bytes
64 bytes from 10.244.1.3: seq=0 ttl=62 time=0.107 ms
64 bytes from 10.244.1.3: seq=1 ttl=62 time=0.094 ms
64 bytes from 10.244.1.3: seq=2 ttl=62 time=0.130 ms
--- 10.244.1.3 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.094/0.110/0.130 ms
rc=0

The ttl=62 is two below the starting value of 64: the packet went through two routing steps, the client's node then the server's node. Still, prefer wget or curl to ping to judge a result. An HTTP exchange goes through the same TCP stack as your applications and escapes ICMP filters, whereas a ping can fail for other reasons: an image without ping, a network policy that filters ICMP.

Where do Pod addresses come from?

Pod addresses come from a range reserved for the cluster, cut into subnets handed out node by node. This split makes it possible to route without a central directory: the prefix of an address designates the destination node. A range that is badly sized, or that overlaps the company network, is hard to fix afterwards.

Cluster CIDR

A CIDR (Classless Inter-Domain Routing) is an address range written address/length, such as 10.244.0.0/16. The cluster receives it as an argument of the kube-controller-manager, which you read like this:

Terminal window
kubectl cluster-info dump | grep -m1 cluster-cidr
"--cluster-cidr=10.244.0.0/16",

The Pod range is 10.244.0.0/16. Services have their own range, 10.96.0.0/16 on this cluster, which you read the same way with grep -m1 service-cluster-ip-range: the kubernetes Service is 10.96.0.1, outside the Pod range. This command reads the arguments as kind sets them; if your distribution sets them differently, look for the range in its documentation. The same move works on a kubeadm cluster, where these ranges can also be read in the kubeadm-config ConfigMap of the kube-system namespace. If your shell enables pipefail, these commands can return code 141 although nothing is broken: grep -m1 closes the pipe after the first line.

CIDR per node

Each node receives a subnet of this range, which the spec.podCIDR field of the node shows:

Terminal window
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDR}{"\n"}{end}'
demo-reseau-control-plane 10.244.0.0/24
demo-reseau-worker 10.244.1.0/24
demo-reseau-worker2 10.244.3.0/24
demo-reseau-worker3 10.244.2.0/24

Two things to read. The control plane also receives a subnet, even though it does not host your Pods: allocation is done per node, before any scheduling.

Next, the subnet number does not follow the worker number: worker2 carries 10.244.3.0/24. So find the subnet of a node with this command, never by guessing. Subnets do not overlap, which lets you read the node of a Pod in its address: 10.244.3.3 belongs to worker2.

This reading holds for kindnet. With Calico, Pod addresses come from /26 blocks of the 10.244.0.0/16 pool and do not follow the podCIDR of the node: on a node with podCIDR 10.244.1.0/24, a Pod can receive 10.244.38.92.

A /24 offers 254 usable addresses, but a node does not accept that many Pods:

Terminal window
kubectl get node demo-reseau-worker -o jsonpath='{.status.capacity.pods}{"\n"}'
110

The node announces a capacity of 110 Pods, which is the default value of the kubelet maxPods. Size the cluster on that figure, not on the one of the subnet.

Who assigns the IPs?

Assignment is the job of the CNI plugin, through its IPAM (IP Address Management) component, which keeps the register of taken and free addresses of the node range. Neither the API server nor the kubelet chooses the address: they receive it from the plugin, and the kubelet publishes it in status.podIP. The plugin configuration file, placed on every node, shows it:

Terminal window
docker exec demo-reseau-worker cat /etc/cni/net.d/10-kindnet.conflist
{
"cniVersion": "0.3.1",
"name": "kindnet",
"plugins": [
{
"type": "ptp",
"ipMasq": false,
"ipam": {
"type": "host-local",
"dataDir": "/run/cni-ipam-state",
"routes": [
{ "dst": "0.0.0.0/0" }
],
"ranges": [
[ { "subnet": "10.244.1.0/24" } ]
]
}
,
"mtu": 1500
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}

The output is shortened by its empty lines. Read three blocks. ipam of type host-local assigns addresses in 10.244.1.0/24, the subnet of this node. ptp connects each Pod through a pair of point-to-point virtual interfaces, without going through a bridge: the next section checks it. portmap handles hostPort. The file is named 10-kindnet.conflist: this is what the runtime looks for in /etc/cni/net.d/ to know which plugin to call.

Which path does a packet take between two Pods?

The model imposes the result, not the path: each plugin chooses its own. On kind, the plugin is kindnet. Two DaemonSets of kube-system show it, since each one runs on every node:

Terminal window
kubectl -n kube-system get daemonset
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kindnet 4 4 4 4 4 kubernetes.io/os=linux 3m54s
kube-proxy 4 4 4 4 4 kubernetes.io/os=linux 3m55s

kindnet is the network plugin and kube-proxy is the implementation of Services, four Pods each, one per node. The next subsections read their work on the node demo-reseau-worker.

On the same node

Two containers of the same Pod talk through localhost, because they share the same network namespace. Between two Pods of the same node, the packet does not leave the machine. To see how each Pod is connected, list the interfaces of the node:

Terminal window
docker exec demo-reseau-worker ip -br link
lo UNKNOWN 00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
eth0@if33 UP a6:94:7c:59:9f:c8 <BROADCAST,MULTICAST,UP,LOWER_UP>
veth2a032149@if2 UP 8e:ff:38:aa:88:f0 <BROADCAST,MULTICAST,UP,LOWER_UP>
veth988a09e0@if2 UP ae:e6:21:ce:e5:c2 <BROADCAST,MULTICAST,UP,LOWER_UP>

A veth pair is a virtual cable with one end in the Pod and the other on the node. Count the veth interfaces of the node: there are two, one per Pod (srv and cli-meme-noeud). Also look for what is missing: there is no bridge interface, neither cni0 nor docker0. In fact, docker exec demo-reseau-worker ip -d link show type bridge returns no line at all.

With kindnet, each Pod therefore has its own veth pair, and the node routes between these pairs. Other plugins connect Pods differently, through a local bridge or through eBPF (see Cilium (in French)): the contract stays the same.

The routing table of the node is what links the veth pairs:

Terminal window
docker exec demo-reseau-worker ip route
default via 172.18.0.1 dev eth0
10.244.0.0/24 via 172.18.0.2 dev eth0
10.244.1.3 dev veth2a032149 scope host
10.244.1.4 dev veth988a09e0 scope host
10.244.2.0/24 via 172.18.0.3 dev eth0
10.244.3.0/24 via 172.18.0.5 dev eth0
172.18.0.0/16 dev eth0 proto kernel scope link src 172.18.0.4

Read the two veth lines: the node has one /32 route per local Pod, 10.244.1.3 to veth2a032149 (srv) and 10.244.1.4 to veth988a09e0 (cli-meme-noeud). There is no line for the local subnet 10.244.1.0/24: each Pod is reached through its own route. The other three 10.244.x.0/24 lines are used to reach the other nodes.

Between different nodes

To reach a Pod on another node, the node follows one of the 10.244.x.0/24 via <address> lines of the previous table. The one for worker2 says: everything going to 10.244.3.0/24 leaves directly for 172.18.0.5, the address of the node demo-reseau-worker2. The packet is neither encapsulated nor translated. The diagram follows a packet from srv to cli-autre-noeud: look at the three steps and at the note that recalls that the Pod address stays intact from start to finish. On the worker2 side, the diagram only shows the result, the delivery to the Pod, which the previous wget and ping already showed.

A packet leaves the Pod srv, 10.244.1.3, through its veth to the worker node. That node reads its route 10.244.3.0/24 via 172.18.0.5 and forwards the packet as is to the worker2 node, with no tunnel and no address translation. Worker2 delivers it to the Pod cli-autre-noeud, 10.244.3.3, which it hosts.Podcli-autre-noeud10.244.3.3Nodeworker2172.18.0.5Nodeworker172.18.0.4Podsrv10.244.1.3route read on the node10.244.3.0/24 via 172.18.0.5no tunnel, noaddresstranslationthe node hands the packetto the Pod it hosts① leaves through itsveth towards10.244.3.3② forwards thepacket unchanged③ hands the packetto 10.244.3.3
Source · SVG
Esc to close, + and − to zoom, scroll to move around. 100 %

This direct route assumes that the network between nodes can carry the packet to a neighboring node. On kind, the four nodes are on the same Docker network, so it can. When the network between nodes cannot carry Pod addresses, other plugins take another path. These three modes cannot be observed on kind; they are described in the documentation of the plugins (see CNI, CSI and CRI):

ModePrincipleExample plugins
Overlay with VXLANthe Pod packet is encapsulated in a UDP packet, port 4789 (RFC 7348)Flannel, Calico, Cilium
IP-in-IPthe Pod packet is encapsulated in another IP packet, protocol 4Calico
Native routing with BGPeach node announces its subnet to the network routers, without encapsulationCalico, Cilium

Native routing avoids the cost of encapsulation but needs a network that speaks BGP. Encapsulated modes work on almost any network, at the price of one more header per packet.

The role of kube-proxy

kube-proxy does not handle direct communication between Pods: the CNI plugin does, as you have just seen. Its role is to implement Services, that is, to redirect a Service address (ClusterIP) to the address of a Pod behind it. To do so, it writes rules on every node (see How worker nodes work). Create a Service in front of srv and query it by name from the client:

Terminal window
kubectl -n ws-podnet expose pod srv --port=80 --name=srv
kubectl -n ws-podnet exec cli-autre-noeud -- wget -qO- -T 3 http://srv >/dev/null; echo "rc=$?"
kubectl -n ws-podnet logs srv --tail=1
service/srv exposed
rc=0
10.244.3.3 - - [12/Nov/2026:05:59:18 +0000] "GET / HTTP/1.1" 200 896 "-" "Wget" "-"

Look at the last line: the nginx log still carries 10.244.3.3. The translation done by kube-proxy replaces the destination (the Service address by the Pod address), and here the source stays the Pod address: this is the case of a Pod calling a ClusterIP with kube-proxy in iptables mode. For access from outside through NodePort or LoadBalancer, kube-proxy can also replace the source address (SNAT). The externalTrafficPolicy field of the Service changes this behavior: with Cluster (the default), the client address can be lost; with Local, it is kept, but only the Pods on the contacted node answer. The Using source IP tutorial covers these cases. The rules can be read in the KUBE-SERVICES chain of the nat table of the node:

Terminal window
docker exec demo-reseau-worker sh -c 'iptables -t nat -S KUBE-SERVICES | head -10'
-N KUBE-SERVICES
-A KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:metrics cluster IP" -m tcp --dport 9153 -j KUBE-SVC-JD5MR3NA4I4DYORP
-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS

Each -j KUBE-SVC-* rule matches one Service port: the kubernetes Service on port 443, and kube-dns on 53 over UDP, 53 over TCP and 9153. The kube-proxy mode changes the way these rules are read. Check it rather than assume it:

Terminal window
kubectl -n kube-system get configmap kube-proxy -o yaml | grep -E '^\s+mode:'
kubectl -n kube-system logs -l k8s-app=kube-proxy --tail=-1 | grep -i proxier
mode: iptables
I1112 05:55:35.080667 1 server_linux.go:144] "Using iptables Proxier"
[...]

The ConfigMap declares mode: iptables and the logs confirm it with Using iptables Proxier (one pair of lines per kube-proxy Pod, four here). The log is the reference if the ConfigMap was changed without restarting the Pods.

According to the official documentation on Services, three modes exist on Linux: iptables, nftables and ipvs, deprecated since 1.35. The nftables mode has been stable since 1.33, and iptables remains the default mode in 1.37.

What do you do when two Pods cannot reach each other?

Most network failures come down to three situations: a Pod that cannot reach another one, a Pod stuck in ContainerCreating because the CNI plugin has not prepared its network, and traffic that fails between two nodes. Each one is recognized by a distinct symptom. The first subsection gives the reading tools, the next three replay one failure each. The plugin comes first in the order of checks: if it does not run, the other tests tell you nothing.

Advanced diagnosis

When an exchange fails, first ask whether the Pod received an interface and a default route. The ip addr and ip route commands assume an image that contains ip: busybox has it, nginx does not.

Terminal window
kubectl -n ws-podnet exec srv -- ip addr; echo "rc=$?"
error: Internal error occurred: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "52c3e3753fffb32dd53ac8152fcc0b48b1f894085d0f0fbfc675e424c44773ad": OCI runtime exec failed: exec failed: unable to start container process: exec: "ip": executable file not found in $PATH
rc=1

The message says executable file not found in $PATH: the fault comes from the image, not from the network. In nginx, cat /proc/net/fib_trie still works and contains the Pod address, 10.244.1.3, on a /32 host LOCAL line. From the busybox Pod:

Terminal window
kubectl -n ws-podnet exec cli-autre-noeud -- ip addr
kubectl -n ws-podnet exec cli-autre-noeud -- ip route
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0@if4: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue qlen 1000
link/ether a2:e9:1a:c8:bc:f9 brd ff:ff:ff:ff:ff:ff
inet 10.244.3.3/24 brd 10.244.3.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::a0e9:1aff:fec8:bcf9/64 scope link
valid_lft forever preferred_lft forever
default via 10.244.3.1 dev eth0
10.244.3.0/24 via 10.244.3.1 dev eth0 src 10.244.3.3
10.244.3.1 dev eth0 scope link src 10.244.3.3

Look at eth0@if4: this is the Pod interface, with its address 10.244.3.3/24, and the @if4 suffix designates the other end of the veth pair, on the node side. The default route goes through 10.244.3.1, the first address of the node subnet. A Pod with no eth0 and no default route has not received its network: go to the ContainerCreating failure.

A Pod cannot reach another Pod

The first thing to observe is the shape of the failure, because it tells you where to look. The Pod srv only listens on port 80; query a closed port of srv, then an address that no Pod carries:

Terminal window
SRV=$(kubectl -n ws-podnet get pod srv -o jsonpath='{.status.podIP}'); time kubectl -n ws-podnet exec cli-autre-noeud -- wget -qO- -T 3 http://$SRV:8081; echo "rc=$?"
time kubectl -n ws-podnet exec cli-autre-noeud -- wget -qO- -T 3 http://10.244.99.99; echo "rc=$?"
wget: can't connect to remote host (10.244.1.3): Connection refused
command terminated with exit code 1
real 0m0.067s
rc=1
wget: download timed out
command terminated with exit code 1
real 0m3.065s
rc=1

The first failure lasts 0.07 s and says Connection refused: the packet arrived, and the Pod answered that no program listens on this port. Look on the application side or the Service side.

The second lasts 3.07 s, the requested timeout, and says download timed out: nobody answered, the packet was lost on the way. Look on the network side: CNI plugin, route, NetworkPolicy. The address 10.244.99.99 is in the 10.244.0.0/16 range but outside the four node subnets, so no node routes it.

To check the network plugin, do not look for it by name. A grep -E 'calico|cilium|flannel' finds nothing on a cluster that works with kindnet, and would wrongly conclude that there is no CNI. Two checks depend on no name: the configuration placed on the node, and the DaemonSets of kube-system, where every network plugin appears since it must run on every node.

Terminal window
docker exec demo-reseau-worker ls /etc/cni/net.d/
kubectl -n kube-system get daemonset
10-kindnet.conflist

The first command must list a file: this is the configuration the runtime uses. The second gives the name of the plugin, already seen above. Once you know this name, read its logs with kubectl -n kube-system logs -l app=<plugin-name> (the label is app=kindnet on kind), then check the routes of the node with ip route.

The Pod is stuck in ContainerCreating

A Pod stuck in ContainerCreating is waiting for a step to finish, and the assignment of the address by the plugin is one of them. Reproduce the failure by removing the plugin configuration file on the node demo-reseau-worker3 (this deliberately breaks that node), then create a Pod on it. Save this Pod as pas-de-reseau.yaml:

apiVersion: v1
kind: Pod
metadata:
name: pas-de-reseau
namespace: ws-podnet
spec:
nodeSelector:
kubernetes.io/hostname: demo-reseau-worker3
containers:
- name: busybox
image: busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4
command: ["sleep", "3600"]
Terminal window
docker exec demo-reseau-worker3 sh -c 'mv /etc/cni/net.d/10-kindnet.conflist /root/10-kindnet.conflist.sauve'
kubectl apply -f pas-de-reseau.yaml
sleep 20
kubectl -n ws-podnet get pod pas-de-reseau
pod/pas-de-reseau created
NAME READY STATUS RESTARTS AGE
pas-de-reseau 0/1 ContainerCreating 0 20s

The Pod stays in ContainerCreating: the events give the exact message of the plugin.

Terminal window
kubectl -n ws-podnet describe pod pas-de-reseau | sed -n '/Events:/,$p'
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 20s default-scheduler Successfully assigned ws-podnet/pas-de-reseau to demo-reseau-worker3
Warning FailedCreatePodSandBox 19s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "3183dc20253bc4cd008968e593e54387e29a719cf8daa3c5539e19cd336800ab": cni plugin not initialized
Warning NetworkNotReady 2s (x4 over 8s) kubelet network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

Read the end of the two Warning messages: cni plugin not initialized. The runtime no longer finds a network configuration, so it cannot prepare the Pod. The node itself is NotReady: kubectl get node demo-reseau-worker3 shows it twenty seconds after the file is removed.

Putting the file back is the obvious move, and it repairs nothing:

Terminal window
docker exec demo-reseau-worker3 sh -c 'mv /root/10-kindnet.conflist.sauve /etc/cni/net.d/10-kindnet.conflist'
kubectl -n ws-podnet wait --for=condition=Ready pod/pas-de-reseau --timeout=120s
error: timed out waiting for the condition on pods/pas-de-reseau

The file is back and the Pod is still not ready after 120 seconds, and the node stays NotReady. The containerd log on the node shows that it saw the file disappear (fs change event(RENAME), but no event when it came back: it never reloaded the configuration.

The move that repairs it is to delete the kindnet Pod of that node: the recreated Pod rewrites the file in a way that containerd detects.

Terminal window
kubectl -n kube-system delete pod -l app=kindnet --field-selector spec.nodeName=demo-reseau-worker3
kubectl wait --for=condition=Ready node/demo-reseau-worker3 --timeout=120s
pod "kindnet-h4l7q" deleted from kube-system namespace
node/demo-reseau-worker3 condition met

The node is Ready again in 14 seconds. Remember the rule: after repairing the configuration of a network plugin, restart its Pod on the affected node instead of waiting.

With Calico, the move is the same: delete the plugin Pod (calico-node) on the affected node, because Calico does not rewrite the CNI file by itself.

Node-to-node communication is down

When two Pods reach each other on the same node but not across nodes, the node does not know where to send the packet. Simulate it by deleting, on demo-reseau-worker, the route to the subnet of worker2:

Terminal window
docker exec demo-reseau-worker ip route show | grep "$(kubectl get node demo-reseau-worker2 -o jsonpath='{.spec.podCIDR}')"
docker exec demo-reseau-worker ip route del 10.244.3.0/24
10.244.3.0/24 via 172.18.0.5 dev eth0

The first command reads the route (10.244.3.0/24 via 172.18.0.5), the second deletes it. Now test both clients toward srv:

Terminal window
SRV=$(kubectl -n ws-podnet get pod srv -o jsonpath='{.status.podIP}'); kubectl -n ws-podnet exec cli-meme-noeud -- wget -qO- -T 3 http://$SRV >/dev/null; echo "rc_meme_noeud=$?"; kubectl -n ws-podnet exec cli-autre-noeud -- wget -qO- -T 3 http://$SRV >/dev/null; echo "rc_autre_noeud=$?"
docker exec demo-reseau-worker ip route get 10.244.3.1
rc_meme_noeud=0
wget: download timed out
command terminated with exit code 1
rc_autre_noeud=1
10.244.3.1 via 172.18.0.1 dev eth0 src 172.18.0.4 uid 0
cache

The pair of tests isolates the path between nodes: the client on the same node succeeds (rc_meme_noeud=0), the one on the other node fails with a timeout. ip route get gives the reason: without the route, the node sends the packet to its default gateway 172.18.0.1 instead of worker2. Success on the same node and failure toward the other point to a missing route or a blocked path between nodes.

Here, kindnet puts the route back by itself: it was back four seconds after its deletion, and both clients succeed again. For a failure that does not repair itself, check in order that the nodes see each other (kubectl get nodes), that spec.podCIDR is allocated, then that ip route get <Pod address> picks the right node.

On a plugin that encapsulates, the cause is most often outside Kubernetes, according to the plugin documentation: UDP port 4789 filtered for VXLAN, IP protocol 4 filtered for IP-in-IP, BGP session not established.

Why nslookup kubernetes.default fails from a busybox Pod

The short name kubernetes.default is the one everyone types, and with busybox it returns NXDOMAIN, although the cluster DNS works.

Terminal window
kubectl -n ws-podnet exec cli-autre-noeud -- nslookup kubernetes.default; echo "rc=$?"
kubectl -n ws-podnet exec cli-autre-noeud -- nslookup kubernetes.default.svc.cluster.local; echo "rc=$?"
Server: 10.96.0.10
Address: 10.96.0.10:53
** server can't find kubernetes.default: NXDOMAIN
** server can't find kubernetes.default: NXDOMAIN
command terminated with exit code 1
rc=1
Server: 10.96.0.10
Address: 10.96.0.10:53
Name: kubernetes.default.svc.cluster.local
Address: 10.96.0.1
rc=0

The short name fails (rc=1), the full name resolves to 10.96.0.1 (rc=0). The cluster is not at fault: from the same Pod, wget -T 3 -S -O- --no-check-certificate https://kubernetes.default prints Connecting to kubernetes.default (10.96.0.1:443) then HTTP/1.1 403 Forbidden. The name resolved, the TLS connection was established, and the API server refuses anonymous access. The tool differs, not the DNS.

The /etc/resolv.conf of the Pod explains the difference:

Terminal window
kubectl -n ws-podnet exec cli-autre-noeud -- cat /etc/resolv.conf
search ws-podnet.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5

The search line lists the suffixes to try to complete an incomplete name. Tools that go through the C library, like wget, apply them. With busybox 1.38.0, nslookup only applies the search list to a name with no dot at all: kubernetes.default contains one, so it is queried as is and nobody serves it.

When diagnosing, use the full name <service>.<namespace>.svc.cluster.local, which depends on no search list. On a short name with no dot, such as web, busybox also prints the NXDOMAIN answers of the suffixes that fail and exits with rc=1 even when it found the address: so do not trust the return code of this nslookup. The next lesson details name resolution.

Which network commands to remember for the CKA?

In the CKA exam, which is timed and taken at the terminal, jsonpath expressions are the slowest to rebuild. Practice typing them without help: these commands cover most of the Pod network questions.

Terminal window
kubectl get pod <name> -o wide
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDR}{"\n"}{end}'
kubectl exec <pod> -- ip addr
kubectl exec <pod> -- wget -qO- -T 3 http://<pod-ip>
kubectl -n kube-system get daemonset
kubectl describe pod <name> | grep -A5 Events

Put it into practice

The guided lab repeats the moves of this lesson on a real cluster. A worker has lost the network of its Pods. You first record the address ranges of the cluster and the address of a Pod, then you repair the node whose CNI file has disappeared. You then prove that one Pod reaches another from one node to the other, and you say which of two Services refuses right away and which one makes you wait. The validation reads six states of the cluster: the recorded ranges, the address of the target Pod, the worker Ready again, the target Pod placed on that worker, the traffic from one node to the other, and the distinction between an immediate refusal and a timeout.

This lab runs on a kubeadm virtual machine with Calico, not on kind. The nodes are machines: the docker exec command of this lesson does not exist there. The lesson Services is a useful support.

How do you clean up the lesson's cluster?

Delete the namespace, then the cluster. Deleting the namespace takes about forty seconds, the time for the sleep Pods to stop.

Terminal window
kubectl delete namespace ws-podnet
kind delete cluster --name demo-reseau

Key takeaways

  • Every Pod has its own IP address, and two Pods reach each other through these addresses with no port to publish and no address translation.
  • The network model is a contract: Kubernetes sets the guarantees, the CNI plugin chooses the path.
  • The cluster reserves a range (--cluster-cidr), cut into one subnet per node (spec.podCIDR), control plane included; the CNI plugin assigns the addresses and the kubelet publishes them in status.podIP.
  • With kindnet, each Pod has its routed veth pair, with no bridge, and the nodes route the subnets to each other directly.
  • kube-proxy implements Services; it does not handle direct traffic between Pods and does not change the source address of a Pod calling a ClusterIP (external access through NodePort or LoadBalancer can replace it, depending on externalTrafficPolicy).
  • An immediate refusal (Connection refused) does not have the same cause as a silence (timed out): the first is on the application side, the second on the network side.
  • After repairing the CNI configuration of a node, restart the plugin Pod on that node instead of waiting.
  • To test a name from busybox, use the full name <service>.<namespace>.svc.cluster.local.

Test your knowledge

This quiz covers the points of the lesson in the format of the CKA questions. The pass mark is set at 80%, stricter than the 66% of the CKA exam.

Knowledge check

Check what you have learned with this interactive quiz

7 questions
5 min
80% to pass

How it works

  • The timer starts when you click Start
  • Multiple choice, true/false and short answer questions
  • You can move between questions
  • Detailed results are shown at the end

Starts the quiz and the timer