Skip to content
Français
Account: not signed in, sync available My learning
Select theme
Français

Securing the infrastructure under the cluster: cloud and nodes

By updated

Improve this page

Logo Kubernetes

Under a Kubernetes cluster sit machines, an operating system, a network and the cloud provider's credentials. These elements do not depend on Kubernetes: RBAC and admission do not protect them. This lesson, which is for anyone who runs or audits a cluster and is preparing for the KCSA, lists what sits under the cluster, says who secures it on a managed cluster and on a self-managed cluster, and explains why compromising a node or a cloud credential bypasses the cluster's controls. It is the "cloud" layer of the 4Cs, seen up close.

What you will learn

  • Tell what the provider secures apart from what your team must secure
  • Name the entry points of the infrastructure under the cluster
  • Explain why a compromised node or cloud credential bypasses the cluster's controls
  • Find the lessons that teach the controls of this layer
  • Interpret a managed or self-managed architecture and say who owns each control

Prerequisites

This lesson extends The 4Cs of cloud native security, whose cloud layer it takes up. It also assumes you know what a worker node is and that you have seen RBAC and NetworkPolicies. No command is to be run here.

What does the provider secure, and what is left to your team?

The cloud provider secures the physical infrastructure it operates: buildings, hardware, hypervisor and physical network. Your team secures what it configures on top: the account, the rights, the network rules, the virtual machines that serve as nodes and their system. The shared responsibility model, explained in Cloud shared responsibility, sets this principle for all types of services.

For Kubernetes, the boundary depends on the offer. On a managed cluster, the provider operates the control plane (API server, etcd, scheduler). It often offers a machine image for the nodes, but it is up to your team to update it and to watch for vulnerabilities, depending on the offer. On a self-managed cluster, nobody else is there: your team carries the control plane, the nodes and the network. The page Shared responsibilities: who does what in security details the split by layer, including for the modes where the provider also manages the nodes.

What are the entry points of the infrastructure under the cluster?

An attacker does not need to go through the Kubernetes API if they can reach what sits below it. Four entry points are worth knowing. The first, metadata, is the one a compromised Pod can reach without leaving the node's network.

The instance metadata service

At a cloud provider, a virtual machine queries a metadata service at the link-local address 169.254.169.254, as explained in Images, cloud-init and the metadata service (in French). This service gives information about the machine and often hands out temporary credentials. A compromised Pod that reaches this address can inherit the node's rights.

Two countermeasures combine. The first is at the network level: the NetworkPolicies lesson shows a rule that excludes this address from a Pod's egress, and Secret leaks: logs, Docker, variables describes how this service leaks.

The second is on the provider side. On AWS, you can accept only version 2 of the service (HttpTokens set to required): it first requires a token, obtained through a PUT request. The response to this PUT crosses only a limited number of network hops, set by HttpPutResponseHopLimit and fixed at 1 by default. A Pod that has its own network namespace goes through the node's network, which adds one hop: in general, it therefore does not get the token. Some node images launch their instances with a limit of 2, which lets a Pod through: check the real value on your nodes in MetadataOptions. You can also disable the service (HttpEndpoint set to disabled). The AWS documentation gives the limit and its default value, not a promise that holds for every network plugin: check the behavior on your cluster.

These two countermeasures close the door. The underlying countermeasure is to give the Pod its own cloud identity, so that it no longer needs to borrow the node's: Machine identity and workload identity compares the mechanisms of GKE, EKS and AKS.

Cloud credentials carried by a node

A node often receives an identity from the provider to call its API, for example to create a volume or a load balancer. If this identity is broad, whoever takes control of it acts on the whole infrastructure, not only on the cluster. The principle is least privilege, covered in IAM: managing identities and permissions in the cloud and in Short-lived machine credentials (in French).

SSH or console access to the nodes

An administrator who connects to a node, over SSH or through the provider's console, acts on the machine outside Kubernetes. This access is therefore very powerful: whoever holds it reads the files of every container on the node. It is limited by reducing the number of accounts, requiring strong keys and logging it, as the lesson Hardening SSH access (in French) shows.

The node image

The machine image fixes the operating system, its kernel and its packages. An old image contains known flaws, and a container that escapes to the kernel takes advantage of those flaws. The provider or your team must therefore renew it regularly, which belongs to host system hardening, whose lessons are listed in the section "Which infrastructure controls are taught elsewhere?".

Why does compromising a node or a cloud credential bypass the cluster's controls?

Kubernetes controls run inside the cluster and assume that the machine and the provider account are sound. When the attacker holds either one, they act under these controls and not through them.

Control metWhat it controlsWhy it does not hinder an attacker one level down
RBACRequests sent to the API serverA node administrator reads the containers' files without sending any request to the API
Admission (for example Pod Security)Write requests sent to the API (creating, modifying, deleting objects)The attacker has no object to create: the Pod already exists and they act on its machine
NetworkPolicyThe Pods' network trafficIt is enforced by the network plugin on the node, therefore by a machine the attacker administers
None, with a stolen cloud credentialNo Kubernetes control appliesThe attacker calls the provider's API directly: depending on the identity's rights, they can delete machines or read disks

Read the right-hand column: in each case, the attacker sends no request the control could refuse. They take a path that does not go through the control.

One nuance concerns the kubelet, the agent of each node. It holds credentials to talk to the API server, and the Node authorization mode limits what they allow, notably reading Secrets only for the Pods scheduled on that node. A compromised node therefore gives access to the Secrets of its Pods, but not to those of the whole cluster: it is a limit that bounds the damage, not a protection of the node itself.

Which infrastructure controls are taught elsewhere?

This lesson does not detail the infrastructure controls: they belong to Linux hardening, IAM and network isolation, which each have their own lessons. The table says where to go for each entry point.

What you want to secureWhere it is taught
The node's operating systemANSSI-BP-028: Linux hardening (in French) and Linux CIS Benchmarks (in French)
SSH access to the nodesHardening SSH access (in French)
The configuration of the cluster componentsKubernetes CIS Benchmark
The nodes' cloud rightsIAM in the cloud and Least privilege applied (in French)
Pods' access to metadataNetworkPolicies
A Pod's own cloud identity, instead of the node'sMachine identity and workload identity
The etcd database and its certificatesetcd and PKI of a Kubernetes cluster

These lessons are not prerequisites of this one: they serve to go deeper into a specific entry point once the map is drawn.

How do you read an architecture and say who owns each control?

To read an architecture, go down layer by layer and ask, for each control, who configures it and who operates it. The course lab cluster serves as an example: with kind, the nodes are containers of a single machine, so the infrastructure under the cluster is that machine, its system and its kernel, which the nodes share. With kubeadm on virtual machines, each node has its own system, accounts and network, which you administer.

The table applies this approach to a managed cluster and to a self-managed cluster.

ControlManaged clusterSelf-managed cluster
Operating the API server and etcdThe provider, depending on the offerYour team
Network access to the API serverYour team, through the service configurationYour team, through network and firewall rules
Node image and patchesShared: provider image, updates by your team depending on the offerYour team
SSH access to the nodesYour team, or no access depending on the offerYour team
Cloud rights of the nodesYour team, through IAMNot applicable outside the cloud: your team then manages its machines' accounts
Metadata seen from the PodsYour team, through NetworkPolicy and provider settingsYour team

The first line goes to the provider and the third is shared. The others stay with your team, with help from the provider in some cases: choosing a managed cluster removes part of the operating work, not the responsibility for access, rights and network.

Key takeaways

  • Under the cluster sit machines, a system, a network and the provider's credentials: RBAC and admission do not protect them, and a NetworkPolicy only filters the Pods' traffic.
  • The provider secures the physical infrastructure; your team secures what it configures on top, and the boundary on Kubernetes depends on the managed or self-managed offer.
  • Four entry points are worth knowing: the instance metadata, the provider credentials carried by a node, SSH or console access, the node image.
  • A compromised node or cloud credential bypasses the cluster's controls, because the attacker sends no request these controls could refuse.
  • A broad cloud credential gives access to the whole infrastructure, not only to the cluster: apply least privilege, and give each Pod its own identity rather than letting it borrow the node's.
  • System hardening, IAM and metadata access have their own dedicated lessons, linked in this page.

Test your knowledge

Eight reasoning questions about the entry points of the infrastructure and the controls they bypass.

Knowledge check

Check what you have learned with this interactive quiz

8 questions
8 min
70% to pass

How it works

  • The timer starts when you click Start
  • Multiple choice, true/false and short answer questions
  • You can move between questions
  • Detailed results are shown at the end

Starts the quiz and the timer