Skip to content
Français
Account: not signed in, sync available My learning
Select theme
Français

Kubernetes High Availability: the Control Plane

By updated

Improve this page

A Kubernetes cluster with a single control plane is a SPOF (Single Point of Failure). If the master node goes down, nobody can deploy, scale or manage workloads any more, even though the applications keep running. High availability (HA) of the control plane removes this risk by spreading the critical components across several nodes.

This guide covers the HA architecture for the CKA certification and for production environments.

What you will learn

  • Why a single control plane is a risk
  • The two HA topologies, stacked etcd and external etcd
  • Setting up an HA cluster with kubeadm
  • The role of the load balancer in front of the API Server
  • The failover scenarios and how to handle them
  • Upgrading a control plane and replacing one
  • Building an HA cluster with three control planes using kind, for practice

Why high availability?

The control plane does not run your applications: it decides what runs and where. As long as it works, its decisions are applied again and again. When it disappears, the cluster freezes in its last known state, and nothing tells the users of the applications. Understanding what exactly stops is the prerequisite for choosing an HA topology.

What is the problem with a single control plane?

In a standard cluster, the control plane (a single node) hosts:

ComponentRoleImpact if unavailable
kube-apiserverEntry point for all requestsNo kubectl command works
etcdStores the state of the clusterConfiguration lost, no resource can be created
kube-schedulerPlaces Pods on nodesNew Pods stay Pending
kube-controller-managerManages Deployments, ReplicaSets...No reconciliation, no autoscaling

One nuance changes how you live through the incident: if the control plane goes down, existing Pods keep running on the workers, and your applications stay online. What you lose is the ability to act: no more creation, no more changes, no more diagnosis through the API. In other words, the outage is invisible to users until something else breaks, at which point you have no way left to intervene.

What does HA bring?

High availability does not make the cluster faster or more powerful: it lets it survive the loss of a machine. Here is what you keep with three control planes, the recommended minimum:

  • one node can fail with no impact for users;
  • kubectl keeps working;
  • you upgrade the nodes one at a time, with no interruption;
  • etcd keeps its quorum with two nodes out of three.

What are the two HA topologies?

Kubernetes supports two architectures for high availability, which differ in where etcd lives.

Each control plane hosts both the Kubernetes components and etcd:

Stacked etcd topology: three control planes each host kube-apiserver, kube-scheduler, kube-controller-manager and one etcd member, behind a single load balancer on port 6443 through which the three workers connect.Control Plane 1Control Plane 2Control Plane 3Load balancersingle entry point, port 6443Worker 1Worker 2Worker 3kube-apiserverkube-schedulerkube-controller-manageretcdkube-apiserverkube-schedulerkube-controller-manageretcdkube-apiserverkube-schedulerkube-controller-manageretcd
Source · SVG
Esc to close, + and − to zoom, scroll to move around. 100 %

This topology is the simplest to deploy and maintain: it needs fewer nodes and reuses the standard kubeadm configuration. Its downside is the coupling: losing a control plane means losing an etcd member at the same time.

External etcd topology

The etcd cluster is separate from the Kubernetes control planes:

External etcd topology: three control planes that host only kube-apiserver, kube-scheduler and kube-controller-manager each query a member of a dedicated three-member etcd cluster replicated by Raft.Dedicated etcd cluster (Raft replication)Control Plane 1Control Plane 2Control Plane 3etcd 1etcd 2etcd 3kube-apiserverkube-schedulerkube-controller-managerkube-apiserverkube-schedulerkube-controller-managerkube-apiserverkube-schedulerkube-controller-manager
Source · SVG
Esc to close, + and − to zoom, scroll to move around. 100 %

This topology isolates failures, since etcd is independent, and can be sized more freely. In return, it is more complex to deploy and needs more nodes, six at a minimum.

For the CKA exam, focus on the stacked etcd topology: it is what kubeadm deploys by default in high availability mode, and therefore what the questions will be about.

What do you need before building an HA cluster?

The setup needs more than a test cluster: you need a stable virtual address in front of the API servers, and an odd number of control plane nodes so that the etcd quorum can break ties.

Infrastructure

Three nodes are the minimum: with two, losing a single one breaks the etcd quorum and the cluster becomes unable to record any change.

  • 3 control plane nodes (2 CPU, 2 GB RAM, lab minimum, not production sizing)
  • 1 redundant load balancer in front of the API Servers (HAProxy + keepalived, Nginx, or a cloud LB)
  • Network connectivity between all nodes
  • Open ports: 6443 (API), 2379-2380 (etcd), 10250-10259 (kubelet/components)

This guide is for self-managed kubeadm clusters, on virtual or physical machines. On a managed cluster, EKS, GKE or AKS, the question does not arise: control plane high availability is part of what you buy, and you have no say in it.

Software

Versions must be aligned before initialization: the kubeadm preflight checks fail if the node's kubelet is newer than the targeted control plane version.

  • kubeadm, kubelet, kubectl installed on all nodes
  • Container runtime (containerd recommended)
  • Same Kubernetes version on all nodes

How do you configure the load balancer?

The load balancer is essential to distribute requests across the API Servers.

With HAProxy

Create /etc/haproxy/haproxy.cfg on the load balancer node:

global
log /dev/log local0
maxconn 2000
user haproxy
group haproxy
daemon
defaults
log global
mode tcp
option tcplog
option dontlognull
timeout connect 5s
timeout client 50s
timeout server 50s
frontend kubernetes-api
bind *:6443
default_backend kubernetes-masters
backend kubernetes-masters
balance roundrobin
option tcp-check
server master1 192.168.1.10:6443 check fall 3 rise 2
server master2 192.168.1.11:6443 check fall 3 rise 2
server master3 192.168.1.12:6443 check fall 3 rise 2

Start HAProxy:

Terminal window
sudo systemctl enable --now haproxy

Check:

Terminal window
# Test the connection to the load balancer
nc -zv <LOAD_BALANCER_IP> 6443

Be careful not to move the problem without solving it. A single HAProxy, with no virtual address and no redundancy, becomes a single point of failure in its own right: three control planes behind a load balancer that goes down is still an unreachable cluster. In production, pair HAProxy with keepalived to get a floating virtual address, or hand the role over to a managed load balancer from your provider.

Advanced check:

Terminal window
# Check that the Kubernetes API really answers
curl -k https://<LB_IP>:6443/readyz

How do you initialize the first control plane?

The first node is special: it creates the cluster CA, the etcd database and the Secret that holds the shared certificates. Two parameters of this step are irreversible or time-limited: controlPlaneEndpoint, which fixes the address everyone will use to reach the API, and the certificate-key produced by --upload-certs, valid for two hours.

  1. Prepare the kubeadm configuration file

    Create kubeadm-config.yaml:

    apiVersion: kubeadm.k8s.io/v1beta4
    kind: ClusterConfiguration
    kubernetesVersion: v1.37.2
    controlPlaneEndpoint: "LOAD_BALANCER_IP:6443"
    networking:
    podSubnet: "10.244.0.0/16"
    ---
    apiVersion: kubeadm.k8s.io/v1beta4
    kind: InitConfiguration
    localAPIEndpoint:
    advertiseAddress: "192.168.1.10"
    bindPort: 6443

    The controlPlaneEndpoint option is mandatory in high availability, and it points to the load balancer. Remember this above all: once the cluster is initialized, it can no longer be changed. That is why you should set it even on a single control plane cluster, or you will have to rebuild everything the day high availability becomes necessary.

    Watch the spelling: the configuration file field is written controlPlaneEndpoint, in camelCase, while the equivalent command-line flag is written --control-plane-endpoint, in lowercase with hyphens. Both forms do the same thing, and kubeadm init --control-plane-endpoint "LOAD_BALANCER_IP:6443" avoids the file for a quick trial. Mixing the two spellings produces an unknown flag with no hint about the right one.

  2. Initialize the cluster

    Terminal window
    sudo kubeadm init --config=kubeadm-config.yaml --upload-certs

    The --upload-certs option encrypts and stores the certificates in a Kubernetes Secret so that the other control planes can fetch them.

  3. Note the join commands

    The output shows two commands:

    • One to join other control planes
    • One to join workers
    # For the control planes
    kubeadm join LOAD_BALANCER_IP:6443 --token <token> \
    --discovery-token-ca-cert-hash sha256:<hash> \
    --control-plane --certificate-key <key>
    # For the workers
    kubeadm join LOAD_BALANCER_IP:6443 --token <token> \
    --discovery-token-ca-cert-hash sha256:<hash>
  4. Configure kubectl

    Terminal window
    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config
  5. Install a CNI

    Flannel is a simple option to start with. You can also choose Calico, Cilium or another CNI depending on your needs. The address below targets a specific Flannel release, not the development branch: the same command therefore always installs the same manifest.

    Terminal window
    # Example with Flannel
    kubectl apply -f https://github.com/flannel-io/flannel/releases/download/v0.28.10/kube-flannel.yml

How do you join the other control planes?

The first control plane produced a special join command, which differs from the workers' one by --control-plane and a certificate key. It is what lets the following nodes fetch the cluster certificates and become etcd members, not mere clients.

On each additional control plane (master2, master3):

Terminal window
sudo kubeadm join LOAD_BALANCER_IP:6443 --token <token> \
--discovery-token-ca-cert-hash sha256:<hash> \
--control-plane --certificate-key <key>

Check:

Terminal window
kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 Ready control-plane 10m v1.37.2
master2 Ready control-plane 5m v1.37.2
master3 Ready control-plane 2m v1.37.2

How do you check the state of the HA cluster?

Three checks, in this order, because they cover different layers: the nodes as Kubernetes sees them, the etcd members as etcd itself sees them, and the internal health of the API Server. A cluster can show three Ready nodes while an etcd member has left the cluster.

Node state

The ROLES column must show control-plane for the three nodes, and kubectl get nodes -o wide also reveals the advertised IP addresses, which help you check that they match the load balancer backends.

Terminal window
kubectl get nodes -o wide

etcd state

kubectl cannot see etcd: you must query the etcd cluster directly, presenting the certificates kubeadm placed in /etc/kubernetes/pki/etcd/. The three members must appear with the started status. The etcd lesson details how it works and how it fails.

The etcdctl binary is not installed on a kubeadm node. That is the first obstacle, and it is a surprise: which etcdctl exits with 1. The binary lives in the etcd container, a distroless image registry.k8s.io/etcd:3.7.0-0, and you reach it with an exec:

Terminal window
kubectl -n kube-system exec etcd-<node-name> -- etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
member list --write-out=table
Output, captured on a cluster with three control planes
┌──────────────────┬─────────┬────────┬─────────────────────────────┬─────────────────────────────┬────────────┐
│ ID │ STATUS │ NAME │ PEER ADDRS │ CLIENT ADDRS │ IS LEARNER │
├──────────────────┼─────────┼────────┼─────────────────────────────┼─────────────────────────────┼────────────┤
│ 3b6a9969c03c53c7 │ started │ ha-cp1 │ https://192.168.122.40:2380 │ https://192.168.122.40:2379 │ false │
│ 4511773aba6df6c3 │ started │ ha-cp2 │ https://192.168.122.41:2380 │ https://192.168.122.41:2379 │ false │
│ c5f3530aa9a1f7da │ started │ ha-cp3 │ https://192.168.122.42:2380 │ https://192.168.122.42:2379 │ false │
└──────────────────┴─────────┴────────┴─────────────────────────────┴─────────────────────────────┴────────────┘

The IS LEARNER column is false for a voting member. A member at true is still catching up and does not count toward the quorum, which explains why a cluster with four rows can have only three that vote.

How do you check the health of each member?

member list says who is declared, not who answers. The distinction matters, and endpoint health --cluster makes it:

Terminal window
kubectl -n kube-system exec etcd-<node-name> -- etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
endpoint health --cluster --write-out=table
Output, one member stopped out of three
│ https://192.168.122.40:2379 │ true │ 3.458728ms │ │
│ https://192.168.122.41:2379 │ true │ 9.71893ms │ │
│ https://192.168.122.42:2379 │ false │ 5.004566097s │ context deadline exceeded │
Error: unhealthy cluster

The command exits with 1 as soon as a single member is missing, even though the cluster works perfectly. So do not wire an alert to this exit code alone: it signals a degradation, not an outage.

How do you check the health of the components?

kubectl get componentstatuses is deprecated and returns incomplete information. The /readyz and /livez endpoints, queried with --raw, list each internal check of the API Server one by one, etcd included.

Terminal window
# Modern endpoints (componentstatuses is deprecated)
kubectl get --raw='/readyz?verbose'
kubectl get --raw='/livez?verbose'

What happens when a control plane fails?

High availability is not declared, it is verified. The test is to deliberately stop a control plane and confirm that write operations still go through. The table that follows generalizes the result to all combinations of failures.

How do you test the failure of one control plane?

Stopping the kubelet is enough: the control plane components are static pods driven by the kubelet, so they stop with it.

  1. Stop a control plane

    Terminal window
    # On master2
    sudo systemctl stop kubelet
  2. Check that the cluster still works

    Terminal window
    kubectl get nodes
    kubectl create deployment test --image=nginx:1.30@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb
    kubectl get pods
  3. Restart the node

    Terminal window
    sudo systemctl start kubelet

How does the cluster behave depending on the failures?

The threshold sits between one and two simultaneous failures out of three nodes. The switch is abrupt: as long as the quorum holds, nothing changes for the user; once it is lost, no write succeeds any more.

FailuresetcdAPI ServerObserved impact
0 of 3quorumservesnormal operation
1 of 3quorum at 2 of 3servesno change for the user
2 of 3quorum lostno longer answersUnable to connect to the server: unexpected EOF
3 of 3stoppedstoppedcluster unusable

The two bold rows were measured on a cluster with three control planes. With one failure, a kubectl run succeeds while the machine is still off: the cluster does not just answer, it accepts writes. With two failures, the connection is cut dead, with no intermediate message and no degraded mode.

Losing the etcd quorum is not a gradual degradation, it is a stop. The cluster can no longer accept any state change. In the measurement above, with two control planes out of three stopped, kubectl receives no answer at all. Do not count on a "read-only" mode.

etcd requires a majority of members to be working, which gives three usual cases: with 3 nodes you need 2, with 5 you need 3, with 7 you need 4. Hence the odd-number rule: going from 3 to 4 members does not change the number of failures tolerated, which stays at one, while adding a machine to maintain and one more member that can fail.

How do you maintain an HA cluster?

An HA cluster is maintained without service interruption, provided you never touch two nodes at the same time: the etcd quorum must be kept for the whole operation. The two procedures below cover the planned upgrade and the replacement of a node that is permanently lost.

How do you upgrade a control plane?

Proceed one node at a time:

Terminal window
# 1. Drain the node
kubectl drain master1 --ignore-daemonsets --delete-emptydir-data
# 2. Upgrade kubeadm (replace X.Y.Z with the target version)
sudo apt-get update && sudo apt-get install -y kubeadm=X.Y.Z-*
# 3. Apply the upgrade
sudo kubeadm upgrade apply vX.Y.Z # First control plane only
# or
sudo kubeadm upgrade node # Other control planes
# 4. Upgrade kubelet and kubectl
sudo apt-get install -y kubelet=X.Y.Z-* kubectl=X.Y.Z-*
sudo systemctl daemon-reload && sudo systemctl restart kubelet
# 5. Uncordon the node
kubectl uncordon master1

Before any version upgrade, check the current stable version on kubernetes.io/releases: it changes every few weeks. The full procedure, with its compatibility rules between versions, is detailed in the lesson upgrade a Kubernetes cluster.

How do you replace a failed control plane?

Replacing a dead control plane does not mean adding another one: you must first remove the old etcd member, or the cluster keeps waiting for it when counting its quorum. The order of operations therefore matters more than the commands themselves.

If a control plane is permanently lost, follow these four steps. The etcdctl commands run in the etcd container of a healthy node, with the certificate options seen above, since the binary is not installed on the node:

  1. Remove the etcd member

    Terminal window
    # Find the ID of the lost member, from the etcd Pod of a healthy node
    kubectl -n kube-system exec etcd-<healthy-node> -- etcdctl \
    --endpoints=https://127.0.0.1:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/server.crt \
    --key=/etc/kubernetes/pki/etcd/server.key \
    member list --write-out=table
    # Remove it, with the ID read from the ID column
    kubectl -n kube-system exec etcd-<healthy-node> -- etcdctl \
    --endpoints=https://127.0.0.1:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/server.crt \
    --key=/etc/kubernetes/pki/etcd/server.key \
    member remove <member-ID>
  2. Delete the node from the cluster

    Terminal window
    kubectl delete node <failed-node>
  3. Set up a new node

    Generate a new token and join:

    Terminal window
    kubeadm token create --print-join-command
    kubeadm init phase upload-certs --upload-certs
  4. Join the new control plane

    Terminal window
    kubeadm join LOAD_BALANCER_IP:6443 --token <token> \
    --discovery-token-ca-cert-hash sha256:<hash> \
    --control-plane --certificate-key <key>

What are the common errors?

The three failures below account for most of the blockages during a first HA setup. They have a common point: the message shown rarely names the real cause, so you have to check the layer underneath.

The join fails with "certificate has expired"

The certificate key produced by kubeadm init has a lifetime of two hours. After that, the join command stays syntactically valid but fails to fetch the certificates, with a message that does not point to the real cause.

Regenerate it by running the certificate upload again:

Terminal window
sudo kubeadm init phase upload-certs --upload-certs

etcd refuses to start after a reboot

A missing or expired certificate prevents etcd from starting. These two commands show whether the files are there and until when they are valid:

Terminal window
ls -la /etc/kubernetes/pki/etcd/
openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -text -noout | grep -A2 Validity

kubectl times out against the load balancer

When kubectl times out while the control planes are healthy, the culprit is almost always the load balancer: wrong address in the kubeconfig, backends declared on the wrong port, or a health check that takes every node out of the service.

These three checks read from the outside in: if the first one fails, the next two will tell you nothing useful.

  1. The load balancer answers: nc -zv <LB_IP> 6443
  2. At least one API Server is alive: curl -k https://<MASTER_IP>:6443/healthz
  3. The HAProxy health checks pass, otherwise the load balancer empties its pool

How do you build an HA cluster with kind for practice?

You do not always have three machines available for an HA control plane. kind creates a cluster inside Docker containers, and it accepts several control plane nodes. Each role: control-plane line in the configuration file adds one control plane node, and therefore one more etcd member.

kind-ha.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: control-plane
- role: control-plane

The cluster is created with this file and a pinned node image. The name ha lets you find its containers with docker ps.

Terminal window
kind create cluster --name ha --config kind-ha.yaml \
--image kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5

Unlike the kubeadm installation seen above, you do not configure a load balancer. As soon as a kind cluster has several control planes, kind starts an extra container itself, ha-external-load-balancer, which plays the role of HAProxy in front of the three API servers. The kubeconfig file that kind writes points to this load balancer, not to one control plane in particular. Check with kubectl get nodes that the three nodes are Ready, then with docker ps --filter name=ha- that the load balancer container is there next to the three nodes. Then stop one control plane container with docker stop: kubectl get nodes must keep answering through the load balancer, which is exactly the property this lesson is trying to achieve.

Put it into practice

The only test that counts is to stop a control plane and watch the API answer. The challenge below asks you to do it without help.

Challenge: install a cluster and make its control plane highly available

The challenge is the unaided proof of the module "Install a cluster", and it is graded on the state of the machines you leave behind, not on the commands you typed. Two jobs await you. On the kubeadm cluster of the lab, a worker has been removed from the cluster and reset: you prepare it and join it so that it survives a reboot, then you force a Pod onto it. On the separate host k8s-kind.lab, you then build a kind cluster named ha, with a pinned kindest/node image, three control planes and the load balancer that kind itself creates in front of their API servers. Its etcd has three members, and the API must keep answering through the load balancer when one of the control planes stops. No procedure is given.

The validation reads seven states, with no solution and no value to copy. It checks that the worker is Ready with a kubelet in version 1.37, that swap is durably off, that containerd uses the right cgroup driver and that the forced Pod runs on the worker. It also reads the three control plane nodes and the three etcd members through the load balancer address. For the last state, the grader stops a control plane itself, queries the API through the load balancer, then starts it again: each check therefore takes more than a minute.

Passing requires 70 points out of 100, and there is no hint. Two states are blocking and must pass whatever the total: the worker joined to the cluster, and the API still answering when a control plane goes down. The challenge builds on this lesson, which also teaches the kind example with three control planes, on Join a worker to a kubeadm cluster and on Install a cluster with kubeadm.

Key takeaways

  1. 3 control planes minimum for high availability
  2. A load balancer is mandatory in front of the API Servers
  3. Stacked etcd topology = simpler, recommended for the CKA
  4. etcd quorum = a majority of members is required
  5. Upgrade one node at a time to keep availability
  6. --upload-certs makes it easier to add control planes
  7. Workloads keep running even if the control plane goes down

Knowledge check

These seven questions, drawn from the course's Kubernetes question bank, cover the etcd quorum, the role of controlPlaneEndpoint and the consequences of a partial control plane failure.

Knowledge check

Check what you have learned with this interactive quiz

7 questions
8 min
70% to pass

How it works

  • The timer starts when you click Start
  • Multiple choice, true/false and short answer questions
  • You can move between questions
  • Detailed results are shown at the end

Starts the quiz and the timer