The issue opens pre-filled with this page title and URL.
A Kubernetes cluster with a single control plane is a SPOF (Single Point of Failure). If the master node goes down, nobody can deploy, scale or manage workloads any more, even though the applications keep running. High availability (HA) of the control plane removes this risk by spreading the critical components across several nodes.
This guide covers the HA architecture for the CKA certification and for production environments.
What you will learn
Why a single control plane is a risk
The two HA topologies, stacked etcd and external etcd
Setting up an HA cluster with kubeadm
The role of the load balancer in front of the API Server
The failover scenarios and how to handle them
Upgrading a control plane and replacing one
Building an HA cluster with three control planes using kind, for practice
Why high availability?
The control plane does not run your applications: it decides what runs and
where. As long as it works, its decisions are applied again and again. When it
disappears, the cluster freezes in its last known state, and nothing
tells the users of the applications. Understanding what exactly stops is the
prerequisite for choosing an HA topology.
What is the problem with a single control plane?
In a standard cluster, the control plane (a single node) hosts:
Component
Role
Impact if unavailable
kube-apiserver
Entry point for all requests
No kubectl command works
etcd
Stores the state of the cluster
Configuration lost, no resource can be created
kube-scheduler
Places Pods on nodes
New Pods stay Pending
kube-controller-manager
Manages Deployments, ReplicaSets...
No reconciliation, no autoscaling
Swipe to see more
One nuance changes how you live through the incident: if the control plane
goes down, existing Pods keep running on the workers, and your
applications stay online. What you lose is the ability to act: no more
creation, no more changes, no more diagnosis through the API. In other words,
the outage is invisible to users until something else breaks, at which
point you have no way left to intervene.
What does HA bring?
High availability does not make the cluster faster or more powerful: it
lets it survive the loss of a machine. Here is what you keep with
three control planes, the recommended minimum:
one node can fail with no impact for users;
kubectl keeps working;
you upgrade the nodes one at a time, with no interruption;
etcd keeps its quorum with two nodes out of three.
What are the two HA topologies?
Kubernetes supports two architectures for high availability, which differ in where etcd lives.
Stacked etcd topology (recommended for the CKA)
Each control plane hosts both the Kubernetes components and etcd:
This topology is the simplest to deploy and maintain: it needs fewer nodes
and reuses the standard kubeadm configuration. Its downside is the coupling:
losing a control plane means losing an etcd member at the same time.
External etcd topology
The etcd cluster is separate from the Kubernetes control planes:
External etcd topology: three control planes that host only kube-apiserver, kube-scheduler and kube-controller-manager each query a member of a dedicated three-member etcd cluster replicated by Raft.Dedicated etcd cluster (Raft replication)Control Plane 1Control Plane 2Control Plane 3etcd 1etcd 2etcd 3kube-apiserverkube-schedulerkube-controller-managerkube-apiserverkube-schedulerkube-controller-managerkube-apiserverkube-schedulerkube-controller-manager
This topology isolates failures, since etcd is independent, and can be sized
more freely. In return, it is more complex to deploy and needs more nodes, six
at a minimum.
For the CKA exam, focus on the stacked etcd topology: it is what
kubeadm deploys by default in high availability mode, and therefore what
the questions will be about.
What do you need before building an HA cluster?
The setup needs more than a test cluster: you need a stable virtual
address in front of the API servers, and an odd number of control plane
nodes so that the etcd quorum can break ties.
Infrastructure
Three nodes are the minimum: with two, losing a single one breaks the etcd
quorum and the cluster becomes unable to record any change.
3 control plane nodes (2 CPU, 2 GB RAM, lab minimum, not production sizing)
1 redundant load balancer in front of the API Servers (HAProxy + keepalived, Nginx, or a cloud LB)
Network connectivity between all nodes
Open ports: 6443 (API), 2379-2380 (etcd), 10250-10259 (kubelet/components)
This guide is for self-managed kubeadm clusters, on virtual or physical
machines. On a managed cluster, EKS, GKE or AKS, the question does not
arise: control plane high availability is part of what you buy, and you have
no say in it.
Software
Versions must be aligned before initialization: the kubeadmpreflight
checks fail if the node's kubelet is newer than the targeted control plane
version.
kubeadm, kubelet, kubectl installed on all nodes
Container runtime (containerd recommended)
Same Kubernetes version on all nodes
How do you configure the load balancer?
The load balancer is essential to distribute requests across the API Servers.
With HAProxy
Create /etc/haproxy/haproxy.cfg on the load balancer node:
global
log /dev/log local0
maxconn 2000
user haproxy
group haproxy
daemon
defaults
log global
mode tcp
option tcplog
option dontlognull
timeout connect 5s
timeout client 50s
timeout server 50s
frontend kubernetes-api
bind *:6443
default_backend kubernetes-masters
backend kubernetes-masters
balance roundrobin
option tcp-check
server master1 192.168.1.10:6443 check fall 3 rise 2
server master2 192.168.1.11:6443 check fall 3 rise 2
server master3 192.168.1.12:6443 check fall 3 rise 2
Start HAProxy:
Terminal window
sudosystemctlenable--nowhaproxy
Check:
Terminal window
# Test the connection to the load balancer
nc-zv<LOAD_BALANCER_IP>6443
Be careful not to move the problem without solving it. A single HAProxy,
with no virtual address and no redundancy, becomes a single point of
failure in its own right: three control planes behind a load balancer that
goes down is still an unreachable cluster. In production, pair HAProxy with
keepalived to get a floating virtual address, or hand the role over to a
managed load balancer from your provider.
Advanced check:
Terminal window
# Check that the Kubernetes API really answers
curl-khttps://<LB_IP>:6443/readyz
How do you initialize the first control plane?
The first node is special: it creates the cluster CA, the etcd database
and the Secret that holds the shared certificates. Two parameters of this
step are irreversible or time-limited: controlPlaneEndpoint, which fixes
the address everyone will use to reach the API, and the certificate-key
produced by --upload-certs, valid for two hours.
Prepare the kubeadm configuration file
Create kubeadm-config.yaml:
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.37.2
controlPlaneEndpoint: "LOAD_BALANCER_IP:6443"
networking:
podSubnet: "10.244.0.0/16"
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: "192.168.1.10"
bindPort: 6443
The controlPlaneEndpoint option is mandatory in high availability, and
it points to the load balancer. Remember this above all: once the
cluster is initialized, it can no longer be changed. That is why you
should set it even on a single control plane cluster, or you will have to
rebuild everything the day high availability becomes necessary.
Watch the spelling: the configuration file field is written
controlPlaneEndpoint, in camelCase, while the equivalent command-line
flag is written --control-plane-endpoint, in lowercase with hyphens.
Both forms do the same thing, and kubeadm init --control-plane-endpoint "LOAD_BALANCER_IP:6443" avoids the file for a
quick trial. Mixing the two spellings produces an unknown flag with no
hint about the right one.
Flannel is a simple option to start with. You can also choose Calico, Cilium or another CNI depending on your needs. The address below targets a specific Flannel release, not the development branch: the same command therefore always installs the same manifest.
The first control plane produced a special join command, which differs
from the workers' one by --control-plane and a certificate key. It is
what lets the following nodes fetch the cluster certificates and become
etcd members, not mere clients.
On each additional control plane (master2, master3):
Three checks, in this order, because they cover different layers: the
nodes as Kubernetes sees them, the etcd members as etcd itself sees
them, and the internal health of the API Server. A cluster can show three
Ready nodes while an etcd member has left the cluster.
Node state
The ROLES column must show control-plane for the three nodes, and
kubectl get nodes -o wide also reveals the advertised IP addresses, which
help you check that they match the load balancer backends.
Terminal window
kubectlgetnodes-owide
etcd state
kubectl cannot see etcd: you must query the etcd cluster directly,
presenting the certificates kubeadm placed in /etc/kubernetes/pki/etcd/.
The three members must appear with the started status. The
etcd lesson details how
it works and how it fails.
The etcdctl binary is not installed on a kubeadm node. That is the first
obstacle, and it is a surprise: which etcdctl exits with 1. The binary lives
in the etcd container, a distroless image registry.k8s.io/etcd:3.7.0-0, and
you reach it with an exec:
The IS LEARNER column is false for a voting member. A member at
true is still catching up and does not count toward the quorum, which
explains why a cluster with four rows can have only three that vote.
How do you check the health of each member?
member list says who is declared, not who answers. The distinction
matters, and endpoint health --cluster makes it:
The command exits with 1 as soon as a single member is missing, even
though the cluster works perfectly. So do not wire an alert to this exit code
alone: it signals a degradation, not an outage.
How do you check the health of the components?
kubectl get componentstatuses is deprecated and returns incomplete
information. The /readyz and /livez endpoints, queried with
--raw, list each internal check of the API Server one by one, etcd
included.
Terminal window
# Modern endpoints (componentstatuses is deprecated)
kubectlget--raw='/readyz?verbose'
kubectlget--raw='/livez?verbose'
What happens when a control plane fails?
High availability is not declared, it is verified. The test is to
deliberately stop a control plane and confirm that write operations
still go through. The table that follows generalizes the result to all
combinations of failures.
How do you test the failure of one control plane?
Stopping the kubelet is enough: the control plane components are static
pods driven by the kubelet, so they stop with it.
How does the cluster behave depending on the failures?
The threshold sits between one and two simultaneous failures out of three
nodes. The switch is abrupt: as long as the quorum holds, nothing
changes for the user; once it is lost, no write succeeds any more.
Failures
etcd
API Server
Observed impact
0 of 3
quorum
serves
normal operation
1 of 3
quorum at 2 of 3
serves
no change for the user
2 of 3
quorum lost
no longer answers
Unable to connect to the server: unexpected EOF
3 of 3
stopped
stopped
cluster unusable
Swipe to see more
The two bold rows were measured on a cluster with three control planes.
With one failure, a kubectl run succeeds while the machine is still off: the
cluster does not just answer, it accepts writes. With two failures, the
connection is cut dead, with no intermediate message and no degraded mode.
Losing the etcd quorum is not a gradual degradation, it is a stop. The
cluster can no longer accept any state change. In the measurement above, with
two control planes out of three stopped, kubectl receives no answer at all.
Do not count on a "read-only" mode.
etcd requires a majority of members to be working, which gives three
usual cases: with 3 nodes you need 2, with 5 you need 3, with 7 you
need 4. Hence the odd-number rule: going from 3 to 4 members does not change
the number of failures tolerated, which stays at one, while adding a machine
to maintain and one more member that can fail.
How do you maintain an HA cluster?
An HA cluster is maintained without service interruption, provided you
never touch two nodes at the same time: the etcd quorum must be kept for
the whole operation. The two procedures below cover the planned upgrade and
the replacement of a node that is permanently lost.
Before any version upgrade, check the current stable version on
kubernetes.io/releases: it changes every
few weeks. The full procedure, with its compatibility rules between versions,
is detailed in the lesson
upgrade a Kubernetes cluster.
How do you replace a failed control plane?
Replacing a dead control plane does not mean adding another one: you must
first remove the old etcd member, or the cluster keeps waiting for it
when counting its quorum. The order of operations therefore matters more
than the commands themselves.
If a control plane is permanently lost, follow these four steps. The etcdctl commands run in the etcd container of a healthy node, with the certificate options seen above, since the binary is not installed on the node:
Remove the etcd member
Terminal window
# Find the ID of the lost member, from the etcd Pod of a healthy node
The three failures below account for most of the blockages during a
first HA setup. They have a common point: the message shown rarely
names the real cause, so you have to check the layer underneath.
The join fails with "certificate has expired"
The certificate key produced by kubeadm init has a lifetime of two
hours. After that, the join command stays syntactically valid but fails to
fetch the certificates, with a message that does not point to the real
cause.
Regenerate it by running the certificate upload again:
Terminal window
sudokubeadminitphaseupload-certs--upload-certs
etcd refuses to start after a reboot
A missing or expired certificate prevents etcd from starting. These two
commands show whether the files are there and until when they are valid:
When kubectltimes out while the control planes are healthy, the culprit
is almost always the load balancer: wrong address in the kubeconfig,
backends declared on the wrong port, or a health check that takes every node
out of the service.
These three checks read from the outside in: if the first one fails, the
next two will tell you nothing useful.
The load balancer answers: nc -zv <LB_IP> 6443
At least one API Server is alive: curl -k https://<MASTER_IP>:6443/healthz
The HAProxy health checks pass, otherwise the load balancer empties its pool
How do you build an HA cluster with kind for practice?
You do not always have three machines available for an HA control plane. kind creates a cluster inside Docker containers, and it accepts several control plane nodes. Each role: control-plane line in the configuration file adds one control plane node, and therefore one more etcd member.
kind-ha.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: control-plane
- role: control-plane
The cluster is created with this file and a pinned node image. The name ha lets you find its containers with docker ps.
Unlike the kubeadm installation seen above, you do not configure a load balancer. As soon as a kind cluster has several control planes, kind starts an extra container itself, ha-external-load-balancer, which plays the role of HAProxy in front of the three API servers. The kubeconfig file that kind writes points to this load balancer, not to one control plane in particular. Check with kubectl get nodes that the three nodes are Ready, then with docker ps --filter name=ha- that the load balancer container is there next to the three nodes. Then stop one control plane container with docker stop: kubectl get nodes must keep answering through the load balancer, which is exactly the property this lesson is trying to achieve.
Put it into practice
The only test that counts is to stop a control plane and watch the API answer. The challenge below asks you to do it without help.
Challenge: install a cluster and make its control plane highly available
The challenge is the unaided proof of the module "Install a cluster", and it is graded on the state of the machines you leave behind, not on the commands you typed. Two jobs await you. On the kubeadm cluster of the lab, a worker has been removed from the cluster and reset: you prepare it and join it so that it survives a reboot, then you force a Pod onto it. On the separate host k8s-kind.lab, you then build a kind cluster named ha, with a pinned kindest/node image, three control planes and the load balancer that kind itself creates in front of their API servers. Its etcd has three members, and the API must keep answering through the load balancer when one of the control planes stops. No procedure is given.
The validation reads seven states, with no solution and no value to copy. It checks that the worker is Ready with a kubelet in version 1.37, that swap is durably off, that containerd uses the right cgroup driver and that the forced Pod runs on the worker. It also reads the three control plane nodes and the three etcd members through the load balancer address. For the last state, the grader stops a control plane itself, queries the API through the load balancer, then starts it again: each check therefore takes more than a minute.
Passing requires 70 points out of 100, and there is no hint. Two states are blocking and must pass whatever the total: the worker joined to the cluster, and the API still answering when a control plane goes down. The challenge builds on this lesson, which also teaches the kind example with three control planes, on Join a worker to a kubeadm cluster and on Install a cluster with kubeadm.
Key takeaways
3 control planes minimum for high availability
A load balancer is mandatory in front of the API Servers
Stacked etcd topology = simpler, recommended for the CKA
etcd quorum = a majority of members is required
Upgrade one node at a time to keep availability
--upload-certs makes it easier to add control planes
Workloads keep running even if the control plane goes down
Knowledge check
These seven questions, drawn from the course's Kubernetes question bank, cover
the etcd quorum, the role of controlPlaneEndpoint and the consequences
of a partial control plane failure.
Knowledge check
Check what you have learned with this interactive quiz
7 questions
8 min
70% to pass
How it works
The timer starts when you click Start
Multiple choice, true/false and short answer questions
You can move between questions
Detailed results are shown at the end
Starts the quiz and the timer
⏱️--:--
0 / 7
Review
(0/0)
--:--
0%
0Correct
0To review
0:00Time
-Level
Score by domain
A domain groups several guides and gets several questions, so its score is more reliable than a per-topic score, which often rests on a single question.
Summary
Strengths
No topic passed at 80% on at least two questions yet.