A healthy Kubernetes cluster is a cluster that is watched regularly. The most costly incidents do not come from sudden failures but from slow drifts: failed Pods that get ignored, orphaned Jobs that pile up, certificates that expire silently. This guide structures the recurring checks and actions of a Kubernetes admin into three rhythms, daily, weekly and monthly, to turn reactive administration into preventive operations.
What you will learn
- Structure a morning check that takes 5 minutes
- Identify the failing Pods and resources before they cause an incident
- Plan the weekly cleanup of orphaned resources (Jobs, ReplicaSets, disk pressure)
- Check certificates and quotas every month
- Organize these tasks with an actionable checklist per frequency
- Keep an application available while a node is drained, in a challenge graded on the cluster state
Prerequisites
- A Kubernetes cluster in production (v1.28+)
kubectlconfigured with administration rights- You have followed the guide Observe the health of the cluster (in French)
- Access to metrics through
kubectl top(metrics-server (in French) installed)
Daily: a 5-minute morning check
Every morning (or at the start of every shift), these 5 commands give an immediate view of the state of the cluster. The goal: detect any anomaly before it becomes an incident.
-
Node health: all nodes must be
Ready.Terminal window kubectl get nodesA
NotReadyorSchedulingDisablednode requires immediate investigation. Check whether maintenance is under way. -
API server health: the cluster must answer positively to all internal checks.
Terminal window kubectl get --raw='/readyz?verbose' | grep -E '\[.\]|readyz'All checks must show
[+]. A[-]signals a degraded component (etcd, informer-sync, etc.). -
Failing Pods: identify the Pods that are not running correctly.
Terminal window kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded | head -30Pods in
Pending,FailedorCrashLoopBackOffshow up here. Prioritize those in the system namespaces (kube-system,monitoring). -
Resource consumption: spot the nodes under pressure.
Terminal window kubectl top nodesThe 80% CPU and 85% memory thresholds are operational reference points to draw attention, not the real eviction thresholds of the kubelet. Evictions are triggered by signals such as
memory.available,nodefs.availableorimagefs.available(see the disk pressure section below). -
Recent events: the warnings of the last few hours.
Terminal window kubectl get events -A --sort-by='.lastTimestamp' --field-selector type=Warning | tail -20Recurring
Warningevents often signal an underlying problem: missing image, failing probe, exceeded quota.
These commands are worth gathering in a script that you run every morning, or in a dashboard. The point is not saving keystrokes: it is to always look at the same things, including on the days when everything is fine, so that you recognize what stands out.
Weekly: orphaned resources and disk pressure
Some drifts are not visible from one day to the next but build up over the week. Schedule these checks once a week.
Completed Pods and Jobs
Succeeded or Failed Pods can pile up in the API, notably with Jobs and batch workloads. Their retention time depends on the controller that created them and on the associated cleanup mechanisms.
# Count the completed podskubectl get pods -A --field-selector=status.phase==Succeeded --no-headers | wc -lkubectl get pods -A --field-selector=status.phase==Failed --no-headers | wc -lJobs and CronJobs create Pods on every run. Without a retention setting, they pile up.
# List the CronJobs and their historykubectl get cronjobs -A
# See the completed Jobskubectl get jobs -A --field-selector=status.successful=1 --no-headers | wc -lSet spec.successfulJobsHistoryLimit and spec.failedJobsHistoryLimit in your CronJobs to limit the history automatically (default: 3 successes, 1 failure). For one-off Jobs, the TTL-after-finished controller (stable since Kubernetes 1.23) automatically deletes finished Jobs after the delay defined by .spec.ttlSecondsAfterFinished.
Orphaned ReplicaSets
Every rolling update creates a new ReplicaSet. Kubernetes keeps a history of them, 10 revisions by default, set by spec.revisionHistoryLimit. Check that this value stays reasonable:
# Count the ReplicaSets with 0 replicas (old revisions)kubectl get rs -A --no-headers | awk '$3 == 0 && $4 == 0' | wc -lFor clusters with frequent deployments, reducing revisionHistoryLimit to 3 or 5 saves objects in etcd.
Quotas and limits
ResourceQuota and LimitRange prevent a namespace from consuming all the resources of the cluster. Check their usage every week.
# See the quotas and their consumptionkubectl get resourcequota -A# Detail of a quotakubectl describe resourcequota -n productionA quota close to its limit, when Used approaches Hard, announces that the next deployments in that namespace will fail. Plan ahead: raise the quota, or clean up unused resources.
Disk pressure
Node disk space is a critical resource that is often forgotten. Container images, logs and ephemeral volumes pile up.
The kubelet watches several eviction thresholds (configurable):
| Signal | Default threshold |
|---|---|
memory.available | 100Mi |
nodefs.available | 10% |
nodefs.inodesFree | 5% |
imagefs.available | 15% |
imagefs.inodesFree | 5% |
These five signals all trigger the same thing: a Pod eviction. Do not confuse them with image garbage collection, which is a separate mechanism, governed by two other settings, imageGCHighThresholdPercent (85% by default) and imageGCLowThresholdPercent (80%). The first triggers the cleanup, the second says when to stop. In other words, the kubelet deletes images long before it gets to evicting Pods.
If kubectl describe node shows DiskPressure: True, the node has reached an eviction threshold. The kubelet then evicts Pods in order of quality-of-service class, BestEffort first.
The kubelet already does this housekeeping on its own, from the 85% threshold seen above. Manual cleanup with crictl rmi --prune should therefore remain a targeted action, decided in front of confirmed disk pressure, and not a recurring task: putting it in a routine means doing by hand, and less well, what the node already does.
In case of confirmed disk pressure (with containerd):
# On the node (via SSH): targeted action, not a routinesudo crictl images | grep -v "IMAGE ID"sudo crictl rmi --pruneMonthly: certificates and capacity trends
The monthly checks cover elements with a longer lifetime: certificates, consumption trends, update backlog.
Control plane certificates (kubeadm)
If your cluster is deployed with kubeadm, the control plane certificates, excluding the certificate authority, expire after 1 year by default. Check their state:
# On a control plane nodesudo kubeadm certs check-expirationRenew them before they expire:
sudo kubeadm certs renew allWhat should you do after renewing the certificates?
After sudo kubeadm certs renew all, some components keep the old certificate in memory until they are restarted. Restart the four static Pods of the control plane: etcd, kube-apiserver, kube-controller-manager and kube-scheduler. On a single control plane, these restarts cut the API for about 41 seconds in total (measured on kubeadm 1.37.2): schedule them away from deployments. Never delete their manifests permanently from /etc/kubernetes/manifests/: the kubelet does not regenerate them by itself.
Then copy /etc/kubernetes/admin.conf over ~/.kube/config: the latter file keeps the old client certificate. It still works, which hides the problem, but kubectl will fail on the day of the old expiry date.
sudo cp /etc/kubernetes/admin.conf ~/.kube/configsudo chown "$(id -u):$(id -g)" ~/.kube/configThe full procedure, with the checks, is in the Renewing certificates section of the kubeadm guide.
Ingress TLS certificates (cert-manager)
If you use cert-manager, check the state of the Certificates:
kubectl get certificates -AThe READY column must be True. A certificate at False calls for an investigation: cert-manager Pod logs, DNS or HTTP challenge in error, missing Secret. To read the expiry date of each certificate, which must be in the future, display the status.notAfter field:
kubectl get certificates -A -o custom-columns='NAMESPACE:.metadata.namespace,NOM:.metadata.name,EXPIRE:.status.notAfter'Capacity trends
Do a monthly review of the overall consumption of the cluster:
kubectl top nodesCompare with the previous months. A steady increase signals a need for capacity: adding nodes, reviewing requests and limits, or cleaning up unused workloads.
Summary checklist by frequency
This table sums up all the checks in one place. Each row is a concrete action, sorted by rhythm.
| Frequency | Check | Key command | Action if there is a problem |
|---|---|---|---|
| Daily | Nodes Ready | kubectl get nodes | Investigate the NotReady node |
| Daily | API server healthy | kubectl get --raw='/readyz?verbose' | Check the logs of the [-] component |
| Daily | Failing Pods | kubectl get pods -A --field-selector=... | Investigate or restart |
| Daily | Resource consumption | kubectl top nodes | Investigate if the operational reference points are exceeded |
| Daily | Warning events | kubectl get events -A ... | Identify recurring patterns |
| Weekly | Completed Pods/Jobs | kubectl get pods --field-selector=status.phase==Succeeded | Configure TTL, clean up if needed after analysis |
| Weekly | Orphaned ReplicaSets | kubectl get rs -A | Adjust revisionHistoryLimit |
| Weekly | Namespace quotas | kubectl get resourcequota -A | Adjust if close to the limit |
| Weekly | Disk pressure | kubectl describe node (DiskPressure) | Targeted cleanup if confirmed |
| Monthly | Control plane certificates | kubeadm certs check-expiration | Renew if < 30 days |
| Monthly | Ingress certificates | kubectl get certificates -A | Check cert-manager |
| Monthly | Capacity trends | kubectl top nodes (history) | Plan adjustments |
When should you escalate?
Not every anomaly needs immediate action. Here is a prioritization guide to decide how urgent an intervention is.
| Situation | Level | Action |
|---|---|---|
1 application Pod in CrashLoopBackOff | Normal | Diagnose within the day |
Persistent problem in kube-system (DNS, CNI, kube-proxy, storage) | Urgent | Priority investigation, core cluster component |
NotReady node | Urgent | Check connectivity, kubelet, disk |
readyz returns a [-] | Critical | Investigate immediately, degraded component |
DiskPressure: True on a node | Urgent | Targeted cleanup, check logs/volumes |
| Certificate expires in < 7 days | Critical | Renew immediately |
| Namespace quota close to the limit | Normal | Plan an increase or a cleanup |
Temper this reading for that namespace, though: a failing Pod in kube-system is not always critical. It may be an old Job Pod, a secondary addon or a harmless transient restart. Focus the investigation on the core components, those whose failure is visible everywhere else: CoreDNS, the CNI, kube-proxy, metrics-server and the storage driver.
This list is a starting point, not a destination. What it checks by hand, an observability chain watches continuously and alerts on without anyone thinking about it. As long as it does not exist, the list plays that role; once it exists, keep the part that the alerts do not cover.
Troubleshooting
The symptoms below are met while doing this routine, not while operating an application. They have one thing in common: they describe a cluster that works but accumulates, and that nobody notices until a limit is reached.
| Symptom | Likely cause | Solution |
|---|---|---|
kubectl top nodes does not work | metrics-server missing or misconfigured | Install metrics-server with --kubelet-insecure-tls if self-signed |
Evicted Pods piling up | Disk or memory pressure on a node | Check the node conditions (DiskPressure, MemoryPressure). Targeted cleanup if confirmed |
Recurring FailedScheduling events | Insufficient resources or untolerated taints | Check requests/limits, taints, and the number of available nodes |
kubeadm certs check-expiration fails | Cluster not deployed with kubeadm | See the documentation of your deployment tool (kubespray, RKE2, etc.) |
| Quota exceeded, deployment refused | ResourceQuota reached in the namespace | kubectl describe resourcequota -n <ns> to see the usage |
Put it into practice
This lesson describes a routine to run by hand, and the module puts it to the test on a concrete case: keeping an application available while a node is drained, then recording the cluster state in a report.
Challenge: guarantee an application's availability during maintenance
The challenge is the unaided proof of the module "Operate Kubernetes", and it is graded on the cluster state you leave behind, not on the commands you type. With no procedure, you run the paiement application with three replicas spread across the nodes, you protect it with a disruption budget, you set probes to keep traffic away from a replica that is not ready yet, then you write the routine report of your cluster in ~/routine/rapport.txt.
The validation reads six states, with no solution and no value to copy. For availability, paiement has three replicas available, a single budget targets its Pods while guaranteeing at least two replicas and still allowing maintenance, the replicas spread across as many nodes as the cluster offers, and the probes are declared. The test then drains the k8s-w1.lab worker and checks that paiement never drops below two available replicas while the drain completes. This drain only empties the paiement Pods, because the cluster is shared. For the routine, the report must match the real state: nodes, Pods in error, completed Jobs that remain and certificate expiry dates.
Passing requires 70 points out of 100. Two requirements are blocking and must pass whatever the total: the disruption budget that guarantees the minimum, and the minimum held during the drain. While you work, dsoxlab check lists what still fails. A failing state tells you what is missing, without saying how to fix it: the lesson to go back to is the one that teaches that state. The challenge relies on this lesson for the report, on Make Kubernetes applications reliable in production, Guarantee application availability (in French), Prepare a cluster maintenance (in French) and Manage nodes day to day (in French).
Key takeaways
- A healthy cluster is watched regularly at three rhythms: daily (5 min), weekly (review of orphaned resources) and monthly (certificates, trends).
- The 80% CPU / 85% memory thresholds are operational reference points, not the real eviction thresholds of the kubelet. Evictions rely on signals such as
memory.availableornodefs.available. - Prefer automatic cleanup (
ttlSecondsAfterFinished,revisionHistoryLimit,successfulJobsHistoryLimit) to systematic manual cleanup. - Control plane certificates expire after 1 year (kubeadm). Check them every month, then after
kubeadm certs renew allrestart the four static Pods and copyadmin.confover~/.kube/config, without deleting the manifests. - A persistent problem on a core
kube-systemcomponent (DNS, CNI, kube-proxy) is to be handled first, even though a failingkube-systemPod is not always critical. A failing application Pod can often wait. - This manual checklist prepares the transition to automated monitoring (Prometheus, Alertmanager) without replacing it.
Test your knowledge
Seven questions on what a routine should really watch: the difference between cleaning up images and evicting Pods, what a saturated quota announces, and why deleting completed Pods erases evidence.
Knowledge check
Check what you have learned with this interactive quiz
How it works
- The timer starts when you click Start
- Multiple choice, true/false and short answer questions
- You can move between questions
- Detailed results are shown at the end
Starts the quiz and the timer
Review
(0/0)Score by domain
A domain groups several guides and gets several questions, so its score is more reliable than a per-topic score, which often rests on a single question.
Summary
Strengths
No topic passed at 80% on at least two questions yet.
Weak areas
No topic below the pass mark.
Recommended next
Hints before you try again?
Continue the course: Level 4 exam: administer a cluster
A new full quiz with random questions
Work again only on the questions you missed