Skip to content
Français
Account: not signed in, sync available My learning
Select theme
Français

Kubernetes operations routine: daily, weekly and monthly

By updated

Improve this page

Logo Kubernetes

A healthy Kubernetes cluster is a cluster that is watched regularly. The most costly incidents do not come from sudden failures but from slow drifts: failed Pods that get ignored, orphaned Jobs that pile up, certificates that expire silently. This guide structures the recurring checks and actions of a Kubernetes admin into three rhythms, daily, weekly and monthly, to turn reactive administration into preventive operations.

What you will learn

  • Structure a morning check that takes 5 minutes
  • Identify the failing Pods and resources before they cause an incident
  • Plan the weekly cleanup of orphaned resources (Jobs, ReplicaSets, disk pressure)
  • Check certificates and quotas every month
  • Organize these tasks with an actionable checklist per frequency
  • Keep an application available while a node is drained, in a challenge graded on the cluster state

Prerequisites

  • A Kubernetes cluster in production (v1.28+)
  • kubectl configured with administration rights
  • You have followed the guide Observe the health of the cluster (in French)
  • Access to metrics through kubectl top (metrics-server (in French) installed)

Daily: a 5-minute morning check

Every morning (or at the start of every shift), these 5 commands give an immediate view of the state of the cluster. The goal: detect any anomaly before it becomes an incident.

  1. Node health: all nodes must be Ready.

    Terminal window
    kubectl get nodes

    A NotReady or SchedulingDisabled node requires immediate investigation. Check whether maintenance is under way.

  2. API server health: the cluster must answer positively to all internal checks.

    Terminal window
    kubectl get --raw='/readyz?verbose' | grep -E '\[.\]|readyz'

    All checks must show [+]. A [-] signals a degraded component (etcd, informer-sync, etc.).

  3. Failing Pods: identify the Pods that are not running correctly.

    Terminal window
    kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded | head -30

    Pods in Pending, Failed or CrashLoopBackOff show up here. Prioritize those in the system namespaces (kube-system, monitoring).

  4. Resource consumption: spot the nodes under pressure.

    Terminal window
    kubectl top nodes

    The 80% CPU and 85% memory thresholds are operational reference points to draw attention, not the real eviction thresholds of the kubelet. Evictions are triggered by signals such as memory.available, nodefs.available or imagefs.available (see the disk pressure section below).

  5. Recent events: the warnings of the last few hours.

    Terminal window
    kubectl get events -A --sort-by='.lastTimestamp' --field-selector type=Warning | tail -20

    Recurring Warning events often signal an underlying problem: missing image, failing probe, exceeded quota.

These commands are worth gathering in a script that you run every morning, or in a dashboard. The point is not saving keystrokes: it is to always look at the same things, including on the days when everything is fine, so that you recognize what stands out.

Weekly: orphaned resources and disk pressure

Some drifts are not visible from one day to the next but build up over the week. Schedule these checks once a week.

Completed Pods and Jobs

Succeeded or Failed Pods can pile up in the API, notably with Jobs and batch workloads. Their retention time depends on the controller that created them and on the associated cleanup mechanisms.

Terminal window
# Count the completed pods
kubectl get pods -A --field-selector=status.phase==Succeeded --no-headers | wc -l
kubectl get pods -A --field-selector=status.phase==Failed --no-headers | wc -l

Jobs and CronJobs create Pods on every run. Without a retention setting, they pile up.

Terminal window
# List the CronJobs and their history
kubectl get cronjobs -A
# See the completed Jobs
kubectl get jobs -A --field-selector=status.successful=1 --no-headers | wc -l

Set spec.successfulJobsHistoryLimit and spec.failedJobsHistoryLimit in your CronJobs to limit the history automatically (default: 3 successes, 1 failure). For one-off Jobs, the TTL-after-finished controller (stable since Kubernetes 1.23) automatically deletes finished Jobs after the delay defined by .spec.ttlSecondsAfterFinished.

Orphaned ReplicaSets

Every rolling update creates a new ReplicaSet. Kubernetes keeps a history of them, 10 revisions by default, set by spec.revisionHistoryLimit. Check that this value stays reasonable:

Terminal window
# Count the ReplicaSets with 0 replicas (old revisions)
kubectl get rs -A --no-headers | awk '$3 == 0 && $4 == 0' | wc -l

For clusters with frequent deployments, reducing revisionHistoryLimit to 3 or 5 saves objects in etcd.

Quotas and limits

ResourceQuota and LimitRange prevent a namespace from consuming all the resources of the cluster. Check their usage every week.

Terminal window
# See the quotas and their consumption
kubectl get resourcequota -A
Terminal window
# Detail of a quota
kubectl describe resourcequota -n production

A quota close to its limit, when Used approaches Hard, announces that the next deployments in that namespace will fail. Plan ahead: raise the quota, or clean up unused resources.

Disk pressure

Node disk space is a critical resource that is often forgotten. Container images, logs and ephemeral volumes pile up.

The kubelet watches several eviction thresholds (configurable):

SignalDefault threshold
memory.available100Mi
nodefs.available10%
nodefs.inodesFree5%
imagefs.available15%
imagefs.inodesFree5%

These five signals all trigger the same thing: a Pod eviction. Do not confuse them with image garbage collection, which is a separate mechanism, governed by two other settings, imageGCHighThresholdPercent (85% by default) and imageGCLowThresholdPercent (80%). The first triggers the cleanup, the second says when to stop. In other words, the kubelet deletes images long before it gets to evicting Pods.

If kubectl describe node shows DiskPressure: True, the node has reached an eviction threshold. The kubelet then evicts Pods in order of quality-of-service class, BestEffort first.

The kubelet already does this housekeeping on its own, from the 85% threshold seen above. Manual cleanup with crictl rmi --prune should therefore remain a targeted action, decided in front of confirmed disk pressure, and not a recurring task: putting it in a routine means doing by hand, and less well, what the node already does.

In case of confirmed disk pressure (with containerd):

Terminal window
# On the node (via SSH): targeted action, not a routine
sudo crictl images | grep -v "IMAGE ID"
sudo crictl rmi --prune

The monthly checks cover elements with a longer lifetime: certificates, consumption trends, update backlog.

Control plane certificates (kubeadm)

If your cluster is deployed with kubeadm, the control plane certificates, excluding the certificate authority, expire after 1 year by default. Check their state:

Terminal window
# On a control plane node
sudo kubeadm certs check-expiration

Renew them before they expire:

Terminal window
sudo kubeadm certs renew all

What should you do after renewing the certificates?

After sudo kubeadm certs renew all, some components keep the old certificate in memory until they are restarted. Restart the four static Pods of the control plane: etcd, kube-apiserver, kube-controller-manager and kube-scheduler. On a single control plane, these restarts cut the API for about 41 seconds in total (measured on kubeadm 1.37.2): schedule them away from deployments. Never delete their manifests permanently from /etc/kubernetes/manifests/: the kubelet does not regenerate them by itself.

Then copy /etc/kubernetes/admin.conf over ~/.kube/config: the latter file keeps the old client certificate. It still works, which hides the problem, but kubectl will fail on the day of the old expiry date.

Terminal window
sudo cp /etc/kubernetes/admin.conf ~/.kube/config
sudo chown "$(id -u):$(id -g)" ~/.kube/config

The full procedure, with the checks, is in the Renewing certificates section of the kubeadm guide.

Ingress TLS certificates (cert-manager)

If you use cert-manager, check the state of the Certificates:

Terminal window
kubectl get certificates -A

The READY column must be True. A certificate at False calls for an investigation: cert-manager Pod logs, DNS or HTTP challenge in error, missing Secret. To read the expiry date of each certificate, which must be in the future, display the status.notAfter field:

Terminal window
kubectl get certificates -A -o custom-columns='NAMESPACE:.metadata.namespace,NOM:.metadata.name,EXPIRE:.status.notAfter'

Do a monthly review of the overall consumption of the cluster:

Terminal window
kubectl top nodes

Compare with the previous months. A steady increase signals a need for capacity: adding nodes, reviewing requests and limits, or cleaning up unused workloads.

Summary checklist by frequency

This table sums up all the checks in one place. Each row is a concrete action, sorted by rhythm.

FrequencyCheckKey commandAction if there is a problem
DailyNodes Readykubectl get nodesInvestigate the NotReady node
DailyAPI server healthykubectl get --raw='/readyz?verbose'Check the logs of the [-] component
DailyFailing Podskubectl get pods -A --field-selector=...Investigate or restart
DailyResource consumptionkubectl top nodesInvestigate if the operational reference points are exceeded
DailyWarning eventskubectl get events -A ...Identify recurring patterns
WeeklyCompleted Pods/Jobskubectl get pods --field-selector=status.phase==SucceededConfigure TTL, clean up if needed after analysis
WeeklyOrphaned ReplicaSetskubectl get rs -AAdjust revisionHistoryLimit
WeeklyNamespace quotaskubectl get resourcequota -AAdjust if close to the limit
WeeklyDisk pressurekubectl describe node (DiskPressure)Targeted cleanup if confirmed
MonthlyControl plane certificateskubeadm certs check-expirationRenew if < 30 days
MonthlyIngress certificateskubectl get certificates -ACheck cert-manager
MonthlyCapacity trendskubectl top nodes (history)Plan adjustments

When should you escalate?

Not every anomaly needs immediate action. Here is a prioritization guide to decide how urgent an intervention is.

SituationLevelAction
1 application Pod in CrashLoopBackOffNormalDiagnose within the day
Persistent problem in kube-system (DNS, CNI, kube-proxy, storage)UrgentPriority investigation, core cluster component
NotReady nodeUrgentCheck connectivity, kubelet, disk
readyz returns a [-]CriticalInvestigate immediately, degraded component
DiskPressure: True on a nodeUrgentTargeted cleanup, check logs/volumes
Certificate expires in < 7 daysCriticalRenew immediately
Namespace quota close to the limitNormalPlan an increase or a cleanup

Temper this reading for that namespace, though: a failing Pod in kube-system is not always critical. It may be an old Job Pod, a secondary addon or a harmless transient restart. Focus the investigation on the core components, those whose failure is visible everywhere else: CoreDNS, the CNI, kube-proxy, metrics-server and the storage driver.

This list is a starting point, not a destination. What it checks by hand, an observability chain watches continuously and alerts on without anyone thinking about it. As long as it does not exist, the list plays that role; once it exists, keep the part that the alerts do not cover.

Troubleshooting

The symptoms below are met while doing this routine, not while operating an application. They have one thing in common: they describe a cluster that works but accumulates, and that nobody notices until a limit is reached.

SymptomLikely causeSolution
kubectl top nodes does not workmetrics-server missing or misconfiguredInstall metrics-server with --kubelet-insecure-tls if self-signed
Evicted Pods piling upDisk or memory pressure on a nodeCheck the node conditions (DiskPressure, MemoryPressure). Targeted cleanup if confirmed
Recurring FailedScheduling eventsInsufficient resources or untolerated taintsCheck requests/limits, taints, and the number of available nodes
kubeadm certs check-expiration failsCluster not deployed with kubeadmSee the documentation of your deployment tool (kubespray, RKE2, etc.)
Quota exceeded, deployment refusedResourceQuota reached in the namespacekubectl describe resourcequota -n <ns> to see the usage

Put it into practice

This lesson describes a routine to run by hand, and the module puts it to the test on a concrete case: keeping an application available while a node is drained, then recording the cluster state in a report.

Challenge: guarantee an application's availability during maintenance

The challenge is the unaided proof of the module "Operate Kubernetes", and it is graded on the cluster state you leave behind, not on the commands you type. With no procedure, you run the paiement application with three replicas spread across the nodes, you protect it with a disruption budget, you set probes to keep traffic away from a replica that is not ready yet, then you write the routine report of your cluster in ~/routine/rapport.txt.

The validation reads six states, with no solution and no value to copy. For availability, paiement has three replicas available, a single budget targets its Pods while guaranteeing at least two replicas and still allowing maintenance, the replicas spread across as many nodes as the cluster offers, and the probes are declared. The test then drains the k8s-w1.lab worker and checks that paiement never drops below two available replicas while the drain completes. This drain only empties the paiement Pods, because the cluster is shared. For the routine, the report must match the real state: nodes, Pods in error, completed Jobs that remain and certificate expiry dates.

Passing requires 70 points out of 100. Two requirements are blocking and must pass whatever the total: the disruption budget that guarantees the minimum, and the minimum held during the drain. While you work, dsoxlab check lists what still fails. A failing state tells you what is missing, without saying how to fix it: the lesson to go back to is the one that teaches that state. The challenge relies on this lesson for the report, on Make Kubernetes applications reliable in production, Guarantee application availability (in French), Prepare a cluster maintenance (in French) and Manage nodes day to day (in French).

Key takeaways

  • A healthy cluster is watched regularly at three rhythms: daily (5 min), weekly (review of orphaned resources) and monthly (certificates, trends).
  • The 80% CPU / 85% memory thresholds are operational reference points, not the real eviction thresholds of the kubelet. Evictions rely on signals such as memory.available or nodefs.available.
  • Prefer automatic cleanup (ttlSecondsAfterFinished, revisionHistoryLimit, successfulJobsHistoryLimit) to systematic manual cleanup.
  • Control plane certificates expire after 1 year (kubeadm). Check them every month, then after kubeadm certs renew all restart the four static Pods and copy admin.conf over ~/.kube/config, without deleting the manifests.
  • A persistent problem on a core kube-system component (DNS, CNI, kube-proxy) is to be handled first, even though a failing kube-system Pod is not always critical. A failing application Pod can often wait.
  • This manual checklist prepares the transition to automated monitoring (Prometheus, Alertmanager) without replacing it.

Test your knowledge

Seven questions on what a routine should really watch: the difference between cleaning up images and evicting Pods, what a saturated quota announces, and why deleting completed Pods erases evidence.

Knowledge check

Check what you have learned with this interactive quiz

7 questions
5 min
80% to pass

How it works

  • The timer starts when you click Start
  • Multiple choice, true/false and short answer questions
  • You can move between questions
  • Detailed results are shown at the end

Starts the quiz and the timer