Safe Cluster Upgrade

A cluster upgrade is one of the most nerve-wracking tasks in Kubernetes — one wrong move can drop your workloads, break APIs, or leave nodes in a NotReady state. Yet staying on an outdated version means missing security patches and new features. The goal of this lesson is to make you confident in performing a cluster upgrade safely, step by step, so you can keep your production environment both modern and stable without downtime.

The problem this lesson solves

Running a Kubernetes cluster long-term means you must upgrade. Vendors support only a limited number of minor versions (e.g., 1.28, 1.29, 1.30). Falling behind exposes you to CVEs, broken compatibility with newer tooling (Helm, Istio, etc.), and eventual inability to upgrade at all because the version gap is too wide. The core pain is fear: “Will my apps go down? Will I break the control plane? How do I test this safely?”

This lesson directly addresses that fear by giving you a repeatable, safe upgrade process. Instead of guessing, you’ll follow a mental model that treats each component as a layer you can upgrade independently, verifying health at every step.

Core concept / mental model

Think of a cluster as a three-tier cake: 1. Control plane (master nodes) — the recipe book and chef. 2. Worker nodes — the kitchen staff carrying out orders. 3. Workloads (pods, deployments) — the dishes being cooked.

Upgrading safely means never upgrading two tiers at the same time. You always upgrade the control plane first, then the workers. Even within the control plane, you upgrade components one at a time (etcd → kube-apiserver → kube-controller-manager → kube-scheduler). If the chef’s book changes, the kitchen can still follow old notes until the staff is retrained.

Key rule: Kubernetes supports upgrading from one minor version to the next minor version only (e.g., 1.28 → 1.29). Skipping versions (1.28 → 1.30) is not supported and will break your cluster.

How it works step by step

1. Preparation — backup and check compatibility

• Back up etcd (the cluster state database).
• Read the changelog for the target version to see deprecations and API removals.
• Ensure your workloads use APIs that still exist in the new version. For example, extensions/v1beta1 Ingress was removed in 1.22.
• Check that add-ons (CNI, CSI, Ingress controller) support the target version.

2. Drain and upgrade the control plane node(s)

• On the first master node: kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
• Install the new kubeadm and kubelet version.
• Run kubeadm upgrade plan to see the upgrade path.
• Execute kubeadm upgrade apply v1.29.0 (or your target version).
• Uncordon the node: kubectl uncordon <node>
• Repeat for remaining control plane nodes if you have multiple.

3. Upgrade worker nodes

• Drain one worker node at a time.
• Install the new kubelet and kubectl versions.
• Run kubeadm upgrade node. This upgrades the local kubelet configuration.
• Restart kubelet: sudo systemctl restart kubelet
• Uncordon the node.
• Move to the next worker node — never drain more than one at a time if you need to maintain capacity.

4. Verify everything

• kubectl get nodes — all nodes should show Ready and the new version.
• kubectl get pods --all-namespaces — all pods are running.
• Test your application endpoints.

Pro tip: If you use kubectl drain, pods managed by a DaemonSet are not evicted unless you use --ignore-daemonsets. Always include that flag.

Hands-on walkthrough

Below is a complete example for upgrading a single‑control‑plane cluster from 1.28 to 1.29. All commands assume you are on the node you are upgrading.

Step 1: Check current version and backup etcd

# Show current versions
kubectl version --short

# Backup etcd (on control plane node)
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /var/backups/etcd-snapshot-$(date +%Y%m%d%H%M%S).db

Step 2: Upgrade control plane

# Drain the control plane node (if single node, you'll be self-evicting — be careful)
kubectl drain $(hostname) --ignore-daemonsets --delete-emptydir-data

# Install the new kubeadm
sudo apt-get update && sudo apt-get install -y kubeadm=1.29.0-00

# Check upgrade plan
sudo kubeadm upgrade plan

# Apply upgrade
sudo kubeadm upgrade apply v1.29.0

# Install new kubelet
sudo apt-get install -y kubelet=1.29.0-00 kubectl=1.29.0-00

# Restart kubelet
sudo systemctl daemon-reload && sudo systemctl restart kubelet

# Uncordon the node
kubectl uncordon $(hostname)

Expected output from kubeadm upgrade apply includes:

[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.29.0". Enjoy!

Step 3: Upgrade a worker node

# From your local machine (or control plane)
kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data

# SSH into worker-1
ssh worker-1

# Install new kubelet and kubeadm
sudo apt-get update && sudo apt-get install -y kubelet=1.29.0-00 kubeadm=1.29.0-00

# Upgrade node configuration
sudo kubeadm upgrade node

# Restart kubelet
sudo systemctl daemon-reload && sudo systemctl restart kubelet

# Exit and uncordon
exit
kubectl uncordon worker-1

Step 4: Verify

kubectl get nodes -o wide
# NAME          STATUS   VERSION
# control-plane Ready   v1.29.0
# worker-1      Ready   v1.29.0

kubectl get pods --all-namespaces | grep -v Running | grep -v Completed
# Should return nothing (or only Completed jobs)

Compare options / when to choose what

When to choose what: - If you must maintain uptime, always upgrade workers one at a time after the control plane is healthy. - If you have many workers (50+), consider a rolling node pool upgrade (cloud‑native way) rather than manual drain/uncordon. - If you are on a strict security audit, upgrade every minor version within the support window (usually 14 months).

Troubleshooting & edge cases

“Node is still NotReady after upgrade”

• Check kubelet logs: journalctl -u kubelet -f
• Verify CNI plugin version is compatible with the new Kubernetes version (e.g., Calico 3.27 works with 1.29).
• Ensure the node’s kubelet.conf points to the correct API server.

“kubeadm upgrade apply fails with 'API version mismatch'”

• You tried to skip a minor version. Only upgrade 1.28 → 1.29, not 1.28 → 1.30 directly.
• Downgrade kubeadm/ kubelet to the intermediate version, then perform two sequential upgrades.

“Pods are stuck in ContainerCreating after upgrade”

• The CSI driver or CNI is outdated. Upgrade those add‑ons before the node upgrade, or immediately after.
• Common symptom: kubectl describe pod <pod> shows failed to create container: unknown runtime.

“etcd cluster health check fails”

• If you run a stacked etcd (etcd runs on same node as control plane), upgrade etcd first via kubeadm upgrade apply. If external etcd, upgrade etcd members independently.
• Use ETCDCTL_API=3 etcdctl endpoint health --cluster to diagnose.

What you learned & what's next

You now understand the safe upgrade pattern: backup → control plane (one node at a time) → workers (one at a time) → validate. This method ensures zero‑downtime application updates if you maintain at least two replicas of each service. You have practised the exact commands to upgrade a kubeadm‑based cluster from 1.28 to 1.29.

Next lesson: After upgrading your cluster, you should verify that all resources are working as expected. The next step in the track is Perform cluster health checks, where you’ll learn to proactively monitor API server responsiveness, node conditions, and etcd latency using kubectl plugins and metrics.