Overview
This article walks the installer through the process of upgrading K8S from a previous version.
In working through the installation, references will be made as appropriate to provide context.
System Parameters/Versions:
-
Base OS:
-
Amazon Linux 2023:
ami-05576a079321f21f8
-
Rocky 8.10
-
K8S
-
current version: v1.30.9
-
upgrade version: v1.31.7
Requirements:
K8S must be upgraded incrementally from each version. As an example, if upgrading from v1.29 to v1.31, you must install each increment and repeat the process until the desired version is installed:
-
v1.29 → v1.30
-
v1.30 → v1.31
In this example, upgrading from v1.30 to v1.31 so only need to apply the process once.
Upgrade K8S
Step 1: Check Current Versions
kubeadm
, kubelet
, and kubectl
versions must match. Verify that the current versions are the same:
kubeadm
version:
Example output:
kubeadm version: &http://version.Info {Major:"1", Minor:"30", GitVersion:"v1.30.9", GitCommit:"a87cd6906120a367bf6787420e943103a463acba", GitTreeState:"clean", BuildDate:"2025-01-15T14:40:03Z", GoVersion:"go1.22.10", Compiler:"gc", Platform:"linux/amd64"}
kubectl
version:
Example output:
Client Version: v1.30.9
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
kubelet
version:
Example output:
NAME STATUS ROLES AGE VERSION
gpu Ready <none> 48d v1.30.9
master Ready control-plane 48d v1.30.9
worker Ready <none> 48d v1.30.9
Step 2: Update Base OS
On ALL nodes in the cluster (master, worker, and gpu). Each base OS has different package updates.
# update OS
dnf check-release-update
sudo dnf update -y
# update system
dnf update -y
Step 3: Update Repository Config
On ALL nodes in the cluster (master, worker, and gpu), update repository config file to desired version. In this example, using v1.31:
# set version
export K8S_V="1.31"
# set repository and repo
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v$K8S_V/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v$K8S_V/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF
Step 4: Upgrade Control Plane Kubeadm
On the master node, upgrade kubeadm
. Worker/GPUs nodes will be updated separately.
Get list of possible upgrade versions:
sudo dnf list --showduplicates kubeadm --disableexcludes=kubernetes
Example output:
Installed Packages
kubeadm.x86_64 1.30.9-150500.1.1 @kubernetes
Available Packages
kubeadm.aarch64 1.31.7-150500.1.1 kubernetes
kubeadm.ppc64le 1.31.7-150500.1.1 kubernetes
kubeadm.s390x 1.31.7-150500.1.1 kubernetes
kubeadm.src 1.31.7-150500.1.1 kubernetes
Install new kubeadm
version:
# upgrade kubeadm
sudo dnf install -y kubeadm-"1.31*" --disableexcludes=kubernetes
Check new version:
Example output:
kubeadm version: &http://version.Info {Major:"1", Minor:"31", GitVersion:"v1.31.7", GitCommit:"da53587841b4960dc3bd2af1ec6101b57c79aff4", GitTreeState:"clean", BuildDate:"2025-03-11T20:02:21Z", GoVersion:"go1.23.6", Compiler:"gc", Platform:"linux/amd64"}
Step 5: Upgrade Control Plane Components
On the master node, upgrade control plane components.
Run a plan to show what can/will be updated:
sudo kubeadm upgrade plan
Example output shows the possible v1.30 and 1.31 minor/patch upgrades. For this example, upgrading minor version to 1.31.7:
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: 1.30.9
[upgrade/versions] kubeadm version: v1.31.7
I0313 23:37:23.319175 600006 version.go:261] remote version is much newer: v1.32.3; falling back to: stable-1.31
[upgrade/versions] Target version: v1.31.7
[upgrade/versions] Latest version in the v1.30 series: v1.30.11
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT NODE CURRENT TARGET
kubelet gpu v1.30.9 v1.30.11
kubelet master v1.30.9 v1.30.11
kubelet worker v1.30.9 v1.30.11
Upgrade to the latest version in the v1.30 series:
COMPONENT NODE CURRENT TARGET
kube-apiserver master v1.30.9 v1.30.11
kube-controller-manager master v1.30.9 v1.30.11
kube-scheduler master v1.30.9 v1.30.11
kube-proxy 1.30.9 v1.30.11
CoreDNS v1.11.3 v1.11.3
etcd master 3.5.15-0 3.5.15-0
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.30.11
_____________________________________________________________________
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT NODE CURRENT TARGET
kubelet gpu v1.30.9 v1.31.7
kubelet master v1.30.9 v1.31.7
kubelet worker v1.30.9 v1.31.7
Upgrade to the latest stable version:
COMPONENT NODE CURRENT TARGET
kube-apiserver master v1.30.9 v1.31.7
kube-controller-manager master v1.30.9 v1.31.7
kube-scheduler master v1.30.9 v1.31.7
kube-proxy 1.30.9 v1.31.7
CoreDNS v1.11.3 v1.11.3
etcd master 3.5.15-0 3.5.15-0
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.31.7
_____________________________________________________________________
Upgrade to minor version v1.31.7 (can take up to five mins to complete):
kubeadm upgrade apply v1.31.7 -y
When complete, view kube-system
pods. If the upgrade was successful, the kube*
pods will have a newer age than the others:
kubectl get pods -n kube-system
Example output (upgrade was executed 13h ago):
NAME READY STATUS RESTARTS AGE
coredns-55cb58b774-m7b6h 1/1 Running 3 (2d ago) 24d
coredns-55cb58b774-s5qqr 1/1 Running 4 (20h ago) 24d
etcd-master 1/1 Running 0 13h
kube-apiserver-master 1/1 Running 0 13h
kube-controller-manager-master 1/1 Running 0 13h
kube-proxy-4bg5s 1/1 Running 0 13h
kube-proxy-4mk8s 1/1 Running 0 13h
kube-proxy-7h8ww 1/1 Running 0 13h
kube-scheduler-master 1/1 Running 0 13h
metrics-server-5f6bd8776f-l6lg8 1/1 Running 3 (20h ago) 22d
Step 6: Upgrade Control Plane Kubectl and Kubelet
On the master node, upgrade kubectl
and kubelet
components. Versions need to match kubeadm
from previous steps (v1.31.7):
sudo dnf install -y kubelet-"1.31.7" kubectl-"1.31.7" --disableexcludes=kubernetes
Restart the kubelet
daemon:
systemctl daemon-reload
systemctl restart kubelet
After upgrading kubectl
, need to export config:
export KUBECONFIG=/etc/kubernetes/admin.conf
# or if not root user
mkdir -p $HOME/.kube
sudo cp -f /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Verify kubectl
version:
Example output:
Client Version: v1.31.7
Kustomize Version: v5.4.2
Server Version: v1.31.7
List all nodes to verify that the control-plane (master node) kubelet
was upgraded successfully to v1.31.7 (other nodes will still show previous version):
Example output:
NAME STATUS ROLES AGE VERSION
gpu Ready <none> 48d v1.30.9
master Ready control-plane 48d v1.31.7
worker Ready <none> 48d v1.30.9
Step 7: Upgrade Worker/GPU Node Components
Although the worker and gpu nodes can be upgraded in parallel, it is highly recommended to only upgrade ONE node at a time.
kubeadm
and kubectl
are not required on the worker/gpu nodes (only kubelet
); however, in this example going to install all three for possible future use.
NOTE: if workloads/pods need to remain running during upgrade, each node must be drained, cordoned, and pods rescheduled. This example assumes that the cluster is not operational and that the pods will not be reachable until the upgrade is complete.
Execute each of the following steps on the worker and gpu nodes.
Upgrade kubeadm
:
sudo dnf install -y kubeadm-"1.31.7" --disableexcludes=kubernetes
Verify kubeadm
upgrade:
Example output:
kubeadm version: &http://version.Info {Major:"1", Minor:"31", GitVersion:"v1.31.7", GitCommit:"da53587841b4960dc3bd2af1ec6101b57c79aff4", GitTreeState:"clean", BuildDate:"2025-03-11T20:02:21Z", GoVersion:"go1.23.6", Compiler:"gc", Platform:"linux/amd64"}
Upgrade kubelet
and kubectl
:
sudo dnf install -y kubelet-"1.31.7" kubectl-"1.31.7" --disableexcludes=kubernetes
Restart the kubelet
daemon:
systemctl daemon-reload
systemctl restart kubelet
Verify kubectl
upgrade:
Example output:
Client Version: v1.31.7
Kustomize Version: v5.4.2
The connection to the server localhost:8080 was refused - did you specify the right host or port?
Step 8: Verify all Nodes Upgraded
On the master node (control plane), confirm the correct kubelet
version for all nodes:
Example output:
NAME STATUS ROLES AGE VERSION
gpu Ready <none> 49d v1.31.7
master Ready control-plane 49d v1.31.7
worker Ready <none> 49d v1.31.7