Upgrade Kubernetes

Overview

This article walks the installer through the process of upgrading K8S from a previous version. In working through the installation, references will be made as appropriate to provide context.

System Parameters/Versions:

Base OS:
- Amazon Linux 2023: ami-05576a079321f21f8
- Rocky 8.10
K8S
- current version: v1.30.9
- upgrade version: v1.31.7

Requirements:

K8S must be upgraded incrementally from each version. As an example, if upgrading from v1.29 to v1.31, you must install each increment and repeat the process until the desired version is installed:

v1.29 → v1.30
v1.30 → v1.31

In this example, upgrading from v1.30 to v1.31 so only need to apply the process once.

Upgrade K8S

Step 1: Check Current Versions

kubeadm, kubelet, and kubectl versions must match. Verify that the current versions are the same: kubeadm version:

kubeadm version

Example output:

kubeadm version: &http://version.Info {Major:"1", Minor:"30", GitVersion:"v1.30.9", GitCommit:"a87cd6906120a367bf6787420e943103a463acba", GitTreeState:"clean", BuildDate:"2025-01-15T14:40:03Z", GoVersion:"go1.22.10", Compiler:"gc", Platform:"linux/amd64"}

kubectl version:

kubectl version

Example output:

Client Version: v1.30.9
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3

kubelet version:

kubectl get nodes

Example output:

NAME     STATUS   ROLES           AGE   VERSION
gpu      Ready    <none>          48d   v1.30.9
master   Ready    control-plane   48d   v1.30.9
worker   Ready    <none>          48d   v1.30.9

Step 2: Update Base OS

On ALL nodes in the cluster (master, worker, and gpu). Each base OS has different package updates.

Amazon Linux 2023:

# update OS
dnf check-release-update
sudo dnf update -y

Rocky 8:

# update system
dnf update -y

Step 3: Update Repository Config

On ALL nodes in the cluster (master, worker, and gpu), update repository config file to desired version. In this example, using v1.31:

# set version
export K8S_V="1.31"

# set repository and repo
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v$K8S_V/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v$K8S_V/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

Step 4: Upgrade Control Plane Kubeadm

On the master node, upgrade kubeadm. Worker/GPUs nodes will be updated separately. Get list of possible upgrade versions:

sudo dnf list --showduplicates kubeadm --disableexcludes=kubernetes

Example output:

Installed Packages
kubeadm.x86_64                                                   1.30.9-150500.1.1                                                  @kubernetes
Available Packages
kubeadm.aarch64                                                  1.31.7-150500.1.1                                                  kubernetes
kubeadm.ppc64le                                                  1.31.7-150500.1.1                                                  kubernetes
kubeadm.s390x                                                    1.31.7-150500.1.1                                                  kubernetes
kubeadm.src                                                      1.31.7-150500.1.1                                                  kubernetes 

Install new kubeadm version:

# upgrade kubeadm
sudo dnf install -y kubeadm-"1.31*" --disableexcludes=kubernetes

Check new version:

kubeadm version

Example output:

kubeadm version: &http://version.Info {Major:"1", Minor:"31", GitVersion:"v1.31.7", GitCommit:"da53587841b4960dc3bd2af1ec6101b57c79aff4", GitTreeState:"clean", BuildDate:"2025-03-11T20:02:21Z", GoVersion:"go1.23.6", Compiler:"gc", Platform:"linux/amd64"}

Step 5: Upgrade Control Plane Components

On the master node, upgrade control plane components. Run a plan to show what can/will be updated:

sudo kubeadm upgrade plan

Example output shows the possible v1.30 and 1.31 minor/patch upgrades. For this example, upgrading minor version to 1.31.7:

[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: 1.30.9
[upgrade/versions] kubeadm version: v1.31.7
I0313 23:37:23.319175  600006 version.go:261] remote version is much newer: v1.32.3; falling back to: stable-1.31
[upgrade/versions] Target version: v1.31.7
[upgrade/versions] Latest version in the v1.30 series: v1.30.11

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   NODE      CURRENT   TARGET
kubelet     gpu       v1.30.9   v1.30.11
kubelet     master    v1.30.9   v1.30.11
kubelet     worker    v1.30.9   v1.30.11

Upgrade to the latest version in the v1.30 series:

COMPONENT                 NODE      CURRENT    TARGET
kube-apiserver            master    v1.30.9    v1.30.11
kube-controller-manager   master    v1.30.9    v1.30.11
kube-scheduler            master    v1.30.9    v1.30.11
kube-proxy                          1.30.9     v1.30.11
CoreDNS                             v1.11.3    v1.11.3
etcd                      master    3.5.15-0   3.5.15-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.30.11

_____________________________________________________________________

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   NODE      CURRENT   TARGET
kubelet     gpu       v1.30.9   v1.31.7
kubelet     master    v1.30.9   v1.31.7
kubelet     worker    v1.30.9   v1.31.7

Upgrade to the latest stable version:

COMPONENT                 NODE      CURRENT    TARGET
kube-apiserver            master    v1.30.9    v1.31.7
kube-controller-manager   master    v1.30.9    v1.31.7
kube-scheduler            master    v1.30.9    v1.31.7
kube-proxy                          1.30.9     v1.31.7
CoreDNS                             v1.11.3    v1.11.3
etcd                      master    3.5.15-0   3.5.15-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.31.7

_____________________________________________________________________

Upgrade to minor version v1.31.7 (can take up to five mins to complete):

kubeadm upgrade apply v1.31.7 -y

When complete, view kube-system pods. If the upgrade was successful, the kube* pods will have a newer age than the others:

kubectl get pods -n kube-system

Example output (upgrade was executed 13h ago):

NAME                              READY   STATUS    RESTARTS      AGE
coredns-55cb58b774-m7b6h          1/1     Running   3 (2d ago)    24d
coredns-55cb58b774-s5qqr          1/1     Running   4 (20h ago)   24d
etcd-master                       1/1     Running   0             13h
kube-apiserver-master             1/1     Running   0             13h
kube-controller-manager-master    1/1     Running   0             13h
kube-proxy-4bg5s                  1/1     Running   0             13h
kube-proxy-4mk8s                  1/1     Running   0             13h
kube-proxy-7h8ww                  1/1     Running   0             13h
kube-scheduler-master             1/1     Running   0             13h
metrics-server-5f6bd8776f-l6lg8   1/1     Running   3 (20h ago)   22d

Step 6: Upgrade Control Plane Kubectl and Kubelet

On the master node, upgrade kubectl and kubelet components. Versions need to match kubeadm from previous steps (v1.31.7):

sudo dnf install -y kubelet-"1.31.7" kubectl-"1.31.7" --disableexcludes=kubernetes

Restart the kubelet daemon:

systemctl daemon-reload
systemctl restart kubelet

After upgrading kubectl, need to export config:

export KUBECONFIG=/etc/kubernetes/admin.conf 
# or if not root user 
mkdir -p $HOME/.kube 
sudo cp -f /etc/kubernetes/admin.conf $HOME/.kube/config 
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Verify kubectl version:

kubectl version

Example output:

Client Version: v1.31.7
Kustomize Version: v5.4.2
Server Version: v1.31.7

List all nodes to verify that the control-plane (master node) kubelet was upgraded successfully to v1.31.7 (other nodes will still show previous version):

kubectl get nodes

Example output:

NAME     STATUS   ROLES           AGE   VERSION
gpu      Ready    <none>          48d   v1.30.9
master   Ready    control-plane   48d   v1.31.7
worker   Ready    <none>          48d   v1.30.9

Step 7: Upgrade Worker/GPU Node Components

Although the worker and gpu nodes can be upgraded in parallel, it is highly recommended to only upgrade ONE node at a time. kubeadm and kubectl are not required on the worker/gpu nodes (only kubelet); however, in this example going to install all three for possible future use.

NOTE: if workloads/pods need to remain running during upgrade, each node must be drained, cordoned, and pods rescheduled. This example assumes that the cluster is not operational and that the pods will not be reachable until the upgrade is complete.

Execute each of the following steps on the worker and gpu nodes. Upgrade kubeadm:

sudo dnf install -y kubeadm-"1.31.7" --disableexcludes=kubernetes

Verify kubeadm upgrade:

kubeadm version

Example output:

kubeadm version: &http://version.Info {Major:"1", Minor:"31", GitVersion:"v1.31.7", GitCommit:"da53587841b4960dc3bd2af1ec6101b57c79aff4", GitTreeState:"clean", BuildDate:"2025-03-11T20:02:21Z", GoVersion:"go1.23.6", Compiler:"gc", Platform:"linux/amd64"}

Upgrade kubelet and kubectl:

sudo dnf install -y kubelet-"1.31.7" kubectl-"1.31.7" --disableexcludes=kubernetes

Restart the kubelet daemon:

systemctl daemon-reload
systemctl restart kubelet

Verify kubectl upgrade:

kubectl version

Example output:

Client Version: v1.31.7
Kustomize Version: v5.4.2
The connection to the server localhost:8080 was refused - did you specify the right host or port?

Step 8: Verify all Nodes Upgraded

On the master node (control plane), confirm the correct kubelet version for all nodes:

kubectl get nodes

Example output:

NAME     STATUS   ROLES           AGE   VERSION
gpu      Ready    <none>          49d   v1.31.7
master   Ready    control-plane   49d   v1.31.7
worker   Ready    <none>          49d   v1.31.7

​Overview

​System Parameters/Versions:

​Requirements:

​Upgrade K8S

​Step 1: Check Current Versions

​Step 2: Update Base OS

​Step 3: Update Repository Config

​Step 4: Upgrade Control Plane Kubeadm

​Step 5: Upgrade Control Plane Components

​Step 6: Upgrade Control Plane Kubectl and Kubelet

​Step 7: Upgrade Worker/GPU Node Components

​Step 8: Verify all Nodes Upgraded

Overview

System Parameters/Versions:

Requirements:

Upgrade K8S

Step 1: Check Current Versions

Step 2: Update Base OS

Step 3: Update Repository Config

Step 4: Upgrade Control Plane Kubeadm

Step 5: Upgrade Control Plane Components

Step 6: Upgrade Control Plane Kubectl and Kubelet

Step 7: Upgrade Worker/GPU Node Components

Step 8: Verify all Nodes Upgraded