> ## Documentation Index
> Fetch the complete documentation index at: https://support.lilt.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Upgrade Kubernetes

## Overview

This article walks the installer through the process of upgrading K8S from a previous version.

In working through the installation, references will be made as appropriate to provide context.

#### System Parameters/Versions:

* Base OS:

  * Amazon Linux 2023: `ami-05576a079321f21f8`

  * Rocky 8.10

* K8S

  * current version: v1.30.9

  * upgrade version: v1.31.7

#### Requirements:

K8S must be upgraded incrementally from each version. As an example, if upgrading from v1.29 to v1.31, you must install each increment and repeat the process until the desired version is installed:

* v1.29 → v1.30

* v1.30 → v1.31

In this example, upgrading from v1.30 to v1.31 so only need to apply the process once.

## Upgrade K8S

#### Step 1: Check Current Versions

`kubeadm`, `kubelet`, and `kubectl` versions must match. Verify that the current versions are the same:

`kubeadm` version:

```bash theme={null}
kubeadm version
```

Example output:

```bash theme={null}
kubeadm version: &http://version.Info {Major:"1", Minor:"30", GitVersion:"v1.30.9", GitCommit:"a87cd6906120a367bf6787420e943103a463acba", GitTreeState:"clean", BuildDate:"2025-01-15T14:40:03Z", GoVersion:"go1.22.10", Compiler:"gc", Platform:"linux/amd64"}
```

`kubectl` version:

```bash theme={null}
kubectl version
```

Example output:

```bash theme={null}
Client Version: v1.30.9
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
```

`kubelet` version:

```bash theme={null}
kubectl get nodes
```

Example output:

```
NAME     STATUS   ROLES           AGE   VERSION
gpu      Ready    <none>          48d   v1.30.9
master   Ready    control-plane   48d   v1.30.9
worker   Ready    <none>          48d   v1.30.9
```

#### Step 2: Update Base OS

On ALL nodes in the cluster (master, worker, and gpu). Each base OS has different package updates.

* *Amazon Linux 2023*:

```bash theme={null}
# update OS
dnf check-release-update
sudo dnf update -y
```

* *Rocky 8*:

```bash theme={null}
# update system
dnf update -y
```

#### Step 3: Update Repository Config

On ALL nodes in the cluster (master, worker, and gpu), update repository config file to desired version. In this example, using v1.31:

```bash theme={null}
# set version
export K8S_V="1.31"

# set repository and repo
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v$K8S_V/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v$K8S_V/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF
```

#### Step 4: Upgrade Control Plane Kubeadm

On the master node, upgrade `kubeadm`. Worker/GPUs nodes will be updated separately.

Get list of possible upgrade versions:

```bash theme={null}
sudo dnf list --showduplicates kubeadm --disableexcludes=kubernetes
```

Example output:

```bash theme={null}
Installed Packages
kubeadm.x86_64                                                   1.30.9-150500.1.1                                                  @kubernetes
Available Packages
kubeadm.aarch64                                                  1.31.7-150500.1.1                                                  kubernetes
kubeadm.ppc64le                                                  1.31.7-150500.1.1                                                  kubernetes
kubeadm.s390x                                                    1.31.7-150500.1.1                                                  kubernetes
kubeadm.src                                                      1.31.7-150500.1.1                                                  kubernetes 
```

Install new `kubeadm` version:

```bash theme={null}
# upgrade kubeadm
sudo dnf install -y kubeadm-"1.31*" --disableexcludes=kubernetes
```

Check new version:

```bash theme={null}
kubeadm version
```

Example output:

```bash theme={null}
kubeadm version: &http://version.Info {Major:"1", Minor:"31", GitVersion:"v1.31.7", GitCommit:"da53587841b4960dc3bd2af1ec6101b57c79aff4", GitTreeState:"clean", BuildDate:"2025-03-11T20:02:21Z", GoVersion:"go1.23.6", Compiler:"gc", Platform:"linux/amd64"}
```

#### Step 5: Upgrade Control Plane Components

On the master node, upgrade control plane components.

Run a plan to show what can/will be updated:

```bash theme={null}
sudo kubeadm upgrade plan
```

Example output shows the possible v1.30 and 1.31 minor/patch upgrades. For this example, upgrading minor version to 1.31.7:

```
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: 1.30.9
[upgrade/versions] kubeadm version: v1.31.7
I0313 23:37:23.319175  600006 version.go:261] remote version is much newer: v1.32.3; falling back to: stable-1.31
[upgrade/versions] Target version: v1.31.7
[upgrade/versions] Latest version in the v1.30 series: v1.30.11

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   NODE      CURRENT   TARGET
kubelet     gpu       v1.30.9   v1.30.11
kubelet     master    v1.30.9   v1.30.11
kubelet     worker    v1.30.9   v1.30.11

Upgrade to the latest version in the v1.30 series:

COMPONENT                 NODE      CURRENT    TARGET
kube-apiserver            master    v1.30.9    v1.30.11
kube-controller-manager   master    v1.30.9    v1.30.11
kube-scheduler            master    v1.30.9    v1.30.11
kube-proxy                          1.30.9     v1.30.11
CoreDNS                             v1.11.3    v1.11.3
etcd                      master    3.5.15-0   3.5.15-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.30.11

_____________________________________________________________________

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   NODE      CURRENT   TARGET
kubelet     gpu       v1.30.9   v1.31.7
kubelet     master    v1.30.9   v1.31.7
kubelet     worker    v1.30.9   v1.31.7

Upgrade to the latest stable version:

COMPONENT                 NODE      CURRENT    TARGET
kube-apiserver            master    v1.30.9    v1.31.7
kube-controller-manager   master    v1.30.9    v1.31.7
kube-scheduler            master    v1.30.9    v1.31.7
kube-proxy                          1.30.9     v1.31.7
CoreDNS                             v1.11.3    v1.11.3
etcd                      master    3.5.15-0   3.5.15-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.31.7

_____________________________________________________________________
```

Upgrade to minor version v1.31.7 (can take up to five mins to complete):

```bash theme={null}
kubeadm upgrade apply v1.31.7 -y
```

When complete, view `kube-system` pods. If the upgrade was successful, the `kube*` pods will have a newer age than the others:

```bash theme={null}
kubectl get pods -n kube-system
```

Example output (upgrade was executed 13h ago):

```
NAME                              READY   STATUS    RESTARTS      AGE
coredns-55cb58b774-m7b6h          1/1     Running   3 (2d ago)    24d
coredns-55cb58b774-s5qqr          1/1     Running   4 (20h ago)   24d
etcd-master                       1/1     Running   0             13h
kube-apiserver-master             1/1     Running   0             13h
kube-controller-manager-master    1/1     Running   0             13h
kube-proxy-4bg5s                  1/1     Running   0             13h
kube-proxy-4mk8s                  1/1     Running   0             13h
kube-proxy-7h8ww                  1/1     Running   0             13h
kube-scheduler-master             1/1     Running   0             13h
metrics-server-5f6bd8776f-l6lg8   1/1     Running   3 (20h ago)   22d
```

#### Step 6: Upgrade Control Plane Kubectl and Kubelet

On the master node, upgrade `kubectl` and `kubelet` components. Versions need to match `kubeadm` from previous steps (v1.31.7):

```bash theme={null}
sudo dnf install -y kubelet-"1.31.7" kubectl-"1.31.7" --disableexcludes=kubernetes
```

Restart the `kubelet` daemon:

```bash theme={null}
systemctl daemon-reload
systemctl restart kubelet
```

After upgrading `kubectl`, need to export config:

```bash theme={null}
export KUBECONFIG=/etc/kubernetes/admin.conf 
# or if not root user 
mkdir -p $HOME/.kube 
sudo cp -f /etc/kubernetes/admin.conf $HOME/.kube/config 
sudo chown $(id -u):$(id -g) $HOME/.kube/config
```

Verify `kubectl` version:

```bash theme={null}
kubectl version
```

Example output:

```bash theme={null}
Client Version: v1.31.7
Kustomize Version: v5.4.2
Server Version: v1.31.7
```

List all nodes to verify that the control-plane (master node) `kubelet` was upgraded successfully to v1.31.7 (other nodes will still show previous version):

```bash theme={null}
kubectl get nodes
```

Example output:

```
NAME     STATUS   ROLES           AGE   VERSION
gpu      Ready    <none>          48d   v1.30.9
master   Ready    control-plane   48d   v1.31.7
worker   Ready    <none>          48d   v1.30.9
```

#### Step 7: Upgrade Worker/GPU Node Components

Although the worker and gpu nodes can be upgraded in parallel, it is highly recommended to only upgrade ONE node at a time.

`kubeadm` and `kubectl` are not required on the worker/gpu nodes (only `kubelet`); however, in this example going to install all three for possible future use.

<Note>
  NOTE: if workloads/pods need to remain running during upgrade, each node must be drained, cordoned, and pods rescheduled. This example assumes that the cluster is not operational and that the pods will not be reachable until the upgrade is complete.
</Note>

Execute each of the following steps on the worker and gpu nodes.

Upgrade `kubeadm`:

```bash theme={null}
sudo dnf install -y kubeadm-"1.31.7" --disableexcludes=kubernetes
```

Verify `kubeadm` upgrade:

```bash theme={null}
kubeadm version
```

Example output:

```bash theme={null}
kubeadm version: &http://version.Info {Major:"1", Minor:"31", GitVersion:"v1.31.7", GitCommit:"da53587841b4960dc3bd2af1ec6101b57c79aff4", GitTreeState:"clean", BuildDate:"2025-03-11T20:02:21Z", GoVersion:"go1.23.6", Compiler:"gc", Platform:"linux/amd64"}
```

Upgrade `kubelet` and `kubectl`:

```bash theme={null}
sudo dnf install -y kubelet-"1.31.7" kubectl-"1.31.7" --disableexcludes=kubernetes
```

Restart the `kubelet` daemon:

```bash theme={null}
systemctl daemon-reload
systemctl restart kubelet
```

Verify `kubectl` upgrade:

```bash theme={null}
kubectl version
```

Example output:

```bash theme={null}
Client Version: v1.31.7
Kustomize Version: v5.4.2
The connection to the server localhost:8080 was refused - did you specify the right host or port?
```

#### Step 8: Verify all Nodes Upgraded

On the master node (control plane), confirm the correct `kubelet` version for all nodes:

```bash theme={null}
kubectl get nodes
```

Example output:

```
NAME     STATUS   ROLES           AGE   VERSION
gpu      Ready    <none>          49d   v1.31.7
master   Ready    control-plane   49d   v1.31.7
worker   Ready    <none>          49d   v1.31.7
```