Upgrade Kubernetes (AWS EKS)

​

Overview

1. The control plane. AWS runs it. You choose the version, and EKS upgrades it one minor version at a time.
2. The nodes. Managed node groups are upgraded separately, after the control plane.
3. The cluster add-ons. kube-proxy and CoreDNS have builds for each Kubernetes version and are updated after the nodes. The VPC CNI, the EBS CSI driver, and the Pod Identity agent support a range of versions.
4. Your tools and controllers. kubectl on the machine you work from, and Helm-installed controllers such as the AWS Load Balancer Controller.

​

Version Rules

​

Why Nodes Must Be Upgraded Every Third Minor Version

​

Before You Begin

​

Check Readiness

• Upgrade insights. In the EKS console, open the cluster and select the Upgrade insights tab. Resolve every item marked as an error for the target version.
• Deprecated APIs. Scan the manifests and Helm releases in the cluster with a tool such as pluto or kubent. Then read the EKS release notes for every version you will pass through.
• Subnet capacity. The control-plane upgrade needs up to five free IP addresses in the cluster subnets.
• Third-party components. Every Helm-installed component must support each version you pass through, not only the final one. Check the support matrix of each component, for example Istio, External Secrets, the NVIDIA device plugin or GPU operator, and the AWS Load Balancer Controller. Upgrade any that do not support a version before the control plane reaches it.

​

Check How the Default Add-ons Are Installed

• As EKS add-ons. They appear in aws eks list-addons, and you update them with aws eks update-addon.
• Self-managed. A cluster created through the EKS API, the AWS CLI, or infrastructure as code without explicit add-on resources gets self-managed copies. They do not appear in aws eks list-addons, and EKS never upgrades them.

CLUSTER=my-cluster
aws eks list-addons --cluster-name $CLUSTER

kubectl -n kube-system get ds aws-node   -o yaml > aws-node.bak.yaml
kubectl -n kube-system get ds kube-proxy -o yaml > kube-proxy.bak.yaml
kubectl -n kube-system get cm coredns    -o yaml > coredns.bak.yaml

# repeat for each missing add-on, using a version from the lookup in "Find Add-on Versions"
aws eks create-addon --cluster-name $CLUSTER --addon-name coredns \
  --addon-version <version> --resolve-conflicts OVERWRITE

​

Plan for Node Replacement

• Spare capacity for at least one extra instance of each instance type, in the same availability zones.
• Enough instance quota. GPU instances count against the account’s On-Demand G and VT instance quota, which is measured in vCPUs. If a second GPU node does not fit within the quota, the update fails. See Upgrading When Capacity or Quota Is Limited.
• Room to evict pods. Pod disruption budgets can block the drain. The --force flag on the node group update evicts the pods anyway.

​

Find Add-on Versions

K=1.36          # target Kubernetes version

for A in kube-proxy coredns vpc-cni aws-ebs-csi-driver eks-pod-identity-agent; do
  V=$(aws eks describe-addon-versions --kubernetes-version $K --addon-name $A \
        --query 'addons[0].addonVersions[].addonVersion' --output text | tr '\t' '\n')
  [ "$A" = "kube-proxy" ] && V=$(echo "$V" | grep "^v${K}\.")      # kube-proxy must match the version
  printf '%-24s %s\n' "$A" "$(echo "$V" | sort -V | tail -1)"
done

for A in $(aws eks list-addons --cluster-name $CLUSTER --query 'addons[]' --output text); do
  aws eks describe-addon --cluster-name $CLUSTER --addon-name $A \
    --query 'addon.[addonName,addonVersion,status]' --output text
done

​

Upgrade kubectl

K=1.36          # target Kubernetes version

TMP=$(mktemp -d)
KVER=$(curl -Ls https://dl.k8s.io/release/stable-${K}.txt)
curl -L -o "$TMP/kubectl"        "https://dl.k8s.io/release/${KVER}/bin/linux/amd64/kubectl"
curl -L -o "$TMP/kubectl.sha256" "https://dl.k8s.io/release/${KVER}/bin/linux/amd64/kubectl.sha256"
echo "$(cat $TMP/kubectl.sha256)  $TMP/kubectl" | sha256sum --check
sudo install -m 0755 "$TMP/kubectl" /usr/local/bin/kubectl
rm -rf "$TMP"

which -a kubectl
kubectl version --client

​

Upgrade One Version

​

Step 1: Prepare

1. Upgrade kubectl to 1.36.
2. Confirm that every node runs the current version:
   kubectl version
   kubectl get nodes
   
   All nodes must be on 1.35 before the control plane moves to 1.36.
3. Find the add-on versions for 1.36.

​

Step 2: Upgrade the Control Plane

CLUSTER=my-cluster
REGION=us-east-1

aws eks update-cluster-version --name $CLUSTER --region $REGION --kubernetes-version 1.36
aws eks wait cluster-active --name $CLUSTER --region $REGION
kubectl version          # Server Version: v1.36.x

​

Step 3: Upgrade the Node Groups

for NG in $(aws eks list-nodegroups --cluster-name $CLUSTER --query 'nodegroups[]' --output text); do
  aws eks update-nodegroup-version --cluster-name $CLUSTER --nodegroup-name $NG --force
done

for NG in $(aws eks list-nodegroups --cluster-name $CLUSTER --query 'nodegroups[]' --output text); do
  aws eks wait nodegroup-active --cluster-name $CLUSTER --nodegroup-name $NG
done
kubectl get nodes        # every node VERSION v1.36.x, STATUS Ready

​

Step 4: Update kube-proxy and CoreDNS

aws eks update-addon --cluster-name $CLUSTER --addon-name kube-proxy \
  --addon-version <kube-proxy 1.36 version> --resolve-conflicts PRESERVE
aws eks update-addon --cluster-name $CLUSTER --addon-name coredns \
  --addon-version <coredns 1.36 version> --resolve-conflicts PRESERVE

aws eks wait addon-active --cluster-name $CLUSTER --addon-name kube-proxy
aws eks wait addon-active --cluster-name $CLUSTER --addon-name coredns

​

Step 5: Update Other Add-ons and Controllers

• VPC CNI, EBS CSI driver, and Pod Identity agent. Update them if the version lookup shows a newer build, or if the installed build does not support the new version. Move the VPC CNI one minor version at a time.
• AWS Load Balancer Controller. See Update the AWS Load Balancer Controller.
• GPU components. Update the NVIDIA device plugin or GPU operator to a release that supports the new version.

​

Step 6: Verify

​

Upgrade Several Versions

​

Option A: Upgrade Everything at Each Step

​

Option B: Let the Nodes Fall Up to Three Versions Behind

• Keep kube-proxy on the nodes’ build until the nodes are upgraded. kube-proxy may be up to three minor versions older than the control plane, never newer.
• Upgrade insights may warn about the version difference between the nodes and the control plane. Within three versions, the difference is supported.
• Steps 5 and 6 can run together. Upgrade the control plane, then the nodes.

​

Option C: Scale the Nodes to Zero

1. Scale every node group to zero:
   for NG in $(aws eks list-nodegroups --cluster-name $CLUSTER --query 'nodegroups[]' --output text); do
     aws eks update-nodegroup-config --cluster-name $CLUSTER --nodegroup-name $NG \
       --scaling-config minSize=0,maxSize=<current max>,desiredSize=0
   done
   kubectl get nodes        # wait for "No resources found"
   
   Record each node group’s current sizes first, so you can restore them.
2. For each version in turn, upgrade kubectl, the control plane, and every node group:
   aws eks update-cluster-version --name $CLUSTER --kubernetes-version 1.33
   aws eks wait cluster-active --name $CLUSTER
   for NG in $(aws eks list-nodegroups --cluster-name $CLUSTER --query 'nodegroups[]' --output text); do
     aws eks update-nodegroup-version --cluster-name $CLUSTER --nodegroup-name $NG
     aws eks wait nodegroup-active --cluster-name $CLUSTER --nodegroup-name $NG
   done
   
   With no nodes, the node group upgrade only changes its configuration. It launches nothing and takes a minute or two. Because the node groups keep pace with the control plane, the nodes start on the final version when they come back. Repeat for 1.34, 1.35, and 1.36.
3. Optional: test that nodes join at an intermediate version. If a version changes the node runtime, for example containerd 2 on EKS 1.34 AMIs, scale one node group to one node after that step. Confirm the node reaches Ready, then scale it back to zero. While the node runs, you can also update kube-proxy to that version’s build. The kube-proxy build then stays within three versions of the final control plane when the nodes return.
4. Start the nodes by restoring each node group’s original sizes:
   aws eks update-nodegroup-config --cluster-name $CLUSTER --nodegroup-name <name> \
     --scaling-config minSize=<min>,maxSize=<max>,desiredSize=<desired>
   
5. Update kube-proxy and CoreDNS to the final version’s builds, as in Step 4. Add-on updates need running nodes, so they wait until this point.

​

Update the AWS Load Balancer Controller

1. Find the available versions:
   helm repo add eks https://aws.github.io/eks-charts
   helm repo update
   helm search repo eks/aws-load-balancer-controller --versions | head -5
   helm list -n kube-system | grep aws-load-balancer-controller        # installed version
   
2. Update the CRDs. helm upgrade does not update CRDs, so apply them first:
   kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller/crds?ref=master"
   
3. Update the IAM policy. New releases can need new permissions. Compare the controller’s IAM policy with the docs/install/iam_policy.json file at the release tag in the controller repository, and update the policy if it differs.
4. Upgrade the release, keeping its current values:
   helm get values aws-load-balancer-controller -n kube-system -o yaml > lbc-values.yaml
   helm upgrade aws-load-balancer-controller eks/aws-load-balancer-controller \
     -n kube-system --version <chart version> -f lbc-values.yaml
   
   If the controller is managed by Terraform with a helm_release resource, change the chart version in your Terraform configuration and apply it instead of running helm upgrade.
5. Verify:
   kubectl -n kube-system rollout status deploy/aws-load-balancer-controller
   kubectl -n kube-system logs deploy/aws-load-balancer-controller --tail=50
   kubectl get svc -A | grep LoadBalancer        # existing load balancers keep their hostnames
   

​

Upgrading With Terraform

• Change one minor version per apply. Set the cluster version variable to the next version, apply, verify, and then repeat.
• Read every plan. An upgrade shows in-place changes (~). If a plan shows a replacement (-/+) of the cluster, a node group, an IAM role, or a launch template, stop. A replacement usually means the wrong variable file or workspace, not an upgrade.
• Know the order of node group changes. The AWS provider applies a node group’s version and launch template change before its size change. If a node group has running nodes and a pending version change, a scale-down in the same apply first tries to replace the nodes. To scale such a node group to zero, change its size with the AWS CLI first, then let Terraform apply the version.
• Target the upgrade. If the configuration has unrelated changes waiting, limit the plan to the cluster, node group, and add-on resources with -target, and save it with -out so that apply runs exactly what you reviewed.
• Do not change settings that force a new cluster. For example, in the AWS provider, changing bootstrap_self_managed_addons on an existing cluster replaces it.

​

Verify the Upgrade

kubectl version                                    # Server Version = target version
kubectl get nodes -o wide                          # every node Ready, on the target version
for A in $(aws eks list-addons --cluster-name $CLUSTER --query 'addons[]' --output text); do
  aws eks describe-addon --cluster-name $CLUSTER --addon-name $A \
    --query 'addon.[addonName,addonVersion,status]' --output text
done                                               # every add-on ACTIVE
kubectl get pods -A | grep -v -E 'Running|Completed'
kubectl get pvc -A | grep -v Bound
kubectl describe nodes | grep -E 'nvidia.com/gpu'  # GPU nodes report their GPUs

​

Roll Back

​

Troubleshooting

CR=$(aws ec2 create-capacity-reservation --instance-type <type> --instance-platform Linux/UNIX \
  --availability-zone <zone> --instance-count 1 --instance-match-criteria targeted \
  --end-date-type unlimited --query 'CapacityReservation.CapacityReservationId' --output text) \
  && echo "available" && aws ec2 cancel-capacity-reservation --capacity-reservation-id $CR

aws service-quotas get-service-quota --service-code ec2 --quota-code L-DB2E81BA \
  --query 'Quota.[QuotaName,Value]' --output text

for id in $(aws eks list-updates --name $CLUSTER --nodegroup-name <name> --query 'updateIds[]' --output text); do
  aws eks describe-update --name $CLUSTER --nodegroup-name <name> --update-id $id \
    --query 'update.[createdAt,type,status]' --output text
done | sort

ASG=$(aws eks describe-nodegroup --cluster-name $CLUSTER --nodegroup-name <name> \
  --query 'nodegroup.resources.autoScalingGroups[0].name' --output text)
aws autoscaling describe-lifecycle-hooks --auto-scaling-group-name $ASG \
  --query 'LifecycleHooks[].[LifecycleHookName,LifecycleTransition]' --output text
aws autoscaling complete-lifecycle-action --auto-scaling-group-name $ASG \
  --lifecycle-hook-name <terminating hook name> --instance-id <instance id> \
  --lifecycle-action-result CONTINUE

​

Tips

​

Find Available GPU Instances in Your Region

REGION=us-east-1

aws ec2 describe-instance-type-offerings --region $REGION \
  --location-type availability-zone \
  --filters "Name=instance-type,Values=g4dn.*,g5.*,g6.*,g6e.*,gr6.*" \
  --query 'InstanceTypeOfferings[].[Location,InstanceType]' --output text | sort

aws ec2 describe-instance-types --region $REGION \
  --instance-types g6.xlarge g6.2xlarge g6.4xlarge g6.8xlarge g6.12xlarge g6.24xlarge g5.xlarge g4dn.xlarge \
  --query 'InstanceTypes[].[InstanceType,GpuInfo.Gpus[0].Count,GpuInfo.Gpus[0].Name,VCpuInfo.DefaultVCpus,MemoryInfo.SizeInMiB]' \
  --output table

ZONE=us-east-1a

for t in g6.xlarge g6.2xlarge g6.4xlarge g6.8xlarge g6.12xlarge g6.24xlarge g5.xlarge g4dn.xlarge; do
  CR=$(aws ec2 create-capacity-reservation --instance-type $t --instance-platform Linux/UNIX \
        --availability-zone $ZONE --instance-count 1 --instance-match-criteria targeted \
        --end-date-type unlimited --query 'CapacityReservation.CapacityReservationId' --output text 2>/dev/null)
  if [ -n "$CR" ]; then
    echo "$ZONE  $t  available"
    aws ec2 cancel-capacity-reservation --capacity-reservation-id $CR >/dev/null
  else
    echo "$ZONE  $t  no capacity"
  fi
done

aws service-quotas get-service-quota --service-code ec2 --quota-code L-DB2E81BA \
  --query 'Quota.[QuotaName,Value]' --output text

​

Reserve GPU Capacity Before Starting Nodes

aws ec2 create-capacity-reservation --instance-type <type> --instance-platform Linux/UNIX \
  --availability-zone <zone> --instance-count 1 --instance-match-criteria open \
  --end-date-type unlimited --query 'CapacityReservation.[CapacityReservationId,State]' --output text

aws ec2 cancel-capacity-reservation --capacity-reservation-id <reservation id>

​

Test With a Smaller GPU Instance

1. Scale the node group to zero.
2. Change the instance type.
3. Scale up and test.
4. Scale back to zero.
5. Change the instance type back.

​

Work Safely With Terraform

• Confirm the environment before every plan. If you keep several environments in one configuration, check that the backend and the variable file match the cluster you mean to change:
  terraform state show aws_eks_cluster.<resource name> | grep -E '^\s+name\s'
  
  Running a plan with another environment’s variable file proposes replacing the cluster and everything attached to it.
• Save plans and apply the saved file. terraform plan -out=<file> followed by terraform apply <file> applies exactly what you reviewed. Note that apply with a saved plan does not ask for confirmation.
• Keep saved plans out of version control. Plan files can contain secret values in plain text. Write them to a dedicated directory, such as plans/, and add it to .gitignore.
• Read the summary line. Plan: X to add, Y to change, Z to destroy. An upgrade should show 0 to destroy. Stop on any destroy or replacement you did not expect.
• A saved plan can go stale. If something changes between plan and apply, Terraform reports Saved plan is stale. Run the plan again, review it, and apply the new file.
• Do not run an apply while an EKS update is in progress. Wait until the cluster and node groups are ACTIVE. Otherwise the apply fails with ResourceInUseException, or it starts a second update that repeats the failure that stopped the first one.

​

Allow Time for First Image Pulls

​

Download Tools Into a Temporary Directory