> ## Documentation Index > Fetch the complete documentation index at: https://support.lilt.com/llms.txt > Use this file to discover all available pages before exploring further. # Upgrade Kubernetes (AWS EKS) > Upgrade an Amazon EKS cluster one Kubernetes version at a time or across several versions, including nodes, kube-proxy, CoreDNS, kubectl, and the AWS Load Balancer Controller. This article describes how to upgrade the Kubernetes version of an Amazon EKS cluster that runs LILT. It covers a single-version upgrade, such as 1.35 to 1.36, and an upgrade across several versions, such as 1.32 to 1.36. For clusters installed with `kubeadm` on your own hosts, see [Upgrade Kubernetes](/self-managed/v6.1/upgrade-kubernetes) instead. ## Overview An EKS upgrade has four parts, and they are upgraded in this order: 1. **The control plane.** AWS runs it. You choose the version, and EKS upgrades it one minor version at a time. 2. **The nodes.** Managed node groups are upgraded separately, after the control plane. 3. **The cluster add-ons.** kube-proxy and CoreDNS have builds for each Kubernetes version and are updated after the nodes. The VPC CNI, the EBS CSI driver, and the Pod Identity agent support a range of versions. 4. **Your tools and controllers.** `kubectl` on the machine you work from, and Helm-installed controllers such as the AWS Load Balancer Controller. EKS can only upgrade the control plane **one minor version at a time**. To go from 1.32 to 1.36, the control plane must pass through 1.33, 1.34, and 1.35. There is no way to skip a version. ## Version Rules These rules decide what you upgrade, and when. They come from the [Kubernetes version skew policy](https://kubernetes.io/releases/version-skew-policy/) and the [Amazon EKS upgrade documentation](https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html). | Component | Rule | What it means for an upgrade | | - | - | - | | Control plane | Upgrades one minor version at a time. | Each version in between is a separate upgrade. | | Nodes (kubelet) | May be up to **three** minor versions older than the control plane, and never newer. | Nodes must be upgraded at least every third minor version. AWS recommends matching them to the control plane before each control-plane upgrade. | | kube-proxy | Must not be newer than the control plane, and may be up to three minor versions older than it. EKS publishes a kube-proxy build for each Kubernetes version. | Update kube-proxy to the build for the cluster's version after the nodes reach that version. | | CoreDNS | Not tied to the Kubernetes version, but EKS publishes a recommended build for each one. | Update it after each control-plane upgrade. | | `kubectl` | Must be within **one** minor version of the control plane, older or newer. | Install the target version before each step. It then works with both the old and the new control plane. | | VPC CNI | One build supports a range of Kubernetes versions. AWS recommends upgrading it one minor version of the CNI at a time. | Usually updated once, before or after the cluster upgrade, rather than at every step. | ### Why Nodes Must Be Upgraded Every Third Minor Version The kubelet on each node talks to the control plane's API server. Kubernetes supports a kubelet that is up to three minor versions older than the API server. A node on 1.32 is therefore supported by control planes on 1.32, 1.33, 1.34, and 1.35, but not 1.36. So when you upgrade across several versions, you can upgrade the control plane up to three times while the nodes stay on their original version. Before the fourth control-plane upgrade, the nodes must be upgraded. In the 1.32 to 1.36 example, the nodes must reach at least 1.33 before the control plane moves to 1.36. Upgrading the nodes at every step, as AWS recommends, always satisfies this rule. Waiting up to three steps is allowed, and it reduces how many times the nodes are replaced. ## Before You Begin ### Check Readiness * **Upgrade insights.** In the EKS console, open the cluster and select the **Upgrade insights** tab. Resolve every item marked as an error for the target version. * **Deprecated APIs.** Scan the manifests and Helm releases in the cluster with a tool such as [`pluto`](https://github.com/FairwindsOps/pluto) or [`kubent`](https://github.com/doitintl/kube-no-trouble). Then read the [EKS release notes](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions-standard.html) for every version you will pass through. * **Subnet capacity.** The control-plane upgrade needs up to five free IP addresses in the cluster subnets. * **Third-party components.** Every Helm-installed component must support each version you pass through, not only the final one. Check the support matrix of each component, for example Istio, External Secrets, the NVIDIA device plugin or GPU operator, and the AWS Load Balancer Controller. Upgrade any that do not support a version **before** the control plane reaches it. ### Check How the Default Add-ons Are Installed The VPC CNI, CoreDNS, and kube-proxy are installed in one of two ways: * **As EKS add-ons.** They appear in `aws eks list-addons`, and you update them with `aws eks update-addon`. * **Self-managed.** A cluster created through the EKS API, the AWS CLI, or infrastructure as code without explicit add-on resources gets self-managed copies. They do not appear in `aws eks list-addons`, and **EKS never upgrades them**. ```bash theme={null} CLUSTER=my-cluster aws eks list-addons --cluster-name $CLUSTER ``` If `vpc-cni`, `coredns`, or `kube-proxy` is missing from the list, adopt it as an EKS add-on before you upgrade. First back up its current configuration. Adopting it overwrites custom settings such as environment variables on `aws-node` or edits to the CoreDNS `Corefile`. ```bash theme={null} kubectl -n kube-system get ds aws-node -o yaml > aws-node.bak.yaml kubectl -n kube-system get ds kube-proxy -o yaml > kube-proxy.bak.yaml kubectl -n kube-system get cm coredns -o yaml > coredns.bak.yaml # repeat for each missing add-on, using a version from the lookup in "Find Add-on Versions" aws eks create-addon --cluster-name $CLUSTER --addon-name coredns \ --addon-version --resolve-conflicts OVERWRITE ``` Pass any custom settings from the backups through `--configuration-values`. ### Plan for Node Replacement When you upgrade a managed node group, EKS launches replacement nodes **before** it drains and removes the old ones. During an upgrade the node group needs: * **Spare capacity** for at least one extra instance of each instance type, in the same availability zones. * **Enough instance quota.** GPU instances count against the account's On-Demand G and VT instance quota, which is measured in vCPUs. If a second GPU node does not fit within the quota, the update fails. See [Upgrading When Capacity or Quota Is Limited](#upgrading-when-capacity-or-quota-is-limited). * **Room to evict pods.** Pod disruption budgets can block the drain. The `--force` flag on the node group update evicts the pods anyway. ## Find Add-on Versions AWS publishes new add-on builds regularly. Look up the current builds for your target version rather than copying versions from older notes: ```bash theme={null} K=1.36 # target Kubernetes version for A in kube-proxy coredns vpc-cni aws-ebs-csi-driver eks-pod-identity-agent; do V=$(aws eks describe-addon-versions --kubernetes-version $K --addon-name $A \ --query 'addons[0].addonVersions[].addonVersion' --output text | tr '\t' '\n') [ "$A" = "kube-proxy" ] && V=$(echo "$V" | grep "^v${K}\.") # kube-proxy must match the version printf '%-24s %s\n' "$A" "$(echo "$V" | sort -V | tail -1)" done ``` To see what the cluster runs now: ```bash theme={null} for A in $(aws eks list-addons --cluster-name $CLUSTER --query 'addons[]' --output text); do aws eks describe-addon --cluster-name $CLUSTER --addon-name $A \ --query 'addon.[addonName,addonVersion,status]' --output text done ``` Add-on versions must be exact, such as `v1.36.0-eksbuild.21`. A pattern such as `v1.36.x` is rejected. ## Upgrade kubectl Install the `kubectl` version that matches the **target** Kubernetes version before each step. Because `kubectl` supports one minor version in either direction, the target version works against the control plane both before and after the upgrade. ```bash theme={null} K=1.36 # target Kubernetes version TMP=$(mktemp -d) KVER=$(curl -Ls https://dl.k8s.io/release/stable-${K}.txt) curl -L -o "$TMP/kubectl" "https://dl.k8s.io/release/${KVER}/bin/linux/amd64/kubectl" curl -L -o "$TMP/kubectl.sha256" "https://dl.k8s.io/release/${KVER}/bin/linux/amd64/kubectl.sha256" echo "$(cat $TMP/kubectl.sha256) $TMP/kubectl" | sha256sum --check sudo install -m 0755 "$TMP/kubectl" /usr/local/bin/kubectl rm -rf "$TMP" which -a kubectl kubectl version --client ``` Replace `linux/amd64` with `linux/arm64`, `darwin/arm64`, or `darwin/amd64` as appropriate. On macOS, use `shasum -a 256 -c` instead of `sha256sum --check`. If `which -a kubectl` lists another copy first, for example one bundled with a cloud SDK, put `/usr/local/bin` earlier in your `PATH`. ## Upgrade One Version This example upgrades from 1.35 to 1.36. To upgrade across several versions, see [Upgrade Several Versions](#upgrade-several-versions). ### Step 1: Prepare 1. [Upgrade kubectl](#upgrade-kubectl) to 1.36. 2. Confirm that every node runs the current version: ```bash theme={null} kubectl version kubectl get nodes ``` All nodes must be on 1.35 before the control plane moves to 1.36. 3. [Find the add-on versions](#find-add-on-versions) for 1.36. ### Step 2: Upgrade the Control Plane ```bash theme={null} CLUSTER=my-cluster REGION=us-east-1 aws eks update-cluster-version --name $CLUSTER --region $REGION --kubernetes-version 1.36 aws eks wait cluster-active --name $CLUSTER --region $REGION kubectl version # Server Version: v1.36.x ``` The control-plane upgrade usually takes 10 to 15 minutes. It cannot be paused or cancelled after it starts. If its health checks fail, EKS reverts it, and the cluster stays on the previous version. ### Step 3: Upgrade the Node Groups ```bash theme={null} for NG in $(aws eks list-nodegroups --cluster-name $CLUSTER --query 'nodegroups[]' --output text); do aws eks update-nodegroup-version --cluster-name $CLUSTER --nodegroup-name $NG --force done for NG in $(aws eks list-nodegroups --cluster-name $CLUSTER --query 'nodegroups[]' --output text); do aws eks wait nodegroup-active --cluster-name $CLUSTER --nodegroup-name $NG done kubectl get nodes # every node VERSION v1.36.x, STATUS Ready ``` With no `--kubernetes-version`, each node group moves to the control plane's version with the latest EKS-optimized AMI. A node group that uses a launch template with a custom AMI needs a new launch template version instead. ### Step 4: Update kube-proxy and CoreDNS ```bash theme={null} aws eks update-addon --cluster-name $CLUSTER --addon-name kube-proxy \ --addon-version --resolve-conflicts PRESERVE aws eks update-addon --cluster-name $CLUSTER --addon-name coredns \ --addon-version --resolve-conflicts PRESERVE aws eks wait addon-active --cluster-name $CLUSTER --addon-name kube-proxy aws eks wait addon-active --cluster-name $CLUSTER --addon-name coredns ``` Update kube-proxy **after** the nodes so that its build matches the nodes it runs on. `PRESERVE` keeps settings you changed on the add-on. If you want EKS to restore its defaults, use `OVERWRITE`. ### Step 5: Update Other Add-ons and Controllers * **VPC CNI, EBS CSI driver, and Pod Identity agent.** Update them if the version lookup shows a newer build, or if the installed build does not support the new version. Move the VPC CNI one minor version at a time. * **AWS Load Balancer Controller.** See [Update the AWS Load Balancer Controller](#update-the-aws-load-balancer-controller). * **GPU components.** Update the NVIDIA device plugin or GPU operator to a release that supports the new version. ### Step 6: Verify See [Verify the Upgrade](#verify-the-upgrade). ## Upgrade Several Versions To go from 1.32 to 1.36, the control plane must pass through four upgrades: 1.33, 1.34, 1.35, and 1.36. You choose how the nodes keep up. The three approaches below trade the number of node replacements against the downtime you can accept. ### Option A: Upgrade Everything at Each Step Repeat [Upgrade One Version](#upgrade-one-version) for each version in turn: 1.33, then 1.34, 1.35, and 1.36. At each step, upgrade `kubectl`, the control plane, the node groups, kube-proxy, and CoreDNS. This is the approach AWS recommends, and the cluster stays in a fully supported state throughout. The nodes are replaced four times, and each replacement needs spare capacity for the surge nodes. ### Option B: Let the Nodes Fall Up to Three Versions Behind Upgrade the control plane up to three times while the nodes stay on their current version, then upgrade the nodes. For 1.32 to 1.36: | Step | Control plane | Nodes | kube-proxy | CoreDNS | | - | - | - | - | - | | 1 | 1.33 | 1.32 | 1.32 build | 1.33 build | | 2 | 1.34 | 1.32 | 1.32 build | 1.34 build | | 3 | 1.35 | 1.32 | 1.32 build | 1.35 build | | 4 | 1.35 | **1.35** | 1.35 build | 1.35 build | | 5 | 1.36 | 1.35 | 1.35 build | 1.36 build | | 6 | 1.36 | **1.36** | 1.36 build | 1.36 build | The nodes are replaced twice instead of four times. * Keep kube-proxy on the nodes' build until the nodes are upgraded. kube-proxy may be up to three minor versions older than the control plane, never newer. * Upgrade insights may warn about the version difference between the nodes and the control plane. Within three versions, the difference is supported. * Steps 5 and 6 can run together. Upgrade the control plane, then the nodes. ### Option C: Scale the Nodes to Zero If the cluster can be offline for a maintenance window, scale every node group to zero, step the control plane through each version, and start the nodes once at the end. This is the fastest approach, and it needs no spare capacity for surge nodes, which helps when GPU capacity or quota is limited. Every workload in the cluster, including LILT, is unavailable from the moment the nodes scale down until they are back and running. 1. **Scale every node group to zero:** ```bash theme={null} for NG in $(aws eks list-nodegroups --cluster-name $CLUSTER --query 'nodegroups[]' --output text); do aws eks update-nodegroup-config --cluster-name $CLUSTER --nodegroup-name $NG \ --scaling-config minSize=0,maxSize=,desiredSize=0 done kubectl get nodes # wait for "No resources found" ``` Record each node group's current sizes first, so you can restore them. 2. **For each version in turn,** upgrade `kubectl`, the control plane, and every node group: ```bash theme={null} aws eks update-cluster-version --name $CLUSTER --kubernetes-version 1.33 aws eks wait cluster-active --name $CLUSTER for NG in $(aws eks list-nodegroups --cluster-name $CLUSTER --query 'nodegroups[]' --output text); do aws eks update-nodegroup-version --cluster-name $CLUSTER --nodegroup-name $NG aws eks wait nodegroup-active --cluster-name $CLUSTER --nodegroup-name $NG done ``` With no nodes, the node group upgrade only changes its configuration. It launches nothing and takes a minute or two. Because the node groups keep pace with the control plane, the nodes start on the final version when they come back. Repeat for 1.34, 1.35, and 1.36. 3. **Optional: test that nodes join at an intermediate version.** If a version changes the node runtime, for example containerd 2 on EKS 1.34 AMIs, scale one node group to one node after that step. Confirm the node reaches `Ready`, then scale it back to zero. While the node runs, you can also update kube-proxy to that version's build. The kube-proxy build then stays within three versions of the final control plane when the nodes return. 4. **Start the nodes** by restoring each node group's original sizes: ```bash theme={null} aws eks update-nodegroup-config --cluster-name $CLUSTER --nodegroup-name \ --scaling-config minSize=,maxSize=,desiredSize= ``` 5. **Update kube-proxy and CoreDNS** to the final version's builds, as in [Step 4](#step-4-update-kube-proxy-and-coredns). Add-on updates need running nodes, so they wait until this point. Add-ons such as CoreDNS can show `DEGRADED` while the nodes are at zero, because their pods have nowhere to run. This clears when the nodes return. In a cluster that spans several availability zones, an EBS volume can only attach to a node in its own zone. If fewer nodes come back than before, a pod whose volume is in another zone stays `Pending` with a `volume node affinity conflict`. Record the zone of each persistent volume before you scale down, and make sure a node returns in each of those zones. ## Update the AWS Load Balancer Controller The AWS Load Balancer Controller is installed with Helm and is not upgraded with the cluster. Check its [releases](https://github.com/kubernetes-sigs/aws-load-balancer-controller/releases) for support of your target Kubernetes version. Starting with controller v3, the Helm chart version matches the controller version, for example chart `3.6.0` installs controller `v3.6.0`. For controller v2 releases, the chart version is 1.x, for example chart `1.13.4` installs controller `v2.13.4`. 1. **Find the available versions:** ```bash theme={null} helm repo add eks https://aws.github.io/eks-charts helm repo update helm search repo eks/aws-load-balancer-controller --versions | head -5 helm list -n kube-system | grep aws-load-balancer-controller # installed version ``` 2. **Update the CRDs.** `helm upgrade` does not update CRDs, so apply them first: ```bash theme={null} kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller/crds?ref=master" ``` 3. **Update the IAM policy.** New releases can need new permissions. Compare the controller's IAM policy with the `docs/install/iam_policy.json` file at the release tag in the controller repository, and update the policy if it differs. 4. **Upgrade the release, keeping its current values:** ```bash theme={null} helm get values aws-load-balancer-controller -n kube-system -o yaml > lbc-values.yaml helm upgrade aws-load-balancer-controller eks/aws-load-balancer-controller \ -n kube-system --version -f lbc-values.yaml ``` If the controller is managed by Terraform with a `helm_release` resource, change the chart version in your Terraform configuration and apply it instead of running `helm upgrade`. 5. **Verify:** ```bash theme={null} kubectl -n kube-system rollout status deploy/aws-load-balancer-controller kubectl -n kube-system logs deploy/aws-load-balancer-controller --tail=50 kubectl get svc -A | grep LoadBalancer # existing load balancers keep their hostnames ``` If you use the controller's Gateway API support, also read the release notes for the Gateway API CRD version that release requires. If another component, such as Istio, owns the Gateway API CRDs in your cluster, upgrade them through that component. ## Upgrading With Terraform If the cluster is managed with Terraform, make the same changes through your configuration instead of the AWS CLI: * **Change one minor version per apply.** Set the cluster version variable to the next version, apply, verify, and then repeat. * **Read every plan.** An upgrade shows in-place changes (`~`). If a plan shows a replacement (`-/+`) of the cluster, a node group, an IAM role, or a launch template, stop. A replacement usually means the wrong variable file or workspace, not an upgrade. * **Know the order of node group changes.** The AWS provider applies a node group's version and launch template change **before** its size change. If a node group has running nodes and a pending version change, a scale-down in the same apply first tries to replace the nodes. To scale such a node group to zero, change its size with the AWS CLI first, then let Terraform apply the version. * **Target the upgrade.** If the configuration has unrelated changes waiting, limit the plan to the cluster, node group, and add-on resources with `-target`, and save it with `-out` so that `apply` runs exactly what you reviewed. * **Do not change settings that force a new cluster.** For example, in the AWS provider, changing `bootstrap_self_managed_addons` on an existing cluster replaces it. ## Verify the Upgrade ```bash theme={null} kubectl version # Server Version = target version kubectl get nodes -o wide # every node Ready, on the target version for A in $(aws eks list-addons --cluster-name $CLUSTER --query 'addons[]' --output text); do aws eks describe-addon --cluster-name $CLUSTER --addon-name $A \ --query 'addon.[addonName,addonVersion,status]' --output text done # every add-on ACTIVE kubectl get pods -A | grep -v -E 'Running|Completed' kubectl get pvc -A | grep -v Bound kubectl describe nodes | grep -E 'nvidia.com/gpu' # GPU nodes report their GPUs ``` Then sign in to LILT, upload a file, and run a pretranslation. ## Roll Back Within 7 days of an upgrade, EKS can roll back the control plane to the previous minor version. See [Roll back a cluster to a previous Kubernetes version](https://docs.aws.amazon.com/eks/latest/userguide/rollback-cluster.html). After that window, the only way back is a new cluster on the earlier version. If you manage the cluster with Terraform, set the version variable back after a rollback, so that the next apply does not upgrade the cluster again. ## Troubleshooting AWS had no free instance of that type in the availability zone for the replacement node. The update fails and the node group returns to `ACTIVE` on its previous version. Retry later, reserve capacity first with an On-Demand Capacity Reservation, or use [Option C](#option-c-scale-the-nodes-to-zero), which launches no replacement nodes during the version steps. To test whether an instance type can start now, reserve one and cancel it at once: ```bash theme={null} CR=$(aws ec2 create-capacity-reservation --instance-type --instance-platform Linux/UNIX \ --availability-zone --instance-count 1 --instance-match-criteria targeted \ --end-date-type unlimited --query 'CapacityReservation.CapacityReservationId' --output text) \ && echo "available" && aws ec2 cancel-capacity-reservation --capacity-reservation-id $CR ``` The replacement node would take the account past its instance quota. This is common with GPU instances. Check the quota: ```bash theme={null} aws service-quotas get-service-quota --service-code ec2 --quota-code L-DB2E81BA \ --query 'Quota.[QuotaName,Value]' --output text ``` Request an increase, or upgrade with [Option C](#option-c-scale-the-nodes-to-zero). Another update on the node group is still running. List the updates and wait for the latest one to finish before starting another: ```bash theme={null} for id in $(aws eks list-updates --name $CLUSTER --nodegroup-name --query 'updateIds[]' --output text); do aws eks describe-update --name $CLUSTER --nodegroup-name --update-id $id \ --query 'update.[createdAt,type,status]' --output text done | sort ``` Managed node groups use a lifecycle hook to drain a node before it terminates. If an update fails part way through, the hook can be left waiting. Release it: ```bash theme={null} ASG=$(aws eks describe-nodegroup --cluster-name $CLUSTER --nodegroup-name \ --query 'nodegroup.resources.autoScalingGroups[0].name' --output text) aws autoscaling describe-lifecycle-hooks --auto-scaling-group-name $ASG \ --query 'LifecycleHooks[].[LifecycleHookName,LifecycleTransition]' --output text aws autoscaling complete-lifecycle-action --auto-scaling-group-name $ASG \ --lifecycle-hook-name --instance-id \ --lifecycle-action-result CONTINUE ``` New nodes pull every image at once. On a network with limited egress to Amazon ECR, pulls can take many minutes and time out at first. Until the `ebs-csi-node` pod on a node is ready, EBS volumes cannot attach to that node, and pods report `timed out waiting for external-attacher`. Check the `ebs-csi-node` pod on the new node. If its images are still pulling, wait. VPC endpoints for `ecr.api`, `ecr.dkr`, and an S3 gateway endpoint make pulls faster and more reliable. Messages such as `failed to sync secret cache: timed out waiting for the condition` in the first minutes after a node joins are usually harmless and clear on retry. An add-on update needs running nodes to roll out. If the cluster has no nodes, the update cannot finish. Start the nodes first, then update the add-ons. If the update fails with a conflict, a field you changed differs from the add-on's default. Use `--resolve-conflicts PRESERVE` to keep your change, or `OVERWRITE` to replace it. ## Tips ### Find Available GPU Instances in Your Region An availability zone can offer an instance type and still have none free to start. Before an upgrade launches GPU nodes, check what each zone offers, how large each type is, whether it can start now, and how much quota the account has. **GPU instance types each zone offers:** ```bash theme={null} REGION=us-east-1 aws ec2 describe-instance-type-offerings --region $REGION \ --location-type availability-zone \ --filters "Name=instance-type,Values=g4dn.*,g5.*,g6.*,g6e.*,gr6.*" \ --query 'InstanceTypeOfferings[].[Location,InstanceType]' --output text | sort ``` To limit the output to the zones your node groups use, add `Name=location,Values=,` to `--filters`. **GPU count, GPU model, vCPUs, and memory of each type:** ```bash theme={null} aws ec2 describe-instance-types --region $REGION \ --instance-types g6.xlarge g6.2xlarge g6.4xlarge g6.8xlarge g6.12xlarge g6.24xlarge g5.xlarge g4dn.xlarge \ --query 'InstanceTypes[].[InstanceType,GpuInfo.Gpus[0].Count,GpuInfo.Gpus[0].Name,VCpuInfo.DefaultVCpus,MemoryInfo.SizeInMiB]' \ --output table ``` **Whether each type can start right now.** No API reports live capacity, so this briefly reserves one instance of each type and cancels the reservation at once. Run it once per zone: ```bash theme={null} ZONE=us-east-1a for t in g6.xlarge g6.2xlarge g6.4xlarge g6.8xlarge g6.12xlarge g6.24xlarge g5.xlarge g4dn.xlarge; do CR=$(aws ec2 create-capacity-reservation --instance-type $t --instance-platform Linux/UNIX \ --availability-zone $ZONE --instance-count 1 --instance-match-criteria targeted \ --end-date-type unlimited --query 'CapacityReservation.CapacityReservationId' --output text 2>/dev/null) if [ -n "$CR" ]; then echo "$ZONE $t available" aws ec2 cancel-capacity-reservation --capacity-reservation-id $CR >/dev/null else echo "$ZONE $t no capacity" fi done ``` `available` means one instance of that type can start in that zone now. `no capacity` means AWS has none free at the moment. Capacity changes over time, so run the check again shortly before you start nodes. **The account's GPU quota:** ```bash theme={null} aws service-quotas get-service-quota --service-code ec2 --quota-code L-DB2E81BA \ --query 'Quota.[QuotaName,Value]' --output text ``` The quota is "Running On-Demand G and VT instances". It is measured in **vCPUs**, not GPUs, and counts every running G-family instance in the account and region, including capacity reservations. For example, a `g6.12xlarge` uses 48 vCPUs. A node group upgrade with running nodes needs room for the old and the replacement node at the same time. ### Reserve GPU Capacity Before Starting Nodes If GPU capacity is scarce in your zone, reserve it before you start or replace GPU nodes. An `open` reservation is used automatically by any matching instance, including one launched by a managed node group: ```bash theme={null} aws ec2 create-capacity-reservation --instance-type --instance-platform Linux/UNIX \ --availability-zone --instance-count 1 --instance-match-criteria open \ --end-date-type unlimited --query 'CapacityReservation.[CapacityReservationId,State]' --output text ``` Once the reservation is `active`, start or upgrade the node group. After the node is `Ready`, cancel the reservation. The running instance keeps running: ```bash theme={null} aws ec2 cancel-capacity-reservation --capacity-reservation-id ``` A reservation is billed at the On-Demand rate while it exists, whether or not an instance uses it, and it counts against the instance quota. ### Test With a Smaller GPU Instance To confirm that GPU nodes join the cluster on a new version when your usual GPU type has no capacity, start a smaller type that uses the same EKS-optimized NVIDIA AMI, such as `g6.xlarge`. A smaller type has fewer GPUs, so switch back before you run production workloads. Change the instance type only while the node group has **zero** nodes. With nodes running, a launch template change replaces them, and the replacement needs capacity for the new type alongside the old one. The safe order is: 1. Scale the node group to zero. 2. Change the instance type. 3. Scale up and test. 4. Scale back to zero. 5. Change the instance type back. ### Work Safely With Terraform * **Confirm the environment before every plan.** If you keep several environments in one configuration, check that the backend and the variable file match the cluster you mean to change: ```bash theme={null} terraform state show aws_eks_cluster. | grep -E '^\s+name\s' ``` Running a plan with another environment's variable file proposes replacing the cluster and everything attached to it. * **Save plans and apply the saved file.** `terraform plan -out=` followed by `terraform apply ` applies exactly what you reviewed. Note that `apply` with a saved plan does not ask for confirmation. * **Keep saved plans out of version control.** Plan files can contain secret values in plain text. Write them to a dedicated directory, such as `plans/`, and add it to `.gitignore`. * **Read the summary line.** `Plan: X to add, Y to change, Z to destroy.` An upgrade should show `0 to destroy`. Stop on any destroy or replacement you did not expect. * **A saved plan can go stale.** If something changes between `plan` and `apply`, Terraform reports `Saved plan is stale`. Run the plan again, review it, and apply the new file. * **Do not run an apply while an EKS update is in progress.** Wait until the cluster and node groups are `ACTIVE`. Otherwise the apply fails with `ResourceInUseException`, or it starts a second update that repeats the failure that stopped the first one. ### Allow Time for First Image Pulls New nodes pull every container image they need at once. On a network with limited egress to Amazon ECR, the first start of a node can take 20 to 30 minutes, and `ImagePullBackOff` events in that window are normal while pulls are retried. GPU nodes take the longest because their images are large. Allow for this in your maintenance window, and don't treat a slow first start as a failure until the pulls have stopped retrying. ### Download Tools Into a Temporary Directory `curl -O` writes into the current directory and fails with `Permission denied` if you can't write there. Download into a directory created with `mktemp -d`, as in [Upgrade kubectl](#upgrade-kubectl), so the commands work from any location. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.